[
https://issues.apache.org/jira/browse/FLINK-40917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129654#comment-18129654
]
Mika Naylor commented on FLINK-40917:
-------------------------------------
Hey [~raminqaf], just wanted to follow up on this issue and ask whether this is
still being worked on?
> Deeply nested VARIANT values overflow the stack in toJson(), CAST to STRING
> and the variant builder
> ---------------------------------------------------------------------------------------------------
>
> Key: FLINK-40917
> URL: https://issues.apache.org/jira/browse/FLINK-40917
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
>
> h3. Problem
> Code that walks a \{{VARIANT}} recurses once per nesting level, so a deeply
> nested value overflows the thread stack. A \{{StackOverflowError}} is an
> \{{Error}}, so handlers that catch \{{Exception}} miss it.
> \{{ExceptionUtils.isJvmFatalError}} does not treat it as fatal either. The
> task fails over and restarts on the same record, without a message that says
> what went wrong.
> The limit depends on the stack size and on the JIT state. On a 1 MiB stack,
> the default for task threads, about 2,000 levels overflow. Such a value takes
> only about 20 KB, far below the 16 MiB size limit of a \{{VARIANT}}.
> h3. Affected code
> ||Recursive walker||Used by||
> |\{{JsonVariantFormatter}}|\{{Variant#toJson()}}, \{{Variant#toString()}},
> \{{JSON_STRING}}, \{{JSON_OBJECT}}, the \{{json}} and \{{raw}} formats|
> |\{{VariantCastUtils.renderNode}}|\{{CAST(v AS STRING)}} and printing results|
> |\{{BinaryVariantInternalBuilder.appendVariantImpl}}|the object and array
> builders of the public \{{VariantBuilder}}, and the cast of an \{{ARRAY}},
> \{{MAP}} or \{{ROW}} to \{{VARIANT}}|
> FLINK-40826 catches the overflow in the cast to \{{VARIANT}}, which fails
> with \{{Cannot cast a value of type ARRAY<VARIANT> to VARIANT because it is
> nested too deeply.}}, and \{{TRY_CAST}} returns \{{NULL}}. That covers one
> call site only. The other walkers still throw a bare \{{StackOverflowError}}.
> h3. Reproduce
> {code:java}
> // [[[...[1]...]]] with 20,000 levels, built without recursion
> BinaryVariantInternalBuilder builder = new
> BinaryVariantInternalBuilder(false);
> builder.appendInt(1);
> for (int i = 0; i < 20_000; i++) {
> builder.finishWritingArray(0, new ArrayList<>(List.of(0)));
> }
> BinaryVariant deep = builder.build();
> deep.toJson(); // StackOverflowError
> Variant.newBuilder().array().add(deep).build(); // StackOverflowError
> {code}
> In SQL, \{{JSON_STRING(v)}} and \{{CAST(v AS STRING)}} fail the same way for
> such a value.
> h3. Where deep values come from
> {\{PARSE_JSON}}, \{{TRY_PARSE_JSON}} and the JSON format parse with Jackson,
> which rejects JSON nested deeper than 1,000 levels by default. That is close
> to the overflow limit, so the margin is thin. Other sources have no limit,
> such as binary variants that a format reads as they are, and values that a
> user function builds with \{{VariantBuilder}}.
> h3. Options
> # Make the walkers iterative, with an explicit stack. This removes the limit
> and changes nothing for valid input. It needs the most code.
> # Enforce a maximum nesting depth wherever a variant is built or read, well
> below the stack limit, so that deeper values fail with a clear error. This is
> simple, but a limit below 1,000 would reject JSON that \{{PARSE_JSON}}
> accepts today.
> # Catch \{{StackOverflowError}} at every call site, as the cast to
> \{{VARIANT}} does. This is the least code, but it is fragile, and every new
> walker has to remember it.
> Option 1 fits best. The three walkers sit in core and in the table runtime,
> and every query that uses \{{VARIANT}} reaches them. Once they are iterative,
> the catch in \{{ToVariantConverter#convert}} can go.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
