Ramin Gharib created FLINK-40917:
------------------------------------
Summary: Deeply nested VARIANT values overflow the stack in
toJson(), CAST to STRING and the variant builder
Key: FLINK-40917
URL: https://issues.apache.org/jira/browse/FLINK-40917
Project: Flink
Issue Type: Bug
Components: API / Core
Reporter: Ramin Gharib
Assignee: Ramin Gharib
h3. Problem
Code that walks a \{{VARIANT}} recurses once per nesting level, so a deeply
nested value overflows the thread stack. A \{{StackOverflowError}} is an
\{{Error}}, so handlers that catch \{{Exception}} miss it.
\{{ExceptionUtils.isJvmFatalError}} does not treat it as fatal either. The task
fails over and restarts on the same record, without a message that says what
went wrong.
The limit depends on the stack size and on the JIT state. On a 1 MiB stack, the
default for task threads, about 2,000 levels overflow. Such a value takes only
about 20 KB, far below the 16 MiB size limit of a \{{VARIANT}}.
h3. Affected code
||Recursive walker||Used by||
|\{{JsonVariantFormatter}}|\{{Variant#toJson()}}, \{{Variant#toString()}},
\{{JSON_STRING}}, \{{JSON_OBJECT}}, the \{{json}} and \{{raw}} formats|
|\{{VariantCastUtils.renderNode}}|\{{CAST(v AS STRING)}} and printing results|
|\{{BinaryVariantInternalBuilder.appendVariantImpl}}|the object and array
builders of the public \{{VariantBuilder}}, and the cast of an \{{ARRAY}},
\{{MAP}} or \{{ROW}} to \{{VARIANT}}|
FLINK-40826 catches the overflow in the cast to \{{VARIANT}}, which fails with
\{{Cannot cast a value of type ARRAY<VARIANT> to VARIANT because it is nested
too deeply.}}, and \{{TRY_CAST}} returns \{{NULL}}. That covers one call site
only. The other walkers still throw a bare \{{StackOverflowError}}.
h3. Reproduce
{code:java}
// [[[...[1]...]]] with 20,000 levels, built without recursion
BinaryVariantInternalBuilder builder = new BinaryVariantInternalBuilder(false);
builder.appendInt(1);
for (int i = 0; i < 20_000; i++) {
builder.finishWritingArray(0, new ArrayList<>(List.of(0)));
}
BinaryVariant deep = builder.build();
deep.toJson(); // StackOverflowError
Variant.newBuilder().array().add(deep).build(); // StackOverflowError
{code}
In SQL, \{{JSON_STRING(v)}} and \{{CAST(v AS STRING)}} fail the same way for
such a value.
h3. Where deep values come from
{\{PARSE_JSON}}, \{{TRY_PARSE_JSON}} and the JSON format parse with Jackson,
which rejects JSON nested deeper than 1,000 levels by default. That is close to
the overflow limit, so the margin is thin. Other sources have no limit, such as
binary variants that a format reads as they are, and values that a user
function builds with \{{VariantBuilder}}.
h3. Options
# Make the walkers iterative, with an explicit stack. This removes the limit
and changes nothing for valid input. It needs the most code.
# Enforce a maximum nesting depth wherever a variant is built or read, well
below the stack limit, so that deeper values fail with a clear error. This is
simple, but a limit below 1,000 would reject JSON that \{{PARSE_JSON}} accepts
today.
# Catch \{{StackOverflowError}} at every call site, as the cast to \{{VARIANT}}
does. This is the least code, but it is fragile, and every new walker has to
remember it.
Option 1 fits best. The three walkers sit in core and in the table runtime, and
every query that uses \{{VARIANT}} reaches them. Once they are iterative, the
catch in \{{ToVariantConverter#convert}} can go.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
