Ramin Gharib created FLINK-41087:
------------------------------------
Summary: TRY_PARSE_JSON builds the PARSE_JSON error for every row
it rejects
Key: FLINK-41087
URL: https://issues.apache.org/jira/browse/FLINK-41087
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner, Table SQL / Runtime
Affects Versions: 2.4.0
Reporter: Ramin Gharib
h3. Problem
Since FLINK-40996, {{PARSE_JSON}} and {{TRY_PARSE_JSON}} share a generated
method that calls {{SqlJsonUtils#parseJsonToVariant}}. When parsing fails, that
method scans the whole input for invalid UTF-8, formats the input into a
message and creates a {{TableRuntimeException}}. The generated code stores it.
{{TRY_PARSE_JSON}} then drops it and returns NULL.
Before FLINK-40996, {{TryParseJsonFunction}} returned NULL right after the
parser failed. Each row that {{TRY_PARSE_JSON}} rejects now also pays for:
* a second exception with its stack trace,
* a scan of the input bytes,
* a copy of the input into the message string.
The results do not change. {{TRY_PARSE_JSON}} is meant for input that may not
be valid JSON, so this can hit a large share of the rows.
h3. Proposal
Let the shared method keep the exception that the parser throws. Build the
{{TableRuntimeException}} with the UTF-8 diagnostics only where {{PARSE_JSON}}
rethrows, for example through a {{SqlJsonUtils}} method that takes the input
and the cause. {{PARSE_JSON}} keeps its messages, such as {{Failed to parse
JSON string: Invalid UTF-8 byte at index 1 of 3.}}. A {{TRY_PARSE_JSON}} that
shares the parse with a {{PARSE_JSON}} keeps returning NULL.
h3. Tests
* The existing {{PARSE_JSON}} error tests in {{JsonFunctionsITCase}} pass
unchanged.
* {{JsonParseReuseTest}}: {{PARSE_JSON}} and {{TRY_PARSE_JSON}} on the same
invalid input still parse once and still fail with {{Failed to parse JSON
string}}.
* {{JsonParseReuseTest}}: the generated code of a query with only
{{TRY_PARSE_JSON}} does not build the error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
