raminqaf opened a new pull request, #29743:
URL: https://github.com/apache/flink/pull/29743
## What is the purpose of the change
`PARSE_JSON` read the first JSON value of its input and dropped the rest
without an error. The `raw` format's `VARIANT` column uses the same parser, so
a message holding two newline-delimited JSON documents kept only the first.
| Input |
Before | After
|
|-------------------------------------------------------------------------|--------------------|-----------------------------------------------------------------------------------------------------------------|
| `PARSE_JSON('{"a":1} xyz')` |
`{"a":1}` | `TableRuntimeException: Failed to parse JSON string:
{"a":1} xyz` |
| `PARSE_JSON('{"a":1}{"a":2}')` |
`{"a":1}` | `TableRuntimeException: Failed to parse JSON string:
{"a":1}{"a":2}` |
| `TRY_PARSE_JSON('{"a":1}{"a":2}')` |
`{"a":1}` | `NULL`
|
| raw `VARIANT` message `{"id":1}\n{"id":2}` without `raw.line-delimiter` |
one row `{"id":1}` | `DeserializationException: Failed to deserialize VARIANT
type. The received data is not a valid JSON document.` |
| `PARSE_JSON(' {"a":1} \n')` |
`{"a":1}` | `{"a":1}`, unchanged
|
**Compatibility.** `PARSE_JSON` has shipped since 2.1.0, so input that
parsed before now fails. FLINK-40225 declined the same check for `IS JSON` and
`JSON_VALUE`. This case differs because `PARSE_JSON` stores what it parsed, so
the dropped text is lost data rather than a looser validity check. The raw
format's `VARIANT` support is new in 2.4.0, so making it strict breaks no
running job.
## Brief change log
- `BinaryVariantInternalBuilder#parseJson(String, boolean)` and
`#parseJson(byte[], boolean)` read one more token after the value and throw a
`JsonParseException` unless it is the end of the input. `parseJson(JsonParser,
boolean)` stays as it is, because its caller owns the parser.
- Docs for `PARSE_JSON`, `TRY_PARSE_JSON` and the raw format's `VARIANT`
type, en and zh.
## Verifying this change
This change added tests and can be verified as follows:
- `BinaryVariantInternalBuilderTest`: both overloads reject trailing
garbage, a stray `]` and a second document. Whitespace around the value still
parses.
- `JsonFunctionsITCase`: for a second document, `PARSE_JSON` fails and
`TRY_PARSE_JSON` returns NULL.
- `RawFormatLineDelimiterTest`: a raw `VARIANT` message with two documents
and no delimiter fails.
The new tests fail without the fix.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes. One
extra `nextToken()` per parsed value, which on valid input only skips trailing
whitespace.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable. The `PARSE_JSON`,
`TRY_PARSE_JSON` and raw format docs now state that the input must be exactly
one JSON value.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.316 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
