raminqaf opened a new pull request, #29760:
URL: https://github.com/apache/flink/pull/29760
## What is the purpose of the change
The `raw` format decoded a `VARIANT` message with `raw.charset` into a
`String` and parsed that text. With `raw.line-delimiter` it decoded the whole
message, split it, then encoded each piece and decoded it again. `PARSE_JSON`
parses UTF-8 bytes directly, and the two paths disagreed on invalid UTF-8:
| Input | Before
| After
|
|--------------------------------------------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `{"a":"<0xFF>"}`, with or without `raw.line-delimiter` | stored as
`{"a":"\uFFFD"}` | `DeserializationException: Failed to deserialize
VARIANT type. The received data is not a valid JSON document.`, cause `Invalid
UTF-8 start byte 0xff`
|
| `VARIANT` column with `'raw.charset' = 'UTF-16'` | decoded as
UTF-16, then parsed | `ValidationException: The 'raw' format requires
'raw.charset' to be UTF-8 for a VARIANT column, but it is 'UTF-16'. To use
another charset, declare the column as STRING and convert it with PARSE_JSON or
JSON_STRING.` |
UTF-8 only follows RFC 8259 §8.1, which requires UTF-8 for JSON exchanged
between systems, and matches `PARSE_JSON`. JSON in another charset can still be
read through a `STRING` column with `raw.charset` and `PARSE_JSON`.
**Compatibility.** `VARIANT` support in the `raw` format is new in 2.4.0
(FLINK-40241) and not released yet. If this lands in 2.4.0 too, it breaks no
running job.
## Brief change log
- `RawFormatFactory` rejects a `VARIANT` column whose `raw.charset` is not
UTF-8, for both the source and the sink. The check runs when a query reads or
writes the table, before the first record.
- The `VARIANT` converter parses the message bytes with
`BinaryVariantInternalBuilder#parseJson(byte[], boolean)`.
- With `raw.line-delimiter`, a `VARIANT` message is split on the UTF-8 bytes
of the delimiter and never decoded. UTF-8 is self-synchronizing, so the
delimiter can only match at a character boundary. The split has to be on bytes
as well: decoding the whole message first would turn invalid bytes into U+FFFD
before the converter sees them. An empty delimiter keeps the old path, which
splits between characters.
- `raw.md`, en and zh: the `VARIANT` row says the charset must be UTF-8,
names the `STRING` workaround and says invalid UTF-8 is rejected.
## Verifying this change
This change added tests and can be verified as follows:
- `RawFormatFactoryTest#testVariantRequiresUtf8Charset`: a `VARIANT` column
with UTF-16 fails for the source and the sink, with the exact message.
- `RawFormatLineDelimiterTest#testDeserializeVariantRejectsInvalidUtf8`:
invalid UTF-8 fails without a delimiter and with `\n`, with the exact message
and cause.
- `RawFormatLineDelimiterTest#testDeserializeVariantWithMultiByteDelimiter`:
a multi-byte `→` delimiter splits non-ASCII JSON at the right places and drops
the trailing empty piece.
- The UTF-16 `VARIANT` round trip in `RawFormatSerDeSchemaTest` is removed,
because that charset is now rejected.
The first two tests fail on master.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes. A
`VARIANT` record is now parsed from its bytes, without decoding it to a
`String` first.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable. The `VARIANT` row
of the raw format docs is updated.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.316 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
