raminqaf opened a new pull request, #29786:
URL: https://github.com/apache/flink/pull/29786
## What is the purpose of the change
Backport of #29760 to release-2.4. `VARIANT` support in the `raw` format
(FLINK-40241) and `raw.line-delimiter` (FLINK-39401) are on release-2.4 and not
released yet. Without this backport, 2.4.0 ships a `VARIANT` column that
accepts any charset and stores invalid UTF-8 as U+FFFD, and a line delimiter
that splits every column type. Adding these restrictions after 2.4.0 would
break running jobs.
## Brief change log
- Cherry-pick of 2180171f1e2, 33a8167b49a and c59810c04f8 with `-x`
- 33a8167b49a had one conflict, in the note at the end of `raw.md`, en and
zh: release-2.4 does not have the sentence that FLINK-41298 added. The new note
replaces that paragraph, so every file this backport touches is identical to
master.
## Verifying this change
`RawFormatFactoryTest`, `RawFormatLineDelimiterTest`,
`RawFormatSerDeSchemaTest` and `BinaryVariantTest` pass on this branch.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes. `RawFormatOptions` gains `VARIANT_ENCODING` and the
`VariantEncoding` enum, as in #29760.
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes. A
`VARIANT` record is parsed from its bytes, and a UTF-8 string message is split
without decoding it, as in #29760.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes, the
`raw.variant-encoding` option.
- If yes, how is the feature documented? docs, on the raw format page.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.316 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
