slachiewicz opened a new pull request, #348:
URL: https://github.com/apache/flink-connector-kafka/pull/348
## What is the purpose of the change
A `kafka` or `dynamic-kafka` table with `key.fields`, the default
`'value.fields-include' = 'ALL'` and a metadata column from the value format
(such as `value.source.timestamp` from `debezium-json`) gets NULL in that
column. The value format's metadata columns were placed after
`keyProjection.length + valueProjection.length`, which counts the key fields
twice when the value also contains them. They now start after the physical
columns. With `EXCEPT_KEY`, or without key fields, the two counts are equal, so
nothing changes there.
The same code is on `v5.0` (both sources) and `v4.0` (`KafkaDynamicSource`);
I will open backports once this is merged.
## Brief change log
- `KafkaDynamicSource` and `DynamicKafkaTableSource` start the value
format's metadata columns at the physical column count
## Verifying this change
This change added tests and can be verified as follows:
- `KafkaTableITCase#testValueFormatMetadataWithKeyFields` and
`DynamicKafkaTableITCase#testValueFormatMetadataWithKeyFields`, for `ALL` and
`EXCEPT_KEY`: write the key and a Debezium value as raw bytes, read them back
with a key column, a second physical column and two value-format metadata
columns. On `main` the `ALL` case returns NULL metadata, as in the ticket
## Does this pull request potentially affect one of the following parts:
- Dependencies: no
- The public API: no
- Checkpointed state, its serializers, or exactly-once delivery: no
- The per-record code paths: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
- If the docs changed, are both `docs/content` and `docs/content.zh`
updated? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]