moggaa opened a new pull request, #17653: URL: https://github.com/apache/iceberg/pull/17653
Closes #17652 `Struct.get(Field)` substitutes the schema default value when the stored value is null, so an explicit `NULL` in the source is silently written to the Iceberg table as the column default. Full analysis, reproduction, and ecosystem precedent are in #17652. ### Changes - **Bundled SMTs** (`DebeziumTransform`, `KafkaMetadataTransform`, `CopyValue`): copy fields with `getWithoutDefault` so the stored null survives the copy, matching Debezium's own `ExtractNewRecordState`. This is not an observable behavior change on the default path: the output schema still carries the `defaultValue`, so any consumer reading with `get()` — including today's `RecordConverter` — receives exactly the same values as before. The new SMT tests assert both halves (the stored null and the retained schema default). - **`IcebergSinkConfig`**: new option `iceberg.tables.replace-null-with-default`, default `true`, which preserves the current behavior. Set to `false` to keep explicit nulls. - **`RecordConverter`**: struct field reads — including the variant conversion path — go through a shared helper gated by the option. - **`RecordUtils` / `SinkWriter`**: route-field extraction is gated by the same option, so that with the option disabled an explicitly-null route field is skipped like any other null route value, keeping the option's semantics uniform across writes and routing. - **Docs**: config table row plus a note on the routing behavior and on the JSON converter's own `replace.null.with.default` setting. ### Testing Unit tests cover the transform, converter (struct and variant), routing, and config-default paths; each new behavior is parameterized over both option values. All `kafka-connect` module tests pass. --- AI tooling was used for research and drafting; the analysis, design decisions, and scope are my own, and every claim was verified against source code (Kafka, Debezium, Confluent JDBC sink). The fix has also been validated in a production deployment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
