vbhanuchander-lang commented on issue #8961:
URL: https://github.com/apache/paimon/issues/8961#issuecomment-5247346734
I traced this through the code and can confirm the diagnosis, including the
asymmetry with the Avro path.
`DebeziumJsonRecordParser` inherits
`AbstractJsonRecordParser.extractPrimaryKeys()`, which reads only the value:
```java
// AbstractJsonRecordParser.java:114
protected List<String> extractPrimaryKeys() {
ArrayNode pkNames = getNodeAs(root, primaryField(), ArrayNode.class);
if (pkNames == null) {
return Collections.emptyList();
}
...
```
while the Avro parser already does the right thing with the Kafka key:
```java
// DebeziumAvroRecordParser.java:123
protected List<String> extractPrimaryKeys() {
if (keyRecord == null) {
return Collections.emptyList();
}
Schema keySchema = sanitizedSchema(keyRecord.getSchema());
return
keySchema.getFields().stream().map(Schema.Field::name).collect(...);
}
```
So the two Debezium formats disagree on where primary keys come from, and
`KafkaDebeziumJsonDeserializationSchema` is already putting the parsed key into
`CdcSourceRecord` — the JSON parser simply never looks at it. `currentRecord`
is available on `AbstractRecordParser`, so the fallback you propose needs no
plumbing changes.
One addition to your suggested behaviour, if it's welcome: the key needs the
same schema-enabled / schema-disabled distinction the value already gets via
`hasSchema`, since a schema-enabled key nests the fields under
`schema.fields[].field` while a schema-disabled key is a flat object. Handling
only one shape would fix half the reports.
@TKilome you ticked "willing to submit a PR" — are you still working on
this? I don't want to duplicate your effort. If you haven't started, I'm happy
to send a PR implementing exactly the four points you listed, with tests for
both key shapes and for the `pkNames`-present case so the existing behaviour
stays covered. I'll wait a couple of days for a reply before opening anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
