The GitHub Actions job "Flink CDC CI" on 
flink-cdc.git/FLINK-38647-postgres-changelog-mode-and-serializer has failed.
Run started by GitHub user cansakiroglu (triggered by leonardBang).

Head commit for run:
48d85f2507ce95280f58522b653a24261f1bdeb4 / Mehmet Can Şakiroğlu 
<[email protected]>
Support changelog-mode=upsert + null-tolerant DataChangeEventSerializer

FLINK-38647 reports a NullPointerException from the Postgres Pipeline source
when a captured table has REPLICA IDENTITY DEFAULT and an UPDATE arrives that
does not change the primary key. Root cause: PostgresDataSource hardcodes
DebeziumChangelogMode.ALL, which makes the deserializer extract a before-image
that is null under DEFAULT replication, NPE'ing in
DebeziumSchemaDataTypeInference.inferStruct.

This change fixes the issue from two angles:

1. Connector side — expose 'changelog-mode' YAML option on the Postgres
   Pipeline connector (mirrors the legacy SQL connector). Accepts 'all'
   (default, current behaviour) or 'upsert'. In upsert mode the source emits
   UPDATE events with before == null and only after populated, so the
   pipeline runs cleanly under REPLICA IDENTITY DEFAULT without requiring
   FULL (which roughly multiplies WAL volume per UPDATE by column count).

2. Runtime side — make DataChangeEventSerializer null-tolerant. Three changes:
   - copy() null-guards each recordDataSerializer.copy() call so the chained
     CopyingChainingOutput.pushToOperator path stops NPE'ing on null
     before/after.
   - serialize() writes 2 leading boolean presence flags before conditionally
     writing each record. Required because the previous serialize() already
     skipped writing null fields, but deserialize() always tried to read two
     records back — the wire format itself couldn't roundtrip a null-before
     UPDATE.
   - deserialize() reads the 2 flag bytes and skips the corresponding read
     when absent.

Wire format change is symmetric but breaking vs current 3.6.x. Production
checkpoints/savepoints written by older versions cannot be restored by the
new code without a snapshot-version-aware deserialize path; see the PR
description for the proposed compat strategy (bumping
DataChangeEventSerializerSnapshot.CURRENT_VERSION to 2 and reading old
format when version == 1). Happy to add that in a follow-up commit on this
PR — wanted reviewers' opinion on the strategy first.

Signed-off-by: Mehmet Can Şakiroğlu <[email protected]>

Report URL: https://github.com/apache/flink-cdc/actions/runs/26956727621

With regards,
GitHub Actions via GitBox

Reply via email to