dev-donghwan commented on PR #4437:
URL: https://github.com/apache/flink-cdc/pull/4437#issuecomment-4921463823
@cansakiroglu
Thank you for this PR.
This PR resolves the source part, but the sink part is still broken.
Case:
1. flink-cdc pipeline -> Paimon sink (`changelog-mode: upsert`)
2. the Paimon sink calls `PaimonWriterHelper.convertEventToFullGenericRows()`
3. the UPDATE branch requires `before` -> NPE, and the job fails permanently
```
SinkWrapperException: Failed to handle event ...
DataChangeEvent{tableId=..., before=null, ..., op=UPDATE}
Caused by: java.lang.NullPointerException:
Cannot invoke "org.apache.flink.cdc.common.data.RecordData.getArity()"
because "recordData" is null
at PaimonWriterHelper.convertRecordDataToGenericRow(...)
at PaimonWriterHelper.convertEventToFullGenericRows(...) // UPDATE
branch reads before() unconditionally
```
The reason is that a PostgreSQL upsert event does not have `before`, only
`after` — the same failure pattern FLINK-39684 fixed for REPLACE in the same
method.
We verified a one-line guard end-to-end (PG -> Paimon, DML + schema
evolution):
```java
case UPDATE:
if (hasPrimaryKey && dataChangeEvent.before() != null) { // skip
UPDATE_BEFORE when there is nothing to build it from
...
```
For primary-key tables this is safe: `+U`-only and `-U`+`+U` converge to the
same merged state, and PK-changing updates never arrive as UPDATE (Debezium
decomposes them into DELETE + CREATE). Repro test:
`DataChangeEvent.updateEvent(tableId, null, afterData)` in
`PaimonWriterHelperTest` — every existing test passes a non-null `before`.
Note: the Kafka sink has the same pattern — `CanalJsonSerializationSchema`
and `DebeziumJsonSerializationSchema` read `before()` for UPDATE (and REPLACE)
too.
Would you prefer to include the sink-side null-tolerance in this PR, or
should we file a separate JIRA as a follow-up to FLINK-39684? Happy to
contribute either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
