dev-donghwan commented on PR #4618: URL: https://github.com/apache/flink-cdc/pull/4618#issuecomment-6830995731
@yuxiqian While addressing the review, I found a case I'd like your opinion on before going further. With REPLICA IDENTITY DEFAULT, PostgreSQL doesn't send unchanged TOASTed values. Debezium 1.9.8 replaces them with `__debezium_unavailable_value` only for string and binary (bytes mode) columns, and this PR now fails fast on those in upsert mode. Arrays and binary values in base64/hex mode, however, are converted to null, which is indistinguishable from a real null in the deserializer. Throwing from the value converter doesn't help either, because `TableSchemaBuilder` catches converter exceptions and leaves the field null. Newer Debezium versions emit type-specific placeholders (`UnchangedToastedPlaceholder`). Options I see: 1. Keep this PR as is (the limitation is documented and covered by ITCases) and open a follow-up JIRA to handle these types, via a Debezium upgrade or a backport of the type-specific placeholders. 2. Reject at startup, in upsert mode, tables without REPLICA IDENTITY FULL that have array columns (or bytea with base64/hex mode). This is safe but also rejects tables whose arrays are never TOASTed. 3. Detect it in `PostgresStreamingChangeEventSource`, using the condition Debezium uses to fill the value from the old tuple (a TOASTed column in the new tuple whose old value is missing or null). A local prototype with pgoutput failed all 5 corrupting cases I tested (non-key UPDATE and PK change, text/array/base64 bytea) with no false positives under REPLICA IDENTITY FULL. I haven't checked other decoding plugins, the existing test suite with it, or the overhead, and it would need an upsert-mode flag on the shared streaming path. I'd lean towards 1, possibly with 3 as the follow-up. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
