[
https://issues.apache.org/jira/browse/FLINK-39495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-39495:
---------------------------------
Description:
FROM_CHANGELOG silently drops input rows when the operation code column
contains a value not present in the op_mapping (or not matching the default
mapping). This makes data loss invisible and very hard to debug — users have no
indication that rows are being discarded.
For example, if a CDC source produces an unexpected operation code like
"SNAPSHOT" or a typo like "INSRT", those rows are silently skipped with no
error or warning.
The fix changes the default behavior to throw a RuntimeException when an
unmapped operation code is encountered, making data issues visible immediately
rather than causing silent data loss.
Before (broken):
{code:java}
-- Row with op='UNKNOWN' is silently dropped — no error, no log
SELECT * FROM FROM_CHANGELOG(input => TABLE cdc_stream) {code}
After (fixed):
{code:java}
-- Row with op='UNKNOWN' throws:
-- "Received invalid op code 'UNKNOWN'. Defined op codes are: [INSERT,
UPDATE_BEFORE, UPDATE_AFTER, DELETE]. Failing as configured."
SELECT * FROM FROM_CHANGELOG(input => TABLE cdc_stream)
{code}
was:
FROM_CHANGELOG silently drops input rows when the operation code column
contains a value not present in the op_mapping (or not matching the default
mapping). This makes data loss invisible and very hard to debug — users have no
indication that rows are being discarded.
For example, if a CDC source produces an unexpected operation code like
"SNAPSHOT" or a typo like "INSRT", those rows are silently skipped with no
error or warning.
The fix changes the default behavior to throw a RuntimeException when an
unmapped operation code is encountered, making data issues visible immediately
rather than causing silent data loss.
Before
(broken):
{code:java}
-- Row with op='UNKNOWN' is silently dropped — no error, no log
SELECT * FROM FROM_CHANGELOG(input => TABLE cdc_stream) {code}
After
(fixed):
{code:java}
-- Row with op='UNKNOWN' throws:
-- "Received invalid op code 'UNKNOWN'. Defined op codes are: [INSERT,
UPDATE_BEFORE, UPDATE_AFTER, DELETE]. Failing as configured."
SELECT * FROM FROM_CHANGELOG(input => TABLE cdc_stream)
{code}
> FROM_CHANGELOG silently drops rows with unmapped operation codes
> -----------------------------------------------------------------
>
> Key: FLINK-39495
> URL: https://issues.apache.org/jira/browse/FLINK-39495
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Ramin Gharib
> Priority: Major
>
> FROM_CHANGELOG silently drops input rows when the operation code column
> contains a value not present in the op_mapping (or not matching the default
> mapping). This makes data loss invisible and very hard to debug — users have
> no indication that rows are being discarded.
> For example, if a CDC source produces an unexpected operation code like
> "SNAPSHOT" or a typo like "INSRT", those rows are silently skipped with no
> error or warning.
> The fix changes the default behavior to throw a RuntimeException when an
> unmapped operation code is encountered, making data issues visible
> immediately rather than causing silent data loss.
> Before (broken):
>
> {code:java}
> -- Row with op='UNKNOWN' is silently dropped — no error, no log
> SELECT * FROM FROM_CHANGELOG(input => TABLE cdc_stream) {code}
>
> After (fixed):
>
> {code:java}
> -- Row with op='UNKNOWN' throws:
> -- "Received invalid op code 'UNKNOWN'. Defined op codes are: [INSERT,
> UPDATE_BEFORE, UPDATE_AFTER, DELETE]. Failing as configured."
> SELECT * FROM FROM_CHANGELOG(input => TABLE cdc_stream)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
