Purushottam Sinha created FLINK-41604:
-----------------------------------------
Summary: [db2] Db2 CDC source cannot restore from a checkpoint or
savepoint taken after it read a change: ClassCastException in
Db2OffsetContext$Loader
Key: FLINK-41604
URL: https://issues.apache.org/jira/browse/FLINK-41604
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.7.0, cdc-3.5.0
Reporter: Purushottam Sinha
h3. Problem
A Db2 CDC job ({{db2-cdc}}, incremental source) cannot restore from a
checkpoint or savepoint taken after it has read at least one change event.
Every restore attempt fails, so the job neither resumes after
stop-with-savepoint nor recovers from a checkpoint after a failure; it keeps
restarting:
{noformat}
java.lang.ClassCastException: class java.lang.String cannot be cast to class
java.lang.Long
at
io.debezium.connector.db2.Db2OffsetContext$Loader.load(Db2OffsetContext.java:152)
at
org.apache.flink.cdc.connectors.db2.source.fetch.Db2SourceFetchTaskContext.loadStartingOffsetState(Db2SourceFetchTaskContext.java:170)
at
org.apache.flink.cdc.connectors.db2.source.fetch.Db2SourceFetchTaskContext.configure(Db2SourceFetchTaskContext.java:113)
at
org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceStreamFetcher.submitTask(IncrementalSourceStreamFetcher.java:88)
{noformat}
A checkpoint or savepoint taken before any change is read (right after the
snapshot) restores fine.
h3. Cause
The Db2 incremental source stores the stream position as strings, including
{{event_serial_no}} ({{db2/source/offset/LsnOffset.java}}):
{code:java}
offsetMap.put(EVENT_SERIAL_NO_KEY, String.valueOf(eventSerialNo));
{code}
On restore, {{Db2SourceFetchTaskContext#loadStartingOffsetState}} passes that
map to Debezium's (1.9.8.Final) {{Db2OffsetContext.Loader}}, which expects a
{{Long}}:
{code:java}
Long eventSerialNo = ((Long) offset.get(EVENT_SERIAL_NO_KEY)); //
Db2OffsetContext.java:152
{code}
Positions saved before any change was read carry no {{event_serial_no}}, so the
cast gets {{null}}; that is why the snapshot and the first start work.
FLINK-36086 hit the same mismatch for SQL Server and added a patched
{{io.debezium.connector.sqlserver.SqlServerOffsetContext}} whose loader parses
the string. {{flink-connector-db2-cdc}} patches {{Db2Connection}} and
{{Db2StreamingChangeEventSource}} but not {{Db2OffsetContext}}. The Db2
{{LsnOffset}} has stored {{event_serial_no}} as a string since the incremental
source was added (FLINK-34842, first released in cdc-3.2.0); master is
unchanged.
{{Db2SourceITCase}}'s stream-phase failover tests do not catch it: they trigger
the failover about 200 ms after making changes, with 1 s checkpoints, so the
restored checkpoint predates the change events.
h3. Reproduction
Apache Flink's official {{flink:1.20.3-java17}} image with
{{flink-sql-connector-db2-cdc-3.5.0.jar}} and {{com.ibm.db2:jcc:11.5.9.0}} in
{{lib/}}; {{icr.io/db2_community/db2:11.5.9.0}} with ASN Capture set up by the
{{db2_server}} scripts of this repository's Db2 tests (table registered with
{{ASNCDC.ADDTABLE}}); checkpoints every 2 s.
{code:sql}
CREATE TABLE src (ID INT NOT NULL, ITEM STRING, QTY INT, PRIMARY KEY (ID) NOT
ENFORCED) WITH (
'connector' = 'db2-cdc', 'hostname' = '...', 'port' = '50000', 'username' =
'db2inst1', 'password' = '...',
'database-name' = 'cdc', 'table-name' = 'DB2INST1.ORDERS');
CREATE TABLE out_rows (ID INT, ITEM STRING, QTY INT) WITH ('connector' =
'print');
INSERT INTO out_rows SELECT * FROM src;
{code}
# Start the job; the 3 snapshot rows are printed.
# Insert, update and delete rows in Db2; the changes are printed.
# Stop the job with a savepoint, change more rows, start the job again from the
savepoint.
# The job restarts in a loop with the exception above and prints nothing.
If the job is stopped right after the snapshot instead (no change read), it
resumes and prints the later changes.
Reproduced with cdc-3.5.0 on Flink 1.20.3 and with cdc-3.7.0 ({{-2.2}} build)
on Flink 2.2.0.
h3. Proposed fix
Add a patched {{io.debezium.connector.db2.Db2OffsetContext}} to
{{flink-connector-db2-cdc}} whose loader accepts a string or a {{Long}} for
{{event_serial_no}}, as FLINK-36086 did for SQL Server; parsing on read also
restores positions saved by current releases. Add a test that restores from a
checkpoint or savepoint taken after a change event was read.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)