sivabalan narayanan created HUDI-8949:
-----------------------------------------
Summary: Hitting checkpoint validation failure for custom sources
in HoodieStreamer
Key: HUDI-8949
URL: https://issues.apache.org/jira/browse/HUDI-8949
Project: Apache Hudi
Issue Type: Improvement
Components: writer-core
Reporter: sivabalan narayanan
https://issues.apache.org/jira/browse/HUDI-8917
when we landed this, we added more stricter checkpoint version validaiton.
but for non core sources are running into issue w/ the validation.
When we tried HoodieStreamer to write table version 6
{code:java}
With table version 6 - Caused by: org.apache.hudi.exception.HoodieException:
java.lang.IllegalStateException: Data source should return checkpoint version
V1. The checkpoint resumed in the iteration is null, whose translated version
is null. The checkpoint returned after the iteration
StreamerCheckpointV2{checkpointKey='1'}.
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:142)
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:104)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalStateException: Data source should return
checkpoint version V1. The checkpoint resumed in the iteration is null, whose
translated version is null. The checkpoint returned after the iteration
StreamerCheckpointV2{checkpointKey='1'}.
at
org.apache.hudi.utilities.sources.Source.assertCheckpointVersion(Source.java:156)
at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:171)
at
org.apache.hudi.utilities.streamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:175)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:710)
at
org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:582)
at
org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:554)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:464)
at
org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.upsert(HoodieDeltaStreamerWrapper.java:48)
at
org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.bulkInsert(HoodieDeltaStreamerWrapper.java:57)
at
org.apache.hudi.integ.testsuite.HoodieInlineTestSuiteWriter.bulkInsert(HoodieInlineTestSuiteWriter.java:150)
at
org.apache.hudi.integ.testsuite.dag.nodes.BulkInsertNode.ingest(BulkInsertNode.java:40)
at
org.apache.hudi.integ.testsuite.dag.nodes.InsertNode.execute(InsertNode.java:53)
at
org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:135)
... 6 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
