[
https://issues.apache.org/jira/browse/HUDI-8949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan closed HUDI-8949.
-------------------------------------
Resolution: Fixed
https://github.com/apache/hudi/commit/1bdba3600a7facd0edc79fa2c5942a10f73ce8a8
> Hitting checkpoint validation failure for custom sources in HoodieStreamer
> --------------------------------------------------------------------------
>
> Key: HUDI-8949
> URL: https://issues.apache.org/jira/browse/HUDI-8949
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: writer-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.1
>
>
> https://issues.apache.org/jira/browse/HUDI-8917
>
> when we landed this, we added more stricter checkpoint version validaiton.
> but for non core sources are running into issue w/ the validation.
>
> When we tried HoodieStreamer to write table version 6
> {code:java}
> With table version 6 - Caused by: org.apache.hudi.exception.HoodieException:
> java.lang.IllegalStateException: Data source should return checkpoint version
> V1. The checkpoint resumed in the iteration is null, whose translated version
> is null. The checkpoint returned after the iteration
> StreamerCheckpointV2{checkpointKey='1'}.
> at
> org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:142)
> at
> org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:104)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.lang.IllegalStateException: Data source should return
> checkpoint version V1. The checkpoint resumed in the iteration is null, whose
> translated version is null. The checkpoint returned after the iteration
> StreamerCheckpointV2{checkpointKey='1'}.
> at
> org.apache.hudi.utilities.sources.Source.assertCheckpointVersion(Source.java:156)
> at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:171)
> at
> org.apache.hudi.utilities.streamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:175)
> at
> org.apache.hudi.utilities.streamer.StreamSync.fetchNextBatchFromSource(StreamSync.java:710)
> at
> org.apache.hudi.utilities.streamer.StreamSync.fetchFromSourceAndPrepareRecords(StreamSync.java:582)
> at
> org.apache.hudi.utilities.streamer.StreamSync.readFromSource(StreamSync.java:554)
> at
> org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:464)
> at
> org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.upsert(HoodieDeltaStreamerWrapper.java:48)
> at
> org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.bulkInsert(HoodieDeltaStreamerWrapper.java:57)
> at
> org.apache.hudi.integ.testsuite.HoodieInlineTestSuiteWriter.bulkInsert(HoodieInlineTestSuiteWriter.java:150)
> at
> org.apache.hudi.integ.testsuite.dag.nodes.BulkInsertNode.ingest(BulkInsertNode.java:40)
> at
> org.apache.hudi.integ.testsuite.dag.nodes.InsertNode.execute(InsertNode.java:53)
> at
> org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:135)
> ... 6 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
