Yanjia Gary Li created HUDI-644: ----------------------------------- Summary: Enable to retrieve checkpoint from previous commits in Delta Streamer Key: HUDI-644 URL: https://issues.apache.org/jira/browse/HUDI-644 Project: Apache Hudi (incubating) Issue Type: Improvement Components: DeltaStreamer Reporter: Yanjia Gary Li Assignee: Yanjia Gary Li
This ticket is to resolve the following problem: The user is using a homebrew Spark data source to read new data and write to Hudi table The user would like to migrate to Delta Streamer But the Delta Streamer only checks the last commit metadata, if there is no checkpoint info, then the Delta Streamer will use the default. For Kafka source, it is LATEST. The user would like to run the homebrew Spark data source reader and Delta Streamer in parallel to prevent data loss, but the Spark data source writer will make commit without checkpoint info, which will reset the delta streamer. So if we have an option to allow the user to retrieve the checkpoint from previous commits instead of the latest commit would be helpful for the migration. -- This message was sent by Atlassian Jira (v8.3.4#803005)