Limess opened a new issue #4146:
URL: https://github.com/apache/hudi/issues/4146
**Describe the problem you faced**
When running deltastreamer with a custom checkpoint value, e.g.
`--checkpoint 0`, the commit is skipped if the resulting checkpoint value
matches the checkpoint value of the previous commit:
```
21/11/25 13:42:50 INFO DeltaSync: Checkpoint to resume from :
Option{val=1637847362361}
21/11/25 13:42:50 INFO DFSPathSelector: Root path =>
/Users/charlie.briggs/Projects/signal/pipeline/pipeline-spark-jobs/experiments/schema_evolution/output/schema_evolution_parquet/
source limit => 9223372036854775807
21/11/25 13:42:50 INFO DeltaSync: No new data, source checkpoint has not
changed. Nothing to commit. Old checkpoint=(Option{val=1637847362361}). New
Checkpoint=(1637847362361)
```
This prevents re-reading a static dataset into the same Hudi Deltastreamer
using the checkpointing mechanism.
**To Reproduce**
Steps to reproduce the behavior:
1. Run Deltastreamer using the DFS datasource against a static parquet
datasource
2. Re-run deltastreamer against the same datasource with the same modified
timestamps, passing `--checkpoint 0` (or any checkpoint value < the previous
commit checkpoint)
3. Observe the commit is skipped
**Expected behavior**
With the DFS datasource, any files modified since the provided checkpoint
timestamp are re-read and upserted, and a commit is made, regardless of the
checkpoint timestamp or modified date of the files.
**Environment Description**
EMR 6.4.0
* Hudi version: 0.9.0
* Spark version :
3.1.2
* Hive version :
Hive 3.1.2
* Hadoop version :
Amazon 3.2.1
* Storage (HDFS/S3/GCS..) :
S3
* Running on Docker? (yes/no) :
no
**Additional context**
This is a minor issue, but causes some confusion - it is intuitive to me
that a checkpoint of `0` should re-read all data possible from the source
datasource.
If a file has changed in the source datasource since the previous commit,
this technique does seem to work as the commit is not skipped (as the
checkpoint has increased from the previously committed checkpoint)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
