codope opened a new issue, #20301: URL: https://github.com/apache/hudi/issues/20301
### Bug Description **What happened:** The config documentation (https://github.com/apache/hudi/blob/0202ca7e24c1662ece6ea4b7400168bac7d5dee7/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala#L119) says: > New data written with completion_time >= START_COMMIT are fetched out. The Spark DataSource incremental relations are built with `RangeType.OPEN_CLOSED` (`HoodieHadoopFsRelationFactory.scala`, the default `rangeType` of the incremental relation factories; [MergeOnReadIncrementalRelationV2.scala:55](https://github.com/apache/hudi/blob/0202ca7e24c1662ece6ea4b7400168bac7d5dee7/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala#L57)). So a commit whose completion time equals the begin value is **not** returned. Users who follow the doc and checkpoint "the last completion time I processed, then read from it" get the behavior they want only by accident. Users who expect the boundary commit back don't get it. **What you expected:** Change the doc to "completion_time > START_COMMIT" (exclusive), and mention that END_COMMIT is inclusive. Alternatively, change the code if inclusive was the intent. Either way the doc and the code should agree, since pipelines checkpoint on this value. **Steps to reproduce:** vidence (Hudi 1.2.1, COW table, commit 1 = bulk insert of 2,880,404 rows, commit 2 = upsert of 50 rows) | begin.instanttime | Rows returned | |---|---| | commit 1 completion time | 50 (only commit 2) | | commit 1 completion time − 1 ms | 2,880,404 (commits 1 and 2) | | commit 2 completion time | 0 | ### Environment **Hudi version:** master **Query engine:** Spark **Relevant configs:** ### Logs and Stack Trace _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
