yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802099864
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -376,14 +382,16 @@ private QueryContext(
List<HoodieInstant> archivedInstants,
List<HoodieInstant> activeInstants,
HoodieTimeline activeTimeline,
- @Nullable HoodieTimeline archivedTimeline) {
+ @Nullable HoodieTimeline archivedTimeline,
+ @Nullable String predicateFilter) {
this.startInstant = Option.ofNullable(startInstant);
this.endInstant = Option.ofNullable(endInstant);
this.archivedInstants = archivedInstants;
this.activeInstants = activeInstants;
this.activeTimeline = activeTimeline;
this.archivedTimeline = archivedTimeline;
this.instants = instants;
+ this.predicateFilter = Option.ofNullable(predicateFilter);
Review Comment:
- The `SnapshotLoadQuerySplitter` may change the end instant time and add
the predicate in the case of full scan (see the following code in
`HoodieIncrSource`). To pass the predicate around, the `QueryContext` instance
is passed to `SnapshotLoadQuerySplitter` for further processing before
returning a new `QueryContext` instance that can contain the predicate.
```
if (instantRange.isEmpty() || shouldFullScan) {
// snapshot query
Dataset<Row> snapshot = reader
.options(readOpts)
.option(DataSourceReadOptions.QUERY_TYPE().key(),
DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
.load(srcPath);
if (snapshotLoadQuerySplitter.isPresent()) {
queryContext =
snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext,
sourceProfileSupplier);
// update endTime/next checkpoint
completionTimeToStopAt =
queryContext.getEndInstant().orElse(queryContext.getLastInstant());
}
snapshot =
queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
source = snapshot
// add filtering so that only interested records are returned.
// completion time comparison uses ( , ], but when comparing start
time we need to use [, ]
.filter(String.format("%s >= '%s'",
HoodieRecord.COMMIT_TIME_METADATA_FIELD,
queryContext.getStartInstant().get()))
.filter(String.format("%s <= '%s'",
HoodieRecord.COMMIT_TIME_METADATA_FIELD,
queryContext.getEndInstant().orElse(queryContext.getLastInstant())))
.filter(String.format("%s IN ('%s')",
HoodieRecord.COMMIT_TIME_METADATA_FIELD,
String.join("','", queryContext.getInstants())));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
