yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1801755856
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +174,25 @@ public Pair<Option<Dataset<Row>>, String>
fetchNextBatch(Option<String> lastCkpt
IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
(containsConfigProperty(props,
HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
- getStringWithAltKeys(props,
HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+ getStringWithAltKeys(props,
HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
: null;
if (readLatestOnMissingCkpt) {
missingCheckpointStrategy =
IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
}
- // Use begin Instant if set and non-empty
- Option<String> beginInstant =
- lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty()
: lastCkptStr : Option.empty();
+ IncrementalQueryAnalyzer analyzer =
IncrSourceHelper.getIncrementalQueryAnalyzer(
+ sparkContext, srcPath, lastCkptStr, missingCheckpointStrategy,
+ getIntWithAltKeys(props,
HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH),
+ getLatestSourceProfile());
Review Comment:
The `IncrementalQueryAnalyzer` scans the archived timeline if the
lastCkptStr falls into the archived timeline range, especially the first
incremental pull without the last checkpoint, and the number of instants
included in the predicate can be huge. So we should see if this part can be
improved.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -277,6 +331,23 @@ public static Dataset<Row> coalesceOrRepartition(Dataset
dataset, int numPartiti
return dataset;
}
+ /**
+ * Copy IncrementalQueryAnalyzer.QueryContext with info from
CheckpointWithPredicates updated
+ * */
+ public static QueryContext withUpdatedCheckpoint(
Review Comment:
```suggestion
public static QueryContext withUpdatedCheckpointAndPredicate(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
