danny0405 commented on code in PR #8876:
URL: https://github.com/apache/hudi/pull/8876#discussion_r1229270268
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/IncrementalInputSplits.java:
##########
@@ -596,21 +599,22 @@ public List<HoodieInstant> filterInstantsWithRange(
/**
* Filters out the unnecessary instants as per user specified configs.
*
- * @param timeline The timeline
+ * @param completedTimeline original completed timeline
+ * @param metaClient
*
* @return the filtered timeline
*/
@VisibleForTesting
- public HoodieTimeline filterInstantsAsPerUserConfigs(HoodieTimeline
timeline) {
- final HoodieTimeline oriTimeline = timeline;
+ public HoodieTimeline filterInstantsAsPerUserConfigs(final HoodieTimeline
completedTimeline, HoodieTableMetaClient metaClient) {
+ HoodieTimeline filteredTimeline = completedTimeline;
if (this.skipCompaction) {
// the compaction commit uses 'commit' as action which is tricky
- timeline = timeline.filter(instant ->
!instant.getAction().equals(HoodieTimeline.COMMIT_ACTION));
+ filteredTimeline = filteredTimeline.filter(instant ->
!instant.getAction().equals(HoodieTimeline.COMMIT_ACTION));
}
if (this.skipClustering) {
- timeline = timeline.filter(instant ->
!ClusteringUtil.isClusteringInstant(instant, oriTimeline));
+ filteredTimeline = filteredTimeline.filter(instant ->
!ClusteringUtil.isClusteringInstant(instant, completedTimeline));
}
- return timeline;
+ return handleHollowCommitIfNeeded(filteredTimeline, metaClient,
hollowCommitHandling);
Review Comment:
We can defer the changes for Flink to release 1.0.0, there are rare use
cases for flink batch incremental queries, we can fix it when we have more
clear semantics on timestamp management of timeline.
For Spark source, it is regarded as a temporal fix i think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
