yihua commented on code in PR #6664:
URL: https://github.com/apache/hudi/pull/6664#discussion_r983713186
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -291,31 +292,23 @@ private void doRefresh() {
validate(activeTimeline, queryInstant);
- if (tableType.equals(HoodieTableType.MERGE_ON_READ) &&
queryType.equals(HoodieTableQueryType.SNAPSHOT)) {
- cachedAllInputFileSlices = partitionFiles.keySet().stream()
- .collect(Collectors.toMap(
- Function.identity(),
- partitionPath ->
- queryInstant.map(instant ->
-
fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path,
queryInstant.get())
+ // NOTE: For MOR table, when the compaction is inflight, we need to not
only fetch the
+ // latest slices, but also include the base and log files of the
second-last version of
+ // the file slice in the same file group as the latest file slice that is
under compaction.
+ // This logic is realized by
`AbstractTableFileSystemView::getLatestMergedFileSlicesBeforeOrOn`
+ // API. Note that for COW table, the merging logic of two slices does not
happen as there
+ // is no compaction, thus there is no performance impact.
+ cachedAllInputFileSlices = partitionFiles.keySet().stream()
+ .collect(Collectors.toMap(
+ Function.identity(),
+ partitionPath ->
+ queryInstant.map(instant ->
+
fileSystemView.getLatestMergedFileSlicesBeforeOrOn(partitionPath.path,
queryInstant.get())
Review Comment:
I added the docs above. Is that sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
