nsivabalan commented on code in PR #9280:
URL: https://github.com/apache/hudi/pull/9280#discussion_r1280073349
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -343,7 +343,8 @@ public static HoodieTimeline
handleHollowCommitIfNeeded(HoodieTimeline completed
switch (handlingMode) {
case FAIL:
throw new HoodieException(String.format(
- "Found hollow commit: '%s'. Adjust config `%s` accordingly if to
avoid throwing this exception.",
+ "Found hollow commit: '%s'. If this happened for an incremental
query, "
Review Comment:
lets also add time travel query in addition to incremental query.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -343,7 +343,8 @@ public static HoodieTimeline
handleHollowCommitIfNeeded(HoodieTimeline completed
switch (handlingMode) {
case FAIL:
throw new HoodieException(String.format(
- "Found hollow commit: '%s'. Adjust config `%s` accordingly if to
avoid throwing this exception.",
+ "Found hollow commit: '%s'. If this happened for an incremental
query, "
+ + "adjust config `%s` accordingly if prefer to avoid throwing
this exception.",
Review Comment:
I am thinking if we can change the naming of this.
ConcurrentWritesReadStrategy
or
OutofOrderCommitsReadStrategy.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -133,10 +134,7 @@ trait HoodieIncrementalRelationTrait extends
HoodieBaseRelation {
// Validate this Incremental implementation is properly configured
validate()
- protected val hollowCommitHandling: HollowCommitHandling =
-
optParams.get(DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT.key)
- .map(HollowCommitHandling.valueOf)
-
.getOrElse(HollowCommitHandling.valueOf(DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT.defaultValue))
+ protected val hollowCommitHandling: HollowCommitHandling =
getHollowCommitHandling(optParams)
Review Comment:
can we add tests for diff HollowCommitHandling ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -343,7 +343,8 @@ public static HoodieTimeline
handleHollowCommitIfNeeded(HoodieTimeline completed
switch (handlingMode) {
case FAIL:
throw new HoodieException(String.format(
- "Found hollow commit: '%s'. Adjust config `%s` accordingly if to
avoid throwing this exception.",
+ "Found hollow commit: '%s'. If this happened for an incremental
query, "
+ + "adjust config `%s` accordingly if prefer to avoid throwing
this exception.",
Review Comment:
Lets add documentation to enum values in L360.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -420,7 +421,9 @@ abstract class HoodieBaseRelation(val sqlContext:
SQLContext,
inMemoryFileIndex.listFiles(partitionFilters, dataFilters)
}
- val fsView = new HoodieTableFileSystemView(metaClient, timeline,
partitionDirs.flatMap(_.files).toArray)
+ // fail time travel query when covers hollow commit
+ val completedTimeline = handleHollowCommitIfNeeded(timeline,
metaClient, HollowCommitHandling.FAIL)
Review Comment:
can we add tests for diff HollowCommitHandling ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
