yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786610
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -74,6 +74,15 @@ public static List<String> listCommitsSince(HoodieStorage
storage, String basePa
.map(HoodieInstant::getTimestamp).collect(Collectors.toList());
}
+ public static List<String> listCompletionTimeSince(FileSystem fs, String
basePath,
Review Comment:
This method is not used.
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -88,6 +97,16 @@ public static String latestCommit(HoodieStorage storage,
String basePath) {
return timeline.lastInstant().get().getTimestamp();
}
+ public static String latestCommitCompletionTime(FileSystem fs, String
basePath) {
+ HoodieTimeline timeline = allCompletedCommitsCompactions(fs, basePath);
+ return timeline.lastInstant().get().getCompletionTime();
+ }
+
+ public static String latestCommitCompletionTime(HoodieStorage storage,
String basePath) {
+ HoodieTimeline timeline = allCompletedCommitsCompactions(storage,
basePath);
+ return timeline.lastInstant().get().getCompletionTime();
+ }
Review Comment:
Similar here, this returns the completion time of the commit which has the
largest instant time, which might not be the max completion time on the
timeline.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -48,7 +48,6 @@ import org.apache.hudi.table.HoodieSparkTable
import org.apache.hudi.testutils.HoodieSparkClientTestBase
import org.apache.hudi.util.JFunction
import org.apache.hudi.{AvroConversionUtils, DataSourceReadOptions,
DataSourceWriteOptions, HoodieDataSourceHelpers, QuickstartUtils,
ScalaAssertionSupport}
-
Review Comment:
Let's keep the import grouping.
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -88,6 +97,16 @@ public static String latestCommit(HoodieStorage storage,
String basePath) {
return timeline.lastInstant().get().getTimestamp();
}
+ public static String latestCommitCompletionTime(FileSystem fs, String
basePath) {
+ HoodieTimeline timeline = allCompletedCommitsCompactions(fs, basePath);
+ return timeline.lastInstant().get().getCompletionTime();
+ }
+
+ public static String latestCommitCompletionTime(HoodieStorage storage,
String basePath) {
+ HoodieTimeline timeline = allCompletedCommitsCompactions(storage,
basePath);
+ return timeline.lastInstant().get().getCompletionTime();
+ }
Review Comment:
Should these be moved to test utils if they are only used by the tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
