satishkotha commented on a change in pull request #2809:
URL: https://github.com/apache/hudi/pull/2809#discussion_r616870583
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -438,11 +437,20 @@ public static HoodieMetadataConfig
buildMetadataConfig(Configuration conf) {
if (LOG.isDebugEnabled()) {
LOG.debug("Hoodie Metadata initialized with completed commit instant
as :" + metaClient);
}
-
HoodieTimeline timeline =
HoodieHiveUtils.getTableTimeline(metaClient.getTableConfig().getTableName(),
job, metaClient);
+
Review comment:
can we combine this into getTableTimeline method?
HoodieHiveUtils.getTableTimeline is anyway getting all the config. So i think
that provides better abstraction to get relevant timeline.
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##########
@@ -137,4 +139,16 @@ public static HoodieTimeline getTableTimeline(final String
tableName, final JobC
// by default return all completed commits.
return
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
}
+
+ public static Option<String> getSnapshotMaxCommitTime(JobConf job, String
tableName) {
+ String maxCommitTime = job.get(getSnapshotMaxCommitKey(tableName));
+ if (maxCommitTime != null) {
Review comment:
consider using !StringUtils.isNullorEmpty() or just simply return
Option.ofNullable(maxCommitTime)
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##########
@@ -68,6 +69,7 @@
public static final String HOODIE_STOP_AT_COMPACTION_PATTERN =
"hoodie.%s.ro.stop.at.compaction";
public static final String INCREMENTAL_SCAN_MODE = "INCREMENTAL";
public static final String SNAPSHOT_SCAN_MODE = "SNAPSHOT";
+ public static final String HOODIE_SNAPSHOT_CONSUME_COMMIT_PATTERN =
"hoodie.%s.consume.snapshot.time";
Review comment:
Do you think we can reuse existing config? perhaps HOODIE_CONSUME_COMMIT?
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##########
@@ -137,4 +139,16 @@ public static HoodieTimeline getTableTimeline(final String
tableName, final JobC
// by default return all completed commits.
return
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
}
+
+ public static Option<String> getSnapshotMaxCommitTime(JobConf job, String
tableName) {
Review comment:
nit: consider adding javadoc for all public methods (I know we dont
follow this consistently, but would be great to add for all new code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
