codope commented on code in PR #10872:
URL: https://github.com/apache/hudi/pull/10872#discussion_r1527412766
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java:
##########
@@ -124,4 +123,4 @@ private JavaRDD<String> postProcess(JavaRDD<String>
jsonStringRDD) {
return processor.process(jsonStringRDD);
}
-}
+}
Review Comment:
nit: new line in the end
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -186,4 +182,4 @@ static Set<String> getPartitionColumns(TypedProperties
props) {
return
Arrays.stream(partitionColumns.split(",")).collect(Collectors.toSet());
}
-}
+}
Review Comment:
nit: new line at the end
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -91,26 +89,24 @@ public static Option<JavaRDD<HoodieRecord>>
createHoodieRecords(HoodieStreamer.C
props.setProperty(KeyGenUtils.RECORD_KEY_GEN_INSTANT_TIME_CONFIG, instantTime);
}
BuiltinKeyGenerator builtinKeyGenerator = (BuiltinKeyGenerator)
HoodieSparkKeyGeneratorFactory.createKeyGenerator(props);
- List<Either<HoodieRecord,String>> avroRecords = new
ArrayList<>();
- while (genericRecordIterator.hasNext()) {
- GenericRecord genRec = genericRecordIterator.next();
+ boolean useConsistentLogicalTimestamp = props.getBoolean(
+
KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.key(),
+
Boolean.parseBoolean(KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.defaultValue()));
Review Comment:
We can even extract this out of `mapPartitions`. This keygen config is not
going to change across partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
