yanghua commented on a change in pull request #2143:
URL: https://github.com/apache/hudi/pull/2143#discussion_r502103169
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieWriteableTestTable.java
##########
@@ -94,6 +106,10 @@ public String withInserts(String partition) throws
Exception {
}
public String withInserts(String partition, HoodieRecord... records) throws
Exception {
+ return withInserts(partition, Arrays.asList(records));
+ }
+
+ public String withInserts(String partition, List<HoodieRecord> records)
throws Exception {
Review comment:
Considering the fluent APIs have one general pattern: method `withXXX`
returns `HoodieWriteableTestTable ` itself. However, this method and above
break this rule. WDYT about renaming them to `getFileIdWithInserts` or
`returnFileIdWithInserts`?
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieWriteableTestTable.java
##########
@@ -128,4 +148,37 @@ public HoodieWriteableTestTable withInserts(String
partition, String fileId, Hoo
return this;
}
+
+ public HoodieWriteableTestTable withLogAppends(HoodieRecord... records)
throws Exception {
+ return withLogAppends(Arrays.asList(records));
+ }
+
+ public HoodieWriteableTestTable withLogAppends(List<HoodieRecord> records)
throws Exception {
+ for (List<HoodieRecord> groupedRecords: records.stream()
+
.collect(Collectors.groupingBy(HoodieRecord::getCurrentLocation)).values()) {
+ appendRecordsToLogFile(groupedRecords);
+ }
+ return this;
+ }
+
+ private void appendRecordsToLogFile(List<HoodieRecord> groupedRecords)
throws Exception {
+ String partitionPath = groupedRecords.get(0).getPartitionPath();
+ HoodieRecordLocation location = groupedRecords.get(0).getCurrentLocation();
+ try (HoodieLogFormat.Writer logWriter =
HoodieLogFormat.newWriterBuilder().onParentPath(new Path(basePath,
partitionPath))
+
.withFileExtension(HoodieLogFile.DELTA_EXTENSION).withFileId(location.getFileId())
+ .overBaseCommit(location.getInstantTime()).withFs(fs).build()) {
+ Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
+ header.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME,
location.getInstantTime());
+ header.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, schema.toString());
+ logWriter.appendBlock(new
HoodieAvroDataBlock(groupedRecords.stream().map(r -> {
+ try {
+ GenericRecord val = (GenericRecord)
r.getData().getInsertValue(schema).get();
+ HoodieAvroUtils.addHoodieKeyToRecord(val, r.getRecordKey(),
r.getPartitionPath(), "");
+ return (IndexedRecord) val;
+ } catch (IOException e) {
Review comment:
Logging the detailed exception information looks better?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
