n3nash commented on a change in pull request #1288: [HUDI-117] Close file
handle before throwing an exception due to append…
URL: https://github.com/apache/incubator-hudi/pull/1288#discussion_r372190536
##########
File path:
hudi-common/src/test/java/org/apache/hudi/common/table/log/TestHoodieLogFormat.java
##########
@@ -1113,6 +1113,99 @@ public void
testAvroLogRecordReaderWithMixedInsertsCorruptsAndRollback()
assertEquals("We would read 0 records", 0, scanner.getTotalLogRecords());
}
+ /*
+ * During a spark stage failure, when the stage is retried, tasks that are
part of the previous attempt
+ * of the stage would continue to run. As a result two different tasks
could be performing the same operation.
+ * When trying to update the log file, only one of the tasks would succeed
(one holding lease on the log file).
+ *
+ * In order to make progress in this scenario, second task attempting to
update the log file would rollover to
+ * a new version of the log file. As a result, we might end up with two log
files with same set of data records
+ * present in both of them.
+ *
+ * Following uint tests mimic this scenario to ensure that the reader can
handle merging multiple log files with
+ * duplicate data.
+ *
+ */
+ private void testAvroLogRecordReaderMergingMultipleLogFiles(int
numRecordsInLog1, int numRecordsInLog2)
+ throws IOException, URISyntaxException, InterruptedException {
+ try {
+ // Write one Data bloc with same InstantTime (written in same batch)
Review comment:
nit : s/bloc/block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
