danny0405 commented on a change in pull request #2929:
URL: https://github.com/apache/hudi/pull/2929#discussion_r629011753
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteFunction.java
##########
@@ -142,6 +142,22 @@
*/
private transient TotalSizeTracer tracer;
+ /**
+ * Flag saying whether the write task is waiting for the checkpoint success
notification
+ * after it finished a checkpoint.
+ *
+ * <p>The flag is needed because the write task does not block during the
waiting time interval,
+ * some data buckets still flush out with old instant time. There are two
cases that the flush may produce
+ * corrupted files if the old instant is committed successfully:
+ * 1) the write handle was writing data but interrupted, left a corrupted
parquet file;
+ * 2) the write handle finished the write but was not closed, left an empty
parquet file.
+ *
+ * <p>To solve, when this flag was set to true, we flush the data buffer
with a new instant time = old instant time + 1ms,
+ * the new instant time would affect the write file name. The filesystem
view does not recognize the file as committed because
+ * it always filters the data files based on successful commit time.
Review comment:
It is not invalid, if the write success and there is no retry, the write
data is still valid and the reader can see the data, for current instant. If
there is a retry for the cases i list here, the data would be seen as invalid
and thus does not cause confuse to the filesystem view, we then can go on with
the write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
