nbalajee commented on code in PR #10789:
URL: https://github.com/apache/hudi/pull/10789#discussion_r1518400672
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java:
##########
@@ -62,15 +61,14 @@ public class HoodieLogFormatWriter implements
HoodieLogFormat.Writer {
Short replication,
Long sizeThreshold,
String rolloverLogWriteToken,
- LogFileCreationCallback fileCreationHook) {
+ LogFileCreationCallback fileCreationCallback) {
Review Comment:
> that's true, the shutdown hook is generally not a good way for releasing
resource purpose, I'm just wondering the background for introducing it in the
first place, what is the effect if the shutdown hook is not there and the
inputstream is not closed ?
HoodieLogWriter has the mechanism to rollover to the next version of the log
file and start writing to a new log file. However, this might leave a trail
of stale/partially written log files (incomplete or corrupt log blocks).
Improvements added for handling corrupt log blocks should be able to ignore
these partial log blocks and recover automatically.
However, from debugging the issues perspective, having too many partially
written log files (log blocks) could be challenging to handle and mask some
real issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
