Alexey Kudinkin created HUDI-3302:
-------------------------------------
Summary: Re-evaluate handling of LogBlock appends when Compaction
is pending
Key: HUDI-3302
URL: https://issues.apache.org/jira/browse/HUDI-3302
Project: Apache Hudi
Issue Type: Task
Reporter: Alexey Kudinkin
Currently, when (async) Compaction for particular File Group has been scheduled
but not yet completed, if writer will try to append additional Log Blocks to
the same file-group following will occur:
# FileSystemView (when fetched), will check whether any compaction is pending
and if it's it will inject "phantom" (ie non-existent) log-file into the
existing FileSlice, which will have the same FileGroup name, but will bear
instant of the scheduled Compaction commit (on the timeline) in its name (as
opposed to the instant of the base-file)
# Writer will pick up such log-file as the latest
# Writer will write into such "phantom" log-file
[REF:
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java#L199|https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java#L199]
This posses following problems:
* Reader now has to be aware of such handling and therefore always include
pending compaction instants into its timeline when fetching the FileSystemView,
as otherwise it will miss newly added log-files.
* This pushes the decision-making point of where writes should be channeled
down into FileSystemView, which is clearly alien to its scope of
responsibilities.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
