sivabalan narayanan created HUDI-4728:
-----------------------------------------
Summary: Add support to skip larger log blocks with minor log
compaction
Key: HUDI-4728
URL: https://issues.apache.org/jira/browse/HUDI-4728
Project: Apache Hudi
Issue Type: Improvement
Components: compaction
Reporter: sivabalan narayanan
Is there a size threshold to exclude big log blocks? Why do log compaction on
log blocks that are big enough? Thoughts
Good point. For initial version we want to target all the blocks. In the coming
iterations I will include block sizes threshold as well. Current logic of
AbstractHoodieLogRecordReader should be able to handle it as well.
For streaming workloads, this might be very heavy. So, we need to support this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
