[
https://issues.apache.org/jira/browse/HUDI-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8653:
------------------------------
Description:
When there is a pending compaction, the new base files to be generated by
compaction is not available during this transaction. Given the log files in MOR
from this transaction can be attached to the base file generated by the
compaction in the latest file slice, the accurate record positions may not be
derived. So we should disable WRITE_RECORD_POSITIONS in this case to avoid
incorrect merging results, as the log merging has not accounted for this yet.
After HUDI-8654 fixes the merging behavior, we can remove this check.
was:When there is a pending compaction, the new base files to be generated by
compaction is not available during this transaction. Given the log files in MOR
from this transaction can be attached to the base file generated by the
compaction in the latest file slice, the accurate record positions may not be
derived. So we should disable WRITE_RECORD_POSITIONS in this case to avoid
incorrect merging results, as the log merging has not accounted for this yet.
> Disable write record positions with pending compaction
> ------------------------------------------------------
>
> Key: HUDI-8653
> URL: https://issues.apache.org/jira/browse/HUDI-8653
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Y Ethan Guo
> Assignee: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.0
>
>
> When there is a pending compaction, the new base files to be generated by
> compaction is not available during this transaction. Given the log files in
> MOR from this transaction can be attached to the base file generated by the
> compaction in the latest file slice, the accurate record positions may not be
> derived. So we should disable WRITE_RECORD_POSITIONS in this case to avoid
> incorrect merging results, as the log merging has not accounted for this yet.
> After HUDI-8654 fixes the merging behavior, we can remove this check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
