[
https://issues.apache.org/jira/browse/HUDI-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-5863:
---------------------------------
Labels: pull-request-available (was: )
> Fix HoodieMetadataFileSystemView serving stale view at the timeline server
> --------------------------------------------------------------------------
>
> Key: HUDI-5863
> URL: https://issues.apache.org/jira/browse/HUDI-5863
> Project: Apache Hudi
> Issue Type: Bug
> Components: timeline-server, writer-core
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.13.1
>
>
> We observe that for MOR table, occasionally (<10% for large tables with
> frequent updates and compactions), the deltacommit after the compaction
> commit may add a new log file to the old file slice, not the latest file
> slice, in the corresponding file group. This happens when both the metadata
> table and timeline server are enabled. If either is disabled, the problem
> does not show up.
> Deeper analysis of the code surfaces that the file system view at the
> timeline server may serve the stale view, causing the issue. This is because
> the sync of HoodieMetadataFileSystemView is not atomic when the metadata
> table is enabled:
>
> {code:java}
> AbstractTableFileSystemView:
> @Override
> public void sync() {
> HoodieTimeline oldTimeline = getTimeline();
> HoodieTimeline newTimeline =
> metaClient.reloadActiveTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
> try {
> writeLock.lock();
> runSync(oldTimeline, newTimeline);
> } finally {
> writeLock.unlock();
> }
> }
> HoodieMetadataFileSystemView:
> @Override
> public void sync() {
> super.sync();
> tableMetadata.reset();
> }
> {code}
> The
> tableMetadata.reset()
> happens outside the write lock, causing the issue during race conditions
> (reading the inconsistent state while the table metadata is being reset).
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
