[
https://issues.apache.org/jira/browse/HUDI-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17398445#comment-17398445
]
Prashant Wason commented on HUDI-2303:
--------------------------------------
The following diff fixes this issue:
{{}}
diff --git
a/hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
b/hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
index e408ad939..f365ed0a7 100644
--- a/hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
+++ b/hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
@@ -348,4 +348,8 @@ public abstract class BaseTableMetadata implements
HoodieTableMetadata {
return
datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant()
.map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
}
+
+ public HoodieMetadataConfig getMetadataConfig() {
+ return metadataConfig;
+ }
}
diff --git
a/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java
b/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java
index a3d0e2dfe..7b0d5daef 100644
---
a/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java
+++
b/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java
@@ -36,7 +36,7 @@ import org.apache.hudi.exception.HoodieException;
*/
public class HoodieMetadataFileSystemView extends HoodieTableFileSystemView {
- private final HoodieTableMetadata tableMetadata;
+ private HoodieTableMetadata tableMetadata;
public HoodieMetadataFileSystemView(HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline,
@@ -73,4 +73,16 @@ public class HoodieMetadataFileSystemView extends
HoodieTableFileSystemView {
throw new HoodieException("Error closing metadata file system view.", e);
}
}
+
+ @Override
+ public void sync() {
+ // Sync the tableMetadata first as super.sync() may call listPartition
+ if (tableMetadata != null) {
+ BaseTableMetadata baseMetadata = (BaseTableMetadata) tableMetadata;
+ tableMetadata = HoodieTableMetadata.create(baseMetadata.getEngineContext(),
baseMetadata.getMetadataConfig(),
+ metaClient.getBasePath(),
FileSystemViewStorageConfig.FILESYSTEM_VIEW_SPILLABLE_DIR.defaultValue());
+ }
+
+ super.sync();
+ }
}
> TestMereIntoLogOnlyTable with metadata enabled surfaces likely bug
> ------------------------------------------------------------------
>
> Key: HUDI-2303
> URL: https://issues.apache.org/jira/browse/HUDI-2303
> Project: Apache Hudi
> Issue Type: Bug
> Components: Common Core
> Reporter: Udit Mehrotra
> Assignee: Prashant Wason
> Priority: Major
>
> While enabling Metadata as part of
> [https://github.com/apache/hudi/pull/3411/] one of the test that fails is
> *TestMereIntoLogOnlyTable*.
> Upon looking a bit, what I found is after the final *Merge* command there is
> an inline compaction that is triggered. The parquet file formed as part of
> the compaction misses out on the data from the latest log file right before
> compaction.
> I think it might be because of metadata returning an incorrect list for
> compaction, missing out on the latest log file.
> cc [~pwason] [~vinoth]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
