[
https://issues.apache.org/jira/browse/HUDI-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Brown reassigned HUDI-6249:
-----------------------------------
Assignee: Timothy Brown
> Make maps in HoodieTableFileSystemView concurrent maps
> ------------------------------------------------------
>
> Key: HUDI-6249
> URL: https://issues.apache.org/jira/browse/HUDI-6249
> Project: Apache Hudi
> Issue Type: Improvement
> Components: timeline-server
> Reporter: Jonathan Vexler
> Assignee: Timothy Brown
> Priority: Major
>
> {code:java}
> // mapping from partition paths to file groups contained within them
> protected Map<String, List<HoodieFileGroup>> partitionToFileGroupsMap;
> /**
> * PartitionPath + File-Id to pending compaction instant time.
> */
> protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>>
> fgIdToPendingCompaction;
> /**
> * PartitionPath + File-Id to pending compaction instant time.
> */
> protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>>
> fgIdToPendingLogCompaction;
> /**
> * PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped).
> */
> protected Map<HoodieFileGroupId, BootstrapBaseFileMapping>
> fgIdToBootstrapBaseFile;
> /**
> * Track replace time for replaced file groups.
> */
> protected Map<HoodieFileGroupId, HoodieInstant> fgIdToReplaceInstants;
> /**
> * Track file groups in pending clustering.
> */
> protected Map<HoodieFileGroupId, HoodieInstant> fgIdToPendingClustering;
> {code}
> The maps should be ConcurrentMaps, because addFilesToView in
> AbstractTableFileSystem can be called on multiple partitions concurrently.
> SpillableMapBasedFileSystemView uses ExternalSpillableMap which is not thread
> safe, so we cannot update HoodieTableFileSystemView because then
> SpillableMapBasedFileSystemView would not implement it correctly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
