Jonathan Vexler created HUDI-6249:
-------------------------------------
Summary: Make maps in HoodieTableFileSystemView concurrent maps
Key: HUDI-6249
URL: https://issues.apache.org/jira/browse/HUDI-6249
Project: Apache Hudi
Issue Type: Improvement
Components: timeline-server
Reporter: Jonathan Vexler
{code:java}
// mapping from partition paths to file groups contained within them
protected Map<String, List<HoodieFileGroup>> partitionToFileGroupsMap;
/**
* PartitionPath + File-Id to pending compaction instant time.
*/
protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>>
fgIdToPendingCompaction;
/**
* PartitionPath + File-Id to pending compaction instant time.
*/
protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>>
fgIdToPendingLogCompaction;
/**
* PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped).
*/
protected Map<HoodieFileGroupId, BootstrapBaseFileMapping>
fgIdToBootstrapBaseFile;
/**
* Track replace time for replaced file groups.
*/
protected Map<HoodieFileGroupId, HoodieInstant> fgIdToReplaceInstants;
/**
* Track file groups in pending clustering.
*/
protected Map<HoodieFileGroupId, HoodieInstant> fgIdToPendingClustering; {code}
The maps should be ConcurrentMaps, because addFilesToView in
AbstractTableFileSystem can be called on multiple partitions concurrently.
SpillableMapBasedFileSystemView uses ExternalSpillableMap which is not thread
safe, so we cannot update HoodieTableFileSystemView because then
SpillableMapBasedFileSystemView would not implement it correctly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
