hudi-bot opened a new issue, #15971:
URL: https://github.com/apache/hudi/issues/15971
{code:java}
// mapping from partition paths to file groups contained within them
protected Map<String, List<HoodieFileGroup>> partitionToFileGroupsMap;
/**
* PartitionPath + File-Id to pending compaction instant time.
*/
protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>>
fgIdToPendingCompaction;
/**
* PartitionPath + File-Id to pending compaction instant time.
*/
protected Map<HoodieFileGroupId, Pair<String, CompactionOperation>>
fgIdToPendingLogCompaction;
/**
* PartitionPath + File-Id to bootstrap base File (Index Only bootstrapped).
*/
protected Map<HoodieFileGroupId, BootstrapBaseFileMapping>
fgIdToBootstrapBaseFile;
/**
* Track replace time for replaced file groups.
*/
protected Map<HoodieFileGroupId, HoodieInstant> fgIdToReplaceInstants;
/**
* Track file groups in pending clustering.
*/
protected Map<HoodieFileGroupId, HoodieInstant> fgIdToPendingClustering;
{code}
The maps should be ConcurrentMaps, because addFilesToView in
AbstractTableFileSystem can be called on multiple partitions concurrently.
SpillableMapBasedFileSystemView uses ExternalSpillableMap which is not
thread safe, so we cannot update HoodieTableFileSystemView because then
SpillableMapBasedFileSystemView would not implement it correctly
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6249
- Type: Improvement
---
## Comments
24/May/23 18:09;jonvex;accidentally closed;;;
---
03/Apr/25 00:24;tim.brown;The issue here is that the write and read locks
were not always being used when required. Raised a PR with locking updates to
handle this https://github.com/apache/hudi/pull/12982;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
