goingforstudying-ctrl opened a new pull request, #16981: URL: https://github.com/apache/iceberg/pull/16981
Was debugging a ClassCastException that kept popping up during overwrite operations on tables with many manifests. Took a while to trace it back to the parallel manifest filtering path. The issue is that `filterManifests()` uses `Tasks.range()` to process manifests concurrently across worker threads. Inside `filterManifestWithDeletedFiles()`, worker threads were directly mutating the shared `deleteFiles` LinkedHashSet via `deleteFiles.add(fileCopy)`. LinkedHashSet is not thread-safe, so concurrent adds from multiple threads corrupt the internal HashMap, producing the `ClassCastException: LinkedHashMap$Entry cannot be cast to HashMap$TreeNode` that was reported in #16978. The fix has three parts: 1. Collect per-manifest deleted files into a `ConcurrentMap` from each worker thread instead of mutating the shared set. After all parallel filtering completes, merge the per-manifest results into `deleteFiles` on the main thread. 2. Replace the `filteredManifestToDeletedFiles` HashMap with a `ConcurrentHashMap` so worker threads can safely put their results without resize races. 3. Add `ThreadSafeFileSet`, a `ConcurrentHashMap`-backed `Set` implementation for defense-in-depth. Use it for the `deleteFiles` collection so even if future code accidentally adds concurrent mutations, the set won't corrupt. I also switched `filteredManifestToDeletedFiles` to `ConcurrentHashMap` because worker threads write to it from `filterManifest()` while the main thread reads after join. The old plain HashMap could resize unpredictably under concurrent writes. Added `TestManifestFilterManagerConcurrency` with two tests that create tables with 32 manifests and perform full overwrites/deletes to exercise the parallel filtering path. Not 100% sure if `ThreadSafeFileSet` should live in `api` or `core` — went with `api` because it's a general utility, but happy to move it if that doesn't feel right. Closes #16978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
