wangyum opened a new issue, #18861: URL: https://github.com/apache/iceberg/issues/18861
### Feature Request / Improvement Scan planning on large tables allocates one `PartitionData` (plus its backing `Object[]`) **per manifest entry**, even when millions of files share a small number of distinct partition values. On merge-on-read tables with many small files and many delete files (e.g. after heavy DV/position-delete churn without regular compaction), this produces millions of redundant `PartitionData` instances on the Spark driver during scan planning and contributes materially to Driver OOM. ### Observed evidence (production heap dump, Spark driver) Table scanned: ~4.4M data files + ~4.34M delete files, only a few thousand distinct physical partitions. | Class | Instance count | Retained | |---|---|---| | `org.apache.iceberg.PartitionData` | 8,738,866 | ~489 MB | | `java.lang.Object[]` (backing arrays) | 9,697,151 | ~614 MB | `PartitionData` count == data files + delete files exactly: each manifest entry pays for a private copy, allocated via `BaseFile`'s copy constructor: ```java // BaseFile (copy constructor) this.partitionData = toCopy.partitionData.copy(); ``` `PartitionData.copy()` / `copyFor()` unconditionally allocate a fresh instance and backing array; there is no deduplication across entries with identical partition values. ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/fix independently ### Proposed approach Route `PartitionData.copy()` / `copyFor()` through a weak Guava `Interner` so entries with equal partition spec and values collapse to a single shared instance. Weak keys ensure interned instances are evicted once nothing references them, so this is safe for long-lived processes planning scans across many tables. Two supporting changes: 1. `PartitionData.equals`/`hashCode` currently use `Arrays.equals(Object[], Object[])` / `Arrays.hashCode(Object[])`, which compare elements by reference — partitions with `BINARY`/`FIXED` columns (stored as `byte[]`) never compare equal. Switch to `Arrays.deepEquals` / `Arrays.deepHashCode` so binary partition values intern correctly (this is independently a latent correctness bug for any `Map`/`Set` keyed on `PartitionData` with binary partition columns). 2. Retain `Interner`/`Interners` in the minimized relocated bundled-guava jar (`GuavaClasses`), which currently strips them. Related work on the same Driver-OOM theme: #18359 (reduces `DeleteFileIndex` memory for file-scoped position deletes/DVs). This improvement is complementary and addresses a different allocation source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
