wangyum opened a new pull request, #18862: URL: https://github.com/apache/iceberg/pull/18862
Fixes #18861. ## Problem Manifest reading and scan planning call `PartitionData.copy()` / `copyFor()` once per manifest entry, unconditionally allocating a fresh instance (and backing `Object[]`) even when many entries share the same partition values. For tables with few distinct partitions but many small files (e.g. merge-on-read tables without regular compaction), this produces millions of redundant `PartitionData` objects on the driver during scan planning. Production heap dump from a Spark driver OOM scanning a table with ~4.4M data files + ~4.34M delete files but only a few thousand distinct partitions: | Class | Instances | Retained | |---|---|---| | `org.apache.iceberg.PartitionData` | 8,738,866 | ~489 MB | | `java.lang.Object[]` (backing) | 9,697,151 | ~614 MB | `PartitionData` instance count equals data files + delete files exactly. ## Approach Route `copy()` / `copyFor()` through a weak `Interner` so entries with equal partition spec and values collapse to a single shared instance: - A **weak** interner is used so entries are evicted automatically once nothing references them anymore — safe for long-lived processes that plan scans across many tables, no manual scope management needed. - Interned instances are treated as immutable snapshots; the existing callers (`BaseFile` copy constructor, `StructLikeMap`/`StructLikeSet`, `TableScanUtil`, `ManifestFilterManager`, `PartitionsTable`, `PartitionStatsHandler`) only read the copied partition and never mutate it. A test guards that mutating a source (reused builder scratch) instance after `copy()` does not affect the interned instance. ## Supporting changes 1. **`equals`/`hashCode`: `Arrays.equals` → `Arrays.deepEquals`** (and `deepHashCode`). `PartitionData` stores `BINARY`/`FIXED` partition values as `byte[]` inside the `Object[] data`; element-wise `equals` on `byte[]` is reference equality, so binary-keyed partitions would never intern. This is also a latent correctness bug for any `Map`/`Set` keyed on `PartitionData` with binary partition columns. 2. **`GuavaClasses`**: retain `Interner`/`Interners` in the minimized relocated bundled-guava jar (they were previously unreferenced and stripped). ## Tests `TestPartitionData`: equal values intern to the same instance; different values stay distinct; `copyFor` interns; source mutation after copy is isolated; empty partition unaffected; binary/fixed partition values intern by content. Related: #18359 (complementary — reduces `DeleteFileIndex` memory for file-scoped position deletes/DVs). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
