stephensunyj opened a new issue, #16854:
URL: https://github.com/apache/iceberg/issues/16854
### Query engine
Spark in Scala. I am using iceberg-spark-runtime-3.5_2.12-1.9.2.jar.
### Question
Been dealing with pretty poor read performance on our Iceberg V2 table on
Apache Spark 3.5 + Iceberg 1.9.2. Our table does have a wide nested schema
(~2,000 leaf paths under struct columns), ~415 GB / ~1.29 B records.
We have `write.distribution-mode=range`, `write.format.default=parquet`,
zstd compression, `write.target-file-size-bytes=250 MiB` originally. As far as
I can tell, the data distribution is bimodal: a small number of hot partitions
hold most rows, and a long tail of small partitions hold relatively few rows
each.
Post-ingest (load + Iceberg MERGE INTO) consistently lands at:
- 38,246 files / **avg 10.86 MB** / median 457 KB / max 256 MB
- Hot partitions: well-sized (~720 files at ~255 MB each)
- Long-tail partitions: many tiny files (a partition with 6K records can
hold
720 files of ~419 KB each)
I want to raise the average file size at write time without changing the
partition spec, ideally at first-write time.
### What we've tried (all measured post-MERGE operation)
| # | Change | File count | Avg size | Result |
|---|--------|-----------:|---------:|--------|
| 1 | Baseline (250 MiB target, shuf=720) | 38,246 | 10.86 MB | (baseline)
|
| 2 | `write.target-file-size-bytes` 250 MiB → 512 MiB | 38,110 | 10.90 MB
| No change |
| 3 | `spark.sql.shuffle.partitions` 720 → 360 | 61,772 | 6.73 MB |
**Worse** |
| 4 | `spark.sql.adaptive.advisoryPartitionSizeInBytes` default 64 MiB →
128 MiB | 38,110 | 10.90 MB | No change |
### Questions
1. Is there any write-side setting that materially raises the average file
size at first-write time for this partition spec? Curious if anyone has
achieved a higher average file size for a similar layout to our table (~7K
partitions, hundreds of GB, wide nested schema).
2. If first-write is bounded, is `CALL system.rewrite_data_files` the
canonical / recommended way to address this? i.e., the expected pattern is
"ingest produces what it produces, then post-write compaction handles it"? If
yes, is it performant to run on tables at this scale (~7K partitions, hundreds
of GB)? If the post-write compaction ends up taking as long as reads/longer
than reads, then maybe even a one-compaction might not be worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
