ammarchalifah opened a new issue, #18597: URL: https://github.com/apache/iceberg/issues/18597
### Feature Request / Improvement ### Data Processing Context One of my major data pipeline relies on Iceberg's `MERGE INTO`, where the table is partitioned on multiple partition keys. For example, the partition keys are `date` and `bucket(N, id)`. Each new data slice that lands on a new `date` might also exist in the target table in a different date, and the pipeline need to deduplicate via `MERGE INTO` with `id` as the only join key. Considering shuffle is the biggest bottleneck on the performance, I'd like to start adopting SPJ, allowing SPJ on subset of partition keys. However, with `preserve-data-grouping` enabled, Spark will load all files that make up the same bucket into a single executor, which will cause OOM. The most obvious solution to me seems require increasing the granularity of the second partition key, e.g. from `bucket(N, id)` to `bucket(N*m, id)`, that will reduce the data loaded to each executor by a factor of `m`, increasing parallelism by a factor of `m`. However, it will also increase the file count by the same factor, and it will introduce new problems related to small files. ### Improvement Idea My first question would be: are there any established pattern/strategy to solve the problem without suffering from small file problem? I can only think about adjusting the partition layout, e.g. from `date` & `bucket(N, id)` into `week(date)` & `bucket(N*m, id)`, increasing the files by a factor of only `m/7`. It can also involves further pruning, such as `month(date)` instead of week to change it into `m/30`. Another idea is to enable node-local shuffle. Instead of pure SPJ, Spark might still do limited shuffle as long as all rows that belong to the same bucket/partition group are loaded into the same node, and shuffle are done purely on node-level without triggering cluster-wide shuffles. ### Query engine Spark ### Willingness to contribute - [ ] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
