adiranjan25 commented on issue #18597:
URL: https://github.com/apache/iceberg/issues/18597#issuecomment-6815723258

   Hi @ammarchalifah ,
   
   I'd like to take ownership of investigating and implementing this 
improvement.
   
   I'm planning to focus on the Spark–Iceberg Storage Partition Join (SPJ) 
behavior described here, particularly the executor memory pressure caused by 
grouping data across multiple partition keys and the trade-off between shuffle 
reduction, parallelism, and small-file proliferation.
   
   My proposed approach is to:
   
   1. Reproduce the behavior using a representative Iceberg table partitioned 
by date and bucketed ID, with a `MERGE INTO` workload joining on ID.
   2. Establish baseline measurements for executor memory consumption, shuffle 
volume, task distribution, and execution time.
   3. Investigate whether finer-grained, locality-aware partitioning or a 
node-local shuffle strategy can reduce executor memory pressure without 
increasing the physical table's bucket count.
   4. Develop a focused implementation with correctness tests and reproducible 
performance benchmarks.
   5. Submit a PR with the implementation details, benchmark results, and 
documented trade-offs.
   
   Before making significant architectural changes, I'll review the existing 
Spark SPJ and Iceberg scan-planning mechanisms and share the proposed design 
here for maintainer feedback.
   
   **I'm actively taking this issue forward and intend to submit a PR.** If 
anyone is already working on an overlapping implementation, please let me know 
so we can coordinate and avoid duplicate effort.
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to