[GitHub] [arrow-ballista] andygrove opened a new issue, #338: Optimize shuffle before coalesce

[GitBox]

andygrove opened a new issue, #338:
URL: https://github.com/apache/arrow-ballista/issues/338

   **Is your feature request related to a problem or challenge? Please describe 
what you are trying to do.**
   
   This looks inefficient. We are writing lots of shuffle files, reading them, 
and coalescing them into a single partition. Can we do the coalesce step before 
the shuffle write in this case? 
   
   
![opt-coalesce](https://user-images.githubusercontent.com/934084/194887825-bfa3f0f6-5fb2-4511-9937-d5506139c622.png)
   
   **Describe the solution you'd like**
   Optimize
   
   **Describe alternatives you've considered**
   None
   
   **Additional context**
   None
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org