mingmwang commented on issue #342: URL: https://github.com/apache/arrow-ballista/issues/342#issuecomment-1276464109
@Dandandan Sounds nice. There are couple of things need to do to support the broadcasting exchange. 1. Rpc protocols to efficiently do broadcasting, something similar to Spark's TorrentBroadcast 2. Executor memory management as you just mentioned. 3. Broadcast lifecycle management, after SQL finish, those broadcasts should be cleaned. 4. Planner need more accurate stats to choose broadcast join or partitioned hash join or even SortMergeJoin, build side selection etc. Need to enhance the stats collection. 6. Broadcast exchange reuse within SQL? 7. Broadcast reuse between SQLs ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
