Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21579 )
Change subject: POC: Add queuing to broadcast data sender ...................................................................... Patch Set 1: (1 comment) The patch is in POC state and not reallyready for human consumption, I just wanted to quickly run the tests :) I hope that I can clean it up in a few days. Simple queries seem to work, but I didn't think about cancellation / closed receivers too much. http://gerrit.cloudera.org:8080/#/c/21579/1/be/src/runtime/krpc-data-stream-sender.cc File be/src/runtime/krpc-data-stream-sender.cc: http://gerrit.cloudera.org:8080/#/c/21579/1/be/src/runtime/krpc-data-stream-sender.cc@878 PS1, Line 878: const int QUEUE_SIZE = 2; > I'm trying to make sense of what this all does. Is it essentially a general The main point is the RpcFinished part - the goal is to be able to immediately start a new RPC after the previous one has finished for a given host (provided that there are more serialized batches in the queue), and not having to wait until all hosts catch up. An example is sending batch1 and batch2 to host1 and host2. 1. serialize batch1 2. start the rpc to host1 3. start the rpc to host2 4. serialize batch2 5. call transmitData on host1, (will block as 2. is not finished yet) If at this point 3 (the rpc to host2) is finished, the existing logic will not send the new batch, as it needs to wait for 2 (rpc to host1) to be finished before it can proceed. With the patch batch2 can be sent to host2 without waiting for host1. The same would be useful with partitioned exchanges too - I started with broadcast because it seemed the more complicated in terms of resource management, as it can only release a batch when all channels are finished with sending it. Another reason is that increasing queue size shouldn't be too risky for broadcast, while in the partitioned case the mem cost needs to be considered. -- To view, visit http://gerrit.cloudera.org:8080/21579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe3d01fada31f1b48de6c1403e7d2955114d8078 Gerrit-Change-Number: 21579 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Thu, 11 Jul 2024 19:57:00 +0000 Gerrit-HasComments: Yes
