[
https://issues.apache.org/jira/browse/IMPALA-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-9134:
----------------------------------
Description:
The data stream sender currently does a synchronous RPC to close each channel
https://github.com/apache/impala/blob/d4648e8/be/src/runtime/krpc-data-stream-sender.cc#L565.
This is suboptimal because it serializes the network round-trips and takes
sum(RTT) over all the destinations in the best case, where no data needs to be
flushed or 2 * sum(RTT) in the worst case if all channels need to flush data.
If the RPCs were done asynchronously and overlapped with each other, we could
get this down to 2 * max(RTT).
I'm including this as a subtask of multi-threading because this is going to
scale poorly as the number of fragment instances increases.
was:
The data stream sender currently does a synchronous RPC to close each channel
https://github.com/apache/impala/blob/d4648e8/be/src/runtime/krpc-data-stream-sender.cc#L565.
This is suboptimal because it serializes the network round-trips and takes
sum(RTT) over all the destinations in the best case, where no data needs to be
flushed or 2 * sum(RTT) in the worst case if all channels need to flush data.
If the RPCs were done asynchronously and overlapped with each other, we could
get this down to 2 * max(RTT).
> Parallelise flush in data stream sender
> ---------------------------------------
>
> Key: IMPALA-9134
> URL: https://issues.apache.org/jira/browse/IMPALA-9134
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: perf, scalability
>
> The data stream sender currently does a synchronous RPC to close each channel
> https://github.com/apache/impala/blob/d4648e8/be/src/runtime/krpc-data-stream-sender.cc#L565.
> This is suboptimal because it serializes the network round-trips and takes
> sum(RTT) over all the destinations in the best case, where no data needs to
> be flushed or 2 * sum(RTT) in the worst case if all channels need to flush
> data.
> If the RPCs were done asynchronously and overlapped with each other, we could
> get this down to 2 * max(RTT).
> I'm including this as a subtask of multi-threading because this is going to
> scale poorly as the number of fragment instances increases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
