Joe McDonnell created IMPALA-15485:
--------------------------------------
Summary: Partitioned exchanges should keep accumulating rows if
there is room in the buffer
Key: IMPALA-15485
URL: https://issues.apache.org/jira/browse/IMPALA-15485
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 5.0.0
Reporter: Joe McDonnell
Some row sizes are dramatically overestimated (e.g. TPC-DS Q67 sees
IMPALA-15089 ). The partitioned exchange sender uses the row size to decide how
many rows to accumulate before sending. Currently, this is a fixed calculation
where it divides the buffer size by the estimated row size. When the row size
estimate is too high, this results in many very small messages.
If we made the logic more dynamic, it could accumulate until the buffer is full
rather than stopping at a fixed number of rows. A prototype of this logic
improves TPC-DS Q67 by ~30% at scale 100.
[https://gerrit.cloudera.org/#/c/24126/]
Since this is pushing message sizes larger, we need to decide whether this also
needs bounding on the message size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
