Hello Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21797
to look at the new patch set (#2).
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
The way the planner estimates mem usage for KrpcDataStreamSender
is very different than how the backend actually uses memory -
the planner assumes that batch_size number of rows are sent at
a time while the BE tries to limit it to
data_stream_sender_buffer_size_ (but doesn't consider var len data).
The Jira has more detail about differences and issues.
This change adds flag data_stream_sender_buffer_size_used_by_planner.
If this is set to 16K (data_stream_sender_buffer_size_ default)
then the estimation will work similarly to BE.
Tested that this can improve both under and overestimations:
peak mem / mem estimate of the first sender:
select distinct * from tpch_parquet.lineitem limit 100000
default:
284.04 KB 2.75 MB
--data_stream_sender_buffer_size_used_by_planner=16384:
282.04 KB 283.39 KB
select distinct l_comment from tpch_parquet.lineitem limit 100000;
default:
747.71 KB 509.94 KB
--data_stream_sender_buffer_size_used_by_planner=16384:
740.71 KB 627.46 KB
The default is not changed to avoid side effects. I would like
to change it once BE's handling of var len data is fixed, which
is a prerequisity to use mem reservation in KrpcDataStreamSender.
Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
6 files changed, 44 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/21797/2
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
