Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21797
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate ...................................................................... IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate The way the planner estimates mem usage for KrpcDataStreamSender is very different than how the backend actually uses memory - the planner assumes that batch_size number of rows are sent at a time while the BE tries to limit it to data_stream_sender_buffer_size_ (but doesn't consider var len data). The Jira has more detail about differences and issues. This change adds flag data_stream_sender_buffer_size_used_by_planner. If this is set to 16K (data_stream_sender_buffer_size_ default) then the estimation will work similarly to BE. Tested that this can improve both under and overestimations: peak mem / mem estimate of the first sender: select distinct * from tpch_parquet.lineitem limit 100000 default: 284.04 KB 2.75 MB --data_stream_sender_buffer_size_used_by_planner=16384: 282.04 KB 283.39 KB select distinct l_comment from tpch_parquet.lineitem limit 100000; default: 747.71 KB 509.94 KB --data_stream_sender_buffer_size_used_by_planner=16384: 740.71 KB 627.46 KB The default is not changed to avoid side effects. I would like to change it once BE's handling of var len data is fixed, which is a prerequisity to use mem reservation in KrpcDataStreamSender. Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java 6 files changed, 42 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/21797/1 -- To view, visit http://gerrit.cloudera.org:8080/21797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f Gerrit-Change-Number: 21797 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
