Michael Ho has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9282 )

Change subject: IMPALA-6116: Bound memory usage of DataStreamSevice's service 
queue
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9282/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9282/1//COMMIT_MSG@20
PS1, Line 20: this flag is left at 0, in which case it will be set to 20% of 
process
> Devoting 20% of the process limit to this seems very high. The buffer pool
I was using the assumption of 32KB avg payload size, 500 nodes, 500 fragments 
per host. Assuming each host has 64GB so process mem limit is 80% of it. The 
queue consumption translates to about 15% of process mem limit and I bumped it 
up to 20% just to be safe as broadcast exchange can have payload much larger 
than 32KB.

The consequence of having a value too low is that RPCs may have to be retried 
multiple times. That said, it sounds like 20% may be  too aggressive. Will 
trimming it down to 5% still too high or should we just hard code a value based 
on the assumption above ?



--
To view, visit http://gerrit.cloudera.org:8080/9282
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9
Gerrit-Change-Number: 9282
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Feb 2018 23:03:57 +0000
Gerrit-HasComments: Yes

Reply via email to