Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9282 )
Change subject: IMPALA-6116: Bound memory usage of DataStreamSevice's service queue ...................................................................... Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9282/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9282/1//COMMIT_MSG@20 PS1, Line 20: this flag is left at 0, in which case it will be set to 20% of process > Devoting 20% of the process limit to this seems very high. The buffer pool I was using the assumption of 32KB avg payload size, 500 nodes, 500 fragments per host. Assuming each host has 64GB so process mem limit is 80% of it. The queue consumption translates to about 15% of process mem limit and I bumped it up to 20% just to be safe as broadcast exchange can have payload much larger than 32KB. The consequence of having a value too low is that RPCs may have to be retried multiple times. That said, it sounds like 20% may be too aggressive. Will trimming it down to 5% still too high or should we just hard code a value based on the assumption above ? -- To view, visit http://gerrit.cloudera.org:8080/9282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9 Gerrit-Change-Number: 9282 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Mon, 12 Feb 2018 23:03:57 +0000 Gerrit-HasComments: Yes
