Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9282 )
Change subject: IMPALA-6116: Bound memory usage of DataStreamSevice's service queue ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/9282/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9282/1//COMMIT_MSG@20 PS1, Line 20: this flag is left at 0, in which case it will be set to 20% of process > I was using the assumption of 32KB avg payload size, 500 nodes, 500 fragmen It seems too aggressive for small-to-medium deployments for sure :). It might be a reasonable value for very large deployments with high concurrency, but I think that would require adjusting buffer_pool_limit too. http://gerrit.cloudera.org:8080/#/c/9282/1/be/src/service/data-stream-service.cc File be/src/service/data-stream-service.cc: http://gerrit.cloudera.org:8080/#/c/9282/1/be/src/service/data-stream-service.cc@37 PS1, Line 37: DEFINE_int64(datastream_service_queue_mem_limit, 0, It would be good to use ParseUtil::ParseMemSpec() for this for consistency and convenience - see what is done for --buffer_pool_limit. This lets users specify it as a bytes amount with the usual suffixes or a percentage. -- To view, visit http://gerrit.cloudera.org:8080/9282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idea4262dfb0e0aa8d58ff6ea6a8aaaa248e880b9 Gerrit-Change-Number: 9282 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Mon, 12 Feb 2018 23:44:57 +0000 Gerrit-HasComments: Yes
