Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 )
Change subject: IMPALA-6035: Add query options to limit thread reservation ...................................................................... Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/10365/9/common/thrift/ImpalaInternalService.thrift File common/thrift/ImpalaInternalService.thrift: http://gerrit.cloudera.org:8080/#/c/10365/9/common/thrift/ImpalaInternalService.thrift@287 PS9, Line 287: 66: optional i32 thread_reservation_limit = 0; > We could also consider choosing a default dynamically based on the OS's set I increased some of the configs (I think I was limited by /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max == 12288) and now it got oom-killed at ~26000 threads. I think unfortunately there are a lot of different OS knobs that impact this and they seem to evolve over time, so it's probably not feasible with a reasonable amount of effort to get it working on all common Linux distros. I was thinking ~5000, since 1000-2000 plan nodes is the most I've seen for a query running successfully in production. Maybe I should do this in a follow-on change, since we probably also want to add a test query at or near this limit. -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Fri, 01 Jun 2018 22:45:19 +0000 Gerrit-HasComments: Yes
