[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 13 Mar 2018 00:39:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. We've observed empirically that giving Impala 80% of system memory doesn't leave enough room for the minicluster and ASAN overhead, leading to the OOM killer striking during test runs (sometimes). This commit reduces the threshold to 70%. This commit also reduces the memory usage of semi-joins-exhaustive.test by roughly halving the number of records it deals with. This was necessary for tests to pass on a machine with 32GB of RAM. Testing: I've run the ASAN build (more) happily with this change. I've run exhaustive tests on a 32GB machine. Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Reviewed-on: http://gerrit.cloudera.org:8080/9395 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M bin/start-impala-cluster.py M testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test 2 files changed, 13 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2084/ -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 12 Mar 2018 20:55:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/9395/2/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test File testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test: http://gerrit.cloudera.org:8080/#/c/9395/2/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test@9 PS2, Line 9: # of the relevant fragment is 3.6GB when tested.) > It might be worth reducing it further, since we could potentially still hav I've not done this at the moment. -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 12 Mar 2018 20:53:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/9395/2/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test File testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test: http://gerrit.cloudera.org:8080/#/c/9395/2/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test@9 PS2, Line 9: # of the relevant fragment is 3.6GB when tested.) It might be worth reducing it further, since we could potentially still have problems if it was running with another test that had high memory consumption. Anyway, this is clearly an improvement. -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 09 Mar 2018 21:31:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 09 Mar 2018 21:30:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 2: Tim, Please take a look at semi-joins-exhaustive.test? Thanks! -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 09 Mar 2018 19:45:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Hello Tim Armstrong, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9395 to look at the new patch set (#2). Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. We've observed empirically that giving Impala 80% of system memory doesn't leave enough room for the minicluster and ASAN overhead, leading to the OOM killer striking during test runs (sometimes). This commit reduces the threshold to 70%. This commit also reduces the memory usage of semi-joins-exhaustive.test by roughly halving the number of records it deals with. This was necessary for tests to pass on a machine with 32GB of RAM. Testing: I've run the ASAN build (more) happily with this change. I've run exhaustive tests on a 32GB machine. Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 --- M bin/start-impala-cluster.py M testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test 2 files changed, 13 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/9395/2 -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 1: Code-Review+2 I'm ok with this once you fix my comments. -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Mar 2018 21:14:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 1: (1 comment) I'm ok with this so long as we can confirm that it works ok for existing builds. I'd also try running DEBUG/exhaustive on a 32GB machine - the exhaustive build is a bit more memory hungry in some places so would be good to check that it still works. http://gerrit.cloudera.org:8080/#/c/9395/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9395/1//COMMIT_MSG@14 PS1, Line 14: Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Change-Id is above Testing -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 22 Feb 2018 17:42:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 1: I see this happening in multiple environments, but the environment I see this most uses a little bit more memory for HiveServer2 than other places. (I have another change that tries to get rid of Yarn which would free up some space.) The math here is fundamentally wrong. If we wanted to do it correctly, we'd say: whole_machine = X GB minicluster = Y GB what's left = X - Y random_spare_space = 2 GB per-impalad: (X - Y - 2) / 3 And, finally, we'd assert that per-impalad is big enough. I've added enough monitoring in https://gerrit.cloudera.org/#/c/9291/ to eventually have enough data to make this more explicit, but given that we've muddled along similarly for a while, I don't think it's worth embarking on a bigger effort like above. -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 22 Feb 2018 17:31:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9395 ) Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. Patch Set 1: Code-Review+1 Would still be good to understand why we are hitting it more frequently now. Maybe it is related to the new buffer pool which might hang on to memory more aggressively than tcmalloc? -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Comment-Date: Thu, 22 Feb 2018 17:15:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9395 Change subject: IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. .. IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. We've observed empirically that giving Impala 80% of system memory doesn't leave enough room for the minicluster and ASAN overhead, leading to the OOM killer striking during test runs (sometimes). This commit reduces the threshold to 70%. Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Testing: I've run the ASAN build (more) happily with this change. --- M bin/start-impala-cluster.py 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/9395/1 -- To view, visit http://gerrit.cloudera.org:8080/9395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Gerrit-Change-Number: 9395 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger
