[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-19 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10051/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/10051/3//COMMIT_MSG@7
PS3, Line 7: 6917
> Um... I know the horse is already gone, but this is actually IMPALA-5917.
:/

We'll survive.



--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 19 Apr 2018 22:32:42 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-19 Thread Laszlo Gaal (Code Review) 
Laszlo Gaal has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10051/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/10051/3//COMMIT_MSG@7
PS3, Line 7: 6917
Um... I know the horse is already gone, but this is actually IMPALA-5917.



--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 19 Apr 2018 22:28:54 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 19 Apr 2018 20:31:33 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..

IMPALA-6917: Limit impalad mem-limit to 12GB.

This changes the memlimit choosing code in start-impala-cluster to have
a maximum of 12GB. In 68GB machines, this has the effect of reducing the
memlimit from ~15.8GB to 12GB. On machines with less than 51.4 GB of
RAM, this has no effect.

I'm tinkering with this threshold because ASAN builds sometimes
die at the hands of the OOM killer on m2.4xlarge machines (68GB).
My theory for why it's only sometimes is that during the parallel
tests, our memory usage could vary widely depending on which tests
and queries execute.

End-users don't see this code; this is only used by our minicluster
tests to test Impala.

I have run the ASAN build with this change successfully, though I've
found this particular OOM seems to come and go.

Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Reviewed-on: http://gerrit.cloudera.org:8080/10051
Reviewed-by: Philip Zeyliger 
Tested-by: Impala Public Jenkins 
---
M bin/start-impala-cluster.py
1 file changed, 8 insertions(+), 2 deletions(-)

Approvals:
  Philip Zeyliger: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2332/


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 19 Apr 2018 16:37:44 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-18 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 1:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2329/


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 19 Apr 2018 05:20:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-18 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2329/


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 18 Apr 2018 21:28:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-16 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2313/


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Tue, 17 Apr 2018 00:58:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-16 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2313/


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Mon, 16 Apr 2018 18:14:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-12 Thread Vuk Ercegovac (Code Review) 
Vuk Ercegovac has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10051 )

Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 12 Apr 2018 21:47:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6917: Limit impalad mem-limit to 12GB.

2018-04-12 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10051


Change subject: IMPALA-6917: Limit impalad mem-limit to 12GB.
..

IMPALA-6917: Limit impalad mem-limit to 12GB.

This changes the memlimit choosing code in start-impala-cluster to have
a maximum of 12GB. In 68GB machines, this has the effect of reducing the
memlimit from ~15.8GB to 12GB. On machines with less than 51.4 GB of
RAM, this has no effect.

I'm tinkering with this threshold because ASAN builds sometimes
die at the hands of the OOM killer on m2.4xlarge machines (68GB).
My theory for why it's only sometimes is that during the parallel
tests, our memory usage could vary widely depending on which tests
and queries execute.

End-users don't see this code; this is only used by our minicluster
tests to test Impala.

I have run the ASAN build with this change successfully, though I've
found this particular OOM seems to come and go.

Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
---
M bin/start-impala-cluster.py
1 file changed, 8 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/10051/1
--
To view, visit http://gerrit.cloudera.org:8080/10051
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Gerrit-Change-Number: 10051
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger