[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-11 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..

IMPALA-6384: RequestPoolService should honor custom group mapping config

Due to the way in which we instantiate fair scheduler allocation
loader, we donot read the config overrides from the HDFS config
files.

This is an unexpected behavior from users' POV since we typically
support overrides like custom user -> group mapping via HDFS
config (for ex: LDAPGroupsMapping) that eventually affects the
query -> pool assignment.

Fix: This patch loads the hadoop default configuration so that the
underlying QueuePlacementPolicy is based on user specified overrides.

Testing (manual): Changed the core-site.xml to use LDAPGroupsMapping
instead of the default ShellBasedUnixGroupsMapping and confirmed that
the correct group mapping plugin is loaded, by adding additional logging.

Also, modified TestRequestPoolService to assert that the core-site xml
overrides are loaded.

Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Reviewed-on: http://gerrit.cloudera.org:8080/9000
Reviewed-by: Bharath Vissapragada 
Tested-by: Impala Public Jenkins
---
M 
common/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
4 files changed, 32 insertions(+), 2 deletions(-)

Approvals:
  Bharath Vissapragada: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-11 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 3: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 22:52:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-11 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1708/


--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 19:16:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-11 Thread Bharath Vissapragada (Code Review) 
Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 3: Code-Review+2

(1 comment)

Thanks Zoram and Tim. Carrying +2.

http://gerrit.cloudera.org:8080/#/c/9000/2/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
File testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl:

http://gerrit.cloudera.org:8080/#/c/9000/2/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl@116
PS2, Line 116: overridde
> nit: overriden
Done



--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 19:16:11 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-11 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 2: Code-Review+2

(1 comment)

The test seems good since it's at least testing the root cause of the problem

http://gerrit.cloudera.org:8080/#/c/9000/2/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
File testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl:

http://gerrit.cloudera.org:8080/#/c/9000/2/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl@116
PS2, Line 116: overriden
nit: overriden



--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 19:06:28 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-11 Thread Zoram Thanga (Code Review) 
Zoram Thanga has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 2: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 18:59:43 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-10 Thread Bharath Vissapragada (Code Review) 
Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9000/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9000/1//COMMIT_MSG@21
PS1, Line 21: Testing(manual): Changed the core-site.xml to use 
LDAPGroupsMapping
> Can we add a regression test to TestRequestPool? This seems subtle and easy
Agree. It doesn't seem easy enough to use LDAPGroupsMapping directly since it 
requires a valid LDAP URI to work, so I added a test to make sure that the 
core-site.xml configs are loaded. Please let me know if thats okay



--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 07:53:11 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-10 Thread Bharath Vissapragada (Code Review) 
Hello Dimitris Tsirogiannis, Zoram Thanga, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9000

to look at the new patch set (#2).

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..

IMPALA-6384: RequestPoolService should honor custom group mapping config

Due to the way in which we instantiate fair scheduler allocation
loader, we donot read the config overrides from the HDFS config
files.

This is an unexpected behavior from users' POV since we typically
support overrides like custom user -> group mapping via HDFS
config (for ex: LDAPGroupsMapping) that eventually affects the
query -> pool assignment.

Fix: This patch loads the hadoop default configuration so that the
underlying QueuePlacementPolicy is based on user specified overrides.

Testing (manual): Changed the core-site.xml to use LDAPGroupsMapping
instead of the default ShellBasedUnixGroupsMapping and confirmed that
the correct group mapping plugin is loaded, by adding additional logging.

Also, modified TestRequestPoolService to assert that the core-site xml
overrides are loaded.

Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
---
M 
common/yarn-extras/src/main/java/org/apache/impala/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
4 files changed, 32 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/9000/2
--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-10 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9000 )

Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9000/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9000/1//COMMIT_MSG@21
PS1, Line 21: Testing(manual): Changed the core-site.xml to use 
LDAPGroupsMapping
Can we add a regression test to TestRequestPool? This seems subtle and easy to 
break.



--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoram Thanga 
Gerrit-Comment-Date: Thu, 11 Jan 2018 01:26:59 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6384: RequestPoolService should honor custom group mapping config

2018-01-10 Thread Bharath Vissapragada (Code Review) 
Bharath Vissapragada has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9000


Change subject: IMPALA-6384: RequestPoolService should honor custom group 
mapping config
..

IMPALA-6384: RequestPoolService should honor custom group mapping config

Due to the way in which we instantiate fair scheduler allocation
loader, we donot read the config overrides from the HDFS config
files.

This is an unexpected behavior from users' POV since we typically
support overrides like custom user -> group mapping via HDFS
config (for ex: LDAPGroupsMapping) that eventually affects the
query -> pool assignment.

Fix: This patch loads the hadoop default configuration so that the
underlying QueuePlacementPolicy is based on user specified overrides.

Testing(manual): Changed the core-site.xml to use LDAPGroupsMapping
instead of the default ShellBasedUnixGroupsMapping and confirmed that
the correct group mapping plugin is loaded, by adding additional logging.

Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
---
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
1 file changed, 3 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/9000/1
--
To view, visit http://gerrit.cloudera.org:8080/9000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibb93870c0cc37e2432a643a274931f1d3d13fb96
Gerrit-Change-Number: 9000
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada