[
https://issues.apache.org/jira/browse/IMPALA-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach Amsden closed IMPALA-5920.
-------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.11.0
IMPALA-5920: Remove admission control dependency on YARN RM jar
Impala's admission controller relies on the YARN
fair-scheduler.xml for configuration. That configuration is
loaded using YARN directly (ie. as a library by the
frontend). In Hadoop 3, a number of changes were made to the
YARN resourcemanager which break Impala. While we eventually
want to rethink the admission control configuration
(IMPALA-4159), in the meantime we at least should avoid
using unsupported YARN APIs.
This patch removes the fe dependency on the YARN artifact
'hadoop-yarn-server-resourcemanager' which contains private
APIs and isn't meant to be used as a library.
A subset of the required code has been added in
'common/yarn-extras', taken from Hadoop 2.6.0 in CDH, e.g.
see [1]. The code is added in packages 'org.apache.impala.*'
instead of 'org.apache.yarn.*'. Some code could be copied
as-is, those files are marked with the comment:
//YARNUTIL: VANILLA
Files that required some modifications are marked with:
//YARNUTIL: MODIFIED
Or, if all code except a dummy interface could be added:
//YARNUTIL: DUMMY IMPL
The goal the 'yarn-extras' is to make Impala's handling of
the AC configuration self-sufficient, i.e. it shouldn't
matter what version of Hadoop exists. As-is, this was tested
and found to work when Hadoop 2.6 is installed. Because
the yarn-extras/pom.xml still references hadoop-common,
hadoop-yarn-common, and hadoop-yarn-api, additional testing
will be required to ensure Impala using yarn-extras works
when installed along side Hadoop 3.
That testing for Hadoop 3 will be done later. Future changes
will make any other changes required for existing code to
work when Hadoop 3 is installed.
Testing:
* Ran existing tests on master.
1:
https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd_cdh_package_tarball_512.html
Change-Id: I7efdd8ebea298836ca2a82c0a4ae037ac9285bcf
Reviewed-on: http://gerrit.cloudera.org:8080/8035
Reviewed-by: Matthew Jacobs <[email protected]>
Tested-by: Matthew Jacobs <[email protected]>
> Remove admission control dependency on YARN resourcemanager
> -----------------------------------------------------------
>
> Key: IMPALA-5920
> URL: https://issues.apache.org/jira/browse/IMPALA-5920
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 2.10.0
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Priority: Critical
> Labels: admission-control, resource-management, rm
> Fix For: Impala 2.11.0
>
>
> Impala's admission controller relies on the YARN fair-scheduler.xml for
> configuration. That configuration is loaded using YARN directly (ie. as a
> library by the frontend). In Hadoop 3, a number of changes were made to the
> YARN resourcemanager which break Impala. While we eventually want to rethink
> the admission control configuration (IMPALA-4159), in the meantime we at
> least should avoid using unsupported YARN APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
