Hello Vihang Karajgaonkar, Joe McDonnell,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/13224
to review the following change.
Change subject: Hive 3: switch to Tez-on-YARN execution
......................................................................
Hive 3: switch to Tez-on-YARN execution
This switches away from Tez local mode to tez-on-YARN. After spending a
couple of days trying to debug issues with Tez local mode, it seemed
like it was just going to be too much of a lift.
This patch switches on the starting of a Yarn RM and NM when
USE_CDP_HIVE is enabled. It also switches to a new yarn-site.xml with a
minimized set of configurations, generated by the new python templating.
In order for everything to work properly I also had to update the Hadoop
dependency to come from CDP instead of CDH when using CDP Hive.
Otherwise, the classpath of the launched Tez containers had conflicting
versions of various Hadoop classes which caused tasks to fail.
I verified that this fixes concurrent query execution by running queries
in parallel in two beeline sessions. With local mode, these queries
would periodically fail due to various races (HIVE-21682). I'm also able
to get farther along in data loading.
Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/generate_xml_config.py
M bin/impala-config.sh
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/test/resources/hive-site.xml.py
M shaded-deps/pom.xml
M testdata/cluster/admin
A testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
D testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.tmpl
11 files changed, 129 insertions(+), 171 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/13224/1
--
To view, visit http://gerrit.cloudera.org:8080/13224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d
Gerrit-Change-Number: 13224
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
