Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13224 )
Change subject: IMPALA-8369 (part 2): Hive 3: switch to Tez-on-YARN execution ...................................................................... IMPALA-8369 (part 2): Hive 3: switch to Tez-on-YARN execution This switches away from Tez local mode to tez-on-YARN. After spending a couple of days trying to debug issues with Tez local mode, it seemed like it was just going to be too much of a lift. This patch switches on the starting of a Yarn RM and NM when USE_CDP_HIVE is enabled. It also switches to a new yarn-site.xml with a minimized set of configurations, generated by the new python templating. In order for everything to work properly I also had to update the Hadoop dependency to come from CDP instead of CDH when using CDP Hive. Otherwise, the classpath of the launched Tez containers had conflicting versions of various Hadoop classes which caused tasks to fail. I verified that this fixes concurrent query execution by running queries in parallel in two beeline sessions. With local mode, these queries would periodically fail due to various races (HIVE-21682). I'm also able to get farther along in data loading. Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d Reviewed-on: http://gerrit.cloudera.org:8080/13224 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Todd Lipcon <[email protected]> --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/generate_xml_config.py M bin/impala-config.sh M bin/jenkins/critique-gerrit-review.py M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java M fe/src/test/resources/hive-site.xml.py M shaded-deps/pom.xml M testdata/cluster/admin A testdata/cluster/node_templates/common/etc/hadoop/conf/capacity-scheduler.xml A testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py D testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.tmpl 13 files changed, 389 insertions(+), 173 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13224 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d Gerrit-Change-Number: 13224 Gerrit-PatchSet: 11 Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Sudhanshu Arora <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Yongzhi Chen <[email protected]>
