Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14527
to look at the new patch set (#4).
Change subject: IMPALA-9071: Handle translated external HDFS table in CATS
......................................................................
IMPALA-9071: Handle translated external HDFS table in CATS
After upgrading Hive-3 to a version containing HIVE-22158, it's not
allowed for managed tables to be non transactional. Creating non ACID
tables will result in creating an external table with table property
'external.table.purge' set to true.
In Hive-3, the default location of external HDFS tables will locate in
'metastore.warehouse.external.dir' if it's set. This property is added
by HIVE-19837 in Hive 2.7, but hasn't been added to Hive in cdh6 yet.
In CTAS statement, we create a temporary HMS Table for the analysis on
the Insert part. The table path is created assuming it's a managed
table, and the Insert part will use this path for insertion. However, in
Hive-3, the created table is translated to an external table. It's not
the same as we passed to the HMS API. The created table is located in
'metastore.warehouse.external.dir', while the table path we assumed is
in 'metastore.warehouse.dir'. This introduces bugs when these two
properties are different. CTAS statement will create table in one place
and insert data in another place.
This patch adds a new method in MetastoreShim to wrap the difference for
getting the default table path for non transactional tables between
Hive-2 and Hive-3.
This patch also bumps the CDP version to contain HIVE-22158. Note that
Ranger version also bumps from 1.2 to 2.0 due to this CDP upgrade, which
causes some test failures. They are disabled for follow-up fixs in
IMPALA-9047.
Changes in the infra:
- To support customizing hive configuration, add a env var,
CUSTOM_CLASSPATH in bin/set-classpath.sh to be put in front of
existing CLASSPATH. The customized hive-site.xml should be put inside
CUSTOM_CLASSPATH.
- Change hive-site.xml.py to generate a hive-site.xml with non default
'metastore.warehouse.external.dir'
- Add an option, --env_vars, in bin/start-impala-cluster.py to pass
down CUSTOM_CLASSPATH.
Tests:
- Add a custom cluster test to start Hive with
metastore.warehouse.external.dir being set to non default value. Run
it locally using CDP components with HIVE-22158.
- Run CORE tests using CDH components
Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9
---
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/set-classpath.sh
M bin/start-impala-cluster.py
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M
fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
M fe/src/test/resources/hive-site.xml.py
M testdata/pom.xml
M tests/authorization/test_authorized_proxy.py
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
A tests/custom_cluster/test_custom_hive_configs.py
15 files changed, 253 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14527/4
--
To view, visit http://gerrit.cloudera.org:8080/14527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9
Gerrit-Change-Number: 14527
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
