Quanlong Huang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14527 )
Change subject: [WIP] IMPALA-9071: Handle translated external HDFS table in CATS ...................................................................... [WIP] IMPALA-9071: Handle translated external HDFS table in CATS After upgrading Hive-3 to a version containing HIVE-22158, it's not allowed for managed tables to be non transactional. Creating non ACID tables will result in creating an external table with table property 'external.table.purge' set to true. In Hive-3, the default location of external HDFS tables will locate in 'metastore.warehouse.external.dir' if it's set. This property is added by HIVE-19837 in Hive 2.7, but hasn't been added to Hive in cdh6 yet. In CTAS statement, we create a temporary HMS Table for the analysis on the Insert part. The table path is created assuming it's a managed table, and the Insert part will use this path for insertion. However, in Hive-3, the created table is translated to an external table. It's not the same as we passed to the HMS API. The created table is located in 'metastore.warehouse.external.dir', while the table path we assumed is in 'metastore.warehouse.dir'. This introduces bugs when these two properties are different. CTAS statement will create table in one place and insert data in another place. This patch adds a new method in MetastoreShim to wrap the difference for getting the default table path for non transactional tables between Hive-2 and Hive-3. This patch also bumps the CDP version to contain HIVE-22158. Test Plan: - Adding a custom cluster test to start Hive with external.dir being set. Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 --- M bin/impala-config.sh M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M testdata/pom.xml 5 files changed, 55 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14527/2 -- To view, visit http://gerrit.cloudera.org:8080/14527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I460a57dc877ef68ad7dd0864a33b1599b1e9a8d9 Gerrit-Change-Number: 14527 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
