[
https://issues.apache.org/jira/browse/HIVE-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-14688:
----------------------------------
Description:
This should be committed to when Hive moves to Hadoop 2.8
In Hadoop 2.8.0 TDE trash collection was fixed through HDFS-8831. This enables
us to make drop table calls for Hive managed tables where Hive metastore
warehouse directory is in encrypted zone. However even with the feature in
HDFS, Hive drop table currently fail:
{noformat}
$ hdfs crypto -listZones
/apps/hive/warehouse key2
$ hdfs dfs -ls /apps/hive/warehouse
Found 1 items
drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54 /apps/hive/warehouse/.Trash
hive> create table abc(a string, b int);
OK
Time taken: 5.538 seconds
hive> dfs -ls /apps/hive/warehouse;
Found 2 items
drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54
/apps/hive/warehouse/.Trash
drwxrwxrwx - deepesh hdfs 0 2016-09-01 17:15 /apps/hive/warehouse/abc
hive> drop table if exists abc;
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
default.abc because it is in an encryption zone and trash is enabled. Use
PURGE option to skip trash.)
{noformat}
The problem lies here:
{code:title=metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java}
private void checkTrashPurgeCombination(Path pathToData, String objectName,
boolean ifPurge)
...
if (trashEnabled) {
try {
HadoopShims.HdfsEncryptionShim shim =
ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf),
hiveConf);
if (shim.isPathEncrypted(pathToData)) {
throw new MetaException("Unable to drop " + objectName + " because
it is in an encryption zone" +
" and trash is enabled. Use PURGE option to skip trash.");
}
} catch (IOException ex) {
MetaException e = new MetaException(ex.getMessage());
e.initCause(ex);
throw e;
}
}
{code}
As we can see that we are making an assumption that delete wouldn't be
successful in encrypted zone. We need to modify this logic.
was:
In Hadoop 2.8.0 TDE trash collection was fixed through HDFS-8831. This enables
us to make drop table calls for Hive managed tables where Hive metastore
warehouse directory is in encrypted zone. However even with the feature in
HDFS, Hive drop table currently fail:
{noformat}
$ hdfs crypto -listZones
/apps/hive/warehouse key2
$ hdfs dfs -ls /apps/hive/warehouse
Found 1 items
drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54 /apps/hive/warehouse/.Trash
hive> create table abc(a string, b int);
OK
Time taken: 5.538 seconds
hive> dfs -ls /apps/hive/warehouse;
Found 2 items
drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54
/apps/hive/warehouse/.Trash
drwxrwxrwx - deepesh hdfs 0 2016-09-01 17:15 /apps/hive/warehouse/abc
hive> drop table if exists abc;
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
default.abc because it is in an encryption zone and trash is enabled. Use
PURGE option to skip trash.)
{noformat}
The problem lies here:
{code:title=metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java}
private void checkTrashPurgeCombination(Path pathToData, String objectName,
boolean ifPurge)
...
if (trashEnabled) {
try {
HadoopShims.HdfsEncryptionShim shim =
ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf),
hiveConf);
if (shim.isPathEncrypted(pathToData)) {
throw new MetaException("Unable to drop " + objectName + " because
it is in an encryption zone" +
" and trash is enabled. Use PURGE option to skip trash.");
}
} catch (IOException ex) {
MetaException e = new MetaException(ex.getMessage());
e.initCause(ex);
throw e;
}
}
{code}
As we can see that we are making an assumption that delete wouldn't be
successful in encrypted zone. We need to modify this logic.
> Hive drop call fails in presence of TDE
> ---------------------------------------
>
> Key: HIVE-14688
> URL: https://issues.apache.org/jira/browse/HIVE-14688
> Project: Hive
> Issue Type: Bug
> Components: Security
> Affects Versions: 1.2.1, 2.0.0
> Reporter: Deepesh Khandelwal
> Assignee: Wei Zheng
> Attachments: HIVE-14688.1.patch, HIVE-14688.2.patch
>
>
> This should be committed to when Hive moves to Hadoop 2.8
> In Hadoop 2.8.0 TDE trash collection was fixed through HDFS-8831. This
> enables us to make drop table calls for Hive managed tables where Hive
> metastore warehouse directory is in encrypted zone. However even with the
> feature in HDFS, Hive drop table currently fail:
> {noformat}
> $ hdfs crypto -listZones
> /apps/hive/warehouse key2
> $ hdfs dfs -ls /apps/hive/warehouse
> Found 1 items
> drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54
> /apps/hive/warehouse/.Trash
> hive> create table abc(a string, b int);
> OK
> Time taken: 5.538 seconds
> hive> dfs -ls /apps/hive/warehouse;
> Found 2 items
> drwxrwxrwt - hdfs hdfs 0 2016-09-01 02:54
> /apps/hive/warehouse/.Trash
> drwxrwxrwx - deepesh hdfs 0 2016-09-01 17:15
> /apps/hive/warehouse/abc
> hive> drop table if exists abc;
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop
> default.abc because it is in an encryption zone and trash is enabled. Use
> PURGE option to skip trash.)
> {noformat}
> The problem lies here:
> {code:title=metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java}
> private void checkTrashPurgeCombination(Path pathToData, String objectName,
> boolean ifPurge)
> ...
> if (trashEnabled) {
> try {
> HadoopShims.HdfsEncryptionShim shim =
>
> ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf),
> hiveConf);
> if (shim.isPathEncrypted(pathToData)) {
> throw new MetaException("Unable to drop " + objectName + "
> because it is in an encryption zone" +
> " and trash is enabled. Use PURGE option to skip trash.");
> }
> } catch (IOException ex) {
> MetaException e = new MetaException(ex.getMessage());
> e.initCause(ex);
> throw e;
> }
> }
> {code}
> As we can see that we are making an assumption that delete wouldn't be
> successful in encrypted zone. We need to modify this logic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
