[
https://issues.apache.org/jira/browse/IMPALA-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089328#comment-18089328
]
ASF subversion and git services commented on IMPALA-14647:
----------------------------------------------------------
Commit 1213504bdea08ff43912f38ad3be24dac1123561 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1213504bd ]
IMPALA-14647: Fix truncate for replicated txn tables always delete data
If a transactional table is being replicated by Hive replication, Impala
uses the truncateTable HMS API to truncate it. However, the last
parameter (boolean deleteData) is not set, which causing HMS always
delete the data.
In HiveServer, there is a configuration, "hive.acid.truncate.usebase",
to control whether to create a new base_* dir (so keep the data files)
for truncate operation. This patch fixes the API invocation to respect
this configuration.
Note that for non-transactional tables, the behavior is still deleting
the data files, which is consistent with Hive.
Tests:
- Added test to verify the data files exist after truncate.
- Added util method to fetch effective hadoop and hive configs from the
/hadoop-varz endpoint.
Change-Id: Ia31991baeb2ef8717c387b841b65cff562dbcae0
Reviewed-on: http://gerrit.cloudera.org:8080/23810
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Csaba Ringhofer <[email protected]>
> Truncate transactional table always delete data files for replicated tables
> ---------------------------------------------------------------------------
>
> Key: IMPALA-14647
> URL: https://issues.apache.org/jira/browse/IMPALA-14647
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> Hive has a configuration, hive.acid.truncate.usebase, in hive-site.xml of
> HiveServer2 to control whether to delete the data files in TRUNCATE for
> transactional tables.
> {code:java}
> HIVE_ACID_TRUNCATE_USE_BASE("hive.acid.truncate.usebase", true,
> "If enabled, truncate for transactional tables will not delete the
> data directories,\n" +
> "rather create a new base directory with no datafiles."),{code}
> [https://github.com/apache/hive/blob/e3cb93958ee470b261f44f99fc49716071f05b58/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L3138-L3140]
> Currently, when the table is under Hive replication, Impala uses the
> truncateTable HMS API without passing the deleteData parameter:
> [https://github.com/apache/impala/blob/1970cc709df637f1e416198fb9a6553b2947bffa/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L3560-L3561]
> [https://github.com/apache/impala/blob/1970cc709df637f1e416198fb9a6553b2947bffa/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#L728]
> This results in deleteData=true:
> [https://github.com/apache/hive/blob/e3cb93958ee470b261f44f99fc49716071f05b58/standalone-metastore/metastore-client/src/main/java/org/apache/hadoop/hive/metastore/client/BaseMetaStoreClient.java#L170]
> Impala should set the deleteData parameter based on
> "hive.acid.truncate.usebase".
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
