[
https://issues.apache.org/jira/browse/IMPALA-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482917#comment-16482917
]
Pranay Singh edited comment on IMPALA-6994 at 5/21/18 7:04 PM:
---------------------------------------------------------------
Checked the code in updatePartitionsFromHms() looks like there is little room
of optimization here, skipping CREATE/RECREATE/DROP of a partition will cause
inconsistency between HMS and catalog will result in failure of DML statements
like INSERT, when used with SYNC_DDL option which requires underlying data and
metadata changes to be propagated to all Impala nodes.
-Pranay
was (Author: pranay_singh):
Checked the code in updatePartitionsFromHms() looks like there is little room
of optimization here, skipping CREATE/RECREATE/DROP of a partition will cause
inconsistency between HMS and catalog
this will result in failure of DML statements like INSERT when used with
SYNC_DDL option which requires underlying data and metadata changes to be
propagated to all Impala nodes.
-Pranay
> Avoid reloading a table's HMS data for file-only operations
> -----------------------------------------------------------
>
> Key: IMPALA-6994
> URL: https://issues.apache.org/jira/browse/IMPALA-6994
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Affects Versions: Impala 2.12.0
> Reporter: Balazs Jeszenszky
> Assignee: Pranay Singh
> Priority: Major
>
> Reloading file metadata for HDFS tables (e.g. as a final step in an 'insert')
> is done via
> https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L628
> , which calls
> https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1243
> HdfsTable.load has no option to only load file metadata. HMS metadata will
> also be reloaded every time, which is an unnecessary overhead (and potential
> point of failure) when adding files to existing locations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
