[
https://issues.apache.org/jira/browse/IMPALA-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17360470#comment-17360470
]
ASF subversion and git services commented on IMPALA-10648:
----------------------------------------------------------
Commit 975047f43c39225ac397e22192a5999f5a270c70 in impala's branch
refs/heads/master from Sourabh Goyal
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=975047f ]
IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.
Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Reviewed-on: http://gerrit.cloudera.org:8080/17298
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Vihang Karajgaonkar <[email protected]>
> Invalidate catalogd cache for non transactional tables when create/alter/drop
> HMS apis are accessed
> ---------------------------------------------------------------------------------------------------
>
> Key: IMPALA-10648
> URL: https://issues.apache.org/jira/browse/IMPALA-10648
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog
> Reporter: Sourabh Goyal
> Priority: Major
> Fix For: Impala 4.1
>
>
> IMPALA-10613 introduced changes to expose table/partition metadata stored in
> catalog cache over HMS APIs.
> In this task, we invalidate the non transactional table from cache if HMS DDL
> apis like create/alter/drop table/partition are accessed from catalogd's
> metastore server. Any subsequent get table request fetches the table from HMS
> and also loads it in cache. This ensures that any get_table/get_partition
> requests after ddl operations on the same table return the most updated table
> cc - [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
