[
https://issues.apache.org/jira/browse/IMPALA-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855089#comment-17855089
]
ASF subversion and git services commented on IMPALA-12712:
----------------------------------------------------------
Commit f98da3315e1e4744ad0e49405a4d1c7f98be85ae in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f98da3315 ]
IMPALA-12712: Invalidate metadata on table should set better
createEventId
"INVALIDATE METADATA <table>" can be used to bring up a table in
Impala's catalog cache if the table exists in HMS. Currently,
createEventId for such tables are always set as -1 which will lead to
always removing the table. Sequence of drop table + create table +
invalidate table can lead to flaky test failures like IMPALA-12266.
Solution:
When Invalidate metadata <table> is fired, fetch the latest eventId
from HMS and set it as createEventId for the table, so that drop table
event that happend before invalidate query will be ignored without
removing the table from cache.
Note: Also removed an unnecessary RPC call to HMS to get table object
since we alrady have required info in table metadata rpc call.
Testing:
- Added an end-to-end test to verify that drop table event happened
before time shouldn't remove the metadata object from cache.
Change-Id: Iff6ac18fe8d9e7b25cc41c7e41eecde251fbccdd
Reviewed-on: http://gerrit.cloudera.org:8080/21402
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> INVALIDATE METADATA <table> should set a better createEventId
> -------------------------------------------------------------
>
> Key: IMPALA-12712
> URL: https://issues.apache.org/jira/browse/IMPALA-12712
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Labels: catalog-2024
>
> "INVALIDATE METADATA <table>" can be used to bring up a table in Impala's
> catalog cache if the table exists in HMS. For instance, when HMS event
> processing is disabled, we can use it in Impala to bring up tables that are
> created outside Impala.
> The createEventId for such tables are always set as -1:
> [https://github.com/apache/impala/blob/6ddd69c605d4c594e33fdd39a2ca888538b4b8d7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2243-L2246]
> This is problematic when event-processing is enabled. DropTable events and
> RenameTable events use the createEventId to decide whether to remove the
> table in catalog cache. -1 will lead to always removing the table. Though it
> might be added back shortly in follow-up CreateTable events, in the period
> between them the table is missing in Impala, causing test failures like
> IMPALA-12266.
> A simpler reproducing of the issue is creating a table in Hive and launching
> Impala with a long event polling interval to mimic the delay on events. Note
> that we start Impala cluster after creating the table so Impala don't need to
> process the CREATE_TABLE event.
> {noformat}
> hive> create table debug_tbl (i int);
> bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=60
> {noformat}
> Drop the table in Impala and recreate it in Hive, so it doesn't exist in the
> catalog cache but exist in HMS. Run "INVALIDATE METADATA <table>" in Impala
> to bring it up before the DROP_TABLE event come.
> {noformat}
> impala> drop table debug_tbl;
> hive> create table debug_tbl (i int, j int);
> impala> invalidate metadata debug_tbl;
> {noformat}
> The table will be dropped by the DROP_TABLE event and then added back by the
> CREATE_TABLE event. Shown in catalogd logs:
> {noformat}
> I0115 16:30:15.376713 3208 JniUtil.java:177]
> 02457b6d5f174d1f:3bdeee1400000000] Finished execDdl request: DROP_TABLE
> default.debug_tbl issued by quanlong. Time spent: 417ms
> I0115 16:30:23.390962 3208 CatalogServiceCatalog.java:2777]
> 1840bd101f78d611:22079a5a00000000] Invalidating table metadata:
> default.debug_tbl
> I0115 16:30:23.404150 3208 Table.java:234]
> 1840bd101f78d611:22079a5a00000000] createEventId_ for table:
> default.debug_tbl set to: -1
> I0115 16:30:23.405138 3208 JniUtil.java:177]
> 1840bd101f78d611:22079a5a00000000] Finished resetMetadata request: INVALIDATE
> TABLE default.debug_tbl issued by quanlong. Time spent: 17ms
> I0115 16:30:55.108006 32760 MetastoreEvents.java:637] EventId: 8668853
> EventType: DROP_TABLE Successfully removed table default.debug_tbl
> I0115 16:30:55.108459 32760 MetastoreEvents.java:637] EventId: 8668855
> EventType: CREATE_TABLE Successfully added table default.debug_tbl
> {noformat}
> CC [~VenuReddy], [~hemanth619]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
