[
https://issues.apache.org/jira/browse/IMPALA-12865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18052782#comment-18052782
]
ASF subversion and git services commented on IMPALA-12865:
----------------------------------------------------------
Commit 547f96ec04ce2736fbf24d2a7253f549fcbbd167 in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=547f96ec0 ]
IMPALA-12865: Fix wrong lastRefreshEventId set by firing RELOAD events
When enable_reload_events is true, catalogd fires RELOAD events after
REFRESH finishes reloading the table/partition. The RELOAD event id is
also used to update lastRefreshEventId of the table/partition. This is
problematic when enable_skipping_older_events is true. HMS events
generated after the reload and before the RELOAD event will be skipped.
Solution:
Fetch the current HMS notification event id before the table/partition
is refreshed, and set it as lastRefreshEventId on the metadata object.
Testing:
- Manually verified the issue is addressed.
- Added an end-to-end test which is close to real time issue.
Change-Id: I90039da77ec561c5aede44456f88c6650582815b
Reviewed-on: http://gerrit.cloudera.org:8080/21665
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> enable_reload_events breaks enable_skipping_older_events by pushing
> lastRefreshEventId too high
> -----------------------------------------------------------------------------------------------
>
> Key: IMPALA-12865
> URL: https://issues.apache.org/jira/browse/IMPALA-12865
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
>
> When enable_reload_events is true, catalogd fires RELOAD events after REFRESH
> finishes reloading the table/partition. The RELOAD event id is also used to
> update lastRefreshEventId of the table/partition.
> [https://github.com/apache/impala/blob/784971c018c2dc44c53d7c0f366ad49cd8681ac6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6807-L6814]
> This is problematic when enable_skipping_older_events is true. HMS events
> generated after the reload and before the RELOAD event will be skipped.
> [https://github.com/apache/impala/blob/784971c018c2dc44c53d7c0f366ad49cd8681ac6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1219-L1221]
> The issue is identified in the review of
> [https://gerrit.cloudera.org/c/21096/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#6799]
> CC [~hemanth619], [~VenuReddy], [~csringhofer]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
