[
https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812214#comment-17812214
]
Maxwell Guo commented on IMPALA-12771:
--------------------------------------
with alter parition event, we may found that database/table is not found , or
table is not loaded see [here
|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1057].
then we just skipped the and made a debug log , the log is : " Ignoring the
event since the table is not found ".
But actually, we skipped the handling of these events when table is not found
.So I think we can also +1 on the events-skipped metric if table is not found
or table is IncompleteTable or table was remove in catalog.
Besides, we just mark the events-skipped metric for event process with
isOlderEvent method and isSelfEvent see
[isSelfEvent|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L857]
and
[isOlderEvent|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1198]
But as for
[canBeSkipped|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1703]
method, the metric is not +1. I think we can also add here.
> Impala catalogd events-skipped may mark the wrong number
> --------------------------------------------------------
>
> Key: IMPALA-12771
> URL: https://issues.apache.org/jira/browse/IMPALA-12771
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Maxwell Guo
> Assignee: Maxwell Guo
> Priority: Minor
>
> See the description of [event-skipped
> metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237]
>
> {code:java}
> // total number of events which are skipped because of the flag setting or
> // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were
> ignored
> // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in
> the catalogd.
> {code}
>
> As for CREATE and DROP event on Database/Table/Partition (Also AddPartition
> is inclued) when we found that the table/database when the database or table
> is not found in the cache then we will skip the event process and make the
> event-skipped metric +1.
> But I found that there is some question here for alter table and Reload event:
> * For alter table if renaming a table , the events-skipped metric will also
> +1 ,see [oldTblRemoved to be false
> |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653]
> * For Reload event that is not describe in the description of events-skipped,
> but the value is +1 when is oldevent;
> * Besides if the table is in blacklist the metric will also +1
> In summary, I think this description is inconsistent with the actual
> implementation.
> So can we also mark the events-skipped metric for alter partition events and
> modify the
> description to be all the events skipped
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
