[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850908#comment-17850908 ] Maxwell Guo commented on IMPALA-12771: -- ping again , and update the pr for the latest master branch code in case of merge conflict. [~mylogi...@gmail.com][~stigahuang][~VenuReddy] > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835458#comment-17835458 ] Maxwell Guo commented on IMPALA-12771: -- ping [~mylogi...@gmail.com][~stigahuang][~VenuReddy]:D > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833010#comment-17833010 ] Maxwell Guo commented on IMPALA-12771: -- hi [~hemanth619], thanks for your review comments, I have responded to your comments and updated the latest code. Looking forward to your reply. :) > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828156#comment-17828156 ] Maxwell Guo commented on IMPALA-12771: -- ping [~stigahuang][~mylogi...@gmail.com] [~VenuReddy]:D > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826933#comment-17826933 ] Maxwell Guo commented on IMPALA-12771: -- [~stigahuang][~mylogi...@gmail.com][~VenuReddy] Hi can you help to take a look again ? > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820535#comment-17820535 ] Maxwell Guo commented on IMPALA-12771: -- The build failed, let me take a look at this case. > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819210#comment-17819210 ] Maxwell Guo commented on IMPALA-12771: -- done , publish it now > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819203#comment-17819203 ] Quanlong Huang commented on IMPALA-12771: - [~maxwellguo] Thanks for uploading a patch. Please publish it by clicking the "publish" button so it can be visible to us. Or if you want to keep it private, add us to the reviewer list. > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819161#comment-17819161 ] Maxwell Guo commented on IMPALA-12771: -- The initial version is here https://gerrit.cloudera.org/#/c/21045/ and I am doing local testing at the same time. CC [~mylogi...@gmail.com][~stigahuang][~VenuReddy] ,let me know if there is something obviously wrong with my modifications. > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812553#comment-17812553 ] Maxwell Guo commented on IMPALA-12771: -- Thanks for reminding me, I think I have used it before > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812545#comment-17812545 ] Quanlong Huang commented on IMPALA-12771: - [~maxwellguo] Thanks for raising this! Agree that the events-skipped metric is inconsistent in some cases. Feel free to submit a patch. Note that we use Gerrit for code review: https://cwiki.apache.org/confluence/display/IMPALA/Using+Gerrit+to+submit+and+review+patches CC [~hemanth619], [~VenuReddy] > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812236#comment-17812236 ] Maxwell Guo commented on IMPALA-12771: -- Besides, I found an interesting piece of codeļ¼ [ here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1023] as the TableLoadingException and DatabaseNotFoundException is catched in the method [here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1015] and the inner function of [reloadTableIfExists|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2902] has already catched the exception and the function do not re-throw the exception out so , the outside function has no need to deal with these two exception in my mind. And I think it is not suitable to print an info level log [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2903]. Warn level logging is better. > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812237#comment-17812237 ] Maxwell Guo commented on IMPALA-12771: -- If you think my suggestion is reasonable, I will submit a PR later > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812215#comment-17812215 ] Maxwell Guo commented on IMPALA-12771: -- ping [~huangqiang] > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12771) Impala catalogd events-skipped may mark the wrong number
[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812214#comment-17812214 ] Maxwell Guo commented on IMPALA-12771: -- with alter parition event, we may found that database/table is not found , or table is not loaded see [here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1057]. then we just skipped the and made a debug log , the log is : " Ignoring the event since the table is not found ". But actually, we skipped the handling of these events when table is not found .So I think we can also +1 on the events-skipped metric if table is not found or table is IncompleteTable or table was remove in catalog. Besides, we just mark the events-skipped metric for event process with isOlderEvent method and isSelfEvent see [isSelfEvent|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L857] and [isOlderEvent|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1198] But as for [canBeSkipped|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1703] method, the metric is not +1. I think we can also add here. > Impala catalogd events-skipped may mark the wrong number > > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org