[
https://issues.apache.org/jira/browse/IMPALA-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825571#comment-17825571
]
Maxwell Guo commented on IMPALA-12468:
--------------------------------------
The reason why I create the jira is that we meet the problem of IMPALA-12832
and we have made some code modification just like IMPALA-12832 but we only
dealt with the situation of NEEDS_INVALIDATE。
After looking at the code of impala, we found that the ERROR status may be
different NEED_INVALIDATE. For example, we have found that impala's schema is
not compatible with hive (but this have been fixed in the new impala version),
it may be necessary to ask the HMS side to do some table operations, so that
the EP can deal with this table correctly . Therefore, in this case, I thought
of manually resetting the EP status after user modifying their schema.
> Add the ability to update EventProcessorStatus
> ----------------------------------------------
>
> Key: IMPALA-12468
> URL: https://issues.apache.org/jira/browse/IMPALA-12468
> Project: IMPALA
> Issue Type: Improvement
> Components: be, Catalog, fe
> Reporter: Maxwell Guo
> Assignee: Maxwell Guo
> Priority: Minor
>
> Once the impala and hive's status is missmatched , and the
> EventProcessorStatus become NEED_INVALIDATE， we usually use invalidate
> metadata to reset the catalog instance. And then impala will update the
> status to ACTIVE .
> But if impala contains many tables , the cost of invalidate is a bit high for
> a global invalidate. So we may invalidate metadata for tables one by one for
> these incremental changed table. For example , we have 1000,000,000,000
> tables but only some of the table event process occurs CatalogException and
> MetastoreNotificationNeedsInvalidateException was thrown. I think there is no
> need to invalidate all table caches in order to reset the catalog instance
> see [here
> |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2088].
>
> MetaStoresProcessor 's async update process will not update the currentStatus
> when the status is not ACTIVE, see
> [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L876]
> So what about add a new SQL grammar : SET EVENT STATUS ${status} ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
