[
https://issues.apache.org/jira/browse/IMPALA-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756090#comment-16756090
]
ASF subversion and git services commented on IMPALA-7970:
---------------------------------------------------------
Commit 389fb169d2782729aa48b820f3cd5bbfcb5f8dd8 in impala's branch
refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=389fb16 ]
IMPALA-7970 : Add support for metastore event based automatic invalidate
This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.
In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.
The change also adds a new test class to test the basic functionality
for each of the event type which is supported.
Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.
Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Reviewed-on: http://gerrit.cloudera.org:8080/12118
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add support for automatic invalidates by polling metastore events
> -----------------------------------------------------------------
>
> Key: IMPALA-7970
> URL: https://issues.apache.org/jira/browse/IMPALA-7970
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Major
>
> This JIRA will add the infrastructure pieces needed to poll metastore for
> events at a configurable interval issue {{invalidate}} on the table objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
