[
https://issues.apache.org/jira/browse/IMPALA-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769942#comment-16769942
]
ASF subversion and git services commented on IMPALA-7976:
---------------------------------------------------------
Commit f4dd98236a45fc0cc2b11a6db7c91a6f04c183d3 in impala's branch
refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f4dd982 ]
IMPALA-7976 : Add a flag to disable sync using events at a table level
This change adds the ability to disable event processing at a table or
database level. If the user adds a property with the key
"impala.disableHmsSync" in the table or database properties, event
processing can be turned on/off at the respective level. The value of
the property is string value of Booleans ("true" or "false"). By default
event processing is enabled when the global config
hms_event_polling_interval_s is set to a non-zero value. The value of
property determines if the event processing needs to be disabled for a
particular table or database. So for example, if
"impala.disableHmsSync" is set to "true" then events for that
table/database are ignored. If the property is unset or a value of
"false" is used, the event processing is enabled (although it is not
necessary to explicitly enable event processing, given the global flag).
When both table and db level properties are set, the table level
property takes precedence. If the table level property is not set, then
database level property is used to evaluate if the event needs to be
processed or not.
Users can issue alter table command to change the value of this
property. When the flag value is changed, the alter table event
generated is always processed and subsequent events are either ignored
or processed based on the new value of table flag and database flag. In
theory, a database level property can also be changed. But Impala
doesn't currently support alter database operations. Users who may wish
to change the database level property should use alter database command
from Apache Hive (or any other metastore client)
When the flag is turned OFF and the previous value of ON or unset, it is
possible that the table does not exist in catalog anymore. For example,
when the table was created the event sync was disabled. Hence the create
event was skipped. Now when the user re-enables the sync again on such
table, event processor has no idea how many events were skipped during
this interval. Event processor errs on the side of caution and changes
the state to NEEDS_INVALIDATE in such case.
Testing done: Added new unit tests which execute all supported DDL
operations with all possible combination of db and table flags. Another
test executes alter table operations for changing flags for all combinations
of transitions at table level.
Change-Id: If116edba51bae139bc003be858b3214c3f0104e3
Reviewed-on: http://gerrit.cloudera.org:8080/12365
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add a flag to disable sync using events at a table level
> --------------------------------------------------------
>
> Key: IMPALA-7976
> URL: https://issues.apache.org/jira/browse/IMPALA-7976
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Major
>
> It is possible that certain tables need not be updated by the event
> mechanism. Possible use-cases could be frequently updated tables (from
> multiple systems) or for tables where we don't really need latest data. We
> should be able to add a flag (parameter) to the table which can be used to
> skip event processing for a given table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
