[
https://issues.apache.org/jira/browse/IMPALA-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved IMPALA-10987.
--------------------------------------------
Fix Version/s: Impala 4.4.0
Resolution: Fixed
> Changing impala.disableHmsSync in Hive can break event processing
> -----------------------------------------------------------------
>
> Key: IMPALA-10987
> URL: https://issues.apache.org/jira/browse/IMPALA-10987
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Csaba Ringhofer
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Fix For: Impala 4.4.0
>
>
> To reproduce, start Impala with event polling:
> {code}
> bin/start-impala-cluster.py --catalogd_args="--hms_event_polling_interval_s=2
> --catalog_topic_mode=minimal" --impalad_args="--use_local_catalog=1"
> {code}
> From Hive:
> {code}
> CREATE DATABASE temp;
> CREATE EXTERNAL TABLE temp.t (i int) PARTITIONED BY (p int)
> TBLPROPERTIES('impala.disableHmsSync'='true');
> ALTER TABLE temp.t SET TBLPROPERTIES ('impala.disableHmsSync'='false');
> {code}
> From this point event sync will be broken in Impala. It can be fixed only
> with global INVALIDATE METADATA (or restarting catalogd)
> catalogd log will include an exception like this:
> {code}
> E1026 10:30:16.151208 22514 MetastoreEventsProcessor.java:653] Event
> processing needs a invalidate command to resolve the state
> Java exception follows:
> org.apache.impala.catalog.events.MetastoreNotificationNeedsInvalidateException:
> EventId: 15956 EventType: ALTER_TABLE Detected that event sync was tur
> ned on for the table temp.t and the table does not exist. Event processing
> cannot be continued further. Issue a invalidate metadata command to reset
> the event processing state
> at
> org.apache.impala.catalog.events.MetastoreEvents$AlterTableEvent.process(MetastoreEvents.java:992)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:345)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:747)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:645)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> and future events will be lead to a log like this:
> {code}
> W1026 10:30:18.151962 22514 MetastoreEventsProcessor.java:638] Event
> processing is skipped since status is NEEDS_INVALIDATE. Last synced event id
> is 15955
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
