[
https://issues.apache.org/jira/browse/IMPALA-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved IMPALA-12637.
--------------------------------------------
Resolution: Fixed
> Failed to process CREATE_TABLE events if the db is unloaded
> -----------------------------------------------------------
>
> Key: IMPALA-12637
> URL: https://issues.apache.org/jira/browse/IMPALA-12637
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
>
> Applying a CREATE_TABLE event could fail by DatabaseNotFoundException is the
> db is unloaded. The exception is thrown by
> CatalogOpExecutor#addTableIfNotRemovedLater():
> [https://github.com/apache/impala/blob/3af193022916e42c33d6eafafb6f9560a0789895/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L826-L831]
> In fact, there are cases that the db exists in HMS but is unloaded in
> catalogd (even if HMS event processing is enabled). For instance, when HMS is
> restarted during the launching period of catalogd, some dbs might be failed
> in loading:
> {noformat}
> W1215 08:27:27.811750 26987 CatalogServiceCatalog.java:2014] Encountered an
> exception while invalidating database: default. Ignoring further load of this
> db.
> Java exception follows:
> MetaException(message:Could not connect to meta store using any of the URIs
> provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:255)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:755)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:448)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient$1.run(RetryingMetaStoreClient.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:187)
> at com.sun.proxy.$Proxy11.getFunctions(Unknown Source)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.invalidateDb(CatalogServiceCatalog.java:1943)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:2103)
> at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:175)
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:607)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:250)
> ... 11 more
> )
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:826)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:448)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient$1.run(RetryingMetaStoreClient.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:187)
> at com.sun.proxy.$Proxy11.getFunctions(Unknown Source)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.invalidateDb(CatalogServiceCatalog.java:1943)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:2103)
> at
> org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:175){noformat}
> I can reproduce the issue by adding a breakpoint in
> CatalogServiceCatalog#invalidateDb() and attach to the process by IntelliJ
> during catalogd is launching. Then kill HMS and resume catalogd. After seeing
> this log, restart HMS and resume catalogd again.
> The event processor will go into the ERROR state when processing any
> CREATE_TABLE event in that db:
> {noformat}
> E1215 08:29:54.121959 27281 MetastoreEventsProcessor.java:979] Unexpected
> exception received while processing event
> Java exception follows:
> org.apache.impala.catalog.events.MetastoreNotificationException: EventId:
> 8411895 EventType: CREATE_TABLE Unable to process event
> at
> org.apache.impala.catalog.events.MetastoreEvents$CreateTableEvent.process(MetastoreEvents.java:1213)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:528)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1153)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:961)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.impala.catalog.DatabaseNotFoundException: default not
> found
> at
> org.apache.impala.service.CatalogOpExecutor.addTableIfNotRemovedLater(CatalogOpExecutor.java:831)
> at
> org.apache.impala.catalog.events.MetastoreEvents$CreateTableEvent.process(MetastoreEvents.java:1200)
> ... 10 more
> E1215 08:29:54.137053 27281 MetastoreEventsProcessor.java:1237] Event id:
> 8411895
> Event Type: CREATE_TABLE
> Event time: 1702600193
> Database name: default
> Table name: newtbl
> Event message:
> H4sIAAAAAAAAAL1V23LaMBD9Fz87xjK5AG+0uNO0DGTAmelMyTACL1itkB1JTkoY/j1rySTYuGn70LxgsUfeyzm7652jQD6AdHqOTiRb6V6rxdMl5UmqdK/rd9qOa66wJdxIJpYsoxwvozVe4DOGFc25xr+aLjigRcCjXvCDIdpmhTH8FoWTUX84j/ofhuEBHC9+fFGpQHw3c8jM6eFDaYmHWelm5uzdmRNUoTKmxdpV7D6ngqdibcFzC7J24YFc+cGl75Nuu4AujiG/sFyeWK6sRcLSHsoUucLYve8l4BL3NHvWmDgTRdL7uwYoiVeqQn3gX/h+S4PSZ49UQpLmClrHpNQKT+XaoxldJuAlNE7TzNvQTELsRfBLX4ss159SuaG6QszvX07YA3j33GOp9xmP12uRSvgK28LbONc1dyWZetXI5RlpIjP4qxSw9WIIPE6ftt4Qf6Zsk3GYghxAhQYs1mpi/LmHp9EGfTDK2RPVLBXeqsy7Z0RDMfaFm05NWvu6b6TqNspuMeK/Gd93d1ZuQv7QS8cBgxpmDiVWr9di1XimIhJURGms8riS7puFXBRERpP+aDrsR+FgHo3nh5k2VEaT29Bc15IKxUDoOadKD2IesQ1Ytl/mz1xc5MufoJlYz3H9KFTGXAoMhk0GUlDumTXhZblcQzVMQ/CS6FpfVTeP7RnSbhbjpHZLrmmi18XyStx5w+Rfj6bhJKqM//EU/MO6qg8WQQlP403DYfjxHePd3gywAd4v3gDr+2/x7l6G5by+wchVbUHjNiq7p3PiNjjsu8vCD+l0SYDei88cNr/SdJMdf35cZ8U4KGyku/0zLOV3nYAHAAA=
> W1215 08:29:54.979072 27281 MetastoreEventsProcessor.java:950] Event
> processing is skipped since status is ERROR. Last synced event id is
> 8411894{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
