Sai Hemanth Gantasala created IMPALA-12982:
----------------------------------------------
Summary: COMMIT_TXN event should skipped by EP if there is
concurrent DROP DB operation
Key: IMPALA-12982
URL: https://issues.apache.org/jira/browse/IMPALA-12982
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala
The error happens in loading a partition while a concurrent DROP DATABASE
CASCADE is running. Impalad logs:
{noformat}
I0217 06:55:58.696295 28500 Frontend.java:2070]
8b4a15df68c9c5dd:f21c5bb200000000] Analyzing query: DROP DATABASE
`test_read_hive_inserts_8bb30aca` CASCADE db: default{noformat}
catalogd logs:
{noformat}
I0217 06:55:58.696962 28650 JniUtil.java:166]
8b4a15df68c9c5dd:f21c5bb200000000] execDdl request: DROP_DATABASE
test_read_hive_inserts_8bb30aca issued by jenkins
I0217 06:55:58.751202 5748 MetastoreEventsProcessor.java:909] Received 2
events. Start event id : 51397
I0217 06:55:58.751662 5748 MetastoreEvents.java:293] Total number of events
received: 2 Total number of events filtered out: 0
I0217 06:55:58.751700 5748 MetastoreEvents.java:297] Incremented skipped
metric to 8174
I0217 06:55:58.751801 5748 CatalogServiceCatalog.java:1060] Not a self-event
since the given version is -1 and service id is empty
I0217 06:55:58.751914 5748 HdfsTable.java:2851] Reloading partition metadata
for table: test_read_hive_inserts_8bb30aca.test_read_hive_inserts
(ALTER_PARTITION EVENT FOR TRANSACTIONAL TABLE)
I0217 06:55:58.752924 5749 MetastoreEventsProcessor.java:1023] Latest event in
HMS: id=51399, time=1708181758. Last synced event: id=51397, time=1708181750.
W0217 06:55:58.752954 5749 MetastoreEventsProcessor.java:1026] Lag: 8s. 2
events pending to be processed.
I0217 06:55:58.754891 5748 HdfsTable.java:2943] Setting the latest refresh
event id to 51399 for the reloaded 1 partitions
I0217 06:55:58.756098 5748 HdfsTable.java:2851] Reloading partition metadata
for table: test_read_hive_inserts_8bb30aca.test_read_hive_inserts (Processing
event id: 51399, event type: COMMIT_TXN)
I0217 06:55:58.756839 5748 HdfsTable.java:2943] Setting the latest refresh
event id to 51401 for the reloaded 1 partitions
I0217 06:55:58.756867 5748 HdfsTable.java:2946] for table
test_read_hive_inserts_8bb30aca.test_read_hive_inserts, file metadataOps:
FORCE_LOAD, refreshing file metadata for 1 out of 1 partitions to reload in
reloadPartitions()
E0217 06:55:58.761873 5748 ParallelFileMetadataLoader.java:194] Refreshing
file and block metadata for 1 paths for table
test_read_hive_inserts_8bb30aca.test_read_hive_inserts encountered an error
loading data for path
hdfs://localhost:20500/test-warehouse/managed/test_read_hive_inserts_8bb30aca.db/test_read_hive_inserts/p=1
Java exception follows:
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
/test-warehouse/managed/test_read_hive_inserts_8bb30aca.db/test_read_hive_inserts/p=1/delta_0000010_0000010_0000/000000_0
...
I0217 06:55:58.762064 5748 CatalogOpExecutor.java:5006] Rolling back the write
id list of table test_read_hive_inserts_8bb30aca.test_read_hive_inserts because
reloading for event 51399 is failed: Refreshing file and block metadata for 1
paths for table test_read_hive_inserts_8bb30aca.test_read_hive_inserts: failed
to load 1 paths. Check the catalog server log for more details.
I0217 06:55:58.762253 5748 MetastoreEventsProcessor.java:1189] Time elapsed in
processing event batch: 10.977ms
E0217 06:55:58.762821 5748 MetastoreEventsProcessor.java:981] Event processing
needs a invalidate command to resolve the state
Java exception follows:
org.apache.impala.catalog.events.MetastoreNotificationNeedsInvalidateException:
EventId: 51399 EventType: COMMIT_TXN Failed to mark committed write ids and
refresh partitions for txn {}. Event processing cannot continue. Issue an
invalidate metadata command to reset event processor.
at
org.apache.impala.compat.MetastoreShim$CommitTxnEvent.process(MetastoreShim.java:870)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:672)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1164)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:972)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Refreshing file and
block metadata for 1 paths for table
test_read_hive_inserts_8bb30aca.test_read_hive_inserts: failed to load 1 paths.
Check the catalog server log for more details.
at
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:210)
at
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:805)
at
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:761)
at
org.apache.impala.catalog.HdfsTable.reloadPartitions(HdfsTable.java:2951)
at
org.apache.impala.catalog.HdfsTable.reloadPartitions(HdfsTable.java:2884)
at
org.apache.impala.catalog.HdfsTable.reloadPartitionsFromEvent(HdfsTable.java:2864)
at
org.apache.impala.service.CatalogOpExecutor.addCommittedWriteIdsAndReloadPartitionsIfExist(CatalogOpExecutor.java:4994)
at
org.apache.impala.compat.MetastoreShim$CommitTxnEvent.addCommittedWriteIdsAndRefreshPartitions(MetastoreShim.java:911)
at
org.apache.impala.compat.MetastoreShim$CommitTxnEvent.process(MetastoreShim.java:865)
... 10 more{noformat}
This seems a bug of event-processor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
