Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24629
Change subject: IMPALA-14618: Fix EventProcessor race with catalog reset ...................................................................... IMPALA-14618: Fix EventProcessor race with catalog reset While processing a CREATE_DATABASE event, the check-and-add in CatalogOpExecutor#addDbIfNotRemovedLater() is not protected by the catalog version lock so has race with the concurrent catalog reset. Here is the sequence when things go wrong: thread1: catalog reset starts thread2: CREATE_DATABASE event finds the db doesn't exist, i.e., getDb() returns null. thread1: catalog reset loads the db with the latest table list thread2: CREATE_DATABASE overwrites the Db instance with a new one that has an empty table list. The table list is expected to be updated by following CREATE_TABLE events. thread1: finishes catalog reset. Update last synced event id and restart EventProcessor to start from it. The CREATE_TABLE events are skipped. Leaving the db with an empty table list. Note that when hierarchical event processing is disabled, this is not a problem since catalog resets can't start concurrently when an HMS event is being processing. This is ensured by the instance lock of MetastoreEventsProcessor: - Catalog resets invoke MetastoreEventsProcessor.pause() at the begining. It's a synchronized method. - Processing an event in the main thread of MetastoreEventsProcessor is guarded by "synchronized (this)". When hierarchical event processing is enabled, the main thread of MetastoreEventsProcessor just dispatch events. Events are processed in other threads so can run concurrently with catalog resets. This patch fixes the issue by moving the check-and-set codes in CatalogOpExecutor#addDbIfNotRemovedLater() into CatalogServiceCatalog so they can be wrapped within WriteLockAndLookupDb. This fixes the race with concurrent catalog reset. DROP_DATABASE event also has the same issue in CatalogOpExecutor#removeDbIfNotAddedLater() that the check-and-remove operations are not protected by the catalog version lock. This patch also moves them into CatalogServiceCatalog to be wrapped with WriteLockAndLookupDb. There is a minor issue that the createEventId of dbs are not set during catalog reset, which impacts the skipping logic of DROP_DATABASE events. This patch also fixes this issue. Fixed missing a comma in the default value of common_hms_event_types. This is used in passing the EventTypeSkipList to HMS while fetching specific types of events. The bug won't impact correctness since the filters are also evaluated in client side (catalogd). Testing - Added e2e test to reproduce the bug by using a debug action that sleep before the CREATE_DATABASE event adding the db to catalog. - Ran test_failover_catchup_timeout_and_reset and test_create_db_table_survives_reset 100 times. Assisted-by: Opus 4.8 (Claude Code) Change-Id: Ifa0833c09cd7ec2461a0f2d1db7ab5a72ab6822b --- M be/src/catalog/catalog-server.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/DebugUtils.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 6 files changed, 125 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/24629/1 -- To view, visit http://gerrit.cloudera.org:8080/24629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifa0833c09cd7ec2461a0f2d1db7ab5a72ab6822b Gerrit-Change-Number: 24629 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
