Hello Yida Wu, [email protected], Sai Hemanth Gantasala, Joe McDonnell,
Csaba Ringhofer, Nandor Kollar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24629
to look at the new patch set (#5).
Change subject: IMPALA-14618: Fix EventProcessor race with catalog reset
......................................................................
IMPALA-14618: Fix EventProcessor race with catalog reset
Race of CREATE_DATABASE event
-----------------------------
While processing a CREATE_DATABASE event, the check-and-add in
CatalogOpExecutor#addDbIfNotRemovedLater() is not protected by the
catalog version lock so has race with the concurrent catalog reset. Here
is the sequence when things go wrong:
thread1: catalog reset starts
thread2: CREATE_DATABASE event finds the db doesn't exist, i.e., getDb()
returns null.
thread1: catalog reset loads the db with the latest table list
thread2: CREATE_DATABASE overwrites the Db instance with a new one that
has an empty table list. The table list is expected to be updated by
following CREATE_TABLE events.
thread1: finishes catalog reset. Update last synced event id and restart
EventProcessor to start from it. The CREATE_TABLE events are
skipped. Leaving the db with an empty table list.
Race of CREATE_TABLE event
-----------------------------
While processing a CREATE_TABLE event, a concurrent reset could finish
and restarts event processor to polling events from a new event id,
which skips some events. If there is a corresponding DROP_TABLE event
not being polled and processed, the CREATE_TABLE event will add a stale
table in the db. Note that the table is expected to be removed by the
DROP_TABLE event which is skipped.
Event processor modes
---------------------
When hierarchical event processing is disabled, these are not problems
since catalog resets can't start concurrently when an HMS event is being
processing. This is ensured by the instance lock of
MetastoreEventsProcessor:
- Catalog resets invoke MetastoreEventsProcessor.pause() at the
begining. It's a synchronized method.
- Processing an event in the main thread of MetastoreEventsProcessor is
guarded by "synchronized (this)".
When hierarchical event processing is enabled, the main thread of
MetastoreEventsProcessor just dispatch events. Events are processed in
other threads so can run concurrently with catalog resets.
Solution
--------
This patch fixes the CREATE_DATABASE issue by moving the check-and-set
codes in CatalogOpExecutor#addDbIfNotRemovedLater() into
CatalogServiceCatalog so they can be wrapped within
WriteLockAndLookupDb. This fixes the race with concurrent catalog reset.
DROP_DATABASE event also has the same issue in
CatalogOpExecutor#removeDbIfNotAddedLater() that the check-and-remove
operations are not protected by the catalog version lock. This patch
also moves them into CatalogServiceCatalog to be wrapped with
WriteLockAndLookupDb.
There is an issue that the createEventId of dbs are not set during
catalog reset, which impacts the skipping logic of DROP_DATABASE events.
This patch also fixes this issue.
The issue of applying stale CREATE_TABLE event is fixed by correctly
skipping the event:
- If the corresponding DROP_TABLE event has been polled from HMS, add
it to the DeleteEventLog when queuing it so the thread processing
CREATE_TABLE event knows the table is removed later.
- If the DROP_TABLE event is not in DeleteEventLog, we fallback to
check the createEventId of the db which will be updated to the latest
event id at which reset starts. If the CREATE_TABLE event id is
smaller than this id, the event can be skipped.
Testing
- Added e2e tests by adding debug actions to inject sleeps.
- Ran test_failover_catchup_timeout_and_reset and
TestEventProcessingRace 100 times.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ifa0833c09cd7ec2461a0f2d1db7ab5a72ab6822b
---
M be/src/catalog/catalog-server.cc
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 150 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/24629/5
--
To view, visit http://gerrit.cloudera.org:8080/24629
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifa0833c09cd7ec2461a0f2d1db7ab5a72ab6822b
Gerrit-Change-Number: 24629
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Nandor Kollar <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Yida Wu <[email protected]>
