Siyao Meng created HDDS-17296:
---------------------------------
Summary: After an OM installs a checkpoint, reconfiguring the
directory deleting interval restarts the replaced DirectoryDeletingService as
well
Key: HDDS-17296
URL: https://issues.apache.org/jira/browse/HDDS-17296
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
Each {{DirectoryDeletingService}} registers a reconfiguration complete callback
in its constructor ({{registerReconfigCallbacks}}), and the callback calls
{{updateAndRestart}} on that instance. {{ReconfigurationHandler}} keeps these
callbacks in a list for the life of the OM process and has no way to remove
one. {{OzoneManager.installCheckpoint}} stops the {{KeyManager}} and then
builds a new {{DirectoryDeletingService}}: {{reloadOMState}} creates a new
metadata manager, a new {{OzoneManagerLock}} and a new {{KeyManagerImpl}}, and
when the old DB is kept the same {{KeyManagerImpl}} is started again. The
callback of the replaced service stays registered.
The next reconfiguration that changes
{{ozone.directory.deleting.service.interval}} or
{{ozone.thread.number.dir.deletion}} therefore restarts the replaced service
too. Its tasks check only the suspend flag and leader readiness, and they look
up the key manager and metadata manager from the OM on every run, so the
revived service does not fail on the closed DB: it works on the installed DB
like the current one. It keeps two things from before the install: the
bootstrap lock it took from the replaced metadata manager in the
{{AbstractKeyDeletingService}} constructor, and that metadata manager's
snapshot chain. {{OMDBCheckpointServlet}} resolves the current lock on every
call, so it does not exclude the revived service. The purge request checks the
expected previous snapshot against the current chain when it is applied, so the
stale chain makes the revived service's purges fail once the chain has changed,
rather than purge wrongly.
h3. Trigger
# An OM installs, or attempts to install, a checkpoint from the leader. This
happens to any follower that lags behind the purged Raft log, and every attempt
that gets as far as stopping the key manager counts, also when the checkpoint
is rejected as not newer or pausing the state machine fails. No admin action is
needed.
# An admin changes the directory deleting interval or thread count in
ozone-site.xml and runs {{ozone admin reconfig --service=OM start}} on that OM.
# The replaced service now runs next to the current one whenever that OM is
leader.
Each further install attempt adds another stale callback, and nothing stops a
revived service until the process exits.
h3. Impact
* Two {{DirectoryDeletingService}} instances submit directory purges on the
leader. The replaced one is not excluded by the checkpoint servlet's bootstrap
write lock, so it keeps changing the DB while a checkpoint for another OM or
Recon is being taken. Both shown in the reproduction, before any snapshot
change after the install.
* Each install leaves the replaced service, and the lock and snapshot chain it
references, reachable from the handler (code reading).
* Argued, not run: two instances can read and submit a purge for the same
deleted directory, and {{OMDirectoriesPurgeRequestWithFSO}} applies the quota
change of a purge without checking that the paths still exist, so the bucket
used bytes and namespace could be counted down twice. That needs both instances
to read the same directory before the first purge is applied; it was not
observed without an artificial delay. Once a snapshot is created or deleted
after the install, the revived service's purges are rejected by the expected
previous snapshot check instead.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC7StaleDirDeletingServiceRestart.java]
({{hadoop-ozone/ozone-manager}}) starts one OM through {{OmTestManagers}},
installs a checkpoint of its own DB with {{installCheckpoint}} (presented with
a newer transaction index so that it proceeds) and runs the reconfiguration
task that {{ozone admin reconfig start}} starts. It suspends the current
service with its own hook, adds a deleted directory, and shows that the
replaced service was restarted, purged the directory from the installed DB, and
still gets its bootstrap read lock while {{OMDBCheckpointServlet.Lock}} holds
the bootstrap write lock (the current service does not). A passing test means
the defect is present.
h3. Patch
[^MC-7-restart-only-current-dir-deleting-service-on-reconfig.patch], against
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
669912118cf.
The per instance callback is removed. {{OzoneManager}} registers one callback
that, when the interval or thread count changes, restarts the service that
{{keyManager.getDirDeletingService()}} returns at that moment. A service
created later reads the new values from the OM configuration, which the
reconfiguration task has already updated. {{DirectoryDeletingService.shutdown}}
now also marks the service closed, and {{updateAndRestart}} checks that mark
under the service's monitor before it starts the executors again, so a restart
that picked up the service just before an install shut it down leaves it
stopped. The callback takes no new lock and does not wait for a running
install. It also removes the {{registerCompleteCallback}} call that the install
thread made on the handler's unsynchronized list while a reconfiguration task
could be iterating it.
Covered by {{testReconfigurationRestartsOnlyCurrentService}} in the existing
{{TestDirectoryDeletingService}}: it replaces the service by stopping and
starting the key manager, as an install does, runs the reconfiguration task and
checks that the replaced service stays shut down, also when
{{updateAndRestart}} is called on it directly, and that the current one has the
new interval. Without the change the test fails because the replaced service is
running again; with only the callback change and without the closed check it
fails at the direct restart. With the change {{TestDirectoryDeletingService}}
and {{TestOzoneIntegrationNonHA}} (including the OM reconfiguration and
reconfig shell tests) pass, and checkstyle is clean. With the patch the
reproduction fails at its first assertion, because the replaced service stays
shut down.
Found by TLA+ model checking and code review of the Ozone Manager lock ordering
paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. Related but not duplicates: HDDS-12554 (added this callback),
HDDS-11513 (extended it to the thread count), HDDS-15429 and HDDS-15405 (other
{{updateAndRestart}} fixes), HDDS-17178 (another component of a replaced OM
state left registered after an install), HDDS-17120 and HDDS-16878 (checkpoint
servlets that keep a DB store an install replaced, the same pattern for another
object). The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
