Siyao Meng created HDDS-17389:
---------------------------------
Summary: Recon export cleanup deletes every directory and .tar
file in ozone.recon.export.directory, not only files created by exports
Key: HDDS-17389
URL: https://issues.apache.org/jira/browse/HDDS-17389
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-export-startup-cleanup-only-owned-entries.patch,
TestBugCR1ExportDirSweep.java
h3. Mechanism
The {{ExportJobManager}} constructor removes leftovers of earlier exports from
{{ozone.recon.export.directory}}. It lists the directory and calls
{{FileUtils.deleteQuietly}} (recursive) on every subdirectory and on every file
ending in {{.tar}}, without looking at the name. The manager itself only
creates {{<jobId>}} working directories (a random UUID) and
{{export_<state>_<millis>.tar}} files, so everything else it deletes was put
there by someone else.
The configured value is used as is. Nothing checks that the directory is
dedicated to exports or differs from {{ozone.recon.db.dir}}, and neither
{{ozone-default.xml}} nor {{ReconServerConfigKeys}} says that its contents are
removed. SCM's container ID export ({{ExportFileManager}}) clears its export
directory on startup more narrowly: it removes only {{export_<UUID>}}
directories and {{*.tar.gz.tmp}} files.
h3. Trigger
# Set {{ozone.recon.export.directory}} to a directory that holds other data,
for example the Recon DB directory itself or a shared scratch directory.
# Start Recon. The cleanup runs when the {{ExportJobManager}} singleton is
created, which happens once per Recon process on first use of the containers
API (argued from the Guice binding, not run in a Recon server).
The default ({{ozone.recon.export.directory}} empty, resolved to
{{<ozone.recon.db.dir>/exports}}) is not affected.
h3. Impact
* With the export directory equal to {{ozone.recon.db.dir}}: Recon's container
key DB, OM snapshot DB, Recon SCM DB and Derby SQL DB directories are deleted,
and so is {{recon/}} with Recon's storage {{current/VERSION}} file (run against
stand-in directories with those names, see reproduction). Because the sweep
runs on the first containers API request, Recon has these databases open at
that point, so they are deleted under a running process (argued from code, not
run).
* With a shared directory: other applications' directories and any {{.tar}}
file (for example backups) are deleted permanently (run). Other files are kept.
* It repeats on every Recon start.
h3. Reproduction
[^TestBugCR1ExportDirSweep.java] ({{hadoop-ozone/recon}}, package
{{org.apache.hadoop.ozone.recon.api}}) builds {{ExportJobManager}} through its
public constructor with real configuration keys, as Guice does; only
{{ContainerHealthSchemaManager}}, which the cleanup does not use, is mocked.
The first test points the export directory at the Recon DB directory holding
{{recon-container-key.db_...}}, {{om.snapshot.db_...}}, {{recon-scm.db}},
{{ozone_recon_derby.db}} and {{recon/current/VERSION}}; after construction the
directory is empty. The second uses a shared directory with
{{other-app-cache/}}, {{operator-backup-2026-10-01.tar}} and {{notes.txt}};
only {{notes.txt}} is left. Both tests pass on unmodified source,
deterministic; a passing test means the defect is present.
h3. Patch
[^CR-1-export-startup-cleanup-only-owned-entries.patch], against
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
9c3e5477ffb.
The cleanup now removes a directory only if its name is a valid UUID
({{UUIDUtil.isValidUuidString}}, as in SCM's {{ExportFileManager}}) and a file
only if its name has the form {{export_<state>_<millis>.tar}}, the names
{{submitJob}} and {{executeExport}} create. The state part accepts any letters,
because {{submitJob}} lower cases it with the JVM default locale (under a
Turkish locale {{MISSING}} gets a dotless i). Leftovers of an interrupted
export are still removed and other files are kept. A directory whose name is a
UUID is still removed, because the job manager names its own working
directories that way. In the default directory, which only holds export files,
the behaviour is unchanged.
Covered in the existing {{TestExportJobManager}}: the new
{{startupCleanupKeepsEntriesNotCreatedByExports}} checks that {{backup.tar}},
{{export_backup.tar}} and a DB directory survive, and
{{startupCleanupRemovesLeftoverTarsAndDirs}} now uses the names the manager
really creates (it used {{stale_export.tar}} and {{stale_job_id/}} before),
including a tar named with the Turkish lower case of {{MISSING}}, and still
expects them removed. The test builds that name with an explicit locale and
does not change the JVM default. Without the change the new test fails because
{{backup.tar}} is deleted. With it {{TestExportJobManager}}, {{TestExportJob}}
and {{TestContainerEndpoint}} (54 tests) pass and checkstyle is clean. With the
patch both tests of the reproduction fail, because all entries are kept.
Found by code review of the Recon unhealthy container export job manager, as
part of the TLA+ verification effort under HDDS-15926, on commit
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
