d-wang-commit opened a new pull request, #19126:
URL: https://github.com/apache/iotdb/pull/19126
## Description
### Keep DiskFull recovery consistent with directory protection
A data or snapshot filesystem can remain full while the aggregate free-space
ratio exceeds the warning threshold. Directory selection then enters
`ReadOnly(DiskFull)`, but the next heartbeat restores `Running` using only the
aggregate ratio. Repeated operations can keep switching the node between these
states without any space being freed.
Record a recovery check for each failed directory source in `CommonConfig`.
The heartbeat restores `Running` only when the aggregate check passes and every
recorded source has an eligible directory with space. Constructor failures use
an immutable folder list as the key; live `FolderManager` instances use their
identity and retain their folder-health restrictions. Repeated reports for the
same key replace the callback. Leaving read-only mode clears the recorded
checks.
Filesystem checks run outside the status lock. Recovery rechecks the exact
`ReadOnly(DiskFull)` state before changing it, preserving manual, stopping, and
unrecoverable-error transitions. The existing IOException retry and
unrecoverable-error handling remain unchanged.
### Tests
- UT: extend `CommonConfigTest` for multiple failed sources, callback
replacement, concurrent higher-priority reasons, and manual reset.
- UT: extend `DataNodeInternalRPCServiceImplDiskTest` for healthy aggregate
metrics with blocked sources, and recovered sources with zero or insufficient
aggregate space.
- UT: add `FolderManagerDiskFullRecoveryTest` for repeated snapshot
initialization failures, multiple directory groups, unhealthy candidates, and
callers that opt out of changing node status. Run the existing
`FolderManagerTest` as regression coverage.
- Validation after rebasing onto upstream/master:
- Full-reactor `mvn -B -ntp -o test
-Dtest=CommonConfigTest,FolderManagerTest,FolderManagerDiskFullRecoveryTest,DataNodeInternalRPCServiceImplDiskTest
-Dsurefire.failIfNoSpecifiedTests=false`: passed; 54 tests, 0 failures/errors,
4 pre-existing skips.
- Full-reactor `mvn -B -ntp -o test-compile -DskipTests -P
with-zh-locale`: passed.
- Spotless formatting and `git diff --check`: passed.
- IT: no integration test was added or run. Disk capacity is simulated in
unit tests; a real ENOSPC cluster test was not run.
### Side effects and risks
Recovery now probes previously failed directories using the existing
`hasSpace` behavior, including its retries, so a heartbeat may spend longer
checking a full filesystem. Callback references are retained until the node
leaves read-only mode, with one entry per distinct key. Folder health uses a
concurrent map because recovery reads it from the heartbeat thread.
Space checks are snapshots. A new disk-full report racing with the final
recovery step can still require the next business operation to trigger
protection again; this change addresses repeated recovery while
already-recorded sources remain full.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `CommonConfig`: records failed sources and coordinates guarded disk
recovery.
- `DirectoryStrategy`, `FolderManager`, `DataRegion`: register the recovery
condition at directory-related DiskFull entries.
- `DataNodeInternalRPCServiceImpl`: uses the shared recovery method after
the aggregate disk check passes.
- `CommonConfigTest`, `DataNodeInternalRPCServiceImplDiskTest`,
`FolderManagerDiskFullRecoveryTest`: cover state transitions, heartbeat gates,
and directory failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]