CRZbulabula opened a new pull request, #17931:
URL: https://github.com/apache/iotdb/pull/17931
## Description
This is a follow-up to #17880 ("Fix empty snapshot loading and region
cleanup"), addressing the second problem reported in the same scenario: a
cluster that contains an **empty `DataRegion`** (auto-created by the ConfigNode
after a scale-out, carrying `0` `SeriesPartitionSlot`) being migrated during
scale operations.
While #17880 fixed the empty-snapshot loading (`SnapshotLoader`) and the
region-cleanup timeout (`TableDiskUsageIndex` / `DataRegion`), the affected
DataNode kept flooding its log with `ERROR` entries like:
```
ERROR o.a.i.m.m.s.SystemMetrics:366 - Failed to statistic the size of /data
(/dev/sdb1), because
java.nio.file.NoSuchFileException: /data/.../data/datanode/system
at ...
at
org.apache.iotdb.metrics.metricsets.system.SystemMetrics.getSystemDiskAvailableSpace(SystemMetrics.java:364)
at
org.apache.iotdb.metrics.core.type.IoTDBAutoGauge.getValue(IoTDBAutoGauge.java:43)
at ...DataNodeInternalRPCServiceImpl.sampleDiskLoad(...)
at ...getDataNodeHeartBeat(...)
```
### Root cause
`SystemMetrics#setDiskDirs` resolves each configured disk directory into a
`java.nio.file.FileStore` **once at startup** and caches the resulting objects.
A `FileStore` pins the exact path it was resolved from; on Linux every
`getTotalSpace()` / `getUnallocatedSpace()` / `getUsableSpace()` call re-runs
`statvfs` on that pinned path.
When that directory is removed while IoTDB is running (e.g. an empty data
region directory is deleted during region migration), the pinned path no longer
exists and every space query throws `NoSuchFileException`. Because disk metrics
are sampled on every DataNode heartbeat (and on every Prometheus scrape), the
stale `FileStore` was logged at `ERROR` on every sampling, never recovered, and
flooded the log.
### Fix
- `SystemMetrics` now also stores the configured disk dirs.
- When a space query against a cached `FileStore` fails, the `FileStore` set
is re-resolved **once** via `FileStoreUtils#getFileStore`, which walks up to an
existing ancestor directory on the same device. The metric then recovers on the
next sampling instead of staying broken forever.
- A failure that persists even after re-resolving (practically impossible,
since the lookup ultimately falls back to an existing directory) is logged at
`WARN` instead of `ERROR`, so it can no longer flood the log.
- `fileStores` / `diskDirs` are made `volatile` and the re-resolution is
done copy-on-write, since the getters are invoked concurrently from the
heartbeat and Prometheus-reporter threads.
### Behavior
- No behavioral change on the happy path: when all directories exist, the
reported total/free/available disk space is identical to before.
- When a backing directory disappears, the metric self-heals on the next
sample (re-binding to a still-existing ancestor on the same device) rather than
returning `0` and spamming `ERROR` logs.
PingCode: V2-974
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `org.apache.iotdb.metrics.metricsets.system.SystemMetrics`
- `org.apache.iotdb.metrics.metricsets.system.SystemMetricsTest` (new)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
