cxzl25 opened a new pull request, #8824:
URL: https://github.com/apache/hadoop/pull/8824
### Description of PR
DirectoryScanner schedules itself with
ScheduledThreadPoolExecutor#scheduleAtFixedRate.
With fixed-rate scheduling the next execution is computed from the previous
*scheduled*
time, not from the time the previous run finished. If a single scan takes
longer than
dfs.datanode.directoryscan.interval, the executor runs the missed executions
back-to-back
until it has caught up with the schedule. On DataNodes with a large number
of blocks the
scanner therefore ends up running almost continuously.
### How was this patch tested?
1. DirectoryScanner#start uses scheduleWithFixedDelay instead of
scheduleAtFixedRate, so
dfs.datanode.directoryscan.interval is measured from the end of one scan
to the start
of the next and a long scan can no longer cause back-to-back executions.
The javadoc and the startup log message are updated accordingly.
2. DirectoryScanner#reconcile logs at INFO when a scan starts and, when it
finishes, the
total duration broken down into the scan phase and the reconcile phase
together with
the number of differences reconciled:
Directory scan starting
Directory scan finished in 5ms: scanning took 5ms, reconciling 0
differences took 0ms
3. The description of dfs.datanode.directoryscan.interval in
hdfs-default.xml documents
the fixed-delay semantics.
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
