errose28 opened a new pull request, #4951:
URL: https://github.com/apache/ozone/pull/4951
## What changes were proposed in this pull request?
Ensure container scanners shut down cleanly without marking containers
unhealthy if ongoing scans are interrupted due to shutdown.
- Remove redundant shutdown hook in `MutableVolumeSet`. The datanode already
has a shutdown hook set up in `HddsDatanodeService#call` that does a superset
of this hook's operations and does them in the correct order.
- The shutdown hook in `MutableVolumeSet` would close the RocksDB before
stopping the scanners, causing an error in the running scanners.
- Check interrupt flag when a scanner thread encounters an exception.
- Various exceptions may be thrown if the thread is interrupted while
doing IO. These include
[`InterruptedIOException`](https://docs.oracle.com/javase/8/docs/api/java/io/InterruptedIOException.html)
and
[`ClosedByInterruptException`](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/ClosedByInterruptException.html).
- Rather than trying to capture all exceptions that could be thrown in
these cases, check the thread's interrupt flag on error, and throw
`InterruptedException` if it is set.
- This indicates that the scan was aborted due to interrupt and should
not be marked as passed or failed.
## What is the link to the Apache JIRA
HDDS-7100
## How was this patch tested?
- Unit tests for interrupt handling added.
- Manual docker testing:
1. Generate data: `docker compose exec om ozone freon ockg --size=10000000
-n100 --type=RATIS -rTHREE`
2. Stop datanodes
3. Inspect logs for errors on scanner shutdown.
- Before this patch, would see log messages about scanners marking
containers unhealthy because RocksDB was closed.
- After this patch, no such errors are observed on shutdown.
- I was unable to organically reproduce interrupted IO exceptions while
reading from a container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
