adoroszlai opened a new pull request #1590: HDDS-2238. Container Data Scrubber spams log in empty cluster URL: https://github.com/apache/hadoop/pull/1590 ## What changes were proposed in this pull request? 1. Add configurable interval for container data scan iterations (`hdds.containerscrub.data.scan.interval`). A small delay will not affect scanning if there is enough I/O work to do. However, it prevents the scanner threads from keeping CPU busy when there are no or few healthy closed containers to be scanned. 2. Only count iterations where at least one container was scanned. 3. Some code cleanup: * get rid of raw Container` usage (add generic parameter `<?>`) * fix some javadoc (add `@param` descriptions, remove `@throws` without description) https://issues.apache.org/jira/browse/HDDS-2238 ## How was this patch tested? Test script: ``` cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone echo 'OZONE-SITE.XML_hdds.containerscrub.enabled=true' >> docker-config echo 'OZONE-SITE.XML_hdds.containerscrub.data.scan.interval=1' >> docker-config KEEP_RUNNING=true ./test.sh for i in 1 2; do docker-compose exec scm ozone scmcli container close ${i} sleep 10 done docker-compose logs datanode | grep \ -e 'Completed an iteration of container data scrubber' \ -e 'Container .* is closed' ``` Result: ``` datanode_1 | 2019-10-04 07:01:00 INFO Container:335 - Container 1 is closed with bcsId 0. datanode_1 | 2019-10-04 07:01:00 INFO ContainerDataScanner:115 - Completed an iteration of container data scrubber in 0 minutes. Number of iterations (since the data-node restart) : 1, Number of containers scanned in this iteration : 1, Number of unhealthy containers found in this iteration : 1 datanode_1 | 2019-10-04 07:01:10 INFO Container:335 - Container 2 is closed with bcsId 0. datanode_1 | 2019-10-04 07:01:10 INFO ContainerDataScanner:115 - Completed an iteration of container data scrubber in 0 minutes. Number of iterations (since the data-node restart) : 2, Number of containers scanned in this iteration : 1, Number of unhealthy containers found in this iteration : 1 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
