[
https://issues.apache.org/jira/browse/HDDS-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-5943:
---------------------------------
Labels: pull-request-available (was: )
> Shutdown ResultHandlerExecutorService for StorageVolumeChecker
> --------------------------------------------------------------
>
> Key: HDDS-5943
> URL: https://issues.apache.org/jira/browse/HDDS-5943
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Mark Gui
> Assignee: Mark Gui
> Priority: Major
> Labels: pull-request-available
>
> During unit test on my laptop, theres OOM during TestStorageVolumeChecker:
> {code:java}
> Exception in thread "VolumeCheck ResultHandler thread 15"
> java.lang.OutOfMemoryError: unable to create new native threadException in
> thread "VolumeCheck ResultHandler thread 15" java.lang.OutOfMemoryError:
> unable to create new native thread2021-11-05 12:55:13,999 [VolumeCheck
> ResultHandler thread 44] INFO volume.TestStorageVolumeChecker
> (TestStorageVolumeChecker.java:schedule(289)) - Returning success for volume
> check at java.lang.Thread.start0(Native Method) at
> java.lang.Thread.start(Thread.java:717) at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
> at
> com.google.common.util.concurrent.ImmediateFuture.addListener(ImmediateFuture.java:47)
> at
> com.google.common.util.concurrent.Futures.addCallback(Futures.java:1047)2021-11-05
> 12:55:13,998 [VolumeCheck ResultHandler thread 57] WARN
> volume.MutableVolumeSet
> (MutableVolumeSet.java:lambda$checkVolumeAsync$1(283)) - checkVolumeAsync
> callback got 1 failed volumes:
> [/var/folders/_s/q95mqw991pj9c2cz4fr8ytnr0000gn/T/junit1980556976380740035/439b9ee5-1037-4830-ba87-c7f7778b387f/hdds]
> at
> org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker.checkVolume(StorageVolumeChecker.java:268)
> at
> org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.checkVolumeAsync(MutableVolumeSet.java:280)
> at
> org.apache.hadoop.ozone.container.common.utils.StorageVolumeUtil.onFailure(StorageVolumeUtil.java:41)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:234)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerFile(KeyValueContainer.java:255)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerData(KeyValueContainer.java:366)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:297)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerSet.lambda$handleVolumeFailures$0(ContainerSet.java:129)
> at java.lang.Iterable.forEach(Iterable.java:75) at
> org.apache.hadoop.ozone.container.common.impl.ContainerSet.handleVolumeFailures(ContainerSet.java:126)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.handleVolumeFailures(OzoneContainer.java:341)
> at
> org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.handleVolumeFailures(MutableVolumeSet.java:267)
> at
> org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.lambda$checkVolumeAsync$1(MutableVolumeSet.java:288)
> at
> org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.invokeCallback(StorageVolumeChecker.java:366)
> at
> org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.cleanup(StorageVolumeChecker.java:359)
> at
> org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.onSuccess(StorageVolumeChecker.java:337)
> at
> org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.onSuccess(StorageVolumeChecker.java:282)
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> We should shutdown the ResultHandlerExecutorService correctly when we
> shutdown StorageVolumeChecker to prevent this OOM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
