Mark Gui created HDDS-5943:
------------------------------
Summary: Shutdown ResultHandlerExecutorService for
StorageVolumeChecker
Key: HDDS-5943
URL: https://issues.apache.org/jira/browse/HDDS-5943
Project: Apache Ozone
Issue Type: Bug
Reporter: Mark Gui
Assignee: Mark Gui
During unit test on my laptop, theres OOM during TestStorageVolumeChecker:
{code:java}
Exception in thread "VolumeCheck ResultHandler thread 15"
java.lang.OutOfMemoryError: unable to create new native threadException in
thread "VolumeCheck ResultHandler thread 15" java.lang.OutOfMemoryError: unable
to create new native thread2021-11-05 12:55:13,999 [VolumeCheck ResultHandler
thread 44] INFO volume.TestStorageVolumeChecker
(TestStorageVolumeChecker.java:schedule(289)) - Returning success for volume
check at java.lang.Thread.start0(Native Method) at
java.lang.Thread.start(Thread.java:717) at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at
com.google.common.util.concurrent.ImmediateFuture.addListener(ImmediateFuture.java:47)
at
com.google.common.util.concurrent.Futures.addCallback(Futures.java:1047)2021-11-05
12:55:13,998 [VolumeCheck ResultHandler thread 57] WARN
volume.MutableVolumeSet (MutableVolumeSet.java:lambda$checkVolumeAsync$1(283))
- checkVolumeAsync callback got 1 failed volumes:
[/var/folders/_s/q95mqw991pj9c2cz4fr8ytnr0000gn/T/junit1980556976380740035/439b9ee5-1037-4830-ba87-c7f7778b387f/hdds]
at
org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker.checkVolume(StorageVolumeChecker.java:268)
at
org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.checkVolumeAsync(MutableVolumeSet.java:280)
at
org.apache.hadoop.ozone.container.common.utils.StorageVolumeUtil.onFailure(StorageVolumeUtil.java:41)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:234)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerFile(KeyValueContainer.java:255)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerData(KeyValueContainer.java:366)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:297)
at
org.apache.hadoop.ozone.container.common.impl.ContainerSet.lambda$handleVolumeFailures$0(ContainerSet.java:129)
at java.lang.Iterable.forEach(Iterable.java:75) at
org.apache.hadoop.ozone.container.common.impl.ContainerSet.handleVolumeFailures(ContainerSet.java:126)
at
org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.handleVolumeFailures(OzoneContainer.java:341)
at
org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.handleVolumeFailures(MutableVolumeSet.java:267)
at
org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.lambda$checkVolumeAsync$1(MutableVolumeSet.java:288)
at
org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.invokeCallback(StorageVolumeChecker.java:366)
at
org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.cleanup(StorageVolumeChecker.java:359)
at
org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.onSuccess(StorageVolumeChecker.java:337)
at
org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.onSuccess(StorageVolumeChecker.java:282)
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
We should shutdown the ResultHandlerExecutorService correctly when we shutdown
StorageVolumeChecker to prevent this OOM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
