[
https://issues.apache.org/jira/browse/HDDS-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17636998#comment-17636998
]
Krishna Kumar Asawa commented on HDDS-5861:
-------------------------------------------
[~deveshsingh] Looks similar to issue you are working on.
> Recon container report processing can slow down when there are a lot of new
> containers to consume
> -------------------------------------------------------------------------------------------------
>
> Key: HDDS-5861
> URL: https://issues.apache.org/jira/browse/HDDS-5861
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Affects Versions: 1.2.0
> Reporter: Aravindan Vijayan
> Assignee: Devesh Kumar Singh
> Priority: Critical
>
> Recon checks and adds a container from SCM whenever it sees it for the first
> time. When there are a lot of new containers for Recon to consume due to it
> being down for a long time, then this report processing can hang on the RPC
> call, or even worse cause more bottleneck issues if SCM is down.
> {code}
> EventQueue-ContainerReportForReconContainerReportHandler
> PRIORITY : 5
> THREAD ID : 0X00007F2A6DDC3000
> NATIVE ID : 0XD324
> NATIVE ID (DECIMAL) : 54052
> STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.ipc.Client$Connection.addCall(Client.java:521)
> - waiting to lock <0x00007f1a70482730> (a
> org.apache.hadoop.ipc.Client$Connection)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:413)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1623)
> at org.apache.hadoop.ipc.Client.call(Client.java:1452)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy41.submitRequest(Unknown Source)
> at jdk.internal.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> - locked <0x00007f1a6ca20ad8> (a
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy41.submitRequest(Unknown Source)
> at
> org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.submitRpcRequest(StorageContainerLocationProtocolClientSideTranslatorPB.java:154)
> at
> org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.submitRequest(StorageContainerLocationProtocolClientSideTranslatorPB.java:144)
> at
> org.apache.hadoop.hdds.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.getContainerWithPipeline(StorageContainerLocationProtocolClientSideTranslatorPB.java:230)
> at
> org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl.getContainerWithPipeline(StorageContainerServiceProviderImpl.java:63)
> at
> org.apache.hadoop.ozone.recon.scm.ReconContainerManager.checkAndAddNewContainer(ReconContainerManager.java:122)
> at
> org.apache.hadoop.ozone.recon.scm.ReconContainerReportHandler.onMessage(ReconContainerReportHandler.java:62)
> at
> org.apache.hadoop.ozone.recon.scm.ReconContainerReportHandler.onMessage(ReconContainerReportHandler.java:38)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor$$Lambda$405/0x00007f19c2857d08.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run([email protected]/Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
