[
https://issues.apache.org/jira/browse/HDDS-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043151#comment-17043151
]
JieWang edited comment on HDDS-3041 at 2/24/20 4:11 AM:
--------------------------------------------------------
[~msingh] Hi, the memory leak happens on grpc-java. The image can not show
completely, you can download the image and open it to see all the information.
1. When Physical memory of s3g is 10GB, jmap shows memory leak happens on jvm
heap.
!image-2020-02-24-12-06-22-248.png!
2. Restart s3g, dump the heap when physical memory cost 3GB, which shows there
exist 1384710 live instances of HashMap, the total size of all HashMap is
about 88MB. Besides there are 22MB of NettyClientTransport and 24MB of
NettyClientHandler.
!screenshot-1.png!
3. Trace the constructor method of HashMap by BTrace. There are a lot of stack
as the image shows. You can find 22MB of NettyClientTransport and 24MB of
NettyClientHandler of the heap are both on the stack, so I'm sure this is the
stack where memory leak happens. And from the stack you can also find grpc will
startNewTransport every time sendRequestAsync in ozone, which is not a normal
behavior. I'm not sure whether the root cause of memory leak is the bug of
grpc-java or error use of grpc in ozone. I will continue to investigate the
root cause, if you have any opinion please share it. Thank you very much.
!screenshot-2.png!
was (Author: yjxxtd):
[~msingh] Hi, the memory leak happens of grpc-java. The image can not show
completely, you can download the image and open it to see all the information.
1. When Physical memory of s3g is 10GB, jmap shows memory leak happens on jvm
heap.
!image-2020-02-24-12-06-22-248.png!
2. Restart s3g, dump the heap when physical memory cost 3GB, which shows there
exist 1384710 live instances of HashMap, the total size of all HashMap is
about 88MB. Besides there are 22MB of NettyClientTransport and 24MB of
NettyClientHandler.
!screenshot-1.png!
3. Trace the constructor method of HashMap by BTrace. There are a lot of stack
as the image shows. You can find 22MB of NettyClientTransport and 24MB of
NettyClientHandler of the heap are both on the stack, so I'm sure this is the
stack where memory leak happens. And from the stack you can also find grpc will
startNewTransport every time sendRequestAsync in ozone, which is not a normal
behavior. I'm not sure whether the root cause of memory leak is the bug of
grpc-java or error use of grpc in ozone. I will continue to investigate the
root cause, if you have any opinion please share it. Thank you very much.
!screenshot-2.png!
> memory leak of s3g
> ------------------
>
> Key: HDDS-3041
> URL: https://issues.apache.org/jira/browse/HDDS-3041
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
> Affects Versions: 0.6.0
> Reporter: JieWang
> Priority: Major
> Attachments: image-2020-02-24-12-06-22-248.png,
> image-2020-02-24-12-10-09-552.png, screenshot-1.png, screenshot-2.png,
> screenshot-3.png, screenshot-4.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
