[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the feedback [~Apache9]. Pushed to trunk.
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16635.patch, HBASE-16635_1.patch
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Attachment: HBASE-16635_1.patch
Updated patch addressing the comments.
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16635.patch, HBASE-16635_1.patch
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Status: Patch Available (was: Open)
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16635.patch, HBASE-16635_1.patch
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Status: Open (was: Patch Available)
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16635.patch
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Attachment: HBASE-16635.patch
Patch for 2.0. On AbstractRpcClient#close() we do cleanUp where the created
Bytebuf are released. Also handles SaslWrapHandler ByteBuf release but am not
sure if the place it is released is fine (should be). Ping for reviews.
[~Apache9]?
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16635.patch
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Status: Patch Available (was: Open)
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16635.patch
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16635:
---
Summary: RpcClient under heavy load leaks some netty bytebuf (was:
RpcClient under heave load leaks some netty bytebuf)
> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
