Also thanks to you, Stefano, indeed for the nice community work you do!
Julian
Von: Stefano Bossi <stefano.bo...@gmail.com>
Datum: Dienstag, 25. August 2020 um 20:20
An: <dev@plc4x.apache.org>, Adam Rossi <ac.ro...@gmail.com>, Julian Feinauer
<j.feina...@pragmaticminds.de>
Betreff: Re: Leaking nioEventLoopGroup threads
Yupiii !!!
good news !!!
Regards,
S.
On 25/08/2020 15:45, Adam Rossi wrote:
Juilan,
My apologies - your fix did indeed work. The issue is that
PooledPlcDriverManager does not seem to be calling the close method on the
connection. Switching back to PlcDriverManager from PooledPlcDriverManager
results in your new log comments showing up in the log, and more
importantly no more leaks of the nioEventLoopGroup threads. I have tested
the code in a loop for an hour or so and it is working perfectly.
The PooledPlcDriverManager seems to intercept the close method on the
plcConnection (lines 125 - 130):
if ("close".equals(method.getName())) {
LOGGER.debug("close called on {}", plcConnection);
proxyInvalidated.set(true);
keyedObjectPool.returnObject(poolKey, plcConnection);
return null;
} else {
Which makes sense as it is trying to keep active connections pooled.
However, when this connection is again retrieved from the pool it seems the
plcConnection connects again and creates an additional nioEventLoopGroup
thread, which is never closed.
I am new to working with this project and Jira. It seems to me that I
should close the issue I just created as your fix does indeed correct the
original issue, and perhaps open another issue on PooledPlcDriverManager
for this thread leak?
Regards, Adam
On Mon, Aug 24, 2020 at 4:39 AM Julian Feinauer <
j.feina...@pragmaticminds.de<mailto:j.feina...@pragmaticminds.de>> wrote:
Hi,
short feedback. I looked into the code and indeed it seems that we had a
bug there which could lead tot he socket leak you described.
I pushed a fix in the branch:
https://github.com/apache/plc4x/tree/bugfix/close-eventloop-after-channel
Would you mind taking a look and testing this with your code @Adam Rossi?
Thanks!
Juliasn
Am 24.08.20, 08:26 schrieb "Julian Feinauer" <
j.feina...@pragmaticminds.de<mailto:j.feina...@pragmaticminds.de>>:
Perhaps, some related questions:
- You are using Linux for your Tests?
- Do you close all Connections properly?
Normally the `PlcConnection.close()` method should close the EventLoop.
Julian
Am 24.08.20, 08:23 schrieb "Julian Feinauer" <
j.feina...@pragmaticminds.de<mailto:j.feina...@pragmaticminds.de>>:
Hi Adam,
I will have a look today!
Do we have a Jira Issue for it already?
Julian
Am 24.08.20, 07:38 schrieb "Christofer Dutz" <
christofer.d...@c-ware.de<mailto:christofer.d...@c-ware.de>>:
Hi Adam,
of course that's unfortunate ... also I will not be able to
address this issue soon as I have to work on the tasks of my research
project.
I have one more month to work on this and I'm months behind
schedule because I have been doing free support way too much lately.
I really hope Julian will be able to help ... he's way more
into the details of Netty than I am (cause he's got the book ;-) )
So Julian? ... it would be super awesome if you could take on
this issue.
Chris
Am 24.08.20, 00:17 schrieb "Adam Rossi"
<ac.ro...@gmail.com><mailto:ac.ro...@gmail.com>:
Thanks, I did test with 0.8.0-SNAPSHOT and see the same
behavior. In every
plcConnection a nioEventLoopGroup thread is created and
does not ever seem
to be destroyed.
I wish I understood the io.netty.channel.EventLoopGroup
class better to be
more helpful here. Would an example program that
reproduces this thread
leak be useful?
jconsole shows the thread data as:
Name: nioEventLoopGroup-19-1
State: RUNNABLE
Total blocked: 0 Total waited: 0
Stack trace:
java.base@13.0.2/sun.nio.ch.EPoll.wait<mailto:java.base@13.0.2/sun.nio.ch.EPoll.wait>(Native
Method)
java.base@13.0.2<mailto:java.base@13.0.2>
/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
java.base@13.0.2<mailto:java.base@13.0.2>
/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
- locked
io.netty.channel.nio.SelectedSelectionKeySet@1838f97
- locked sun.nio.ch.EPollSelectorImpl@1f49287
java.base@13.0.2/sun.nio.ch<mailto:java.base@13.0.2/sun.nio.ch>
.SelectorImpl.select(SelectorImpl.java:141)
app//io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base@13.0.2/java.lang.Thread.run<mailto:java.base@13.0.2/java.lang.Thread.run>(Thread.java:830)
Regards, Adam
On Sun, Aug 23, 2020 at 1:00 PM Christofer Dutz <
christofer.d...@c-ware.de<mailto:christofer.d...@c-ware.de>>
wrote:
> Hi Adam,
>
> the Apache SNAPSHOT repo is at:
>
https://repository.apache.org/content/repositories/snapshots
>
> Adding this to your pom should help:
>
> <!-- Make Snapshots of Apache projects available -->
> <repositories>
> <repository>
> <id>apache-snapshots</id>
> <url>
https://repository.apache.org/content/repositories/snapshots
> </url>
> <releases>
> <enabled>false</enabled>
> </releases>
> <snapshots>
> <enabled>true</enabled>
> </snapshots>
> </repository>
> </repositories>
>
> <!-- Make Snapshots of Apache plugins available -->
> <pluginRepositories>
> <pluginRepository>
> <id>apache-snapshots</id>
> <url>
https://repository.apache.org/content/repositories/snapshots
> </url>
> <releases>
> <enabled>false</enabled>
> </releases>
> <snapshots>
> <enabled>true</enabled>
> </snapshots>
> </pluginRepository>
> </pluginRepositories>
>
> Chris
> Am 23.08.20, 18:56 schrieb "Adam Rossi" <
ac.ro...@gmail.com<mailto:ac.ro...@gmail.com>>:
>
> Sure thing. Is 0.8.0-snapshot hosted anywhere or is
that something
> that needs to be built? Regards Adam
>
> > On Aug 23, 2020, at 12:31 PM, Christofer Dutz <
> christofer.d...@c-ware.de<mailto:christofer.d...@c-ware.de>>
wrote:
> >
> > Hmm ...
> >
> > Could you possibly give 0.8.0-SNAPSHOT a try? ....
or 0.6.x? ...
> 0.7.0 was the first of the new generation drivers. We're
maintaining the
> 0.6 branch and working hard on making the new generation
drivers 100%
> production ready.
> >
> > Chris
> >
> >
> > Am 23.08.20, 18:06 schrieb "Adam Rossi" <
ac.ro...@gmail.com<mailto:ac.ro...@gmail.com>>:
> >
> > This is the latest 0.7.0 release from Maven.
> >
> >
> > Regards Adam
> >
> >> On Aug 23, 2020, at 11:56 AM, Christofer Dutz <
> christofer.d...@c-ware.de<mailto:christofer.d...@c-ware.de>>
wrote:
> >>
> >> Hi Adam,
> >>
> >> which version of PLC4X are you using? I know we
had similar reports
> some time ago, but had thought we had fixed them
> >>
> >> Chris
> >>
> >>
> >>
> >> Am 23.08.20, 16:40 schrieb "Adam Rossi" <
ac.ro...@gmail.com<mailto:ac.ro...@gmail.com>>:
> >>
> >> Howdy. I am seeing a persistent thread being
created for every
> >> plcConnection connect which looks like the
following:
> >>
> >> Name: nioEventLoopGroup-11-1
> >> State: RUNNABLE
> >> Total blocked: 0 Total waited: 0
> >>
> >> Stack trace:
> >>
java.base@13.0.2/sun.nio.ch.EPoll.wait<mailto:java.base@13.0.2/sun.nio.ch.EPoll.wait>(Native
Method)
> >> java.base@13.0.2<mailto:java.base@13.0.2>
> >>
/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
> >> java.base@13.0.2<mailto:java.base@13.0.2>
> >>
/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
> >> - locked
io.netty.channel.nio.SelectedSelectionKeySet@f6df6b
> >> - locked sun.nio.ch.EPollSelectorImpl@7ffc19
> >>
java.base@13.0.2/sun.nio.ch<mailto:java.base@13.0.2/sun.nio.ch>
> .SelectorImpl.select(SelectorImpl.java:141)
> >>
>
app//io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
> >>
>
app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
> >>
app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
> >>
>
app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> >>
>
app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> >>
>
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> >> java.base@13.0.2<mailto:java.base@13.0.2>
/java.lang.Thread.run(Thread.java:830)
> >>
> >>
> >> This happens both when creating a connection
directly or when
> using the
> >> plc4x connection pool.
> >>
> >> In both cases I am obtaining the connection,
reading some modbus
> values,
> >> and then closing the connection. From the
plcConnection metadata
> the
> >> connection type I am using is:
> >>
> >>
org.apache.plc4x.java.spi.connection.DefaultNettyPlcConnection
> >>
> >> Over time thousands of these nioEventLoopGroup
threads are
> created and I
> >> experience an OutOfMemory error in my program.
> >>
> >> Has anyone experienced this problem? Is there a
way to force
> these threads
> >> to close, timeout, or to specify the total
thread limit?
> >>
> >> Thanks and Regards, Adam
> >>
> >
>
>
