[
https://issues.apache.org/jira/browse/FLINK-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249688#comment-16249688
]
Piotr Nowojski commented on FLINK-7845:
---------------------------------------
Please do not close this ticket, since
{code:java}
java.lang.IllegalAccessError:
org/apache/flink/runtime/io/network/netty/NettyMessage
{code}
is indeed a bug in Flink. It was showing up only after 20-60 iterations because
only at that point JIT was deciding to compile this method:
{code:java}
106188 15451 3
org.apache.flink.shaded.netty4.io.netty.util.internal.__matchers__.org.apache.flink.runtime.io.network.netty.NettyMessageMatcher::match
(5 bytes)
{code}
Before compilation, for some strange reasons interpreted version of this method
is working fine. Apparently interpreted and compiled code were slightly
different. It might have something to do with {{NoOpTypeParameterMatcher}}
being super simple and always returning {{true}}.
[~f.pompermaier] if you will be able to find something new regarding this
suspected memory leak please open new ticket or write on the user mailing list.
However keep in mind that in use cases were you are submitting indefinitely
multiple jobs to the stand alone Flink cluster, that it's important that your
job should always clean up after themselves. Especially pay attention at the
3rd party libraries, if they are not initialising some static contexts that are
never deinitialised. Sometimes it might be helpful to make sure that number of
threads is not increasing overtime. If it is, find out to whom those threads
belong.
> Netty Exception when submitting batch job repeatedly
> ----------------------------------------------------
>
> Key: FLINK-7845
> URL: https://issues.apache.org/jira/browse/FLINK-7845
> Project: Flink
> Issue Type: Bug
> Components: Core, Network
> Affects Versions: 1.3.2
> Reporter: Flavio Pompermaier
> Attachments: Screen Shot 2017-11-13 at 14.54.38.png
>
>
> We had some problems with Flink and Netty so we wrote a small unit test to
> reproduce the memory issues we have in production. It happens that we have to
> restart the Flink cluster because the memory is always increasing from job to
> job.
> The github project is https://github.com/okkam-it/flink-memory-leak and the
> JUnit test is contained in the MemoryLeakTest class (within src/main/test).
> I don't know if this is the root of our problems but at some point, usually
> around the 28th loop, the job fails with the following exception (actually we
> never faced that in production but maybe is related to the memory issue
> somehow...):
> {code:java}
> Caused by: java.lang.IllegalAccessError:
> org/apache/flink/runtime/io/network/netty/NettyMessage
> at
> io.netty.util.internal.__matchers__.org.apache.flink.runtime.io.network.netty.NettyMessageMatcher.match(NoOpTypeParameterMatcher.java)
> at
> io.netty.channel.SimpleChannelInboundHandler.acceptInboundMessage(SimpleChannelInboundHandler.java:95)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
