Re: java.lang.StackOverflowError

2020-01-23 文章 Piotr Nowojski
Hi,

Thanks for reporting the issue. Could you first try to upgrade to Flink 1.6.4? 
This might be a known issue fixed in a later bug fix release [1].

Also, are you sure you are using (unmodified) Flink 1.6.2? Stack traces somehow 
do not match with the 1.6.2 release tag in the repository, for example this one:

at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
 

Doesn’t match the code [2]. (But first try to upgrade the Flink)

Piotrek


[1] https://issues.apache.org/jira/browse/FLINK-10367 
<https://issues.apache.org/jira/browse/FLINK-10367>
[2] 
https://github.com/apache/flink/blob/3456ad0dcacb3163e8aecf8fbe2dd684404cf37d/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java#L380
 
<https://github.com/apache/flink/blob/3456ad0dcacb3163e8aecf8fbe2dd684404cf37d/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java#L380>

> On 22 Jan 2020, at 03:24, 刘建刚  wrote:
> 
> I am using flink 1.6.2 on yarn. State backend is rocksdb. 
> 
>> 2020年1月22日 上午10:15,刘建刚 > <mailto:liujiangangp...@gmail.com>> 写道:
>> 
>>   I have a flink job which fails occasionally. I am eager to avoid this 
>> problem. Can anyone help me? The error stacktrace is as following:
>> java.io.IOException: java.lang.StackOverflowError
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
>>  at 
>> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
>>  at 
>> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:236)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
>>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
>>  at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.StackOverflowError
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.setError(InputChannel.java:203)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
>>  at 
>> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>  at 
>> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>  at 
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>  at 
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>  at 
>> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>  at 
>> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>  at 
>> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>  at 
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>  at 
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>  at 
>> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>  at 
>> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>  at 
>> org.a

Re: java.lang.StackOverflowError

2020-01-22 文章 刘建刚
多谢,已经找到解决的issue了:https://issues.apache.org/jira/browse/FLINK-10367 
<https://issues.apache.org/jira/browse/FLINK-10367>

> 2020年1月22日 下午4:48,zhisheng  写道:
> 
> 1、建议问题别同时发到三个邮件去
> 2、找找还有没有更加明显的异常日志
> 
> 刘建刚  于2020年1月22日周三 上午10:25写道:
> 
>> I am using flink 1.6.2 on yarn. State backend is rocksdb.
>> 
>>> 2020年1月22日 上午10:15,刘建刚  写道:
>>> 
>>>  I have a flink job which fails occasionally. I am eager to avoid
>> this problem. Can anyone help me? The error stacktrace is as following:
>>> java.io.IOException: java.lang.StackOverflowError
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
>>>  at
>> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
>>>  at org.apache.flink.streaming.runtime.io
>> .StreamInputProcessor.processInput(StreamInputProcessor.java:236)
>>>  at
>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
>>>  at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
>>>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
>>>  at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.StackOverflowError
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.InputChannel.setError(InputChannel.java:203)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputCha

Re: java.lang.StackOverflowError

2020-01-22 文章 zhisheng
1、建议问题别同时发到三个邮件去
2、找找还有没有更加明显的异常日志

刘建刚  于2020年1月22日周三 上午10:25写道:

> I am using flink 1.6.2 on yarn. State backend is rocksdb.
>
> > 2020年1月22日 上午10:15,刘建刚  写道:
> >
> >   I have a flink job which fails occasionally. I am eager to avoid
> this problem. Can anyone help me? The error stacktrace is as following:
> > java.io.IOException: java.lang.StackOverflowError
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
> >   at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
> >   at org.apache.flink.streaming.runtime.io
> .StreamInputProcessor.processInput(StreamInputProcessor.java:236)
> >   at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
> >   at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
> >   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
> >   at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.StackOverflowError
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.InputChannel.setError(InputChannel.java:203)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByte

Re: java.lang.StackOverflowError

2020-01-21 文章 刘建刚
I am using flink 1.6.2 on yarn. State backend is rocksdb. 

> 2020年1月22日 上午10:15,刘建刚  写道:
> 
>   I have a flink job which fails occasionally. I am eager to avoid this 
> problem. Can anyone help me? The error stacktrace is as following:
> java.io.IOException: java.lang.StackOverflowError
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:236)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.StackOverflowError
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.InputChannel.setError(InputChannel.java:203)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
>   at 
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>   at 
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>   at 
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>   at 
> org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>   at 
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>   at 
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>  

java.lang.StackOverflowError

2020-01-21 文章 刘建刚
  I have a flink job which fails occasionally. I am eager to avoid this
problem. Can anyone help me? The error stacktrace is as following:

java.io.IOException: java.lang.StackOverflowError
at 
org.apache.flink.runtime.io.network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
at 
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
at 
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
at 
org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
at 
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:236)
at 
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.StackOverflowError
at 
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
at 
org.apache.flink.runtime.io.network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
at 
org.apache.flink.runtime.io.network.partition.consumer.InputChannel.setError(InputChannel.java:203)
at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
at 
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
at 
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
at 
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
at 
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
at 
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
at 
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
at 
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
at 
org.apache.flink.runtime.io.network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
at 
org.apache.flink.shaded.netty4