Re: java.lang.StackOverflowError

2020-01-22 文章 刘建刚 
多谢，已经找到解决的issue了：https://issues.apache.org/jira/browse/FLINK-10367 


> 2020年1月22日 下午4:48，zhisheng  写道：
> 
> 1、建议问题别同时发到三个邮件去
> 2、找找还有没有更加明显的异常日志
> 
> 刘建刚  于2020年1月22日周三 上午10:25写道：
> 
>> I am using flink 1.6.2 on yarn. State backend is rocksdb.
>> 
>>> 2020年1月22日 上午10:15，刘建刚  写道：
>>> 
>>>  I have a flink job which fails occasionally. I am eager to avoid
>> this problem. Can anyone help me? The error stacktrace is as following:
>>> java.io.IOException: java.lang.StackOverflowError
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
>>>  at
>> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
>>>  at org.apache.flink.streaming.runtime.io
>> .StreamInputProcessor.processInput(StreamInputProcessor.java:236)
>>>  at
>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
>>>  at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
>>>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
>>>  at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.StackOverflowError
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.InputChannel.setError(InputChannel.java:203)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
>>>  at org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
>>>  at org.apache.flink.runtime.io
>> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
>>>  at
>> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
>>>  at org.apache.flink.runtime.io
>>

[State Processor API] how to convert savepoint back to broadcast state

2020-01-22 文章 Jin Yi 
Hi there,

I would like to read the savepoints (for broadcast state) back into the
broadcast state, how should I do it?

// load the existingSavepoint;
ExistingSavepoint existingSavepoint = Savepoint.load(environment,
"file:///tmp/new_savepoints", new MemoryStateBackend());

// read state from existing savepoint
dataSet = existingSavepoint.readBroadcastState(OPERATOR_UID,
"largeKeySetStateDescription", BasicTypeInfo.STRING_TYPE_INFO,
BasicTypeInfo.STRING_TYPE_INFO);

// TODO in BoradcastProcessFunction, how can I put the savepoint
dataset back into BroadcastState?

Thanks!

Eleanore

Re: request for a flink sink

2020-01-22 文章 zhisheng 
 hi，flink don't have facebook faiss connector now, you can custom Sink
(implement SinkFunction)

容祖儿  于2020年1月22日周三 下午7:55写道：

> Hi members,
>
> Do you know if there is a sink who writes data to facebook faiss[1]?
> I am looking for a sink class like this one [2].
>
> [1] https://github.com/facebookresearch/faiss
> [2]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/zh/dev/connectors/rabbitmq.html
>
> Thank you.
>

request for a flink sink

2020-01-22 文章 容祖儿 
Hi members,

Do you know if there is a sink who writes data to facebook faiss[1]?
I am looking for a sink class like this one [2].

[1] https://github.com/facebookresearch/faiss
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/zh/dev/connectors/rabbitmq.html

Thank you.

Re: java.lang.StackOverflowError

2020-01-22 文章 zhisheng 
1、建议问题别同时发到三个邮件去
2、找找还有没有更加明显的异常日志

刘建刚  于2020年1月22日周三 上午10:25写道：

> I am using flink 1.6.2 on yarn. State backend is rocksdb.
>
> > 2020年1月22日 上午10:15，刘建刚  写道：
> >
> >   I have a flink job which fails occasionally. I am eager to avoid
> this problem. Can anyone help me? The error stacktrace is as following:
> > java.io.IOException: java.lang.StackOverflowError
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.InputChannel.checkError(InputChannel.java:191)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.getNextBuffer(RemoteInputChannel.java:194)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:589)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:546)
> >   at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:175)
> >   at org.apache.flink.streaming.runtime.io
> .StreamInputProcessor.processInput(StreamInputProcessor.java:236)
> >   at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
> >   at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
> >   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:754)
> >   at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.StackOverflowError
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.notifyChannelNonEmpty(SingleInputGate.java:656)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.InputChannel.notifyChannelNonEmpty(InputChannel.java:125)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.InputChannel.setError(InputChannel.java:203)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:403)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.notifyBufferAvailable(RemoteInputChannel.java:380)
> >   at org.apache.flink.runtime.io
> .network.buffer.LocalBufferPool.recycle(LocalBufferPool.java:282)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.deallocate(NetworkBuffer.java:172)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:95)
> >   at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:84)
> >   at org.apache.flink.runtime.io
> .network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:147)
> >   at org.apache.flink.runtime.io
>

Re: flink on yarn任务启动报错 The assigned slot container_e10_1579661300080_0005_01_000002_0 was removed.

2020-01-22 文章 zhisheng 
应该是你作业之前挂过了

郑 洁锋  于2020年1月22日周三 上午11:16写道：

> 大家好，
>flink on yarn任务启动时，发现报错了The assigned slot
> container_e10_1579661300080_0005_01_02_0 was removed.
>环境：flink1.8.1，cdh5.14.2，kafka0.10，jdk1.8.0_241
>
> flink版本为1.8.1，yarn上的日志：
>
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:
> 
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  Starting
> YarnJobClusterEntrypoint (Version: , Rev:7297bac, Date:24.06.2019
> @ 23:04:28 CST)
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  OS current user:
> cloudera-scm
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  Current
> Hadoop/Kerberos user: root
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  JVM: Java
> HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.241-b07
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  Maximum heap size:
> 406 MiBytes
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  JAVA_HOME:
> /usr/java/default
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  Hadoop version: 2.6.5
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  JVM Options:
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xms424m
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint: -Xmx424m
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  Program Arguments:
> (none)
> 20/01/22 11:07:53 INFO entrypoint.ClusterEntrypoint:  Classpath:
>