Re: 大家怎么学习flink的呢

好的，感谢指导 > 在 2019年2月14日，下午1:19，Tzu-Li (Gordon) Tai 写道： > > Hi, > > 除了 Apache Flink 官方文件以外 [1]，我個人也建議可以看看 Ververica 這一系列的 Flink training 題材: > https://training.ververica.com/ > 除此之外，學習過程中有遇到任何問題也歡迎可以直接發信件跟我們詢問。 > > - Gordon > > [1] https://flink.apache.org/ > > On Thu, Feb 14, 2019 at 11:44

Re: flink使用异常超时

Hi 可以考虑是否 taskmanager 的 GC 比较严重 Best, Congxian cousin-gmail 于2019年2月14日周四 下午2:34写道： > 嘿，我这里使用flink on yarn中，经常报出异常，然后flink就自己关闭了。 > > 里面具体的逻辑是从kafka中接收数据，然后按照enentTime中的window滑动窗口滑动， > 窗口大小为1小时，滑动间隔是5秒。聚集数据后，就写到redis中。 > > 一般运行了2个小时候，就报异常，然后就结束了任务。其中，jobmanager的日志中显 > 示为： >

flink使用异常超时

嘿，我这里使用flink on yarn中，经常报出异常，然后flink就自己关闭了。 里面具体的逻辑是从kafka中接收数据，然后按照enentTime中的window滑动窗口滑动， 窗口大小为1小时，滑动间隔是5秒。聚集数据后，就写到redis中。 一般运行了2个小时候，就报异常，然后就结束了任务。其中，jobmanager的日志中显 示为： java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_e23_1545597259276_0273_01_001220 timed

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi, @Averell I renamed the `ElasticsearchFailureHandlerIndexer` to be `BufferingNoOpRequestIndexer`, which explains why you can't find it. The voting thread for RC#1 of 1.7.2 can be found at [1]. The actual commits which fixes the problem are d9c45af to 2f52227. Cheers, Gordon [1]

Re: 大家怎么学习flink的呢

Hi, 除了 Apache Flink 官方文件以外 [1]，我個人也建議可以看看 Ververica 這一系列的 Flink training 題材: https://training.ververica.com/ 除此之外，學習過程中有遇到任何問題也歡迎可以直接發信件跟我們詢問。 - Gordon [1] https://flink.apache.org/ On Thu, Feb 14, 2019 at 11:44 AM shen lei wrote: > 有木有好的经验或者方法分享一下，感谢。最近学的，感觉还是不系统。

Re: Impact of occasional big pauses in stream processing

Hi Ajay, Flink handles "backpressure" in a graceful way so that it doesn't get affected when your processing pipeline is occasionally slowed down. I think the following articles will help [1,2]. In your specific case: the "KeyBy" operation will re-hash data so they can be reshuffled from all

大家怎么学习flink的呢

有木有好的经验或者方法分享一下，感谢。最近学的，感觉还是不系统。

Re: Production readiness

Hi, Andrey I have another question that if i do not set the maximum parallelism first(which be set to 128 by default), and then rescale to a parallelism bigger than 128. In this scenario，will the state lost? Thanks, Aitozi -- Sent from:

Re: flink restoring from state

Hi, Avi I think the "*Checkpoint failed: The assigned slot container_e02_1550091678485_0001_01_23_7 was removed"*(this may be a container failure or something else, could double check the taskamanger log for more information)and *"**Checkpoint failed: Checkpoint Coordinator is suspending"

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi Averell, https://github.com/apache/flink/commit/35af99391dac431c85e30bcc98b89cba79bccfea#diff-51a12ea54593424e195dd5874309a08d …is the commit where Gordon made his changes

Re: [ANNOUNCE] New Flink PMC member Thomas Weise

Congrats Thomas! Best, Congxian françois lacombe 于2019年2月13日周三 下午4:52写道： > Congratulation Thomas > > Thanks for help you provide and useful inputs > > François > > Le mer. 13 févr. 2019 à 03:13, Kurt Young a écrit : > >> Congrats Thomas! >> >> Best, >> Kurt >> >> >> On Wed, Feb 13, 2019 at

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi Ken, Thanks for that. But I could not find the changes included in Gordon's mentioned pull request in the repository you gave me (e.g: the new class /ElasticsearchFailureHandlerIndexer/). I have found this folder https://dist.apache.org/repos/dist/dev/flink/flink-1.7.2-rc1/, but it also

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi Averell, You can get release candidates from the Apache release candidate maven repo. For 1.7.2, I think it’s in: https://repository.apache.org/content/repositories/orgapacheflink-1206/ So just edit your pom.xml to

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Hi Gordon, Sorry for a noob question: How can I get the RC 1.7.2 build / code to build? I could not find any branch like that in Github. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: [DISCUSS] Adding a mid-term roadmap to the Flink website

Thanks Stephan for the great proposal. This would not only be beneficial for new users but also for contributors to keep track on all upcoming features. I think that better window operator support can also be separately group into its own category, as they affects both future DataStream API and

Impact of occasional big pauses in stream processing

I was wondering what is the impact if one of the stream operator function occasionally takes too long to process the event. Given the following simple flink job inputStream .KeyBy (“tenantId”) .process ( new MyKeyedProcessFunction()) , if occasionally

Re: Using RemoteExecutionEnvironment from IDE

Hi Benjamin, what kind of problem with RemoteInputChannels are you debugging? If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager. Best, Andrey [1]

Re: Production readiness

Hi Avi, The maximum parallelism is not an easy parameter to change for a job, once the job is started. The checkpoints/savepoints of the job will need migration to rehash the keyed state entries to the different number of key groups (unit of keyed state storage). You can try Bravo tool for it

Production readiness

Hi Looking at the production readiness checklist - is there any rule of thumb to determine the maximum parallelism ? we have a stateful pipeline with high throughput

Re: Broadcast state before events stream consumption

Hi Konstantin, For the second solution, would savepoint persist the Broadcast state in State backend? Because I am aware that Broadcast state is not checkpointed.  Is that correct? Thanks, Chirag Sent from Yahoo Mail on Android On Mon, 11 Feb 2019 at 2:39 PM, Konstantin Knauf wrote: Hi

Re: Data loss when restoring from savepoint

Sorry not posting on the mail list was my mistake :/ On Wed, 13 Feb 2019 at 15:01, Juho Autio wrote: > Thanks for stepping in, did you post outside of the mailing list on > purpose btw? > > This I did long time ago: > > To rule out for good any questions about sink behaviour, the job was >>

Re: Data loss when restoring from savepoint

Stefan (or anyone!), please, could I have some feedback on the findings that I reported on Dec 21, 2018? This is still a major blocker.. On Thu, Jan 31, 2019 at 11:46 AM Juho Autio wrote: > Hello, is there anyone that could help with this? > > On Fri, Jan 11, 2019 at 8:14 AM Juho Autio wrote:

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Thanks for testing it out. Will be great to get your feedback on whether the release candidate for 1.7.2 fixes this for you. On Wed, Feb 13, 2019 at 7:38 PM Averell wrote: > Thank you Gordon. > > That's my exact problem. Will try the fix in 1.7.2 now. > > Thanks and regards, > Averell > > > >

Re: No resource available error while testing HA

Hi Gary, Thanks for the suggestion. How about changing the configuration of the Flink job itself during runtime? What I have to do now is to take a savepoint, stop the job, change the configuration, and then restore the job from the save point. Is there any easier way to do that? Thanks and

Re: [DISCUSS] Adding a mid-term roadmap to the Flink website

Very excited and thank you for launching such a great discussion, Stephan ! Here only a little suggestion that in the Batch Streaming Unification section, do we need to add an item: - Same window operators on bounded/unbounded Table API and DataStream API (currently OVER window only exists in

Re: ElasticSearchSink - retrying doesn't work in ActionRequestFailureHandler

Thank you Gordon. That's my exact problem. Will try the fix in 1.7.2 now. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[DISCUSS] Adding a mid-term roadmap to the Flink website

Hi all! Recently several contributors, committers, and users asked about making it more visible in which way the project is currently going. Users and developers can track the direction by following the discussion threads and JIRA, but due to the mass of discussions and open issues, it is very

Using RemoteExecutionEnvironment from IDE

Hi all, I’m currently debugging and enjoy it executing my task directly from the IDE. The IDE starts a LocalExecutionEnvironment and immediately runs the task. Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.) To run

Re: [ANNOUNCE] New Flink PMC member Thomas Weise

Congratulation Thomas Thanks for help you provide and useful inputs François Le mer. 13 févr. 2019 à 03:13, Kurt Young a écrit : > Congrats Thomas! > > Best, > Kurt > > > On Wed, Feb 13, 2019 at 10:02 AM Shaoxuan Wang > wrote: > >> Congratulations, Thomas! >> >> On Tue, Feb 12, 2019 at 5:59