Re: Bootstrapping

Hi, I see this coming up more and more often these days. For now, the solution of doing a savepoint and switching sources should work but I've had it in my head for a while now to add functionality for bootstrapping inputs in the API. An operator would read from the bootstrap stream (which is

Re: Understanding Restart Strategy

Thanks for the update! > On 25. Jan 2018, at 04:12, Ashish Pokharel wrote: > > FYI, > > I think I have gotten to the bottom this situation. For anyone who might be > in situation hopefully my observations will help. > > In my case, it had nothing to do with Flink Restart

Re: Forcing consuming one stream completely prior to another starting

Hi, How can you determine whether the required decoration data for an event from the main stream is there? If it works via event-time you could think about buffering main-input events in the operator until the corresponding decoration arrives. On a side note, we're currently working on

Re: flink read hdfs file error

Hi, It seems you are using Akka in your user program and the reference.conf in your Jar is clashing with the same File from Flink. What is the reason for having Akka (or reference.conf) in your user jar? Best, Aljoscha > On 22. Jan 2018, at 11:09, 韩宁宁 <453673...@qq.com> wrote: > > Dear All >

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Great, thank you! Hopefully, this pushes the PR forward. Thanks, Fabian 2018-01-25 22:30 GMT+01:00 Christophe Jolif : > Hi Fabian, > > FYI I rebased the branch and tested it and it worked OK on a sample. > > -- > Christophe > > On Mon, Jan 22, 2018 at 2:53 PM, Fabian Hueske

Re: Scheduling of GroupByKey and CombinePerKey operations

Hi, What are the other stages in that program? Best, Aljoscha > On 18. Jan 2018, at 16:22, Fabian Hueske wrote: > > Hi Pawel, > > This question might be better suited for the Beam user list. > Beam includes the Beam Flink runner which translates Beam programs into Flink >

Re: Low throughput when trying to send data with Sockets

Taking a step back: why do you want to manually implement communication via sockets in the first place? With this you will not get any fault-tolerance guarantees and I would guess that maintaining a custom solution is more difficult than using, say, Kafka. Best, Aljoscha > On 16. Jan 2018, at

Re: Flip-6 + Dynamic scaling

Hi Jayant, I am working on FLIP-6, and I can tell you that we are aiming at shipping it with Flink 1.5. For the scope and release timeline see this blog post: https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html Best, Gary On Wed, Jan 24, 2018 at 7:02 AM, Jayant Ameta

Ingesting a table from a retraction stream !!

Hi I know currently Ingesting a table from a retraction stream is not supported yet. is there any plan to include in upcoming releases. -- *Cheers * *Puneet Kinra* *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com * *e-mail

Re: Trigger not firing when using BoundedOutOfOrdernessTimestampExtractor

A while back I wrote this slightly more elaborate extractor that will advance the watermark independently after the stream is idle for a while:

Re: Ingesting a table from a retraction stream !!

Hi Puneet, I'm very sure that this feature will be included in upcoming releases. However, such an interface has to be carefully designed because built-in operators assume that only records that have been emitted previously are retracted. Maybe it will be part of Flink 1.6 but 1.7 should

RE: How to deal with dynamic types

We’ve been using genericRecords with custom serializers to do exactly this. We need to run the same flink pipeline for 10s of thousands of different schemas for our use cases and code gening or building that many different jars just isn’t practical. From: madan

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Ok, I got it "done". I have a PR for ES5.3 (FLINK-7386) just rebasing the original one that was never merged (#4675). And added ES 6.X through RestHighLevelClient on top (FLINK-8101). This is: https://github.com/apache/flink/pull/5374. And believe it or not but someone else submitted a PR for

RE: Two operators consuming from same stream

We've seen the same thing here. We read files twice for the same reason, it's simply faster to do that than to connect the two pipes to the same input. From: Sofer, Tovi [mailto:tovi.so...@citi.com] Sent: Monday, January 01, 2018 8:51 AM To: user@flink.apache.org Subject: Two operators

Re: Ingesting a table from a retraction stream !!

Hi Puneet, I agree with Timo, such a feature should be carefully designed. Besides, I'm very interested with your business scenario. I am wondering where your retraction stream come from (Maybe mysql binlog? or ?). It would be great if you can tell more about the details. Maybe we can figure out