Re: Sliding window based on event arrival

Hi Xingcan, Thanks for the clarification. I was able to implement the scenario with the example you provided. Thanks,Miyuru On Monday, 12 March 2018, 12:53, Xingcan Cui wrote: Hi Miyuru, what you need should be something like a `SlidingCountWindow`. Flink

Re: POJO default constructor - how is it used by Flink?

Hi Alex, I guess your "real" constuctor is invoked by your code within your Flink program. The default constructor is used during serialization between operators. If you are interested in the internals, you can have a look at the PojoSerializer [1]. The POJO is created with the default

Re: Running the executables from Flink distribution of the source build

Can you try to look into log folder for error? May be some clue. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Record Delivery Guarantee with Kafka 1.0.0

Hi, I am trying to use Kafka Sink 0.11 with ATLEAST_ONCE semantic and experiencing some data loss on Task Manager failure. Its a simple job with parallelism=1 and a single Task Manager. After a few checkpoints(kafka flush's) i kill one of my Task Manager running as a container on Docker Swarm. 

Re: Share state across operators

Hey Flinker! Anyone? Anybody? Someone with experience or any idea on the question above? Best, Max -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Sliding window based on event arrival

Hi Miyuru, what you need should be something like a `SlidingCountWindow`. Flink Datastream API has already provided a `countWindow()` method for that and a related example can be found here

Re: Partial aggregation result sink

Hi LiYue, This should help : Apache Flink 1.5-SNAPSHOT Documentation: Windows | | | | Apache Flink 1.5-SNAPSHOT Documentation: Windows | | | So basically you need to register a processing time trigger at every 10 minutes and on callback, you can FIRE the window result like this:  

Re: HDFS data locality and distribution

Relevant versions: Beam 2.1, Flink 1.3. From: Reinier Kip Sent: 12 March 2018 13:45:47 To: user@flink.apache.org Subject: HDFS data locality and distribution Hey all, I'm trying to batch-process 30-ish files from HDFS, but I see that data is

HDFS data locality and distribution

Hey all, I'm trying to batch-process 30-ish files from HDFS, but I see that data is distributed very badly across slots. 4 out of 32 slots get 4/5ths of the data, another 3 slots get about 1/5th and a last slot just a few records. This probably triggers disk spillover on these slots and slows

Re: Share state across operators

Hi Max, I would go with the Either approach if you want to ensure that the initital state and the first element arrive in the right order. Performance-wise there should not be a big different between both approaches. The side outputs are more meant for have a side channel beside the

Re: Running the executables from Flink distribution of the source build

The scala-shell does not currently work out-of-the-box in 1.5-SNAPSHOT/1.6-SNAPSHOT. As a workaround please add "mode: old" to conf/flink-conf.yaml. See also: https://issues.apache.org/jira/browse/FLINK-8795 On 12.03.2018 02:19, kedar mhaswade wrote: Flink gurus! I have built Flink from

Re: Partial aggregation result sink

Hi Chirag, Thank for your reply! I found a provided ContinuousEventTimeTrigger should be worked in my situation. Most examples are based on Table API like ‘ds.keyBy(0).window().trigger(MyTrigger.of())…’, But how to apply the trigger to a pure Flink SQL Application ? > 在

Production-readyness of Flink SQL

Hi! I am quite sure this was asked before. But why is Flink SQL is declared to be in in Beta as given by the header in https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/ ? Are there specific functions in SQL which are not production ready or is it meant for the whole Flink

Re: [DISCUSS] Inverted (child-first) class loading

Hi Kedar, See some thoughts inline below. I will admit that classpath issues (as in “I can see my damn class in the file, I can load it from my code, but I’m getting a freakin’ class not found exception at runtime…arghhh”) have been one of the biggest hassles with Flink-based projects that

Re: Production-readyness of Flink SQL

Hi Philip, you are absolutely right that Flink SQL is definitely production ready. It has been developed for 2 years now and is used at Uber, Alibaba, Huawei and many other companies. We usually only merge production ready code or add an explicit warning about it. I will finally remove this

Global Window, Trigger and Watermarks, Parallelism

Could someone clarify how exactly event time/watermarks and allow lateness work. I have created the program below and I have an input file such as... device_id,trigger_id,event_time,messageId 1,START,1520433909396,1 1,TRACKING,1520433914398,2 1,TRACKING,1520433919398,3

migrate AbstractStreamOperator from 1.0 to 1.4

Hi, I'm newbie in Flink and in streaming Engine and I'm starting to get familiar with and understand with the Stream API using examples that creates Its own Operator The examples used AbstractStreamOperator at version 1.0 and updating library to 1.4 I notice that restoreState

Re: [DISCUSS] Inverted (child-first) class loading

Hi Kedar, There is this section in the Flink docs: https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html Best, Aljoscha > On 10. Mar 2018, at 05:53, kedar

Re: [DISCUSS] Inverted (child-first) class loading

Many thanks Aljoscha! I am sorry I missed this section. Regards, Kedar On Mon, Mar 12, 2018 at 9:16 AM, Aljoscha Krettek wrote: > Hi Kedar, > > There is this section in the Flink docs: https://ci.apache.org/ >