Behavior of time based operators

Hello, I have recently started reading Stream Processing with Apache Flink by Fabian and Vasiliki. In Chapter 3 of the book there is a statement that says: None of the functions expose an API to set time stamps of emitted records, manipulate the event-time clock of a task, or emit watermarks.

Re: Order of events in a Keyed Stream

me partition* of >>> Kafka topic >>> >>> 2018-07-29 11:01 GMT+08:00 Hequn Cheng : >>> >>>> Hi harshvardhan, >>>> If 1.the messages exist on the same topic and 2.there are no rebalance >>>> and 3.keyby on the same field with same value,

Re: Order of events in a Keyed Stream

if you perform keyBy(), you should keyBy on a field the consecutive > two messages share the same value. > > Best, Hequn > > On Sat, Jul 28, 2018 at 12:11 AM, Harshvardhan Agrawal < > harshvardhan.ag...@gmail.com> wrote: > >> Hi, >> >> >> We are cur

Order of events in a Keyed Stream

Hi, We are currently using Flink to process financial data. We are getting position data from Kafka and we enrich the positions with account and product information. We are using Ingestion time while processing events. The question I have is: say I key the position datasream by account number.

Re: Implement Joins with Lookup Data

it can also be an All data cache. The All data > cache can be updated, say each 2h, according to our requirement. > > Adding a cache can not only simplify your pipeline but also improve the > job performance. > > Best, Hequn > > > On Mon, Aug 20, 2018 at 5:42 AM, Harsh

Re: Implement Joins with Lookup Data

e: > > Yes, using Kafka which you initialize with the initial values and then > feed changes to the Kafka topic from which you consume could be a solution. > > On Tue, Jul 24, 2018 at 3:58 PM Harshvardhan Agrawal < > harshvardhan.ag...@gmail.com> wrote: > > Hi Till, >

Implement Joins with Lookup Data

with the first approach or the second one. If the second one, how can I implement the join? -- *Regards,Harshvardhan Agrawal*

Triggers for late elements

to understand the how this works? -- *Regards,Harshvardhan Agrawal*

Behaviour of triggers in Flink

here? -- *Regards,Harshvardhan Agrawal* *267.991.6618 | LinkedIn <https://www.linkedin.com/in/harshvardhanagr/>*

Re: Behaviour of triggers in Flink

flink-docs-master/dev/stream/operators/windows.html#default-triggers-of-windowassigners > > On Sun, Jul 22, 2018 at 11:59 PM, Harshvardhan Agrawal < > harshvardhan.ag...@gmail.com> wrote: > >> Hi, >> >> I have been trying to understand how triggers work in Flink. W

Re: Implement Joins with Lookup Data

d changes to the Kafka topic from which you consume could be a solution. > > On Tue, Jul 24, 2018 at 3:58 PM Harshvardhan Agrawal < > harshvardhan.ag...@gmail.com> wrote: > > Hi Till, > > How would we do the initial hydration of the Product and Account data > since i

Re: Implement Joins with Lookup Data

; incoming event. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/ > [2] > https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/co/CoMapFunction.html > > Cheers, > Till > > On Tue, J

Re: Implement Joins with Lookup Data

cron to enrich later since > your processing doesn’t seem to require absolute real time. > > > > Thanks > > Ankit > > > > *From: *Jörn Franke > *Date: *Monday, July 23, 2018 at 10:10 PM > *To: *Harshvardhan Agrawal > *Cc: * > *Subject: *Re: Implement Jo

Behaviour of Process Window Function

in the clear method will do the trick? -- *Regards,Harshvardhan Agrawal*

Re: Behaviour of Process Window Function

siderable flexibility, you can >> based on processing time / event time / timer / it's clear method / >> customized implementation, the specific design depends on your business >> logic, how long you need to save the cache. >> >> Thanks, vino. >> >>

Between Checkpoints in Kafka 11

Hi, I was going through the blog post on how TwoPhaseCommitSink function works with Kafka 11. One of the things I don’t understand is: What is the behavior of the Kafka 11 Producer between two checkpoints? Say that the time interval between two checkpoints is set to 15 minutes. Will Flink buffer

Take RocksDB state dump

Hello, We are currently using a RocksDBStateBackend for our Flink pipeline. We want to analyze the data that is stored in Rocksdb state.Is there a recommended process to do that? The sst_dump tool available from RocksDB isn’t working for us and we keep on getting errors like “Snappy not

Re: Between Checkpoints in Kafka 11

Hi, Can someone please help me understand how does the exactly once semantic work with Kafka 11 in Flink? Thanks, Harsh On Tue, Sep 11, 2018 at 10:54 AM Harshvardhan Agrawal < harshvardhan.ag...@gmail.com> wrote: > Hi, > > I was going through the blog post on how TwoPhaseComm

Writing a custom Rocksdb statistics collector

te RocksDB options object for each of the slots. Is this a good way to approach this problem? Do you think this will work? Thanks in advance! :) -- *Regards,Harshvardhan Agrawal*

Re: Writing a custom Rocksdb statistics collector

gt; Best > Yun Tang > -- > *From:* Harshvardhan Agrawal > *Sent:* Thursday, January 31, 2019 0:23 > *To:* user > *Subject:* Writing a custom Rocksdb statistics collector > > > Hi, > > I am currently trying to integrate RocksDB statist

Unsubscribe

-- Regards, Harshvardhan