lement2 a single time? How does the
> framework decide when to switch the stream processing when the streams are
> connected?
>
>
>
> Regards,
>
> Alexis.
>
>
>
> *From:* David Morávek
> *Sent:* Donnerstag, 2. Dezember 2021 17:18
> *To:* Alexis Sarda-Espinos
: Buffering when connecting streams
Hi Alexis and David,
This actually can not happen. There are mechanisms in the code to make sure
none of the input is starved IF there is some data to be read.
The only time when input can be blocked is during the alignment phase of
aligned checkpoints under
pin...@microfocus.com> wrote:
>>
>> Yes, that sounds right, but with my current KeyedCoProcessFunction I
>> can’t tell Flink to "halt" processElement1 and switch to the other stream
>> depending on watermarks. I could look into TwoInputStreamOperator
gt;
>
> *From:* David Morávek
> *Sent:* Donnerstag, 2. Dezember 2021 17:18
> *To:* Alexis Sarda-Espinosa
> *Cc:* user@flink.apache.org
> *Subject:* Re: Buffering when connecting streams
>
>
>
> Even with the TwoInputStreamOperator you can not "halt"
. Dezember 2021 17:18
To: Alexis Sarda-Espinosa
Cc: user@flink.apache.org
Subject: Re: Buffering when connecting streams
Even with the TwoInputStreamOperator you can not "halt" the processing. You
need to buffer these elements for example in the ListState for later
processing. At th
k
> *Sent:* Donnerstag, 2. Dezember 2021 16:59
> *To:* Alexis Sarda-Espinosa
> *Cc:* user@flink.apache.org
> *Subject:* Re: Buffering when connecting streams
>
>
>
> I think this would require using lower level API and implementing a custom
> `TwoInputStreamOpera
ávek
Sent: Donnerstag, 2. Dezember 2021 16:59
To: Alexis Sarda-Espinosa
Cc: user@flink.apache.org
Subject: Re: Buffering when connecting streams
I think this would require using lower level API and implementing a custom
`TwoInputStreamOperator`. Then you can hook to `processWatemark{1,2}` metho
t;
>
>
> *From:* David Morávek
> *Sent:* Donnerstag, 2. Dezember 2021 15:45
> *To:* Alexis Sarda-Espinosa
> *Cc:* user@flink.apache.org
> *Subject:* Re: Buffering when connecting streams
>
>
>
> You can not rely on order of the two streams that easily. In case
reaches
processElement1, even when considering watermarks.
Regards,
Alexis.
From: David Morávek
Sent: Donnerstag, 2. Dezember 2021 15:45
To: Alexis Sarda-Espinosa
Cc: user@flink.apache.org
Subject: Re: Buffering when connecting streams
You can not rely on order of the two streams that easily
om open
> to close) triggered on second 17, and my windows are evaluated every
> minute, so it wasn’t a race condition.
>
>
>
> Regards,
>
> Alexis.
>
>
>
> *From:* David Morávek
> *Sent:* Donnerstag, 2. Dezember 2021 14:52
> *To:* Alexis Sarda-Espinosa
>
n
second 17, and my windows are evaluated every minute, so it wasn’t a race
condition.
Regards,
Alexis.
From: David Morávek
Sent: Donnerstag, 2. Dezember 2021 14:52
To: Alexis Sarda-Espinosa
Cc: user@flink.apache.org
Subject: Re: Buffering when connecting streams
Hi Alexis,
I'm not sure w
Hi Alexis,
I'm not sure what "watermark" step refers to in you graph, but in general
I'd say your intuition is correct.
For the "buffering" part, each sub-task needs to send data via data
exchange (last operator in chain) has an output buffer and the operator
that consumes this data (maybe on
Hello,
I have a use case with event-time processing that ends up with a DAG roughly
like this:
source -> filter -> keyBy -> watermark -> window -> process -> keyBy_2 ->
connect (KeyedCoProcessFunction) -> sink
| /
13 matches
Mail list logo
