Thanks for the response guys.
Based on Niels response, it seems like a keyby immediately after reading
from the source should map all messages with the account number on the same
slot.
On Sun, Jul 29, 2018 at 05:33 Renjie Liu wrote:
> Hi,
> Another way to ensure order is by adding a logical
Hi,
Another way to ensure order is by adding a logical version number for each
message so that earlier version will not override later version. Timestamp
depends on your ntp server works correctly.
On Sun, Jul 29, 2018 at 3:52 PM Niels Basjes wrote:
> Hi,
>
> The basic thing is that you will
Hi,
The basic thing is that you will only get the messages in a guaranteed
order if the order is maintained in all steps from creation to use.
In Kafka order is only guaranteed for messages in the same partition.
So if you need them in order by account then the producing system must use
the
Hi,
Maybe the messages of the same key should be in the *same partition* of
Kafka topic
2018-07-29 11:01 GMT+08:00 Hequn Cheng :
> Hi harshvardhan,
> If 1.the messages exist on the same topic and 2.there are no rebalance and
> 3.keyby on the same field with same value, the answer is yes.
>
>
Hi harshvardhan,
If 1.the messages exist on the same topic and 2.there are no rebalance and
3.keyby on the same field with same value, the answer is yes.
Best, Hequn
On Sun, Jul 29, 2018 at 3:56 AM, Harshvardhan Agrawal <
harshvardhan.ag...@gmail.com> wrote:
> Hey,
>
> The messages will exist
Hey,
The messages will exist on the same topic. I intend to keyby on the same
field. The question is that will the two messages be mapped to the same
task manager and on the same slot. Also will they be processed in correct
order given they have the same keys?
On Fri, Jul 27, 2018 at 21:28 Hequn
Hi Harshvardhan,
There are a number of factors to consider.
1. the consecutive Kafka messages must exist in a same topic of kafka.
2. the data should not been rebalanced. For example, operators should be
chained in order to avoid rebalancing.
3. if you perform keyBy(), you should keyBy on a field
Hi,
We are currently using Flink to process financial data. We are getting
position data from Kafka and we enrich the positions with account and
product information. We are using Ingestion time while processing events.
The question I have is: say I key the position datasream by account number.
