Edi,
Looking at your config again, you’ll want to also ensure that on the Publisher,
you set the partition to `${kafka.partition}` so that it goes to the same
partition on the destination system. You’ll also went to ensure that you set
“Failure Strategy” to “Rollback” - otherwise any failure would route to
‘failure’ relationship and change the ordering. You’ll also need to limit the
concurrent tasks on the publisher to 1 concurrent task, to ensure that you’re
not sending multiple FlowFiles out of order.
Thanks
-Mark
On Dec 15, 2023, at 2:26 AM, edi mari <[email protected]> wrote:
Hi Mark,
I tried the combination of FIFO and setting the back pressure to 10k, but it
didn't preserve the order.
Thanks
Edi
On Wed, Dec 13, 2023 at 3:47 PM Mark Payne
<[email protected]<mailto:[email protected]>> wrote:
Hey Edi,
By default, nifi doesn’t preserve ordering but you can have it do so by
updating the connection’s configuration and adding the First In First Out
Prioritizer.
Also of note you will want to keep the backpressure threshold set to 10,000
objects rather than increasing it as shown in the image.
Thanks
Mark
Sent from my iPhone
On Dec 13, 2023, at 8:19 AM, edi mari
<[email protected]<mailto:[email protected]>> wrote:
Hello ,
I'm using NIFI v1.20.0 to replicate 250 million messages between Kafka topics.
The problem is that NIFI replicates messages in a non-sequential order,
resulting in the destination topic storing messages differently than the source
topic.
for example
source topic - partition 0
offset:5 key:a value:v1
offset:6 key:a value:v2
offset:7 key:a value:v3
destination topic - partition 0
offset:5 key:a value:v2
offset:6 key:a value:v1
offset:7 key:a value:v3
The topics are configured with a cleanup policy: compact.
I'm using ConsumeKafka and PublishKafka processors to replicate topics.
<image.png>
<image.png>
<image.png>
<image.png>
<image.png>
Thanks
Edi
