Hello Team, We have a apache Nifi cluster with 3 nodes and 3 nodes kafka cluster.We are receiving some files which has transactions in orders.(A-type first and than B-type) These events are in order but may come is different files.For example A-event for id 111 can be present in file 1 and B-event can come in immedaite file2 [B will always come after A-type for any ID].We want data need to be puslished in the same order as it is received.
We developed a flow using ListSFTP+FecthFTP+publishkafka combination in order ,have also done partitioning on kafka topic[9 partitions] on the basics of a key column and same key is used in Publish Kafka Processor. Al the events are published to the same partition but are going out of order but within the partition are out of order. Example B-type events are coming before A-Type in kafka topic TEST. Now i have some queries regarding the above What i understood is that since the ListSFTP+FecthFTP improves load balancing but does it ensures ordering? File1 may go to Node1 and File2 may go to Node2 , and Node 2 can publish the record to the same partition on kafka before Node1? Is there any way to gaurantee load order of files in Apache Nifi in cluster Mode keeping perfomance in mind.? Since each task in PublishKafka processor is one publisher , if we run the publish kafka on only primary node and pass only one broker-id does it will do the trick? -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
