I am retrieving many small csv files each of size 1MB (total folder size around
~100GB). In update step, I am doing some enrichment on ingress csv. Anyway my
flow doesn’t do anything with the stop the world time right?
Can you please tell me about flowfile checkpointing related tunings?
Thanks & Regards,
From: Mike Thomsen [mailto:mikerthom...@gmail.com]
Sent: Wednesday, June 13, 2018 2:33 PM
Subject: Re: NiFi Performance Analysis Clarification
What are you retrieving (particularly size) and what happens in the "update"
On Wed, Jun 13, 2018 at 4:10 AM V, Prashanth (Nokia - IN/Bangalore)
I am doing some performance testing in NiFi. WorkFlow is GetSFTP -> update ->
PutKafka. I want to tune my setup to achieve high throughput without much
But my throughput average drops during flowfile checkpointing duration. I
believe stop-the-world is happening during that time.
I can roughly read ~100MB/s from SFTP and send almost same to Kafka. But every
2 mins, it stops the complete execution. Check below logs
2018-06-13 13:24:21,160 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
2018-06-13 13:24:49,420 INFO [Write-Ahead Local State Provider Maintenance]
checkpointed with 23 Records and 0 Swap Files in 39353 milliseconds
(Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 3 millis), max
Transaction ID 68
2018-06-13 13:25:00,165 INFO [pool-10-thread-1]
o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 7
Records and 0 Swap Files in 39002 milliseconds (Stop-the-world time = 28275
milliseconds), max Transaction ID 316705
2018-06-13 13:25:00,169 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 7 records in 39008 milliseconds
I think all processor goes in idle state for 39 seconds ☹ .. Please guide how
to tune it..
I changed the heap memory with 32G [I am testing on 12 core, 48G machine]. I
disabled content-repository archiving. All other properties remains same.
Thanks & Regards,