I am doing some performance testing in NiFi. WorkFlow is GetSFTP -> update ->
PutKafka. I want to tune my setup to achieve high throughput without much
But my throughput average drops during flowfile checkpointing duration. I
believe stop-the-world is happening during that time.
I can roughly read ~100MB/s from SFTP and send almost same to Kafka. But every
2 mins, it stops the complete execution. Check below logs
2018-06-13 13:24:21,160 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
2018-06-13 13:24:49,420 INFO [Write-Ahead Local State Provider Maintenance]
org.wali.MinimalLockingWriteAheadLog@cf82c58 checkpointed with 23 Records and 0
Swap Files in 39353 milliseconds (Stop-the-world time = 3 milliseconds, Clear
Edit Logs time = 3 millis), max Transaction ID 68
2018-06-13 13:25:00,165 INFO [pool-10-thread-1]
o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 7
Records and 0 Swap Files in 39002 milliseconds (Stop-the-world time = 28275
milliseconds), max Transaction ID 316705
2018-06-13 13:25:00,169 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 7 records in 39008 milliseconds
I think all processor goes in idle state for 39 seconds ☹ .. Please guide how
to tune it..
I changed the heap memory with 32G [I am testing on 12 core, 48G machine]. I
disabled content-repository archiving. All other properties remains same.
Thanks & Regards,