Hi Team,

I am doing some performance testing in NiFi. WorkFlow is GetSFTP -> update -> 
PutKafka. I want to tune my setup to achieve high throughput without much 
queuing.
But my throughput average drops during flowfile checkpointing duration. I 
believe stop-the-world  is happening during that time.

I can roughly read ~100MB/s from SFTP and send almost same to Kafka. But every 
2 mins, it stops the complete execution. Check below logs

2018-06-13 13:24:21,160 INFO [pool-10-thread-1] 
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile 
Repository
2018-06-13 13:24:49,420 INFO [Write-Ahead Local State Provider Maintenance] 
org.wali.MinimalLockingWriteAheadLog 
org.wali.MinimalLockingWriteAheadLog@cf82c58 checkpointed with 23 Records and 0 
Swap Files in 39353 milliseconds (Stop-the-world time = 3 milliseconds, Clear 
Edit Logs time = 3 millis), max Transaction ID 68
2018-06-13 13:25:00,165 INFO [pool-10-thread-1] 
o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 7 
Records and 0 Swap Files in 39002 milliseconds (Stop-the-world time = 28275 
milliseconds), max Transaction ID 316705
2018-06-13 13:25:00,169 INFO [pool-10-thread-1] 
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
Repository with 7 records in 39008 milliseconds

I think all processor goes in idle state for 39 seconds ☹ .. Please guide how 
to tune it..
I changed the heap memory with 32G [I am testing on 12 core, 48G machine]. I 
disabled content-repository archiving. All other properties remains same.

Thanks & Regards,
Prashanth

Reply via email to