Hi Mike, I am retrieving many small csv files each of size 1MB (total folder size around ~100GB). In update step, I am doing some enrichment on ingress csv. Anyway my flow doesn’t do anything with the stop the world time right?
Can you please tell me about flowfile checkpointing related tunings? Thanks & Regards, Prashanth From: Mike Thomsen [mailto:mikerthom...@gmail.com] Sent: Wednesday, June 13, 2018 2:33 PM To: users@nifi.apache.org Subject: Re: NiFi Performance Analysis Clarification What are you retrieving (particularly size) and what happens in the "update" step? Thanks, Mike On Wed, Jun 13, 2018 at 4:10 AM V, Prashanth (Nokia - IN/Bangalore) <prashant...@nokia.com<mailto:prashant...@nokia.com>> wrote: Hi Team, I am doing some performance testing in NiFi. WorkFlow is GetSFTP -> update -> PutKafka. I want to tune my setup to achieve high throughput without much queuing. But my throughput average drops during flowfile checkpointing duration. I believe stop-the-world is happening during that time. I can roughly read ~100MB/s from SFTP and send almost same to Kafka. But every 2 mins, it stops the complete execution. Check below logs 2018-06-13 13:24:21,160 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2018-06-13 13:24:49,420 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@cf82c58<mailto:org.wali.MinimalLockingWriteAheadLog@cf82c58> checkpointed with 23 Records and 0 Swap Files in 39353 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 3 millis), max Transaction ID 68 2018-06-13 13:25:00,165 INFO [pool-10-thread-1] o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 7 Records and 0 Swap Files in 39002 milliseconds (Stop-the-world time = 28275 milliseconds), max Transaction ID 316705 2018-06-13 13:25:00,169 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 7 records in 39008 milliseconds I think all processor goes in idle state for 39 seconds ☹ .. Please guide how to tune it.. I changed the heap memory with 32G [I am testing on 12 core, 48G machine]. I disabled content-repository archiving. All other properties remains same. Thanks & Regards, Prashanth
