markap14 commented on issue #3131: NIFI-3229 When a queue contains only Penalized FlowFile's the next pr… URL: https://github.com/apache/nifi/pull/3131#issuecomment-463333448 Hey @patricker sorry for the delay in getting back to this. I looked into the unit tests, and did some quick profiling. It looks like the reason that it seemed to "go out to lunch" as you say was because of Mockito objects being used. They are super useful but don't provide great performance. I updated the test so that instead of using Mockito it just created simple objects like Funnels or implemented interfaces directly without much of anything happening. That resulted in much better performance. I updated the test about to run in multiple threads, as well, because this is where we are going to see the heavy contention and therefore the performance concerns. When I run with a single thread, we see pretty comparable results between the existing implementation and the new implementation that checks for penalization: > 1M checks for FlowFiles, non-empty queue: 57 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 101 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 47 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 67 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 60 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 77 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 38 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 66 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 34 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 54 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 28 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 91 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 67 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 96 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 35 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 59 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 29 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 55 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 36 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 50 millis > -------------------- > The difference is measurable but not necessarily concerning. At 2 threads, the performance is more concerning but still not necessarily a deal breaker: > 1M checks for FlowFiles, non-empty queue: 314 millis > 1M checks for FlowFiles, non-empty queue: 315 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 410 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 412 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 240 millis > 1M checks for FlowFiles, non-empty queue: 240 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 166 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 167 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 235 millis > 1M checks for FlowFiles, non-empty queue: 236 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 510 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 510 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 225 millis > 1M checks for FlowFiles, non-empty queue: 227 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 3 seconds, 994 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 3 seconds, 995 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 239 millis > 1M checks for FlowFiles, non-empty queue: 239 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 102 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 102 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 229 millis > 1M checks for FlowFiles, non-empty queue: 231 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 634 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 634 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 234 millis > 1M checks for FlowFiles, non-empty queue: 235 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 3 seconds, 732 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 3 seconds, 732 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 231 millis > 1M checks for FlowFiles, non-empty queue: 233 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 252 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 253 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 231 millis > 1M checks for FlowFiles, non-empty queue: 231 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 195 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 195 millis > -------------------- > 1M checks for FlowFiles, non-empty queue: 235 millis > 1M checks for FlowFiles, non-empty queue: 237 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 89 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 4 seconds, 89 millis > -------------------- > However, when we update the test to use 12 threads, we see some results that are more concerning: > 1M checks for FlowFiles, non-empty queue: 4 seconds, 430 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 463 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 486 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 331 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 316 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 484 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 517 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 357 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 357 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 349 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 478 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 372 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 435 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 31 seconds, 434 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 142 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 422 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 444 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 513 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 520 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 567 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 572 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 579 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 587 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 586 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 582 millis > 1M checks for FlowFiles, non-empty queue: 4 seconds, 594 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 21 seconds, 787 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 755 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 772 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 777 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 780 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 794 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 804 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 813 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 819 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 819 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 807 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 810 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 802 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 8 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 9 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 16 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 16 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 15 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 10 seconds, 16 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 469 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 481 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 479 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 521 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 544 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 553 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 575 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 566 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 581 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 582 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 583 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 572 millis > -------------------- > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 926 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 960 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-penalized method, non-empty queue: 1 minutes, 19 seconds, 965 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 724 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 745 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 761 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 764 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 776 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 774 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 800 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 800 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 802 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 803 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 805 millis > 1M checks for FlowFiles, non-empty queue: 2 seconds, 795 millis > So with 12 threads, each checking 1 million times, the difference in performance/latency goes from 2-3 seconds to 70-90 seconds. We can also see what happens to the CPU utilization as it switches between the version that checks penalization and the one that doesn't:  So at one thread, I'm not concerned about the performance difference. But if you start adding several threads to a processor, the performance concern is definitely more significant and I think is concerning.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
