Did you add the no-op appender to Ceki’s project? Or are you using our NullAppender? I have my doubts about using that NullAppender as it does nothing - not even render the Layout, so it may get completely optimized away.
I’d still like to know what kind of disks Remko did his original testing. The page says he got 18.495 million messages per second. Each message would be about 130 characters long from I can tell. That means the destination has to be able to support about at least 2.4 GB per second. In Ceki’s test the message is only 23 characters long (vs 100 in Remko’s). That means the message size is only going to be about 60 characters long. For me the async test is writing at about 1700 ops/ms which equates to 102 MBs, yet I am still seeing the queue backed up. So how we are writing must be terribly inefficient. Logback is getting about 2000 ops/ms, which is still only 120 MBs. Your test below would be generating about 197 MBs. One thing I did notice that made a big difference is that Ceki has Logback configured to use a buffer size of 256K while Log4j2 uses the default of 8K. Setting Log4j2 to 256K considerably improved the performance. Ralph > On Aug 25, 2021, at 8:42 AM, Carter Kozak <[email protected]> wrote: > >> If we would move the appender performance aside, am I right to conclude >> that the entire complex async. framework built atop Disruptor with all its >> whistles and bells is yielding a diminishing performance gain compared to a >> simple off the shelf blocking queue? > > I haven't seen any data that would give me such an impression -- the file > appender > itself is slower than the thread producing events, which appears to limit our > throughput. > Even then, log4j2 does much better in my testing on a linux workstation with > 20+ producer > threads compared to logback. > Any time the queue is constantly full, performance will suffer (and this > applies to standard > blocking queues as well as disruptor, however disruptor will fare worse when > the queue > is full). > The results using a single thread are roughly reproducible on the non-async > test, so I think > the problem is somewhere between the layout, and appender/manager. > > Results running locally with no-op appender implementations: > > 1 thread: > > Benchmark Mode Cnt Score > Error Units > AsyncWithFileAppenderBenchmark.log4j2AsyncFile thrpt 4 3285.743 ± > 427.963 ops/ms > AsyncWithFileAppenderBenchmark.logbackFile thrpt 4 2383.532 ± > 136.034 ops/ms > > 20 threads: > > Benchmark Mode Cnt Score > Error Units > AsyncWithFileAppenderBenchmark.log4j2AsyncFile thrpt 4 1177.746 ± > 919.560 ops/ms > AsyncWithFileAppenderBenchmark.logbackFile thrpt 4 602.614 ± > 47.485 ops/ms
