That's what I thought at first, but he means non-human readable formats since we all use tools to parse logs as it is (Splunk and ELK are the big two I know of).
On 2 March 2016 at 02:15, Remko Popma <[email protected]> wrote: > Re: binary logging, I think he's talking about providing an API to log > objects directly into byte buffers without turning them into Strings first. > > https://issues.apache.org/jira/browse/LOG4J2-1274 and > https://issues.apache.org/jira/browse/LOG4J2-506 > > were created with that in mind and should be a good step in that > direction. > > Sent from my iPhone > > On 2016/03/02, at 15:11, Gary Gregory <[email protected]> wrote: > > Well, I've often wondered about creating a binary format but it seems that > you could use JSON+ZIP or BSON and get most of the advantages. > > Gary > > On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker <[email protected]> wrote: > >> One other interesting thing I learned is that improper use of logging is >> a huge source of performance problems. The GC-free parameterized message >> factory will help with one aspect of that (I suggested parameterized >> messages, but he countered with the Object[] that is created), and >> encouraging users to use a Supplier<String> instead of passing parameters >> should help as well (especially when those parameters have to be computed). >> He had some strong criticisms of logging APIs promoting bad practices which >> stems all the way back to log4j1 and affects pretty much every logging API >> in Java (some criticisms were actually outdated or didn't consider newer >> features of the API like markers and the huge amount of filters available). >> >> His other big idea was promoting the use of binary logging formats >> because humans rarely read the raw log files as it is, but it's not like >> there's a standard way to do that. >> >> Now I kinda wonder if he'll find this thread one day and tell me how I >> misinterpreted him or something. ;) >> >> On 1 March 2016 at 22:28, Matt Sicker <[email protected]> wrote: >> >>> Alright, I learned some interesting things. I'm going to get us some >>> tools we can use to try and profile this. Otherwise, he did suggest trying >>> out this project: >>> https://github.com/RichardWarburton/honest-profiler >>> >>> On 1 March 2016 at 19:31, Matt Sicker <[email protected]> wrote: >>> >>>> So far he's said something about using lambdas for lazy evaluation >>>> (though I don't think that would actually help us at all). I'll try to talk >>>> to him one-on-one afterward to delve more into this. >>>> >>>> On 1 March 2016 at 18:13, Ralph Goers <[email protected]> >>>> wrote: >>>> >>>>> Actually, most of the tests have the commands in the comments right in >>>>> the class. Just cut and past. >>>>> >>>>> Ralph >>>>> >>>>> On Mar 1, 2016, at 1:43 PM, Matt Sicker <[email protected]> wrote: >>>>> >>>>> I can't even figure out how to execute the simple perf test class. >>>>> IntelliJ gives me some annotation processing error, and doing it from the >>>>> command line is turning into a classpath nightmare to figure out what jars >>>>> are needed to execute the test manually. >>>>> >>>>> On 1 March 2016 at 11:34, Gary Gregory <[email protected]> wrote: >>>>> >>>>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you >>>>>> available after the preso to talk about some issue we are seeing? >>>>>> >>>>>> Gary >>>>>> On Mar 1, 2016 8:29 AM, "Matt Sicker" <[email protected]> wrote: >>>>>> >>>>>>> I'm attending a JUG meetup tonight with Kirk Pepperdine presenting. >>>>>>> It's supposed to be a Java performance workshop type of thing, so if >>>>>>> you've >>>>>>> got a decent way to ask about it, I could see if he can help figure out >>>>>>> this regression. I can at least show off the SimplePerfTest and any >>>>>>> microbenchmarks we have. >>>>>>> >>>>>>> On 28 February 2016 at 11:54, Matt Sicker <[email protected]> wrote: >>>>>>> >>>>>>>> Take a look at the git bisect command. Might help you find which >>>>>>>> changes caused the problem. >>>>>>>> >>>>>>>> >>>>>>>> On Sunday, 28 February 2016, Gary Gregory <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thank you for digging in Remko. This is will be a nice theme to >>>>>>>>> publicize when you get it figured out. >>>>>>>>> >>>>>>>>> Gary >>>>>>>>> On Feb 28, 2016 4:08 AM, "Remko Popma" <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> After removing the potential impact of appenders and layouts by >>>>>>>>>> testing with >>>>>>>>>> log4j-core\src\test\resources\perf-CountingNoOpAppender.xml >>>>>>>>>> and org.apache.logging.log4j.core.async.perftest.SimplePerfTest, I've >>>>>>>>>> confirmed my initial numbers: >>>>>>>>>> >>>>>>>>>> 2.0: 7.5M ops/sec >>>>>>>>>> 2.1: 6M ops/sec >>>>>>>>>> 2.2: 6M ops/sec >>>>>>>>>> 2.3: 6M ops/sec >>>>>>>>>> 2.4: 4.5M ops/sec >>>>>>>>>> 2.5: 4M ops/sec >>>>>>>>>> 2.6: 2M ops/sec >>>>>>>>>> >>>>>>>>>> I tried reverting various changes made to AsyncLogger since 2.0, >>>>>>>>>> performance improves a little up to 4M ops/sec. >>>>>>>>>> However, when completely reverting AsyncLogger source to the 2.0 >>>>>>>>>> version, performance is back to 7.5M ops/sec. >>>>>>>>>> >>>>>>>>>> I'll try starting from the 2.0 source and getting back to 2.6 >>>>>>>>>> functionality without losing performance... >>>>>>>>>> (Lengthy process...) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Feb 27, 2016 at 12:18 PM, Remko Popma < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> This is the PerfTestDriver test class (in log4j-core/test, >>>>>>>>>>> package ...async.perf). >>>>>>>>>>> Mainly perf3PlainNoLocation.xml: >>>>>>>>>>> RollingRandomAccessFileAppender, PatternLayout, all loggers are >>>>>>>>>>> AsyncLoggers, logging a simple string without parameters. >>>>>>>>>>> >>>>>>>>>>> Profiling with YourKit did not tell me anything useful. >>>>>>>>>>> >>>>>>>>>>> I'm now eliminating the effect of Layouts/Appenders, using >>>>>>>>>>> CountingNoOpAppender, and seeing similar numbers. So this seems to >>>>>>>>>>> be >>>>>>>>>>> mostly an issue in AsyncLogger. >>>>>>>>>>> >>>>>>>>>>> I'll let you know when I find out more. >>>>>>>>>>> There's a lot of trial and error here, so this may take a >>>>>>>>>>> while... >>>>>>>>>>> >>>>>>>>>>> Remko >>>>>>>>>>> >>>>>>>>>>> Sent from my iPhone >>>>>>>>>>> >>>>>>>>>>> On 2016/02/26, at 21:02, Mikael Ståldal < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> Which components (appenders, layouts) are involved in the tests? >>>>>>>>>>> Would it be possible to do some profiling to see if there is any >>>>>>>>>>> particular >>>>>>>>>>> component which is to blame? >>>>>>>>>>> >>>>>>>>>>> On Fri, Feb 26, 2016 at 12:51 PM, Remko Popma < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> To give you some rough impression on concrete numbers for this >>>>>>>>>>>> trend: >>>>>>>>>>>> 2.0: ~6M ops/sec >>>>>>>>>>>> 2.1-2.2: ~5M ops/sec >>>>>>>>>>>> 2.3-2.4: ~3-4M ops/sec >>>>>>>>>>>> 2.5: ~3M ops/sec >>>>>>>>>>>> 2.6: ~2M ops/sec >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Friday, 26 February 2016, Remko Popma <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> You're absolutely right. I still have quite a few unit tests >>>>>>>>>>>>> to add. >>>>>>>>>>>>> >>>>>>>>>>>>> Initial perf testing shows a downward trend in Async Logger >>>>>>>>>>>>> performance with every release. (Logging simple string >>>>>>>>>>>>> messages without params.) This is worrisome and I'm focusing on >>>>>>>>>>>>> figuring >>>>>>>>>>>>> that out first: this will likely involve additional code changes >>>>>>>>>>>>> and I'll >>>>>>>>>>>>> add more tests after that. >>>>>>>>>>>>> >>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>> >>>>>>>>>>>>> On 2016/02/26, at 10:38, Gary Gregory <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Wow, I love the activity we are seeing toward 2.6! All the >>>>>>>>>>>>> perf work on top of an existing sizable change set. Very exciting >>>>>>>>>>>>> indeed. >>>>>>>>>>>>> >>>>>>>>>>>>> There sure are a lot of changes coming in. I hope that we all >>>>>>>>>>>>> can pitch in to make sure most if not all of these changes get >>>>>>>>>>>>> code >>>>>>>>>>>>> coverage from unit tests. I've not checked closely, but it seems >>>>>>>>>>>>> like we >>>>>>>>>>>>> may not have good coverage _yet_, or do I have the wrong >>>>>>>>>>>>> impression? >>>>>>>>>>>>> >>>>>>>>>>>>> I want to make sure we keep our stability in tip top shape :-) >>>>>>>>>>>>> and that we have no regression from previous releases. >>>>>>>>>>>>> >>>>>>>>>>>>> Gary >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> E-Mail: [email protected] | [email protected] >>>>>>>>>>>>> Java Persistence with Hibernate, Second Edition >>>>>>>>>>>>> <http://www.manning.com/bauer3/> >>>>>>>>>>>>> JUnit in Action, Second Edition >>>>>>>>>>>>> <http://www.manning.com/tahchiev/> >>>>>>>>>>>>> Spring Batch in Action <http://www.manning.com/templier/> >>>>>>>>>>>>> Blog: http://garygregory.wordpress.com >>>>>>>>>>>>> Home: http://garygregory.com/ >>>>>>>>>>>>> Tweet! http://twitter.com/GaryGregory >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> [image: MagineTV] >>>>>>>>>>> >>>>>>>>>>> *Mikael Ståldal* >>>>>>>>>>> Senior software developer >>>>>>>>>>> >>>>>>>>>>> *Magine TV* >>>>>>>>>>> [email protected] >>>>>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>>>>>> >>>>>>>>>>> Privileged and/or Confidential Information may be contained in >>>>>>>>>>> this message. If you are not the addressee indicated in this message >>>>>>>>>>> (or responsible for delivery of the message to such a person), >>>>>>>>>>> you may not copy or deliver this message to anyone. In such case, >>>>>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>>>>> reply email. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Matt Sicker <[email protected]> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Matt Sicker <[email protected]> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Matt Sicker <[email protected]> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Matt Sicker <[email protected]> >>>> >>> >>> >>> >>> -- >>> Matt Sicker <[email protected]> >>> >> >> >> >> -- >> Matt Sicker <[email protected]> >> > > > > -- > E-Mail: [email protected] | [email protected] > Java Persistence with Hibernate, Second Edition > <http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory > > -- Matt Sicker <[email protected]>
