That's what I meant, I didn't make myself clear. For example, we could offer a simple binary layout: time stamp (8 bytes) level int (4 bytes) thread ID (4 bytes) - Thread names in separate file Logger ID (4 bytes) - Logger names in separate file. message length (4 bytes) message type (2 bytes) message data (variable length) throwable length (4 bytes) throwable data (variable length)
It's a very different approach to logging. On the plus side, this would be extremely compact and very fast to write. On the other hand it would require a separate tool to decode/display the data in human readable form. Such a tool should handle text messages out of the box, but for custom messages I image there could be some plugin mechanism for custom decoders. All very interesting... :-) Sent from my iPhone > On 2016/03/03, at 0:01, Matt Sicker <[email protected]> wrote: > > That's what I thought at first, but he means non-human readable formats since > we all use tools to parse logs as it is (Splunk and ELK are the big two I > know of). > >> On 2 March 2016 at 02:15, Remko Popma <[email protected]> wrote: >> Re: binary logging, I think he's talking about providing an API to log >> objects directly into byte buffers without turning them into Strings first. >> >> https://issues.apache.org/jira/browse/LOG4J2-1274 and >> https://issues.apache.org/jira/browse/LOG4J2-506 >> >> were created with that in mind and should be a good step in that direction. >> >> Sent from my iPhone >> >>> On 2016/03/02, at 15:11, Gary Gregory <[email protected]> wrote: >>> >>> Well, I've often wondered about creating a binary format but it seems that >>> you could use JSON+ZIP or BSON and get most of the advantages. >>> >>> Gary >>> >>>> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker <[email protected]> wrote: >>>> One other interesting thing I learned is that improper use of logging is a >>>> huge source of performance problems. The GC-free parameterized message >>>> factory will help with one aspect of that (I suggested parameterized >>>> messages, but he countered with the Object[] that is created), and >>>> encouraging users to use a Supplier<String> instead of passing parameters >>>> should help as well (especially when those parameters have to be >>>> computed). He had some strong criticisms of logging APIs promoting bad >>>> practices which stems all the way back to log4j1 and affects pretty much >>>> every logging API in Java (some criticisms were actually outdated or >>>> didn't consider newer features of the API like markers and the huge amount >>>> of filters available). >>>> >>>> His other big idea was promoting the use of binary logging formats because >>>> humans rarely read the raw log files as it is, but it's not like there's a >>>> standard way to do that. >>>> >>>> Now I kinda wonder if he'll find this thread one day and tell me how I >>>> misinterpreted him or something. ;) >>>> >>>>> On 1 March 2016 at 22:28, Matt Sicker <[email protected]> wrote: >>>>> Alright, I learned some interesting things. I'm going to get us some >>>>> tools we can use to try and profile this. Otherwise, he did suggest >>>>> trying out this project: >>>>> https://github.com/RichardWarburton/honest-profiler >>>>> >>>>>> On 1 March 2016 at 19:31, Matt Sicker <[email protected]> wrote: >>>>>> So far he's said something about using lambdas for lazy evaluation >>>>>> (though I don't think that would actually help us at all). I'll try to >>>>>> talk to him one-on-one afterward to delve more into this. >>>>>> >>>>>>> On 1 March 2016 at 18:13, Ralph Goers <[email protected]> >>>>>>> wrote: >>>>>>> Actually, most of the tests have the commands in the comments right in >>>>>>> the class. Just cut and past. >>>>>>> >>>>>>> Ralph >>>>>>> >>>>>>>> On Mar 1, 2016, at 1:43 PM, Matt Sicker <[email protected]> wrote: >>>>>>>> >>>>>>>> I can't even figure out how to execute the simple perf test class. >>>>>>>> IntelliJ gives me some annotation processing error, and doing it from >>>>>>>> the command line is turning into a classpath nightmare to figure out >>>>>>>> what jars are needed to execute the test manually. >>>>>>>> >>>>>>>> On 1 March 2016 at 11:34, Gary Gregory <[email protected]> wrote: >>>>>>>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you >>>>>>>>> available after the preso to talk about some issue we are seeing? >>>>>>>>> >>>>>>>>> Gary >>>>>>>>> >>>>>>>>>> On Mar 1, 2016 8:29 AM, "Matt Sicker" <[email protected]> wrote: >>>>>>>>>> I'm attending a JUG meetup tonight with Kirk Pepperdine presenting. >>>>>>>>>> It's supposed to be a Java performance workshop type of thing, so if >>>>>>>>>> you've got a decent way to ask about it, I could see if he can help >>>>>>>>>> figure out this regression. I can at least show off the >>>>>>>>>> SimplePerfTest and any microbenchmarks we have. >>>>>>>>>> >>>>>>>>>>> On 28 February 2016 at 11:54, Matt Sicker <[email protected]> wrote: >>>>>>>>>>> Take a look at the git bisect command. Might help you find which >>>>>>>>>>> changes caused the problem. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Sunday, 28 February 2016, Gary Gregory <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> Thank you for digging in Remko. This is will be a nice theme to >>>>>>>>>>>> publicize when you get it figured out. >>>>>>>>>>>> >>>>>>>>>>>> Gary >>>>>>>>>>>> >>>>>>>>>>>>> On Feb 28, 2016 4:08 AM, "Remko Popma" <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> After removing the potential impact of appenders and layouts by >>>>>>>>>>>>> testing with >>>>>>>>>>>>> log4j-core\src\test\resources\perf-CountingNoOpAppender.xml and >>>>>>>>>>>>> org.apache.logging.log4j.core.async.perftest.SimplePerfTest, I've >>>>>>>>>>>>> confirmed my initial numbers: >>>>>>>>>>>>> >>>>>>>>>>>>> 2.0: 7.5M ops/sec >>>>>>>>>>>>> 2.1: 6M ops/sec >>>>>>>>>>>>> 2.2: 6M ops/sec >>>>>>>>>>>>> 2.3: 6M ops/sec >>>>>>>>>>>>> 2.4: 4.5M ops/sec >>>>>>>>>>>>> 2.5: 4M ops/sec >>>>>>>>>>>>> 2.6: 2M ops/sec >>>>>>>>>>>>> >>>>>>>>>>>>> I tried reverting various changes made to AsyncLogger since 2.0, >>>>>>>>>>>>> performance improves a little up to 4M ops/sec. >>>>>>>>>>>>> However, when completely reverting AsyncLogger source to the 2.0 >>>>>>>>>>>>> version, performance is back to 7.5M ops/sec. >>>>>>>>>>>>> >>>>>>>>>>>>> I'll try starting from the 2.0 source and getting back to 2.6 >>>>>>>>>>>>> functionality without losing performance... >>>>>>>>>>>>> (Lengthy process...) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Feb 27, 2016 at 12:18 PM, Remko Popma >>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>> This is the PerfTestDriver test class (in log4j-core/test, >>>>>>>>>>>>>> package ...async.perf). >>>>>>>>>>>>>> Mainly perf3PlainNoLocation.xml: >>>>>>>>>>>>>> RollingRandomAccessFileAppender, PatternLayout, all loggers are >>>>>>>>>>>>>> AsyncLoggers, logging a simple string without parameters. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Profiling with YourKit did not tell me anything useful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm now eliminating the effect of Layouts/Appenders, using >>>>>>>>>>>>>> CountingNoOpAppender, and seeing similar numbers. So this seems >>>>>>>>>>>>>> to be mostly an issue in AsyncLogger. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'll let you know when I find out more. >>>>>>>>>>>>>> There's a lot of trial and error here, so this may take a >>>>>>>>>>>>>> while... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Remko >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2016/02/26, at 21:02, Mikael Ståldal >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Which components (appenders, layouts) are involved in the >>>>>>>>>>>>>>> tests? Would it be possible to do some profiling to see if >>>>>>>>>>>>>>> there is any particular component which is to blame? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Feb 26, 2016 at 12:51 PM, Remko Popma >>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>> To give you some rough impression on concrete numbers for this >>>>>>>>>>>>>>>> trend: >>>>>>>>>>>>>>>> 2.0: ~6M ops/sec >>>>>>>>>>>>>>>> 2.1-2.2: ~5M ops/sec >>>>>>>>>>>>>>>> 2.3-2.4: ~3-4M ops/sec >>>>>>>>>>>>>>>> 2.5: ~3M ops/sec >>>>>>>>>>>>>>>> 2.6: ~2M ops/sec >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Friday, 26 February 2016, Remko Popma >>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>> You're absolutely right. I still have quite a few unit tests >>>>>>>>>>>>>>>>> to add. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Initial perf testing shows a downward trend in Async Logger >>>>>>>>>>>>>>>>> performance with every release. (Logging simple string >>>>>>>>>>>>>>>>> messages without params.) This is worrisome and I'm focusing >>>>>>>>>>>>>>>>> on figuring that out first: this will likely involve >>>>>>>>>>>>>>>>> additional code changes and I'll add more tests after that. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2016/02/26, at 10:38, Gary Gregory >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Wow, I love the activity we are seeing toward 2.6! All the >>>>>>>>>>>>>>>>>> perf work on top of an existing sizable change set. Very >>>>>>>>>>>>>>>>>> exciting indeed. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> There sure are a lot of changes coming in. I hope that we >>>>>>>>>>>>>>>>>> all can pitch in to make sure most if not all of these >>>>>>>>>>>>>>>>>> changes get code coverage from unit tests. I've not checked >>>>>>>>>>>>>>>>>> closely, but it seems like we may not have good coverage >>>>>>>>>>>>>>>>>> _yet_, or do I have the wrong impression? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I want to make sure we keep our stability in tip top shape >>>>>>>>>>>>>>>>>> :-) and that we have no regression from previous releases. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Gary >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> E-Mail: [email protected] | [email protected] >>>>>>>>>>>>>>>>>> Java Persistence with Hibernate, Second Edition >>>>>>>>>>>>>>>>>> JUnit in Action, Second Edition >>>>>>>>>>>>>>>>>> Spring Batch in Action >>>>>>>>>>>>>>>>>> Blog: http://garygregory.wordpress.com >>>>>>>>>>>>>>>>>> Home: http://garygregory.com/ >>>>>>>>>>>>>>>>>> Tweet! http://twitter.com/GaryGregory >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mikael Ståldal >>>>>>>>>>>>>>> Senior software developer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Magine TV >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | >>>>>>>>>>>>>>> www.magine.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Privileged and/or Confidential Information may be contained in >>>>>>>>>>>>>>> this message. If you are not the addressee indicated in this >>>>>>>>>>>>>>> message >>>>>>>>>>>>>>> (or responsible for delivery of the message to such a person), >>>>>>>>>>>>>>> you may not copy or deliver this message to anyone. In such >>>>>>>>>>>>>>> case, >>>>>>>>>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>>>>>>>>> reply email. >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Matt Sicker <[email protected]> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Matt Sicker <[email protected]> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Matt Sicker <[email protected]> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Matt Sicker <[email protected]> >>>>> >>>>> >>>>> >>>>> -- >>>>> Matt Sicker <[email protected]> >>>> >>>> >>>> >>>> -- >>>> Matt Sicker <[email protected]> >>> >>> >>> >>> -- >>> E-Mail: [email protected] | [email protected] >>> Java Persistence with Hibernate, Second Edition >>> JUnit in Action, Second Edition >>> Spring Batch in Action >>> Blog: http://garygregory.wordpress.com >>> Home: http://garygregory.com/ >>> Tweet! http://twitter.com/GaryGregory > > > > -- > Matt Sicker <[email protected]>
