I think we'd need to provide all LogEvent fields. Gary
On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma <[email protected]> wrote: > That's what I meant, I didn't make myself clear. For example, we could > offer a simple binary layout: > time stamp (8 bytes) > level int (4 bytes) > thread ID (4 bytes) - Thread names in separate file > Logger ID (4 bytes) - Logger names in separate file. > message length (4 bytes) > message type (2 bytes) > message data (variable length) > throwable length (4 bytes) > throwable data (variable length) > > It's a very different approach to logging. On the plus side, this would be > extremely compact and very fast to write. > > On the other hand it would require a separate tool to decode/display the > data in human readable form. Such a tool should handle text messages out of > the box, but for custom messages I image there could be some plugin > mechanism for custom decoders. > > All very interesting... > :-) > > Sent from my iPhone > > On 2016/03/03, at 0:01, Matt Sicker <[email protected]> wrote: > > That's what I thought at first, but he means non-human readable formats > since we all use tools to parse logs as it is (Splunk and ELK are the big > two I know of). > > On 2 March 2016 at 02:15, Remko Popma <[email protected]> wrote: > >> Re: binary logging, I think he's talking about providing an API to log >> objects directly into byte buffers without turning them into Strings first. >> >> https://issues.apache.org/jira/browse/LOG4J2-1274 and >> https://issues.apache.org/jira/browse/LOG4J2-506 >> >> were created with that in mind and should be a good step in that >> direction. >> >> Sent from my iPhone >> >> On 2016/03/02, at 15:11, Gary Gregory <[email protected]> wrote: >> >> Well, I've often wondered about creating a binary format but it seems >> that you could use JSON+ZIP or BSON and get most of the advantages. >> >> Gary >> >> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker <[email protected]> wrote: >> >>> One other interesting thing I learned is that improper use of logging is >>> a huge source of performance problems. The GC-free parameterized message >>> factory will help with one aspect of that (I suggested parameterized >>> messages, but he countered with the Object[] that is created), and >>> encouraging users to use a Supplier<String> instead of passing parameters >>> should help as well (especially when those parameters have to be computed). >>> He had some strong criticisms of logging APIs promoting bad practices which >>> stems all the way back to log4j1 and affects pretty much every logging API >>> in Java (some criticisms were actually outdated or didn't consider newer >>> features of the API like markers and the huge amount of filters available). >>> >>> His other big idea was promoting the use of binary logging formats >>> because humans rarely read the raw log files as it is, but it's not like >>> there's a standard way to do that. >>> >>> Now I kinda wonder if he'll find this thread one day and tell me how I >>> misinterpreted him or something. ;) >>> >>> On 1 March 2016 at 22:28, Matt Sicker <[email protected]> wrote: >>> >>>> Alright, I learned some interesting things. I'm going to get us some >>>> tools we can use to try and profile this. Otherwise, he did suggest trying >>>> out this project: >>>> https://github.com/RichardWarburton/honest-profiler >>>> >>>> On 1 March 2016 at 19:31, Matt Sicker <[email protected]> wrote: >>>> >>>>> So far he's said something about using lambdas for lazy evaluation >>>>> (though I don't think that would actually help us at all). I'll try to >>>>> talk >>>>> to him one-on-one afterward to delve more into this. >>>>> >>>>> On 1 March 2016 at 18:13, Ralph Goers <[email protected]> >>>>> wrote: >>>>> >>>>>> Actually, most of the tests have the commands in the comments right >>>>>> in the class. Just cut and past. >>>>>> >>>>>> Ralph >>>>>> >>>>>> On Mar 1, 2016, at 1:43 PM, Matt Sicker <[email protected]> wrote: >>>>>> >>>>>> I can't even figure out how to execute the simple perf test class. >>>>>> IntelliJ gives me some annotation processing error, and doing it from the >>>>>> command line is turning into a classpath nightmare to figure out what >>>>>> jars >>>>>> are needed to execute the test manually. >>>>>> >>>>>> On 1 March 2016 at 11:34, Gary Gregory <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you >>>>>>> available after the preso to talk about some issue we are seeing? >>>>>>> >>>>>>> Gary >>>>>>> On Mar 1, 2016 8:29 AM, "Matt Sicker" <[email protected]> wrote: >>>>>>> >>>>>>>> I'm attending a JUG meetup tonight with Kirk Pepperdine presenting. >>>>>>>> It's supposed to be a Java performance workshop type of thing, so if >>>>>>>> you've >>>>>>>> got a decent way to ask about it, I could see if he can help figure out >>>>>>>> this regression. I can at least show off the SimplePerfTest and any >>>>>>>> microbenchmarks we have. >>>>>>>> >>>>>>>> On 28 February 2016 at 11:54, Matt Sicker <[email protected]> wrote: >>>>>>>> >>>>>>>>> Take a look at the git bisect command. Might help you find which >>>>>>>>> changes caused the problem. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sunday, 28 February 2016, Gary Gregory <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thank you for digging in Remko. This is will be a nice theme to >>>>>>>>>> publicize when you get it figured out. >>>>>>>>>> >>>>>>>>>> Gary >>>>>>>>>> On Feb 28, 2016 4:08 AM, "Remko Popma" <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> After removing the potential impact of appenders and layouts by >>>>>>>>>>> testing with >>>>>>>>>>> log4j-core\src\test\resources\perf-CountingNoOpAppender.xml >>>>>>>>>>> and org.apache.logging.log4j.core.async.perftest.SimplePerfTest, >>>>>>>>>>> I've >>>>>>>>>>> confirmed my initial numbers: >>>>>>>>>>> >>>>>>>>>>> 2.0: 7.5M ops/sec >>>>>>>>>>> 2.1: 6M ops/sec >>>>>>>>>>> 2.2: 6M ops/sec >>>>>>>>>>> 2.3: 6M ops/sec >>>>>>>>>>> 2.4: 4.5M ops/sec >>>>>>>>>>> 2.5: 4M ops/sec >>>>>>>>>>> 2.6: 2M ops/sec >>>>>>>>>>> >>>>>>>>>>> I tried reverting various changes made to AsyncLogger since 2.0, >>>>>>>>>>> performance improves a little up to 4M ops/sec. >>>>>>>>>>> However, when completely reverting AsyncLogger source to the 2.0 >>>>>>>>>>> version, performance is back to 7.5M ops/sec. >>>>>>>>>>> >>>>>>>>>>> I'll try starting from the 2.0 source and getting back to 2.6 >>>>>>>>>>> functionality without losing performance... >>>>>>>>>>> (Lengthy process...) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Feb 27, 2016 at 12:18 PM, Remko Popma < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> This is the PerfTestDriver test class (in log4j-core/test, >>>>>>>>>>>> package ...async.perf). >>>>>>>>>>>> Mainly perf3PlainNoLocation.xml: >>>>>>>>>>>> RollingRandomAccessFileAppender, PatternLayout, all loggers are >>>>>>>>>>>> AsyncLoggers, logging a simple string without parameters. >>>>>>>>>>>> >>>>>>>>>>>> Profiling with YourKit did not tell me anything useful. >>>>>>>>>>>> >>>>>>>>>>>> I'm now eliminating the effect of Layouts/Appenders, using >>>>>>>>>>>> CountingNoOpAppender, and seeing similar numbers. So this seems to >>>>>>>>>>>> be >>>>>>>>>>>> mostly an issue in AsyncLogger. >>>>>>>>>>>> >>>>>>>>>>>> I'll let you know when I find out more. >>>>>>>>>>>> There's a lot of trial and error here, so this may take a >>>>>>>>>>>> while... >>>>>>>>>>>> >>>>>>>>>>>> Remko >>>>>>>>>>>> >>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>> >>>>>>>>>>>> On 2016/02/26, at 21:02, Mikael Ståldal < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Which components (appenders, layouts) are involved in the >>>>>>>>>>>> tests? Would it be possible to do some profiling to see if there >>>>>>>>>>>> is any >>>>>>>>>>>> particular component which is to blame? >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Feb 26, 2016 at 12:51 PM, Remko Popma < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> To give you some rough impression on concrete numbers for this >>>>>>>>>>>>> trend: >>>>>>>>>>>>> 2.0: ~6M ops/sec >>>>>>>>>>>>> 2.1-2.2: ~5M ops/sec >>>>>>>>>>>>> 2.3-2.4: ~3-4M ops/sec >>>>>>>>>>>>> 2.5: ~3M ops/sec >>>>>>>>>>>>> 2.6: ~2M ops/sec >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Friday, 26 February 2016, Remko Popma < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> You're absolutely right. I still have quite a few unit tests >>>>>>>>>>>>>> to add. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Initial perf testing shows a downward trend in Async Logger >>>>>>>>>>>>>> performance with every release. (Logging simple string >>>>>>>>>>>>>> messages without params.) This is worrisome and I'm focusing on >>>>>>>>>>>>>> figuring >>>>>>>>>>>>>> that out first: this will likely involve additional code changes >>>>>>>>>>>>>> and I'll >>>>>>>>>>>>>> add more tests after that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2016/02/26, at 10:38, Gary Gregory <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Wow, I love the activity we are seeing toward 2.6! All the >>>>>>>>>>>>>> perf work on top of an existing sizable change set. Very >>>>>>>>>>>>>> exciting indeed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> There sure are a lot of changes coming in. I hope that we all >>>>>>>>>>>>>> can pitch in to make sure most if not all of these changes get >>>>>>>>>>>>>> code >>>>>>>>>>>>>> coverage from unit tests. I've not checked closely, but it seems >>>>>>>>>>>>>> like we >>>>>>>>>>>>>> may not have good coverage _yet_, or do I have the wrong >>>>>>>>>>>>>> impression? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I want to make sure we keep our stability in tip top shape >>>>>>>>>>>>>> :-) and that we have no regression from previous releases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Gary >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> E-Mail: [email protected] | [email protected] >>>>>>>>>>>>>> Java Persistence with Hibernate, Second Edition >>>>>>>>>>>>>> <http://www.manning.com/bauer3/> >>>>>>>>>>>>>> JUnit in Action, Second Edition >>>>>>>>>>>>>> <http://www.manning.com/tahchiev/> >>>>>>>>>>>>>> Spring Batch in Action <http://www.manning.com/templier/> >>>>>>>>>>>>>> Blog: http://garygregory.wordpress.com >>>>>>>>>>>>>> Home: http://garygregory.com/ >>>>>>>>>>>>>> Tweet! http://twitter.com/GaryGregory >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> [image: MagineTV] >>>>>>>>>>>> >>>>>>>>>>>> *Mikael Ståldal* >>>>>>>>>>>> Senior software developer >>>>>>>>>>>> >>>>>>>>>>>> *Magine TV* >>>>>>>>>>>> [email protected] >>>>>>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | >>>>>>>>>>>> www.magine.com >>>>>>>>>>>> >>>>>>>>>>>> Privileged and/or Confidential Information may be contained in >>>>>>>>>>>> this message. If you are not the addressee indicated in this >>>>>>>>>>>> message >>>>>>>>>>>> (or responsible for delivery of the message to such a person), >>>>>>>>>>>> you may not copy or deliver this message to anyone. In such case, >>>>>>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>>>>>> reply email. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Matt Sicker <[email protected]> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Matt Sicker <[email protected]> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Matt Sicker <[email protected]> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Matt Sicker <[email protected]> >>>>> >>>> >>>> >>>> >>>> -- >>>> Matt Sicker <[email protected]> >>>> >>> >>> >>> >>> -- >>> Matt Sicker <[email protected]> >>> >> >> >> >> -- >> E-Mail: [email protected] | [email protected] >> Java Persistence with Hibernate, Second Edition >> <http://www.manning.com/bauer3/> >> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >> Spring Batch in Action <http://www.manning.com/templier/> >> Blog: http://garygregory.wordpress.com >> Home: http://garygregory.com/ >> Tweet! http://twitter.com/GaryGregory >> >> > > > -- > Matt Sicker <[email protected]> > > -- E-Mail: [email protected] | [email protected] Java Persistence with Hibernate, Second Edition <http://www.manning.com/bauer3/> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> Spring Batch in Action <http://www.manning.com/templier/> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory
