Oh neat, I didn't know that. On 12 January 2016 at 13:41, Gary Gregory <[email protected]> wrote:
> BSON can be used anywhere, it's a partially compressed JSON format. > > Right now, I'm using CSV for large local logs I can later process or > import in a database. > > To save space, I'd like a format that is compressed but can also be > processed as is, BSON might help. > > Yes, I know I can rollover and compress ;-) > > Gary > > On Tue, Jan 12, 2016 at 11:26 AM, Matt Sicker <[email protected]> wrote: > >> Isn't BSON only used by Mongo? >> >> On 12 January 2016 at 12:51, Gary Gregory <[email protected]> wrote: >> >>> And I'm wondering if we should have a BSON layout, that would also be >>> with Jackson. >>> >>> Gayr >>> >>> On Tue, Jan 12, 2016 at 10:51 AM, Gary Gregory <[email protected]> >>> wrote: >>> >>>> Keep in mind that we use Jackson so almost all the code is the same for >>>> JSON and XML. >>>> >>>> Gary >>>> >>>> On Tue, Jan 12, 2016 at 8:34 AM, Robin Coe <[email protected]> >>>> wrote: >>>> >>>>> I would be happy to test but haven't looked at the writer >>>>> implementations, so speedy would not be the effort. But, it's a >>>>> performance issue only...maybe...so no hurry anyway. >>>>> >>>>> On Tue, Jan 12, 2016 at 11:30 AM, Gary Gregory <[email protected] >>>>> > wrote: >>>>> >>>>>> If the reason to do this is performance, it should be proved by a >>>>>> benchmark... >>>>>> >>>>>> Gary >>>>>> On Jan 12, 2016 8:28 AM, "Gary Gregory" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Maybe something like mdcStyle="this/that/future"? >>>>>>> >>>>>>> This would also apply to the XML layout? >>>>>>> >>>>>>> Gary >>>>>>> On Jan 12, 2016 8:26 AM, "Mikael Ståldal" <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Ah yes, that's possible. It would be nice. >>>>>>>> >>>>>>>> On Tue, Jan 12, 2016 at 5:24 PM, Ralph Goers < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> If there is a flag that causes the new structure to be generated >>>>>>>>> then he would get the performance gain when it is enabled. The current >>>>>>>>> structure would be generated when the flag is not set. >>>>>>>>> >>>>>>>>> Ralph >>>>>>>>> >>>>>>>>> On Jan 12, 2016, at 9:14 AM, Mikael Ståldal < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>> But I guess that you won't get any performance gain if we keep the >>>>>>>>> old structure besides the new one, since then both will be parsed. >>>>>>>>> >>>>>>>>> On Tue, Jan 12, 2016 at 3:15 PM, Robin Coe <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> I agree that if it were changed there may be some compatibility >>>>>>>>>> issues. But, if it's doable, then introducing a new property could >>>>>>>>>> bridge >>>>>>>>>> the change. Not saying it's doable, because I haven't looked, but a >>>>>>>>>> new >>>>>>>>>> property and a deprecation warning (in docs, I expect) would allow >>>>>>>>>> the >>>>>>>>>> change to happen. Very preliminary data showed me that parsing 1000 >>>>>>>>>> events >>>>>>>>>> slowed my parser from < 500 ms (w/o contextMap) to 2000 ms when each >>>>>>>>>> event >>>>>>>>>> contained 2 contextMap entries, requiring the list of maps to be >>>>>>>>>> converted >>>>>>>>>> to a single map. Not sure what the time would be to parse a >>>>>>>>>> multi-valued >>>>>>>>>> map, though, so I can't be sure of the overhead of walking the list >>>>>>>>>> wrapper. >>>>>>>>>> >>>>>>>>>> On Tue, Jan 12, 2016 at 6:05 AM, Mikael Ståldal < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I think that the current JSONLayout format is unfortunate, and I >>>>>>>>>>> would prefer to have it as you propose. But we cannot change it now >>>>>>>>>>> since >>>>>>>>>>> that will break backwards compatibility. >>>>>>>>>>> >>>>>>>>>>> See: https://issues.apache.org/jira/browse/LOG4J2-623 >>>>>>>>>>> >>>>>>>>>>> Perhaps GELFLayout would work better for you. >>>>>>>>>>> >>>>>>>>>>> On Mon, Jan 4, 2016 at 10:00 PM, Gary Gregory < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> The point I was trying to make is that you cannot describe what >>>>>>>>>>>> you are asking for with a generic XML schema, not sure about JSON >>>>>>>>>>>> schema, >>>>>>>>>>>> but the idea is the same. Since we use Jackson, that also means we >>>>>>>>>>>> use the >>>>>>>>>>>> same code to emit JSON and XML. >>>>>>>>>>>> >>>>>>>>>>>> Gary >>>>>>>>>>>> On Jan 4, 2016 12:25 PM, "Robin Coe" <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I can see that XML entities requires conforming to a schema >>>>>>>>>>>>> but isn't the writer implementation capable of wrapping the map >>>>>>>>>>>>> entries >>>>>>>>>>>>> when required? Seems like it's making the JSON representation >>>>>>>>>>>>> more complex >>>>>>>>>>>>> (and less performant) at the cost of some wrapper code for the >>>>>>>>>>>>> xml writer. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jan 4, 2016 at 3:19 PM, Gary Gregory < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, that is because we can define this kind of structure >>>>>>>>>>>>>> with XML/JSON schema with ease. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Gary >>>>>>>>>>>>>> On Jan 4, 2016 11:55 AM, "Robin Coe" <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I was trying to deserialize a log event written by the >>>>>>>>>>>>>>> JSONLayout appender, which uses Jackson. I therefore also am >>>>>>>>>>>>>>> using Jackson >>>>>>>>>>>>>>> but with the MrBeanModule, which is a POJO materializer. After >>>>>>>>>>>>>>> much >>>>>>>>>>>>>>> difficulty with Jackson throwing deserialization exceptions >>>>>>>>>>>>>>> with the >>>>>>>>>>>>>>> "contextMap" field, I learned that the map is actually written >>>>>>>>>>>>>>> out as a >>>>>>>>>>>>>>> List of Maps (i.e. List<Map<String,String>>. I've included one >>>>>>>>>>>>>>> such event >>>>>>>>>>>>>>> here, with unnecessary fields shortened: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> {"timeMillis":...,"thread":"...","level":"OFF","loggerName":"...","message":"...","endOfBatch":false,"loggerFqcn":"...","contextMap":[{"key":"LOGROLL","value":"com.xxx.xxx.handler.event.FailoverHandler"},{"key":"ROUTINGKEY","value":"elasticsearch-rollover"}]} >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm curious why the contextMap is represented as the more >>>>>>>>>>>>>>> complex List of single entry Maps, as opposed to a single >>>>>>>>>>>>>>> multi-valued >>>>>>>>>>>>>>> Map? So, instead of something that looks like: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> {"contextMap":[{"key":"key1"},{"value":"value1"},{"key":"key2"},{"value":"value2"},...] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would expect the much simpler (and easily parseable): >>>>>>>>>>>>>>> {"contextMap":{"key1":"value1","key2":"value2",...}. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is this intended? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Robin. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> [image: MagineTV] >>>>>>>>>>> >>>>>>>>>>> *Mikael Ståldal* >>>>>>>>>>> Senior software developer >>>>>>>>>>> >>>>>>>>>>> *Magine TV* >>>>>>>>>>> [email protected] >>>>>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>>>>>> >>>>>>>>>>> Privileged and/or Confidential Information may be contained in >>>>>>>>>>> this message. If you are not the addressee indicated in this message >>>>>>>>>>> (or responsible for delivery of the message to such a person), >>>>>>>>>>> you may not copy or deliver this message to anyone. In such case, >>>>>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>>>>> reply email. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> [image: MagineTV] >>>>>>>>> >>>>>>>>> *Mikael Ståldal* >>>>>>>>> Senior software developer >>>>>>>>> >>>>>>>>> *Magine TV* >>>>>>>>> [email protected] >>>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>>>> >>>>>>>>> Privileged and/or Confidential Information may be contained in >>>>>>>>> this message. If you are not the addressee indicated in this message >>>>>>>>> (or responsible for delivery of the message to such a person), you >>>>>>>>> may not copy or deliver this message to anyone. In such case, >>>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>>> reply email. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> [image: MagineTV] >>>>>>>> >>>>>>>> *Mikael Ståldal* >>>>>>>> Senior software developer >>>>>>>> >>>>>>>> *Magine TV* >>>>>>>> [email protected] >>>>>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com >>>>>>>> >>>>>>>> Privileged and/or Confidential Information may be contained in this >>>>>>>> message. If you are not the addressee indicated in this message >>>>>>>> (or responsible for delivery of the message to such a person), you >>>>>>>> may not copy or deliver this message to anyone. In such case, >>>>>>>> you should destroy this message and kindly notify the sender by >>>>>>>> reply email. >>>>>>>> >>>>>>> >>>>> >>>> >>>> >>>> -- >>>> E-Mail: [email protected] | [email protected] >>>> Java Persistence with Hibernate, Second Edition >>>> <http://www.manning.com/bauer3/> >>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >>>> Spring Batch in Action <http://www.manning.com/templier/> >>>> Blog: http://garygregory.wordpress.com >>>> Home: http://garygregory.com/ >>>> Tweet! http://twitter.com/GaryGregory >>>> >>> >>> >>> >>> -- >>> E-Mail: [email protected] | [email protected] >>> Java Persistence with Hibernate, Second Edition >>> <http://www.manning.com/bauer3/> >>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >>> Spring Batch in Action <http://www.manning.com/templier/> >>> Blog: http://garygregory.wordpress.com >>> Home: http://garygregory.com/ >>> Tweet! http://twitter.com/GaryGregory >>> >> >> >> >> -- >> Matt Sicker <[email protected]> >> > > > > -- > E-Mail: [email protected] | [email protected] > Java Persistence with Hibernate, Second Edition > <http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory > -- Matt Sicker <[email protected]>
