Can't you use a footer for any terminator you wish? Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <[email protected]> wrote: > Hi. > > (Moving this discussion to logging dev mailing list.) > > Have you tried to use: > <JsonLayout properties="true" eventEol="true" compact="true"/> > > Then each log event will be terminated by end-of-line (\r\n). > > I think it would be easy to implement 0-byte terminated log events in > JsonLayout, and that would make sense since we have implemented support for > that in GelfLayout. Created a JIRA issue for it: > https://issues.apache.org/jira/browse/LOG4J2-1981 > > As for other tools, the only receivers for Log4j 2 SerializedLayout we > know are Log4j's own SocketServer and Lilith. Chainsaw currently only > support Log4j 1 SerializedLayout. Log4j's own SocketServer support > JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout > by default. > > > On 2017-07-17 15:01, Joern Huxhorn wrote: > >> Hi Mikael, >> >> I've just taken a look at the JsonLayout and see no way to parse its >> output event by event. >> >> It's semi-fine for files because then it will result in a well-formed >> array at the root level that can be read in one big swoop. It isn't >> really ideal for that use case either since it means that the whole >> array containing all events needs to be read into memory in one piece, >> causing OOM in case of huge files. >> >> But in Lilith, I really need to process events one by one. This is >> absolutely crucial. >> >> The only way to achieve this would essentially require me to >> re-implement a JSON parser that can cope with data like this. >> >> I implemented a similar kludge to support reading of log4j xml files >> (see >> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ >> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java >> ) but this was way easier since i could simply search for >> "</log4j:event>" to find the position at which I could split the stream >> of events into a valid XML document for every single event . This was >> still kind of nasty but doable, especially since processing an "offline" >> XML file doesn't have the same performance restriction as handling of >> live events. >> >> I wouldn't have the luxury of such a unique split signal in JSON, though. >> >> I would need to detect the "}" at the level of the root array which >> would not simply involve counting of opening and closing brackets but >> also ignoring of brackets inside of strings. This would boil down to >> implementing a custom JSON reader and I won't be doing this. >> >> This isn't just me being lazy either. Such an implementation wouldn't be >> resource (CPU/memory) friendly anyway. >> >> In my own JSON receiver using my own JSON format there are two ways to >> send those events: >> - write an int containing the amount of bytes representing the event, >> then the bytes of the event. This type of events also supports >> compression. >> - write the bytes of the event, followed by a 0-byte. This works fine >> and JavaScript/ActionScript is able to produce events like that while >> they are unable (or were unable 7 years ago) to count the bytes on their >> own. This type of events only supports plain text like JSON or XML and >> no compression since compressed events could contain the 0-byte while >> XML and JSON both won't allow it. >> >> Both are essentially "binary" formats because of either the raw ints or >> the 0-bytes. >> >> I'm not sure why you deprecate SerializedLayout since the security >> issues arise only while deserializing the events, not while serializing >> them. Is this just meant to educate the user about the issue? >> >> I fixed the remote code execution exploit by implementing >> https://github.com/huxi/lilith/blob/master/lilith-engine/ >> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc >> er/WhitelistObjectInputStream.java >> >> This still doesn't fix DoS scenarios (endless loops or OOM in case of >> malicious data) but is close enough for me, mainly because fixing the >> DoS issues would really have to be handled by Java, not by user code. >> Manually re-implementing deserialization would likely be fragile and >> error prone, possibly even introducing other security issues on its own. >> >> I'd be interested how other tools are tackling this issue. Is Chainsaw >> officially dead or are they implementing receivers for event streams >> like this? >> >> I'm totally fine with implementing a new receiver as a replacement for >> log4j2 SerializedLayout (even though I won't be able to get rid of the >> Serializable receiver anyway) but the current JsonLayout just isn't up >> to the job at the moment. >> >> Feel free to share this mail with your fellow developers. I'm open for >> suggestions. >> >> I hope this doesn't come across in a harsh way. It isn't meant to be. >> >> Cheers, >> Jörn. >> > >
