No. A Footer is only used at end of file. He needs to know how long each event is or when it is the start of a new event.
Ralph > On Jul 17, 2017, at 12:32 PM, Gary Gregory <[email protected]> wrote: > > Can't you use a footer for any terminator you wish? > > Gary > > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <[email protected]> wrote: > >> Hi. >> >> (Moving this discussion to logging dev mailing list.) >> >> Have you tried to use: >> <JsonLayout properties="true" eventEol="true" compact="true"/> >> >> Then each log event will be terminated by end-of-line (\r\n). >> >> I think it would be easy to implement 0-byte terminated log events in >> JsonLayout, and that would make sense since we have implemented support for >> that in GelfLayout. Created a JIRA issue for it: >> https://issues.apache.org/jira/browse/LOG4J2-1981 >> >> As for other tools, the only receivers for Log4j 2 SerializedLayout we >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support >> JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout >> by default. >> >> >> On 2017-07-17 15:01, Joern Huxhorn wrote: >> >>> Hi Mikael, >>> >>> I've just taken a look at the JsonLayout and see no way to parse its >>> output event by event. >>> >>> It's semi-fine for files because then it will result in a well-formed >>> array at the root level that can be read in one big swoop. It isn't >>> really ideal for that use case either since it means that the whole >>> array containing all events needs to be read into memory in one piece, >>> causing OOM in case of huge files. >>> >>> But in Lilith, I really need to process events one by one. This is >>> absolutely crucial. >>> >>> The only way to achieve this would essentially require me to >>> re-implement a JSON parser that can cope with data like this. >>> >>> I implemented a similar kludge to support reading of log4j xml files >>> (see >>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ >>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java >>> ) but this was way easier since i could simply search for >>> "</log4j:event>" to find the position at which I could split the stream >>> of events into a valid XML document for every single event . This was >>> still kind of nasty but doable, especially since processing an "offline" >>> XML file doesn't have the same performance restriction as handling of >>> live events. >>> >>> I wouldn't have the luxury of such a unique split signal in JSON, though. >>> >>> I would need to detect the "}" at the level of the root array which >>> would not simply involve counting of opening and closing brackets but >>> also ignoring of brackets inside of strings. This would boil down to >>> implementing a custom JSON reader and I won't be doing this. >>> >>> This isn't just me being lazy either. Such an implementation wouldn't be >>> resource (CPU/memory) friendly anyway. >>> >>> In my own JSON receiver using my own JSON format there are two ways to >>> send those events: >>> - write an int containing the amount of bytes representing the event, >>> then the bytes of the event. This type of events also supports >>> compression. >>> - write the bytes of the event, followed by a 0-byte. This works fine >>> and JavaScript/ActionScript is able to produce events like that while >>> they are unable (or were unable 7 years ago) to count the bytes on their >>> own. This type of events only supports plain text like JSON or XML and >>> no compression since compressed events could contain the 0-byte while >>> XML and JSON both won't allow it. >>> >>> Both are essentially "binary" formats because of either the raw ints or >>> the 0-bytes. >>> >>> I'm not sure why you deprecate SerializedLayout since the security >>> issues arise only while deserializing the events, not while serializing >>> them. Is this just meant to educate the user about the issue? >>> >>> I fixed the remote code execution exploit by implementing >>> https://github.com/huxi/lilith/blob/master/lilith-engine/ >>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc >>> er/WhitelistObjectInputStream.java >>> >>> This still doesn't fix DoS scenarios (endless loops or OOM in case of >>> malicious data) but is close enough for me, mainly because fixing the >>> DoS issues would really have to be handled by Java, not by user code. >>> Manually re-implementing deserialization would likely be fragile and >>> error prone, possibly even introducing other security issues on its own. >>> >>> I'd be interested how other tools are tackling this issue. Is Chainsaw >>> officially dead or are they implementing receivers for event streams >>> like this? >>> >>> I'm totally fine with implementing a new receiver as a replacement for >>> log4j2 SerializedLayout (even though I won't be able to get rid of the >>> Serializable receiver anyway) but the current JsonLayout just isn't up >>> to the job at the moment. >>> >>> Feel free to share this mail with your fellow developers. I'm open for >>> suggestions. >>> >>> I hope this doesn't come across in a harsh way. It isn't meant to be. >>> >>> Cheers, >>> Jörn. >>> >> >>
