Do we want a general event header and footer then? Gary
On Jul 17, 2017 14:43, "Ralph Goers" <ralph.go...@dslextreme.com> wrote: > No. A Footer is only used at end of file. He needs to know how long each > event is or when it is the start of a new event. > > Ralph > > > On Jul 17, 2017, at 12:32 PM, Gary Gregory <garydgreg...@gmail.com> > wrote: > > > > Can't you use a footer for any terminator you wish? > > > > Gary > > > > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <mi...@apache.org> > wrote: > > > >> Hi. > >> > >> (Moving this discussion to logging dev mailing list.) > >> > >> Have you tried to use: > >> <JsonLayout properties="true" eventEol="true" compact="true"/> > >> > >> Then each log event will be terminated by end-of-line (\r\n). > >> > >> I think it would be easy to implement 0-byte terminated log events in > >> JsonLayout, and that would make sense since we have implemented support > for > >> that in GelfLayout. Created a JIRA issue for it: > >> https://issues.apache.org/jira/browse/LOG4J2-1981 > >> > >> As for other tools, the only receivers for Log4j 2 SerializedLayout we > >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only > >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support > >> JsonLayout and XmlLayout as well, and we are changing it to use > JsonLayout > >> by default. > >> > >> > >> On 2017-07-17 15:01, Joern Huxhorn wrote: > >> > >>> Hi Mikael, > >>> > >>> I've just taken a look at the JsonLayout and see no way to parse its > >>> output event by event. > >>> > >>> It's semi-fine for files because then it will result in a well-formed > >>> array at the root level that can be read in one big swoop. It isn't > >>> really ideal for that use case either since it means that the whole > >>> array containing all events needs to be read into memory in one piece, > >>> causing OOM in case of huge files. > >>> > >>> But in Lilith, I really need to process events one by one. This is > >>> absolutely crucial. > >>> > >>> The only way to achieve this would essentially require me to > >>> re-implement a JSON parser that can cope with data like this. > >>> > >>> I implemented a similar kludge to support reading of log4j xml files > >>> (see > >>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/ > >>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java > >>> ) but this was way easier since i could simply search for > >>> "</log4j:event>" to find the position at which I could split the stream > >>> of events into a valid XML document for every single event . This was > >>> still kind of nasty but doable, especially since processing an > "offline" > >>> XML file doesn't have the same performance restriction as handling of > >>> live events. > >>> > >>> I wouldn't have the luxury of such a unique split signal in JSON, > though. > >>> > >>> I would need to detect the "}" at the level of the root array which > >>> would not simply involve counting of opening and closing brackets but > >>> also ignoring of brackets inside of strings. This would boil down to > >>> implementing a custom JSON reader and I won't be doing this. > >>> > >>> This isn't just me being lazy either. Such an implementation wouldn't > be > >>> resource (CPU/memory) friendly anyway. > >>> > >>> In my own JSON receiver using my own JSON format there are two ways to > >>> send those events: > >>> - write an int containing the amount of bytes representing the event, > >>> then the bytes of the event. This type of events also supports > >>> compression. > >>> - write the bytes of the event, followed by a 0-byte. This works fine > >>> and JavaScript/ActionScript is able to produce events like that while > >>> they are unable (or were unable 7 years ago) to count the bytes on > their > >>> own. This type of events only supports plain text like JSON or XML and > >>> no compression since compressed events could contain the 0-byte while > >>> XML and JSON both won't allow it. > >>> > >>> Both are essentially "binary" formats because of either the raw ints or > >>> the 0-bytes. > >>> > >>> I'm not sure why you deprecate SerializedLayout since the security > >>> issues arise only while deserializing the events, not while serializing > >>> them. Is this just meant to educate the user about the issue? > >>> > >>> I fixed the remote code execution exploit by implementing > >>> https://github.com/huxi/lilith/blob/master/lilith-engine/ > >>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc > >>> er/WhitelistObjectInputStream.java > >>> > >>> This still doesn't fix DoS scenarios (endless loops or OOM in case of > >>> malicious data) but is close enough for me, mainly because fixing the > >>> DoS issues would really have to be handled by Java, not by user code. > >>> Manually re-implementing deserialization would likely be fragile and > >>> error prone, possibly even introducing other security issues on its > own. > >>> > >>> I'd be interested how other tools are tackling this issue. Is Chainsaw > >>> officially dead or are they implementing receivers for event streams > >>> like this? > >>> > >>> I'm totally fine with implementing a new receiver as a replacement for > >>> log4j2 SerializedLayout (even though I won't be able to get rid of the > >>> Serializable receiver anyway) but the current JsonLayout just isn't up > >>> to the job at the moment. > >>> > >>> Feel free to share this mail with your fellow developers. I'm open for > >>> suggestions. > >>> > >>> I hope this doesn't come across in a harsh way. It isn't meant to be. > >>> > >>> Cheers, > >>> Jörn. > >>> > >> > >> > > >
