Gary, what do you say? I think you once added that comment.
On 2017-07-22 16:27, Jörn Huxhorn wrote:
Well… it says
/**
* A Jackson JSON {@link ObjectMapper} initialized for Log4j.
* <p>
* <em>Consider this class private.</em>
* </p>
*/
so I wanted to holler about how to continue.
I’ll give it a shot.
On 22. July 2017 at 15:06:28, Mikael Ståldal (mi...@apache.org) wrote:
Would it work to use
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
public?
See here how it is used:
https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java
On 2017-07-22 13:47, Jörn Huxhorn wrote:
It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module
used for
JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)
It would make sense to reuse your ObjectMapper/Module in my code instead of
reimplementing
them to prevent future compatibility issues down the road.
On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxh...@googlemail.com) wrote:
I’ll give this a shot over the next view days.
Thanks!
On 19. July 2017 at 21:33:49, Mikael Ståldal (mi...@apache.org) wrote:
JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
termination of log events. Will be part of the upcoming 2.9 release.
On 2017-07-18 11:48, Jörn Huxhorn wrote:
A general event header and footer would probably be overkill.
I’m using the following interface in Lilith:
public interface WriteByteStrategy {
void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
throws IOException;
}
One implementation writes an int with the length of bytes before “bytes", the
other
implementation writes a 0-byte after “bytes".
The problem with that approach is that it requires the intermediate “bytes”
array
and
is thus probably incompatible with your zero garbage approach.
I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this
with
you.
I would also have to implement a new receiver type whereas I could just reuse
an existing
one in case of 0-byte termination.
I’d prefer using 0-byte termination since it would be less fragile than
end-of-line.
The end-of-line approach would break if compact=“false” was set erroneously
while
you’d still have the option of using either compact=“true” or compact=“false” in
case
of 0-byte termination. 0-byte termination is also slightly more efficient while
reading
since the splitting can operate on bytes instead of chars or Strings - another
reason
why I didn’t just implement it and instead wanted to discuss this with you
first.
Joern
On 18. July 2017 at 03:26:32, Ralph Goers (ralph.go...@dslextreme.com) wrote:
Right now that is all handled by the specific layouts. For example, by default
the
RFC5424Layout
doesn’t append newlines so when writing to a file they will all be on the same
“line”,
but
it has an option to append one if needed. Doing the same would be another
option for
the
JSONLayout.
I’d be ok with event header and footers but ONLY if they have zero overhead
when not
present.
IOW, when the Layout is initialized it would have to wire in the appropriate
encode
method.
Ralph
On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
Do we want a general event header and footer then?
Gary
On Jul 17, 2017 14:43, "Ralph Goers" wrote:
No. A Footer is only used at end of file. He needs to know how long each
event is or when it is the start of a new event.
Ralph
On Jul 17, 2017, at 12:32 PM, Gary Gregory
wrote:
Can't you use a footer for any terminator you wish?
Gary
On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
wrote:
Hi.
(Moving this discussion to logging dev mailing list.)
Have you tried to use:
Then each log event will be terminated by end-of-line (\r\n).
I think it would be easy to implement 0-byte terminated log events in
JsonLayout, and that would make sense since we have implemented support
for
that in GelfLayout. Created a JIRA issue for it:
https://issues.apache.org/jira/browse/LOG4J2-1981
As for other tools, the only receivers for Log4j 2 SerializedLayout we
know are Log4j's own SocketServer and Lilith. Chainsaw currently only
support Log4j 1 SerializedLayout. Log4j's own SocketServer support
JsonLayout and XmlLayout as well, and we are changing it to use
JsonLayout
by default.
On 2017-07-17 15:01, Joern Huxhorn wrote:
Hi Mikael,
I've just taken a look at the JsonLayout and see no way to parse its
output event by event.
It's semi-fine for files because then it will result in a well-formed
array at the root level that can be read in one big swoop. It isn't
really ideal for that use case either since it means that the whole
array containing all events needs to be read into memory in one piece,
causing OOM in case of huge files.
But in Lilith, I really need to process events one by one. This is
absolutely crucial.
The only way to achieve this would essentially require me to
re-implement a JSON parser that can cope with data like this.
I implemented a similar kludge to support reading of log4j xml files
(see
https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
) but this was way easier since i could simply search for
"" to find the position at which I could split the stream
of events into a valid XML document for every single event . This was
still kind of nasty but doable, especially since processing an
"offline"
XML file doesn't have the same performance restriction as handling of
live events.
I wouldn't have the luxury of such a unique split signal in JSON,
though.
I would need to detect the "}" at the level of the root array which
would not simply involve counting of opening and closing brackets but
also ignoring of brackets inside of strings. This would boil down to
implementing a custom JSON reader and I won't be doing this.
This isn't just me being lazy either. Such an implementation wouldn't
be
resource (CPU/memory) friendly anyway.
In my own JSON receiver using my own JSON format there are two ways to
send those events:
- write an int containing the amount of bytes representing the event,
then the bytes of the event. This type of events also supports
compression.
- write the bytes of the event, followed by a 0-byte. This works fine
and JavaScript/ActionScript is able to produce events like that while
they are unable (or were unable 7 years ago) to count the bytes on
their
own. This type of events only supports plain text like JSON or XML and
no compression since compressed events could contain the 0-byte while
XML and JSON both won't allow it.
Both are essentially "binary" formats because of either the raw ints or
the 0-bytes.
I'm not sure why you deprecate SerializedLayout since the security
issues arise only while deserializing the events, not while serializing
them. Is this just meant to educate the user about the issue?
I fixed the remote code execution exploit by implementing
https://github.com/huxi/lilith/blob/master/lilith-engine/
src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
er/WhitelistObjectInputStream.java
This still doesn't fix DoS scenarios (endless loops or OOM in case of
malicious data) but is close enough for me, mainly because fixing the
DoS issues would really have to be handled by Java, not by user code.
Manually re-implementing deserialization would likely be fragile and
error prone, possibly even introducing other security issues on its
own.
I'd be interested how other tools are tackling this issue. Is Chainsaw
officially dead or are they implementing receivers for event streams
like this?
I'm totally fine with implementing a new receiver as a replacement for
log4j2 SerializedLayout (even though I won't be able to get rid of the
Serializable receiver anyway) but the current JsonLayout just isn't up
to the job at the moment.
Feel free to share this mail with your fellow developers. I'm open for
suggestions.
I hope this doesn't come across in a harsh way. It isn't meant to be.
Cheers,
Jörn.
