> On Mar 3, 2020, at 2:12 AM, Volkan Yazıcı <volkan.yaz...@gmail.com> wrote:
>
> Hello,
>
> Given I am done with my JsonTemplateLayout PR and waiting for the
> review (I am looking at you Ralph),
I promise I will get to this very soon.
> I have started working on removing
> Jackson dependency. This would allow the layout to be included in the
> core. A major obstacle I encounter is the implementation of the
> following two configuration knobs: maxStringLength (the maximum
> allowed length for a string field in JSON) and maxLength (the maximum
> allowed length of the resultant JSON string length). It is pretty
> difficult to implement them without getting a notable performance hit.
> I am considering leaving these two features out. This opens up the
> following set of questions: Who is responsible for
> truncating/discarding excessive (malicious?) log payloads? Is it
> possible to handle such payloads earlier in the chain by means of a
> built-in filter?
>
> Note that filtering out excessive payloads after layout kicks in is
> not a good idea. Consider the following: An excessive LogEvent makes
> its way up to the layout and causes the internal StringBuilder to grow
> excessively. Later on, the output of the layout is discarded by, say,
> the appender or some other filter. Now... Who is gonna trim the
> internal char[] used by the StringBuilder? The filtering should happen
> before the explosive reaches to the layout. It is not impossible to
> implement all these measures in the layout, but pretty misplaced and
> technically challenging to get it right efficiently.
>
> How one would determine excessive LogEvents is also a challenge on its
> own. The notion of excessive very well depends on the layout. That is,
> for a particular LogEvent, the output of a compressed binary layout
> can be considered okay while its JSON representation might appear
> excessive. But I've just aforementioned filtering should kick in
> before layout. Sort of a chicken and egg problem.
>
> For the records, I've discussed this issue with the colleagues at
> work, where we use LogstashLayout with its excessive payload
> prevention mechanisms in tens of thousands of services, they really
> would like to have either it or a replacement, FYI.
>
> I am pretty much lost at this stage. Any thoughts?
Only a couple of our Layouts actually check for a maxLength, and then only
because the spec they are implementing requires it. Also, performing a length
check is extremely difficult since we are publishing directly into the output
stream. Performing a length check would require creating the stream and then
validating the length, which would be terrible for performance.
Ralph
