Hello, Given I am done with my JsonTemplateLayout PR and waiting for the review (I am looking at you Ralph), I have started working on removing Jackson dependency. This would allow the layout to be included in the core. A major obstacle I encounter is the implementation of the following two configuration knobs: maxStringLength (the maximum allowed length for a string field in JSON) and maxLength (the maximum allowed length of the resultant JSON string length). It is pretty difficult to implement them without getting a notable performance hit. I am considering leaving these two features out. This opens up the following set of questions: Who is responsible for truncating/discarding excessive (malicious?) log payloads? Is it possible to handle such payloads earlier in the chain by means of a built-in filter?
Note that filtering out excessive payloads after layout kicks in is not a good idea. Consider the following: An excessive LogEvent makes its way up to the layout and causes the internal StringBuilder to grow excessively. Later on, the output of the layout is discarded by, say, the appender or some other filter. Now... Who is gonna trim the internal char[] used by the StringBuilder? The filtering should happen before the explosive reaches to the layout. It is not impossible to implement all these measures in the layout, but pretty misplaced and technically challenging to get it right efficiently. How one would determine excessive LogEvents is also a challenge on its own. The notion of excessive very well depends on the layout. That is, for a particular LogEvent, the output of a compressed binary layout can be considered okay while its JSON representation might appear excessive. But I've just aforementioned filtering should kick in before layout. Sort of a chicken and egg problem. For the records, I've discussed this issue with the colleagues at work, where we use LogstashLayout with its excessive payload prevention mechanisms in tens of thousands of services, they really would like to have either it or a replacement, FYI. I am pretty much lost at this stage. Any thoughts? Kinds regards.
