[
https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895202#comment-16895202
]
Otto Fowler edited comment on METRON-2195 at 7/29/19 12:28 PM:
---------------------------------------------------------------
[~tigerquoll] ( METRON-2196 )
Trace logging messages such as at
https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
are causing a performance regression due to the logging function arguments not
being lazily evaluated (the logging string is lazily constructed, but
largeObject.toJsonString() calls are being evaluated (with all the performance
and GC overhead) even if the log message is not going to be emitted due to the
configured log level.
Potential fixes include:
* "If" statements around logging messages, testing for log level before
proceeding
* Using native java logging with slf4j bridge with JDK8+ which supports lazily
evaluating logging method parameters.
* Upgrade slf4j to version 2.x which supports lazily evaluating logging method
parameters.
* Create or use an existing custom log function wrapper around our existing
slf4j 1.x version that will support lazily evaluation of logging method
parameters.
Options 2 or 3 is the architecturally cleaner solutions. Option 4 is the
lightest touch solution classpath wise. Option 1 should not be considered by
anybody who values the lives of kittens.
was (Author: ottobackwards):
[~tigerquoll] ( METRON-2196 )
Trace logging messages such as at
https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
are causing a performance regression due to the logging function arguments not
being lazily evaluated (the logging string is lazily constructed, but
largeObject.toJsonString() calls are being evaluated (with all the performance
and GC overhead) even if the log message is not going to be emitted due to the
configured log level.
Potential fixes include:
"If" statements around logging messages, testing for log level before proceeding
Using native java logging with slf4j bridge with JDK8+ which supports lazily
evaluating logging method parameters.
Upgrade slf4j to version 2.x which supports lazily evaluating logging method
parameters.
Create or use an existing custom log function wrapper around our existing slf4j
1.x version that will support lazily evaluation of logging method parameters.
Options 2 or 3 is the architecturally cleaner solutions. Option 4 is the
lightest touch solution classpath wise. Option 1 should not be considered by
anybody who values the lives of kittens.
> Add defensive log level checks when constructing logs is expensive
> ------------------------------------------------------------------
>
> Key: METRON-2195
> URL: https://issues.apache.org/jira/browse/METRON-2195
> Project: Metron
> Issue Type: Improvement
> Reporter: Otto Fowler
> Priority: Critical
>
> There are instances where we log, and output strings for json objects and
> other things that are quite expensive.
> These are done regardless of the log level being enabled and can increase
> performance significantly:
> https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
> We need to find places where this happens, and employ the best practice check
> for the log level being enabled ( best practice if your parameters or log
> message construction takes a lot of time ).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
