[jira] [Comment Edited] (LOG4J2-1229) Error creating logger

[Xiaopeng Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177439#comment-15177439
 ] 

Xiaopeng Liu edited comment on LOG4J2-1229 at 3/3/16 7:49 AM:
--

Sorry, it has been fixed by setting different filename attribute in  and 
 tags.

{noformat}











   

























{noformat}


was (Author: xiaopeng.liu):
Sorry, it has been fixed by setting different filename attribute in  and 
 tags.

{quote}











   

























{quote}

> Error creating logger
> -
>
> Key: LOG4J2-1229
> URL: https://issues.apache.org/jira/browse/LOG4J2-1229
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Appenders
>Affects Versions: 2.4.1, 2.5
> Environment: Ubuntu 14.04
>Reporter: Dmitriy Dumanskiy
> Fix For: 2.6
>
>
> I have a lot of VMs where everything is fine. But on some of them  I get 
> below stack trace. I have a feeling it somehow related with access rights. 
> Here is stack trace :
> {code}
> 2015-12-21 08:05:06,036 AsyncLogger-1 ERROR Unable to invoke factory method 
> in class class org.apache.logging.log4j.core.appender.RollingFileAppender for 
> element RollingFile. java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
> at 
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
> at 
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
> at 
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:152)
> at 
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:136)
> at 
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:110)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at 
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:390)
> at 
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:378)
> at 
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:362)
> at 
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
> at 
> org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:385)
> at 
> org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:103)
> at 
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:43)
> at 
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:28)
> at 
> 

[jira] [Commented] (LOG4J2-1229) Error creating logger

[Xiaopeng Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177439#comment-15177439
 ] 

Xiaopeng Liu commented on LOG4J2-1229:
--

Sorry, it has been fixed by setting different filename attribute in  and 
 tags.

{quote}











   

























{quote}

> Error creating logger
> -
>
> Key: LOG4J2-1229
> URL: https://issues.apache.org/jira/browse/LOG4J2-1229
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Appenders
>Affects Versions: 2.4.1, 2.5
> Environment: Ubuntu 14.04
>Reporter: Dmitriy Dumanskiy
> Fix For: 2.6
>
>
> I have a lot of VMs where everything is fine. But on some of them  I get 
> below stack trace. I have a feeling it somehow related with access rights. 
> Here is stack trace :
> {code}
> 2015-12-21 08:05:06,036 AsyncLogger-1 ERROR Unable to invoke factory method 
> in class class org.apache.logging.log4j.core.appender.RollingFileAppender for 
> element RollingFile. java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
> at 
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
> at 
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
> at 
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:152)
> at 
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:136)
> at 
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:110)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at 
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at 
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:390)
> at 
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:378)
> at 
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:362)
> at 
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
> at 
> org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:385)
> at 
> org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:103)
> at 
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:43)
> at 
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:28)
> at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: ManagerFactory 
> [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@346629f5]
>  unable to create manager for [/data//1_v4.csv] with data 
> [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@6d48f4b0]
> at 
> org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
> at 
> org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
> at 
> 

[jira] [Closed] (LOG4J2-1217) Layout option to limit length of text

[Thies Wellpott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thies Wellpott closed LOG4J2-1217.
--

Thanks Matt for including it in the log4j2 source+doc environment, looks good.

> Layout option to limit length of text
> -
>
> Key: LOG4J2-1217
> URL: https://issues.apache.org/jira/browse/LOG4J2-1217
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Pattern Converters
>Affects Versions: 2.4.1
>Reporter: Thies Wellpott
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> PatternLayout does not provide the ability to limit the length of the text 
> without space padding (with the numbers directly after the %).
> The following Converter provides this possibility:
> {{%maxLen\{pattern}\{max. length\}}}, e.g. {{%maxLen\{%m}\{90\}}}
> use {{%maxLen}} or {{%maxLength}}
> {code:title=MaxLengthConverter.java}
> package de.ittw.log4j;
> import java.util.List;
> import org.apache.logging.log4j.core.LogEvent;
> import org.apache.logging.log4j.core.appender.AbstractAppender;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.plugins.Plugin;
> import org.apache.logging.log4j.core.layout.PatternLayout;
> import org.apache.logging.log4j.core.pattern.ConverterKeys;
> import org.apache.logging.log4j.core.pattern.LogEventPatternConverter;
> import org.apache.logging.log4j.core.pattern.PatternConverter;
> import org.apache.logging.log4j.core.pattern.PatternFormatter;
> import org.apache.logging.log4j.core.pattern.PatternParser;
> /**
>  * Max length pattern converter. Limit contained text to a maximum length.
>  * On invalid length the default value 100 is used (and an error message is 
> logged).
>  * If max length is greater than 20, an abbreviated text will get ellipsis 
> ("...") appended.
>  * Example usage (for email subject):
>  *"%maxLen{[AppName, ${hostName}, ${web:contextPath}] %p: %c{1} - 
> %m%notEmpty{ =>%ex{short}}}{160}"
>  *
>  * @author Thies Wellpott
>  */
> @Plugin(name = "maxLength", category = PatternConverter.CATEGORY)
> @ConverterKeys({ "maxLength", "maxLen" })
> public final class MaxLengthConverter extends LogEventPatternConverter
> {
> /**
>  * Gets an instance of the class.
>  *
>  * @param config
>  *The current Configuration.
>  * @param options
>  *pattern options, an array of two elements: pattern, max 
> length (defaults to 100 on invalid value).
>  * @return instance of class.
>  */
> public static MaxLengthConverter newInstance(final Configuration config, 
> final String[] options) {
> if (options.length != 2) {
> LOGGER.error("Incorrect number of options on maxLength: expected 
> 2 received {}: {}", options.length, options);
> return null;
> }
> if (options[0] == null) {
> LOGGER.error("No pattern supplied on maxLength");
> return null;
> }
> if (options[1] == null) {
> LOGGER.error("No length supplied on maxLength");
> return null;
> }
> final PatternParser parser = 
> PatternLayout.createPatternParser(config);
> final List formatters = parser.parse(options[0]);
> return new MaxLengthConverter(formatters, 
> AbstractAppender.parseInt(options[1], 100));
> }
> private final List formatters;
> private final int maxLength;
> /**
>  * Construct the converter.
>  *
>  * @param formatters
>  *The PatternFormatters to generate the text to manipulate.
>  * @param maxLength
>  *The max. length of the resultung string. Ellipses ("...") 
> is appended on shorted string, if greater than 20.
>  */
> private MaxLengthConverter(final List formatters, final 
> int maxLength) {
> super("MaxLength", "maxLength");
> this.maxLength = maxLength;
> this.formatters = formatters;
> LOGGER.trace("new MaxLengthConverter with {}", maxLength);
> }
> @Override
> public void format(final LogEvent event, final StringBuilder toAppendTo) {
> final StringBuilder buf = new StringBuilder();
> for (final PatternFormatter formatter : formatters) {
> formatter.format(event, buf);
> if (buf.length() > maxLength) {// stop early
> break;
> }
> }
> if (buf.length() > maxLength) {
> buf.setLength(maxLength);
> if (maxLength > 20) {// only append ellipses if length is 
> not very short
> buf.append("...");
> }
> }
> toAppendTo.append(buf);
> }
> }
> {code}
> You may include this in log4j2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LOG4J2-1216) Nested pattern layout options broken

[Thies Wellpott (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177413#comment-15177413
 ] 

Thies Wellpott commented on LOG4J2-1216:


Sorry Matt (and a bit shame on me), but I did not write unit tests. I just 
tested it directly in my project using log4j2 with some quite deeply nested 
options (example see issue description).

> Nested pattern layout options broken
> 
>
> Key: LOG4J2-1216
> URL: https://issues.apache.org/jira/browse/LOG4J2-1216
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Pattern Converters
>Affects Versions: 2.4.1
>Reporter: Thies Wellpott
>  Labels: easyfix
>
> Parsing the "deeply" nested PatternLayout
> {code}
> %maxLen{[XXX, ${hostName}, ${web:contextPath}] %p: %c{1} - %m%notEmpty{ 
> =>%ex{short}}}{160}
> {code}
> (with %maxLen being a custom Converter)
> results in wrong parsing.
> Patternparser.extractOptions() gets the nesting wrong.
> Solution:
> {code}
> private static int extractOptions(final String pattern, final int start, 
> final List options) {
> int i = start;
> while (i < pattern.length()  &&  pattern.charAt(i) == '{') {
> i++;  // skip opening "{"
> final int begin = i;  // position of first real char
> int depth = 1;// already inside one level
> while (depth > 0  &&  i < pattern.length()) {
> char c = pattern.charAt(i);
> if (c == '{') {
> depth++;
> } else if (c == '}') {
> depth--;
> // TODO(?) maybe escaping of { and } with \ or %
> }
> i++;
> } // while
> if (depth > 0) {  // option not closed, continue with pattern 
> on opening "{"
> return begin-1;
> }
> options.add(pattern.substring(begin, i-1));
> } // while
> return i;
> }
> {code}
> This should also be faster than the current implementation because the 
> pattern ist only walked once, not multiple times with indexOf().
> (LOG4J2-107 is a similar but old bug)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Updated] (LOG4J2-1305) Binary Layout

[Remko Popma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remko Popma updated LOG4J2-1305:

Description: 
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary to either 
display or convert to human-readable form. 

This ticket proposes a simple BinaryLayout, where each LogEvent is logged in a 
binary format.

*Example BinaryLayout format*
||Offset||Type||Description||
|0|long|TimeMillis|
|8|long|NanoTime|
|16|int|Level|
|20|int|Logger name index - string value in separate file|
|24|int|Thread name index - string value in separate file|
|28|long|Thread ID|
|36|int|Marker index - value & hierarchy in separate file|
|40|int|Message length|
|44|int|Message type: 0=text, 1+=custom message type|
|48|byte[]|Message data - below offset assumes 12 bytes of message data|
|60|int| Throwable data length|
|64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
|80|int|ThreadContext key/value pair count|
|84|int|ThreadContext key index - string value in separate file|
|88|int|ThreadContext value index - string value in separate file|

*Versioning*
The binary file must start with a header, indicating version information and 
perhaps schema information providing meta data on the log record. Schema 
information may make it possible to include/exclude fields. For version 1.0, 
the schema can either be fixed like the above example, or it could be a simple 
bitmask for the fields mentioned above.

*Custom Messages*
Note: custom Messages that implement the {{Encoder}} interface (introduced with 
LOG4J2-1274) can be written in binary form directly without first being 
converted to text (LOG4J2-506). Any specialized tool for reading binary log 
files should handle messages of type "text" out of the box, but could have some 
plugin mechanism for decoding custom messages.

A more flexible variation of this is to have a registry of Encoders that map 
Classes to the associated Encoder. That would allow any ObjectMessage to be 
encoded in binary form without demanding a custom Message implementation for 
each Object.

*Byte Order*
TBD: Are multi-byte values like ints and longs written in big Endian or little 
Endian? This could be specified in the header, or we could fix it to either 
one. Exchange protocols like ITCH tend to select a fixed byte order (ITCH uses 
big Endian - network byte order). I like the simplicity of this approach.

*Multiple Files*
Repeating String data like thread names, logger names, marker names and 
ThreadContextMap keys and values are saved to a separate string-data file. The 
main log file contains an index (the line number, zero-based) into the 
string-data file instead of the full string. Index -1 means the String value 
was {{null}}. The format of the string-data file can simply be: each unique 
string on a separate line (separated by '\n' (0x0A) character). Any '\n' 
characters embedded in the string value are Unicode escaped and writen as 
"\u000A".

TBD: as Matt points out in the comment, Markers are special since they are 
hierarchic. One way to deal with this is to manage a separate file to save the 
Marker hierarchy. Another way is to do something similar to PatternLayout: 
treat it as a String value, where the string includes hierarchy information. I 
like the simplicity of the latter approach.

  was:
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary 

[jira] [Commented] (LOG4J2-1305) Binary Layout

[Gary Gregory (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177388#comment-15177388
 ] 

Gary Gregory commented on LOG4J2-1305:
--

I just added the thread priority to the LogEvent, we should have it here as 
well.

> Binary Layout
> -
>
> Key: LOG4J2-1305
> URL: https://issues.apache.org/jira/browse/LOG4J2-1305
> Project: Log4j 2
>  Issue Type: New Feature
>  Components: Layouts
>Reporter: Remko Popma
>  Labels: binary
>
> Logging in a binary format instead of in text can give large performance 
> improvements. 
> Logging text means going from a LogEvent object to formatted text, and then 
> converting this text to bytes. Performance investigations with text-based 
> logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
> bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
> expensive and imposes limits on the performance that can be achieved. 
> A different approach would be to convert the LogEvent to a binary 
> representation directly without creating a text representation first. This 
> would result in extremely compact log files that are fast to write. The 
> trade-off is that a binary log cannot easily be read in a general-purpose 
> editor like VI or Notepad. A specialized tool would be necessary to either 
> display or convert to human-readable form. 
> This ticket proposes a simple BinaryLayout, where each LogEvent is logged in 
> a binary format.
> *Example BinaryLayout format*
> ||Offset||Type||Description||
> |0|long|TimeMillis|
> |8|long|NanoTime|
> |16|int|Level|
> |20|int|Logger name index - string value in separate file|
> |24|int|Thread name index - string value in separate file|
> |28|long|Thread ID|
> |36|int|Marker index - value & hierarchy in separate file|
> |40|int|Message length|
> |44|int|Message type: 0=text, 1+=custom message type|
> |48|byte[]|Message data - below offset assumes 12 bytes of message data|
> |60|int| Throwable data length|
> |64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
> |80|int|ThreadContext key/value pair count|
> |84|int|ThreadContext key index - string value in separate file|
> |88|int|ThreadContext value index - string value in separate file|
> *Versioning*
> The binary file must start with a header, indicating version information and 
> perhaps schema information providing meta data on the log record. Schema 
> information may make it possible to include/exclude fields. For version 1.0, 
> the schema can either be fixed like the above example, or it could be a 
> simple bitmask for the fields mentioned above.
> *Custom Messages*
> Note: custom Messages that implement the {{Encoder}} interface (introduced 
> with LOG4J2-1274) can be written in binary form directly without first being 
> converted to text (LOG4J2-506). Any specialized tool for reading binary log 
> files should handle messages of type "text" out of the box, but could have 
> some plugin mechanism for decoding custom messages.
> *Byte Order*
> TBD: Are multi-byte values like ints and longs written in big Endian or 
> little Endian? This could be specified in the header, or we could fix it to 
> either one. Exchange protocols like ITCH tend to select a fixed byte order 
> (ITCH uses big Endian - network byte order). I like the simplicity of this 
> approach.
> *Multiple Files*
> Repeating String data like thread names, logger names, marker names and 
> ThreadContextMap keys and values are saved to a separate string-data file. 
> The main log file contains an index (the line number, zero-based) into the 
> string-data file instead of the full string. Index -1 means the String value 
> was {{null}}. The format of the string-data file can simply be: each unique 
> string on a separate line (separated by '\n' (0x0A) character). Any '\n' 
> characters embedded in the string value are Unicode escaped and writen as 
> "\u000A".
> TBD: as Matt points out in the comment, Markers are special since they are 
> hierarchic. One way to deal with this is to manage a separate file to save 
> the Marker hierarchy. Another way is to do something similar to 
> PatternLayout: treat it as a String value, where the string includes 
> hierarchy information. I like the simplicity of the latter approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Updated] (LOG4J2-1305) Binary Layout

[Remko Popma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remko Popma updated LOG4J2-1305:

Description: 
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary to either 
display or convert to human-readable form. 

This ticket proposes a simple BinaryLayout, where each LogEvent is logged in a 
binary format.

*Example BinaryLayout format*
||Offset||Type||Description||
|0|long|TimeMillis|
|8|long|NanoTime|
|16|int|Level|
|20|int|Logger name index - string value in separate file|
|24|int|Thread name index - string value in separate file|
|28|long|Thread ID|
|36|int|Marker index - value & hierarchy in separate file|
|40|int|Message length|
|44|int|Message type: 0=text, 1+=custom message type|
|48|byte[]|Message data - below offset assumes 12 bytes of message data|
|60|int| Throwable data length|
|64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
|80|int|ThreadContext key/value pair count|
|84|int|ThreadContext key index - string value in separate file|
|88|int|ThreadContext value index - string value in separate file|

*Versioning*
The binary file must start with a header, indicating version information and 
perhaps schema information providing meta data on the log record. Schema 
information may make it possible to include/exclude fields. For version 1.0, 
the schema can either be fixed like the above example, or it could be a simple 
bitmask for the fields mentioned above.

*Custom Messages*
Note: custom Messages that implement the {{Encoder}} interface (introduced with 
LOG4J2-1274) can be written in binary form directly without first being 
converted to text (LOG4J2-506). Any specialized tool for reading binary log 
files should handle messages of type "text" out of the box, but could have some 
plugin mechanism for decoding custom messages.

*Byte Order*
TBD: Are multi-byte values like ints and longs written in big Endian or little 
Endian? This could be specified in the header, or we could fix it to either 
one. Exchange protocols like ITCH tend to select a fixed byte order (ITCH uses 
big Endian - network byte order). I like the simplicity of this approach.

*Multiple Files*
Repeating String data like thread names, logger names, marker names and 
ThreadContextMap keys and values are saved to a separate string-data file. The 
main log file contains an index (the line number, zero-based) into the 
string-data file instead of the full string. Index -1 means the String value 
was {{null}}. The format of the string-data file can simply be: each unique 
string on a separate line (separated by '\n' (0x0A) character). Any '\n' 
characters embedded in the string value are Unicode escaped and writen as 
"\u000A".

TBD: as Matt points out in the comment, Markers are special since they are 
hierarchic. One way to deal with this is to manage a separate file to save the 
Marker hierarchy. Another way is to do something similar to PatternLayout: 
treat it as a String value, where the string includes hierarchy information. I 
like the simplicity of the latter approach.

  was:
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary to either 
display or convert to human-readable form. 

This ticket proposes a simple BinaryLayout, where each LogEvent is logged in a 
binary format.

*Example BinaryLayout format*
||Offset||Type||Description||
|0|long|TimeMillis|

[jira] [Comment Edited] (LOG4J2-1305) Binary Layout

[Gary Gregory (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177388#comment-15177388
 ] 

Gary Gregory edited comment on LOG4J2-1305 at 3/3/16 7:11 AM:
--

I just added the thread priority (int) to the LogEvent, we should have it here 
as well.


was (Author: garydgregory):
I just added the thread priority to the LogEvent, we should have it here as 
well.

> Binary Layout
> -
>
> Key: LOG4J2-1305
> URL: https://issues.apache.org/jira/browse/LOG4J2-1305
> Project: Log4j 2
>  Issue Type: New Feature
>  Components: Layouts
>Reporter: Remko Popma
>  Labels: binary
>
> Logging in a binary format instead of in text can give large performance 
> improvements. 
> Logging text means going from a LogEvent object to formatted text, and then 
> converting this text to bytes. Performance investigations with text-based 
> logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
> bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
> expensive and imposes limits on the performance that can be achieved. 
> A different approach would be to convert the LogEvent to a binary 
> representation directly without creating a text representation first. This 
> would result in extremely compact log files that are fast to write. The 
> trade-off is that a binary log cannot easily be read in a general-purpose 
> editor like VI or Notepad. A specialized tool would be necessary to either 
> display or convert to human-readable form. 
> This ticket proposes a simple BinaryLayout, where each LogEvent is logged in 
> a binary format.
> *Example BinaryLayout format*
> ||Offset||Type||Description||
> |0|long|TimeMillis|
> |8|long|NanoTime|
> |16|int|Level|
> |20|int|Logger name index - string value in separate file|
> |24|int|Thread name index - string value in separate file|
> |28|long|Thread ID|
> |36|int|Marker index - value & hierarchy in separate file|
> |40|int|Message length|
> |44|int|Message type: 0=text, 1+=custom message type|
> |48|byte[]|Message data - below offset assumes 12 bytes of message data|
> |60|int| Throwable data length|
> |64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
> |80|int|ThreadContext key/value pair count|
> |84|int|ThreadContext key index - string value in separate file|
> |88|int|ThreadContext value index - string value in separate file|
> *Versioning*
> The binary file must start with a header, indicating version information and 
> perhaps schema information providing meta data on the log record. Schema 
> information may make it possible to include/exclude fields. For version 1.0, 
> the schema can either be fixed like the above example, or it could be a 
> simple bitmask for the fields mentioned above.
> *Custom Messages*
> Note: custom Messages that implement the {{Encoder}} interface (introduced 
> with LOG4J2-1274) can be written in binary form directly without first being 
> converted to text (LOG4J2-506). Any specialized tool for reading binary log 
> files should handle messages of type "text" out of the box, but could have 
> some plugin mechanism for decoding custom messages.
> *Byte Order*
> TBD: Are multi-byte values like ints and longs written in big Endian or 
> little Endian? This could be specified in the header, or we could fix it to 
> either one. Exchange protocols like ITCH tend to select a fixed byte order 
> (ITCH uses big Endian - network byte order). I like the simplicity of this 
> approach.
> *Multiple Files*
> Repeating String data like thread names, logger names, marker names and 
> ThreadContextMap keys and values are saved to a separate string-data file. 
> The main log file contains an index (the line number, zero-based) into the 
> string-data file instead of the full string. Index -1 means the String value 
> was {{null}}. The format of the string-data file can simply be: each unique 
> string on a separate line (separated by '\n' (0x0A) character). Any '\n' 
> characters embedded in the string value are Unicode escaped and writen as 
> "\u000A".
> TBD: as Matt points out in the comment, Markers are special since they are 
> hierarchic. One way to deal with this is to manage a separate file to save 
> the Marker hierarchy. Another way is to do something similar to 
> PatternLayout: treat it as a String value, where the string includes 
> hierarchy information. I like the simplicity of the latter approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Updated] (LOG4J2-1305) Binary Layout

[Remko Popma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remko Popma updated LOG4J2-1305:

Description: 
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary to either 
display or convert to human-readable form. 

This ticket proposes a simple BinaryLayout, where each LogEvent is logged in a 
binary format.

*Example BinaryLayout format*
||Offset||Type||Description||
|0|long|TimeMillis|
|8|long|NanoTime|
|16|int|Level|
|20|int|Logger name index - string value in separate file|
|24|int|Thread name index - string value in separate file|
|28|long|Thread ID|
|36|int|Marker index - value & hierarchy in separate file|
|40|int|Message length|
|44|int|Message type: 0=text, 1+=custom message type|
|48|byte[]|Message data - below offset assumes 12 bytes of message data|
|60|int| Throwable data length|
|64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
|80|int|ThreadContext key/value pair count|
|84|int|ThreadContext key index - string value in separate file|
|88|int|ThreadContext value index - string value in separate file|

*Versioning*
The binary file must start with a header, indicating version information and 
perhaps schema information providing meta data on the log record. Schema 
information may make it possible to include/exclude fields. For version 1.0, 
the schema can either be fixed like the above example, or it could be a simple 
bitmask for the fields mentioned above.

*Custom Messages*
Note: custom Messages that implement the {{Encoder}} interface (introduced with 
LOG4J2-1274) can be written in binary form directly without first being 
converted to text (LOG4J2-506). Any specialized tool for reading binary log 
files should handle messages of type "text" out of the box, but could have some 
plugin mechanism for decoding custom messages.

*Byte Order*
TBD: Are multi-byte values like ints and longs written in big Endian or little 
Endian? This could be specified in the header, or we could fix it to either 
one. Exchange protocols like ITCH tend to select a fixed byte order (ITCH uses 
big Endian - network byte order). I like the simplicity of this approach.

*Multiple Files*
Repeating String data like thread names, logger names, marker names and 
ThreadContextMap keys and values are saved to a separate string-data file. The 
main log file contains an index (the line number, zero-based) into the 
string-data file instead of the full string. The format of this file can simply 
be: each unique string on a separate line (separated by '\n' (0x0A) character). 
Any '\n' characters embedded in the string value are Unicode escaped and writen 
as "\u000A".

TBD: as Matt points out in the comment, Markers are special since they are 
hierarchic. One way to deal with this is to manage a separate file to save the 
Marker hierarchy. Another way is to do something similar to PatternLayout: 
treat it as a String value, where the string includes hierarchy information. I 
like the simplicity of the latter approach.

  was:
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary to either 
display or convert to human-readable form. 

Note: custom Messages that implement the {{Encoder}} interface (introduced with 
LOG4J2-1274) can be written in binary form directly without first being 
converted to text (LOG4J2-506). Any specialized tool for reading binary log 
files should 

[jira] [Commented] (LOG4J2-1305) Binary Layout

[Remko Popma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177370#comment-15177370
 ] 

Remko Popma commented on LOG4J2-1305:
-

Thanks for pointing that out. I simplified the format and added a note.

> Binary Layout
> -
>
> Key: LOG4J2-1305
> URL: https://issues.apache.org/jira/browse/LOG4J2-1305
> Project: Log4j 2
>  Issue Type: New Feature
>  Components: Layouts
>Reporter: Remko Popma
>  Labels: binary
>
> Logging in a binary format instead of in text can give large performance 
> improvements. 
> Logging text means going from a LogEvent object to formatted text, and then 
> converting this text to bytes. Performance investigations with text-based 
> logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
> bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
> expensive and imposes limits on the performance that can be achieved. 
> A different approach would be to convert the LogEvent to a binary 
> representation directly without creating a text representation first. This 
> would result in extremely compact log files that are fast to write. The 
> trade-off is that a binary log cannot easily be read in a general-purpose 
> editor like VI or Notepad. A specialized tool would be necessary to either 
> display or convert to human-readable form. 
> Note: custom Messages that implement the {{Encoder}} interface (introduced 
> with LOG4J2-1274) can be written in binary form directly without first being 
> converted to text (LOG4J2-506). Any specialized tool for reading binary log 
> files should handle messages of type "text" out of the box, but could have 
> some plugin mechanism for decoding custom messages.
> This ticket proposes a simple BinaryLayout, where each LogEvent is logged in 
> a binary format.
> *Example BinaryLayout format*
> ||Offset||Type||Description||
> |0|long|TimeMillis|
> |8|long|NanoTime|
> |16|int|Level|
> |20|int|Logger name index - string value in separate file|
> |24|int|Thread name index - string value in separate file|
> |28|long|Thread ID|
> |36|int|Marker index - value & hierarchy in separate file|
> |40|int|Message length|
> |44|int|Message type|
> |48|byte[]|Message data - below offset assumes 12 bytes of message data|
> |60|int| Throwable data length|
> |64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
> |80|int|ThreadContext key/value pair count|
> |84|int|ThreadContext key index - string value in separate file|
> |88|int|ThreadContext value index - string value in separate file|
> *Versioning*
> The binary file must start with a header, indicating version information and 
> perhaps schema information providing meta data on the log record. Schema 
> information may make it possible to include/exclude fields. For version 1.0, 
> the schema can either be fixed like the above example, or it could be a 
> simple bitmask for the fields mentioned above.
> *Byte Order*
> TBD: Are multi-byte values like ints and longs written in big Endian or 
> little Endian? This could be specified in the header, or we could fix it to 
> either one. Exchange protocols like ITCH tend to select a fixed byte order 
> (ITCH uses big Endian - network byte order). I like the simplicity of this 
> approach.
> *Multiple Files*
> Repeating String data like thread names, logger names, marker names and 
> ThreadContextMap keys and values are saved to a separate string-data file. 
> The main log file contains an index (the line number, zero-based) into the 
> string-data file instead of the full string. The format of this file can 
> simply be: each unique string on a separate line (separated by '\n' (0x0A) 
> character). Any '\n' characters embedded in the string value are Unicode 
> escaped and writen as "\u000A".
> TBD, as Matt points out in the comment, Markers are special since they are 
> hierarchic. One way to deal with this is to manage a separate file to save 
> the Marker hierarchy. Another way is to do something similar to 
> PatternLayout: treat it as a String value, where the string includes 
> hierarchy information. I like the simplicity of the latter approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Updated] (LOG4J2-1305) Binary Layout

[Remko Popma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remko Popma updated LOG4J2-1305:

 Labels: binary  (was: )
Description: 
Logging in a binary format instead of in text can give large performance 
improvements. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. Performance investigations with text-based 
logging formats like PatternLayout (see LOG4J2-930), and encoding Strings to 
bytes (LOG4J2-935, LOG4J2-1151) suggest that formatting and encoding text is 
expensive and imposes limits on the performance that can be achieved. 

A different approach would be to convert the LogEvent to a binary 
representation directly without creating a text representation first. This 
would result in extremely compact log files that are fast to write. The 
trade-off is that a binary log cannot easily be read in a general-purpose 
editor like VI or Notepad. A specialized tool would be necessary to either 
display or convert to human-readable form. 

Note: custom Messages that implement the {{Encoder}} interface (introduced with 
LOG4J2-1274) can be written in binary form directly without first being 
converted to text (LOG4J2-506). Any specialized tool for reading binary log 
files should handle messages of type "text" out of the box, but could have some 
plugin mechanism for decoding custom messages.

This ticket proposes a simple BinaryLayout, where each LogEvent is logged in a 
binary format.

*Example BinaryLayout format*
||Offset||Type||Description||
|0|long|TimeMillis|
|8|long|NanoTime|
|16|int|Level|
|20|int|Logger name index - string value in separate file|
|24|int|Thread name index - string value in separate file|
|28|long|Thread ID|
|36|int|Marker index - value & hierarchy in separate file|
|40|int|Message length|
|44|int|Message type|
|48|byte[]|Message data - below offset assumes 12 bytes of message data|
|60|int| Throwable data length|
|64|byte[]|Throwable data - below offset assumes 16 bytes of Throwable data|
|80|int|ThreadContext key/value pair count|
|84|int|ThreadContext key index - string value in separate file|
|88|int|ThreadContext value index - string value in separate file|

*Versioning*
The binary file must start with a header, indicating version information and 
perhaps schema information providing meta data on the log record. Schema 
information may make it possible to include/exclude fields. For version 1.0, 
the schema can either be fixed like the above example, or it could be a simple 
bitmask for the fields mentioned above.

*Byte Order*
TBD: Are multi-byte values like ints and longs written in big Endian or little 
Endian? This could be specified in the header, or we could fix it to either 
one. Exchange protocols like ITCH tend to select a fixed byte order (ITCH uses 
big Endian - network byte order). I like the simplicity of this approach.

*Multiple Files*
Repeating String data like thread names, logger names, marker names and 
ThreadContextMap keys and values are saved to a separate string-data file. The 
main log file contains an index (the line number, zero-based) into the 
string-data file instead of the full string. The format of this file can simply 
be: each unique string on a separate line (separated by '\n' (0x0A) character). 
Any '\n' characters embedded in the string value are Unicode escaped and writen 
as "\u000A".

TBD, as Matt points out in the comment, Markers are special since they are 
hierarchic. One way to deal with this is to manage a separate file to save the 
Marker hierarchy. Another way is to do something similar to PatternLayout: 
treat it as a String value, where the string includes hierarchy information. I 
like the simplicity of the latter approach.

  was:
Logging in a binary format instead of in text can give large performance 
improvements. Text-based logging formats are supported by layouts like 
PatternLayout, and performance investigations like done in LOG4J2-930 suggest 
that it may be difficult to achieve good performance when logging text. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. A different approach would be to convert the 
LogEvent to a binary representation directly without creating a text 
representation first. This may make fast synchronous logging possible, although 
that would additionally require a lock-free appender (LOG4J2-928).

This proposes a simple BinaryLayout, where each LogEvent is logged in a binary 
record like this:
||Offset||Type||Description||
|0|long|TimeMillis|
|8|long|NanoTime|
|16|int|Level|
|20|int|Logger name index - string value in separate file|
|24|int|Thread name index - string value in separate file|
|28|long|Thread ID|
|36|short|Marker count|
|38|int|marker name index - string value in separate file ... (below offset 
assumes only one marker)|
|42|int|Message length|

Re: [1/3] logging-log4j2 git commit: LOG4J2-1227 Test if the event is null before using it, to avoid NullPointerException. Add some unit tests. Submitted by: Olivier Lemasle <o.lema...@gmail.com>

[Ralph Goers]

It appears that the newly added testEventMapMessage is not working.

Ralph

> On Mar 2, 2016, at 8:08 PM, mattsic...@apache.org wrote:
> 
> Repository: logging-log4j2
> Updated Branches:
>  refs/heads/master 60d1ccd93 -> a57fc35b9
> 
> 
> LOG4J2-1227
>Test if the event is null before using it, to avoid NullPointerException.
>Add some unit tests.
>Submitted by: Olivier Lemasle 
> 
> 
> Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
> Commit: http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/33ee4bfd
> Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/33ee4bfd
> Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/33ee4bfd
> 
> Branch: refs/heads/master
> Commit: 33ee4bfd0aa1f58a705771222132a42d1dfb328c
> Parents: 13b0dd8
> Author: Olivier Lemasle 
> Authored: Mon Dec 21 16:53:36 2015 +0100
> Committer: Olivier Lemasle 
> Committed: Mon Dec 21 16:53:36 2015 +0100
> 
> --
> .../logging/log4j/core/lookup/MapLookup.java|  2 +-
> .../log4j/core/lookup/MapLookupTest.java| 25 
> 2 files changed, 26 insertions(+), 1 deletion(-)
> --
> 
> 
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/33ee4bfd/log4j-core/src/main/java/org/apache/logging/log4j/core/lookup/MapLookup.java
> --
> diff --git 
> a/log4j-core/src/main/java/org/apache/logging/log4j/core/lookup/MapLookup.java
>  
> b/log4j-core/src/main/java/org/apache/logging/log4j/core/lookup/MapLookup.java
> index c369a0b..c00645e 100644
> --- 
> a/log4j-core/src/main/java/org/apache/logging/log4j/core/lookup/MapLookup.java
> +++ 
> b/log4j-core/src/main/java/org/apache/logging/log4j/core/lookup/MapLookup.java
> @@ -118,7 +118,7 @@ public class MapLookup implements StrLookup {
> 
> @Override
> public String lookup(final LogEvent event, final String key) {
> -final boolean isMapMessage = event.getMessage() instanceof 
> MapMessage;
> +final boolean isMapMessage = event != null && event.getMessage() 
> instanceof MapMessage;
> if (map == null && !isMapMessage) {
> return null;
> }
> 
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/33ee4bfd/log4j-core/src/test/java/org/apache/logging/log4j/core/lookup/MapLookupTest.java
> --
> diff --git 
> a/log4j-core/src/test/java/org/apache/logging/log4j/core/lookup/MapLookupTest.java
>  
> b/log4j-core/src/test/java/org/apache/logging/log4j/core/lookup/MapLookupTest.java
> index be550e3..5f26d15 100644
> --- 
> a/log4j-core/src/test/java/org/apache/logging/log4j/core/lookup/MapLookupTest.java
> +++ 
> b/log4j-core/src/test/java/org/apache/logging/log4j/core/lookup/MapLookupTest.java
> @@ -20,6 +20,9 @@ import static org.junit.Assert.assertEquals;
> 
> import java.util.HashMap;
> 
> +import org.apache.logging.log4j.core.LogEvent;
> +import org.apache.logging.log4j.core.impl.Log4jLogEvent;
> +import org.apache.logging.log4j.message.MapMessage;
> import org.junit.Test;
> 
> /**
> @@ -64,4 +67,26 @@ public class MapLookupTest {
> assertEquals(null, lookup.lookup("foo.txt"));
> }
> 
> +@Test
> +public void testEventMapMessage() {
> +  final HashMap map = new HashMap<>();
> +  map.put("A", "B");
> +  final HashMap eventMap = new HashMap<>();
> +  eventMap.put("A1", "B1");
> +  final MapMessage message = new MapMessage(eventMap);
> +  final LogEvent event = Log4jLogEvent.newBuilder()
> +.setMessage(message)
> +.build();
> +  final MapLookup lookup = new MapLookup(map);
> +  assertEquals("B", lookup.lookup(event, "A"));
> +  assertEquals("B1", lookup.lookup(event, "A"));
> +}
> +
> +@Test
> +public void testNullEvent() {
> +  final HashMap map = new HashMap<>();
> +  map.put("A", "B");
> +  final MapLookup lookup = new MapLookup(map);
> +  assertEquals("B", lookup.lookup(null, "A"));
> +}
> }
> 
> 



-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Comment Edited] (LOG4J2-935) Performance: which String.getBytes method to use

[Remko Popma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942442#comment-14942442
 ] 

Remko Popma edited comment on LOG4J2-935 at 3/3/16 5:32 AM:


Doing research for LOG4J2-1142 I discovered again just how much processing time 
is spent in converting Strings to byte[] arrays. I will just post these 
benchmark results here for now.

{code}
// serialize a log event and convert by calling String.getBytes(CharSet)
@Benchmark
public byte[] newInstance() {
return serializeWithNewInstance(LOG4J2EVENT).getBytes(CHARSET_DEFAULT);
}

// serialize a log event and leave as String
@Benchmark
public String _stringNewInstance() {
return serializeWithNewInstance(LOG4J2EVENT);
}
{code}

One thread:
{noformat}
java -jar log4j-perf/target/benchmarks.jar ".*ThreadLocalVsPool.*" -f 1 -wi 10 
-i 20 -tu ns -bm sample

# Run complete. Total time: 00:03:45

Benchmark  Mode  Samples
 ScoreError  Units
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark._stringNewInstancesample   260215   
689.856 ± 14.194  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark._stringObjectPool sample   340346   
533.474 ±  7.776  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark._stringThreadLocalsample   213014   
434.266 ±  9.796  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark.newInstance   sample   275170  
1279.410 ± 19.773  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark.objectPoolsample   308588  
1147.695 ± 12.717  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark.threadLocal   sample   328446  
1085.917 ± 14.877  ns/op
{noformat}



was (Author: rem...@yahoo.com):
Doing research for LOG4J2-1142 I discovered again just how much processing time 
is spent in converting Strings to byte[] arrays. I will just post these 
benchmark results here for now.

{code}
// serialize a log event and convert by calling String.getBytes(CharSet)
@Benchmark
public byte[] newInstance() {
return serializeWithNewInstance(LOG4J2EVENT).getBytes(CHARSET_DEFAULT);
}

// serialize a log event and leave as String
@Benchmark
public String _stringNewInstance() {
return serializeWithNewInstance(LOG4J2EVENT);
}
{code}

One thread:
{code}
java -jar log4j-perf/target/benchmarks.jar ".*ThreadLocalVsPool.*" -f 1 -wi 10 
-i 20 -tu ns -bm sample

# Run complete. Total time: 00:03:45

Benchmark  Mode  Samples
 ScoreError  Units
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark._stringNewInstancesample   260215   
689.856 ± 14.194  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark._stringObjectPool sample   340346   
533.474 ±  7.776  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark._stringThreadLocalsample   213014   
434.266 ±  9.796  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark.newInstance   sample   275170  
1279.410 ± 19.773  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark.objectPoolsample   308588  
1147.695 ± 12.717  ns/op
o.a.l.l.p.j.ThreadLocalVsPoolBenchmark.threadLocal   sample   328446  
1085.917 ± 14.877  ns/op
{code}


> Performance: which String.getBytes method to use
> 
>
> Key: LOG4J2-935
> URL: https://issues.apache.org/jira/browse/LOG4J2-935
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Appenders, Core
>Affects Versions: 2.1
>Reporter: Remko Popma
>Assignee: Remko Popma
> Fix For: 2.4.1
>
>
> I did some investigation to compare the performance of 
> {{String.getBytes(String)}} to {{String.getBytes(Charset)}} and it turns out 
> that the String variant is significantly faster (at least on Windows 7 and 
> Solaris 10).
> I am not sure what causes this but the code path for 
> {{String.getBytes(Charset)}} leads to 
> {{java.lang.StringCoding.encode(Charset, char[], int, int)}} which creates a 
> new {{StringEncoder}} for each invocation. By contrast, 
> {{String.getBytes(String)}} leads to {{java.lang.StringCoding.encode(String, 
> char[], int, int)}} which uses a cached {{StringEncoder}} (at least in Java 6 
> and 7). This may be one reason.
> I also compared Java 6, 7 and 8. Note that with each upgrade performance is 
> getting better, with a huge jump in Java 8 for ISO-8859-1 (I suspect this is 
> done by simply casting each char to a byte, dropping the 8 most significant 
> bits of each 16-bit char).
> Benchmark results follow below, sorted by throughput (best first).
> {{java -jar benchmarks.jar ".\*StringEncodingBenchmark.\*" -f 1 -t 1 -wi 5 -i 
> 5}}
> *Windows 7 (64bit) 2-core Intel i5-2500 CPU @3.30Ghz*
> {noformat}
> jdk1.6.0_45
> Benchmark  Mode  Samples  
>Score   Error Units
> o.a.l.l.p.j.StringEncBenchmark.GetBytesString88591thrpt5  

[jira] [Updated] (LOG4J2-1151) Performance improvement: backport fast Java 8 ISO-8859-1 String to byte[] encoder to AbstractStringLayout

[Remko Popma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remko Popma updated LOG4J2-1151:

Description: 
Java 8 made large improvements to String encoding performance for all character 
sets. Especially for the ISO8859_1 encoding, Java 8 introduced logic that 
simply casts each char in the array to a byte, giving a huge performance 
increase.

This ticket proposes to backport that logic into AbstractStringLayout to get 
the same performance in Java 7.

Focussing on the results for ISO8859_1:
{quote}
Java 7 String.getBytes(): ~200-300 ns/op
Java 8 String.getBytes(): ~70-80 ns/op
Custom logic: ~90 ns/op (on both Java 7 and 8)
{quote}

Benchmarks below are with a representative message string of 100 characters, on 
64 bit Windows 10, Dual Intel Core i5-3317U @1.70GHz with hyperthreading on (4 
cores).

*jdk 1.8.0_60 (64 bit Hotspot)*

{noformat}
Benchmark  Mode  
SamplesScoreError  Units
o.a.l.l.p.j.StringEncodingBenchmark.customIso8859_1  sample   
156848   94.014 ±  2.362  ns/op <- custom
o.a.l.l.p.j.StringEncodingBenchmark.encoderIso8859_1 sample   
103773  859.529 ± 27.461  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytes   sample   
125785  363.129 ± 17.674  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesCharSet88591   sample   
117191   72.606 ±  2.771  ns/op <- variation 1
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesCharSetShiftJISsample   
102914  452.721 ± 21.274  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesString88591sample   
181547   79.653 ±  1.735  ns/op <- variation 2
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesStringShiftJIS sample   
125817  354.961 ±  4.072  ns/op
{noformat}


*jdk 1.7.0_55 (64 bit Hotspot)*

{noformat}
Benchmark  Mode  
Samples ScoreError  Units
o.a.l.l.p.j.StringEncodingBenchmark.customIso8859_1  sample   
14845693.248 ±  2.193  ns/op <- custom
o.a.l.l.p.j.StringEncodingBenchmark.encoderIso8859_1 sample   
176198   516.731 ± 24.054  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytes   sample   
121660   731.530 ± 27.116  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesCharSet88591   sample
99636   234.709 ±  3.234  ns/op <- variation 1
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesCharSetShiftJISsample   
125754  1401.008 ± 25.385  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesString88591sample   
147124   316.411 ± 21.454  ns/op <- variation 2
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesStringShiftJIS sample   
114733   808.315 ± 27.177  ns/op
{noformat}


  was:
Java 8 made large improvements to String encoding performance for all character 
sets. Especially for the ISO8859_1 encoding, Java 8 introduced logic that 
simply casts each char in the array to a byte, giving a huge performance 
increase.

This ticket proposes to backport that logic into AbstractStringLayout to get 
the same performance in Java 7.

Focussing on the results for ISO8859_1:
{quote}
Java 7 String.getBytes(): ~200-300 ns/op
Java 8 String.getBytes(): ~70-80 ns/op
Custom logic: ~90 ns/op (on both Java 7 and 8)
{quote}

Benchmarks below are with a representative message string of 100 characters, on 
64 bit Windows 10, Dual Intel Core i5-3317U @1.70GHz with hyperthreading on (4 
cores).

*jdk 1.8.0_60 (64 bit Hotspot)*

{code}
Benchmark  Mode  
SamplesScoreError  Units
o.a.l.l.p.j.StringEncodingBenchmark.customIso8859_1  sample   
156848   94.014 ±  2.362  ns/op <- custom
o.a.l.l.p.j.StringEncodingBenchmark.encoderIso8859_1 sample   
103773  859.529 ± 27.461  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytes   sample   
125785  363.129 ± 17.674  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesCharSet88591   sample   
117191   72.606 ±  2.771  ns/op <- variation 1
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesCharSetShiftJISsample   
102914  452.721 ± 21.274  ns/op
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesString88591sample   
181547   79.653 ±  1.735  ns/op <- variation 2
o.a.l.l.p.j.StringEncodingBenchmark.stringGetBytesStringShiftJIS sample   
125817  354.961 ±  4.072  ns/op
{code}


*jdk 1.7.0_55 (64 bit Hotspot)*

{code}
Benchmark  Mode  
Samples ScoreError  Units
o.a.l.l.p.j.StringEncodingBenchmark.customIso8859_1  sample   
14845693.248 ±  2.193  ns/op <- custom
o.a.l.l.p.j.StringEncodingBenchmark.encoderIso8859_1 sample   
176198   516.731 ± 24.054  ns/op

Jenkins build is still unstable: Log4j 2.x #1755

[Apache Jenkins Server]

See 


-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: [jira] [Updated] (INFRA-8279) Add Log4j2 to Windows Buildbot

[Ralph Goers]

I would suggest that if we want a build job for Windows that we do it in 
Jenkins.  We seem to have very little control over buildbot and watching failed 
builds go by for the last year without the ability to do anything has 
completely persuaded me to stay away from it.

Ralph


> On Mar 2, 2016, at 9:50 PM, Chris Lambertus (JIRA)  wrote:
> 
> 
> [ 
> https://issues.apache.org/jira/browse/INFRA-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>  ]
> 
> Chris Lambertus updated INFRA-8279:
> ---
>Status: Waiting for user  (was: Waiting for Infra)
> 
> Are you asking for log4j2 installed on the windows build slaves, or are you 
> asking for windows build jobs? This is a 2 year old issue, so this may all be 
> moot. Please let us know one way or the other.
> 
>> Add Log4j2 to Windows Buildbot
>> --
>> 
>>Key: INFRA-8279
>>URL: https://issues.apache.org/jira/browse/INFRA-8279
>>Project: Infrastructure
>> Issue Type: New Feature
>> Security Level: public(Regular issues) 
>>   Reporter: Matt Sicker
>> 
>> We've had numerous issues in the past regarding Windows-specific issues that 
>> weren't noticed for some time or are hard to reproduce outside a Windows 
>> environment. To help our code quality, we'd like to have an additional 
>> buildbot configuration that runs on one of the Windows buildslaves.
>> Environmentally, we'd still need the same things as on the Linux buildslave: 
>> Maven 3.0.x, Java 1.6, and Git.
>> If this would be more appropriate on a different build system (e.g., 
>> Jenkins), please let us know so we can do it there instead. Thanks!
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
> 



-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: [3/3] logging-log4j2 git commit: Add changelog entry for LOG4J2-1227.

[Matt Sicker]

Got it.

On 2 March 2016 at 22:17, Remko Popma  wrote:

> Typo: "is the event" -> "if the event"
>
>
> On Thursday, 3 March 2016,  wrote:
>
>> Add changelog entry for LOG4J2-1227.
>>
>> This closes #20
>>
>>
>> Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
>> Commit:
>> http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/a57fc35b
>> Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/a57fc35b
>> Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/a57fc35b
>>
>> Branch: refs/heads/master
>> Commit: a57fc35b97bda2fec4715040f58f048288f5a2b5
>> Parents: 6362a38
>> Author: Matt Sicker 
>> Authored: Wed Mar 2 21:08:37 2016 -0600
>> Committer: Matt Sicker 
>> Committed: Wed Mar 2 21:08:37 2016 -0600
>>
>> --
>>  src/changes/changes.xml | 3 +++
>>  1 file changed, 3 insertions(+)
>> --
>>
>>
>>
>> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/a57fc35b/src/changes/changes.xml
>> --
>> diff --git a/src/changes/changes.xml b/src/changes/changes.xml
>> index 646a41f..80a67f7 100644
>> --- a/src/changes/changes.xml
>> +++ b/src/changes/changes.xml
>> @@ -184,6 +184,9 @@
>>
>>  JeroMqAppender should support layouts.
>>
>> +  > due-to="Olivier Lemasle">
>> +NullPointerException in MapLookup.lookup is the event is null.
>> +  
>>  
>>  
>>
>>
>>


-- 
Matt Sicker 

[jira] [Resolved] (LOG4J2-1217) Layout option to limit length of text

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1217.
-
   Resolution: Fixed
Fix Version/s: 2.6

I've added your plugin along with a few unit tests and a manual entry. Please 
verify and close.

> Layout option to limit length of text
> -
>
> Key: LOG4J2-1217
> URL: https://issues.apache.org/jira/browse/LOG4J2-1217
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Pattern Converters
>Affects Versions: 2.4.1
>Reporter: Thies Wellpott
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> PatternLayout does not provide the ability to limit the length of the text 
> without space padding (with the numbers directly after the %).
> The following Converter provides this possibility:
> {{%maxLen\{pattern}\{max. length\}}}, e.g. {{%maxLen\{%m}\{90\}}}
> use {{%maxLen}} or {{%maxLength}}
> {code:title=MaxLengthConverter.java}
> package de.ittw.log4j;
> import java.util.List;
> import org.apache.logging.log4j.core.LogEvent;
> import org.apache.logging.log4j.core.appender.AbstractAppender;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.plugins.Plugin;
> import org.apache.logging.log4j.core.layout.PatternLayout;
> import org.apache.logging.log4j.core.pattern.ConverterKeys;
> import org.apache.logging.log4j.core.pattern.LogEventPatternConverter;
> import org.apache.logging.log4j.core.pattern.PatternConverter;
> import org.apache.logging.log4j.core.pattern.PatternFormatter;
> import org.apache.logging.log4j.core.pattern.PatternParser;
> /**
>  * Max length pattern converter. Limit contained text to a maximum length.
>  * On invalid length the default value 100 is used (and an error message is 
> logged).
>  * If max length is greater than 20, an abbreviated text will get ellipsis 
> ("...") appended.
>  * Example usage (for email subject):
>  *"%maxLen{[AppName, ${hostName}, ${web:contextPath}] %p: %c{1} - 
> %m%notEmpty{ =>%ex{short}}}{160}"
>  *
>  * @author Thies Wellpott
>  */
> @Plugin(name = "maxLength", category = PatternConverter.CATEGORY)
> @ConverterKeys({ "maxLength", "maxLen" })
> public final class MaxLengthConverter extends LogEventPatternConverter
> {
> /**
>  * Gets an instance of the class.
>  *
>  * @param config
>  *The current Configuration.
>  * @param options
>  *pattern options, an array of two elements: pattern, max 
> length (defaults to 100 on invalid value).
>  * @return instance of class.
>  */
> public static MaxLengthConverter newInstance(final Configuration config, 
> final String[] options) {
> if (options.length != 2) {
> LOGGER.error("Incorrect number of options on maxLength: expected 
> 2 received {}: {}", options.length, options);
> return null;
> }
> if (options[0] == null) {
> LOGGER.error("No pattern supplied on maxLength");
> return null;
> }
> if (options[1] == null) {
> LOGGER.error("No length supplied on maxLength");
> return null;
> }
> final PatternParser parser = 
> PatternLayout.createPatternParser(config);
> final List formatters = parser.parse(options[0]);
> return new MaxLengthConverter(formatters, 
> AbstractAppender.parseInt(options[1], 100));
> }
> private final List formatters;
> private final int maxLength;
> /**
>  * Construct the converter.
>  *
>  * @param formatters
>  *The PatternFormatters to generate the text to manipulate.
>  * @param maxLength
>  *The max. length of the resultung string. Ellipses ("...") 
> is appended on shorted string, if greater than 20.
>  */
> private MaxLengthConverter(final List formatters, final 
> int maxLength) {
> super("MaxLength", "maxLength");
> this.maxLength = maxLength;
> this.formatters = formatters;
> LOGGER.trace("new MaxLengthConverter with {}", maxLength);
> }
> @Override
> public void format(final LogEvent event, final StringBuilder toAppendTo) {
> final StringBuilder buf = new StringBuilder();
> for (final PatternFormatter formatter : formatters) {
> formatter.format(event, buf);
> if (buf.length() > maxLength) {// stop early
> break;
> }
> }
> if (buf.length() > maxLength) {
> buf.setLength(maxLength);
> if (maxLength > 20) {// only append ellipses if length is 
> not very short
> buf.append("...");
> }
> }
> toAppendTo.append(buf);
> }
> }
> {code}
> You may include this in log4j2

Jenkins build is still unstable: Log4j 2.x #1754

[Apache Jenkins Server]

See 


-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: [3/3] logging-log4j2 git commit: Add changelog entry for LOG4J2-1227.

[Remko Popma]

Typo: "is the event" -> "if the event"


On Thursday, 3 March 2016,  wrote:

> Add changelog entry for LOG4J2-1227.
>
> This closes #20
>
>
> Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
> Commit:
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/a57fc35b
> Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/a57fc35b
> Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/a57fc35b
>
> Branch: refs/heads/master
> Commit: a57fc35b97bda2fec4715040f58f048288f5a2b5
> Parents: 6362a38
> Author: Matt Sicker >
> Authored: Wed Mar 2 21:08:37 2016 -0600
> Committer: Matt Sicker >
> Committed: Wed Mar 2 21:08:37 2016 -0600
>
> --
>  src/changes/changes.xml | 3 +++
>  1 file changed, 3 insertions(+)
> --
>
>
>
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/a57fc35b/src/changes/changes.xml
> --
> diff --git a/src/changes/changes.xml b/src/changes/changes.xml
> index 646a41f..80a67f7 100644
> --- a/src/changes/changes.xml
> +++ b/src/changes/changes.xml
> @@ -184,6 +184,9 @@
>
>  JeroMqAppender should support layouts.
>
> +   due-to="Olivier Lemasle">
> +NullPointerException in MapLookup.lookup is the event is null.
> +  
>  
>  
>
>
>

Re: Fast and Furious

[Remko Popma]

I created https://issues.apache.org/jira/browse/LOG4J2-1305 as a write up
of my current thinking on the topic and a place to discuss ideas. Still
need to add some things we discussed here (tools, endianness, versioning
etc).

It's a fascinating topic but I still have a lot of work to do on the
GC-free epic before I can start working on this one.


On Thursday, 3 March 2016, Remko Popma  wrote:

> Not Java Serialization, I was thinking simple ByteBuffer.putLong, putInt,
> putBytes. This is much more performant (
> http://mechanical-sympathy.blogspot.jp/2012/07/native-cc-like-performance-for-java.html).
> SBE (Simple Binary Encoding) seems overkill, but open to other opinions.
>
> The less conditional logic in there the better, so I'm not that interested
> in configurability. All log event fields, ok. ThreadContextMap/Stack keys
> and values: similarly to other repeating strings like logger names: write
> to separate mapping file & only write int values (for count as well as
> key/value indices) to the "main" log file.
>
> Two things we would need to decide is how to handle versioning, and what
> Endianness to use.
>
> Version information (possibly with schema info) could be written to the
> log file header.
>
> Endianness could also be written to the header, or we could simply say we
> use network byte order (big endian). Intel chips use little endian, but
> apparently swapping is implemented with an intrinsic and is very fast.
>
> On Thursday, 3 March 2016, Matt Sicker  > wrote:
>
>> At that point, it would be nice if it were extensible. There are some
>> neat binary formats we could use that would allow for that.
>>
>> On 2 March 2016 at 12:09, Gary Gregory  wrote:
>>
>>> I think we'd need to provide all LogEvent fields.
>>>
>>> Gary
>>>
>>> On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma 
>>> wrote:
>>>
 That's what I meant, I didn't make myself clear. For example, we could
 offer a simple binary layout:
 time stamp (8 bytes)
 level int (4 bytes)
 thread ID (4 bytes) - Thread names in separate file
 Logger ID (4 bytes) - Logger names in separate file.
 message length (4 bytes)
 message type (2 bytes)
 message data (variable length)
 throwable length (4 bytes)
 throwable data (variable length)

 It's a very different approach to logging. On the plus side, this would
 be extremely compact and very fast to write.

 On the other hand it would require a separate tool to decode/display
 the data in human readable form. Such a tool should handle text messages
 out of the box, but for custom messages I image there could be some plugin
 mechanism for custom decoders.

 All very interesting...
 :-)

 Sent from my iPhone

 On 2016/03/03, at 0:01, Matt Sicker  wrote:

 That's what I thought at first, but he means non-human readable formats
 since we all use tools to parse logs as it is (Splunk and ELK are the big
 two I know of).

 On 2 March 2016 at 02:15, Remko Popma  wrote:

> Re: binary logging, I think he's talking about providing an API to log
> objects directly into byte buffers without turning them into Strings 
> first.
>
> https://issues.apache.org/jira/browse/LOG4J2-1274 and
> https://issues.apache.org/jira/browse/LOG4J2-506
>
> were created with that in mind and should be a good step in that
> direction.
>
> Sent from my iPhone
>
> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>
> Well, I've often wondered about creating a binary format but it seems
> that you could use JSON+ZIP or BSON and get most of the advantages.
>
> Gary
>
> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>
>> One other interesting thing I learned is that improper use of logging
>> is a huge source of performance problems. The GC-free parameterized 
>> message
>> factory will help with one aspect of that (I suggested parameterized
>> messages, but he countered with the Object[] that is created), and
>> encouraging users to use a Supplier instead of passing parameters
>> should help as well (especially when those parameters have to be 
>> computed).
>> He had some strong criticisms of logging APIs promoting bad practices 
>> which
>> stems all the way back to log4j1 and affects pretty much every logging 
>> API
>> in Java (some criticisms were actually outdated or didn't consider newer
>> features of the API like markers and the huge amount of filters 
>> available).
>>
>> His other big idea was promoting the use of binary logging formats
>> because humans rarely read the raw log files as it is, but it's not like
>> there's a 

[jira] [Assigned] (LOG4J2-1217) Layout option to limit length of text

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker reassigned LOG4J2-1217:
---

Assignee: Matt Sicker

> Layout option to limit length of text
> -
>
> Key: LOG4J2-1217
> URL: https://issues.apache.org/jira/browse/LOG4J2-1217
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Pattern Converters
>Affects Versions: 2.4.1
>Reporter: Thies Wellpott
>Assignee: Matt Sicker
>
> PatternLayout does not provide the ability to limit the length of the text 
> without space padding (with the numbers directly after the %).
> The following Converter provides this possibility:
> {{%maxLen\{pattern}\{max. length\}}}, e.g. {{%maxLen\{%m}\{90\}}}
> use {{%maxLen}} or {{%maxLength}}
> {code:title=MaxLengthConverter.java}
> package de.ittw.log4j;
> import java.util.List;
> import org.apache.logging.log4j.core.LogEvent;
> import org.apache.logging.log4j.core.appender.AbstractAppender;
> import org.apache.logging.log4j.core.config.Configuration;
> import org.apache.logging.log4j.core.config.plugins.Plugin;
> import org.apache.logging.log4j.core.layout.PatternLayout;
> import org.apache.logging.log4j.core.pattern.ConverterKeys;
> import org.apache.logging.log4j.core.pattern.LogEventPatternConverter;
> import org.apache.logging.log4j.core.pattern.PatternConverter;
> import org.apache.logging.log4j.core.pattern.PatternFormatter;
> import org.apache.logging.log4j.core.pattern.PatternParser;
> /**
>  * Max length pattern converter. Limit contained text to a maximum length.
>  * On invalid length the default value 100 is used (and an error message is 
> logged).
>  * If max length is greater than 20, an abbreviated text will get ellipsis 
> ("...") appended.
>  * Example usage (for email subject):
>  *"%maxLen{[AppName, ${hostName}, ${web:contextPath}] %p: %c{1} - 
> %m%notEmpty{ =>%ex{short}}}{160}"
>  *
>  * @author Thies Wellpott
>  */
> @Plugin(name = "maxLength", category = PatternConverter.CATEGORY)
> @ConverterKeys({ "maxLength", "maxLen" })
> public final class MaxLengthConverter extends LogEventPatternConverter
> {
> /**
>  * Gets an instance of the class.
>  *
>  * @param config
>  *The current Configuration.
>  * @param options
>  *pattern options, an array of two elements: pattern, max 
> length (defaults to 100 on invalid value).
>  * @return instance of class.
>  */
> public static MaxLengthConverter newInstance(final Configuration config, 
> final String[] options) {
> if (options.length != 2) {
> LOGGER.error("Incorrect number of options on maxLength: expected 
> 2 received {}: {}", options.length, options);
> return null;
> }
> if (options[0] == null) {
> LOGGER.error("No pattern supplied on maxLength");
> return null;
> }
> if (options[1] == null) {
> LOGGER.error("No length supplied on maxLength");
> return null;
> }
> final PatternParser parser = 
> PatternLayout.createPatternParser(config);
> final List formatters = parser.parse(options[0]);
> return new MaxLengthConverter(formatters, 
> AbstractAppender.parseInt(options[1], 100));
> }
> private final List formatters;
> private final int maxLength;
> /**
>  * Construct the converter.
>  *
>  * @param formatters
>  *The PatternFormatters to generate the text to manipulate.
>  * @param maxLength
>  *The max. length of the resultung string. Ellipses ("...") 
> is appended on shorted string, if greater than 20.
>  */
> private MaxLengthConverter(final List formatters, final 
> int maxLength) {
> super("MaxLength", "maxLength");
> this.maxLength = maxLength;
> this.formatters = formatters;
> LOGGER.trace("new MaxLengthConverter with {}", maxLength);
> }
> @Override
> public void format(final LogEvent event, final StringBuilder toAppendTo) {
> final StringBuilder buf = new StringBuilder();
> for (final PatternFormatter formatter : formatters) {
> formatter.format(event, buf);
> if (buf.length() > maxLength) {// stop early
> break;
> }
> }
> if (buf.length() > maxLength) {
> buf.setLength(maxLength);
> if (maxLength > 20) {// only append ellipses if length is 
> not very short
> buf.append("...");
> }
> }
> toAppendTo.append(buf);
> }
> }
> {code}
> You may include this in log4j2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, 

[jira] [Commented] (LOG4J2-1216) Nested pattern layout options broken

[Matt Sicker (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177110#comment-15177110
 ] 

Matt Sicker commented on LOG4J2-1216:
-

Do you have a unit test for this?

> Nested pattern layout options broken
> 
>
> Key: LOG4J2-1216
> URL: https://issues.apache.org/jira/browse/LOG4J2-1216
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Pattern Converters
>Affects Versions: 2.4.1
>Reporter: Thies Wellpott
>  Labels: easyfix
>
> Parsing the "deeply" nested PatternLayout
> {code}
> %maxLen{[XXX, ${hostName}, ${web:contextPath}] %p: %c{1} - %m%notEmpty{ 
> =>%ex{short}}}{160}
> {code}
> (with %maxLen being a custom Converter)
> results in wrong parsing.
> Patternparser.extractOptions() gets the nesting wrong.
> Solution:
> {code}
> private static int extractOptions(final String pattern, final int start, 
> final List options) {
> int i = start;
> while (i < pattern.length()  &&  pattern.charAt(i) == '{') {
> i++;  // skip opening "{"
> final int begin = i;  // position of first real char
> int depth = 1;// already inside one level
> while (depth > 0  &&  i < pattern.length()) {
> char c = pattern.charAt(i);
> if (c == '{') {
> depth++;
> } else if (c == '}') {
> depth--;
> // TODO(?) maybe escaping of { and } with \ or %
> }
> i++;
> } // while
> if (depth > 0) {  // option not closed, continue with pattern 
> on opening "{"
> return begin-1;
> }
> options.add(pattern.substring(begin, i-1));
> } // while
> return i;
> }
> {code}
> This should also be faster than the current implementation because the 
> pattern ist only walked once, not multiple times with indexOf().
> (LOG4J2-107 is a similar but old bug)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Closed] (LOG4J2-1306) JeroMqAppender should use ShutdownCallbackRegistry instead of runtime hooks

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker closed LOG4J2-1306.
---
   Resolution: Fixed
Fix Version/s: 2.6

In master.

> JeroMqAppender should use ShutdownCallbackRegistry instead of runtime hooks
> ---
>
> Key: LOG4J2-1306
> URL: https://issues.apache.org/jira/browse/LOG4J2-1306
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Appenders
>Affects Versions: 2.5
>Reporter: Matt Sicker
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> It currently uses {{Runtime.addShutdownHook()}} and should use the more 
> extensible {{ShutdownCallbackRegistry}} provided by {{Log4jContextFactory}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Jenkins build is still unstable: Log4j 2.x #1753

[Matt Sicker]

I honestly didn't think there would be a test verifying an exception for
that scenario.

On 2 March 2016 at 21:12, Gary Gregory  wrote:

> This is a case of a change made without adjusting the unit test methinks.
>
> G
>
> -- Forwarded message --
> From: Apache Jenkins Server 
> Date: Wed, Mar 2, 2016 at 6:59 PM
> Subject: Jenkins build is still unstable: Log4j 2.x #1753
> To: log4j-dev@logging.apache.org
>
>
> See 
>
>
> -
> To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
> For additional commands, e-mail: log4j-dev-h...@logging.apache.org
>
>
>
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>



-- 
Matt Sicker 

[jira] [Created] (LOG4J2-1306) JeroMqAppender should use ShutdownCallbackRegistry instead of runtime hooks

[Matt Sicker (JIRA)]

Matt Sicker created LOG4J2-1306:
---

 Summary: JeroMqAppender should use ShutdownCallbackRegistry 
instead of runtime hooks
 Key: LOG4J2-1306
 URL: https://issues.apache.org/jira/browse/LOG4J2-1306
 Project: Log4j 2
  Issue Type: Improvement
  Components: Appenders
Affects Versions: 2.5
Reporter: Matt Sicker
Assignee: Matt Sicker


It currently uses {{Runtime.addShutdownHook()}} and should use the more 
extensible {{ShutdownCallbackRegistry}} provided by {{Log4jContextFactory}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: logging-log4j2 git commit: LOG4J2-1252 - JeroMqAppender should support layouts

[Matt Sicker]

Good point. Let me see what I can do.

On 2 March 2016 at 21:10, Gary Gregory  wrote:

> No tests? ;-)
>
> Gary
>
> -- Forwarded message --
> From: 
> Date: Wed, Mar 2, 2016 at 6:41 PM
> Subject: logging-log4j2 git commit: LOG4J2-1252 - JeroMqAppender should
> support layouts
> To: comm...@logging.apache.org
>
>
> Repository: logging-log4j2
> Updated Branches:
>   refs/heads/master ebc53bbdc -> 60d1ccd93
>
>
> LOG4J2-1252 - JeroMqAppender should support layouts
>
>
> Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
> Commit:
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/60d1ccd9
> Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/60d1ccd9
> Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/60d1ccd9
>
> Branch: refs/heads/master
> Commit: 60d1ccd9349e4601464d53926595146a59ac4beb
> Parents: ebc53bb
> Author: Matt Sicker 
> Authored: Wed Mar 2 20:42:04 2016 -0600
> Committer: Matt Sicker 
> Committed: Wed Mar 2 20:42:04 2016 -0600
>
> --
>  .../logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java  | 5 +++--
>  src/changes/changes.xml | 3 +++
>  2 files changed, 6 insertions(+), 2 deletions(-)
> --
>
>
>
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/60d1ccd9/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
> --
> diff --git
> a/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
> b/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
> index 974477b..99968e4 100644
> ---
> a/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
> +++
> b/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
> @@ -244,8 +244,9 @@ public final class JeroMqAppender extends
> AbstractAppender {
>
>  @Override
>  public synchronized void append(final LogEvent event) {
> -final String formattedMessage =
> event.getMessage().getFormattedMessage();
> -if (getPublisher().send(formattedMessage, 0)) {
> +final Layout layout = getLayout();
> +final byte[] formattedMessage = layout.toByteArray(event);
> +if (getPublisher().send(getLayout().toByteArray(event))) {
>  sendRcTrue++;
>  } else {
>  sendRcFalse++;
>
>
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/60d1ccd9/src/changes/changes.xml
> --
> diff --git a/src/changes/changes.xml b/src/changes/changes.xml
> index 30f7a67..646a41f 100644
> --- a/src/changes/changes.xml
> +++ b/src/changes/changes.xml
> @@ -181,6 +181,9 @@
>
>  Stop throwing unnecessary exception in
> Log4jServletContextListener.contextDestroyed().
>
> +  
> +JeroMqAppender should support layouts.
> +  
>  
>  
>
>
>
>
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>



-- 
Matt Sicker 

Fwd: Jenkins build is still unstable: Log4j 2.x #1753

[Gary Gregory]

This is a case of a change made without adjusting the unit test methinks.

G

-- Forwarded message --
From: Apache Jenkins Server 
Date: Wed, Mar 2, 2016 at 6:59 PM
Subject: Jenkins build is still unstable: Log4j 2.x #1753
To: log4j-dev@logging.apache.org


See 


-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org




-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: Jenkins build is still unstable: Log4j 2.x #1753

[Matt Sicker]

Fixed.

On 2 March 2016 at 20:59, Apache Jenkins Server 
wrote:

> See 
>
>
> -
> To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
> For additional commands, e-mail: log4j-dev-h...@logging.apache.org
>
>


-- 
Matt Sicker 

Fwd: logging-log4j2 git commit: LOG4J2-1252 - JeroMqAppender should support layouts

[Gary Gregory]

No tests? ;-)

Gary

-- Forwarded message --
From: 
Date: Wed, Mar 2, 2016 at 6:41 PM
Subject: logging-log4j2 git commit: LOG4J2-1252 - JeroMqAppender should
support layouts
To: comm...@logging.apache.org


Repository: logging-log4j2
Updated Branches:
  refs/heads/master ebc53bbdc -> 60d1ccd93


LOG4J2-1252 - JeroMqAppender should support layouts


Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
Commit:
http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/60d1ccd9
Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/60d1ccd9
Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/60d1ccd9

Branch: refs/heads/master
Commit: 60d1ccd9349e4601464d53926595146a59ac4beb
Parents: ebc53bb
Author: Matt Sicker 
Authored: Wed Mar 2 20:42:04 2016 -0600
Committer: Matt Sicker 
Committed: Wed Mar 2 20:42:04 2016 -0600

--
 .../logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java  | 5 +++--
 src/changes/changes.xml | 3 +++
 2 files changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/60d1ccd9/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
--
diff --git
a/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
b/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
index 974477b..99968e4 100644
---
a/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
+++
b/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/jeromq/JeroMqAppender.java
@@ -244,8 +244,9 @@ public final class JeroMqAppender extends
AbstractAppender {

 @Override
 public synchronized void append(final LogEvent event) {
-final String formattedMessage =
event.getMessage().getFormattedMessage();
-if (getPublisher().send(formattedMessage, 0)) {
+final Layout layout = getLayout();
+final byte[] formattedMessage = layout.toByteArray(event);
+if (getPublisher().send(getLayout().toByteArray(event))) {
 sendRcTrue++;
 } else {
 sendRcFalse++;

http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/60d1ccd9/src/changes/changes.xml
--
diff --git a/src/changes/changes.xml b/src/changes/changes.xml
index 30f7a67..646a41f 100644
--- a/src/changes/changes.xml
+++ b/src/changes/changes.xml
@@ -181,6 +181,9 @@
   
 Stop throwing unnecessary exception in
Log4jServletContextListener.contextDestroyed().
   
+  
+JeroMqAppender should support layouts.
+  
 
 
   




-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

[jira] [Resolved] (LOG4J2-1227) NullPointerException in MapLookup.lookup is the event is null

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1227.
-
Resolution: Fixed

I've merged your pull request, thanks for the patch! Please verify and close.

> NullPointerException in MapLookup.lookup is the event is null
> -
>
> Key: LOG4J2-1227
> URL: https://issues.apache.org/jira/browse/LOG4J2-1227
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 2.4, 2.4.1, 2.5
>Reporter: Olivier Lemasle
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> A NullPointerException is thrown in 
> org.apache.logging.log4j.core.lookup.MapLookup.lookup, when the event is null 
> and the map is not null.
> The event can indeed be null, for example if used from StrSubstitutor, with 
> the method replace(final StringBuilder source).
> Exception:
> java.lang.NullPointerException: null
>   at 
> org.apache.logging.log4j.core.lookup.MapLookup.lookup(MapLookup.java:121)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Commented] (LOG4J2-1227) NullPointerException in MapLookup.lookup is the event is null

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177072#comment-15177072
 ] 

ASF GitHub Bot commented on LOG4J2-1227:


Github user asfgit closed the pull request at:

https://github.com/apache/logging-log4j2/pull/20


> NullPointerException in MapLookup.lookup is the event is null
> -
>
> Key: LOG4J2-1227
> URL: https://issues.apache.org/jira/browse/LOG4J2-1227
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 2.4, 2.4.1, 2.5
>Reporter: Olivier Lemasle
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> A NullPointerException is thrown in 
> org.apache.logging.log4j.core.lookup.MapLookup.lookup, when the event is null 
> and the map is not null.
> The event can indeed be null, for example if used from StrSubstitutor, with 
> the method replace(final StringBuilder source).
> Exception:
> java.lang.NullPointerException: null
>   at 
> org.apache.logging.log4j.core.lookup.MapLookup.lookup(MapLookup.java:121)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Assigned] (LOG4J2-1227) NullPointerException in MapLookup.lookup is the event is null

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker reassigned LOG4J2-1227:
---

Assignee: Matt Sicker

> NullPointerException in MapLookup.lookup is the event is null
> -
>
> Key: LOG4J2-1227
> URL: https://issues.apache.org/jira/browse/LOG4J2-1227
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 2.4, 2.4.1, 2.5
>Reporter: Olivier Lemasle
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> A NullPointerException is thrown in 
> org.apache.logging.log4j.core.lookup.MapLookup.lookup, when the event is null 
> and the map is not null.
> The event can indeed be null, for example if used from StrSubstitutor, with 
> the method replace(final StringBuilder source).
> Exception:
> java.lang.NullPointerException: null
>   at 
> org.apache.logging.log4j.core.lookup.MapLookup.lookup(MapLookup.java:121)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Jenkins build is still unstable: Log4j 2.x #1753

[Apache Jenkins Server]

See 


-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Commented] (LOG4J2-1305) Binary Layout

[Matt Sicker (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177053#comment-15177053
 ] 

Matt Sicker commented on LOG4J2-1305:
-

Markers are hierarchical though, that's only a linear representation.

> Binary Layout
> -
>
> Key: LOG4J2-1305
> URL: https://issues.apache.org/jira/browse/LOG4J2-1305
> Project: Log4j 2
>  Issue Type: New Feature
>  Components: Layouts
>Reporter: Remko Popma
>
> Logging in a binary format instead of in text can give large performance 
> improvements. Text-based logging formats are supported by layouts like 
> PatternLayout, and performance investigations like done in LOG4J2-930 suggest 
> that it may be difficult to achieve good performance when logging text. 
> Logging text means going from a LogEvent object to formatted text, and then 
> converting this text to bytes. A different approach would be to convert the 
> LogEvent to a binary representation directly without creating a text 
> representation first. This may make fast synchronous logging possible, 
> although that would additionally require a lock-free appender (LOG4J2-928).
> This proposes a simple BinaryLayout, where each LogEvent is logged in a 
> binary record like this:
> ||Offset||Type||Description||
> |0|long|TimeMillis|
> |8|long|NanoTime|
> |16|int|Level|
> |20|int|Logger name index - string value in separate file|
> |24|int|Thread name index - string value in separate file|
> |28|long|Thread ID|
> |36|short|Marker count|
> |38|int|marker name index - string value in separate file ... (below offset 
> assumes only one marker)|
> |42|int|Message length|
> |46|int|Message type|
> |50|byte[]|Message data - below offset assumes 10 bytes of message data|
> |60|int| Throwable data length|
> |64|byte[]|Throwable data - below offset assumes 10 bytes of Throwable data|
> |74|int|ThreadContext key/value pair count|
> |78|int|ThreadContext key index - string value in separate file|
> |82|int|ThreadContext value index - string value in separate file|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Resolved] (LOG4J2-1252) JeroMqAppender should support layouts

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1252.
-
   Resolution: Fixed
Fix Version/s: 2.6

Fixed in master.

> JeroMqAppender should support layouts
> -
>
> Key: LOG4J2-1252
> URL: https://issues.apache.org/jira/browse/LOG4J2-1252
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Appenders
>Affects Versions: 2.5
>Reporter: Mikael Ståldal
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> JeroMqAppender does not support layouts, even though the documentation says 
> it does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Created] (LOG4J2-1305) Binary Layout

[Remko Popma (JIRA)]

Remko Popma created LOG4J2-1305:
---

 Summary: Binary Layout
 Key: LOG4J2-1305
 URL: https://issues.apache.org/jira/browse/LOG4J2-1305
 Project: Log4j 2
  Issue Type: New Feature
  Components: Layouts
Reporter: Remko Popma


Logging in a binary format instead of in text can give large performance 
improvements. Text-based logging formats are supported by layouts like 
PatternLayout, and performance investigations like done in LOG4J2-930 suggest 
that it may be difficult to achieve good performance when logging text. 

Logging text means going from a LogEvent object to formatted text, and then 
converting this text to bytes. A different approach would be to convert the 
LogEvent to a binary representation directly without creating a text 
representation first. This may make fast synchronous logging possible, although 
that would additionally require a lock-free appender (LOG4J2-928).

This proposes a simple BinaryLayout, where each LogEvent is logged in a binary 
record like this:
||Offset||Type||Description||
|0|long|TimeMillis|
|8|long|NanoTime|
|16|int|Level|
|20|int|Logger name index - string value in separate file|
|24|int|Thread name index - string value in separate file|
|28|long|Thread ID|
|36|short|Marker count|
|38|int|marker name index - string value in separate file ... (below offset 
assumes only one marker)|
|42|int|Message length|
|46|int|Message type|
|50|byte[]|Message data - below offset assumes 10 bytes of message data|
|60|int| Throwable data length|
|64|byte[]|Throwable data - below offset assumes 10 bytes of Throwable data|
|74|int|ThreadContext key/value pair count|
|78|int|ThreadContext key index - string value in separate file|
|82|int|ThreadContext value index - string value in separate file|






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Commented] (LOG4J2-1252) JeroMqAppender should support layouts

[Matt Sicker (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177022#comment-15177022
 ] 

Matt Sicker commented on LOG4J2-1252:
-

You know what's interesting is that there isn't even a server for receiving 
these log messages, so there isn't even a standard format being sent. This is 
pretty simple to fix.

> JeroMqAppender should support layouts
> -
>
> Key: LOG4J2-1252
> URL: https://issues.apache.org/jira/browse/LOG4J2-1252
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Appenders
>Affects Versions: 2.5
>Reporter: Mikael Ståldal
>Assignee: Matt Sicker
>
> JeroMqAppender does not support layouts, even though the documentation says 
> it does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Assigned] (LOG4J2-1252) JeroMqAppender should support layouts

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker reassigned LOG4J2-1252:
---

Assignee: Matt Sicker

> JeroMqAppender should support layouts
> -
>
> Key: LOG4J2-1252
> URL: https://issues.apache.org/jira/browse/LOG4J2-1252
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Appenders
>Affects Versions: 2.5
>Reporter: Mikael Ståldal
>Assignee: Matt Sicker
>
> JeroMqAppender does not support layouts, even though the documentation says 
> it does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Deleted] (LOG4J2-1287) Chnage flow logging text from "entry' to

[Gary Gregory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Gregory deleted LOG4J2-1287:
-


> Chnage flow logging text from "entry' to 
> -
>
> Key: LOG4J2-1287
> URL: https://issues.apache.org/jira/browse/LOG4J2-1287
> Project: Log4j 2
>  Issue Type: Bug
>Reporter: Gary Gregory
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Resolved] (LOG4J2-1262) Log4jServletContextListener unnecessary exception

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1262.
-
   Resolution: Fixed
Fix Version/s: 2.6

Fixed in master. Please verify and close.

> Log4jServletContextListener unnecessary exception
> -
>
> Key: LOG4J2-1262
> URL: https://issues.apache.org/jira/browse/LOG4J2-1262
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Web/Servlet
>Affects Versions: 2.5
>Reporter: Philip Helger
>Assignee: Matt Sicker
> Fix For: 2.6
>
>
> {{Log4jServletContextListener.contextDestroyed}} throws an 
> {{IllegalStateException}} if contextInitialized was never called. In certain 
> rare situations where a server startup fails due to previous errors this 
> Exception is unnecessary.
> So I suggest to now throw an Exception and instead log it as a warning and 
> end the method gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Assigned] (LOG4J2-1262) Log4jServletContextListener unnecessary exception

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker reassigned LOG4J2-1262:
---

Assignee: Matt Sicker

> Log4jServletContextListener unnecessary exception
> -
>
> Key: LOG4J2-1262
> URL: https://issues.apache.org/jira/browse/LOG4J2-1262
> Project: Log4j 2
>  Issue Type: Bug
>  Components: Web/Servlet
>Affects Versions: 2.5
>Reporter: Philip Helger
>Assignee: Matt Sicker
>
> {{Log4jServletContextListener.contextDestroyed}} throws an 
> {{IllegalStateException}} if contextInitialized was never called. In certain 
> rare situations where a server startup fails due to previous errors this 
> Exception is unnecessary.
> So I suggest to now throw an Exception and instead log it as a warning and 
> end the method gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Resolved] (LOG4J2-1275) RollingAppenderNoUnconditionalDeleteTest leaves the rollingtest.log which cause the test to fail when rerunning the test

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1275.
-
   Resolution: Fixed
 Assignee: Matt Sicker
Fix Version/s: 2.6

Thanks for the patch, I can confirm that it also works for me as well on repeat 
test runs. I've pushed it to master. Please verify and close.

> RollingAppenderNoUnconditionalDeleteTest leaves the rollingtest.log which 
> cause the test to fail when rerunning the test
> 
>
> Key: LOG4J2-1275
> URL: https://issues.apache.org/jira/browse/LOG4J2-1275
> Project: Log4j 2
>  Issue Type: Test
>  Components: Core
>Affects Versions: 2.6
>Reporter: Ludovic HOCHET
>Assignee: Matt Sicker
>Priority: Trivial
> Fix For: 2.6
>
> Attachments: patch-log4j2-1275.diff
>
>
> When running mvn install on core after a mvn clean install, the following 
> tests fail:
>   RollingAppenderNoUnconditionalDeleteTest.testAppender:95 rolled over lines 
> expected:<17> but was:<18>
>   RollingAppenderNoUnconditionalDeleteTest.testAppender:95 rolled over lines 
> expected:<17> but was:<18>
>   RollingAppenderNoUnconditionalDeleteTest.testAppender:95 rolled over lines 
> expected:<17> but was:<18>
> This seems to result from the rollingtest.log file being still present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Commented] (LOG4J2-1287) Chnage flow logging text from "entry' to

[Matt Sicker (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176965#comment-15176965
 ] 

Matt Sicker commented on LOG4J2-1287:
-

Hey you still need this bug for something?

> Chnage flow logging text from "entry' to 
> -
>
> Key: LOG4J2-1287
> URL: https://issues.apache.org/jira/browse/LOG4J2-1287
> Project: Log4j 2
>  Issue Type: Bug
>Reporter: Gary Gregory
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Resolved] (LOG4J2-1302) log4j-slf4j-impl should provide a runtime dependency on log4j-core

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1302.
-
Resolution: Information Provided

Well, there is the problem that it's an optional dependency. Though this is 
documented in LOG4J2-1303 now.

> log4j-slf4j-impl should provide a runtime dependency on log4j-core
> --
>
> Key: LOG4J2-1302
> URL: https://issues.apache.org/jira/browse/LOG4J2-1302
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: SLF4J Bridge
>Reporter: Steve Davids
>
> After pulling in the log4j-slf4j-impl I was surprised to find out that I also 
> needed to add the the log4j-core dependency myself instead of the dependency 
> pulled in automatically from the log4j-slf4j-impl pom. I was expecting the 
> behavior provided by slf4j-log4j12 which does pull in the appropriate 
> implementation to get going immediately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Resolved] (LOG4J2-1303) Please provide guidance on libraries necessary for slf4j bindings

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker resolved LOG4J2-1303.
-
Resolution: Fixed

Added the runtime dependencies link to all the component pages to help 
eliminate the confusion. Go ahead and close this issue if this was enough info 
for you.

> Please provide guidance on libraries necessary for slf4j bindings
> -
>
> Key: LOG4J2-1303
> URL: https://issues.apache.org/jira/browse/LOG4J2-1303
> Project: Log4j 2
>  Issue Type: Documentation
>Reporter: Steve Davids
>Assignee: Matt Sicker
>
> The documentation 
> (http://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html) is minimal 
> on the details of what dependencies you should bring in for your SLF4j to 
> properly work. The spring-boot-starter-log4j2 
> (http://central.maven.org/maven2/org/springframework/boot/spring-boot-starter-log4j2/1.3.3.RELEASE/spring-boot-starter-log4j2-1.3.3.RELEASE.pom)
>  provides a decent starting point of what is needed, but it doesn't include 
> the log4j-over-slf4j dependency for the legacy log4j 1.x which makes me 
> wonder if some logging messages are getting dropped?
> Your guidance would be greatly appreciated as to a typical app that would 
> like to use the SLF4j API + bridge support for JCL, JUL, Log4j 1.x + Log4j 
> 2.x SLF4j implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Comment Edited] (LOG4J2-1303) Please provide guidance on libraries necessary for slf4j bindings

[Matt Sicker (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176921#comment-15176921
 ] 

Matt Sicker edited comment on LOG4J2-1303 at 3/3/16 1:36 AM:
-

That's all covered in the [FAQ|http://logging.apache.org/log4j/2.x/faq.html] 
and again in [Runtime 
Dependencies|http://logging.apache.org/log4j/2.x/runtime-dependencies.html], 
but I agree, the component pages should have some better intros.


was (Author: jvz):
That's all covered in the [FAQ|http://logging.apache.org/log4j/2.x/faq.html], 
but I agree, the component pages should have some better intros.

> Please provide guidance on libraries necessary for slf4j bindings
> -
>
> Key: LOG4J2-1303
> URL: https://issues.apache.org/jira/browse/LOG4J2-1303
> Project: Log4j 2
>  Issue Type: Documentation
>Reporter: Steve Davids
>Assignee: Matt Sicker
>
> The documentation 
> (http://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html) is minimal 
> on the details of what dependencies you should bring in for your SLF4j to 
> properly work. The spring-boot-starter-log4j2 
> (http://central.maven.org/maven2/org/springframework/boot/spring-boot-starter-log4j2/1.3.3.RELEASE/spring-boot-starter-log4j2-1.3.3.RELEASE.pom)
>  provides a decent starting point of what is needed, but it doesn't include 
> the log4j-over-slf4j dependency for the legacy log4j 1.x which makes me 
> wonder if some logging messages are getting dropped?
> Your guidance would be greatly appreciated as to a typical app that would 
> like to use the SLF4j API + bridge support for JCL, JUL, Log4j 1.x + Log4j 
> 2.x SLF4j implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Commented] (LOG4J2-1303) Please provide guidance on libraries necessary for slf4j bindings

[Matt Sicker (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176921#comment-15176921
 ] 

Matt Sicker commented on LOG4J2-1303:
-

That's all covered in the [FAQ|http://logging.apache.org/log4j/2.x/faq.html], 
but I agree, the component pages should have some better intros.

> Please provide guidance on libraries necessary for slf4j bindings
> -
>
> Key: LOG4J2-1303
> URL: https://issues.apache.org/jira/browse/LOG4J2-1303
> Project: Log4j 2
>  Issue Type: Documentation
>Reporter: Steve Davids
>
> The documentation 
> (http://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html) is minimal 
> on the details of what dependencies you should bring in for your SLF4j to 
> properly work. The spring-boot-starter-log4j2 
> (http://central.maven.org/maven2/org/springframework/boot/spring-boot-starter-log4j2/1.3.3.RELEASE/spring-boot-starter-log4j2-1.3.3.RELEASE.pom)
>  provides a decent starting point of what is needed, but it doesn't include 
> the log4j-over-slf4j dependency for the legacy log4j 1.x which makes me 
> wonder if some logging messages are getting dropped?
> Your guidance would be greatly appreciated as to a typical app that would 
> like to use the SLF4j API + bridge support for JCL, JUL, Log4j 1.x + Log4j 
> 2.x SLF4j implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Assigned] (LOG4J2-1303) Please provide guidance on libraries necessary for slf4j bindings

[Matt Sicker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Sicker reassigned LOG4J2-1303:
---

Assignee: Matt Sicker

> Please provide guidance on libraries necessary for slf4j bindings
> -
>
> Key: LOG4J2-1303
> URL: https://issues.apache.org/jira/browse/LOG4J2-1303
> Project: Log4j 2
>  Issue Type: Documentation
>Reporter: Steve Davids
>Assignee: Matt Sicker
>
> The documentation 
> (http://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html) is minimal 
> on the details of what dependencies you should bring in for your SLF4j to 
> properly work. The spring-boot-starter-log4j2 
> (http://central.maven.org/maven2/org/springframework/boot/spring-boot-starter-log4j2/1.3.3.RELEASE/spring-boot-starter-log4j2-1.3.3.RELEASE.pom)
>  provides a decent starting point of what is needed, but it doesn't include 
> the log4j-over-slf4j dependency for the legacy log4j 1.x which makes me 
> wonder if some logging messages are getting dropped?
> Your guidance would be greatly appreciated as to a typical app that would 
> like to use the SLF4j API + bridge support for JCL, JUL, Log4j 1.x + Log4j 
> 2.x SLF4j implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Log multiplexing/demultiplexing (was: Fast and Furious)

[Robin Coe]

Thanks for taking it off-topic.  I responded in that thread, because I
thought the input handlers could be extended to consume from multiple
sources and re-integrate a log event.  The processor as I've written it
will consume events from a file but wouldn't integrate a single event from
multiple sources.  I also thought the input stream handlers could be
extended to include binary.  I had thought of using thrift for that but
didn't see the use case.

You've summed it up nicely.  In the simplest case, it's a wrapper, so the
event timestamp, etc., comes from the processor while the message comes
from the stream.  However, given a utf8 serialized event from a compatible
emitter, the log processor will re-materialize the original as much as
possible.  Logj doesn't have an open api that would allow me to do that, so
the event is still a wrapped message, but the i tent is to pull data of
interest from the original and use the mdc as the structure to hold it.
Gives better analysis in elasticsearch, I've found.

In doing this work, I have found certain issues with failover, like that it
won't complete the initialization of a stateful appender if the appender is
offline when logj initializes.  Also had to jump through a few hoops to
avoid unnecessary object creation when creating an event.  Pretty closed
api in places.  But otherwise, it works really well.
On Mar 2, 2016 7:25 PM, "Remko Popma"  wrote:

> (Changing the subject since this should be a separate mail thread)
>
> So if I understand correctly, it's like your app has one or more threads
> that read from a stream, wrap that data in a LogEvent, and then use a log4j
> logger to log the event. So log4j acts as a multiplexer (combining multiple
> inputs into a single output). This allows you to use the existing appenders
> and other log4j functionality for persisting and/or distributing the data.
> (And you can demultiplex by having multiple appenders.)
>
> So essentially this is to solve the problem of log aggregation. This is an
> interesting problem, and currently log4j doesn't have anything in that
> space.
>
> By the way, failover is notoriously hard, and I'm not sure that Log4j's
> current failover mechanism covers all or even most corner cases. It is
> definitely a "best effort, no guarantees" kind of thing.
>
> On Thursday, 3 March 2016, Robin Coe  wrote:
>
>> The scope was to create a bridge for non java apps to get access to
>> log4j2's functionality.  I like the configurable appenders, especially the
>> failover.  I liked the plugin API and that other projects were using it to
>> create custom appenders, like elasticsearch.  I don't like having to go
>> through something like logstash, which throws away events when its buffer
>> fills up, instead prefering to talk to the data sink.  I also like that
>> it's simple to multiplex events to multiple endpoints.  My expectation is
>> that most languages have variants of log4*, so should have appenders that
>> would emit utf8 in some form, allowing me to create something like chainsaw
>> but using streams, not files.
>>
>> There are a lot of streaming processors out there but I couldn't find any
>> that had recoverable failover.  Other solutions have a lot of complexity in
>> their deployments, too, so having a simple runtime that uses existing tech
>> to stream events was what I was looking for.  The stream handlers are also
>> intended to be a public API, so I wanted to provide extensible event
>> handlers that could be used to vet event payloads, for security/data
>> integrity.
>> On Mar 2, 2016 5:52 PM, "Remko Popma"  wrote:
>>
>>> Robin, two questions:
>>> First, what is the problem you're trying to solve?
>>> Second, I'm having trouble understanding how your ideas fit in the
>>> log4j design. Do you intend to feed info *into* log4j (something like
>>> custom messages) or process info *coming out* of log4j (like a custom
>>> appender)? Or both? (But if both, why use log4j?)
>>>
>>> On Thursday, 3 March 2016, Robin Coe  wrote:
>>>
 Idea is a lightweight service that starts TCP listeners that consume
 streams and parses them according to a layout, e.g., syslog, pipe, regex,
 etc.  The configuration is via yaml, whereby a listener is coupled to a
 codec.  The codec is the input stream layout coupled to the output stream
 log4j2 route.

 Simplest use case is taking a stream, say stdout (12factor
 architecture), and coupling that to a log4j2 route.  Other possibilities
 are to consume json and create log event data structures from the
 document.  By extension, any UTF8 stream could be parsed with regex, fields
 of interest extracted and injecged into a log event, and passed to log4j2
 to route.  The log4j2 routes I have set up use a failover strategy, whereby
 upstream sources that go offline cause log events to be written to a local
 file in json format.  On 

Log multiplexing/demultiplexing (was: Fast and Furious)

[Remko Popma]

(Changing the subject since this should be a separate mail thread)

So if I understand correctly, it's like your app has one or more threads
that read from a stream, wrap that data in a LogEvent, and then use a log4j
logger to log the event. So log4j acts as a multiplexer (combining multiple
inputs into a single output). This allows you to use the existing appenders
and other log4j functionality for persisting and/or distributing the data.
(And you can demultiplex by having multiple appenders.)

So essentially this is to solve the problem of log aggregation. This is an
interesting problem, and currently log4j doesn't have anything in that
space.

By the way, failover is notoriously hard, and I'm not sure that Log4j's
current failover mechanism covers all or even most corner cases. It is
definitely a "best effort, no guarantees" kind of thing.

On Thursday, 3 March 2016, Robin Coe  wrote:

> The scope was to create a bridge for non java apps to get access to
> log4j2's functionality.  I like the configurable appenders, especially the
> failover.  I liked the plugin API and that other projects were using it to
> create custom appenders, like elasticsearch.  I don't like having to go
> through something like logstash, which throws away events when its buffer
> fills up, instead prefering to talk to the data sink.  I also like that
> it's simple to multiplex events to multiple endpoints.  My expectation is
> that most languages have variants of log4*, so should have appenders that
> would emit utf8 in some form, allowing me to create something like chainsaw
> but using streams, not files.
>
> There are a lot of streaming processors out there but I couldn't find any
> that had recoverable failover.  Other solutions have a lot of complexity in
> their deployments, too, so having a simple runtime that uses existing tech
> to stream events was what I was looking for.  The stream handlers are also
> intended to be a public API, so I wanted to provide extensible event
> handlers that could be used to vet event payloads, for security/data
> integrity.
> On Mar 2, 2016 5:52 PM, "Remko Popma"  > wrote:
>
>> Robin, two questions:
>> First, what is the problem you're trying to solve?
>> Second, I'm having trouble understanding how your ideas fit in the log4j
>> design. Do you intend to feed info *into* log4j (something like custom
>> messages) or process info *coming out* of log4j (like a custom
>> appender)? Or both? (But if both, why use log4j?)
>>
>> On Thursday, 3 March 2016, Robin Coe > > wrote:
>>
>>> Idea is a lightweight service that starts TCP listeners that consume
>>> streams and parses them according to a layout, e.g., syslog, pipe, regex,
>>> etc.  The configuration is via yaml, whereby a listener is coupled to a
>>> codec.  The codec is the input stream layout coupled to the output stream
>>> log4j2 route.
>>>
>>> Simplest use case is taking a stream, say stdout (12factor
>>> architecture), and coupling that to a log4j2 route.  Other possibilities
>>> are to consume json and create log event data structures from the
>>> document.  By extension, any UTF8 stream could be parsed with regex, fields
>>> of interest extracted and injecged into a log event, and passed to log4j2
>>> to route.  The log4j2 routes I have set up use a failover strategy, whereby
>>> upstream sources that go offline cause log events to be written to a local
>>> file in json format.  On service recovery, I dispatch a worker that
>>> rematerializes those events and sends them upstream.  Recovery begins with
>>> a rollover to a file which uses naming by convention, using the route, the
>>> worker parses the file and sends the events up.  This gives me best-effort
>>> eventual consistency of log events in one or more upstream sinks.
>>> Obviously, this implies a client agent.
>>>
>>> My original architecture was based on Kafka for
>>> consistency/durability/performance but I couldn't guarantee delivery of
>>> events from the emitter to the sink.  When I saw that log4j2 had failover,
>>> I came up with this solution.  I just had to build the healthcheck service
>>> and recovery worker.  My plan was to follow the log4j2 plugin architecture
>>> and use annotations to declare log event handlers, allowing extension to
>>> the log processor.  That part's not done.
>>>
>>>
>>> ​
>>>
>>

Re: Fast and Furious

[Robin Coe]

The scope was to create a bridge for non java apps to get access to
log4j2's functionality.  I like the configurable appenders, especially the
failover.  I liked the plugin API and that other projects were using it to
create custom appenders, like elasticsearch.  I don't like having to go
through something like logstash, which throws away events when its buffer
fills up, instead prefering to talk to the data sink.  I also like that
it's simple to multiplex events to multiple endpoints.  My expectation is
that most languages have variants of log4*, so should have appenders that
would emit utf8 in some form, allowing me to create something like chainsaw
but using streams, not files.

There are a lot of streaming processors out there but I couldn't find any
that had recoverable failover.  Other solutions have a lot of complexity in
their deployments, too, so having a simple runtime that uses existing tech
to stream events was what I was looking for.  The stream handlers are also
intended to be a public API, so I wanted to provide extensible event
handlers that could be used to vet event payloads, for security/data
integrity.
On Mar 2, 2016 5:52 PM, "Remko Popma"  wrote:

> Robin, two questions:
> First, what is the problem you're trying to solve?
> Second, I'm having trouble understanding how your ideas fit in the log4j
> design. Do you intend to feed info *into* log4j (something like custom
> messages) or process info *coming out* of log4j (like a custom appender)?
> Or both? (But if both, why use log4j?)
>
> On Thursday, 3 March 2016, Robin Coe  wrote:
>
>> Idea is a lightweight service that starts TCP listeners that consume
>> streams and parses them according to a layout, e.g., syslog, pipe, regex,
>> etc.  The configuration is via yaml, whereby a listener is coupled to a
>> codec.  The codec is the input stream layout coupled to the output stream
>> log4j2 route.
>>
>> Simplest use case is taking a stream, say stdout (12factor architecture),
>> and coupling that to a log4j2 route.  Other possibilities are to consume
>> json and create log event data structures from the document.  By extension,
>> any UTF8 stream could be parsed with regex, fields of interest extracted
>> and injecged into a log event, and passed to log4j2 to route.  The log4j2
>> routes I have set up use a failover strategy, whereby upstream sources that
>> go offline cause log events to be written to a local file in json format.
>> On service recovery, I dispatch a worker that rematerializes those events
>> and sends them upstream.  Recovery begins with a rollover to a file which
>> uses naming by convention, using the route, the worker parses the file and
>> sends the events up.  This gives me best-effort eventual consistency of log
>> events in one or more upstream sinks.  Obviously, this implies a client
>> agent.
>>
>> My original architecture was based on Kafka for
>> consistency/durability/performance but I couldn't guarantee delivery of
>> events from the emitter to the sink.  When I saw that log4j2 had failover,
>> I came up with this solution.  I just had to build the healthcheck service
>> and recovery worker.  My plan was to follow the log4j2 plugin architecture
>> and use annotations to declare log event handlers, allowing extension to
>> the log processor.  That part's not done.
>>
>>
>> ​
>>
>

[jira] [Commented] (LOG4J2-1301) Automatic reconfiguration with xml and xinclude

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176689#comment-15176689
 ] 

ASF GitHub Bot commented on LOG4J2-1301:


Github user alanachtenberg commented on the pull request:

https://github.com/apache/logging-log4j2/pull/25#issuecomment-191488100
  
On the retrieval of xinclude files for the automatic watching route.

I was trying out this neat little library for xml . 
https://github.com/jOOQ/jOOX 

From their wiki.

>  jOOX stands for Java Object Oriented XML. It is a simple wrapper for the 
org.w3c.dom package, to allow for fluent XML document creation and manipulation 
where DOM is required but too verbose. jOOX only wraps the underlying document 
and can be used to enhance DOM, not as an alternative.

Usage as follows works in my simple test program.
`Document document = $(xmlFile).document();
Match includeElements  = $(document).find("include");
for (Element element : includeElements){
String includeFilePath = element.getAttribute("href");
}`

I can retrieve the include elements because the builder they use does not 
have xinclude enabled by default. Therefore the include elements are not 
processed by the builder. Which made me realize if we wanted to retrieve the 
included files we could parse the file for them before enabling xinclude. It 
would require parsing the file twice(once to get the file names, and once for 
actual configuration) which is not ideal however.


> Automatic reconfiguration with xml and xinclude
> ---
>
> Key: LOG4J2-1301
> URL: https://issues.apache.org/jira/browse/LOG4J2-1301
> Project: Log4j 2
>  Issue Type: Bug
>Affects Versions: 2.3
>Reporter: Alan Achtenberg
> Attachments: GroupFileConfigurationMonitor.java, 
> MyXmlConfiguration.java, MyXmlConfigurationFactory.java
>
>
> Automatic reconfiguration does not work with xinclude/xml.
> Modifying xinclude files is not detected as a file change on on the main 
> config file and as such it is not reconfigured when updates are made to files 
> included with xinclude (ie. log4j2-appenders.xml).
> Desired functionality would be to also listen for changes of files that are 
> included by main configuration file as well and update the configuration as 
> needed.
> An alternative would be to allow user to specify additional files to monitor 
> for changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Fast and Furious

[Remko Popma]

Robin, two questions:
First, what is the problem you're trying to solve?
Second, I'm having trouble understanding how your ideas fit in the log4j
design. Do you intend to feed info *into* log4j (something like custom
messages) or process info *coming out* of log4j (like a custom appender)?
Or both? (But if both, why use log4j?)

On Thursday, 3 March 2016, Robin Coe  wrote:

> Idea is a lightweight service that starts TCP listeners that consume
> streams and parses them according to a layout, e.g., syslog, pipe, regex,
> etc.  The configuration is via yaml, whereby a listener is coupled to a
> codec.  The codec is the input stream layout coupled to the output stream
> log4j2 route.
>
> Simplest use case is taking a stream, say stdout (12factor architecture),
> and coupling that to a log4j2 route.  Other possibilities are to consume
> json and create log event data structures from the document.  By extension,
> any UTF8 stream could be parsed with regex, fields of interest extracted
> and injecged into a log event, and passed to log4j2 to route.  The log4j2
> routes I have set up use a failover strategy, whereby upstream sources that
> go offline cause log events to be written to a local file in json format.
> On service recovery, I dispatch a worker that rematerializes those events
> and sends them upstream.  Recovery begins with a rollover to a file which
> uses naming by convention, using the route, the worker parses the file and
> sends the events up.  This gives me best-effort eventual consistency of log
> events in one or more upstream sinks.  Obviously, this implies a client
> agent.
>
> My original architecture was based on Kafka for
> consistency/durability/performance but I couldn't guarantee delivery of
> events from the emitter to the sink.  When I saw that log4j2 had failover,
> I came up with this solution.  I just had to build the healthcheck service
> and recovery worker.  My plan was to follow the log4j2 plugin architecture
> and use annotations to declare log event handlers, allowing extension to
> the log processor.  That part's not done.
>
>
> ​
>

Re: Fast and Furious

[Remko Popma]

Not Java Serialization, I was thinking simple ByteBuffer.putLong, putInt,
putBytes. This is much more performant (
http://mechanical-sympathy.blogspot.jp/2012/07/native-cc-like-performance-for-java.html).
SBE (Simple Binary Encoding) seems overkill, but open to other opinions.

The less conditional logic in there the better, so I'm not that interested
in configurability. All log event fields, ok. ThreadContextMap/Stack keys
and values: similarly to other repeating strings like logger names: write
to separate mapping file & only write int values (for count as well as
key/value indices) to the "main" log file.

Two things we would need to decide is how to handle versioning, and what
Endianness to use.

Version information (possibly with schema info) could be written to the log
file header.

Endianness could also be written to the header, or we could simply say we
use network byte order (big endian). Intel chips use little endian, but
apparently swapping is implemented with an intrinsic and is very fast.

On Thursday, 3 March 2016, Matt Sicker  wrote:

> At that point, it would be nice if it were extensible. There are some neat
> binary formats we could use that would allow for that.
>
> On 2 March 2016 at 12:09, Gary Gregory  > wrote:
>
>> I think we'd need to provide all LogEvent fields.
>>
>> Gary
>>
>> On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma > > wrote:
>>
>>> That's what I meant, I didn't make myself clear. For example, we could
>>> offer a simple binary layout:
>>> time stamp (8 bytes)
>>> level int (4 bytes)
>>> thread ID (4 bytes) - Thread names in separate file
>>> Logger ID (4 bytes) - Logger names in separate file.
>>> message length (4 bytes)
>>> message type (2 bytes)
>>> message data (variable length)
>>> throwable length (4 bytes)
>>> throwable data (variable length)
>>>
>>> It's a very different approach to logging. On the plus side, this would
>>> be extremely compact and very fast to write.
>>>
>>> On the other hand it would require a separate tool to decode/display the
>>> data in human readable form. Such a tool should handle text messages out of
>>> the box, but for custom messages I image there could be some plugin
>>> mechanism for custom decoders.
>>>
>>> All very interesting...
>>> :-)
>>>
>>> Sent from my iPhone
>>>
>>> On 2016/03/03, at 0:01, Matt Sicker >> > wrote:
>>>
>>> That's what I thought at first, but he means non-human readable formats
>>> since we all use tools to parse logs as it is (Splunk and ELK are the big
>>> two I know of).
>>>
>>> On 2 March 2016 at 02:15, Remko Popma >> > wrote:
>>>
 Re: binary logging, I think he's talking about providing an API to log
 objects directly into byte buffers without turning them into Strings first.

 https://issues.apache.org/jira/browse/LOG4J2-1274 and
 https://issues.apache.org/jira/browse/LOG4J2-506

 were created with that in mind and should be a good step in that
 direction.

 Sent from my iPhone

 On 2016/03/02, at 15:11, Gary Gregory > wrote:

 Well, I've often wondered about creating a binary format but it seems
 that you could use JSON+ZIP or BSON and get most of the advantages.

 Gary

 On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker > wrote:

> One other interesting thing I learned is that improper use of logging
> is a huge source of performance problems. The GC-free parameterized 
> message
> factory will help with one aspect of that (I suggested parameterized
> messages, but he countered with the Object[] that is created), and
> encouraging users to use a Supplier instead of passing parameters
> should help as well (especially when those parameters have to be 
> computed).
> He had some strong criticisms of logging APIs promoting bad practices 
> which
> stems all the way back to log4j1 and affects pretty much every logging API
> in Java (some criticisms were actually outdated or didn't consider newer
> features of the API like markers and the huge amount of filters 
> available).
>
> His other big idea was promoting the use of binary logging formats
> because humans rarely read the raw log files as it is, but it's not like
> there's a standard way to do that.
>
> Now I kinda wonder if he'll find this thread one day and tell me how I
> misinterpreted him or something. ;)
>
> On 1 March 2016 at 22:28, Matt Sicker  

Re: Build failed in Jenkins: Log4j 2.x #1750

[Gary Gregory]

Ah, thank you Ralph then :-)

G

On Wed, Mar 2, 2016 at 1:54 PM, Ralph Goers 
wrote:

> Actually, I fixed it.  The class was extending the class you removed the
> import for.  I guess you didn’t try to run a build before you committed.
>
> Ralph
>
> On Mar 2, 2016, at 2:18 PM, Gary Gregory  wrote:
>
> Thanks to Matt for the fix. I think this was a Javadoc needing the
> import...
>
> Gary
>
> On Wed, Mar 2, 2016 at 5:40 AM, Ralph Goers 
> wrote:
>
>> Gary,  Did you remove more than you should have?
>>
>> Ralph
>>
>> > On Mar 2, 2016, at 2:56 AM, Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>> >
>> > See 
>> >
>> > Changes:
>> >
>> > [ggregory] [LOG4J2-1299] Add pattern converter for thread id and
>> priority in
>> >
>> > [ggregory] Add missing annotations.
>> >
>> > [ggregory] Remove unused imports
>> >
>> > [ggregory] Use final.
>> >
>> > --
>> > [...truncated 3171 lines...]
>> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043
>> sec - in org.apache.logging.log4j.web.Log4jServletFilterTest
>> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19
>> sec - in org.apache.logging.log4j.web.Log4jServletContextListenerTest
>> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec
>> - in org.apache.logging.log4j.web.WebLookupTest
>> > Mar 02, 2016 9:55:30 AM org.springframework.mock.web.MockServletContext
>> log
>> > INFO: This is a test
>> >
>> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec
>> - in org.apache.logging.log4j.web.ServletAppenderTest
>> > ERROR StatusLogger No log4j2 configuration file found. Using default
>> configuration: logging only errors to the console.
>> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.29
>> sec - in org.apache.logging.log4j.web.Log4jWebInitializerImplTest
>> >
>> > Results :
>> >
>> > Tests run: 29, Failures: 0, Errors: 0, Skipped: 0
>> >
>> > [JENKINS] Recording test results
>> > [INFO]
>> > [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ log4j-web ---
>> > [INFO] Building jar: <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-web/target/log4j-web-2.6-SNAPSHOT.jar
>> >
>> > [INFO]
>> > [INFO] --- maven-failsafe-plugin:2.19.1:integration-test
>> (integration-tests) @ log4j-web ---
>> > [JENKINS] Recording test results
>> > [INFO]
>> > [INFO] --- maven-failsafe-plugin:2.19.1:verify (verify) @ log4j-web ---
>> > [JENKINS] Recording test results[INFO]
>> > [INFO] --- maven-source-plugin:3.0.0:jar-no-fork (attach-sources) @
>> log4j-web ---
>> >
>> > [INFO] Building jar: <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-web/target/log4j-web-2.6-SNAPSHOT-sources.jar
>> >
>> > [INFO]
>> > [INFO] --- maven-install-plugin:2.3.1:install (default-install) @
>> log4j-web ---
>> > [INFO] Installing <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-web/target/log4j-web-2.6-SNAPSHOT.jar>
>> to
>> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.jar
>> > [INFO] Installing <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-web/pom.xml> to
>> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.pom
>> > [INFO] Installing <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-web/target/log4j-web-2.6-SNAPSHOT-sources.jar>
>> to
>> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT-sources.jar
>> > [INFO]
>> > [INFO]
>> 
>> > [INFO] Building Apache Log4j Tag Library 2.6-SNAPSHOT
>> > [INFO]
>> 
>> > [INFO]
>> > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
>> log4j-taglib ---
>> > [INFO] Deleting <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-taglib/target>
>> > [INFO]
>> > [INFO] --- maven-remote-resources-plugin:1.1:process (default) @
>> log4j-taglib ---
>> > [INFO]
>> > [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
>> log4j-taglib ---
>> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
>> > [INFO] Copying 1 resource
>> > [INFO] Copying 3 resources
>> > [INFO]
>> > [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @
>> log4j-taglib ---
>> > [INFO] Changes detected - recompiling the module!
>> > [INFO] Compiling 20 source files to <
>> https://builds.apache.org/job/Log4j%202.x/ws/log4j-taglib/target/classes>
>> > [INFO]
>> > [INFO] --- maven-bundle-plugin:2.5.3:manifest (default) @ log4j-taglib
>> ---
>> > [INFO]
>> > [INFO] --- maven-resources-plugin:2.7:testResources
>> (default-testResources) @ log4j-taglib ---
>> > [INFO] Using 'UTF-8' encoding to 

Re: Build failed in Jenkins: Log4j 2.x #1750

[Ralph Goers]

Actually, I fixed it.  The class was extending the class you removed the import 
for.  I guess you didn’t try to run a build before you committed.

Ralph

> On Mar 2, 2016, at 2:18 PM, Gary Gregory  wrote:
> 
> Thanks to Matt for the fix. I think this was a Javadoc needing the import...
> 
> Gary
> 
> On Wed, Mar 2, 2016 at 5:40 AM, Ralph Goers  > wrote:
> Gary,  Did you remove more than you should have?
> 
> Ralph
> 
> > On Mar 2, 2016, at 2:56 AM, Apache Jenkins Server 
> > > wrote:
> >
> > See  > >
> >
> > Changes:
> >
> > [ggregory] [LOG4J2-1299] Add pattern converter for thread id and priority in
> >
> > [ggregory] Add missing annotations.
> >
> > [ggregory] Remove unused imports
> >
> > [ggregory] Use final.
> >
> > --
> > [...truncated 3171 lines...]
> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043 sec - 
> > in org.apache.logging.log4j.web.Log4jServletFilterTest
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19 sec - 
> > in org.apache.logging.log4j.web.Log4jServletContextListenerTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec - 
> > in org.apache.logging.log4j.web.WebLookupTest
> > Mar 02, 2016 9:55:30 AM org.springframework.mock.web.MockServletContext log
> > INFO: This is a test
> >
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec - 
> > in org.apache.logging.log4j.web.ServletAppenderTest
> > ERROR StatusLogger No log4j2 configuration file found. Using default 
> > configuration: logging only errors to the console.
> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.29 sec - 
> > in org.apache.logging.log4j.web.Log4jWebInitializerImplTest
> >
> > Results :
> >
> > Tests run: 29, Failures: 0, Errors: 0, Skipped: 0
> >
> > [JENKINS] Recording test results
> > [INFO]
> > [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ log4j-web ---
> > [INFO] Building jar: 
> >  >  
> > >
> > [INFO]
> > [INFO] --- maven-failsafe-plugin:2.19.1:integration-test 
> > (integration-tests) @ log4j-web ---
> > [JENKINS] Recording test results
> > [INFO]
> > [INFO] --- maven-failsafe-plugin:2.19.1:verify (verify) @ log4j-web ---
> > [JENKINS] Recording test results[INFO]
> > [INFO] --- maven-source-plugin:3.0.0:jar-no-fork (attach-sources) @ 
> > log4j-web ---
> >
> > [INFO] Building jar: 
> >  >  
> > >
> > [INFO]
> > [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ log4j-web 
> > ---
> > [INFO] Installing 
> >  >  
> > >
> >  to 
> > /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.jar
> > [INFO] Installing 
> >  > > to 
> > /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.pom
> > [INFO] Installing 
> >  >  
> > >
> >  to 
> > /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT-sources.jar
> > [INFO]
> > [INFO] 
> > 
> > [INFO] Building Apache Log4j Tag Library 2.6-SNAPSHOT
> > [INFO] 
> > 
> > [INFO]
> > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ log4j-taglib ---
> > [INFO] Deleting 
> >  > >
> > [INFO]
> > [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ 
> > log4j-taglib ---
> > [INFO]
> > [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
> > log4j-taglib ---
> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
> > [INFO] Copying 1 resource
> > [INFO] Copying 3 

Re: Fast and Furious

[Robin Coe]

Idea is a lightweight service that starts TCP listeners that consume
streams and parses them according to a layout, e.g., syslog, pipe, regex,
etc.  The configuration is via yaml, whereby a listener is coupled to a
codec.  The codec is the input stream layout coupled to the output stream
log4j2 route.

Simplest use case is taking a stream, say stdout (12factor architecture),
and coupling that to a log4j2 route.  Other possibilities are to consume
json and create log event data structures from the document.  By extension,
any UTF8 stream could be parsed with regex, fields of interest extracted
and injecged into a log event, and passed to log4j2 to route.  The log4j2
routes I have set up use a failover strategy, whereby upstream sources that
go offline cause log events to be written to a local file in json format.
On service recovery, I dispatch a worker that rematerializes those events
and sends them upstream.  Recovery begins with a rollover to a file which
uses naming by convention, using the route, the worker parses the file and
sends the events up.  This gives me best-effort eventual consistency of log
events in one or more upstream sinks.  Obviously, this implies a client
agent.

My original architecture was based on Kafka for
consistency/durability/performance but I couldn't guarantee delivery of
events from the emitter to the sink.  When I saw that log4j2 had failover,
I came up with this solution.  I just had to build the healthcheck service
and recovery worker.  My plan was to follow the log4j2 plugin architecture
and use annotations to declare log event handlers, allowing extension to
the log processor.  That part's not done.


​

Re: Fast and Furious

[Gary Gregory]

Can you describe a use case please, I having trouble visualizing what you
propose.

Thank you,
Gary
On Mar 2, 2016 11:58 AM, "Robin Coe"  wrote:

> I recently started work on a listener interface to log4j2, intended to
> provide off-stack apps with the features of log4j2.  I intended it to be a
> streaming processor that could pipe messages upstream to log4j2 routes.
> One of the intended behaviours was to be a pre-processor.  Effectively,
> it's like logstash/fluentd but with log4j2 features, like failover (and
> recovery).  It's pretty early in its implementation but might serve the
> purpose you're discussing.  I am hopingto get my company interested in open
> sourcing it.  Would there be interest?
>
> On Wed, Mar 2, 2016 at 1:15 PM, Matt Sicker  wrote:
>
>> At that point, it would be nice if it were extensible. There are some
>> neat binary formats we could use that would allow for that.
>>
>> On 2 March 2016 at 12:09, Gary Gregory  wrote:
>>
>>> I think we'd need to provide all LogEvent fields.
>>>
>>> Gary
>>>
>>> On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma 
>>> wrote:
>>>
 That's what I meant, I didn't make myself clear. For example, we could
 offer a simple binary layout:
 time stamp (8 bytes)
 level int (4 bytes)
 thread ID (4 bytes) - Thread names in separate file
 Logger ID (4 bytes) - Logger names in separate file.
 message length (4 bytes)
 message type (2 bytes)
 message data (variable length)
 throwable length (4 bytes)
 throwable data (variable length)

 It's a very different approach to logging. On the plus side, this would
 be extremely compact and very fast to write.

 On the other hand it would require a separate tool to decode/display
 the data in human readable form. Such a tool should handle text messages
 out of the box, but for custom messages I image there could be some plugin
 mechanism for custom decoders.

 All very interesting...
 :-)

 Sent from my iPhone

 On 2016/03/03, at 0:01, Matt Sicker  wrote:

 That's what I thought at first, but he means non-human readable formats
 since we all use tools to parse logs as it is (Splunk and ELK are the big
 two I know of).

 On 2 March 2016 at 02:15, Remko Popma  wrote:

> Re: binary logging, I think he's talking about providing an API to log
> objects directly into byte buffers without turning them into Strings 
> first.
>
> https://issues.apache.org/jira/browse/LOG4J2-1274 and
> https://issues.apache.org/jira/browse/LOG4J2-506
>
> were created with that in mind and should be a good step in that
> direction.
>
> Sent from my iPhone
>
> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>
> Well, I've often wondered about creating a binary format but it seems
> that you could use JSON+ZIP or BSON and get most of the advantages.
>
> Gary
>
> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>
>> One other interesting thing I learned is that improper use of logging
>> is a huge source of performance problems. The GC-free parameterized 
>> message
>> factory will help with one aspect of that (I suggested parameterized
>> messages, but he countered with the Object[] that is created), and
>> encouraging users to use a Supplier instead of passing parameters
>> should help as well (especially when those parameters have to be 
>> computed).
>> He had some strong criticisms of logging APIs promoting bad practices 
>> which
>> stems all the way back to log4j1 and affects pretty much every logging 
>> API
>> in Java (some criticisms were actually outdated or didn't consider newer
>> features of the API like markers and the huge amount of filters 
>> available).
>>
>> His other big idea was promoting the use of binary logging formats
>> because humans rarely read the raw log files as it is, but it's not like
>> there's a standard way to do that.
>>
>> Now I kinda wonder if he'll find this thread one day and tell me how
>> I misinterpreted him or something. ;)
>>
>> On 1 March 2016 at 22:28, Matt Sicker  wrote:
>>
>>> Alright, I learned some interesting things. I'm going to get us some
>>> tools we can use to try and profile this. Otherwise, he did suggest 
>>> trying
>>> out this project:
>>> https://github.com/RichardWarburton/honest-profiler
>>>
>>> On 1 March 2016 at 19:31, Matt Sicker  wrote:
>>>
 So far he's said something about using lambdas for lazy evaluation
 (though I don't think that would actually help us at all). I'll try to 
 talk
 to him one-on-one afterward 

Re: One Include to rule them all

[Matt Sicker]

I'd probably go with the smallest positive value. The only scenario to
ignore all files would be if none specified a non-zero interval.

On 2 March 2016 at 14:00, Ralph Goers  wrote:

> Yes, I would only check the files all at the same time.  Which value to
> use would most likely follow the rules for merging all duplicate attributes.
>
> Ralph
>
> On Mar 2, 2016, at 12:21 PM, Matt Sicker  wrote:
>
> With a multi-configuration, would the file watcher thread sync up the
> times together? I wouldn't want them to alternate and have to reload the
> configuration more times than necessary because I updated two config files
> at the same time.
>
> On 2 March 2016 at 13:10, Gary Gregory  wrote:
>
>> Which ever way we do it, merge vs. include, the big picture item is that
>> Log4j knows about these files and therefore can watch them and reconfigure
>> itself. I'm OK with either approach. The multiple configuration is simpler
>> in the sense that it does not require new Configuration elements or
>> attributes. I assume that you just list them one after the other in some
>> sys prop. Otherwise, I would not want Log4j hunting all over my classpath
>> for config files and merging them all, that would not be good IMO.
>>
>> Gary
>>
>> On Wed, Mar 2, 2016 at 10:54 AM, Ralph Goers 
>> wrote:
>>
>>> I never really wanted to do includes.  I would prefer to support
>>> multiple configuration files that are merged - see LOG4J2-494.  I view the
>>> XInclude for XML files as a special case.
>>>
>>> If I did want to support includes I would not want to allow a
>>> monitorInterval on the include element. The value on the configuration
>>> should be used.  I have no idea what it would mean to have a
>>> monitorInterval of 0 on the main configuration and a non-zero value on an
>>> include. Likewise, having a main monitorInterval of 60 and an interval of
>>> 30 on an include also doesn’t seem right.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Mar 2, 2016, at 11:39 AM, Gary Gregory 
>>> wrote:
>>>
>>> Stemming from discussion in
>>> https://github.com/apache/logging-log4j2/pull/25
>>>
>>> How about finally adding our own include mechanism:
>>>
>>> file://...
>>>
>>> If Configuration has a monitorInterval, then Includes inherit the
>>> setting, if you set an Include monitorInterval to 0 then, then it is
>>> not watched.
>>>
>>> Thoughts?
>>>
>>> Gary
>>>
>>> --
>>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>>> Java Persistence with Hibernate, Second Edition
>>> 
>>> JUnit in Action, Second Edition 
>>> Spring Batch in Action 
>>> Blog: http://garygregory.wordpress.com
>>> Home: http://garygregory.com/
>>> Tweet! http://twitter.com/GaryGregory
>>>
>>>
>>>
>>
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> 
>> JUnit in Action, Second Edition 
>> Spring Batch in Action 
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>
>
>
> --
> Matt Sicker 
>
>
>


-- 
Matt Sicker 

Re: One Include to rule them all

[Ralph Goers]

Yes, I would only check the files all at the same time.  Which value to use 
would most likely follow the rules for merging all duplicate attributes.

Ralph

> On Mar 2, 2016, at 12:21 PM, Matt Sicker  wrote:
> 
> With a multi-configuration, would the file watcher thread sync up the times 
> together? I wouldn't want them to alternate and have to reload the 
> configuration more times than necessary because I updated two config files at 
> the same time.
> 
> On 2 March 2016 at 13:10, Gary Gregory  > wrote:
> Which ever way we do it, merge vs. include, the big picture item is that 
> Log4j knows about these files and therefore can watch them and reconfigure 
> itself. I'm OK with either approach. The multiple configuration is simpler in 
> the sense that it does not require new Configuration elements or attributes. 
> I assume that you just list them one after the other in some sys prop. 
> Otherwise, I would not want Log4j hunting all over my classpath for config 
> files and merging them all, that would not be good IMO.
> 
> Gary
> 
> On Wed, Mar 2, 2016 at 10:54 AM, Ralph Goers  > wrote:
> I never really wanted to do includes.  I would prefer to support multiple 
> configuration files that are merged - see LOG4J2-494.  I view the XInclude 
> for XML files as a special case.
> 
> If I did want to support includes I would not want to allow a monitorInterval 
> on the include element. The value on the configuration should be used.  I 
> have no idea what it would mean to have a monitorInterval of 0 on the main 
> configuration and a non-zero value on an include. Likewise, having a main 
> monitorInterval of 60 and an interval of 30 on an include also doesn’t seem 
> right.
> 
> Ralph
> 
> 
> 
>> On Mar 2, 2016, at 11:39 AM, Gary Gregory > > wrote:
>> 
>> Stemming from discussion in https://github.com/apache/logging-log4j2/pull/25 
>> 
>> 
>> How about finally adding our own include mechanism:
>> 
>> file://...
>> 
>> If Configuration has a monitorInterval, then Includes inherit the setting, 
>> if you set an Include monitorInterval to 0 then, then it is not watched.
>> 
>> Thoughts?
>> 
>> Gary
>> 
>> -- 
>> E-Mail: garydgreg...@gmail.com  | 
>> ggreg...@apache.org  
>> Java Persistence with Hibernate, Second Edition 
>> 
>> JUnit in Action, Second Edition 
>> Spring Batch in Action 
>> Blog: http://garygregory.wordpress.com  
>> Home: http://garygregory.com/ 
>> Tweet! http://twitter.com/GaryGregory 
> 
> 
> 
> -- 
> E-Mail: garydgreg...@gmail.com  | 
> ggreg...@apache.org  
> Java Persistence with Hibernate, Second Edition 
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com  
> Home: http://garygregory.com/ 
> Tweet! http://twitter.com/GaryGregory 
> 
> 
> -- 
> Matt Sicker >

Re: Fast and Furious

[Robin Coe]

I recently started work on a listener interface to log4j2, intended to
provide off-stack apps with the features of log4j2.  I intended it to be a
streaming processor that could pipe messages upstream to log4j2 routes.
One of the intended behaviours was to be a pre-processor.  Effectively,
it's like logstash/fluentd but with log4j2 features, like failover (and
recovery).  It's pretty early in its implementation but might serve the
purpose you're discussing.  I am hopingto get my company interested in open
sourcing it.  Would there be interest?

On Wed, Mar 2, 2016 at 1:15 PM, Matt Sicker  wrote:

> At that point, it would be nice if it were extensible. There are some neat
> binary formats we could use that would allow for that.
>
> On 2 March 2016 at 12:09, Gary Gregory  wrote:
>
>> I think we'd need to provide all LogEvent fields.
>>
>> Gary
>>
>> On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma 
>> wrote:
>>
>>> That's what I meant, I didn't make myself clear. For example, we could
>>> offer a simple binary layout:
>>> time stamp (8 bytes)
>>> level int (4 bytes)
>>> thread ID (4 bytes) - Thread names in separate file
>>> Logger ID (4 bytes) - Logger names in separate file.
>>> message length (4 bytes)
>>> message type (2 bytes)
>>> message data (variable length)
>>> throwable length (4 bytes)
>>> throwable data (variable length)
>>>
>>> It's a very different approach to logging. On the plus side, this would
>>> be extremely compact and very fast to write.
>>>
>>> On the other hand it would require a separate tool to decode/display the
>>> data in human readable form. Such a tool should handle text messages out of
>>> the box, but for custom messages I image there could be some plugin
>>> mechanism for custom decoders.
>>>
>>> All very interesting...
>>> :-)
>>>
>>> Sent from my iPhone
>>>
>>> On 2016/03/03, at 0:01, Matt Sicker  wrote:
>>>
>>> That's what I thought at first, but he means non-human readable formats
>>> since we all use tools to parse logs as it is (Splunk and ELK are the big
>>> two I know of).
>>>
>>> On 2 March 2016 at 02:15, Remko Popma  wrote:
>>>
 Re: binary logging, I think he's talking about providing an API to log
 objects directly into byte buffers without turning them into Strings first.

 https://issues.apache.org/jira/browse/LOG4J2-1274 and
 https://issues.apache.org/jira/browse/LOG4J2-506

 were created with that in mind and should be a good step in that
 direction.

 Sent from my iPhone

 On 2016/03/02, at 15:11, Gary Gregory  wrote:

 Well, I've often wondered about creating a binary format but it seems
 that you could use JSON+ZIP or BSON and get most of the advantages.

 Gary

 On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:

> One other interesting thing I learned is that improper use of logging
> is a huge source of performance problems. The GC-free parameterized 
> message
> factory will help with one aspect of that (I suggested parameterized
> messages, but he countered with the Object[] that is created), and
> encouraging users to use a Supplier instead of passing parameters
> should help as well (especially when those parameters have to be 
> computed).
> He had some strong criticisms of logging APIs promoting bad practices 
> which
> stems all the way back to log4j1 and affects pretty much every logging API
> in Java (some criticisms were actually outdated or didn't consider newer
> features of the API like markers and the huge amount of filters 
> available).
>
> His other big idea was promoting the use of binary logging formats
> because humans rarely read the raw log files as it is, but it's not like
> there's a standard way to do that.
>
> Now I kinda wonder if he'll find this thread one day and tell me how I
> misinterpreted him or something. ;)
>
> On 1 March 2016 at 22:28, Matt Sicker  wrote:
>
>> Alright, I learned some interesting things. I'm going to get us some
>> tools we can use to try and profile this. Otherwise, he did suggest 
>> trying
>> out this project:
>> https://github.com/RichardWarburton/honest-profiler
>>
>> On 1 March 2016 at 19:31, Matt Sicker  wrote:
>>
>>> So far he's said something about using lambdas for lazy evaluation
>>> (though I don't think that would actually help us at all). I'll try to 
>>> talk
>>> to him one-on-one afterward to delve more into this.
>>>
>>> On 1 March 2016 at 18:13, Ralph Goers 
>>> wrote:
>>>
 Actually, most of the tests have the commands in the comments right
 in the class. Just cut and past.

 Ralph

 On Mar 1, 

Re: One Include to rule them all

[Matt Sicker]

With a multi-configuration, would the file watcher thread sync up the times
together? I wouldn't want them to alternate and have to reload the
configuration more times than necessary because I updated two config files
at the same time.

On 2 March 2016 at 13:10, Gary Gregory  wrote:

> Which ever way we do it, merge vs. include, the big picture item is that
> Log4j knows about these files and therefore can watch them and reconfigure
> itself. I'm OK with either approach. The multiple configuration is simpler
> in the sense that it does not require new Configuration elements or
> attributes. I assume that you just list them one after the other in some
> sys prop. Otherwise, I would not want Log4j hunting all over my classpath
> for config files and merging them all, that would not be good IMO.
>
> Gary
>
> On Wed, Mar 2, 2016 at 10:54 AM, Ralph Goers 
> wrote:
>
>> I never really wanted to do includes.  I would prefer to support multiple
>> configuration files that are merged - see LOG4J2-494.  I view the XInclude
>> for XML files as a special case.
>>
>> If I did want to support includes I would not want to allow a
>> monitorInterval on the include element. The value on the configuration
>> should be used.  I have no idea what it would mean to have a
>> monitorInterval of 0 on the main configuration and a non-zero value on an
>> include. Likewise, having a main monitorInterval of 60 and an interval of
>> 30 on an include also doesn’t seem right.
>>
>> Ralph
>>
>>
>>
>> On Mar 2, 2016, at 11:39 AM, Gary Gregory  wrote:
>>
>> Stemming from discussion in
>> https://github.com/apache/logging-log4j2/pull/25
>>
>> How about finally adding our own include mechanism:
>>
>> file://...
>>
>> If Configuration has a monitorInterval, then Includes inherit the
>> setting, if you set an Include monitorInterval to 0 then, then it is not
>> watched.
>>
>> Thoughts?
>>
>> Gary
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> 
>> JUnit in Action, Second Edition 
>> Spring Batch in Action 
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>>
>>
>
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>



-- 
Matt Sicker 

Re: One Include to rule them all

[Paul Benedict]

You're right that XInclude only works with XML. Perhaps this is why Ralph
said it's a "special case". It's a pretty powerful standard feature; yet it
won't fit the bill if you want to watch files. Sorry, I know no such way of
knowing what files are included through XInclude because it's meant to be
transparent. Don't take my word for it but I have yet to find a way (or
need a way).

Cheers,
Paul

On Wed, Mar 2, 2016 at 1:06 PM, Gary Gregory  wrote:

> So, with XInclude, there is no way to ask for "What files are you really
> included when processing this document" unless you get in the guts of the
> private Xerces copy which is in Oracle Java 7 (but not in Java 8), which we
> do not want to do. Also XInclude does not work with JSON and YAML.
>
> G
>
> On Wed, Mar 2, 2016 at 10:52 AM, Paul Benedict 
> wrote:
>
>> I agree with the initial assessment that XInclude is the way to go. If
>> someone wants to use XInclude, they should provide the right parser
>> implementation to make it available. I wouldn't bend over backward to do
>> your own inclusion mechanism; put it on the consumer to provide the
>> appropriate parser.
>>
>> Cheers,
>> Paul
>>
>> On Wed, Mar 2, 2016 at 12:39 PM, Gary Gregory 
>> wrote:
>>
>>> Stemming from discussion in
>>> https://github.com/apache/logging-log4j2/pull/25
>>>
>>> How about finally adding our own include mechanism:
>>>
>>> file://...
>>>
>>> If Configuration has a monitorInterval, then Includes inherit the
>>> setting, if you set an Include monitorInterval to 0 then, then it is
>>> not watched.
>>>
>>> Thoughts?
>>>
>>> Gary
>>>
>>> --
>>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>>> Java Persistence with Hibernate, Second Edition
>>> 
>>> JUnit in Action, Second Edition 
>>> Spring Batch in Action 
>>> Blog: http://garygregory.wordpress.com
>>> Home: http://garygregory.com/
>>> Tweet! http://twitter.com/GaryGregory
>>>
>>
>>
>
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>

Re: One Include to rule them all

[Gary Gregory]

Which ever way we do it, merge vs. include, the big picture item is that
Log4j knows about these files and therefore can watch them and reconfigure
itself. I'm OK with either approach. The multiple configuration is simpler
in the sense that it does not require new Configuration elements or
attributes. I assume that you just list them one after the other in some
sys prop. Otherwise, I would not want Log4j hunting all over my classpath
for config files and merging them all, that would not be good IMO.

Gary

On Wed, Mar 2, 2016 at 10:54 AM, Ralph Goers 
wrote:

> I never really wanted to do includes.  I would prefer to support multiple
> configuration files that are merged - see LOG4J2-494.  I view the XInclude
> for XML files as a special case.
>
> If I did want to support includes I would not want to allow a
> monitorInterval on the include element. The value on the configuration
> should be used.  I have no idea what it would mean to have a
> monitorInterval of 0 on the main configuration and a non-zero value on an
> include. Likewise, having a main monitorInterval of 60 and an interval of
> 30 on an include also doesn’t seem right.
>
> Ralph
>
>
>
> On Mar 2, 2016, at 11:39 AM, Gary Gregory  wrote:
>
> Stemming from discussion in
> https://github.com/apache/logging-log4j2/pull/25
>
> How about finally adding our own include mechanism:
>
> file://...
>
> If Configuration has a monitorInterval, then Includes inherit the
> setting, if you set an Include monitorInterval to 0 then, then it is not
> watched.
>
> Thoughts?
>
> Gary
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>
>
>


-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: One Include to rule them all

[Gary Gregory]

So, with XInclude, there is no way to ask for "What files are you really
included when processing this document" unless you get in the guts of the
private Xerces copy which is in Oracle Java 7 (but not in Java 8), which we
do not want to do. Also XInclude does not work with JSON and YAML.

G

On Wed, Mar 2, 2016 at 10:52 AM, Paul Benedict  wrote:

> I agree with the initial assessment that XInclude is the way to go. If
> someone wants to use XInclude, they should provide the right parser
> implementation to make it available. I wouldn't bend over backward to do
> your own inclusion mechanism; put it on the consumer to provide the
> appropriate parser.
>
> Cheers,
> Paul
>
> On Wed, Mar 2, 2016 at 12:39 PM, Gary Gregory 
> wrote:
>
>> Stemming from discussion in
>> https://github.com/apache/logging-log4j2/pull/25
>>
>> How about finally adding our own include mechanism:
>>
>> file://...
>>
>> If Configuration has a monitorInterval, then Includes inherit the
>> setting, if you set an Include monitorInterval to 0 then, then it is not
>> watched.
>>
>> Thoughts?
>>
>> Gary
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> 
>> JUnit in Action, Second Edition 
>> Spring Batch in Action 
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>
>


-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: One Include to rule them all

[Ralph Goers]

I never really wanted to do includes.  I would prefer to support multiple 
configuration files that are merged - see LOG4J2-494.  I view the XInclude for 
XML files as a special case.

If I did want to support includes I would not want to allow a monitorInterval 
on the include element. The value on the configuration should be used.  I have 
no idea what it would mean to have a monitorInterval of 0 on the main 
configuration and a non-zero value on an include. Likewise, having a main 
monitorInterval of 60 and an interval of 30 on an include also doesn’t seem 
right.

Ralph



> On Mar 2, 2016, at 11:39 AM, Gary Gregory  wrote:
> 
> Stemming from discussion in https://github.com/apache/logging-log4j2/pull/25 
> 
> 
> How about finally adding our own include mechanism:
> 
> file://...
> 
> If Configuration has a monitorInterval, then Includes inherit the setting, if 
> you set an Include monitorInterval to 0 then, then it is not watched.
> 
> Thoughts?
> 
> Gary
> 
> -- 
> E-Mail: garydgreg...@gmail.com  | 
> ggreg...@apache.org  
> Java Persistence with Hibernate, Second Edition 
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com  
> Home: http://garygregory.com/ 
> Tweet! http://twitter.com/GaryGregory 

Re: One Include to rule them all

[Paul Benedict]

PS: Are we talking the same thing? Document inclusion is NOT the same as
watching an external file. The former creates one seamless document; the
second one is aware that the referenced file is distinct.

Cheers,
Paul

On Wed, Mar 2, 2016 at 12:52 PM, Paul Benedict  wrote:

> I agree with the initial assessment that XInclude is the way to go. If
> someone wants to use XInclude, they should provide the right parser
> implementation to make it available. I wouldn't bend over backward to do
> your own inclusion mechanism; put it on the consumer to provide the
> appropriate parser.
>
> Cheers,
> Paul
>
> On Wed, Mar 2, 2016 at 12:39 PM, Gary Gregory 
> wrote:
>
>> Stemming from discussion in
>> https://github.com/apache/logging-log4j2/pull/25
>>
>> How about finally adding our own include mechanism:
>>
>> file://...
>>
>> If Configuration has a monitorInterval, then Includes inherit the
>> setting, if you set an Include monitorInterval to 0 then, then it is not
>> watched.
>>
>> Thoughts?
>>
>> Gary
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> 
>> JUnit in Action, Second Edition 
>> Spring Batch in Action 
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>
>

Re: One Include to rule them all

[Paul Benedict]

I agree with the initial assessment that XInclude is the way to go. If
someone wants to use XInclude, they should provide the right parser
implementation to make it available. I wouldn't bend over backward to do
your own inclusion mechanism; put it on the consumer to provide the
appropriate parser.

Cheers,
Paul

On Wed, Mar 2, 2016 at 12:39 PM, Gary Gregory 
wrote:

> Stemming from discussion in
> https://github.com/apache/logging-log4j2/pull/25
>
> How about finally adding our own include mechanism:
>
> file://...
>
> If Configuration has a monitorInterval, then Includes inherit the
> setting, if you set an Include monitorInterval to 0 then, then it is not
> watched.
>
> Thoughts?
>
> Gary
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> 
> JUnit in Action, Second Edition 
> Spring Batch in Action 
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>

One Include to rule them all

[Gary Gregory]

Stemming from discussion in https://github.com/apache/logging-log4j2/pull/25

How about finally adding our own include mechanism:

file://...

If Configuration has a monitorInterval, then Includes inherit the setting,
if you set an Include monitorInterval to 0 then, then it is not watched.

Thoughts?

Gary

-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

[jira] [Commented] (LOG4J2-1301) Automatic reconfiguration with xml and xinclude

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176196#comment-15176196
 ] 

ASF GitHub Bot commented on LOG4J2-1301:


Github user garydgregory commented on the pull request:

https://github.com/apache/logging-log4j2/pull/25#issuecomment-191363016
  
We are on Java 7 for now so we cannot use Java 8 features unless we do so 
in a way is not used on Java 7.

The bigger picture item is: If we invented a Log4j include mechanism that 
works across all of our configuration formats (XML, JSON, YAML), then we would 
not need a new watchFiles attribute for this used case because we would know 
which files to watch. We could even enable/disable watching on a per-include 
basis:

file://...

This monitorIntervalSeconds would override the value set on the 
Configuration element.

Sorry, I am getting side tracked... I need to bring this up on the dev ML.


> Automatic reconfiguration with xml and xinclude
> ---
>
> Key: LOG4J2-1301
> URL: https://issues.apache.org/jira/browse/LOG4J2-1301
> Project: Log4j 2
>  Issue Type: Bug
>Affects Versions: 2.3
>Reporter: Alan Achtenberg
> Attachments: GroupFileConfigurationMonitor.java, 
> MyXmlConfiguration.java, MyXmlConfigurationFactory.java
>
>
> Automatic reconfiguration does not work with xinclude/xml.
> Modifying xinclude files is not detected as a file change on on the main 
> config file and as such it is not reconfigured when updates are made to files 
> included with xinclude (ie. log4j2-appenders.xml).
> Desired functionality would be to also listen for changes of files that are 
> included by main configuration file as well and update the configuration as 
> needed.
> An alternative would be to allow user to specify additional files to monitor 
> for changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Fast and Furious

[Matt Sicker]

At that point, it would be nice if it were extensible. There are some neat
binary formats we could use that would allow for that.

On 2 March 2016 at 12:09, Gary Gregory  wrote:

> I think we'd need to provide all LogEvent fields.
>
> Gary
>
> On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma  wrote:
>
>> That's what I meant, I didn't make myself clear. For example, we could
>> offer a simple binary layout:
>> time stamp (8 bytes)
>> level int (4 bytes)
>> thread ID (4 bytes) - Thread names in separate file
>> Logger ID (4 bytes) - Logger names in separate file.
>> message length (4 bytes)
>> message type (2 bytes)
>> message data (variable length)
>> throwable length (4 bytes)
>> throwable data (variable length)
>>
>> It's a very different approach to logging. On the plus side, this would
>> be extremely compact and very fast to write.
>>
>> On the other hand it would require a separate tool to decode/display the
>> data in human readable form. Such a tool should handle text messages out of
>> the box, but for custom messages I image there could be some plugin
>> mechanism for custom decoders.
>>
>> All very interesting...
>> :-)
>>
>> Sent from my iPhone
>>
>> On 2016/03/03, at 0:01, Matt Sicker  wrote:
>>
>> That's what I thought at first, but he means non-human readable formats
>> since we all use tools to parse logs as it is (Splunk and ELK are the big
>> two I know of).
>>
>> On 2 March 2016 at 02:15, Remko Popma  wrote:
>>
>>> Re: binary logging, I think he's talking about providing an API to log
>>> objects directly into byte buffers without turning them into Strings first.
>>>
>>> https://issues.apache.org/jira/browse/LOG4J2-1274 and
>>> https://issues.apache.org/jira/browse/LOG4J2-506
>>>
>>> were created with that in mind and should be a good step in that
>>> direction.
>>>
>>> Sent from my iPhone
>>>
>>> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>>>
>>> Well, I've often wondered about creating a binary format but it seems
>>> that you could use JSON+ZIP or BSON and get most of the advantages.
>>>
>>> Gary
>>>
>>> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>>>
 One other interesting thing I learned is that improper use of logging
 is a huge source of performance problems. The GC-free parameterized message
 factory will help with one aspect of that (I suggested parameterized
 messages, but he countered with the Object[] that is created), and
 encouraging users to use a Supplier instead of passing parameters
 should help as well (especially when those parameters have to be computed).
 He had some strong criticisms of logging APIs promoting bad practices which
 stems all the way back to log4j1 and affects pretty much every logging API
 in Java (some criticisms were actually outdated or didn't consider newer
 features of the API like markers and the huge amount of filters available).

 His other big idea was promoting the use of binary logging formats
 because humans rarely read the raw log files as it is, but it's not like
 there's a standard way to do that.

 Now I kinda wonder if he'll find this thread one day and tell me how I
 misinterpreted him or something. ;)

 On 1 March 2016 at 22:28, Matt Sicker  wrote:

> Alright, I learned some interesting things. I'm going to get us some
> tools we can use to try and profile this. Otherwise, he did suggest trying
> out this project:
> https://github.com/RichardWarburton/honest-profiler
>
> On 1 March 2016 at 19:31, Matt Sicker  wrote:
>
>> So far he's said something about using lambdas for lazy evaluation
>> (though I don't think that would actually help us at all). I'll try to 
>> talk
>> to him one-on-one afterward to delve more into this.
>>
>> On 1 March 2016 at 18:13, Ralph Goers 
>> wrote:
>>
>>> Actually, most of the tests have the commands in the comments right
>>> in the class. Just cut and past.
>>>
>>> Ralph
>>>
>>> On Mar 1, 2016, at 1:43 PM, Matt Sicker  wrote:
>>>
>>> I can't even figure out how to execute the simple perf test class.
>>> IntelliJ gives me some annotation processing error, and doing it from 
>>> the
>>> command line is turning into a classpath nightmare to figure out what 
>>> jars
>>> are needed to execute the test manually.
>>>
>>> On 1 March 2016 at 11:34, Gary Gregory 
>>> wrote:
>>>
 Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you
 available after the preso to talk about some issue we are seeing?

 Gary
 On Mar 1, 2016 8:29 AM, "Matt Sicker"  wrote:

> I'm attending a JUG meetup tonight 

Re: Fast and Furious

[Gary Gregory]

I think we'd need to provide all LogEvent fields.

Gary

On Wed, Mar 2, 2016 at 9:13 AM, Remko Popma  wrote:

> That's what I meant, I didn't make myself clear. For example, we could
> offer a simple binary layout:
> time stamp (8 bytes)
> level int (4 bytes)
> thread ID (4 bytes) - Thread names in separate file
> Logger ID (4 bytes) - Logger names in separate file.
> message length (4 bytes)
> message type (2 bytes)
> message data (variable length)
> throwable length (4 bytes)
> throwable data (variable length)
>
> It's a very different approach to logging. On the plus side, this would be
> extremely compact and very fast to write.
>
> On the other hand it would require a separate tool to decode/display the
> data in human readable form. Such a tool should handle text messages out of
> the box, but for custom messages I image there could be some plugin
> mechanism for custom decoders.
>
> All very interesting...
> :-)
>
> Sent from my iPhone
>
> On 2016/03/03, at 0:01, Matt Sicker  wrote:
>
> That's what I thought at first, but he means non-human readable formats
> since we all use tools to parse logs as it is (Splunk and ELK are the big
> two I know of).
>
> On 2 March 2016 at 02:15, Remko Popma  wrote:
>
>> Re: binary logging, I think he's talking about providing an API to log
>> objects directly into byte buffers without turning them into Strings first.
>>
>> https://issues.apache.org/jira/browse/LOG4J2-1274 and
>> https://issues.apache.org/jira/browse/LOG4J2-506
>>
>> were created with that in mind and should be a good step in that
>> direction.
>>
>> Sent from my iPhone
>>
>> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>>
>> Well, I've often wondered about creating a binary format but it seems
>> that you could use JSON+ZIP or BSON and get most of the advantages.
>>
>> Gary
>>
>> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>>
>>> One other interesting thing I learned is that improper use of logging is
>>> a huge source of performance problems. The GC-free parameterized message
>>> factory will help with one aspect of that (I suggested parameterized
>>> messages, but he countered with the Object[] that is created), and
>>> encouraging users to use a Supplier instead of passing parameters
>>> should help as well (especially when those parameters have to be computed).
>>> He had some strong criticisms of logging APIs promoting bad practices which
>>> stems all the way back to log4j1 and affects pretty much every logging API
>>> in Java (some criticisms were actually outdated or didn't consider newer
>>> features of the API like markers and the huge amount of filters available).
>>>
>>> His other big idea was promoting the use of binary logging formats
>>> because humans rarely read the raw log files as it is, but it's not like
>>> there's a standard way to do that.
>>>
>>> Now I kinda wonder if he'll find this thread one day and tell me how I
>>> misinterpreted him or something. ;)
>>>
>>> On 1 March 2016 at 22:28, Matt Sicker  wrote:
>>>
 Alright, I learned some interesting things. I'm going to get us some
 tools we can use to try and profile this. Otherwise, he did suggest trying
 out this project:
 https://github.com/RichardWarburton/honest-profiler

 On 1 March 2016 at 19:31, Matt Sicker  wrote:

> So far he's said something about using lambdas for lazy evaluation
> (though I don't think that would actually help us at all). I'll try to 
> talk
> to him one-on-one afterward to delve more into this.
>
> On 1 March 2016 at 18:13, Ralph Goers 
> wrote:
>
>> Actually, most of the tests have the commands in the comments right
>> in the class. Just cut and past.
>>
>> Ralph
>>
>> On Mar 1, 2016, at 1:43 PM, Matt Sicker  wrote:
>>
>> I can't even figure out how to execute the simple perf test class.
>> IntelliJ gives me some annotation processing error, and doing it from the
>> command line is turning into a classpath nightmare to figure out what 
>> jars
>> are needed to execute the test manually.
>>
>> On 1 March 2016 at 11:34, Gary Gregory 
>> wrote:
>>
>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you
>>> available after the preso to talk about some issue we are seeing?
>>>
>>> Gary
>>> On Mar 1, 2016 8:29 AM, "Matt Sicker"  wrote:
>>>
 I'm attending a JUG meetup tonight with Kirk Pepperdine presenting.
 It's supposed to be a Java performance workshop type of thing, so if 
 you've
 got a decent way to ask about it, I could see if he can help figure out
 this regression. I can at least show off the SimplePerfTest and any
 microbenchmarks we have.

[jira] [Commented] (LOG4J2-1301) Automatic reconfiguration with xml and xinclude

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LOG4J2-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176083#comment-15176083
 ] 

ASF GitHub Bot commented on LOG4J2-1301:


Github user alanachtenberg commented on the pull request:

https://github.com/apache/logging-log4j2/pull/25#issuecomment-191348199
  
I agree it would be nice to automatically pick up xincluded files; It would 
good if it was a "just works" solution for the end user without having to 
select files to monitor, but that solution would not be very portable to the 
other file formats. If specifying a list of files is deemed acceptable this 
would be easy to reproduce for json and probably YAML(I have no experience with 
it).

Also being able to specify the files gives the user alot more power. In my 
case what if I only want to monitor the log4j-xinclude-loggers.xml(changed 
often) for changes and not the appenders(changed rarely, usually with a new 
release if at all) to reduce overhead with the checks. Secondly a user could 
enable and disable monitoring of files at anytime since the main configuration 
will always be monitored. Lastly I think there could be some good use cases for 
monitoring non log4j2 files for changes (Especially when you consider 
extensions or plugins of log4j2) . For example an application that uses a 
single generic configuration/properties file(client accessible) for application 
wide configuration could set some properties to be used in the logging system 
initialization.

I am a happy camper either way if i can get reconfiguration based on the 
xincluded files :)

Thanks for the feedback and let me know if you find anything with regards 
to a "java 8 way". I looked into where the xinclude is processed and it seemed 
it would be difficult to retrieve the files without hacking up the xml code. 
However, I have relatively little experience and you might see a better 
solution than me.

-Alan


> Automatic reconfiguration with xml and xinclude
> ---
>
> Key: LOG4J2-1301
> URL: https://issues.apache.org/jira/browse/LOG4J2-1301
> Project: Log4j 2
>  Issue Type: Bug
>Affects Versions: 2.3
>Reporter: Alan Achtenberg
> Attachments: GroupFileConfigurationMonitor.java, 
> MyXmlConfiguration.java, MyXmlConfigurationFactory.java
>
>
> Automatic reconfiguration does not work with xinclude/xml.
> Modifying xinclude files is not detected as a file change on on the main 
> config file and as such it is not reconfigured when updates are made to files 
> included with xinclude (ie. log4j2-appenders.xml).
> Desired functionality would be to also listen for changes of files that are 
> included by main configuration file as well and update the configuration as 
> needed.
> An alternative would be to allow user to specify additional files to monitor 
> for changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Fast and Furious

[Matt Sicker]

I like the idea. Would this be Java-specific due to serialization? Also, we
could provide a basic tool to decode the format and see about getting other
projects to support it.

On 2 March 2016 at 11:13, Remko Popma  wrote:

> That's what I meant, I didn't make myself clear. For example, we could
> offer a simple binary layout:
> time stamp (8 bytes)
> level int (4 bytes)
> thread ID (4 bytes) - Thread names in separate file
> Logger ID (4 bytes) - Logger names in separate file.
> message length (4 bytes)
> message type (2 bytes)
> message data (variable length)
> throwable length (4 bytes)
> throwable data (variable length)
>
> It's a very different approach to logging. On the plus side, this would be
> extremely compact and very fast to write.
>
> On the other hand it would require a separate tool to decode/display the
> data in human readable form. Such a tool should handle text messages out of
> the box, but for custom messages I image there could be some plugin
> mechanism for custom decoders.
>
> All very interesting...
> :-)
>
> Sent from my iPhone
>
> On 2016/03/03, at 0:01, Matt Sicker  wrote:
>
> That's what I thought at first, but he means non-human readable formats
> since we all use tools to parse logs as it is (Splunk and ELK are the big
> two I know of).
>
> On 2 March 2016 at 02:15, Remko Popma  wrote:
>
>> Re: binary logging, I think he's talking about providing an API to log
>> objects directly into byte buffers without turning them into Strings first.
>>
>> https://issues.apache.org/jira/browse/LOG4J2-1274 and
>> https://issues.apache.org/jira/browse/LOG4J2-506
>>
>> were created with that in mind and should be a good step in that
>> direction.
>>
>> Sent from my iPhone
>>
>> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>>
>> Well, I've often wondered about creating a binary format but it seems
>> that you could use JSON+ZIP or BSON and get most of the advantages.
>>
>> Gary
>>
>> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>>
>>> One other interesting thing I learned is that improper use of logging is
>>> a huge source of performance problems. The GC-free parameterized message
>>> factory will help with one aspect of that (I suggested parameterized
>>> messages, but he countered with the Object[] that is created), and
>>> encouraging users to use a Supplier instead of passing parameters
>>> should help as well (especially when those parameters have to be computed).
>>> He had some strong criticisms of logging APIs promoting bad practices which
>>> stems all the way back to log4j1 and affects pretty much every logging API
>>> in Java (some criticisms were actually outdated or didn't consider newer
>>> features of the API like markers and the huge amount of filters available).
>>>
>>> His other big idea was promoting the use of binary logging formats
>>> because humans rarely read the raw log files as it is, but it's not like
>>> there's a standard way to do that.
>>>
>>> Now I kinda wonder if he'll find this thread one day and tell me how I
>>> misinterpreted him or something. ;)
>>>
>>> On 1 March 2016 at 22:28, Matt Sicker  wrote:
>>>
 Alright, I learned some interesting things. I'm going to get us some
 tools we can use to try and profile this. Otherwise, he did suggest trying
 out this project:
 https://github.com/RichardWarburton/honest-profiler

 On 1 March 2016 at 19:31, Matt Sicker  wrote:

> So far he's said something about using lambdas for lazy evaluation
> (though I don't think that would actually help us at all). I'll try to 
> talk
> to him one-on-one afterward to delve more into this.
>
> On 1 March 2016 at 18:13, Ralph Goers 
> wrote:
>
>> Actually, most of the tests have the commands in the comments right
>> in the class. Just cut and past.
>>
>> Ralph
>>
>> On Mar 1, 2016, at 1:43 PM, Matt Sicker  wrote:
>>
>> I can't even figure out how to execute the simple perf test class.
>> IntelliJ gives me some annotation processing error, and doing it from the
>> command line is turning into a classpath nightmare to figure out what 
>> jars
>> are needed to execute the test manually.
>>
>> On 1 March 2016 at 11:34, Gary Gregory 
>> wrote:
>>
>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you
>>> available after the preso to talk about some issue we are seeing?
>>>
>>> Gary
>>> On Mar 1, 2016 8:29 AM, "Matt Sicker"  wrote:
>>>
 I'm attending a JUG meetup tonight with Kirk Pepperdine presenting.
 It's supposed to be a Java performance workshop type of thing, so if 
 you've
 got a decent way to ask about it, I could see if he can help figure out
 

Re: Fast and Furious

[Remko Popma]

That's what I meant, I didn't make myself clear. For example, we could offer a 
simple binary layout:
time stamp (8 bytes)
level int (4 bytes)
thread ID (4 bytes) - Thread names in separate file
Logger ID (4 bytes) - Logger names in separate file. 
message length (4 bytes)
message type (2 bytes)
message data (variable length)
throwable length (4 bytes)
throwable data (variable length)

It's a very different approach to logging. On the plus side, this would be 
extremely compact and very fast to write. 

On the other hand it would require a separate tool to decode/display the data 
in human readable form. Such a tool should handle text messages out of the box, 
but for custom messages I image there could be some plugin mechanism for custom 
decoders. 

All very interesting...
:-)

Sent from my iPhone

> On 2016/03/03, at 0:01, Matt Sicker  wrote:
> 
> That's what I thought at first, but he means non-human readable formats since 
> we all use tools to parse logs as it is (Splunk and ELK are the big two I 
> know of).
> 
>> On 2 March 2016 at 02:15, Remko Popma  wrote:
>> Re: binary logging, I think he's talking about providing an API to log 
>> objects directly into byte buffers without turning them into Strings first. 
>> 
>> https://issues.apache.org/jira/browse/LOG4J2-1274 and 
>> https://issues.apache.org/jira/browse/LOG4J2-506
>> 
>> were created with that in mind and should be a good step in that direction. 
>> 
>> Sent from my iPhone
>> 
>>> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>>> 
>>> Well, I've often wondered about creating a binary format but it seems that 
>>> you could use JSON+ZIP or BSON and get most of the advantages.
>>> 
>>> Gary
>>> 
 On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
 One other interesting thing I learned is that improper use of logging is a 
 huge source of performance problems. The GC-free parameterized message 
 factory will help with one aspect of that (I suggested parameterized 
 messages, but he countered with the Object[] that is created), and 
 encouraging users to use a Supplier instead of passing parameters 
 should help as well (especially when those parameters have to be 
 computed). He had some strong criticisms of logging APIs promoting bad 
 practices which stems all the way back to log4j1 and affects pretty much 
 every logging API in Java (some criticisms were actually outdated or 
 didn't consider newer features of the API like markers and the huge amount 
 of filters available).
 
 His other big idea was promoting the use of binary logging formats because 
 humans rarely read the raw log files as it is, but it's not like there's a 
 standard way to do that.
 
 Now I kinda wonder if he'll find this thread one day and tell me how I 
 misinterpreted him or something. ;)
 
> On 1 March 2016 at 22:28, Matt Sicker  wrote:
> Alright, I learned some interesting things. I'm going to get us some 
> tools we can use to try and profile this. Otherwise, he did suggest 
> trying out this project:
> https://github.com/RichardWarburton/honest-profiler
> 
>> On 1 March 2016 at 19:31, Matt Sicker  wrote:
>> So far he's said something about using lambdas for lazy evaluation 
>> (though I don't think that would actually help us at all). I'll try to 
>> talk to him one-on-one afterward to delve more into this.
>> 
>>> On 1 March 2016 at 18:13, Ralph Goers  
>>> wrote:
>>> Actually, most of the tests have the commands in the comments right in 
>>> the class. Just cut and past.
>>> 
>>> Ralph
>>> 
 On Mar 1, 2016, at 1:43 PM, Matt Sicker  wrote:
 
 I can't even figure out how to execute the simple perf test class. 
 IntelliJ gives me some annotation processing error, and doing it from 
 the command line is turning into a classpath nightmare to figure out 
 what jars are needed to execute the test manually.
 
 On 1 March 2016 at 11:34, Gary Gregory  wrote:
> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you 
> available after the preso to talk about some issue we are seeing?
> 
> Gary
> 
>> On Mar 1, 2016 8:29 AM, "Matt Sicker"  wrote:
>> I'm attending a JUG meetup tonight with Kirk Pepperdine presenting. 
>> It's supposed to be a Java performance workshop type of thing, so if 
>> you've got a decent way to ask about it, I could see if he can help 
>> figure out this regression. I can at least show off the 
>> SimplePerfTest and any microbenchmarks we have.
>> 
>>> On 28 February 2016 at 11:54, Matt Sicker  wrote:

Re: Jenkins build is unstable: Log4j 2.x #1752

[Matt Sicker]

Also, since the test deletes the directory afterward, we can't even
download the bz2 file to see if it's valid.

On 2 March 2016 at 11:03, Matt Sicker  wrote:

> Just the bzip2 test failing again.
>
> On 2 March 2016 at 11:01, Apache Jenkins Server  > wrote:
>
>> See 
>>
>>
>> -
>> To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
>> For additional commands, e-mail: log4j-dev-h...@logging.apache.org
>>
>>
>
>
> --
> Matt Sicker 
>



-- 
Matt Sicker 

Re: Jenkins build is unstable: Log4j 2.x #1752

[Matt Sicker]

Just the bzip2 test failing again.

On 2 March 2016 at 11:01, Apache Jenkins Server 
wrote:

> See 
>
>
> -
> To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
> For additional commands, e-mail: log4j-dev-h...@logging.apache.org
>
>


-- 
Matt Sicker 

Jenkins build is unstable: Log4j 2.x #1752

[Apache Jenkins Server]

See 


-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Can't build log4j from IntelliJ, any ideas?

[Matt Sicker]

It worked for me after I restarted IntelliJ. Might have been a cache issue
or something. Hopefully I don't get that error again.

On 2 March 2016 at 03:01, Mikael Ståldal  wrote:

> It works fine for me with IntelliJ IDEA 15.0.4 community on Linux, latest
> from master.
>
> On Tue, Mar 1, 2016 at 9:35 PM, Matt Sicker  wrote:
>
>> If I try to execute any unit tests (or even "make" the project) I get the
>> following error:
>>
>> Error:java: Bad service configuration file, or exception thrown while
>> constructing Processor object: javax.annotation.processing.Processor:
>> Provider
>> org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor
>> could not be instantiated: java.lang.NoClassDefFoundError:
>> org/apache/logging/log4j/core/config/plugins/processor/PluginCache
>>
>> Everything works fine in Maven. I enabled annotation processing, yet I
>> get this error every time in log4j-core.
>>
>> --
>> Matt Sicker 
>>
>
>
>
> --
> [image: MagineTV]
>
> *Mikael Ståldal*
> Senior software developer
>
> *Magine TV*
> mikael.stal...@magine.com
> Grev Turegatan 3  | 114 46 Stockholm, Sweden  |   www.magine.com
>
> Privileged and/or Confidential Information may be contained in this
> message. If you are not the addressee indicated in this message
> (or responsible for delivery of the message to such a person), you may not
> copy or deliver this message to anyone. In such case,
> you should destroy this message and kindly notify the sender by reply
> email.
>



-- 
Matt Sicker 

Re: Fast and Furious

[Matt Sicker]

That's what I thought at first, but he means non-human readable formats
since we all use tools to parse logs as it is (Splunk and ELK are the big
two I know of).

On 2 March 2016 at 02:15, Remko Popma  wrote:

> Re: binary logging, I think he's talking about providing an API to log
> objects directly into byte buffers without turning them into Strings first.
>
> https://issues.apache.org/jira/browse/LOG4J2-1274 and
> https://issues.apache.org/jira/browse/LOG4J2-506
>
> were created with that in mind and should be a good step in that
> direction.
>
> Sent from my iPhone
>
> On 2016/03/02, at 15:11, Gary Gregory  wrote:
>
> Well, I've often wondered about creating a binary format but it seems that
> you could use JSON+ZIP or BSON and get most of the advantages.
>
> Gary
>
> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>
>> One other interesting thing I learned is that improper use of logging is
>> a huge source of performance problems. The GC-free parameterized message
>> factory will help with one aspect of that (I suggested parameterized
>> messages, but he countered with the Object[] that is created), and
>> encouraging users to use a Supplier instead of passing parameters
>> should help as well (especially when those parameters have to be computed).
>> He had some strong criticisms of logging APIs promoting bad practices which
>> stems all the way back to log4j1 and affects pretty much every logging API
>> in Java (some criticisms were actually outdated or didn't consider newer
>> features of the API like markers and the huge amount of filters available).
>>
>> His other big idea was promoting the use of binary logging formats
>> because humans rarely read the raw log files as it is, but it's not like
>> there's a standard way to do that.
>>
>> Now I kinda wonder if he'll find this thread one day and tell me how I
>> misinterpreted him or something. ;)
>>
>> On 1 March 2016 at 22:28, Matt Sicker  wrote:
>>
>>> Alright, I learned some interesting things. I'm going to get us some
>>> tools we can use to try and profile this. Otherwise, he did suggest trying
>>> out this project:
>>> https://github.com/RichardWarburton/honest-profiler
>>>
>>> On 1 March 2016 at 19:31, Matt Sicker  wrote:
>>>
 So far he's said something about using lambdas for lazy evaluation
 (though I don't think that would actually help us at all). I'll try to talk
 to him one-on-one afterward to delve more into this.

 On 1 March 2016 at 18:13, Ralph Goers 
 wrote:

> Actually, most of the tests have the commands in the comments right in
> the class. Just cut and past.
>
> Ralph
>
> On Mar 1, 2016, at 1:43 PM, Matt Sicker  wrote:
>
> I can't even figure out how to execute the simple perf test class.
> IntelliJ gives me some annotation processing error, and doing it from the
> command line is turning into a classpath nightmare to figure out what jars
> are needed to execute the test manually.
>
> On 1 March 2016 at 11:34, Gary Gregory  wrote:
>
>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you
>> available after the preso to talk about some issue we are seeing?
>>
>> Gary
>> On Mar 1, 2016 8:29 AM, "Matt Sicker"  wrote:
>>
>>> I'm attending a JUG meetup tonight with Kirk Pepperdine presenting.
>>> It's supposed to be a Java performance workshop type of thing, so if 
>>> you've
>>> got a decent way to ask about it, I could see if he can help figure out
>>> this regression. I can at least show off the SimplePerfTest and any
>>> microbenchmarks we have.
>>>
>>> On 28 February 2016 at 11:54, Matt Sicker  wrote:
>>>
 Take a look at the git bisect command. Might help you find which
 changes caused the problem.


 On Sunday, 28 February 2016, Gary Gregory 
 wrote:

> Thank you for digging in Remko. This is will be a nice theme to
> publicize when you get it figured out.
>
> Gary
> On Feb 28, 2016 4:08 AM, "Remko Popma" 
> wrote:
>
>> After removing the potential impact of appenders and layouts by
>> testing with 
>> log4j-core\src\test\resources\perf-CountingNoOpAppender.xml
>> and org.apache.logging.log4j.core.async.perftest.SimplePerfTest, I've
>> confirmed my initial numbers:
>>
>> 2.0: 7.5M ops/sec
>> 2.1: 6M ops/sec
>> 2.2: 6M ops/sec
>> 2.3: 6M ops/sec
>> 2.4: 4.5M ops/sec
>> 2.5: 4M ops/sec
>> 2.6: 2M ops/sec
>>
>> I tried reverting various changes made to AsyncLogger since 2.0,
>> 

DefaultRolloverStrategy; reducing max backup index

[Greg Thomas]

We're using a RollingFile appender with the DefaultRolloverStrategy;
we've noticed that if we reduce the max backup index (e.g. from
max="10" to max="5") after files have been created then we end up with
some files that look like they are current, but never get deleted,
i.e.

test.log
test.1.log
test.2.log
test.3.log
test.4.log
test.5.log
^^^ These files are rolled, according to their size
test.6.log
test.7.log
test.8.log
test.9.log
test.10.log
^^^ These files remain forever, which leads to confusion.

Is this
a) A bug,
b) An undocumented, but otherwise desirable feature, or
c) An undesirable feature that ideally should be fixed
?

Thanks,

Greg

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: Build failed in Jenkins: Log4j 2.x #1750

[Ralph Goers]

Gary,  Did you remove more than you should have?

Ralph

> On Mar 2, 2016, at 2:56 AM, Apache Jenkins Server  
> wrote:
> 
> See 
> 
> Changes:
> 
> [ggregory] [LOG4J2-1299] Add pattern converter for thread id and priority in
> 
> [ggregory] Add missing annotations.
> 
> [ggregory] Remove unused imports
> 
> [ggregory] Use final.
> 
> --
> [...truncated 3171 lines...]
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043 sec - 
> in org.apache.logging.log4j.web.Log4jServletFilterTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19 sec - in 
> org.apache.logging.log4j.web.Log4jServletContextListenerTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec - in 
> org.apache.logging.log4j.web.WebLookupTest
> Mar 02, 2016 9:55:30 AM org.springframework.mock.web.MockServletContext log
> INFO: This is a test
> 
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec - in 
> org.apache.logging.log4j.web.ServletAppenderTest
> ERROR StatusLogger No log4j2 configuration file found. Using default 
> configuration: logging only errors to the console.
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.29 sec - 
> in org.apache.logging.log4j.web.Log4jWebInitializerImplTest
> 
> Results :
> 
> Tests run: 29, Failures: 0, Errors: 0, Skipped: 0
> 
> [JENKINS] Recording test results
> [INFO] 
> [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ log4j-web ---
> [INFO] Building jar: 
> 
> [INFO] 
> [INFO] --- maven-failsafe-plugin:2.19.1:integration-test (integration-tests) 
> @ log4j-web ---
> [JENKINS] Recording test results
> [INFO] 
> [INFO] --- maven-failsafe-plugin:2.19.1:verify (verify) @ log4j-web ---
> [JENKINS] Recording test results[INFO] 
> [INFO] --- maven-source-plugin:3.0.0:jar-no-fork (attach-sources) @ log4j-web 
> ---
> 
> [INFO] Building jar: 
> 
> [INFO] 
> [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ log4j-web 
> ---
> [INFO] Installing 
> 
>  to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.jar
> [INFO] Installing 
>  to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.pom
> [INFO] Installing 
> 
>  to 
> /home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT-sources.jar
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Apache Log4j Tag Library 2.6-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ log4j-taglib ---
> [INFO] Deleting 
> 
> [INFO] 
> [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ log4j-taglib 
> ---
> [INFO] 
> [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
> log4j-taglib ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 1 resource
> [INFO] Copying 3 resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ 
> log4j-taglib ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 20 source files to 
> 
> [INFO] 
> [INFO] --- maven-bundle-plugin:2.5.3:manifest (default) @ log4j-taglib ---
> [INFO] 
> [INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
> log4j-taglib ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 1 resource
> [INFO] Copying 3 resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ 
> log4j-taglib ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 13 source files to 
> 
> [INFO] 
> [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ log4j-taglib ---
> 
> ---
> T E S T S
> ---
> Running org.apache.logging.log4j.taglib.TagUtilsScopeTest
> Tests run: 5, Failures: 0, Errors: 0, 

Re: Can't build log4j from IntelliJ, any ideas?

[Mikael Ståldal]

It works fine for me with IntelliJ IDEA 15.0.4 community on Linux, latest
from master.

On Tue, Mar 1, 2016 at 9:35 PM, Matt Sicker  wrote:

> If I try to execute any unit tests (or even "make" the project) I get the
> following error:
>
> Error:java: Bad service configuration file, or exception thrown while
> constructing Processor object: javax.annotation.processing.Processor:
> Provider
> org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor
> could not be instantiated: java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/core/config/plugins/processor/PluginCache
>
> Everything works fine in Maven. I enabled annotation processing, yet I get
> this error every time in log4j-core.
>
> --
> Matt Sicker 
>



-- 
[image: MagineTV]

*Mikael Ståldal*
Senior software developer

*Magine TV*
mikael.stal...@magine.com
Grev Turegatan 3  | 114 46 Stockholm, Sweden  |   www.magine.com

Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply
email.

Build failed in Jenkins: Log4j 2.x #1751

[Apache Jenkins Server]

See 

Changes:

[ggregory] [LOG4J2-1304] Update Jackson from 2.7.0 to 2.7.2.

--
[...truncated 3169 lines...]
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.271 sec - in 
org.apache.logging.log4j.web.Log4jServletContainerInitializerTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.366 sec - in 
org.apache.logging.log4j.web.Log4jServletFilterTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.532 sec - in 
org.apache.logging.log4j.web.Log4jServletContextListenerTest
ERROR StatusLogger No log4j2 configuration file found. Using default 
configuration: logging only errors to the console.
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.983 sec - in 
org.apache.logging.log4j.web.Log4jWebInitializerImplTest
Mar 02, 2016 10:14:48 AM org.springframework.mock.web.MockServletContext log
INFO: This is a test

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.997 sec - in 
org.apache.logging.log4j.web.ServletAppenderTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.019 sec - in 
org.apache.logging.log4j.web.WebLookupTest

Results :

Tests run: 29, Failures: 0, Errors: 0, Skipped: 0

[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ log4j-web ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.19.1:integration-test (integration-tests) @ 
log4j-web ---
[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-failsafe-plugin:2.19.1:verify (verify) @ log4j-web ---
[JENKINS] Recording test results[INFO] 
[INFO] --- maven-source-plugin:3.0.0:jar-no-fork (attach-sources) @ log4j-web 
---
[INFO] Building jar: 


[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ log4j-web ---
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.jar
[INFO] Installing 
 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT-sources.jar
[INFO] 
[INFO] 
[INFO] Building Apache Log4j Tag Library 2.6-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ log4j-taglib ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.1:process (default) @ log4j-taglib 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
log4j-taglib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ log4j-taglib 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 20 source files to 

[INFO] 
[INFO] --- maven-bundle-plugin:2.5.3:manifest (default) @ log4j-taglib ---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
log4j-taglib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ 
log4j-taglib ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 13 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ log4j-taglib ---

---
 T E S T S
---
Running org.apache.logging.log4j.taglib.CatchingTagTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.015 sec - in 
org.apache.logging.log4j.taglib.CatchingTagTest
Running org.apache.logging.log4j.taglib.SetLoggerTagTest
ERROR StatusLogger No log4j2 configuration file found. Using default 
configuration: logging only errors to the console.
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.654 

Build failed in Jenkins: Log4j 2.x #1750

[Apache Jenkins Server]

See 

Changes:

[ggregory] [LOG4J2-1299] Add pattern converter for thread id and priority in

[ggregory] Add missing annotations.

[ggregory] Remove unused imports

[ggregory] Use final.

--
[...truncated 3171 lines...]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043 sec - in 
org.apache.logging.log4j.web.Log4jServletFilterTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19 sec - in 
org.apache.logging.log4j.web.Log4jServletContextListenerTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec - in 
org.apache.logging.log4j.web.WebLookupTest
Mar 02, 2016 9:55:30 AM org.springframework.mock.web.MockServletContext log
INFO: This is a test

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.5 sec - in 
org.apache.logging.log4j.web.ServletAppenderTest
ERROR StatusLogger No log4j2 configuration file found. Using default 
configuration: logging only errors to the console.
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.29 sec - in 
org.apache.logging.log4j.web.Log4jWebInitializerImplTest

Results :

Tests run: 29, Failures: 0, Errors: 0, Skipped: 0

[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ log4j-web ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.19.1:integration-test (integration-tests) @ 
log4j-web ---
[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-failsafe-plugin:2.19.1:verify (verify) @ log4j-web ---
[JENKINS] Recording test results[INFO] 
[INFO] --- maven-source-plugin:3.0.0:jar-no-fork (attach-sources) @ log4j-web 
---

[INFO] Building jar: 

[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ log4j-web ---
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.jar
[INFO] Installing 
 to 
/home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/1/org/apache/logging/log4j/log4j-web/2.6-SNAPSHOT/log4j-web-2.6-SNAPSHOT-sources.jar
[INFO] 
[INFO] 
[INFO] Building Apache Log4j Tag Library 2.6-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ log4j-taglib ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.1:process (default) @ log4j-taglib 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
log4j-taglib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ log4j-taglib 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 20 source files to 

[INFO] 
[INFO] --- maven-bundle-plugin:2.5.3:manifest (default) @ log4j-taglib ---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
log4j-taglib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ 
log4j-taglib ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 13 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ log4j-taglib ---

---
 T E S T S
---
Running org.apache.logging.log4j.taglib.TagUtilsScopeTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.291 sec - in 
org.apache.logging.log4j.taglib.TagUtilsScopeTest
Running org.apache.logging.log4j.taglib.LoggingMessageTagSupportTest
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.535 sec - in 
org.apache.logging.log4j.taglib.LoggingMessageTagSupportTest
Running org.apache.logging.log4j.taglib.CatchingTagTest
Tests run: 3, 

[jira] [Closed] (LOG4J2-1304) Update Jackson from 2.7.0 to 2.7.2

[Gary Gregory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Gregory closed LOG4J2-1304.

Resolution: Fixed

In Git master.

> Update Jackson from 2.7.0 to 2.7.2
> --
>
> Key: LOG4J2-1304
> URL: https://issues.apache.org/jira/browse/LOG4J2-1304
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Layouts
>Affects Versions: 2.5
>Reporter: Gary Gregory
>Assignee: Gary Gregory
> Fix For: 2.6
>
>
> Update Jackson from 2.6.0 to 2.7.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Updated] (LOG4J2-1304) Update Jackson from 2.7.0 to 2.7.2

[Gary Gregory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Gregory updated LOG4J2-1304:
-
Summary: Update Jackson from 2.7.0 to 2.7.2  (was: Update Jackson from 
2.6.0 to 2.7.7)

> Update Jackson from 2.7.0 to 2.7.2
> --
>
> Key: LOG4J2-1304
> URL: https://issues.apache.org/jira/browse/LOG4J2-1304
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Layouts
>Affects Versions: 2.5
>Reporter: Gary Gregory
>Assignee: Gary Gregory
> Fix For: 2.6
>
>
> Update Jackson from 2.6.0 to 2.7.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Created] (LOG4J2-1304) Update Jackson from 2.6.0 to 2.7.2

[Gary Gregory (JIRA)]

Gary Gregory created LOG4J2-1304:


 Summary: Update Jackson from 2.6.0 to 2.7.2
 Key: LOG4J2-1304
 URL: https://issues.apache.org/jira/browse/LOG4J2-1304
 Project: Log4j 2
  Issue Type: Improvement
  Components: Layouts
Affects Versions: 2.5
Reporter: Gary Gregory
Assignee: Gary Gregory
 Fix For: 2.6


Update Jackson from 2.6.0 to 2.7.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

[jira] [Updated] (LOG4J2-1304) Update Jackson from 2.6.0 to 2.7.7

[Gary Gregory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LOG4J2-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Gregory updated LOG4J2-1304:
-
Summary: Update Jackson from 2.6.0 to 2.7.7  (was: Update Jackson from 
2.6.0 to 2.7.2)

> Update Jackson from 2.6.0 to 2.7.7
> --
>
> Key: LOG4J2-1304
> URL: https://issues.apache.org/jira/browse/LOG4J2-1304
> Project: Log4j 2
>  Issue Type: Improvement
>  Components: Layouts
>Affects Versions: 2.5
>Reporter: Gary Gregory
>Assignee: Gary Gregory
> Fix For: 2.6
>
>
> Update Jackson from 2.6.0 to 2.7.2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org

Re: [jira] [Commented] (LOG4J2-1299) Add pattern converter for thread id in PatternLayout

[Gary Gregory]

Hm, yeah, we'll have to remember to do that! ;-)

Gary

On Tue, Mar 1, 2016 at 10:52 AM, Matt Sicker  wrote:

> Yeah probably. Just note it in the release notes?
>
> On 1 March 2016 at 12:50, Gary Gregory  wrote:
>
>> Yeah, I did not add custom read/write, perhaps it's a YAGNI item?
>>
>> Gary
>>
>> On Tue, Mar 1, 2016 at 10:11 AM, Matt Sicker  wrote:
>>
>>> New fields is backwards compatible, but not forwards compatible. You
>>> could add custom readObject and writeObject methods if we really need BC
>>> and FC.
>>>
>>> On 1 March 2016 at 11:40, Gary Gregory  wrote:
>>>
 New fields for thread ID and thread priority in the log event class,
 which is of course serializable.

 Gary
 On Mar 1, 2016 8:27 AM, "Matt Sicker"  wrote:

> How does it break serialization?
>
> On 1 March 2016 at 10:13, Gary Gregory (JIRA)  wrote:
>
>>
>> [
>> https://issues.apache.org/jira/browse/LOG4J2-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173959#comment-15173959
>> ]
>>
>> Gary Gregory commented on LOG4J2-1299:
>> --
>>
>> My patch is almost ready. Note that this will break log event
>> serialization with 2.5.
>>
>> > Add pattern converter for thread id in PatternLayout
>> > 
>> >
>> > Key: LOG4J2-1299
>> > URL:
>> https://issues.apache.org/jira/browse/LOG4J2-1299
>> > Project: Log4j 2
>> >  Issue Type: Improvement
>> >  Components: Core, Pattern Converters
>> >Affects Versions: 2.5
>> >Reporter: Remko Popma
>> >Assignee: Gary Gregory
>> >
>> > As requested here:
>> >
>> http://stackoverflow.com/questions/14359044/display-thread-id-instead-thread-name-in-log
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
>> -
>> To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
>> For additional commands, e-mail: log4j-dev-h...@logging.apache.org
>>
>>
>
>
> --
> Matt Sicker 
>

>>>
>>>
>>> --
>>> Matt Sicker 
>>>
>>
>>
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> 
>> JUnit in Action, Second Edition 
>> Spring Batch in Action 
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>
>
>
> --
> Matt Sicker 
>



-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: [1/2] logging-log4j2 git commit: [LOG4J2-1299] Add pattern converter for thread id and priority in PatternLayout.

[Gary Gregory]

Yep, darn, good catch. Fixing...

Gary

On Wed, Mar 2, 2016 at 12:17 AM, Remko Popma  wrote:

> Should "ThreadProperty" not be "ThreadPriority"?
>
> On Wednesday, 2 March 2016,  wrote:
>
>> Repository: logging-log4j2
>> Updated Branches:
>>   refs/heads/master c83366edb -> 9c9bc95d3
>>
>>
>> [LOG4J2-1299] Add pattern converter for thread id and priority in
>> PatternLayout.
>>
>> Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
>> Commit:
>> http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/9ebf0720
>> Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/9ebf0720
>> Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/9ebf0720
>>
>> Branch: refs/heads/master
>> Commit: 9ebf07202e981a9a90a4b076486cd678d097b2a4
>> Parents: b2ec305
>> Author: ggregory 
>> Authored: Tue Mar 1 16:55:56 2016 -0800
>> Committer: ggregory 
>> Committed: Tue Mar 1 16:55:56 2016 -0800
>>
>> --
>>  log4j-core/src/main/resources/Log4j-events.xsd | 2 ++
>>  1 file changed, 2 insertions(+)
>> --
>>
>>
>>
>> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/9ebf0720/log4j-core/src/main/resources/Log4j-events.xsd
>> --
>> diff --git a/log4j-core/src/main/resources/Log4j-events.xsd
>> b/log4j-core/src/main/resources/Log4j-events.xsd
>> index 153302a..875df91 100644
>> --- a/log4j-core/src/main/resources/Log4j-events.xsd
>> +++ b/log4j-core/src/main/resources/Log4j-events.xsd
>> @@ -58,7 +58,9 @@
>>  > use="required"/>
>>  > use="optional"/>
>>  > type="log4j:LevelEnum" use="required"/>
>> +> use="required"/>
>>  > use="required"/>
>> +> type="xs:integer" use="required"/>
>>  
>>  
>>  
>>
>>


-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition

JUnit in Action, Second Edition 
Spring Batch in Action 
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Re: [1/2] logging-log4j2 git commit: [LOG4J2-1299] Add pattern converter for thread id and priority in PatternLayout.

[Remko Popma]

Should "ThreadProperty" not be "ThreadPriority"?

On Wednesday, 2 March 2016,  wrote:

> Repository: logging-log4j2
> Updated Branches:
>   refs/heads/master c83366edb -> 9c9bc95d3
>
>
> [LOG4J2-1299] Add pattern converter for thread id and priority in
> PatternLayout.
>
> Project: http://git-wip-us.apache.org/repos/asf/logging-log4j2/repo
> Commit:
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/commit/9ebf0720
> Tree: http://git-wip-us.apache.org/repos/asf/logging-log4j2/tree/9ebf0720
> Diff: http://git-wip-us.apache.org/repos/asf/logging-log4j2/diff/9ebf0720
>
> Branch: refs/heads/master
> Commit: 9ebf07202e981a9a90a4b076486cd678d097b2a4
> Parents: b2ec305
> Author: ggregory >
> Authored: Tue Mar 1 16:55:56 2016 -0800
> Committer: ggregory >
> Committed: Tue Mar 1 16:55:56 2016 -0800
>
> --
>  log4j-core/src/main/resources/Log4j-events.xsd | 2 ++
>  1 file changed, 2 insertions(+)
> --
>
>
>
> http://git-wip-us.apache.org/repos/asf/logging-log4j2/blob/9ebf0720/log4j-core/src/main/resources/Log4j-events.xsd
> --
> diff --git a/log4j-core/src/main/resources/Log4j-events.xsd
> b/log4j-core/src/main/resources/Log4j-events.xsd
> index 153302a..875df91 100644
> --- a/log4j-core/src/main/resources/Log4j-events.xsd
> +++ b/log4j-core/src/main/resources/Log4j-events.xsd
> @@ -58,7 +58,9 @@
>   use="required"/>
>   use="optional"/>
>   use="required"/>
> + use="required"/>
>   use="required"/>
> + type="xs:integer" use="required"/>
>  
>  
>  
>
>

Re: Fast and Furious

[Remko Popma]

Re: binary logging, I think he's talking about providing an API to log objects 
directly into byte buffers without turning them into Strings first. 

https://issues.apache.org/jira/browse/LOG4J2-1274 and 
https://issues.apache.org/jira/browse/LOG4J2-506

were created with that in mind and should be a good step in that direction. 

Sent from my iPhone

> On 2016/03/02, at 15:11, Gary Gregory  wrote:
> 
> Well, I've often wondered about creating a binary format but it seems that 
> you could use JSON+ZIP or BSON and get most of the advantages.
> 
> Gary
> 
>> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker  wrote:
>> One other interesting thing I learned is that improper use of logging is a 
>> huge source of performance problems. The GC-free parameterized message 
>> factory will help with one aspect of that (I suggested parameterized 
>> messages, but he countered with the Object[] that is created), and 
>> encouraging users to use a Supplier instead of passing parameters 
>> should help as well (especially when those parameters have to be computed). 
>> He had some strong criticisms of logging APIs promoting bad practices which 
>> stems all the way back to log4j1 and affects pretty much every logging API 
>> in Java (some criticisms were actually outdated or didn't consider newer 
>> features of the API like markers and the huge amount of filters available).
>> 
>> His other big idea was promoting the use of binary logging formats because 
>> humans rarely read the raw log files as it is, but it's not like there's a 
>> standard way to do that.
>> 
>> Now I kinda wonder if he'll find this thread one day and tell me how I 
>> misinterpreted him or something. ;)
>> 
>>> On 1 March 2016 at 22:28, Matt Sicker  wrote:
>>> Alright, I learned some interesting things. I'm going to get us some tools 
>>> we can use to try and profile this. Otherwise, he did suggest trying out 
>>> this project:
>>> https://github.com/RichardWarburton/honest-profiler
>>> 
 On 1 March 2016 at 19:31, Matt Sicker  wrote:
 So far he's said something about using lambdas for lazy evaluation (though 
 I don't think that would actually help us at all). I'll try to talk to him 
 one-on-one afterward to delve more into this.
 
> On 1 March 2016 at 18:13, Ralph Goers  wrote:
> Actually, most of the tests have the commands in the comments right in 
> the class. Just cut and past.
> 
> Ralph
> 
>> On Mar 1, 2016, at 1:43 PM, Matt Sicker  wrote:
>> 
>> I can't even figure out how to execute the simple perf test class. 
>> IntelliJ gives me some annotation processing error, and doing it from 
>> the command line is turning into a classpath nightmare to figure out 
>> what jars are needed to execute the test manually.
>> 
>> On 1 March 2016 at 11:34, Gary Gregory  wrote:
>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you 
>>> available after the preso to talk about some issue we are seeing?
>>> 
>>> Gary
>>> 
 On Mar 1, 2016 8:29 AM, "Matt Sicker"  wrote:
 I'm attending a JUG meetup tonight with Kirk Pepperdine presenting. 
 It's supposed to be a Java performance workshop type of thing, so if 
 you've got a decent way to ask about it, I could see if he can help 
 figure out this regression. I can at least show off the SimplePerfTest 
 and any microbenchmarks we have.
 
> On 28 February 2016 at 11:54, Matt Sicker  wrote:
> Take a look at the git bisect command. Might help you find which 
> changes caused the problem.
> 
> 
>> On Sunday, 28 February 2016, Gary Gregory  
>> wrote:
>> Thank you for digging in Remko. This is will be a nice theme to 
>> publicize when you get it figured out.
>> 
>> Gary
>> 
>>> On Feb 28, 2016 4:08 AM, "Remko Popma"  
>>> wrote:
>>> After removing the potential impact of appenders and layouts by 
>>> testing with 
>>> log4j-core\src\test\resources\perf-CountingNoOpAppender.xml and 
>>> org.apache.logging.log4j.core.async.perftest.SimplePerfTest, I've 
>>> confirmed my initial numbers:
>>> 
>>> 2.0: 7.5M ops/sec
>>> 2.1: 6M ops/sec
>>> 2.2: 6M ops/sec
>>> 2.3: 6M ops/sec
>>> 2.4: 4.5M ops/sec
>>> 2.5: 4M ops/sec
>>> 2.6: 2M ops/sec
>>> 
>>> I tried reverting various changes made to AsyncLogger since 2.0, 
>>> performance improves a little up to 4M ops/sec.
>>> However, when completely reverting AsyncLogger source to the 2.0 
>>> version, performance is back to 7.5M ops/sec.