Re: Json Template Layout

[Ralph Goers]

See https://cwiki.apache.org/confluence/display/LOGGING/Logging+to+ELK 
.  This is 
what I get when using the Layout and configuration in Logging in the Cloud.

Ralph

> On May 17, 2020, at 4:18 PM, Ralph Goers  wrote:
> 
> 
> 
>> On May 17, 2020, at 2:37 PM, Volkan Yazıcı  wrote:
>> 
>> Thanks so much for the thorough review Ralph, really appreciated! I
>> will address issues you have raised.
>> 
>> [As a side note, I have pushed changes containing performance
>> improvements and benchmark results. The module is still dependency
>> free and performance-wise pretty good.]
>> 
>>> all the default templates separate the message from the exception
>> 
>> Yes, this I have also realized while studying the source code of
>> GelfLayout. Some random thoughts:
>> 
>> 1. How about introducing a firstNonNull-like operator:
>>  ${json:op:firstNonNull:${json:message},${json:exception:message}}.
>>  Parsing this would not be trivial, given the list of variables
>>  can include comma in various forms. Though the rest should
>>  be tractable. (Feels like JsonTemplateLayout directives will at
>>  some point catch up the with ones in PatternLayout.)
> 
> The problem here is that I want newlines in the message. I’m not sure what 
> firstNonNull means in the context of a message. The message itself won’t have 
> nulls. The Gelf format dictates the null is at the end of the JSON string.
> 
>> 
>> 2. How about introducing a fallback parameter to the "message"
>>  directive: ${json:message:fallback=exception:message}.
> 
> Again, I don’t know what that means or how it solves my problem of wanting 
> newlines in the message.
> 
>> 
>> 3. May I challenge the request: Do one really need it? You store
>>  individual values, i.e, message and exception, in individual fields.
>>  When one is missing and the other is present, isn't it just a
>>  presentation layer issue to display it properly?
> 
> Picture how a log event looks in a file with a “standard” PatternLayout. In 
> Kibana by default you get 3 columns, the timestamp, a field I don’t care 
> about and the message. I want the message to appear just as it would in a log 
> file without the timestamp. Yes, I can click on each message and see its 
> attributes, but that takes longer. And I want exceptions to jump out at me 
> when I am visually scrolling through messages. In fact, I typically don’t 
> want the stack trace in a variable of its own.   Kibana doesn’t take the 
> pieces of the message and put them together like the pattern layout does. 
> Instead it lets you create columns for individual attributes. The problem 
> with that is the columns waste a lot of space, especially if they frequently 
> are empty as a stack trace would be.
> 
> So yes, I really need it.
> 
>> 
>>> In the logged event all newlines seem to be stripped from the exception.
>> 
>> I really don't get this Ralph. Have you seen the tests in LogstashIT?
>> I have just added a new one testing only newlines. Would you mind
>> fiddling around with LogstashIT to reproduce the issue, please? Or
>> sharing a recipe for me to reproduce it?
> 
> The way I do it is to install Logstash, ElasticSearch and Kibana locally on 
> my Mac as well as Docker Desktop for Mac. I then run SpringCloud config 
> server from the Log4j Spring Cloud Config project and tailor the log4j2.xml 
> that is in the config-repo directory of that project. Note that all Log4j 
> Lookup variables have to have the $ escaped or Spring Cloud Config will try 
> to resolve them. It also requires having the Socket appender connect to a 
> host of “host.docker.internal” to be able to connect to Logstash outside of 
> the docker container. I configure Logstash using the configuration documented 
> at http://logging.apache.org/log4j/2.x/manual/cloud.html 
>  
>  >, at least when using 
> Gelf. I configure Kibana with an index of app-*.
> 
> I then modifythe Spring Cloud Config Sample app as needed to work with this 
> setup and start it using docker/restartApp.sh. 
> 
> After starting I do a few things:
> 1. Enter “docker logs app-container”. This should show it is running and the 
> last message will be the Spring logo.
> 2. Look in Kibana and click on the logs. You should see logs from the server.
> 3. In a browser enter http://localhost:8080/sample/exception 
>  
>  > - unless you have changed the port. 
> For some reason I had something running on 8080 and had to change the sample 
> app to use 8090. This will generate a stack trace on the web page and in the 
> log.
> 4. Look in Kibana. You should see the exception and the stack trace properly 
> formatted in the message column. It should also 

Re: Json Template Layout

[Ralph Goers]

To be clear, while I could easily merge the PR for you I am sure you would like 
to do it yourself now that you have commit privileges.

Ralph

> On May 17, 2020, at 2:37 PM, Volkan Yazıcı  wrote:
> 
> Thanks so much for the thorough review Ralph, really appreciated! I
> will address issues you have raised.
> 
> [As a side note, I have pushed changes containing performance
> improvements and benchmark results. The module is still dependency
> free and performance-wise pretty good.]
> 
>> all the default templates separate the message from the exception
> 
> Yes, this I have also realized while studying the source code of
> GelfLayout. Some random thoughts:
> 
> 1. How about introducing a firstNonNull-like operator:
>   ${json:op:firstNonNull:${json:message},${json:exception:message}}.
>   Parsing this would not be trivial, given the list of variables
>   can include comma in various forms. Though the rest should
>   be tractable. (Feels like JsonTemplateLayout directives will at
>   some point catch up the with ones in PatternLayout.)
> 
> 2. How about introducing a fallback parameter to the "message"
>   directive: ${json:message:fallback=exception:message}.
> 
> 3. May I challenge the request: Do one really need it? You store
>   individual values, i.e, message and exception, in individual fields.
>   When one is missing and the other is present, isn't it just a
>   presentation layer issue to display it properly?
> 
>> In the logged event all newlines seem to be stripped from the exception.
> 
> I really don't get this Ralph. Have you seen the tests in LogstashIT?
> I have just added a new one testing only newlines. Would you mind
> fiddling around with LogstashIT to reproduce the issue, please? Or
> sharing a recipe for me to reproduce it?
> 
>> Logstash does not seem to be getting a null character
>> (I tried with both “\0” and “\u”)
> 
> This is weird. Below is how GelfLayout does it:
> 
>if (includeNullDelimiter) {
>builder.append('\0');
>}
> 
> And here is how JsonTemplateLayout does it:
> 
>stringBuilder.append(eventDelimiter);
> 
> Further, there is even a test for this:
> JsonTemplateLayout#test_null_eventDelimiter():
> 
>final String serializedLogEvent = layout.toSerializable(logEvent);
>assertThat(serializedLogEvent).isEqualTo(eventTemplate + '\0');
> 
> Any ideas on what might I be missing?
> 
>> This isn’t valid in the Gelf spec as additional attributes in Gelf
>> must match the regex - ^[\w\.\-]*$. This means you need to
>> parse the MDC and add each key as an additional to be able
>> to create valid Gelf. Personally, I’d like that option anyway.
> 
> For this purpose, I need to introduce an operator similar to
> unquote-splicing in Lisp to merge the contents of the child with the
> parent. That said, I have my doubts about the practical benefit of
> such a feature. To the best of my knowledge, almost every (sanely
> configured?) log pipeline ends up with a structured storage system
> containing whitelisted fields. Consider ELK stack. Would you allow
> developers to introduce fields at their will? If so, it is a matter of
> time somebody will take down the entire Elasticsearch cluster by
> flooding it with ContextData.put(userInput, "doh"). Hence, you employ
> the identical whitelisting strategy at the layout as well:
> 
>{
>  "version": "1.1",
>  "host": "${hostName}",
>  ...,
>  "_mdc.loginId": "${json:mdc:loginId}",
>  "_mdc.requestId": "${json:mdc:requestId}"
>}
> 
> What do you think? Shall we really introduce unquote-splicing?
> 
> Kind regards.
> 
> On Sat, May 16, 2020 at 1:28 AM Ralph Goers  
> wrote:
>> 
>> I finally got time to spend looking at the Layout. I now understand why it 
>> works for you and sort of understand why it doesn’t work for me.
>> 
>> First, all the default templates separate the message from the exception. In 
>> the logged event all newlines seem to be stripped from the exception. This 
>> makes viewing them in Kibana pretty awful, but the do get logged 
>> successfully.
>> 
>> To include newlines in a log event you either have to play games in Logstash 
>> (or Fluentd) and try to put the event back together again after it has been 
>> separated into multiple lines or you have to use a different delimiter. Gelf 
>> specifies that a null character be used.
>> 
>> I then tried to use GelfLayout.json with
>> 
>> >host="\${sys:logstash.host:-host.docker.internal}"
>>port="1"
>>protocol="tcp"
>>bufferedIo="true">
>>  > eventDelimiter="\0"
>>  locationInfoEnabled="true">
>>
>>  
>>  
>>  > value="$\${lower:\${spring:spring.application.name:-spring}}"/>
>>
>>  
>> 
>> but this did not work at all. Logstash does not seem to be getting a null 
>> character (I tried with both “\0” and “\u”) and all the events buffer in 
>> Logstash until I shutdown the application. Worse the formatting looks like 
>> escaped JSON rather than raw json.
>> 
>> 

Re: Json Template Layout

[Ralph Goers]

> On May 17, 2020, at 2:37 PM, Volkan Yazıcı  wrote:
> 
> Thanks so much for the thorough review Ralph, really appreciated! I
> will address issues you have raised.
> 
> [As a side note, I have pushed changes containing performance
> improvements and benchmark results. The module is still dependency
> free and performance-wise pretty good.]
> 
>> all the default templates separate the message from the exception
> 
> Yes, this I have also realized while studying the source code of
> GelfLayout. Some random thoughts:
> 
> 1. How about introducing a firstNonNull-like operator:
>   ${json:op:firstNonNull:${json:message},${json:exception:message}}.
>   Parsing this would not be trivial, given the list of variables
>   can include comma in various forms. Though the rest should
>   be tractable. (Feels like JsonTemplateLayout directives will at
>   some point catch up the with ones in PatternLayout.)

The problem here is that I want newlines in the message. I’m not sure what 
firstNonNull means in the context of a message. The message itself won’t have 
nulls. The Gelf format dictates the null is at the end of the JSON string.

> 
> 2. How about introducing a fallback parameter to the "message"
>   directive: ${json:message:fallback=exception:message}.

Again, I don’t know what that means or how it solves my problem of wanting 
newlines in the message.

> 
> 3. May I challenge the request: Do one really need it? You store
>   individual values, i.e, message and exception, in individual fields.
>   When one is missing and the other is present, isn't it just a
>   presentation layer issue to display it properly?

Picture how a log event looks in a file with a “standard” PatternLayout. In 
Kibana by default you get 3 columns, the timestamp, a field I don’t care about 
and the message. I want the message to appear just as it would in a log file 
without the timestamp. Yes, I can click on each message and see its attributes, 
but that takes longer. And I want exceptions to jump out at me when I am 
visually scrolling through messages. In fact, I typically don’t want the stack 
trace in a variable of its own.   Kibana doesn’t take the pieces of the message 
and put them together like the pattern layout does. Instead it lets you create 
columns for individual attributes. The problem with that is the columns waste a 
lot of space, especially if they frequently are empty as a stack trace would be.

So yes, I really need it.

> 
>> In the logged event all newlines seem to be stripped from the exception.
> 
> I really don't get this Ralph. Have you seen the tests in LogstashIT?
> I have just added a new one testing only newlines. Would you mind
> fiddling around with LogstashIT to reproduce the issue, please? Or
> sharing a recipe for me to reproduce it?

The way I do it is to install Logstash, ElasticSearch and Kibana locally on my 
Mac as well as Docker Desktop for Mac. I then run SpringCloud config server 
from the Log4j Spring Cloud Config project and tailor the log4j2.xml that is in 
the config-repo directory of that project. Note that all Log4j Lookup variables 
have to have the $ escaped or Spring Cloud Config will try to resolve them. It 
also requires having the Socket appender connect to a host of 
“host.docker.internal” to be able to connect to Logstash outside of the docker 
container. I configure Logstash using the configuration documented at 
http://logging.apache.org/log4j/2.x/manual/cloud.html 
, at least when using 
Gelf. I configure Kibana with an index of app-*.

I then modifythe Spring Cloud Config Sample app as needed to work with this 
setup and start it using docker/restartApp.sh. 

After starting I do a few things:
1. Enter “docker logs app-container”. This should show it is running and the 
last message will be the Spring logo.
2. Look in Kibana and click on the logs. You should see logs from the server.
3. In a browser enter http://localhost:8080/sample/exception 
 - unless you have changed the port. 
For some reason I had something running on 8080 and had to change the sample 
app to use 8090. This will generate a stack trace on the web page and in the 
log.
4. Look in Kibana. You should see the exception and the stack trace properly 
formatted in the message column. It should also display the thread id, 
log-level and other items configured in the pattern.

> 
>> Logstash does not seem to be getting a null character
>> (I tried with both “\0” and “\u”)
> 
> This is weird. Below is how GelfLayout does it:
> 
>if (includeNullDelimiter) {
>builder.append('\0');
>}
> 
> And here is how JsonTemplateLayout does it:
> 
>stringBuilder.append(eventDelimiter);
> 
> Further, there is even a test for this:
> JsonTemplateLayout#test_null_eventDelimiter():
> 
>final String serializedLogEvent = layout.toSerializable(logEvent);
>assertThat(serializedLogEvent).isEqualTo(eventTemplate + 

Re: Json Template Layout

[Volkan Yazıcı]

Thanks so much for the thorough review Ralph, really appreciated! I
will address issues you have raised.

[As a side note, I have pushed changes containing performance
improvements and benchmark results. The module is still dependency
free and performance-wise pretty good.]

> all the default templates separate the message from the exception

Yes, this I have also realized while studying the source code of
GelfLayout. Some random thoughts:

1. How about introducing a firstNonNull-like operator:
   ${json:op:firstNonNull:${json:message},${json:exception:message}}.
   Parsing this would not be trivial, given the list of variables
   can include comma in various forms. Though the rest should
   be tractable. (Feels like JsonTemplateLayout directives will at
   some point catch up the with ones in PatternLayout.)

2. How about introducing a fallback parameter to the "message"
   directive: ${json:message:fallback=exception:message}.

3. May I challenge the request: Do one really need it? You store
   individual values, i.e, message and exception, in individual fields.
   When one is missing and the other is present, isn't it just a
   presentation layer issue to display it properly?

> In the logged event all newlines seem to be stripped from the exception.

I really don't get this Ralph. Have you seen the tests in LogstashIT?
I have just added a new one testing only newlines. Would you mind
fiddling around with LogstashIT to reproduce the issue, please? Or
sharing a recipe for me to reproduce it?

> Logstash does not seem to be getting a null character
> (I tried with both “\0” and “\u”)

This is weird. Below is how GelfLayout does it:

if (includeNullDelimiter) {
builder.append('\0');
}

And here is how JsonTemplateLayout does it:

stringBuilder.append(eventDelimiter);

Further, there is even a test for this:
JsonTemplateLayout#test_null_eventDelimiter():

final String serializedLogEvent = layout.toSerializable(logEvent);
assertThat(serializedLogEvent).isEqualTo(eventTemplate + '\0');

Any ideas on what might I be missing?

> This isn’t valid in the Gelf spec as additional attributes in Gelf
> must match the regex - ^[\w\.\-]*$. This means you need to
> parse the MDC and add each key as an additional to be able
> to create valid Gelf. Personally, I’d like that option anyway.

For this purpose, I need to introduce an operator similar to
unquote-splicing in Lisp to merge the contents of the child with the
parent. That said, I have my doubts about the practical benefit of
such a feature. To the best of my knowledge, almost every (sanely
configured?) log pipeline ends up with a structured storage system
containing whitelisted fields. Consider ELK stack. Would you allow
developers to introduce fields at their will? If so, it is a matter of
time somebody will take down the entire Elasticsearch cluster by
flooding it with ContextData.put(userInput, "doh"). Hence, you employ
the identical whitelisting strategy at the layout as well:

{
  "version": "1.1",
  "host": "${hostName}",
  ...,
  "_mdc.loginId": "${json:mdc:loginId}",
  "_mdc.requestId": "${json:mdc:requestId}"
}

What do you think? Shall we really introduce unquote-splicing?

Kind regards.

On Sat, May 16, 2020 at 1:28 AM Ralph Goers  wrote:
>
> I finally got time to spend looking at the Layout. I now understand why it 
> works for you and sort of understand why it doesn’t work for me.
>
> First, all the default templates separate the message from the exception. In 
> the logged event all newlines seem to be stripped from the exception. This 
> makes viewing them in Kibana pretty awful, but the do get logged successfully.
>
> To include newlines in a log event you either have to play games in Logstash 
> (or Fluentd) and try to put the event back together again after it has been 
> separated into multiple lines or you have to use a different delimiter. Gelf 
> specifies that a null character be used.
>
> I then tried to use GelfLayout.json with
>
>  host="\${sys:logstash.host:-host.docker.internal}"
> port="1"
> protocol="tcp"
> bufferedIo="true">
>eventDelimiter="\0"
>   locationInfoEnabled="true">
> 
>   
>   
>value="$\${lower:\${spring:spring.application.name:-spring}}"/>
> 
>   
> 
> but this did not work at all. Logstash does not seem to be getting a null 
> character (I tried with both “\0” and “\u”) and all the events buffer in 
> Logstash until I shutdown the application. Worse the formatting looks like 
> escaped JSON rather than raw json.
>
> Also, the GelfTemplate contains
> {
>   "version": "1.1",
>   "host": "${hostName}",
>   "short_message": "${json:message}",
>   "full_message": "${json:exception:stackTrace:text}",
>   "timestamp": "${json:timestamp:epoch:secs}",
>   "level": "${json:level:severity:code}",
>   "_logger": "${json:logger:name}",
>   "_thread": "${json:thread:name}",
>   "_mdc": "${json:mdc}"
> 

Re: [Log4j] Potential simplification to java9 builds?

[Ralph Goers]

I kind of doubt it. Logging runs up against this primarily because Oracle 
dropped support for sun.reflect.Reflection at the same time they added 
StackWalker. Most everything else had years to switch to a replacement.  

https://www.elastic.co/blog/elasticsearch-java-9-and-beyond 
 - Elastic only 
tests against the Java version you have chosen to compile with. They rely on 
their CI tools to run multiple builds to these the combinations.

I have seen mentions of other components that are delivered as multi-release 
jars but as far as I can tell Log4j 2 was the first mainstream library to do it 
so many of the problems with it have been encountered by us.

Ralph



> On May 17, 2020, at 11:03 AM, Volkan Yazıcı  wrote:
> 
> Maybe a naive question, but... Does anybody know how other Apache
> projects deal with this? Do they also require multiple JDKs to be
> present at compile time? Do they also employ `java9` directory work
> arounds as in log4j?
> 
> On Sat, Apr 11, 2020 at 6:39 PM Matt Sicker  wrote:
>> 
>> I was playing around with the pom a little bit yesterday when I came
>> across a relatively new maven-compiler-plugin configuration element
>> called  which can be used to output the compiled
>> classes relative to META-INF/versions/N rather than the root. This
>> looks like it would likely remove the need to use
>> maven-assembly-plugin as an intermediary step.
>> 
>> I found an interesting approach linked in [1] as the multi-release
>> parent strategy with source code at [2]. I attempted to refactor
>> log4j-api to use this pattern, but I couldn't figure out how to make
>> the same pattern work for test classes (which made it impossible to
>> compile log4j-api/src/test/java9).
>> 
>> I'm going to continue experimenting a bit with this, but has anyone
>> tried out the more recent multi-version tooling support? We were early
>> users of some things, so I'd imagine tooling has caught up by now.
>> 
>> [1]: https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
>> [2]: https://github.com/meterware/multirelease-parent/blob/master/pom.xml
>> 
>> --
>> Matt Sicker 
> 

Re: [Log4j] Potential simplification to java9 builds?

[Matt Sicker]

Maven and Ant are a couple places that likely needed custom solutions,
but probably not applicable here. Maybe something in Commons? I tried
looking at a few random projects, but I'm not seeing anything. Spring
Framework might have something.

On Sun, 17 May 2020 at 13:03, Volkan Yazıcı  wrote:
>
> Maybe a naive question, but... Does anybody know how other Apache
> projects deal with this? Do they also require multiple JDKs to be
> present at compile time? Do they also employ `java9` directory work
> arounds as in log4j?
>
> On Sat, Apr 11, 2020 at 6:39 PM Matt Sicker  wrote:
> >
> > I was playing around with the pom a little bit yesterday when I came
> > across a relatively new maven-compiler-plugin configuration element
> > called  which can be used to output the compiled
> > classes relative to META-INF/versions/N rather than the root. This
> > looks like it would likely remove the need to use
> > maven-assembly-plugin as an intermediary step.
> >
> > I found an interesting approach linked in [1] as the multi-release
> > parent strategy with source code at [2]. I attempted to refactor
> > log4j-api to use this pattern, but I couldn't figure out how to make
> > the same pattern work for test classes (which made it impossible to
> > compile log4j-api/src/test/java9).
> >
> > I'm going to continue experimenting a bit with this, but has anyone
> > tried out the more recent multi-version tooling support? We were early
> > users of some things, so I'd imagine tooling has caught up by now.
> >
> > [1]: 
> > https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
> > [2]: https://github.com/meterware/multirelease-parent/blob/master/pom.xml
> >
> > --
> > Matt Sicker 



-- 
Matt Sicker 

Re: [Log4j] Potential simplification to java9 builds?

[Volkan Yazıcı]

Maybe a naive question, but... Does anybody know how other Apache
projects deal with this? Do they also require multiple JDKs to be
present at compile time? Do they also employ `java9` directory work
arounds as in log4j?

On Sat, Apr 11, 2020 at 6:39 PM Matt Sicker  wrote:
>
> I was playing around with the pom a little bit yesterday when I came
> across a relatively new maven-compiler-plugin configuration element
> called  which can be used to output the compiled
> classes relative to META-INF/versions/N rather than the root. This
> looks like it would likely remove the need to use
> maven-assembly-plugin as an intermediary step.
>
> I found an interesting approach linked in [1] as the multi-release
> parent strategy with source code at [2]. I attempted to refactor
> log4j-api to use this pattern, but I couldn't figure out how to make
> the same pattern work for test classes (which made it impossible to
> compile log4j-api/src/test/java9).
>
> I'm going to continue experimenting a bit with this, but has anyone
> tried out the more recent multi-version tooling support? We were early
> users of some things, so I'd imagine tooling has caught up by now.
>
> [1]: https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
> [2]: https://github.com/meterware/multirelease-parent/blob/master/pom.xml
>
> --
> Matt Sicker 

Re: [ANNOUNCEMENT] Welcome Volkan Yazici

[Volkan Yazıcı]

I am speechless, thanks so much Ralph! That means a lot coming from
you. I will do my best push the project forward.

On Sun, May 17, 2020 at 12:19 AM Ralph Goers  wrote:
>
>
>
> Hi,
>
> it is my pleasure to announce to the community that Volkan Yazici has joined 
> our ranks as a new Committer.
>
> He made remarkable contributions to the JSON Template Layout and we all 
> believe he is a valuable member of Apache Logging Services now and in future.
>
> Kind regards,
> Ralph Goers

Re: [Log4j] Potential simplification to java9 builds?

[Matt Sicker]

Yikes, that sounds similar to OSGi. Adding modules makes testing into more
of an integration exercise since traditional JUnit usage assumes a flat
class path.

On Sun, May 17, 2020 at 00:10 Ralph Goers 
wrote:

> I played around with this today. I don’t think it is possible to make the
> tests work.  The reason is that from what I understand the multi-release
> format only works when the classes are packaged in a jar, which hasn’t
> happened when the tests are compiled. This causes the compile to fail since
> it can’t find the Java 9+ classes.
>
> The only way around that would be to package the files in a jar before
> running the tests, which is opposite of what Maven is trying to do.
>
> Ralph
>
> > On Apr 11, 2020, at 9:39 AM, Matt Sicker  wrote:
> >
> > I was playing around with the pom a little bit yesterday when I came
> > across a relatively new maven-compiler-plugin configuration element
> > called  which can be used to output the compiled
> > classes relative to META-INF/versions/N rather than the root. This
> > looks like it would likely remove the need to use
> > maven-assembly-plugin as an intermediary step.
> >
> > I found an interesting approach linked in [1] as the multi-release
> > parent strategy with source code at [2]. I attempted to refactor
> > log4j-api to use this pattern, but I couldn't figure out how to make
> > the same pattern work for test classes (which made it impossible to
> > compile log4j-api/src/test/java9).
> >
> > I'm going to continue experimenting a bit with this, but has anyone
> > tried out the more recent multi-version tooling support? We were early
> > users of some things, so I'd imagine tooling has caught up by now.
> >
> > [1]:
> https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
> > [2]:
> https://github.com/meterware/multirelease-parent/blob/master/pom.xml
> >
> > --
> > Matt Sicker 
> >
>
>
> --
Matt Sicker