RE: logging during multithreaded builds (follow-up)

2015-07-17 Thread Sander Verhagen
update your address book. Thank you! -Original Message- From: Igor Fedorenko [mailto:i...@ifedorenko.com] Sent: Tuesday, June 09, 2015 12:15 To: Maven Developers List; Sander Verhagen Subject: Re: logging during multithreaded builds (follow-up) Based on the feedback I got on this list

Re: logging during multithreaded builds (follow-up)

2015-07-17 Thread Igor Fedorenko
logging during multithreaded builds. It sounded like there was an addition to Maven being built (whether it be in the core or some extension). Is that somehow generally available, be it in released form or as (alpha) source code that I can build myself? Much appreciated, Sander

logging during multithreaded builds (follow-up)

2015-06-09 Thread Sander Verhagen
Hi group, Sorry if it's inappropriate for me to ask this in the dev list, but this is where this was previously discussed. I'm interested in the work that Igor mentioned around logging during multithreaded builds. It sounded like there was an addition to Maven being built (whether

Re: logging during multithreaded builds (follow-up)

2015-06-09 Thread Igor Fedorenko
Verhagen san...@sanderverhagen.net wrote: Hi group, Sorry if it's inappropriate for me to ask this in the dev list, but this is where this was previously discussed. I'm interested in the work that Igor mentioned around logging during multithreaded builds. It sounded like there was an addition

Re: logging during multithreaded builds

2015-05-28 Thread Anders Hammar
I agree with Jason, it would be better to keep this outside of core (the core distro). /Anders On Thu, May 28, 2015 at 5:21 AM, Jason van Zyl ja...@takari.io wrote: I honestly don't think an optional feature relying on an optional dependency belongs in the core itself. On May 27, 2015, at

Re: logging during multithreaded builds

2015-05-28 Thread Kristian Rosenvold
I know it's not directly related, but any general purpose algorithm that captures to memory buffers needs an overflow to disk mechanism - every time a ByteArrayOutputStream is used for this, some guy with a *huge* output from his build gets an OOM. It's nice you're looking into this issue. As for

Re: logging during multithreaded builds

2015-05-27 Thread Igor Fedorenko
So I went ahead and implemented these changes, including working (but not terribly well tested) logback appenders to buffer-and-group project console log messages and create per-project build.log files. Does anyone see a problem if I check in these appenders in maven core source tree or you

Re: logging during multithreaded builds

2015-05-27 Thread Igor Fedorenko
There are three semi-related parts to my implementation 1. SLF4J MDC management, basically setting and removing project attributes in a thread-local map. Truly reliable implementation will need to be coded in all Builders. Alternatively, it should be possible to use existing lifecycle callbacks

Re: logging during multithreaded builds

2015-05-27 Thread Jason van Zyl
I honestly don't think an optional feature relying on an optional dependency belongs in the core itself. On May 27, 2015, at 10:34 PM, Igor Fedorenko i...@ifedorenko.com wrote: There are three semi-related parts to my implementation 1. SLF4J MDC management, basically setting and removing

Re: logging during multithreaded builds

2015-05-27 Thread Jason van Zyl
On May 27, 2015, at 3:55 PM, Igor Fedorenko i...@ifedorenko.com wrote: So I went ahead and implemented these changes, including working (but not terribly well tested) logback appenders to buffer-and-group project console log messages and create per-project build.log files. What changes

Re: logging during multithreaded builds

2015-05-26 Thread Igor Fedorenko
I spent some time looking into this, and I think project-level logging will require several semi-related changes. * As Ralph pointed out, Maven needs to use SLF4J MDC to associate log messages with individual projects being built. This is required to enable any project-related logging approach

Re: logging during multithreaded builds

2015-05-26 Thread Jeffrey E Care
Igor Fedorenko i...@ifedorenko.com wrote on 05/25/2015 07:28:45 AM: I had to troubleshoot a large multithreaded build last week and that proved to be rather difficult mostly because build log was a jumble of messages produced by concurrently running threads. It was not possible to tell which

Re: logging during multithreaded builds

2015-05-26 Thread Ralph Goers
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and 2.x - you can include anything stored in the MDC on every line of log output. Just use %X to include all MDC items or %MDC{key} to include the specific key. This would require storing the value(s) at the beginning of

logging during multithreaded builds

2015-05-25 Thread Igor Fedorenko
I had to troubleshoot a large multithreaded build last week and that proved to be rather difficult mostly because build log was a jumble of messages produced by concurrently running threads. It was not possible to tell which message came from which thread, which made the build log more or less

Re: logging during multithreaded builds

2015-05-25 Thread Igor Fedorenko
Yes, thread-id will help to some degree, but maven uses multiline log messages quite often and these will still be mangled and unreadable. Per-project build log files is the only solution I found to preserve readable logs. Also, each project build is mostly independent from the rest and I find

Re: logging during multithreaded builds

2015-05-25 Thread Sean Busbey
In multithreaded builds we could add a thread ID to each output line. That would make it easier to read and filter in different files in post processing. -- Sean On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote: I had to troubleshoot a large multithreaded build last week and