Upon further inspection there seems to be an issue we may have overlooked: In cluster mode, some of the runners will have dependencies added directly to the classpath by the cluster, and since SLF4J can only work with one binding, the first one in the classpath will be used.
So while what we suggested would work in local mode, the user's chosen binding and configuration might be ignored in cluster mode, which is detrimental to what we wanted to accomplish. So I believe what we should do instead is: 1. Add better documentation regarding logging in each runner, which binding is used, perhaps examples of how to configure logging for that runner. 2. Have direct runner use the most common binding among runners (this appears to be log4j which is used by Spark runner, Flink runner and Apex runner). On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <aljos...@apache.org> wrote: > Yes, I think we can exclude log4j from the Flink dependencies. It’s > somewhat annoying that they are there in the first place. > > The Flink doc has this to say about the topic: > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html > > On 3. Apr 2017, at 17:56, Aviem Zur <aviem...@gmail.com> wrote: > > > >> * java.util.logging could be a good choice for the Direct Runner > > Yes, this will be great for users (Instead of having no logging when > using > > direct runner). > > > >> * Logging backend could be runner-specific, particularly if it needs to > >> integrate into some other experience > > Good point, let's take a look at the current state of runners: > > Direct runner - will use JUL as suggested. > > Dataflow runner - looks like there is already no binding (There is a > > binding in tests only). > > Spark runner - currently uses slf4j-log4j12. does not require any > specific > > logger, we can change this to no binding. > > Flink runner - uses slf4j-log4j12 transitively from Flink dependencies. > I'm > > assuming this is not a must and we can default to no binding here. > > @aljoscha please confirm. > > Apex runner - uses slf4j-log4j12 transitively from Apex dependencies. I'm > > assuming this is not a must and we can default to no binding here. @thw > > please confirm. > > > > It might be a good idea to use a consistent binding in tests (Since we'll > > use JUL for direct runner, let this be JUL). > > > > On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote: > > > > +1 on consistency across Beam modules on the logging facade > > +1 on enforcing consistency > > +1 on clearly documenting how to do logging > > > > Mixed feelings: > > * Logging backend could be runner-specific, particularly if it needs to > > integrate into some other experience > > * java.util.logging could be a good choice for the Direct Runner > > > > On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid> > > wrote: > > > >> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com> > >> wrote: > >> > >>> This is a great idea! > >>> > >>> I believe Python-SDK's logging could also be enhanced (a bit > >> differently): > >>> Currently we are not instantiating the logger, just using the class > what > >>> logging package provides. > >>> Shortcoming of this approach is that the user cannot set the log level > > on > >>> a per module basis as all log messages > >>> end up in the root level. > >>> > >> > >> +1 to this. Python SDK needs to expands its logging capabilities. Filed > > [1] > >> for this. > >> > >> Ahmet > >> > >> [1] https://issues.apache.org/jira/browse/BEAM-1825 > >> > >> > >>> > >>> On 3/22/17, 5:46 AM, "Aviem Zur" <aviem...@gmail.com> wrote: > >>> > >>> +1 to what JB said. > >>> > >>> Will just have to be documented well as if we provide no binding > >> there > >>> will > >>> be no logging out of the box unless the user adds a binding. > >>> > >>> On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré < > >> j...@nanthrax.net> > >>> wrote: > >>> > >>>> Hi Aviem, > >>>> > >>>> Good point. > >>>> > >>>> I think, in our dependencies set, we should just depend to > >> slf4j-api > >>> and > >>>> let the > >>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple, > >>> whatever). > >>>> > >>>> We define a binding only with test scope in our modules. > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On 03/22/2017 04:58 AM, Aviem Zur wrote: > >>>>> Hi all, > >>>>> > >>>>> There have been a few reports lately (On JIRA [1] and on Slack) > >>> from > >>>> users > >>>>> regarding inconsistent loggers used across Beam's modules. > >>>>> > >>>>> While we use SLF4J, different modules use a different logger > >>> behind it > >>>>> (JUL, log4j, etc) > >>>>> So when people add a log4j.properties file to their classpath > > for > >>>> instance, > >>>>> they expect this to affect all of their dependencies on Beam > >>> modules, but > >>>>> it doesn’t and they miss out on some logs they thought they > > would > >>> see. > >>>>> > >>>>> I think we should strive for consistency in which logger is used > >>> behind > >>>>> SLF4J, and try to enforce this in our modules. > >>>>> I for one think it should be slf4j-log4j. However, if > > performance > >>> of > >>>>> logging is critical we might want to consider logback. > >>>>> > >>>>> Note: SLF4J will still be the facade for logging across the > >>> project. The > >>>>> only change would be the logger SLF4J delegates to. > >>>>> > >>>>> Once we have something like this it would also be useful to add > >>>>> documentation on logging in Beam to the website. > >>>>> > >>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757 > >>>>> > >>>> > >>>> -- > >>>> Jean-Baptiste Onofré > >>>> jbono...@apache.org > >>>> http://blog.nanthrax.net > >>>> Talend - http://www.talend.com > >>>> > >>> > >>> > >>> > >> > >
