Re: [discuss] Using a logger hierarchy in Python

2019-11-26 Thread Pablo Estrada
Ah I'll try to add this tomorrow before going out for the weekend. -P. On Wed, Nov 20, 2019 at 12:15 PM Valentyn Tymofieiev wrote: > Based on my recent debugging experience for > https://issues.apache.org/jira/browse/BEAM-8651, I think it may be > helpful to include thread IDs, into the log

Re: [discuss] Using a logger hierarchy in Python

2019-11-20 Thread Valentyn Tymofieiev
Based on my recent debugging experience for https://issues.apache.org/jira/browse/BEAM-8651, I think it may be helpful to include thread IDs, into the log entries, or have an option to easily enable this. I imagine that having process ID may also be helpful in other situations. On Tue, Nov 19,

Re: [discuss] Using a logger hierarchy in Python

2019-11-19 Thread Chad Dombrova
Pablo, it might be necessary to setup a root logging handler if one does not exist already. I noticed that a LocalJobServicer that I was testing against stopped emitting tracebacks when I rebased onto the latest from master. Setting up the root handler fixed it. I'm still testing this, and I

Re: [discuss] Using a logger hierarchy in Python

2019-11-15 Thread Pablo Estrada
Thanks all, 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to switch the variable name to _log via sed. Best -P. On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver wrote: > +1 for per-module loggers (what Robert said). > > On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri wrote: > >> +1, but

Re: [discuss] Using a logger hierarchy in Python

2019-11-15 Thread Udi Meiri
+1, but can we use something less verbose and shift key heavy than _LOGGER like log or _log? Also please dedupe with these existing bugs: https://issues.apache.org/jira/browse/BEAM-3523 https://issues.apache.org/jira/browse/BEAM-1825 On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: >

Re: [discuss] Using a logger hierarchy in Python

2019-11-14 Thread Thomas Weise
Awesome, thanks Chad! On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova wrote: > Hi Thomas, > > >> Will this include the ability for users to configure logging via pipeline >> options? >> > > We're working on a proposal to allow pluggable logging handlers that can > be configured via pipeline

Re: [discuss] Using a logger hierarchy in Python

2019-11-13 Thread Chad Dombrova
Hi Thomas, > Will this include the ability for users to configure logging via pipeline > options? > We're working on a proposal to allow pluggable logging handlers that can be configured via pipeline options. For example, it would allow you to add a new logging handler for StackDriver or

Re: [discuss] Using a logger hierarchy in Python

2019-11-13 Thread Pablo Estrada
Okay, I've just gone and done this for most modules: Runners modules: https://github.com/apache/beam/pull/10097 IO modules: https://github.com/apache/beam/pull/10099 Other modules (testing, utils): https://github.com/apache/beam/pull/10100 I imagine the trickier one will be runners, since

Re: [discuss] Using a logger hierarchy in Python

2019-11-13 Thread Chad Dombrova
On Wed, Nov 13, 2019 at 10:52 AM Robert Bradshaw wrote: > I would be in favor of using module-level loggers as well. +1

Re: [discuss] Using a logger hierarchy in Python

2019-11-13 Thread Robert Bradshaw
I would be in favor of using module-level loggers as well. I think per-class would be overkill and unlike Java not everything is in a class, as well as being more conventional in Python (where modules are generally seen as the unit of compilation, vs. Java where classes are the unit of compilation

Re: [discuss] Using a logger hierarchy in Python

2019-11-13 Thread Luke Cwik
That doesn't seem like a very invasive change so if we adopt it we should adopt it everywhere in the same CL so people see the common pattern and use it. I'm for using a named logger and would rather that it is per class instead of per module since many of the modules have lots of classes but +1

[discuss] Using a logger hierarchy in Python

2019-11-12 Thread Pablo Estrada
Hi all, as of today, the Python SDK uses the root logger wherever we log. This means that it's impossible to have different logging levels depending on the section of the code that we want to debug most. I have been doing some work on the FnApiRunner, and adding logging for it. I would like to