Re: [discuss] Using a logger hierarchy in Python
Ah I'll try to add this tomorrow before going out for the weekend. -P. On Wed, Nov 20, 2019 at 12:15 PM Valentyn Tymofieiev wrote: > Based on my recent debugging experience for > https://issues.apache.org/jira/browse/BEAM-8651, I think it may be > helpful to include thread IDs, into the log entries, or have an option to > easily enable this. I imagine that having process ID may also be helpful in > other situations. > > On Tue, Nov 19, 2019 at 11:17 AM Chad Dombrova wrote: > >> Pablo, it might be necessary to setup a root logging handler if one does >> not exist already. I noticed that a LocalJobServicer that I was testing >> against stopped emitting tracebacks when I rebased onto the latest from >> master. Setting up the root handler fixed it. I'm still testing this, and >> I might be misinterpreting what I saw, but I wanted to get eyes on it in >> case I don't have time to get a definitive answer. >> >> -chad >> >> >> >> On Fri, Nov 15, 2019 at 4:30 PM Pablo Estrada wrote: >> >>> Thanks all, >>> 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to >>> switch the variable name to _log via sed. >>> Best >>> -P. >>> >>> On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver wrote: >>> +1 for per-module loggers (what Robert said). On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri wrote: > +1, but can we use something less verbose and shift key heavy than > _LOGGER like log or _log? > > Also please dedupe with these existing bugs: > https://issues.apache.org/jira/browse/BEAM-3523 > https://issues.apache.org/jira/browse/BEAM-1825 > > On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: > >> Awesome, thanks Chad! >> >> On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova >> wrote: >> >>> Hi Thomas, >>> >>> Will this include the ability for users to configure logging via pipeline options? >>> >>> We're working on a proposal to allow pluggable logging handlers that >>> can be configured via pipeline options. For example, it would allow >>> you to >>> add a new logging handler for StackDriver or Elasticsearch. Will >>> hopefully >>> have a document to share soon. >>> >>> -chad >>> >>>
Re: [discuss] Using a logger hierarchy in Python
Based on my recent debugging experience for https://issues.apache.org/jira/browse/BEAM-8651, I think it may be helpful to include thread IDs, into the log entries, or have an option to easily enable this. I imagine that having process ID may also be helpful in other situations. On Tue, Nov 19, 2019 at 11:17 AM Chad Dombrova wrote: > Pablo, it might be necessary to setup a root logging handler if one does > not exist already. I noticed that a LocalJobServicer that I was testing > against stopped emitting tracebacks when I rebased onto the latest from > master. Setting up the root handler fixed it. I'm still testing this, and > I might be misinterpreting what I saw, but I wanted to get eyes on it in > case I don't have time to get a definitive answer. > > -chad > > > > On Fri, Nov 15, 2019 at 4:30 PM Pablo Estrada wrote: > >> Thanks all, >> 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to >> switch the variable name to _log via sed. >> Best >> -P. >> >> On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver wrote: >> >>> +1 for per-module loggers (what Robert said). >>> >>> On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri wrote: >>> +1, but can we use something less verbose and shift key heavy than _LOGGER like log or _log? Also please dedupe with these existing bugs: https://issues.apache.org/jira/browse/BEAM-3523 https://issues.apache.org/jira/browse/BEAM-1825 On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: > Awesome, thanks Chad! > > On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova > wrote: > >> Hi Thomas, >> >> >>> Will this include the ability for users to configure logging via >>> pipeline options? >>> >> >> We're working on a proposal to allow pluggable logging handlers that >> can be configured via pipeline options. For example, it would allow you >> to >> add a new logging handler for StackDriver or Elasticsearch. Will >> hopefully >> have a document to share soon. >> >> -chad >> >>
Re: [discuss] Using a logger hierarchy in Python
Pablo, it might be necessary to setup a root logging handler if one does not exist already. I noticed that a LocalJobServicer that I was testing against stopped emitting tracebacks when I rebased onto the latest from master. Setting up the root handler fixed it. I'm still testing this, and I might be misinterpreting what I saw, but I wanted to get eyes on it in case I don't have time to get a definitive answer. -chad On Fri, Nov 15, 2019 at 4:30 PM Pablo Estrada wrote: > Thanks all, > 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to > switch the variable name to _log via sed. > Best > -P. > > On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver wrote: > >> +1 for per-module loggers (what Robert said). >> >> On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri wrote: >> >>> +1, but can we use something less verbose and shift key heavy than >>> _LOGGER like log or _log? >>> >>> Also please dedupe with these existing bugs: >>> https://issues.apache.org/jira/browse/BEAM-3523 >>> https://issues.apache.org/jira/browse/BEAM-1825 >>> >>> On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: >>> Awesome, thanks Chad! On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova wrote: > Hi Thomas, > > >> Will this include the ability for users to configure logging via >> pipeline options? >> > > We're working on a proposal to allow pluggable logging handlers that > can be configured via pipeline options. For example, it would allow you > to > add a new logging handler for StackDriver or Elasticsearch. Will > hopefully > have a document to share soon. > > -chad > >
Re: [discuss] Using a logger hierarchy in Python
Thanks all, 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to switch the variable name to _log via sed. Best -P. On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver wrote: > +1 for per-module loggers (what Robert said). > > On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri wrote: > >> +1, but can we use something less verbose and shift key heavy than >> _LOGGER like log or _log? >> >> Also please dedupe with these existing bugs: >> https://issues.apache.org/jira/browse/BEAM-3523 >> https://issues.apache.org/jira/browse/BEAM-1825 >> >> On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: >> >>> Awesome, thanks Chad! >>> >>> On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova >>> wrote: >>> Hi Thomas, > Will this include the ability for users to configure logging via > pipeline options? > We're working on a proposal to allow pluggable logging handlers that can be configured via pipeline options. For example, it would allow you to add a new logging handler for StackDriver or Elasticsearch. Will hopefully have a document to share soon. -chad
Re: [discuss] Using a logger hierarchy in Python
+1, but can we use something less verbose and shift key heavy than _LOGGER like log or _log? Also please dedupe with these existing bugs: https://issues.apache.org/jira/browse/BEAM-3523 https://issues.apache.org/jira/browse/BEAM-1825 On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: > Awesome, thanks Chad! > > On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova wrote: > >> Hi Thomas, >> >> >>> Will this include the ability for users to configure logging via >>> pipeline options? >>> >> >> We're working on a proposal to allow pluggable logging handlers that can >> be configured via pipeline options. For example, it would allow you to add >> a new logging handler for StackDriver or Elasticsearch. Will hopefully >> have a document to share soon. >> >> -chad >> >> smime.p7s Description: S/MIME Cryptographic Signature
Re: [discuss] Using a logger hierarchy in Python
Awesome, thanks Chad! On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova wrote: > Hi Thomas, > > >> Will this include the ability for users to configure logging via pipeline >> options? >> > > We're working on a proposal to allow pluggable logging handlers that can > be configured via pipeline options. For example, it would allow you to add > a new logging handler for StackDriver or Elasticsearch. Will hopefully > have a document to share soon. > > -chad > >
Re: [discuss] Using a logger hierarchy in Python
Hi Thomas, > Will this include the ability for users to configure logging via pipeline > options? > We're working on a proposal to allow pluggable logging handlers that can be configured via pipeline options. For example, it would allow you to add a new logging handler for StackDriver or Elasticsearch. Will hopefully have a document to share soon. -chad
Re: [discuss] Using a logger hierarchy in Python
Okay, I've just gone and done this for most modules: Runners modules: https://github.com/apache/beam/pull/10097 IO modules: https://github.com/apache/beam/pull/10099 Other modules (testing, utils): https://github.com/apache/beam/pull/10100 I imagine the trickier one will be runners, since there's a few files that do handling of logs coming from the SDK to the runner. Other than that, the changes should be relatively straightforward. Best -P. On Wed, Nov 13, 2019 at 3:45 PM Thomas Weise wrote: > +1 for using a logger hierarchy. > > Will this include the ability for users to configure logging via pipeline > options? > > > On Wed, Nov 13, 2019 at 11:04 AM Chad Dombrova wrote: > >> >> On Wed, Nov 13, 2019 at 10:52 AM Robert Bradshaw >> wrote: >> >>> I would be in favor of using module-level loggers as well. >> >> >> +1 >> >> >
Re: [discuss] Using a logger hierarchy in Python
On Wed, Nov 13, 2019 at 10:52 AM Robert Bradshaw wrote: > I would be in favor of using module-level loggers as well. +1
Re: [discuss] Using a logger hierarchy in Python
I would be in favor of using module-level loggers as well. I think per-class would be overkill and unlike Java not everything is in a class, as well as being more conventional in Python (where modules are generally seen as the unit of compilation, vs. Java where classes are the unit of compilation and there's really no analogous of modules (short of say a static-member only class, packages aren't the same thing)). On Wed, Nov 13, 2019 at 9:22 AM Luke Cwik wrote: > > That doesn't seem like a very invasive change so if we adopt it we should > adopt it everywhere in the same CL so people see the common pattern and use > it. > > I'm for using a named logger and would rather that it is per class instead of > per module since many of the modules have lots of classes but +1 from me > overall. > > On Tue, Nov 12, 2019 at 4:37 PM Pablo Estrada wrote: >> >> Hi all, >> as of today, the Python SDK uses the root logger wherever we log. This means >> that it's impossible to have different logging levels depending on the >> section of the code that we want to debug most. >> >> I have been doing some work on the FnApiRunner, and adding logging for it. I >> would like to start using a logger hierarchy, and slowly transition the rest >> of the project to use per-module loggers. >> >> On each module, we could have a line like so: >> >> _LOGGER = logging.getLogger(__name__) >> >> and simply log everything on that _LOGGER. Is that an acceptable thing to do >> for everyone? >> >> If I see no objections, I will change the FnApiRunner to use a logger like >> this, and change other sections of the code as I interact with them. >> Best >> -P.
Re: [discuss] Using a logger hierarchy in Python
That doesn't seem like a very invasive change so if we adopt it we should adopt it everywhere in the same CL so people see the common pattern and use it. I'm for using a named logger and would rather that it is per class instead of per module since many of the modules have lots of classes but +1 from me overall. On Tue, Nov 12, 2019 at 4:37 PM Pablo Estrada wrote: > Hi all, > as of today, the Python SDK uses the root logger wherever we log. This > means that it's impossible to have different logging levels depending on > the section of the code that we want to debug most. > > I have been doing some work on the FnApiRunner, and adding logging for it. > I would like to start using a logger hierarchy, and slowly transition the > rest of the project to use per-module loggers. > > On each module, we could have a line like so: > > _LOGGER = logging.getLogger(__name__) > > and simply log everything on that _LOGGER. Is that an acceptable thing to > do for everyone? > > If I see no objections, I will change the FnApiRunner to use a logger like > this, and change other sections of the code as I interact with them. > Best > -P. >
[discuss] Using a logger hierarchy in Python
Hi all, as of today, the Python SDK uses the root logger wherever we log. This means that it's impossible to have different logging levels depending on the section of the code that we want to debug most. I have been doing some work on the FnApiRunner, and adding logging for it. I would like to start using a logger hierarchy, and slowly transition the rest of the project to use per-module loggers. On each module, we could have a line like so: _LOGGER = logging.getLogger(__name__) and simply log everything on that _LOGGER. Is that an acceptable thing to do for everyone? If I see no objections, I will change the FnApiRunner to use a logger like this, and change other sections of the code as I interact with them. Best -P.
