I always fantasized of a way to filter logs through metadata, so that I could filter by a combination of application, module, and so on. Never got around to talking about it to José though :).
On Fri, 8 Dec 2017 at 17:30, Isaac Whitfield <[email protected]> wrote: > Doh, missed that you already filed one! Here's it is for everyone's > reference: https://github.com/elixir-lang/elixir/issues/7095 > > > On Friday, December 8, 2017 at 8:03:42 AM UTC-8, Isaac Whitfield wrote: >> >> Agreed; it seems like a lot of what's needed is already in place. Would >> you recommend I file an issue at this point? >> >> I completely agree that bumping the default to info is the best path >> forward (and yes, you're correct on the goal). I figured changing the >> default might be considered a "breaking" change, so a new level avoided >> that. Either way is perfectly fine by me! >> >> Thanks for the feedback! >> Isaac Whitfield >> >> On Friday, December 8, 2017 at 12:43:12 AM UTC-8, José Valim wrote: >>> >>> The minimum we can do for now is to allow compile time level purging per >>> application. This means the work is all done at compile time and there is >>> no runtime overhead. This fits well with issue #7046 >>> <https://github.com/elixir-lang/elixir/issues/7046> which aims to make >>> all logger calls purgeable without side-effects. >>> >>> In regards to adding a new level, I still don't believe it would add >>> anything new. If the goal is to reduce the default output, then I would >>> rather bump the default level to info than introduce a new level. >>> >>> >>> >>> *José Valimwww.plataformatec.com.br >>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>> >>> On Fri, Dec 8, 2017 at 12:28 AM, David Antaramian <[email protected]> >>> wrote: >>> >>>> I agree with Paul that this is something that should be solved at the >>>> Logger level and not the backend level for consistency purposes. I'm >>>> responsible for maintaining a backend. The backend *could* allow users >>>> to pass a list of applications and keep/evict matching logs based on the >>>> application metadata key available on each log event. But having this >>>> as custom logic doesn't make sense because it seems to be a more general >>>> concern. >>>> >>>> Approaching this from a different angle, one of the major differences >>>> between Elixir's Logger and logging packages in other languages is that the >>>> Logger is a singleton (for lack of a better term). For example, in Java you >>>> could instantiate a Logger instance for a specific class and then manage >>>> that logger and its level separate from others. >>>> >>>> To throw the idea out there, would it be more useful to be able to >>>> create a named Logger process separate from the main Logger process? I >>>> don't have a concrete idea for how that would interface with the rest of >>>> the logging pipeline in Elixir. My thought, though, is that only events >>>> from the main Logger process would be sent to backends by default; named >>>> Logger processes would have to be "opt-in" via explicit configuration. >>>> >>>> The additional benefit of this is that I could run the different named >>>> Logger processes at different levels, similar to Paul's namespace >>>> suggestion above. For example, I could set the named :cachex logger to >>>> run at :info by default. If I have an issue, I could switch it to run >>>> at :debug to gather diagnostic logs without the other running loggers >>>> also switching to :debug (and inundating me with a ton of other logs I >>>> don't need). >>>> >>>> - David >>>> >>>> On Thursday, December 7, 2017 at 5:43:52 PM UTC-5, Isaac Whitfield >>>> wrote: >>>>> >>>>> Hey Paul, >>>>> >>>>> Using the module is actually pretty compelling; I guess we could even >>>>> reproduce the application just by using the parent module of an >>>>> application? So we would support both with a single syntax (no need to >>>>> check for module vs. application). I would like that solution too; I'm >>>>> just >>>>> not sure if we'd want to stick to the OTP style here with applications, or >>>>> go for the more Elixir-y route of modules? >>>>> >>>>> I completely agree that it grants a lot of control over your logging >>>>> without too much churn in the Logger itself (as far as I know). Like you, >>>>> I >>>>> came from JVM languages where I'm totally used to having this readily >>>>> available to me; I found it quite surprising that there's nothing in the >>>>> Logger as it exists for it. What do you think about the extra log level >>>>> which exists below the default log level? >>>>> >>>>> Thank you both for input so far! >>>>> Isaac >>>>> >>>>> On Thursday, December 7, 2017 at 2:21:00 PM UTC-8, Paul Schoenfelder >>>>> wrote: >>>>>> >>>>>> I personally think that it would be nice to be able to override the >>>>>> log level for a specific application and/or namespace (i.e. something >>>>>> like >>>>>> a pattern, where setting the level for `Foo.Bar` will set the level for >>>>>> logs produced by any module starting with `Foo.Bar`). This provides >>>>>> control >>>>>> to silence noisy logging on a per-application or per-module level, for >>>>>> example when a dependency logs a lot of unnecessary information at too >>>>>> high >>>>>> of a log level (i.e. debug info at the info level), and you want to strip >>>>>> just the annoying bits; it also provides the ability to log more >>>>>> verbosely >>>>>> from a component without opening the floodgates by verbosely logging from >>>>>> everything in the system. >>>>>> >>>>>> While a custom logger backend could probably accomplish some or all >>>>>> of this, what about when you are already using multiple backends (say >>>>>> console and file) which both require this filtering? You either need to >>>>>> write a filter (and I'm not sure if that's practical here, as it seems >>>>>> like >>>>>> it might incur significant overhead, but I haven't looked into it in much >>>>>> detail) or reimplement all the backends you are using to do this >>>>>> filtering. >>>>>> >>>>>> This seems like pretty basic functionality that should be present in >>>>>> Logger itself, and since the pieces are already there, it's really a >>>>>> matter >>>>>> of putting them together. Having spent many years in .NET and JVM >>>>>> languages, the ability to specify backends, log levels, and even format >>>>>> on >>>>>> a per-appdomain/namespace/class basis was pretty much table stakes in >>>>>> loggers in those languages - I see no reason why we shouldn't have that >>>>>> capability in Elixir. >>>>>> >>>>>> Paul >>>>>> >>>>>> >>>>>> On Thu, Dec 7, 2017 at 2:56 PM, Isaac Whitfield <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Using "abusing OTP internals" makes me think that we should avoid >>>>>>> that bit; but I'm admittedly unfamiliar with how the backends work. >>>>>>> When I >>>>>>> say "application" above, I'm mainly using it to mean modules that >>>>>>> belong to >>>>>>> the application - not processes. This would mean that GenServers started >>>>>>> using application modules would also be included in the definitions. >>>>>>> >>>>>>> I think this would be more expected; what do you think? >>>>>>> >>>>>>> Isaac Whitfield >>>>>>> >>>>>>> On Thursday, December 7, 2017 at 11:18:36 AM UTC-8, Michał Muskała >>>>>>> wrote: >>>>>>>> >>>>>>>> There's always a question of what this setting would actually mean >>>>>>>> - should it affect logging from modules that belong to that >>>>>>>> application or >>>>>>>> should it affect logging from processes belonging to that application. >>>>>>>> If >>>>>>>> the later, it's already possible with a custom logger backend by >>>>>>>> leveraging >>>>>>>> the group leader mechanism and abusing OTP internals a little bit. >>>>>>>> >>>>>>>> Each log has the pid of the originating process - using >>>>>>>> :application.get_application(pid) one can obtain the application where >>>>>>>> this >>>>>>>> process runs. The backend can dispatch the log based on the >>>>>>>> configuration >>>>>>>> and only log messages from some of the applications. >>>>>>>> >>>>>>>> Michał. >>>>>>>> >>>>>>>> On 7 Dec 2017, 20:01 +0100, Isaac Whitfield <[email protected]>, >>>>>>>> wrote: >>>>>>>> >>>>>>>> This is based on previous conversation in >>>>>>>> https://groups.google.com/d/msg/elixir-lang-core/C5-ixZVau3U/Gkb4PyOOAQAJ, >>>>>>>> but this thread proposes an actual solution which I believe will >>>>>>>> benefit >>>>>>>> everyone. >>>>>>>> >>>>>>>> The Logger in Elixir is widely used due to being in the standard >>>>>>>> library, but misses out on a few key use cases that are familiar for >>>>>>>> developers coming from other languages. For one, I believe it's time >>>>>>>> to add >>>>>>>> functionality to suppress logs on a per-application basis. As the >>>>>>>> application is known to the logging macros it can easily filter out at >>>>>>>> compile time. This can also be a backwards-compatible option inside the >>>>>>>> Logger configuration to override the default log level on a per >>>>>>>> application >>>>>>>> basis: >>>>>>>> >>>>>>>> >>>>>>>> config :logger, :my_logger, >>>>>>>> level: :info, >>>>>>>> applications: [ >>>>>>>> library1: :debug >>>>>>>> ] >>>>>>>> >>>>>>>> >>>>>>>> This allows developers to allow logs to be noisier for specific >>>>>>>> applications (such as their own), without being swamped with noise from >>>>>>>> other applications. The :applications key will just override the >>>>>>>> default >>>>>>>> level provided in the :level key. If the :applications key is not >>>>>>>> ideal, we >>>>>>>> can name it something else (:apps, :override, etc). We could even just >>>>>>>> combine the two in :levels and use a :default for the default level. >>>>>>>> >>>>>>>> In addition to these changes, I propose that a new logging level be >>>>>>>> added which is lower than the default log level of :debug, with the >>>>>>>> intent >>>>>>>> that all developers log to this unless there is explicit reason to have >>>>>>>> their logs seen by default. The combination of this and the above >>>>>>>> :applications key allows developers to easily turn on logs from >>>>>>>> applications they're interested in at the time. It allows the >>>>>>>> application >>>>>>>> developer to decide what is important enough for another developer to >>>>>>>> see, >>>>>>>> rather than the developer having to (typically) guess. I propose that >>>>>>>> this >>>>>>>> new logging level should be named any of :trace, :verbose or :quiet >>>>>>>> (with >>>>>>>> :trace being preferred for brevity/familiarity). >>>>>>>> >>>>>>>> There are many other languages which encourage this style of >>>>>>>> logging. Here are a few I know of: >>>>>>>> >>>>>>>> - Node.js has the "debug" package, which is widely popular as it >>>>>>>> allows users to opt-in to seeing logs from specific namespaces (which >>>>>>>> roughly translates to OTP applications in our use case). >>>>>>>> - Java has SLF4J which hides debug logs by default, allowing you to >>>>>>>> opt in on package or class level (also hugely popular). >>>>>>>> - Rust's official logging framework defaults to actually hiding all >>>>>>>> logging, making you explicitly ask to see logs (of any kind). >>>>>>>> >>>>>>>> The interesting thing with the packages from Node.s/Java is that >>>>>>>> they're libraries and not in the standard library; they evolved >>>>>>>> because of >>>>>>>> the standard logging missing this stuff (although to be fair, Node.js >>>>>>>> only >>>>>>>> has console). As the Logger is so widely used in Elixir, it feels that >>>>>>>> we're already at the point where this should be supported. Making this >>>>>>>> an >>>>>>>> external library in Elixir would be wasted effort as we would then have >>>>>>>> logging from different sources with different behaviour, thus lowering >>>>>>>> the >>>>>>>> utility. I think that including this support in Logger will make >>>>>>>> logging >>>>>>>> feel more familiar to developers coming from the languages above, >>>>>>>> without >>>>>>>> making it unfamiliar to (e.g.) Ruby developers where the behaviour >>>>>>>> matches >>>>>>>> what Elixir currently has. >>>>>>>> >>>>>>>> I appreciate any support and/or suggestions with this. I feel >>>>>>>> strongly about not having a "quiet" log level, as it causes a very >>>>>>>> painful >>>>>>>> balance for a library developer when determining how verbose to be with >>>>>>>> logging. >>>>>>>> >>>>>>>> Thanks in advance for all replies, feel free to ask questions! >>>>>>>> IW >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elixir-lang-core" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/606029ea-9855-44c6-ba94-69113ffed45c%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/606029ea-9855-44c6-ba94-69113ffed45c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elixir-lang-core" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/61adb36f-e448-46c2-abf9-b1a5757a14d3%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/61adb36f-e448-46c2-abf9-b1a5757a14d3%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elixir-lang-core/ddf53d78-124b-4adb-8f6f-6deb5a52d0eb%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elixir-lang-core/ddf53d78-124b-4adb-8f6f-6deb5a52d0eb%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/8ca8d1ce-8996-4b55-923b-f8e0b99bf054%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/8ca8d1ce-8996-4b55-923b-f8e0b99bf054%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Andrea Leopardi [email protected] -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAM9Rf%2BLh24O26-p2hcpPXDd2-%3DzKswYWp7%2BKRm-qhnB0gXFDUQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
