I looked at Bunyan, and it probably closer to what I'm looker for than anything else, but it still doesn't solve the problem of configuring logging output from nested reusable packages.
For simple applications, you're right that libraries should not make noise, but when working with larger applications and libraries, I've found it very useful to have separate log files that show what each component is doing (at info and trace levels, separately). This is what I've found lacking in the Node world right now, and this is what I'm trying to create. This would be intended not for normal command-line tools, but for long-running servers or generators. (things that run for long enough to need diagnostics) My goal is that popular libraries (especially things like Socket.io) continue to work as you described for simple usage, but can then be easily plugged in to an application-level logging framework when necessary. JSON certainly is the best way to output detailed logs (especially with context), although I will probably also have a simpler output format that can be read directly from the console. I should probably have separate output formats optimized to work with various existing tools like Bunyan's viewer, Kibana / logstash, or pupergrep. But basically, my whole goal here is to provide a framework that is meant to be used by libraries as well. Do you think that fits in with the Node philosophy? Are there any problems with this approach that I'm not seeing? On Friday, April 19, 2013 11:58:37 AM UTC-4, Isaac Schlueter wrote: > > Check out Bunyan. That's what Joyent uses for all our various > services and such. You can configure separate loggers for each > component, and it just spews json to stdio. > > Bunyan also comes along with a command-line utility that consumes > bunyan json and outputs something pretty and easier for humans to look > at. You can use that util to find all the 304 responses your site > sent, or all the warnings, etc. And because the logs are just > one-line-per-message json, they're easy to consume with whatever other > tools you want. They gzip well, and can be rotated out when they get > large, etc. > > I would *not* recommend anything as opinionated as Bunyan for most > command line tools, and certainly libs and reusable modules should be > 100% silent unless it is their primary job to make noise (or something > disastrous is happening!) But for the situation you describe, Bunyan > sounds about perfect. > > > On Thu, Apr 18, 2013 at 10:03 PM, Schabse Laks > <[email protected]<javascript:>> > wrote: > > <tldr> > > I'm creating a reusable and externally configurable logging framework, > > intended to be used by both libraries and applications. > > What features would you want to see in it? > > </tldr> > > > > What I want > > I'm running a large Node.js website (https://unroll.me), and I want to > > configure logging for each of the components used. (eg, put all trace > from > > each package in its own file, and have a single file with info or > higher). > > Coming to the Node.js world from Java, I really miss the existence of a > > widespread and externally configurable logging framework like Logback > (with > > SLF4J). > > > > In my mind, a good logging framework should: > > > > Have a separate emitter component which would be used by (ideally, all) > > libraries (npm packages) that want to log things > > Be version resilient (if the application uses v1.2, and it references > > different libraries that use v1.1 and v1.3, everything should still > work) > > Include the package (and ideally file) that emitted each log message in > each > > line of output > > Allow applications (entry points) to configure log levels and outputs > > (appenders, transports, whatever you call them) for logs from different > > packages, with different outputs, filters, and log levels for different > > packages > > (Ideally) Allow this configuration to be changed at runtime without > > restarting (so you can start gathering additional log data when things > start > > going wrong) > > Allow both applications and libraries to attach context information that > > will flow down asynchronous call chains and be associated with all log > > messages made in that context, such as user ID or HTTP request ID. > (like > > Logback's MDC) > > Be runnable in a browser (with and without browserify), logging to the > JS > > console or to a server over websockets (the server should then run it > > through its regular logging outputs) > > Not send logged messages from the browser if the server isn't writing > those > > levels (per component) > > Include the package and file name even from browserified packages > > Support existing Winston transports > > Be easily pluggable into older libraries that take a log function as a > > config parameter (eg, Socket.io) > > Have a JSON output writer coupled to a web-based streaming viewer for > nice > > filtering and grouping (like pupergrep), which should: > > > > Highlight related messages when hovering a message (by context key) > > Show a menu to filter by context key when clicking a message > > Stream new events over web sockets, either by reading a file live or by > a > > separate appender directly from the program being logged > > Support search-as-you-type > > Aggregate logs from multiple programs or servers (interleaved by > timestamp) > > Show different groups of messages (by source or context key or package) > > side-by-side > > Use bookmarkable / sharable URLs for filtered and/or timeranged views > > I will only start writing this after I finish everything else; it's much > > less urgent for us. > > > > Why I feel the need to re-invent the wheel > > The important feature that no existing framework (that I've been able to > > find) is the ability to use it from a library. > > This means three things: > > > > You must be able to create a logger by specifying a name only, and have > it > > pick up settings from a central (application-level) configuration > > Loggers created by different versions must still share configuration (in > > case different referenced packages reference different versions of the > > logging framework > > Log messages emitted from each package must include the name of the > package > > > > Winston and Bunyan come close, but they don't have any kind of > configuration > > mechanism or global storage, making them impractical to use in nested > > library packages. > > > > I also haven't found any libraries that support contextual data. > > > > Where I want this to end up > > If done right, this should end up being used by all reusable libraries > so > > that applications can easily capture their log outputs. > > For example, today, if I write a reusable library that wraps socket.io, > > there is no decent way for consumers of my library to handle my log > output > > and socket.io's log output differently (unless the library takes lots > of > > config log functions) > > > > To this end, the library should have two separate npm packages: > > > > An emitter, used by libraries to emit log messages (like SLF4J in the > Java > > world) > > A sink / server / host (I'm trying to find the best term), used by > > applications to configure how messages from different emitters are > consumed > > (console, file, etc) > > > > Applications would also use the emitter to perform application-level > > logging; the second package would only be used at launch time to set > things > > up. > > > > For when the emitter is used in a process that has not set up the sink / > > host, the emitter should have a basic console output built-in (probably > only > > logging info or higher). > > This means that people can get started with libraries that use the > > framework, without having to add extra packages or configure anything. > > It should probably also print a message instructing people to install > the > > second component to fully configure logging. > > > > What I'm asking > > > > Is there an existing solution for this that I'm missing? > > What features would you want to see in this? > > Do you disagree with any parts of this design? What would you change? > > Once I finish writing & testing it, what would it take to get this used > by > > most libraries? (to replace the more-annoying log callback config > option) > > Do you want a pony? (no guarantees; sorry) > > > > -- > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to [email protected]<javascript:> > > To unsubscribe from this group, send email to > > [email protected] <javascript:> > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "nodejs" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
