On 18.07.2017 21:27, Lars Kellogg-Stedman wrote: > Our current model for logging in a containerized deployment has pretty > much everything logging to files in a directory that has been > bind-mounted from the host. This has some advantages: primarily, it > makes it easy for an operator on the local system to find logs, > particularly if they have had some previous exposure to > non-containerized deployments. > > There is strong demand for a centralized logging solution. We've got > one potential solution right now in the form of the fluentd service > introduced in Newton, but this requires explicit registration of log > files for every service. I don't think it's an ideal solution, and I > would like to explore some alternatives. > > Logging via syslog > ============== > > For the purposes of the following, I'm going to assume that we're > deploying on an EL-variant (RHEL/CentOS/etc), which means (a) journald > owns /dev/log and (b) we're running rsyslog on the host and using the > omjournal plugin to read messages from journald. > > If we bind mount /dev/log into containers and configure openstack > services to log via syslog rather than via files, we get the following > for free: > > - We get message-based rather than line-based logging. This means that > multiline tracebacks are handled correctly. > > - A single point of collection for logs. If your host has been > configured to ship logs to a centralized collector, logs from all of > your services will be sent there without any additional configuration. > > - We get per-service message rate limiting from journald. > > - Log messages are annotated by journald with a variety of useful > metadata, including the container id and a high resolution timestamp. > > - We can configure the syslog service on the host to continue to write > files into legacy locations, so an operator looking to run grep against > local log files will still have that ability. > > - Ryslog itself can send structured messages directly to an Elastic > instance, which means that in a many deployments we would not require > fluentd and its dependencies. > > - This plays well in environments where some services are running in > containers and others are running on the host, because everything simply > logs to /dev/log.
Plus it solves the log rotation (still have to be addressed [0] for Pike though), out-of-box. > > Logging via stdin/stdout > ================== > > A common pattern in the container world is to log everything to > stdout/stderr. This has some of the advantages of the above: > > - We can configure the container orchestration service to send logs to > the journal or to another collector. > > - We get a different set of annotations on log messages. > > - This solution may play better with frameworks like Kubernetes that > tend to isolate containers from the host a little more than using Docker > or similar tools straight out of the box. > > But there are some disadvantages: > > - Some services only know how to log via syslog (e.g., swift and haproxy) > > - We're back to line-based vs. message-based logging. > > - It ends up being more difficult to expose logs at legacy locations. > > - The container orchestration layer may not implement the same message > rate limiting we get with fluentd. > > Based on the above, I would like to suggest exploring a syslog-based > logging model moving forward. What do people think about this idea? I've > started putting together a spec > at https://review.openstack.org/#/c/484922/ and I would welcome your input. My vote goes for this option, but TBD for Queens. It won't make it for Pike, as it looks too late for such amount of drastic changes, like switching all OpenStack services to syslog, deploying additional required components, and so on. [0] https://bugs.launchpad.net/tripleo/+bug/1700912 [1] https://review.openstack.org/#/c/462900/ > > Cheers, > > -- > Lars Kellogg-Stedman <l...@redhat.com <mailto:l...@redhat.com>> > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Bogdan Dobrelya, Irc #bogdando __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
