By default, docker collects stdout and stderr of the containerized process. Docker doesn't know about the log format of the application, it can be python logging, glog, log4j or whatever, it only known the stream this message came from.
Fluentd collects information from docker, including what stream this message has been written to. It also doesn't know by default what is the format of these logs, only the stream. At this point it decides to mark messages sent to stdout as INFO and messages sent to stderr as ERROR. This is a known problem, for example for glog all messages used to be ingested as errors (glog logs everything to stderr). To make further distinction, you should teach fluentd to parse your custom format. Example of such change: https://github.com/kubernetes/kubernetes/pull/39151 Unfortunately, right now changing the configuration of your fluentd instance can be really clumsy for two reasons: 1. Fluentd configuration is embedded into image, you have to make your own fluentd image with the new configuration. 2. Fluentd is distributed in cluster as static pod, it's bundled with a node basically, it's hard to change the configuration of every fluentd instance at once Both problems are being addressed in Kubernetes 1.6, there will be detailed instructions for ingesting your logs' format somewhere there: http://k8s.io/docs/user-guide/logging/overview Also, in 1.6 glog format will be ingested by default, if you're interested. Do you want assistance right now or are you ready to wait for 1.6 to have a much easier and a well-documented solution? ETA for 1.6 is the end of March On Fri, Feb 17, 2017 at 11:14 AM, Filip Grzadkowski <[email protected]> wrote: > +Mik Vyatskov > +Piotr Szczesniak > > -- > Filip > > On Fri, Feb 17, 2017 at 7:24 AM, Michael Dillon <[email protected]> > wrote: >> >> I'm using a GKE cluster and all of my container log messages appear in the >> Google (or Stackdriver?) logging dashboard with no problems. >> >> However, messages are only tagged with 'ERROR' or 'INFO'. For example, a >> 'DEBUG' message is always tagged with a severity of 'error'. I frequently >> use warning and debug and things aren't integrated as well as I'd like. >> >> Is there a specific log format I should output in my containers? >> >> Thanks! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
