Thanks! I suspected the fluentd config on the node was responsible but it seemed strange to meddle with it since google manages those instances... I can wait until 1.6
On Friday, February 17, 2017 at 3:40:50 AM UTC-7, Mik Vyatskov wrote: > > By default, docker collects stdout and stderr of the containerized > process. Docker doesn't know about the log format of the application, > it can be python logging, glog, log4j or whatever, it only known the > stream this message came from. > > Fluentd collects information from docker, including what stream this > message has been written to. It also doesn't know by default what is > the format of these logs, only the stream. At this point it decides to > mark messages sent to stdout as INFO and messages sent to stderr as > ERROR. This is a known problem, for example for glog all messages used > to be ingested as errors (glog logs everything to stderr). > > To make further distinction, you should teach fluentd to parse your > custom format. Example of such change: > https://github.com/kubernetes/kubernetes/pull/39151 > > Unfortunately, right now changing the configuration of your fluentd > instance can be really clumsy for two reasons: > > 1. Fluentd configuration is embedded into image, you have to make your > own fluentd image with the new configuration. > 2. Fluentd is distributed in cluster as static pod, it's bundled with > a node basically, it's hard to change the configuration of every > fluentd instance at once > > Both problems are being addressed in Kubernetes 1.6, there will be > detailed instructions for ingesting your logs' format somewhere there: > http://k8s.io/docs/user-guide/logging/overview Also, in 1.6 glog > format will be ingested by default, if you're interested. > > Do you want assistance right now or are you ready to wait for 1.6 to > have a much easier and a well-documented solution? ETA for 1.6 is the > end of March > > On Fri, Feb 17, 2017 at 11:14 AM, Filip Grzadkowski <[email protected] > <javascript:>> wrote: > > +Mik Vyatskov > > +Piotr Szczesniak > > > > -- > > Filip > > > > On Fri, Feb 17, 2017 at 7:24 AM, Michael Dillon <[email protected] > <javascript:>> > > wrote: > >> > >> I'm using a GKE cluster and all of my container log messages appear in > the > >> Google (or Stackdriver?) logging dashboard with no problems. > >> > >> However, messages are only tagged with 'ERROR' or 'INFO'. For example, > a > >> 'DEBUG' message is always tagged with a severity of 'error'. I > frequently > >> use warning and debug and things aren't integrated as well as I'd like. > >> > >> Is there a specific log format I should output in my containers? > >> > >> Thanks! > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Kubernetes user discussion and Q&A" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected] <javascript:>. > >> To post to this group, send email to [email protected] > <javascript:>. > >> Visit this group at https://groups.google.com/group/kubernetes-users. > >> For more options, visit https://groups.google.com/d/optout. > > > > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
