>From my previous experience with K8s, I'd assume that the cluster itself already has some ELK attached to it and all stdout/err is collected automatically.
So if you want to also add logs, I'd make that configurable and I'm torn what the default should be as both make sense. However, thinking a bit further, it sounds as if the current way is already very inconsistent. We should probably either have log files for everything or log everything to stdout/err and let K8s deal with it. And then we should have some configuration to toggle between the modes. On Tue, May 5, 2020 at 2:50 PM Stephan Ewen <se...@apache.org> wrote: > @Patrick could you chime in? > > We should at least understand the original motivation before simply > changing the way it works. > > On Tue, May 5, 2020 at 2:49 PM Stephan Ewen <se...@apache.org> wrote: > > > I think Patrick originally introduced the foreground mode, and I believe > > it had indeed something to do with container use and logging. > > > > IIRC the default assumption in docker and Kubernetes is that the logs > come > > on stdout (or stderr) so after "principle of least astonishment" the idea > > was to give a similar experience with Flink. > > > > On Tue, May 5, 2020 at 10:49 AM David Anderson <da...@alpinegizmo.com> > > wrote: > > > >> I like this idea because it should improve the experience (and reduce > >> confusion) for folks having their first flink experience via one of the > >> docker playgrounds. Right now it gives the impression that something is > >> broken out-of-the-box. > >> > >> Regards, > >> David > >> > >> On Mon, May 4, 2020 at 6:01 PM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >> > >> > Hi everyone, > >> > > >> > thanks for starting this discussion Chesnay. > >> > > >> > I think it would be nice if we also displayed the logs when starting > the > >> > process in the foreground. > >> > > >> > The repercussions could be mitigated if the default logger > >> configurations > >> > would contain file rolling with a max log file size. > >> > > >> > @Yang I think there are solutions how to redirect stdout and stderr > into > >> > separate files using tee without duplication [1]. > >> > > >> > [1] http://www.softpanorama.org/Tools/tee.shtml > >> > > >> > Cheers, > >> > Till > >> > > >> > On Wed, Apr 29, 2020 at 4:28 AM Yang Wang <danrtsey...@gmail.com> > >> wrote: > >> > > >> > > Thanks for Chesnay starting this discussion. > >> > > > >> > > In FLINK-17166 implementation[1], we are trying to use "tee" instead > >> of > >> > > introducing the stream redirection(redirect the out/err to files). > >> > However, > >> > > a side effect is that the logging will be duplicated both in .log > and > >> > .out > >> > > files. > >> > > Then it may consume more disk space. However it is not a very > critical > >> > > problem since we could use log4j/logback configuration to control > the > >> > > rolling > >> > > files and max size. > >> > > > >> > > Also, it only happens in docker/K8s deployment. For YARN/Mesos > >> > deployment, > >> > > the behavior is just same as before. > >> > > > >> > > > >> > > [1]. https://github.com/apache/flink/pull/11839 > >> > > > >> > > Best, > >> > > Yang > >> > > > >> > > Chesnay Schepler <ches...@apache.org> 于2020年4月29日周三 上午12:30写道： > >> > > > >> > > > Currently, processes started in the foreground (like in the case > of > >> > > > Docker) output all logging/stdout directly to the console, without > >> > > > creating any logging files. > >> > > > > >> > > > The downside of this approach, as outlined in FLIP-111, is that > the > >> > > > WebUI is not able to display the logs since it relies on these > very > >> > > > files to exist. > >> > > > > >> > > > In FLINK-17166 (part of FLIP-111) we are trying to change this > such > >> > that > >> > > > we always created .log/.out files. It seems like a reasonable > >> change to > >> > > > do, but it could have repercussions on existing deployments since > we > >> > > > will naturally use more disk space (logs gotta go somewhere). > >> > > > > >> > > > I'm curious what people think about this. > >> > > > > >> > > > > >> > > > >> > > >> > > > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
