Hi all, Yes, indeed. After I did some investigation on similar features provided by the Cloud platforms, I actually found several popular Clouds have already offered this.
- AWS Kinesis: Setting the Application Logging Level [1], which is implemented by UpdateApplication API [2]. - Ververica: Logging & Metrics[3], by changing the template. - Alicloud: Configure job logs [4], which is quite similar to Ververica also by changing the template - Cloudera: Enabling Flink DEBUG logging[5], by changing the Configuration and triggering a restart It looks like this feature is not necessary. It has been developed in one way or another by so many platforms in the ecosystem. [1]: https://docs.aws.amazon.com/kinesisanalytics/latest/java/cloudwatch-logs.html#cloudwatch-level [2]: https://docs.aws.amazon.com/kinesisanalytics/latest/apiv2/API_UpdateApplication.html [3]: https://docs.ververica.com/platform_operations/logging_metrics.html [4]: https://www.alibabacloud.com/help/doc-detail/173646.htm#title-1ay-hju-pka [5]: https://docs.cloudera.com/csa/1.6.0/monitoring/topics/csa-ssb-enabling-debug-logging.html Thanks, Wenhao On Tue, Jan 11, 2022 at 8:24 PM Martijn Visser <mart...@ververica.com> wrote: > > Hi all, > > I agree with Konstantin, this feels like a problem that shouldn't be solved > via Apache Flink but via the logging ecosystem itself. > > Best regards, > > Martijn > > On Tue, 11 Jan 2022 at 13:11, Konstantin Knauf <kna...@apache.org> wrote: > > > I've now read over the discussion on the ticket, and I am personally not in > > favor of adding this functionality to Flink via the REST API or Web UI. I > > believe that changing the logging configuration via the existing > > configuration files (log4j or logback) is good enough, to justify not > > increasing the scope of Flink in that direction. As you specifically > > mention YARN: doesn't Cloudera's Hadoop platform, for example, offer means > > to manage the configuration files for all worker nodes from a central > > configuration management system? It overall feels like we are trying to > > solve a problem in Apache Flink that is already solved in its ecosystem and > > increases the scope of the project without adding core value. I also expect > > that over time the exposed logging configuration options would become more > > and more complex. > > > > I am curious to hear what others think. > > > > On Tue, Jan 11, 2022 at 10:34 AM Chesnay Schepler <ches...@apache.org> > > wrote: > > > > > Reloading the config from the filesystem is already enabled by default; > > > that was one of the things that made us switch to Log4j 2. > > > > > > The core point of contention w.r.t. this topic is whether having the > > > admin ssh into the machine is too inconvenient. > > > > > > Personally I still think that the the current capabilities are > > > sufficient, and I do not want us to rely on internals of the logging > > > backends in production code. > > > > > > On 10/01/2022 17:26, Konstantin Knauf wrote: > > > > Thank you for starting the discussion. Being able to change the logging > > > > level at runtime is very valuable in my experience. > > > > > > > > Instead of introducing our own API (and eventually even persistence), > > > could > > > > we just periodically reload the log4j or logback configuration from the > > > > environment/filesystem? I only quickly googled the topic and [1,2] > > > suggest > > > > that this might be possible? > > > > > > > > [1] https://stackoverflow.com/a/16216956/6422562? > > > > [2] https://logback.qos.ch/manual/configuration.html#autoScan > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 10, 2022 at 5:10 PM Wenhao Ji <predator....@gmail.com> > > > wrote: > > > > > > > >> Hi everyone, > > > >> > > > >> Hope you enjoyed the Holiday Season. > > > >> > > > >> I would like to start the discussion on the improvement purpose > > > >> FLIP-210 [1] which aims to provide a way to change log levels at > > > >> runtime to simplify issues and bugs detection as reported in the > > > >> ticket FLINK-16478 [2]. > > > >> Firstly, thanks Xingxing Di and xiaodao for their previous effort. The > > > >> FLIP I drafted is largely influenced by their previous designs [3][4]. > > > >> Although we have reached some agreements under the jira comments about > > > >> the scope of this feature, we still have the following questions > > > >> listed below ready to be discussed in this thread. > > > >> > > > >> ## Question 1 > > > >> > > > >>> Creating as custom DSL and implementing it for several logging > > backend > > > >> sounds like quite a maintenance burden. Extensions to the DSL, and > > > >> supported backends, could become quite an effort. (by Chesnay > > Schepler) > > > >> > > > >> I tried to design the API of the logging backend to stay away from the > > > >> details of implementations but I did not find any slf4j-specific API > > > >> that is available to change the log level of a logger. So what I did > > > >> is to introduce another kind of abstraction on top of the slf4j / > > > >> log4j / logback so that we will not depend on the logging provider's > > > >> api directly. It will be convenient for us to adopt any other logging > > > >> providers. Please see the "Logging Abstraction" section. > > > >> > > > >> ## Question 2 > > > >> > > > >>> Do we know whether other systems support this kind of feature? If > > yes, > > > >> how do they solve it for different logging backends? (by Till > > Rohrmann) > > > >> > > > >> I investigated several Java frameworks including Spark, Storm, and > > > >> Spring Boot. Here is what I found. > > > >> Spark & Storm directly depend on the log4j implementations, which > > > >> means they do not support any other slf4j implementation at all. They > > > >> simply call the log4j api directly. (see SparkContext.scala#L381 [5], > > > >> Utils.scala#L2443 [6] in Spark, and LogConfigManager.java#L144 [7] in > > > >> Storm). They are pretty different from what Flink provides. > > > >> However, I found Spring Boot has implemented what we are interested > > > >> in. Just as Flink, Spring boot also supports many slf4j > > > >> implementations. Users are not limited to log4j. They have the ability > > > >> to declare different logging frameworks by importing certain > > > >> dependencies. After that spring will decide the activated one by > > > >> scanning its classpath and context. (see LoggingSystem.java#L164 [8] > > > >> and LoggersEndpoint.java#L99 [9]) > > > >> > > > >> ## Question 3 > > > >> > > > >> Besides the questions raised in the jira comments, I also find another > > > >> thing that has not been discussed. Considering this feature as an MVP, > > > >> do we need to introduce a HighAvailabilityService to store the log > > > >> settings so that they can be synced to newly-joined task managers and > > > >> also job manager followers for consistency? This issue is included in > > > >> the "Limitations" section in the flip. > > > >> > > > >> Finally, thanks for your time for joining this discussion and > > > >> reviewing this FLIP. I would appreciate it if you could have any > > > >> comments or suggestions on this. > > > >> > > > >> > > > >> [1]: > > > >> > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-210%3A+Change+logging+level+dynamically+at+runtime > > > >> [2]: https://issues.apache.org/jira/browse/FLINK-16478 > > > >> [3]: > > > >> > > > > > https://docs.google.com/document/d/1Q02VSSBzlZaZzvxuChIo1uinw8KDQsyTZUut6_IDErY > > > >> [4]: > > > >> > > > > > https://docs.google.com/document/d/19AyuTHeERP6JKmtHYnCdBw29LnZpRkbTS7K12q4OfbA > > > >> [5]: > > > >> > > > > > https://github.com/apache/spark/blob/11596b3b17b5e0f54e104cd49b1397c33c34719d/core/src/main/scala/org/apache/spark/SparkContext.scala#L381 > > > >> [6]: > > > >> > > > > > https://github.com/apache/spark/blob/11596b3b17b5e0f54e104cd49b1397c33c34719d/core/src/main/scala/org/apache/spark/util/Utils.scala#L2433 > > > >> [7]: > > > >> > > > > > https://github.com/apache/storm/blob/3f96c249cbc17ce062491bfbb39d484e241ab168/storm-client/src/jvm/org/apache/storm/daemon/worker/LogConfigManager.java#L144 > > > >> [8]: > > > >> > > > > > https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/LoggingSystem.java#L164 > > > >> [9]: > > > >> > > > > > https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/logging/LoggersEndpoint.java#L99 > > > >> > > > >> Thanks, > > > >> Wenhao > > > >> > > > > > > > > > > > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > >
