Thanks Wenhao for referring the JMX solution. I believe it is a good alternative for who they do not have the ssh permission of the production machines.
Best, Yang Wenhao Ji <predator....@gmail.com> 于2022年1月16日周日 20:20写道： > The FLIP has been marked as rejected. > > Anyone landing here looking for a solution to change the log level > without modifying configuration files may use the existing JMX APIs > exposed by log4j2 [1] or logback [2]. > > [1]: https://logging.apache.org/log4j/2.x/manual/jmx.html > [2]: https://logback.qos.ch/manual/jmxConfig.html > > Thanks, > Wenhao > > On Fri, Jan 14, 2022 at 12:12 AM Konstantin Knauf > <konstan...@ververica.com> wrote: > > > > Thanks, Wenhao. > > > > On Thu, Jan 13, 2022 at 4:19 PM Wenhao Ji <predator....@gmail.com> > wrote: > > > > > It seems that we have reached a consensus that the proposal will not > > > be implemented in Flink. I will mark the FLIP as discarded if there > > > are no objections. > > > > > > Thanks, everyone, for joining the discussion again! > > > > > > Wenhao > > > > > > > > > On Tue, Jan 11, 2022 at 11:12 PM Wenhao Ji <predator....@gmail.com> > wrote: > > > > > > > > Hi all, > > > > > > > > Yes, indeed. > > > > After I did some investigation on similar features provided by the > > > > Cloud platforms, I actually found several popular Clouds have already > > > > offered this. > > > > > > > > - AWS Kinesis: Setting the Application Logging Level [1], which is > > > > implemented by UpdateApplication API [2]. > > > > - Ververica: Logging & Metrics[3], by changing the template. > > > > - Alicloud: Configure job logs [4], which is quite similar to > > > > Ververica also by changing the template > > > > - Cloudera: Enabling Flink DEBUG logging[5], by changing the > > > > Configuration and triggering a restart > > > > > > > > It looks like this feature is not necessary. It has been developed in > > > > one way or another by so many platforms in the ecosystem. > > > > > > > > [1]: > > > > https://docs.aws.amazon.com/kinesisanalytics/latest/java/cloudwatch-logs.html#cloudwatch-level > > > > [2]: > > > > https://docs.aws.amazon.com/kinesisanalytics/latest/apiv2/API_UpdateApplication.html > > > > [3]: > https://docs.ververica.com/platform_operations/logging_metrics.html > > > > [4]: > > > > https://www.alibabacloud.com/help/doc-detail/173646.htm#title-1ay-hju-pka > > > > [5]: > > > > https://docs.cloudera.com/csa/1.6.0/monitoring/topics/csa-ssb-enabling-debug-logging.html > > > > > > > > Thanks, > > > > Wenhao > > > > > > > > On Tue, Jan 11, 2022 at 8:24 PM Martijn Visser < > mart...@ververica.com> > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > I agree with Konstantin, this feels like a problem that shouldn't > be > > > solved > > > > > via Apache Flink but via the logging ecosystem itself. > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > On Tue, 11 Jan 2022 at 13:11, Konstantin Knauf <kna...@apache.org> > > > wrote: > > > > > > > > > > > I've now read over the discussion on the ticket, and I am > personally > > > not in > > > > > > favor of adding this functionality to Flink via the REST API or > Web > > > UI. I > > > > > > believe that changing the logging configuration via the existing > > > > > > configuration files (log4j or logback) is good enough, to > justify not > > > > > > increasing the scope of Flink in that direction. As you > specifically > > > > > > mention YARN: doesn't Cloudera's Hadoop platform, for example, > offer > > > means > > > > > > to manage the configuration files for all worker nodes from a > central > > > > > > configuration management system? It overall feels like we are > trying > > > to > > > > > > solve a problem in Apache Flink that is already solved in its > > > ecosystem and > > > > > > increases the scope of the project without adding core value. I > also > > > expect > > > > > > that over time the exposed logging configuration options would > > > become more > > > > > > and more complex. > > > > > > > > > > > > I am curious to hear what others think. > > > > > > > > > > > > On Tue, Jan 11, 2022 at 10:34 AM Chesnay Schepler < > > > ches...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Reloading the config from the filesystem is already enabled by > > > default; > > > > > > > that was one of the things that made us switch to Log4j 2. > > > > > > > > > > > > > > The core point of contention w.r.t. this topic is whether > having > > > the > > > > > > > admin ssh into the machine is too inconvenient. > > > > > > > > > > > > > > Personally I still think that the the current capabilities are > > > > > > > sufficient, and I do not want us to rely on internals of the > > > logging > > > > > > > backends in production code. > > > > > > > > > > > > > > On 10/01/2022 17:26, Konstantin Knauf wrote: > > > > > > > > Thank you for starting the discussion. Being able to change > the > > > logging > > > > > > > > level at runtime is very valuable in my experience. > > > > > > > > > > > > > > > > Instead of introducing our own API (and eventually even > > > persistence), > > > > > > > could > > > > > > > > we just periodically reload the log4j or logback > configuration > > > from the > > > > > > > > environment/filesystem? I only quickly googled the topic and > > > [1,2] > > > > > > > suggest > > > > > > > > that this might be possible? > > > > > > > > > > > > > > > > [1] https://stackoverflow.com/a/16216956/6422562? > > > > > > > > [2] > https://logback.qos.ch/manual/configuration.html#autoScan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 10, 2022 at 5:10 PM Wenhao Ji < > > > predator....@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > >> Hi everyone, > > > > > > > >> > > > > > > > >> Hope you enjoyed the Holiday Season. > > > > > > > >> > > > > > > > >> I would like to start the discussion on the improvement > purpose > > > > > > > >> FLIP-210 [1] which aims to provide a way to change log > levels at > > > > > > > >> runtime to simplify issues and bugs detection as reported > in the > > > > > > > >> ticket FLINK-16478 [2]. > > > > > > > >> Firstly, thanks Xingxing Di and xiaodao for their previous > > > effort. The > > > > > > > >> FLIP I drafted is largely influenced by their previous > designs > > > [3][4]. > > > > > > > >> Although we have reached some agreements under the jira > > > comments about > > > > > > > >> the scope of this feature, we still have the following > questions > > > > > > > >> listed below ready to be discussed in this thread. > > > > > > > >> > > > > > > > >> ## Question 1 > > > > > > > >> > > > > > > > >>> Creating as custom DSL and implementing it for several > logging > > > > > > backend > > > > > > > >> sounds like quite a maintenance burden. Extensions to the > DSL, > > > and > > > > > > > >> supported backends, could become quite an effort. (by > Chesnay > > > > > > Schepler) > > > > > > > >> > > > > > > > >> I tried to design the API of the logging backend to stay > away > > > from the > > > > > > > >> details of implementations but I did not find any > > > slf4j-specific API > > > > > > > >> that is available to change the log level of a logger. So > what > > > I did > > > > > > > >> is to introduce another kind of abstraction on top of the > slf4j > > > / > > > > > > > >> log4j / logback so that we will not depend on the logging > > > provider's > > > > > > > >> api directly. It will be convenient for us to adopt any > other > > > logging > > > > > > > >> providers. Please see the "Logging Abstraction" section. > > > > > > > >> > > > > > > > >> ## Question 2 > > > > > > > >> > > > > > > > >>> Do we know whether other systems support this kind of > feature? > > > If > > > > > > yes, > > > > > > > >> how do they solve it for different logging backends? (by > Till > > > > > > Rohrmann) > > > > > > > >> > > > > > > > >> I investigated several Java frameworks including Spark, > Storm, > > > and > > > > > > > >> Spring Boot. Here is what I found. > > > > > > > >> Spark & Storm directly depend on the log4j implementations, > > > which > > > > > > > >> means they do not support any other slf4j implementation at > > > all. They > > > > > > > >> simply call the log4j api directly. (see > > > SparkContext.scala#L381 [5], > > > > > > > >> Utils.scala#L2443 [6] in Spark, and > LogConfigManager.java#L144 > > > [7] in > > > > > > > >> Storm). They are pretty different from what Flink provides. > > > > > > > >> However, I found Spring Boot has implemented what we are > > > interested > > > > > > > >> in. Just as Flink, Spring boot also supports many slf4j > > > > > > > >> implementations. Users are not limited to log4j. They have > the > > > ability > > > > > > > >> to declare different logging frameworks by importing certain > > > > > > > >> dependencies. After that spring will decide the activated > one by > > > > > > > >> scanning its classpath and context. (see > > > LoggingSystem.java#L164 [8] > > > > > > > >> and LoggersEndpoint.java#L99 [9]) > > > > > > > >> > > > > > > > >> ## Question 3 > > > > > > > >> > > > > > > > >> Besides the questions raised in the jira comments, I also > find > > > another > > > > > > > >> thing that has not been discussed. Considering this feature > as > > > an MVP, > > > > > > > >> do we need to introduce a HighAvailabilityService to store > the > > > log > > > > > > > >> settings so that they can be synced to newly-joined task > > > managers and > > > > > > > >> also job manager followers for consistency? This issue is > > > included in > > > > > > > >> the "Limitations" section in the flip. > > > > > > > >> > > > > > > > >> Finally, thanks for your time for joining this discussion > and > > > > > > > >> reviewing this FLIP. I would appreciate it if you could > have any > > > > > > > >> comments or suggestions on this. > > > > > > > >> > > > > > > > >> > > > > > > > >> [1]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-210%3A+Change+logging+level+dynamically+at+runtime > > > > > > > >> [2]: https://issues.apache.org/jira/browse/FLINK-16478 > > > > > > > >> [3]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Q02VSSBzlZaZzvxuChIo1uinw8KDQsyTZUut6_IDErY > > > > > > > >> [4]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/19AyuTHeERP6JKmtHYnCdBw29LnZpRkbTS7K12q4OfbA > > > > > > > >> [5]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://github.com/apache/spark/blob/11596b3b17b5e0f54e104cd49b1397c33c34719d/core/src/main/scala/org/apache/spark/SparkContext.scala#L381 > > > > > > > >> [6]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://github.com/apache/spark/blob/11596b3b17b5e0f54e104cd49b1397c33c34719d/core/src/main/scala/org/apache/spark/util/Utils.scala#L2433 > > > > > > > >> [7]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://github.com/apache/storm/blob/3f96c249cbc17ce062491bfbb39d484e241ab168/storm-client/src/jvm/org/apache/storm/daemon/worker/LogConfigManager.java#L144 > > > > > > > >> [8]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/LoggingSystem.java#L164 > > > > > > > >> [9]: > > > > > > > >> > > > > > > > > > > > > > > > > > https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/logging/LoggersEndpoint.java#L99 > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> Wenhao > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Konstantin Knauf > > > > > > > > > > > > https://twitter.com/snntrable > > > > > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > > > > -- > > > > Konstantin Knauf | Head of Product > > > > +49 160 91394525 > > > > > > Follow us @VervericaData Ververica <https://www.ververica.com/> > > > > > > -- > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > Conference > > > > Stream Processing | Event Driven | Real Time > > > > -- > > > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > > > -- > > Ververica GmbH > > Registered at Amtsgericht Charlottenburg: HRB 158244 B > > Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason, > > Jinwei (Kevin) Zhang >
