Thank you all. I will change the configuration file to and deploy next week.
Best regards Eyal Peer From: Chesnay Schepler <[email protected]> Sent: Wednesday, March 11, 2020 11:10 AM To: Yang Wang <[email protected]>; miki haiat <[email protected]> Cc: Eyal Pe'er <[email protected]>; Rafi Aroch <[email protected]>; user <[email protected]>; StartApp R&D Data Platform <[email protected]> Subject: Re: Setting app Flink logger @Eyal: The image you are using is for 1.9.2, but the logging configuration you fetched was from master. In 1.9.2 we use Log4j1, but on master we switched to Log4j2 instead, which uses a different configuration syntax. Log4j1 pretty much ignores the entire file, causing the error. Please use the configuration file from the release-1.9 branch. On 11/03/2020 03:50, Yang Wang wrote: Since you are using log4j2, the java dynamic property should not be "log4j.configuration". Please use "log4j.configurationFile" instead. Maybe it is not your problem, there is something wrong with the docker image. The log4j2 properties in "flink-console.sh" are not configured correctly. Best, Yang miki haiat <[email protected]<mailto:[email protected]>> 于2020年3月10日周二 下午11:50写道： Which image are you using ? On Tue, Mar 10, 2020, 16:27 Eyal Pe'er <[email protected]<mailto:[email protected]>> wrote: Hi Rafi, The file exists (and is the file from the official image☺, please see below). The user is root and it has permission. I am running in HA mode using docker. cat /opt/flink/conf/log4j-console.properties ################################################################################ # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ################################################################################ # This affects logging for both user code and Flink rootLogger.level = INFO rootLogger.appenderRef.console.ref = ConsoleAppender # Uncomment this if you want to _only_ change Flink's logging #log4j.logger.org.apache.flink=INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. logger.akka.name<http://logger.akka.name> = akka logger.akka.level = INFO logger.kafka.name<http://logger.kafka.name>= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name<http://logger.hadoop.name> = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name<http://logger.zookeeper.name> = org.apache.zookeeper logger.zookeeper.level = INFO # Log all infos to the console appender.console.name<http://appender.console.name> = ConsoleAppender appender.console.type = CONSOLE appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Suppress the irrelevant (wrong) warnings from the Netty channel handler logger.netty.name<http://logger.netty.name> = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF Best regards Eyal Peer / Data Platform Developer From: Rafi Aroch <[email protected]<mailto:[email protected]>> Sent: Tuesday, March 10, 2020 3:55 PM To: Eyal Pe'er <[email protected]<mailto:[email protected]>> Cc: user <[email protected]<mailto:[email protected]>>; StartApp R&D Data Platform <[email protected]<mailto:[email protected]>> Subject: Re: Setting app Flink logger Hi Eyal, Sounds trivial, but can you verify that the file actually exists in /opt/flink/conf/log4j-console.properties? Also, verify that the user running the process has read permissions to that file. You said you use Flink in YARN mode, but the the example above you run inside a docker image so this is a bit confusing. Notice that the official docker images run as "flink" user and group ids. If you wish to try to use Logback instead, you can place you logback.xml file as part of your project resources folder to include it in the classpath. That should automatically get detected on startup. Hope this helps, Rafi On Tue, Mar 10, 2020 at 1:42 PM Eyal Pe'er <[email protected]<mailto:[email protected]>> wrote: Hi, I am running Flink in YARN mode using the official image with few additional files. I’ve noticed that my logger failed to initialize: root:~# docker logs flink-task-manager Starting taskexecutor as a console application on host ***. log4j:WARN No appenders could be found for logger (org.apache.flink.runtime.taskexecutor.TaskManagerRunner). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. I followed the documentation<https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html> and seems like all related configuration files exist. Currently, I am using the default files from the official image https://github.com/apache/flink/tree/master/flink-dist/src/main/flink-bin/conf In addition, seems like the process got the right parameters: root 21892 21866 1 08:29 ? 00:02:06 /usr/local/openjdk-8/bin/java -XX:+UseG1GC -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties<file://opt/flink/conf/log4j-console.properties> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml<file://opt/flink/conf/logback-console.xml> -classpath /opt/flink/lib/flink-metrics-prometheus-1.9.2.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.2.jar:/opt/flink/lib/flink-table_2.11-1.9.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.9.2.jar::: org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir /opt/flink/conf What am I doing wrong ? How can I turn logger on? Best regards Eyal Peer
