Hey I encountered the same issue while upgrading from v1.12.2 to 1.14.2 few weeks ago. starting from v1.13.0, deployment has been changed in K8s native mode.(detailed about its flow [1])
user input properties get overriden in flink-console.sh. which means, pointing to xml file(-Dlog4j2....) gets ignored. I created a ticket[2] and open MR with a fix[3] unfortunately, E2E keep failing(Not sure whether it's related to change or not, to be honest, did not have time to check the E2E pipeline) [1] https://lists.apache.org/thread/b24g1nd00q5pln5h9w2mh1s3ocxwb61b [2] https://issues.apache.org/jira/browse/FLINK-25762 [3] https://github.com/apache/flink/pull/18447/files# Hope it helps, Tamir ________________________________ From: Chesnay Schepler <ches...@apache.org> Sent: Wednesday, February 16, 2022 11:15 AM To: jonas eyob <jonas.e...@gmail.com> Cc: user <user@flink.apache.org> Subject: Re: Log4j2 configuration EXTERNAL EMAIL hmm....yes then it is indeed weird that it can't find the logger, but their error messages are notorious for being misleading in my experience. Can you set the log4j2.debug system property (to any value, even an empty string) and try again? If that doesn't reveal anything I would try the following: * run a job without bundling any logging classes * if that works, add one jar at a time to the fat jar and see if it continues to work * check what happens if the logging classes are added to lib/ instead of being bundled in the fat jar You can also test all of this with a standalone cluster and a dummy job if that makes things easier; I don't think there is anything Kubernetes-specific going on. On 15/02/2022 18:18, jonas eyob wrote: 1. Ok, thanks! 2. We are using application mode. No changes to the distribution other than updating the log4j-console.properties file. content of /lib/: * flink-csv-1.14.3.jar * flink-json-1.14.3.jar * flink-table_2.12-1.14.3.jar * log4j-api-2.17.1.jar * log4j-slf4j-impl-2.17.1.jar * flink-dist_2.12-1.14.3.jar * flink-shaded-zookeeper-3.4.14.jar * log4j-1.2-api-2.17.1.jar * log4j-core-2.17.1.jar Den tis 15 feb. 2022 kl 16:30 skrev Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>>: 1) You either need to modify the log4j-console.properties file, or explicitly set the log4j.configurationFile property to point to your .xml file. 2) Have you made modifications to the distribution (e.g., removing other logging jars from the lib directory)? Are you using application mode, or session clusters? On 15/02/2022 16:41, jonas eyob wrote: Hey, We are deploying our Flink Cluster on a standalone Kubernetes with the longrunning job written in scala. We recently upgraded our Flink cluster from 1.12 to 1.14.3 - after which we started seeing a few problems related to logging which I have been struggling to fix for the past days). Related is also an attempt to add, we are also attempting to add a Sentry integration for our error logs. PROBLEM 1 - Error logs not being sent to Sentry. We are bundling our code and dependencies into a FAT jar, which includes a log4j2.xml specifying the Sentry Appender. But if I understand the documentation<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/advanced/logging/#configuring-log4j-2> correctly our log4j2.xml won't be picked up by Flink as it already defines a set of default logging configurations files (e.g. log4j and logback). Q: How does Flink resolve logging configurations to use? I can see the following JVM override params provided when running in our dockerized version locally. -Dlog.file=/opt/flink/log/flink--taskexecutor-0-thoros-taskmanager-6b9785d4df-c28n4.log 2022-02-15 10:01:59,826 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties 2022-02-15 10:01:59,827 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties 2022-02-15 10:01:59,830 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml Content of the log4j2.xml (path: src/main/resources): <?xml version="1.0" encoding="UTF-8"?> <Configuration status="warn" packages="org.apache.logging.log4j.core,io.sentry.log4j2"> <Appenders> <Console name="Console" target="SYSTEM_OUT"> <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/> </Console> <Sentry name="Sentry" dsn="<DSN>" minimumEventLevel="ERROR" /> </Appenders> <Loggers> <Root level="info"> <AppenderRef ref="Sentry"/> <AppenderRef ref="Console"/> </Root> </Loggers> </Configuration> For our kubernetes deployment we have followed the reference example here https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions. My assumption is that I would need to also provide the Sentry-related configuration to the "log4-console.properties" for it to be picked up by the Taskmanager and JobManager? PROBLEM 2: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console I am not sure what's going on here. Following dependencies are bundled with the FAT jar "com.typesafe.scala-logging" %% "scala-logging" % scalaLoggingVersion, "org.slf4j" % "slf4j-api" % "1.7.33", "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.17.0", "org.apache.logging.log4j" % "log4j-core" % "2.17.0", "org.apache.logging.log4j" %% "log4j-api-scala" % "12.0", "io.sentry" % "sentry-log4j2" % "5.6.0", Confused about what is going on here, possible this might not be Flink related matter but I am not sure..any tips on how to best debug this would be much appreciated. -- Thanks, Jonas -- Med Vänliga Hälsningar Jonas Eyob Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
