Could you find the logs under /opt/flink/log/jobmanager.log? If not, please share the commands the JobManager and TaskManager are using? If the command is correct and the log4j under /opt/flink/conf is expected, it is so curious why we could not get the logs.
Best, Yang Li Peng <li.p...@doordash.com> 于2019年12月11日周三 下午1:24写道： > Ah I see. I think the Flink app is reading files from > /opt/flink/conf correctly as it is, since changes I make to flink-conf are > picked up as expected, it's just the log4j properties that are either not > being used, or don't apply to stdout or whatever source k8 uses for its > logs? Given that the pods don't seem to have logs written to file > anywhere, contrary to the properties, I'm inclined to say it's the former > and that the log4j properties just aren't being picked up. Still have no > idea why though. > > On Tue, Dec 10, 2019 at 6:56 PM Yun Tang <myas...@live.com> wrote: > >> Sure, /opt/flink/conf is mounted as a volume from the configmap. >> >> >> >> Best >> >> Yun Tang >> >> >> >> *From: *Li Peng <li.p...@doordash.com> >> *Date: *Wednesday, December 11, 2019 at 9:37 AM >> *To: *Yang Wang <danrtsey...@gmail.com> >> *Cc: *vino yang <yanghua1...@gmail.com>, user <user@flink.apache.org> >> *Subject: *Re: Flink on Kubernetes seems to ignore log4j.properties >> >> >> >> 1. Hey Yun, I'm calling /opt/flink/bin/standalone-job.sh and >> /opt/flink/bin/taskmanager.sh on my job and task managers respectively. >> It's based on the setup described here: >> http://shzhangji.com/blog/2019/08/24/deploy-flink-job-cluster-on-kubernetes/ >> . >> I haven't tried the configmap approach yet, does it also replace the conf >> files in /opt/flink/conf? >> >> 2. Hey Vino, here's a sample of the kubernetes: >> https://pastebin.com/fqJrgjZu I didn't change any patterns from the >> default, so the string patterns should look the same, but as you can see >> it's full of info checkpoint logs that I originally was trying to suppress. >> Based on my log4j.properties, the level should be set to WARN. I couldn't >> actually find any .out files on the pod, this is from the kubectl logs >> command. I also didn't see any files in /opt/flink/log, which I thought my >> log4j was specified to do, hence me thinking that the properties weren't >> actually being consumed. I also have the same properties in my >> src/main/resources folder. >> >> 3. Hey Yang, yes this is a standalone session cluster. I did specify in >> the docker file to copy the log4j.properties to the /opt/flink/conf folder >> on the image, and I confirmed that the properties are correct when I bash'd >> into the pod and viewed them manually. >> >> >> >> Incidentally, I also tried passing the -Dlog4j.configuration argument to >> the programs, and it doesn't work either. And based on what I'm reading on >> jira, that option is not really supported anymore? >> >> >> >> Thanks for your responses, folks! >> >> Li >> >> >> >> On Mon, Dec 9, 2019 at 7:10 PM Yang Wang <danrtsey...@gmail.com> wrote: >> >> Hi Li Peng, >> >> >> >> You are running standalone session cluster or per-job cluster on >> kubernetes. Right? >> >> If so, i think you need to check your log4j.properties in the image, not >> local. The log is >> >> stored to /opt/flink/log/jobmanager.log by default. >> >> >> >> If you are running active Kubernetes integration for a fresh taste. The >> following cli option >> >> could be used to remove the redirect. >> >> -Dkubernetes.container-start-command-template="%java% %classpath% >> %jvmmem% %jvmopts% %logging% %class% %args%" >> >> >> >> Best, >> >> Yang >> >> >> >> vino yang <yanghua1...@gmail.com> 于2019年12月10日周二 上午10:55写道： >> >> Hi Li, >> >> >> >> A potential reason could be conflicting logging frameworks. Can you share >> the log in your .out file and let us know if the print format of the log is >> the same as the configuration file you gave. >> >> >> >> Best, >> >> Vino >> >> >> >> Li Peng <li.p...@doordash.com> 于2019年12月10日周二 上午10:09写道： >> >> Hey folks, I noticed that my kubernetes flink logs (reached via *kubectl >> logs <pod-name>*) completely ignore any of the configurations I put into >> /flink/conf/. I set the logger level to WARN, yet I still see INFO level >> logging from flink loggers >> like org.apache.flink.runtime.checkpoint.CheckpointCoordinator. I even made >> copied the same properties to /flink/conf/log4j-console.properties >> and log4j-cli.properties. >> >> >> >> From what I can tell, kubernetes just listens to stdout and stderr, so >> shouldn't the log4j.properties control output to them? Anyone seen this >> issue before? >> >> >> >> Here is my log4j.properties: >> >> >> # This affects logging for both user code and Flink >> log4j.rootLogger=WARN, file, console, stdout >> >> # Uncomment this if you want to _only_ change Flink's logging >> log4j.logger.org.apache.flink=WARN >> >> # The following lines keep the log level of common libraries/connectors on >> # log level INFO. The root logger does not override this. You have to >> manually >> # change the log levels here. >> log4j.logger.akka=INFO >> log4j.logger.org.apache.kafka=INFO >> log4j.logger.org.apache.hadoop=INFO >> log4j.logger.org.apache.zookeeper=INFO >> >> # Log all infos in the given file >> log4j.appender.file=org.apache.log4j.FileAppender >> log4j.appender.file.file=${log.file} >> log4j.appender.file.append=false >> log4j.appender.file.layout=org.apache.log4j.PatternLayout >> log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} >> %-5p %-60c %x - %m%n >> >> # Log all infos to the console >> log4j.appender.console=org.apache.log4j.ConsoleAppender >> log4j.appender.console.layout=org.apache.log4j.PatternLayout >> log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} >> %-5p %-60c %x - %m%n >> >> # Suppress the irrelevant (wrong) warnings from the Netty channel handler >> log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, >> file, console >> log4j.logger.org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction=WARN >> log4j.logger.org.apache.flink.runtime.checkpoint=WARN >> >> Thanks, >> >> Li >> >>
