If you have already updated the log4j.properties, and it still could not
work, then I
suggest to log in the Yarn NodeManager machine and check the
log4j.properties
in the container workdir is correct. Also you could have a look at the java
dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options are
set correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha <diwakar.n...@gmail.com> 于2020年11月2日周一 下午4:32写道：
>
>
> Hi Yang,
>
> Thank you so much for taking a look at the log files. I changed my
> log4j.properties. Below is the actual file that I got from EMR 6.1.0
> distribution of flink 1.11. I observed that it is different from Flink 1.11
> that i downloaded so i changed it. Still I didn't see any logs.
>
> *Actual*
> log4j.rootLogger=INFO,file
>
> # Log all infos in the given file
> log4j.appender.file=org.apache.log4j.FileAppender
> log4j.appender.file.file=${log.file}
> log4j.appender.file.append=false
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>
>
> *modified : *commented the above and added new logging from actual flink
> application log4.properties file
>
> #log4j.rootLogger=INFO,file
>
> # Log all infos in the given file
> #log4j.appender.file=org.apache.log4j.FileAppender
> #log4j.appender.file.file=${log.file}
> #log4j.appender.file.append=false
> #log4j.appender.file.layout=org.apache.log4j.PatternLayout
> #log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>
> # This affects logging for both user code and Flink
> rootLogger.level = INFO
> rootLogger.appenderRef.file.ref = MainAppender
>
> # Uncomment this if you want to _only_ change Flink's logging
> #logger.flink.name = org.apache.flink
> #logger.flink.level = INFO
>
> # The following lines keep the log level of common libraries/connectors on
> # log level INFO. The root logger does not override this. You have to
> manually
> # change the log levels here.
> logger.akka.name = akka
> logger.akka.level = INFO
> logger.kafka.name= org.apache.kafka
> logger.kafka.level = INFO
> logger.hadoop.name = org.apache.hadoop
> logger.hadoop.level = INFO
> logger.zookeeper.name = org.apache.zookeeper
> logger.zookeeper.level = INFO
>
> # Log all infos in the given file
> appender.main.name = MainAppender
> appender.main.type = File
> appender.main.append = false
> appender.main.fileName = ${sys:log.file}
> appender.main.layout.type = PatternLayout
> appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x -
> %m%n
>
> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
> logger.netty.level = OFF
>
> **********************************
> I also think its related to the log4j setting but I'm not able to figure
> it out.
> Please let me know if you want any other log files or configuration.
>
> Thanks.
>
> On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> Hi Diwakar Jha,
>>
>> From the logs you have provided, everything seems working as expected.
>> The JobManager and TaskManager
>> java processes have been started with correct dynamic options, especially
>> for the logging.
>>
>> Could you share the content of $FLINK_HOME/conf/log4j.properties? I think
>> there's something wrong with the
>> log4j config file. For example, it is a log4j1 format. But we are using
>> log4j2 in Flink 1.11.
>>
>>
>> Best,
>> Yang
>>
>> Diwakar Jha <diwakar.n...@gmail.com> 于2020年11月2日周一 上午1:57写道：
>>
>>> Hi
>>> I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine
>>> but i'm not seeing any taskmanager/jobmanager logs.
>>> I see the below error in stdout.
>>> 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR
>>> org.apache.flink.runtime.rest.handler.taskmanager.
>>> TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor
>>> container_1604033334508_0001_01_000004.
>>> java.util.concurrent.CompletionException: org.apache.flink.util.
>>> FlinkException: The file LOG does not exist on the TaskExecutor.
>>>
>>> I'm stuck at this step for a couple of days now and not able to migrate
>>> to Flink 1.11. I would appreciate it if anyone can help me.
>>> i have the following setup :
>>> a) i'm deploying flink using yarn. I have attached yarn application id
>>> logs.
>>> c) stsd setup
>>>
>>> metrics.reporters: stsd
>>> metrics.reporter.stsd.factory.class:
>>> org.apache.flink.metrics.statsd.StatsDReporterFactory
>>> metrics.reporter.stsd.host: localhost
>>> metrics.reporter.stsd.port: 8125
>>>
>>>
