Just share another method about how to access the finished TaskManager logs on Yarn.
Currently, only when a Yarn application finished/failed/killed, the logs will be aggregated to HDFS. That means if the Flink application is still running, you could still use the Yarn NodeManager webUI to access the TM logs. Please build the url in the following schema. http://<NodeManagerAddress:Port>/node/containerlogs/container_xxxxxx/<User>/taskmanager.log <http://i22a11409.sqa.eu95.tbsite.net:8042/node/containerlogs/container_e07_1593487926057_0852_01_000002/admin/taskmanager.log> Best, Yang Robert Metzger <rmetz...@apache.org> 于2020年7月27日周一 下午2:42写道： > Hi Richard, > thanks for forwarding my answer to the list! > > I fear that Flink does not have a built-in solution for serving the logs > of a finished TaskManager while a YARN session is still running. > > I agree with Yangze that you probably have to rely on an external logging > service, such as ElasticSearch or Splunk to index your log events. > Maybe there's also some tooling from Cloudera specifically made for YARN? > > > > On Mon, Jul 27, 2020 at 3:46 AM Yangze Guo <karma...@gmail.com> wrote: > >> Hi, Richard >> >> Before the session has been terminated, you could not fetch the >> terminated TM logs. One possible solution could be leveraging the >> log4j2 appenders[1]. Flink uses log4j2 as default in the latest >> release 1.11. >> >> [1] https://logging.apache.org/log4j/2.x/manual/appenders.html >> >> Best, >> Yangze Guo >> >> On Sat, Jul 25, 2020 at 2:37 AM Richard Moorhead >> <richard.moorh...@gmail.com> wrote: >> > >> > >> > >> > ---------- Forwarded message --------- >> > From: Robert Metzger <rmetz...@apache.org> >> > Date: Fri, Jul 24, 2020 at 1:09 PM >> > Subject: Re: Flink Session TM Logs >> > To: Richard Moorhead <richard.moorh...@gmail.com> >> > >> > >> > I accidentally replied to you directly, not to the user@ mailing list. >> Is it okay for you to publish the thread on the list again? >> > >> > >> > >> > On Fri, Jul 24, 2020 at 8:01 PM Richard Moorhead < >> richard.moorh...@gmail.com> wrote: >> >> >> >> It is enabled. The issue is that for a long running flink session >> -which may execute many jobs- the task managers, after a job is completed, >> are gone, and their logs arent available. >> >> >> >> What I have noticed is that when the session is terminated I am able >> to find the logs in the job history server under the associated yarn >> application id. >> >> >> >> On Fri, Jul 24, 2020 at 12:51 PM Robert Metzger <rmetz...@apache.org> >> wrote: >> >>> >> >>> Hi Richard, >> >>> >> >>> you need to enable YARN log aggregation to access logs of finished >> YARN applications. >> >>> >> >>> On Fri, Jul 24, 2020 at 5:58 PM Richard Moorhead < >> richard.moorh...@gmail.com> wrote: >> >>>> >> >>>> When running a flink session on YARN, task manager logs for a job >> are not available after completion. How do we locate these logs? >> >>>> >> >
