Thanks Navina. What I found is that containers logs were gone after days locally even when the job is still running.
Chen On Mon, Aug 3, 2015 at 6:06 PM, Navina Ramesh <nram...@linkedin.com.invalid> wrote: > Hi Chen, > You can set the yarn.nodemanage.delete.debug-delay-sec config in > yarn-site.xml, in order to retain the logs after the job crashes. Refer > > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml > > > (This was intended for debug purpose). > > If you need to retain log for longer period of time for purposes other than > debugging, you should look into yarn's log aggregation features. > Alternatively, you can use Samza's StreamAppender to produce the log data > to a stream. You can then publish it to ELK for further lookup. It really > depends on your requirements. > Please note that the StreamAppender has a bug in the master branch that is > pending resolution in https://issues.apache.org/jira/browse/SAMZA-723 > > Cheers! > Navina > > > On Mon, Aug 3, 2015 at 2:26 PM, Chen Song <chen.song...@gmail.com> wrote: > > > Dump question. > > > > When running samza job on YARN, I found that container logs gone after > > days. Is there a YARN config to keep the logs from being deleted for > > long-lived streaming job? > > > > -- > > Chen Song > > > > > > -- > Navina R. > -- Chen Song
