Re: Where can I find logs set inside RDD processing functions?

2015-02-06 Thread Nitin kak
yarn.nodemanager.remote-app-log-dir is set to /tmp/logs On Fri, Feb 6, 2015 at 4:14 PM, Ted Yu wrote: > To add to What Petar said, when YARN log aggregation is enabled, consider > specifying yarn.nodemanager.remote-app-log-dir which is where aggregated > logs are saved. > > Cheers > > On Fri, F

Re: Where can I find logs set inside RDD processing functions?

2015-02-06 Thread Nitin kak
The yarn log aggregation is enabled and the logs which I get through "yarn logs -applicationId " are no different than what I get through logs in Yarn Application tracking URL. They still dont have the above logs. On Fri, Feb 6, 2015 at 3:36 PM, Petar Zecevic wrote: > > You can enable YARN log a

Re: Where can I find logs set inside RDD processing functions?

2015-02-06 Thread Ted Yu
To add to What Petar said, when YARN log aggregation is enabled, consider specifying yarn.nodemanager.remote-app-log-dir which is where aggregated logs are saved. Cheers On Fri, Feb 6, 2015 at 12:36 PM, Petar Zecevic wrote: > > You can enable YARN log aggregation (yarn.log-aggregation-enable to

Re: Where can I find logs set inside RDD processing functions?

2015-02-06 Thread Petar Zecevic
You can enable YARN log aggregation (yarn.log-aggregation-enable to true) and execute command yarn logs -applicationId after your application finishes. Or you can look at them directly in HDFS in /tmp/logs//logs// On 6.2.2015. 19:50, nitinkak001 wrote: I am trying to debug my mapPartition