Interesting use-case and setup. We never had this use-case in mind so far - we so far assumed a history-server per YARN cluster. You may be running into some issues where this assumption is not valid.
Why do you need two separate YARN clusters for the same underlying data on HDFS? And if that can't change, why can't you have two history-servers? +Vinod On Feb 20, 2014, at 6:08 PM, Anfernee Xu <[email protected]> wrote: > Hi, > > I'm at 2.2.0 release and I have a HDFS cluster which is shared by 2 YARN(MR) > cluster, also I have a single shared history server, what I'm seeing is I can > see all job summary for all jobs from history server UI, I also can see task > log for jobs running in one cluster, but if I want to see log for jobs > running in another cluster, it showed me below error > > Logs not available for attempt_1392933787561_0024_m_000000_0. Aggregation may > not be complete, Check back later or try the nodemanager at > slc03jvt.mydomain.com:31303 > > Here's my configuration: > > Note: my history server is running on RM node of the MR cluster where I can > see the log. > > > --------mapred-site.xml > <property> > <name>mapreduce.jobhistory.address</name> > <value>slc00dgd:10020</value> > <description>MapReduce JobHistory Server IPC host:port</description> > </property> > > <property> > <name>mapreduce.jobhistory.webapp.address</name> > <value>slc00dgd:19888</value> > <description>MapReduce JobHistory Server Web UI host:port</description> > </property> > > ----------yarn-site.xml > <property> > <name>yarn.log-aggregation-enable</name> > <value>true</value> > </property> > > <property> > <name>yarn.nodemanager.remote-app-log-dir-suffix</name> > <value>dc</value> > </property> > > Above configuration are almost same for both clusters, the only difference is > "yarn.nodemanager.remote-app-log-dir-suffix", they have different suffix. > > > > -- > --Anfernee -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
signature.asc
Description: Message signed with OpenPGP using GPGMail
