Can we see if we can avoid creating TezLocalRunner that extends TezClient? We already have TezSession and TezClient and there is an open jira to unify them. Having a third client side launcher is probably not aligned with making things easier for the users.
Bikas -----Original Message----- From: Chen He [mailto:[email protected]] Sent: Wednesday, March 19, 2014 6:20 PM To: [email protected] Subject: Re: Question about how TezConfiguration Hi Sid Thank you for the reply. Actually, I think it related to Tez-710 and Tez-717. I created TezLocalRunner which extends TezClient. The TezLocalRunner need to start DAGAppMaster which is in charge of maintaining Tez DAG and let LocalContainerLauncher run tasks in a single node.(Tez-710). Here is the error when LocalContainerLauncher launches task: java.io.FileNotFoundException: tez-conf.pb (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at org.apache.tez.common.TezUtils.addUserSpecifiedTezConfiguration(TezUtils.j ava:58) at org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:296) at org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContaine rLauncher.java:263) at java.lang.Thread.run(Thread.java:744) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java :615) at java.lang.Thread.run(Thread.java:744) I did some research and found out that the tez-conf.pb was successfully written to the Tez staging directory. But the YarnTezDagChild can not get the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null") Any suggestion? Thank you in advance! Regards! Chen On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <[email protected]> wrote: > Right, the configuration provided by the user gets written out and is > made available to the AM and tasks. > > I雋 not sure what you mean by OEI always get 軟ull硃 - assuming this > is while trying to read a specific key. The Tez client filters out all > non-tez configuration parameters before writing out this Configuration > - which would cause an empty lookup. > > Could you describe why you need to pass in a Configuration - > typically, I/P/O should have their own payload - which is what they > use to configure themselves. > > > On 3/19/14, 3:57 PM, "Chen He" <[email protected]> wrote: > > >Dear Tez developers: > > > >I met a problem when I was working on TezLocalRunner. I need to know > >how user's configuration is passed to the YarnTezDagChild so that it > >can load it using: > >TezUtils.addUserSpecifiedTezConfiguration(conf); > > > >Here is my understanding: > > > >1) Users create their TezConfiguration and the TezClientUtils will > >write it to staging directory; > >2) DAGAppMaster's main method load TezConfiguration using > >TezUtils.addUserSpecifiedTezConfiguration(conf); > >3) when the YarnTezDagChild starts to run, its main method also call > >TezUtils.addUserSpecifiedTezConfiguration(conf); > > > >I met a problem that I always get "null" whenever I call the > >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient. > > > >Any reply will be appreciated! > > > >Regards! > > > >Chen > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
