Varun Vasudev commented on YARN-3084:

[~Xquery] My apologies for not replying for so long. For some reason I didn't 
get any notification that you had replied to me. In response to your questions -

1. First, for launching the application master, I see I need to provide it in 
the request. I tried to do it, but couldn’t find AppMaster.jar on the 
HDFS/local FS. I assume there is AppMaster per YARN-based application (MR, Pig, 
Hive etc.). Can you let me know how can I find/install/download such AppMaster 

I would recommend using the DistributedShell jar. You can build it yourself 
from the Hadoop source code. The jar can be found at 
hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target. Copy 
this jar to a location on HDFS that's readable by the user running the job. 
I've attached a sample json file which you can use to submit to the REST API. 
One thing you should note - please update the values for "size" and "timestamp" 
under the "AppMaster.jar" key. The value for "resource" should be the full path 
to the jar on HDFS.

The DistributedShell app runs a command on multiple machines and exits. The 
commands write their output to a local file and that output is aggregated at 
the end of the job if log aggregation is enabled.

In the json I've uploaded, copy the script you want to run to HDFS(making sure 
it's readable by the user submitting the job). Set the value of the key 
"DISTRIBUTEDSHELLSCRIPTTIMESTAMP" to the timestamp for this script(on HDFS), 
the value of the key "DISTRIBUTEDSHELLSCRIPTLEN" to the size of the script and 
the value of "DISTRIBUTEDSHELLSCRIPTLOCATION" to the location on HDFS.

The 'num_containers' parameter(part of the "command" key) is the number of 
containers you wish to launch.

2. After I launched the application master, in order to run the map reduce 
remotely, I need to run another rest api request (I guess), but couldn’t find 
any example for it. Do you have REST API example of how to run map reduce using 
REST? (or an explained how to/steps)

I don't have an example for running MapReduce using REST. The MapReduce client 
for YARN is a thick client which does a lot of calculations such as creating 
the splits for the map before it submits the job. You will have to implement 
that logic yourself if you wish to submit MapReduce jobs. You don't need to run 
any other API once you submit the job. The AppMaster is responsible for 
scheduling your mappers and reducers.

Also, if I want to run the application as user A password B, where I supposed 
to add my credentials and Identify; When I submit my map reduce job, isn’t yarn 
expects me to identify? 

Hadoop requires you to setup kerberos for secure mode. In secure mode, jobs are 
executed as the user who submitted the job. Credentials are picked up when you 
submit the job.

I'm going to close this issue since it doesn't seem like an issue with the REST 
API itself. If you have any further questions, we can discuss them offline.

> YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes 
> to run
> ------------------------------------------------------------------------------------
>                 Key: YARN-3084
>                 URL: https://issues.apache.org/jira/browse/YARN-3084
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager, webapp
>    Affects Versions: 2.6.0
>         Environment: Using eclipse on windows 7 (client)to run the map reduce 
> job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version 
> 6.0.2 build-1744117)
>            Reporter: Michael Br
>            Priority: Minor
>         Attachments: submit-app.json, 
> yarn-yarn-resourcemanager-sandbox.hortonworks.com.log
> Hello,
> 1.    I want to run the simple Map Reduce job example (with the REST API 2.6 
> for yarn applications) and to calculate PI… for now it doesn’t work.
> When I use the command in the hortonworks terminal it works: “hadoop jar 
> /usr/hdp/
>  pi 10 10”.
> But I want to submit the job with the REST API and not in the terminal as a 
> command line. 
> [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application]
> 2.    I do succeed with other REST API requests: get state, get new 
> application id and even kill(change state), but when I try to submit my 
> example, the response is:
> --------------------------------------------------
> --------------------------------------------------
> The Response Header:
> Key : null ,Value : [HTTP/1.1 202 Accepted]
> Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 
> GMT]
> Key : Content-Length ,Value : [0]
> Key : Expires ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 
> 07:47:24 GMT]
> Key : Location ,Value : [http://[my 
> port]:8088/ws/v1/cluster/apps/application_1421661392788_0038]
> Key : Content-Type ,Value : [application/json]
> Key : Server ,Value : [Jetty(6.1.26.hwx)]
> Key : Pragma ,Value : [no-cache, no-cache]
> Key : Cache-Control ,Value : [no-cache]
> The Respone Body:
> Null (No Response)
> --------------------------------------------------
> --------------------------------------------------
> 3.    I need help with the http request body filling. I am doing a POST http 
> request and I know that I am doing it right (in java).
> 4.    I think the problem is in the request body.
> 5.    I used this guy’s answer to help me build my map reduce example xml but 
> it does not work: 
> [http://hadoop-forum.org/forum/general-hadoop-discussion/miscellaneous/2136-how-can-i-run-mapreduce-job-by-rest-api].
> 6.    What am I missing? (the description is not clear to me in the submit 
> section of the rest api 2.6)
> 7.    Does someone have an xml example for using a simple MR job?
> 8.    Thanks! Here is the XML file I am using for the request body:
> --------------------------------------------------
> --------------------------------------------------
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <application-submission-context>    
>       <application-id>application_1421661392788_0038</application-id>
>     <application-name>test_21_1</application-name>    
>       <queue>default</queue>
>     <priority>3</priority>
>     <am-container-spec>      
>               <environment>                   
>                       <entry>                   
>                               <key>CLASSPATH</key>
> <value>/usr/hdp/;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;&lt;CPS&gt;/usr/share/java/mysql-connector-java-5.1.17.jar&lt;CPS&gt;/usr/share/java/mysql-connector-java.jar&lt;CPS&gt;/usr/hdp/current/hadoop-mapreduce-client/*&lt;CPS&gt;/usr/hdp/current/tez-client/*&lt;CPS&gt;/usr/hdp/current/tez-client/lib/*&lt;CPS&gt;/etc/tez/conf/&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/usr/hdp/*&lt;CPS&gt;/etc/tez/conf</value>
>                       </entry>
>               </environment>
>               <commands>
>                       <command>hadoop jar 
> /usr/hdp/
>  pi 10 10</command>
>               </commands>
>     </am-container-spec>
>     <unmanaged-AM>false</unmanaged-AM>
>     <max-app-attempts>2</max-app-attempts>
>     <resource>      
>               <memory>1024</memory>
>               <vCores>1</vCores>
>     </resource>    
>       <application-type>MAPREDUCE</application-type>
> <keep-containers-across-application-attempts>false</keep-containers-across-application-attempts>
>     <application-tags>      
>               <tag>Michael</tag>      
>               <tag>PI example</tag>    
>       </application-tags>
> </application-submission-context>
> --------------------------------------------------
> --------------------------------------------------

This message was sent by Atlassian JIRA

Reply via email to