[jira] [Commented] (YARN-3084) YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run
[
https://issues.apache.org/jira/browse/YARN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383715#comment-14383715
]
Varun Vasudev commented on YARN-3084:
-
[~Xquery] My apologies for not replying for so long. For some reason I didn't
get any notification that you had replied to me. In response to your questions -
{quote{
1. First, for launching the application master, I see I need to provide it in
the request. I tried to do it, but couldn’t find AppMaster.jar on the
HDFS/local FS. I assume there is AppMaster per YARN-based application (MR, Pig,
Hive etc.). Can you let me know how can I find/install/download such AppMaster
jar?
{quote}
I would recommend using the DistributedShell jar. You can build it yourself
from the Hadoop source code. The jar can be found at
hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target. Copy
this jar to a location on HDFS that's readable by the user running the job.
I've attached a sample json file which you can use to submit to the REST API.
One thing you should note - please update the values for size and timestamp
under the AppMaster.jar key. The value for resource should be the full path
to the jar on HDFS.
The DistributedShell app runs a command on multiple machines and exits. The
commands write their output to a local file and that output is aggregated at
the end of the job if log aggregation is enabled.
In the json I've uploaded, copy the script you want to run to HDFS(making sure
it's readable by the user submitting the job). Set the value of the key
DISTRIBUTEDSHELLSCRIPTTIMESTAMP to the timestamp for this script(on HDFS),
the value of the key DISTRIBUTEDSHELLSCRIPTLEN to the size of the script and
the value of DISTRIBUTEDSHELLSCRIPTLOCATION to the location on HDFS.
The 'num_containers' parameter(part of the command key) is the number of
containers you wish to launch.
{quote}
2. After I launched the application master, in order to run the map reduce
remotely, I need to run another rest api request (I guess), but couldn’t find
any example for it. Do you have REST API example of how to run map reduce using
REST? (or an explained how to/steps)
{quote}
I don't have an example for running MapReduce using REST. The MapReduce client
for YARN is a thick client which does a lot of calculations such as creating
the splits for the map before it submits the job. You will have to implement
that logic yourself if you wish to submit MapReduce jobs. You don't need to run
any other API once you submit the job. The AppMaster is responsible for
scheduling your mappers and reducers.
{quote}
Also, if I want to run the application as user A password B, where I supposed
to add my credentials and Identify; When I submit my map reduce job, isn’t yarn
expects me to identify?
{quote}
Hadoop requires you to setup kerberos for secure mode. In secure mode, jobs are
executed as the user who submitted the job. Credentials are picked up when you
submit the job.
I'm going to close this issue since it doesn't seem like an issue with the REST
API itself. If you have any further questions, we can discuss them offline.
YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes
to run
Key: YARN-3084
URL: https://issues.apache.org/jira/browse/YARN-3084
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, webapp
Affects Versions: 2.6.0
Environment: Using eclipse on windows 7 (client)to run the map reduce
job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version
6.0.2 build-1744117)
Reporter: Michael Br
Priority: Minor
Attachments: submit-app.json,
yarn-yarn-resourcemanager-sandbox.hortonworks.com.log
Hello,
1.I want to run the simple Map Reduce job example (with the REST API 2.6
for yarn applications) and to calculate PI… for now it doesn’t work.
When I use the command in the hortonworks terminal it works: “hadoop jar
/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar
pi 10 10”.
But I want to submit the job with the REST API and not in the terminal as a
command line.
[http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application]
2.I do succeed with other REST API requests: get state, get new
application id and even kill(change state), but when I try to submit my
example, the response is:
--
--
The Response Header:
Key : null ,Value : [HTTP/1.1 202 Accepted]
Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24
GMT]
Key : Content-Length
[jira] [Commented] (YARN-3084) YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run
[ https://issues.apache.org/jira/browse/YARN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335311#comment-14335311 ] Sean Roberts commented on YARN-3084: Apologies, didn't mean to hit submit. I submitted with that job. Interestingly, the 'pi' runs and is successful but the parent job reports a failure. Application application_1424804952495_0004 failed 2 times due to AM Container for appattempt_1424804952495_0004_02 exited with exitCode: 0 Attaching resource manager logs as yarn-yarn-resourcemanager-sandbox.hortonworks.com.log YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run Key: YARN-3084 URL: https://issues.apache.org/jira/browse/YARN-3084 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, webapp Affects Versions: 2.6.0 Environment: Using eclipse on windows 7 (client)to run the map reduce job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version 6.0.2 build-1744117) Reporter: Michael Br Priority: Minor Attachments: yarn-yarn-resourcemanager-sandbox.hortonworks.com.log Hello, 1.I want to run the simple Map Reduce job example (with the REST API 2.6 for yarn applications) and to calculate PI… for now it doesn’t work. When I use the command in the hortonworks terminal it works: “hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10”. But I want to submit the job with the REST API and not in the terminal as a command line. [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application] 2.I do succeed with other REST API requests: get state, get new application id and even kill(change state), but when I try to submit my example, the response is: -- -- The Response Header: Key : null ,Value : [HTTP/1.1 202 Accepted] Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT] Key : Content-Length ,Value : [0] Key : Expires ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT] Key : Location ,Value : [http://[my port]:8088/ws/v1/cluster/apps/application_1421661392788_0038] Key : Content-Type ,Value : [application/json] Key : Server ,Value : [Jetty(6.1.26.hwx)] Key : Pragma ,Value : [no-cache, no-cache] Key : Cache-Control ,Value : [no-cache] The Respone Body: Null (No Response) -- -- 3.I need help with the http request body filling. I am doing a POST http request and I know that I am doing it right (in java). 4.I think the problem is in the request body. 5.I used this guy’s answer to help me build my map reduce example xml but it does not work: [http://hadoop-forum.org/forum/general-hadoop-discussion/miscellaneous/2136-how-can-i-run-mapreduce-job-by-rest-api]. 6.What am I missing? (the description is not clear to me in the submit section of the rest api 2.6) 7.Does someone have an xml example for using a simple MR job? 8.Thanks! Here is the XML file I am using for the request body: -- -- ?xml version=1.0 encoding=UTF-8 standalone=yes? application-submission-context application-idapplication_1421661392788_0038/application-id application-nametest_21_1/application-name queuedefault/queue priority3/priority am-container-spec environment entry keyCLASSPATH/key value/usr/hdp/2.2.0.0-2041/hadoop/conflt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*lt;CPSgt;lt;CPSgt;/usr/share/java/mysql-connector-java-5.1.17.jarlt;CPSgt;/usr/share/java/mysql-connector-java.jarlt;CPSgt;/usr/hdp/current/hadoop-mapreduce-client/*lt;CPSgt;/usr/hdp/current/tez-client/*lt;CPSgt;/usr/hdp/current/tez-client/lib/*lt;CPSgt;/etc/tez/conf/lt;CPSgt;/usr/hdp/2.2.0.0-2041/tez/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/tez/lib/*lt;CPSgt;/etc/tez/conf/value /entry
[jira] [Commented] (YARN-3084) YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run
[
https://issues.apache.org/jira/browse/YARN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335288#comment-14335288
]
Sean Roberts commented on YARN-3084:
I ran the same but with a simplified job request:
{code}
{
application-id:application_1424804952495_0004,
application-name:seanpi2,
am-container-spec:
{
commands:
{
command:hadoop jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 2 2
}
},
application-type:YARN
}
{code}
YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes
to run
Key: YARN-3084
URL: https://issues.apache.org/jira/browse/YARN-3084
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, webapp
Affects Versions: 2.6.0
Environment: Using eclipse on windows 7 (client)to run the map reduce
job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version
6.0.2 build-1744117)
Reporter: Michael Br
Priority: Minor
Hello,
1.I want to run the simple Map Reduce job example (with the REST API 2.6
for yarn applications) and to calculate PI… for now it doesn’t work.
When I use the command in the hortonworks terminal it works: “hadoop jar
/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar
pi 10 10”.
But I want to submit the job with the REST API and not in the terminal as a
command line.
[http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application]
2.I do succeed with other REST API requests: get state, get new
application id and even kill(change state), but when I try to submit my
example, the response is:
--
--
The Response Header:
Key : null ,Value : [HTTP/1.1 202 Accepted]
Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24
GMT]
Key : Content-Length ,Value : [0]
Key : Expires ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015
07:47:24 GMT]
Key : Location ,Value : [http://[my
port]:8088/ws/v1/cluster/apps/application_1421661392788_0038]
Key : Content-Type ,Value : [application/json]
Key : Server ,Value : [Jetty(6.1.26.hwx)]
Key : Pragma ,Value : [no-cache, no-cache]
Key : Cache-Control ,Value : [no-cache]
The Respone Body:
Null (No Response)
--
--
3.I need help with the http request body filling. I am doing a POST http
request and I know that I am doing it right (in java).
4.I think the problem is in the request body.
5.I used this guy’s answer to help me build my map reduce example xml but
it does not work:
[http://hadoop-forum.org/forum/general-hadoop-discussion/miscellaneous/2136-how-can-i-run-mapreduce-job-by-rest-api].
6.What am I missing? (the description is not clear to me in the submit
section of the rest api 2.6)
7.Does someone have an xml example for using a simple MR job?
8.Thanks! Here is the XML file I am using for the request body:
--
--
?xml version=1.0 encoding=UTF-8 standalone=yes?
application-submission-context
application-idapplication_1421661392788_0038/application-id
application-nametest_21_1/application-name
queuedefault/queue
priority3/priority
am-container-spec
environment
entry
keyCLASSPATH/key
value/usr/hdp/2.2.0.0-2041/hadoop/conflt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*lt;CPSgt;lt;CPSgt;/usr/share/java/mysql-connector-java-5.1.17.jarlt;CPSgt;/usr/share/java/mysql-connector-java.jarlt;CPSgt;/usr/hdp/current/hadoop-mapreduce-client/*lt;CPSgt;/usr/hdp/current/tez-client/*lt;CPSgt;/usr/hdp/current/tez-client/lib/*lt;CPSgt;/etc/tez/conf/lt;CPSgt;/usr/hdp/2.2.0.0-2041/tez/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/tez/lib/*lt;CPSgt;/etc/tez/conf/value
/entry
/environment
commands
commandhadoop jar
[jira] [Commented] (YARN-3084) YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run
[ https://issues.apache.org/jira/browse/YARN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287336#comment-14287336 ] Michael Br commented on YARN-3084: -- Hi, first thanks for your quick reply... 1. Where can i see the queue? i looked for it but i still dont get how to get to it 2. I cant reach the logs... The only clue I have regarding this job fails is: Thats the only information i can see from the URL: http://192.168.38.133:8088/ws/v1/cluster/apps/application_1421661392788_0039 (the links there when i try to use them for the logs dont workhttp://sandbox.hortonworks.com:8088/proxy/application_1421661392788_0039/ [i replace and insert my host ip] and also the link for the container logs dont work. http://sandbox.hortonworks.com:8042/node/containerlogs/container_1421661392788_0039_02_01/dr.who -- app idapplication_1421661392788_0039/id userdr.who/user nametest_33/name queuedefault/queue stateFAILED/state finalStatusFAILED/finalStatus progress0.0/progress trackingUIHistory/trackingUI trackingUrl http://sandbox.hortonworks.com:8088/cluster/app/application_1421661392788_0039 /trackingUrl diagnostics Application application_1421661392788_0039 failed 2 times due to AM Container for appattempt_1421661392788_0039_02 exited with exitCode: 0 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/proxy/application_1421661392788_0039/Then, click on links to logs of each attempt. Diagnostics: Failing this attempt. Failing the application. /diagnostics clusterId1421661392788/clusterId applicationTypeMAPREDUCE/applicationType applicationTagsmichael,pi example/applicationTags startedTime1421923561425/startedTime finishedTime1421923723426/finishedTime elapsedTime162001/elapsedTime amContainerLogs http://sandbox.hortonworks.com:8042/node/containerlogs/container_1421661392788_0039_02_01/dr.who /amContainerLogs amHostHttpAddresssandbox.hortonworks.com:8042/amHostHttpAddress allocatedMB-1/allocatedMB allocatedVCores-1/allocatedVCores runningContainers-1/runningContainers memorySeconds200857/memorySeconds vcoreSeconds160/vcoreSeconds preemptedResourceMB0/preemptedResourceMB preemptedResourceVCores0/preemptedResourceVCores numNonAMContainerPreempted0/numNonAMContainerPreempted numAMContainerPreempted0/numAMContainerPreempted /app YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run Key: YARN-3084 URL: https://issues.apache.org/jira/browse/YARN-3084 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, webapp Affects Versions: 2.6.0 Environment: Using eclipse on windows 7 (client)to run the map reduce job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version 6.0.2 build-1744117) Reporter: Michael Br Priority: Minor Hello, 1.I want to run the simple Map Reduce job example (with the REST API 2.6 for yarn applications) and to calculate PI… for now it doesn’t work. When I use the command in the hortonworks terminal it works: “hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10”. But I want to submit the job with the REST API and not in the terminal as a command line. [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application] 2.I do succeed with other REST API requests: get state, get new application id and even kill(change state), but when I try to submit my example, the response is: -- -- The Response Header: Key : null ,Value : [HTTP/1.1 202 Accepted] Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT] Key : Content-Length ,Value : [0] Key : Expires ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT] Key : Location ,Value : [http://[my port]:8088/ws/v1/cluster/apps/application_1421661392788_0038] Key : Content-Type ,Value : [application/json] Key : Server ,Value : [Jetty(6.1.26.hwx)] Key : Pragma ,Value : [no-cache, no-cache] Key : Cache-Control ,Value : [no-cache] The Respone Body: Null (No Response) -- -- 3.I need help with the http request body filling. I am doing a POST http request and I know that I am doing it right (in java). 4.I think the problem is in the request body. 5.I used this guy’s answer to help me build my map reduce example xml but it does not work:
[jira] [Commented] (YARN-3084) YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run
[ https://issues.apache.org/jira/browse/YARN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287214#comment-14287214 ] Steve Loughran commented on YARN-3084: -- 202, accepted, looks like the RM accepted it. # does it appear in the queue of job submissions?? # what does the RM log say? YARN REST API 2.6 - can't submit simple job in hortonworks-allways job failes to run Key: YARN-3084 URL: https://issues.apache.org/jira/browse/YARN-3084 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, webapp Affects Versions: 2.6.0 Environment: Using eclipse on windows 7 (client)to run the map reduce job on the host of Hortonworks HDP 2.2 (hortonworks is on vmware version 6.0.2 build-1744117) Reporter: Michael Br Priority: Minor Hello, 1.I want to run the simple Map Reduce job example (with the REST API 2.6 for yarn applications) and to calculate PI… for now it doesn’t work. When I use the command in the hortonworks terminal it works: “hadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10”. But I want to submit the job with the REST API and not in the terminal as a command line. [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application] 2.I do succeed with other REST API requests: get state, get new application id and even kill(change state), but when I try to submit my example, the response is: -- -- The Response Header: Key : null ,Value : [HTTP/1.1 202 Accepted] Key : Date ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT] Key : Content-Length ,Value : [0] Key : Expires ,Value : [Thu, 22 Jan 2015 07:47:24 GMT, Thu, 22 Jan 2015 07:47:24 GMT] Key : Location ,Value : [http://[my port]:8088/ws/v1/cluster/apps/application_1421661392788_0038] Key : Content-Type ,Value : [application/json] Key : Server ,Value : [Jetty(6.1.26.hwx)] Key : Pragma ,Value : [no-cache, no-cache] Key : Cache-Control ,Value : [no-cache] The Respone Body: Null (No Response) -- -- 3.I need help with the http request body filling. I am doing a POST http request and I know that I am doing it right (in java). 4.I think the problem is in the request body. 5.I used this guy’s answer to help me build my map reduce example xml but it does not work: [http://hadoop-forum.org/forum/general-hadoop-discussion/miscellaneous/2136-how-can-i-run-mapreduce-job-by-rest-api]. 6.What am I missing? (the description is not clear to me in the submit section of the rest api 2.6) 7.Does someone have an xml example for using a simple MR job? 8.Thanks! Here is the XML file I am using for the request body: -- -- ?xml version=1.0 encoding=UTF-8 standalone=yes? application-submission-context application-idapplication_1421661392788_0038/application-id application-nametest_21_1/application-name queuedefault/queue priority3/priority am-container-spec environment entry keyCLASSPATH/key value/usr/hdp/2.2.0.0-2041/hadoop/conflt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*lt;CPSgt;lt;CPSgt;/usr/share/java/mysql-connector-java-5.1.17.jarlt;CPSgt;/usr/share/java/mysql-connector-java.jarlt;CPSgt;/usr/hdp/current/hadoop-mapreduce-client/*lt;CPSgt;/usr/hdp/current/tez-client/*lt;CPSgt;/usr/hdp/current/tez-client/lib/*lt;CPSgt;/etc/tez/conf/lt;CPSgt;/usr/hdp/2.2.0.0-2041/tez/*lt;CPSgt;/usr/hdp/2.2.0.0-2041/tez/lib/*lt;CPSgt;/etc/tez/conf/value /entry /environment commands commandhadoop jar /usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar pi 10 10/command /commands /am-container-spec unmanaged-AMfalse/unmanaged-AM max-app-attempts2/max-app-attempts resource
