[
https://issues.apache.org/jira/browse/KNOX-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830302#comment-15830302
]
Kuldeep Kulkarni commented on KNOX-857:
---------------------------------------
Okay, so this is what exactly is happening.
This is my workflow.xml
{code}
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
<start to="mr-node"/>
<action name="mr-node">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete
path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.mapper.class</name>
<value>org.apache.oozie.example.SampleMapper</value>
</property>
<property>
<name>mapred.reducer.class</name>
<value>org.apache.oozie.example.SampleReducer</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>1</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
</property>
<property>
<name>mapred.output.dir</name>
<value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
{code}
*Note the value of outputDir in above config file*
This is my job.xml.bug
{code}
<configuration>
<property>
<name>outputDir</name>
<value>map-reduce</value>
</property>
<property>
<name>examplesRoot</name>
<value>examples</value>
</property>
<property>
<name>oozie.wf.application.path</name>
<value>hdfs://KK/user/ambari-qa/examples/apps/map-reduce/workflow.xml</value>
</property>
<property>
<name>queueName</name>
<value>default</value>
</property>
<property>
<name>user.name</name>
<value>ambari-qa</value>
</property>
<property>
<name>jobTracker</name>
<value>oozieknox3.openstacklocal:8050</value>
</property>
<property>
<name>mapreduce.job.user.name</name>
<value>ambari-qa</value>
</property>
<property>
<name>nameNode</name>
<value>hdfs://KK</value>
</property>
</configuration>
{code}
I submit the job
{code}
[ambari-qa@oozieknox1 ~]$ curl -i -k -u ambari-qa:hadoop -H
Content-Type:application/xml -T job.xml.bug -X POST
'https://oozieknox3.openstacklocal:8443/gateway/mytopology/oozie/v1/jobs?action=start'
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Set-Cookie:
JSESSIONID=kwip9wajqmb71ebg53g93pe3j;Path=/gateway/mytopology;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Server: Apache-Coyote/1.1
Date: Thu, 19 Jan 2017 17:23:15 GMT
Content-Type: application/json;charset=UTF-8
Content-Length: 45
{"id":"0000021-170118140025343-oozie-oozi-W"}
{code}
Now, when I open Oozie UI --> Active Jobs --> Select job with workflow
ID=0000021-170118140025343-oozie-oozi-W --> Click Job configuration --> Check
value of outputDir variable.
*Value of variable outputDir automatically gets changed from "map-reduce" to
"hdfs://oozieknox2.openstacklocal:8020/map-reduce"*
I think Knox is adding extra "hdfs://oozieknox2.openstacklocal:8020/" part.
Thanks,
Kuldeep
> Incorrect formation of <delete path> causes oozie mapreduce action failure
> --------------------------------------------------------------------------
>
> Key: KNOX-857
> URL: https://issues.apache.org/jira/browse/KNOX-857
> Project: Apache Knox
> Issue Type: Bug
> Components: KnoxCLI
> Affects Versions: 0.6.0
> Environment: HDP-2.4.0.0
> Reporter: Kuldeep Kulkarni
> Priority: Critical
> Attachments: job.xml.bug, knox.conf.tar.gz, knox.log.tz
>
>
> When Oozie workflow is submitted via Knox containing mapreduce-action. it
> always gets failed with below error:
> {code}
> Heart beat
> {"properties":[{"key":"oozie.launcher.job.id","value":"job_1484748086731_0022","isFinal":false,"resource":"programatically"},{"key":"oozie.job.id","value":"0000017-170118140025343-oozie-oozi-W","isFinal":false,"resource":"programatically"},{"key":"oozie.action.id","value":"0000017-170118140025343-oozie-oozi-W@mr-node","isFinal":false,"resource":"programatically"},{"key":"mapreduce.job.tags","value":"oozie-20b9b1ecd11f08843b078e701ce5c1ad","isFinal":false,"resource":"programatically"}]}Starting
> the execution of prepare actions
> Prepare execution in the Launcher Mapper has failed
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.MapReduceMain], exception invoking main(),
> java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs:
> org.apache.oozie.action.hadoop.LauncherException:
> java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs:
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:193)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.oozie.action.hadoop.LauncherException:
> java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs:
> at
> org.apache.oozie.action.hadoop.PrepareActionsDriver.doOperations(PrepareActionsDriver.java:77)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.executePrepare(LauncherMapper.java:494)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:189)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Expected scheme-specific part at index 5: hdfs:
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.<init>(Path.java:171)
> at org.apache.hadoop.fs.Path.<init>(Path.java:93)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:211)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)
> at
> org.apache.oozie.action.hadoop.FSLauncherURIHandler.delete(FSLauncherURIHandler.java:59)
> at
> org.apache.oozie.action.hadoop.PrepareActionsDriver.execute(PrepareActionsDriver.java:90)
> at
> org.apache.oozie.action.hadoop.PrepareActionsDriver.doOperations(PrepareActionsDriver.java:68)
> ... 10 more
> Caused by: java.net.URISyntaxException: Expected scheme-specific part at
> index 5: hdfs:
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.failExpecting(URI.java:2854)
> at java.net.URI$Parser.parse(URI.java:3057)
> at java.net.URI.<init>(URI.java:746)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> ... 17 more
> Oozie Launcher failed, finishing Hadoop job gracefully
> {code}
> This happens because, <delete path> in the action configuration gets
> generated incorrectly.
> {code}
> <delete
> path="hdfs://oozieknox2.openstacklocal:8020/user/ambari-qa/examples/output-data/hdfs://oozieknox2.openstacklocal:8020/map-reduce"
> />
> {code}
> Expected value is:
> {code}
> <delete
> path="hdfs://oozieknox2.openstacklocal:8020/user/ambari-qa/examples/output-data/map-reduce"
> />
> {code}
> Command used to submit oozie job is:
> {code}
> [ambari-qa@oozieknox1 ~]$ curl -i -k -u ambari-qa:hadoop -H
> Content-Type:application/xml -T job.xml.bug -X POST
> 'https://oozieknox3.openstacklocal:8443/gateway/mytopology/oozie/v1/jobs?action=start'
> HTTP/1.1 100 Continue
> HTTP/1.1 201 Created
> Set-Cookie:
> JSESSIONID=13d8gzukx9aq019rdkb06li3pj;Path=/gateway/mytopology;Secure;HttpOnly
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Server: Apache-Coyote/1.1
> Date: Thu, 19 Jan 2017 12:06:39 GMT
> Content-Type: application/json;charset=UTF-8
> Content-Length: 45
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
