hi,all:
I want to use distcp tp copy ftp files to hdfs . I tried distcp action ,but
I got error with:java.lang.IllegalArgumentException: begin > end in range
(begin, end): (1415382327807, 1415382308114)
but it was succeed with command:
hadoop distcp ftp://youeftp:[email protected]/biz/biz_20160803.zip
hdfs://dev0:9000/sourcedata/biz/201608/biz_20160803.zip
so I write the command into a sh file and try to use oozie shell action:
but the job run succeed with out running the command. my workflow.xml like
this:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-distcp">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>distcp.sh</exec>
<!--
<argument>my_output=Hello Oozie</argument>
-->
<file>/user/hadoop/examples/apps/youedata/weixin/distcp.sh#distcp.sh</file>
<capture-output/>
</shell>
<ok to="check-output"/>
<error to="fail"/>
</action>
<decision name="check-output">
<switch>
<case to="end">
${wf:actionData('shell-node')['my_output'] eq 'Hello Oozie'}
</case>
<default to="fail-output"/>
</switch>
</decision>
<kill name="fail">
<message>Shell action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was
[${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app>
--------------------------------------------------------------------------------------------------------------
and distcp.sh:
#!/bin/bash
hadoop distcp ftp://youeftp:[email protected]/weixin/biz/biz_20161025.zip
hdfs://dev0:9000/sourcedata/weixin/biz/biz/biz_20161025.zip
echo "my_output=Hello Oozie"
2016-10-27
lk_hadoop
