[
https://issues.apache.org/jira/browse/OOZIE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuaishuai Nie updated OOZIE-1521:
----------------------------------
Description: Now Sqoop work flow launched by Oozie will copy the jar files
in %SQOOP_HOME%\lib folder to the job cache each time a Sqoop job is launched.
We should add these jars to Sqoop sharelib so that for each worker node, the
jars in the Sqoop sharelib only copied to distribute cache and localize to each
worker node once and reuse by all the subsequent Sqoop jobs. In Sqoop-1192, an
option '--skip-dist-cache' is introduced which disable Sqoop add lib jars to
job cache action and reuse the jars in the Sqoop sharelib. In the patch
attached, Oozie will add "--skip-dist-cache" option to Sqoop argument if share
lib is used (was: Now Sqoop work flow launched by Oozie will copy the jar
files in %SQOOP_HOME%\lib folder to the job cache each time a Sqoop job is
launched. We should add these jars to Sqoop sharelib so that for each worker
node, the jars in the Sqoop sharelib only copied to distribute cache and
localize to each worker node once and reuse by all the subsequent Sqoop jobs.
In Sqoop-1192, an option "--skip-dist-cache" is introduced which disable Sqoop
add lib jars to job cache action and reuse the jars in the Sqoop sharelib. In
the patch attached, Oozie will add "--skip-dist-cache" option to Sqoop argument
if share lib is used)
> Optimize I/O for Sqoop workflow launched by Oozie
> -------------------------------------------------
>
> Key: OOZIE-1521
> URL: https://issues.apache.org/jira/browse/OOZIE-1521
> Project: Oozie
> Issue Type: Improvement
> Reporter: Shuaishuai Nie
>
> Now Sqoop work flow launched by Oozie will copy the jar files in
> %SQOOP_HOME%\lib folder to the job cache each time a Sqoop job is launched.
> We should add these jars to Sqoop sharelib so that for each worker node, the
> jars in the Sqoop sharelib only copied to distribute cache and localize to
> each worker node once and reuse by all the subsequent Sqoop jobs. In
> Sqoop-1192, an option '--skip-dist-cache' is introduced which disable Sqoop
> add lib jars to job cache action and reuse the jars in the Sqoop sharelib. In
> the patch attached, Oozie will add "--skip-dist-cache" option to Sqoop
> argument if share lib is used
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
