Author: rkanter
Date: Thu Dec 19 02:30:46 2013
New Revision: 1552208
URL: http://svn.apache.org/r1552208
Log:
OOZIE-1576 Add documentation for Oozie Sqoop CLI (bowenzhangusa via rkanter)
Modified:
oozie/trunk/docs/src/site/twiki/DG_CommandLineTool.twiki
oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki
oozie/trunk/release-log.txt
Modified: oozie/trunk/docs/src/site/twiki/DG_CommandLineTool.twiki
URL:
http://svn.apache.org/viewvc/oozie/trunk/docs/src/site/twiki/DG_CommandLineTool.twiki?rev=1552208&r1=1552207&r2=1552208&view=diff
==============================================================================
--- oozie/trunk/docs/src/site/twiki/DG_CommandLineTool.twiki (original)
+++ oozie/trunk/docs/src/site/twiki/DG_CommandLineTool.twiki Thu Dec 19
02:30:46 2013
@@ -111,6 +111,14 @@ usage:
-oozie <arg> Oozie URL
-P <property=value> set parameters for script
.
+ oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments
after '-X' are put in <configuration>
+ -auth <arg> select authentication type
[SIMPLE|KERBEROS]
+ -config <arg> job configuration file '.properties'
+ -D <property=value> set/override value for given property
+ -doas <arg> doAs user, impersonates as the
specified user
+ -command <arg> sqoop command
+ -oozie <arg> Oozie URL
+.
oozie info <OPTIONS> : get more detailed info about specific topics
-timezones display a list of available time zones
.
@@ -1079,6 +1087,52 @@ Multiple -D and -P arguments can be spec
The job will be created and run right away.
+---++ Sqoop Operations
+
+---+++ Submitting a sqoop job through HTTP
+
+Syntax:
+
+<verbatim>
+$ oozie sqoop [-Dkey=value] -command completeSqoopCommand -config OOZIE-CONFIG
[-X [-Dkey=value opts for Launcher/Job configuration]]
+</verbatim>
+
+Example:
+
+<verbatim>
+$ oozie sqoop -oozie http://localhost:11000/oozie
-Dfs.default.name=hdfs://localhost:8020 -command import --connect
jdbc:mysql://localhost:3306/oozie --username oozie --password oozie --table
WF_JOBS --target-dir '/user/${wf:user()}/${examplesRoot}/output-data/sqoop' -m
1 -config job.properties -X -Dmapred.job.queue.name=default
+.
+job: 14-20090525161322-oozie-joe-W
+.
+</verbatim>
+
+Sqoop Freeform Example:
+<verbatim>
+$ oozie sqoop -oozie http://localhost:11000/oozie -command import --connect
jdbc:mysql://localhost:3306/oozie --username oozie --password oozie --query
"SELECT a.id FROM WF_JOBS a WHERE \$CONDITIONS" --target-dir
'/user/${wf:user()}/${examplesRoot}/output-data/sqoop' -m 1 -config
job.properties -X -Dmapred.job.queue.name=default
+.
+job: 14-20090525161321-oozie-joe-W
+.
+$cat job.properties
+fs.default.name=hdfs://localhost:8020
+mapreduce.jobtracker.kerberos.principal=ccc
+dfs.namenode.kerberos.principal=ddd
+oozie.libpath=hdfs://localhost:8020/user/oozie/sqoop/lib/
+</verbatim>
+
+The parameters for the job must be provided in a Java Properties file
(.properties). jobtracker, namenode,
+libpath must be specified in this file. All jar files (including sqoop jar
file) and all other files needed by the
+sqoop job need to be uploaded onto HDFS under libpath beforehand. The
workflow.xml will be created in Oozie server
+internally. Users can get the workflow.xml from console or command
line(-definition).
+The -D options passed after the -X will be placed into the generated
workflow's <code><configuration> elements</code>
+(and make it to the configuration used by Sqoop); Multiple -D arguments can be
specified.
+
+The job will be created and run right away.
+
+Note: in the freeform query example, the "select" query itself must be double
quoted and the "$" sign in the query is
+properly escaped by "\". And all other variables containing "$" within sqoop
command are escaped by single quoting the
+variable itself like the value of "--target-dir". All the "-D" arguments
before "-X" that are overriding given property
+must be placed before the "-command" argument.
+
---++ Info Operations
The Info sub-command provides a convenient place for Oozie to display misc
information.
Modified: oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki
URL:
http://svn.apache.org/viewvc/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki?rev=1552208&r1=1552207&r2=1552208&view=diff
==============================================================================
--- oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki (original)
+++ oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki Thu Dec 19 02:30:46
2013
@@ -787,6 +787,80 @@ Content-Type: application/json;charset=U
}
</verbatim>
+---++++ Proxy Sqoop Job Submission
+
+You can submit a Workflow that contains a single Sqoop command without writing
a workflow.xml. Any requred Jars or other
+ files must already exist in HDFS.
+
+The following properties are required:
+ * =fs.default.name=: The NameNode
+ * =mapred.job.tracker=: The JobTracker
+ * =user.name=: The username of the user submitting the job
+ * =oozie.sqoop.command=: The sqoop command you want to run where each
argument occupies one line or separated by "\n"
+ * =oozie.libpath=: A directory in HDFS that contains necessary Jars for
your job
+ * =oozie.proxysubmission=: Must be set to =true=
+
+The following properties are optional:
+ * =oozie.sqoop.options.size=: The number of options you'll be passing to
Sqoop Hadoop job
+ * =oozie.sqoop.options.n=: An argument to pass to Sqoop hadoop job conf,
the 'n' should be an integer starting with 0 to indicate the option number
+
+*Request:*
+
+<verbatim>
+POST /oozie/v1/jobs?jobtype=sqoop
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>fs.default.name</name>
+ <value>hdfs://localhost:8020</value>
+ </property>
+ <property>
+ <name>mapred.job.tracker</name>
+ <value>localhost:8021</value>
+ </property>
+ <property>
+ <name>user.name</name>
+ <value>bzhang</value>
+ </property>
+ <property>
+ <name>oozie.sqoop.command</name>
+ <value>
+ import
+ --connect
+ jdbc:mysql://localhost:3306/oozie
+ --username
+ oozie
+ --password
+ oozie
+ --table
+ WF_JOBS
+ --target-dir
+ /user/${wf:user()}/${examplesRoot}/output-data/sqoop
+ </value>
+ </property>
+ <name>oozie.libpath</name>
+ <value>hdfs://localhost:8020/user/bzhang/share/lib/sqoop</value>
+ </property>
+ <property>
+ <name>oozie.proxysubmission</name>
+ <value>true</value>
+ </property>
+</configuration>
+</verbatim>
+
+*Response:*
+
+<verbatim>
+HTTP/1.1 201 CREATED
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "job-3"
+}
+</verbatim>
+
---++++ Managing a Job
A HTTP PUT request starts, suspends, resumes, kills, or dryruns a job.
Modified: oozie/trunk/release-log.txt
URL:
http://svn.apache.org/viewvc/oozie/trunk/release-log.txt?rev=1552208&r1=1552207&r2=1552208&view=diff
==============================================================================
--- oozie/trunk/release-log.txt (original)
+++ oozie/trunk/release-log.txt Thu Dec 19 02:30:46 2013
@@ -1,5 +1,6 @@
-- Oozie 4.1.0 release (trunk - unreleased)
+OOZIE-1576 Add documentation for Oozie Sqoop CLI (bowenzhangusa via rkanter)
OOZIE-1616 Add sharelib and launcherlib locations to the instrumentation info
(rkanter)
OOZIE-1647 oozie-setup.sh doesn't check exit code of java executions (alazarev
via rkanter)
OOZIE-1642 writeUTF 64k limit for counters (puru via rohini)
