Updated Branches: refs/heads/sqoop2 ae075f2e3 -> 135f00cf1
SQOOP-925: Sqoop2: Devguide: Describe client API for embedding Sqoop client in applications (Vasanth kumar RJ via Jarek Jarcec Cecho) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/135f00cf Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/135f00cf Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/135f00cf Branch: refs/heads/sqoop2 Commit: 135f00cf1285a63611bf33ee66a29d3204452982 Parents: ae075f2 Author: Jarek Jarcec Cecho <[email protected]> Authored: Sat Apr 13 11:50:23 2013 -0700 Committer: Jarek Jarcec Cecho <[email protected]> Committed: Sat Apr 13 11:50:23 2013 -0700 ---------------------------------------------------------------------- docs/src/site/sphinx/ClientAPI.rst | 319 +++++++++++++++++++++++++++++++ docs/src/site/sphinx/conf.py | 2 +- docs/src/site/sphinx/index.rst | 1 + 3 files changed, 321 insertions(+), 1 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/sqoop/blob/135f00cf/docs/src/site/sphinx/ClientAPI.rst ---------------------------------------------------------------------- diff --git a/docs/src/site/sphinx/ClientAPI.rst b/docs/src/site/sphinx/ClientAPI.rst new file mode 100644 index 0000000..a9d39fb --- /dev/null +++ b/docs/src/site/sphinx/ClientAPI.rst @@ -0,0 +1,319 @@ +.. Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +====================== +Sqoop Client API Guide +====================== + +This document will explain how to use Sqoop Client API with external application. Client API allows you to execute the functions of sqoop commands. It requires Sqoop Client JAR and its dependencies. + +Client API is explained using Generic JDBC Connector properties. Before executing the application using the sqoop client API, check whether sqoop server is running. + +Workflow +======== + +Given workflow has to be followed for executing a job in Sqoop server. + + 1. Create connection using Connector ID (cid) - Creates connection and returns connection ID (xid) + 2. Create Job using Connection ID (xid) - Create job and returns Job ID (jid) + 3. Job submission with Job ID (jid) - Submit sqoop Job to server + +Project Dependencies +==================== +Here given maven dependency + +:: + + <dependency> + <groupId>org.apache.sqoop</groupId> + <artifactId>sqoop-client</artifactId> + <version>${requestedVersion}</version> + </dependency> + +Initialization +============== + +First initialize the SqoopClient class with server URL as argument. + +:: + + String url = "http://localhost:12000/sqoop/";; + SqoopClient client = new SqoopClient(url); + +Server URL value can be modfied by setting value to setServerUrl(String) method + +:: + + client.setServerUrl(newUrl); + + +Connection +========== + +Client API allows you to create, update and delete connection. For creating or updating connection requires Connector forms and Framwork Forms. User has to retrive the connector and framework forms, then update the values. + +Create Connection +----------------- + +First create a new connection by invoking newConnection(cid) method with connector ID and returns a MConnection object with dummy id. Then fill the connection and framework forms as given below. Invoke create connection with updated connection object. + +:: + + //Dummy connection object + MConnection newCon = client.newConnection(1); + + //Get connection and framework forms. Set name for connection + MConnectionForms conForms = newCon.getConnectorPart(); + MConnectionForms frameworkForms = newCon.getFrameworkPart(); + newCon.setName("MyConnection"); + + //Set connection forms values + conForms.getStringInput("connection.connectionString").setValue("jdbc:mysql://localhost/my"); + conForms.getStringInput("connection.jdbcDriver").setValue("com.mysql.jdbc.Driver"); + conForms.getStringInput("connection.username").setValue("root"); + conForms.getStringInput("connection.password").setValue("root"); + + frameworkForms.getIntegerInput("security.maxConnections").setValue(0); + + Status status = client.createConnection(newCon); + if(status.canProceed()) { + System.out.println("Created. New Connection ID : " +newCon.getPersistenceId()); + } else { + System.out.println("Check for status and forms error "); + } + +status.canProceed() returns true if status is FINE or ACCEPTABLE. Above code has given status after validation of connector and framework forms. + +On successful execution, new connection ID is assigned for the connection. getPersistenceId() method returns ID. +User can retrieve a connection using below methods + ++----------------------------+--------------------------------------+ +| Method | Description | ++============================+======================================+ +| ``getConnection(xid)`` | Returns a connection object. | ++----------------------------+--------------------------------------+ +| ``getConnections()`` | Returns list of connection object | ++----------------------------+--------------------------------------+ + +List of status code +------------------- + ++------------------+------------------------------------------------------------------------------------------------------------+ +| Function | Description | ++==================+============================================================================================================+ +| ``FINE`` | There are no issues, no warnings. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``ACCEPTABLE`` | Validated entity is correct enough to be processed. There might be some warnings, but no errors. | ++------------------+------------------------------------------------------------------------------------------------------------+ +| ``UNACCEPTABLE`` | There are serious issues with validated entity. We can't proceed until reported issues will be resolved. | ++------------------+------------------------------------------------------------------------------------------------------------+ + +View Error or Warning message +----------------------------- + +In case of any UNACCEPTABLE AND ACCEPTABLE status, user has to iterate the connector part forms and framework part forms for getting actual error or warning message. Below piece of code describe how to itereate over the forms for input message. + +:: + + printMessage(newCon.getConnectorPart().getForms()); + printMessage(newCon.getFrameworkPart().getForms()); + + private static void printMessage(List<MForm> formList) { + for(MForm form : formList) { + List<MInput<?>> inputlist = form.getInputs(); + if (form.getValidationMessage() != null) { + System.out.println("Form message: " + form.getValidationMessage()); + } + for (MInput minput : inputlist) { + if (minput.getValidationStatus() == Status.ACCEPTABLE) { + System.out.println("Warning:" + minput.getValidationMessage()); + } else if (minput.getValidationStatus() == Status.UNACCEPTABLE) { + System.out.println("Error:" + minput.getValidationMessage()); + } + } + } + } + +Job +=== + +A job object holds database configurations, input or output configurations and resources required for executing as a hadoop job. Create job object requires filling connector part and framework part forms. + +Below given code shows how to create a import job + +:: + + String url = "http://localhost:12000/sqoop/";; + SqoopClient client = new SqoopClient(url); + //Creating dummy job object + MJob newjob = client.newJob(1, org.apache.sqoop.model.MJob.Type.IMPORT); + MJobForms connectorForm = newjob.getConnectorPart(); + MJobForms frameworkForm = newjob.getFrameworkPart(); + + newjob.setName("ImportJob"); + //Database configuration + connectorForm.getStringInput("table.schemaName").setValue(""); + //Input either table name or sql + connectorForm.getStringInput("table.tableName").setValue("table"); + //connectorForm.getStringInput("table.sql").setValue("select id,name from table where ${CONDITIONS}"); + connectorForm.getStringInput("table.columns").setValue("id,name"); + connectorForm.getStringInput("table.partitionColumn").setValue("id"); + //Set boundary value only if required + //connectorForm.getStringInput("table.boundaryQuery").setValue(""); + + //Output configurations + frameworkForm.getEnumInput("output.storageType").setValue("HDFS"); + frameworkForm.getEnumInput("output.outputFormat").setValue("TEXT_FILE");//Other option: SEQUENCE_FILE + frameworkForm.getStringInput("output.outputDirectory").setValue("/output"); + + //Job resources + frameworkForm.getIntegerInput("throttling.extractors").setValue(1); + frameworkForm.getIntegerInput("throttling.loaders").setValue(1); + + Status status = client.createJob(newjob); + if(status.canProceed()) { + System.out.println("New Job ID: "+ newjob.getPersistenceId()); + } else { + System.out.println("Check for status and forms error "); + } + + //Print errors or warnings + printMessage(newjob.getConnectorPart().getForms()); + printMessage(newjob.getFrameworkPart().getForms()); + + +Export job creation is same as import job, but only few input configuration changes + +:: + + String url = "http://localhost:12000/sqoop/";; + SqoopClient client = new SqoopClient(url); + MJob newjob = client.newJob(1, org.apache.sqoop.model.MJob.Type.EXPORT); + MJobForms connectorForm = newjob.getConnectorPart(); + MJobForms frameworkForm = newjob.getFrameworkPart(); + + newjob.setName("ExportJob"); + //Database configuration + connectorForm.getStringInput("table.schemaName").setValue(""); + //Input either table name or sql + connectorForm.getStringInput("table.tableName").setValue("table"); + //connectorForm.getStringInput("table.sql").setValue("select id,name from table where ${CONDITIONS}"); + connectorForm.getStringInput("table.columns").setValue("id,name"); + + //Input configurations + frameworkForm.getStringInput("input.inputDirectory").setValue("/input"); + + //Job resources + frameworkForm.getIntegerInput("throttling.extractors").setValue(1); + frameworkForm.getIntegerInput("throttling.loaders").setValue(1); + + Status status = client.createJob(newjob); + if(status.canProceed()) { + System.out.println("New Job ID: "+ newjob.getPersistenceId()); + } else { + System.out.println("Check for status and forms error "); + } + + //Print errors or warnings + printMessage(newjob.getConnectorPart().getForms()); + printMessage(newjob.getFrameworkPart().getForms()); + +Managing connection and job +--------------------------- +After creating connection or job object, you can update or delete a connection or job using given functions + ++----------------------------------+------------------------------------------------------------------------------------+ +| Method | Description | ++==================================+====================================================================================+ +| ``updateConnection(connection)`` | Invoke update with connection object and check status for any errors or warnings | ++----------------------------------+------------------------------------------------------------------------------------+ +| ``deleteConnection(xid)`` | Delete connection. Deletes only if specified connection is used by any job | ++----------------------------------+------------------------------------------------------------------------------------+ +| ``updateJob(job)`` | Invoke update with job object and check status for any errors or warnings | ++----------------------------------+------------------------------------------------------------------------------------+ +| ``deleteJob(jid)`` | Delete job | ++----------------------------------+------------------------------------------------------------------------------------+ + +Job Submission +============== + +Job submission requires a job id. On successful submission, getStatus() method returns "BOOTING" or "RUNNING". + +:: + + //Job submission start + MSubmission submission = client.startSubmission(1); + System.out.println("Status : " + submission.getStatus()); + if(submission.getStatus().isRunning() && submission.getProgress() != -1) { + System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100)); + } + System.out.println("Hadoop job id :" + submission.getExternalId()); + System.out.println("Job link : " + submission.getExternalLink()); + Counters counters = submission.getCounters(); + if(counters != null) { + System.out.println("Counters:"); + for(CounterGroup group : counters) { + System.out.print("\t"); + System.out.println(group.getName()); + for(Counter counter : group) { + System.out.print("\t\t"); + System.out.print(counter.getName()); + System.out.print(": "); + System.out.println(counter.getValue()); + } + } + } + if(submission.getExceptionInfo() != null) { + System.out.println("Exception info : " +submission.getExceptionInfo()); + } + + + //Check job status + MSubmission submission = client.getSubmissionStatus(1); + if(submission.getStatus().isRunning() && submission.getProgress() != -1) { + System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100)); + } + + //Stop a running job + submission.stopSubmission(jid); + +Describe Forms +========================== + +You can view the connection or job forms input values with labels of built-in resource bundle. + +:: + + String url = "http://localhost:12000/sqoop/";; + SqoopClient client = new SqoopClient(url); + //Use getJob(jid) for describing job. + //While printing connection forms, pass connector id to getResourceBundle(cid). + describe(client.getConnection(1).getConnectorPart().getForms(), client.getResourceBundle(1)); + describe(client.getConnection(1).getFrameworkPart().getForms(), client.getFrameworkResourceBundle()); + + void describe(List<MForm> forms, ResourceBundle resource) { + for (MForm mf : forms) { + System.out.println(resource.getString(mf.getLabelKey())+":"); + List<MInput<?>> mis = mf.getInputs(); + for (MInput mi : mis) { + System.out.println(resource.getString(mi.getLabelKey()) + " : " + mi.getValue()); + } + System.out.println(); + } + } + + +Above Sqoop 2 Client API tutorial explained you how to create connection, create job and submit job. http://git-wip-us.apache.org/repos/asf/sqoop/blob/135f00cf/docs/src/site/sphinx/conf.py ---------------------------------------------------------------------- diff --git a/docs/src/site/sphinx/conf.py b/docs/src/site/sphinx/conf.py index 642d065..6a9bf31 100644 --- a/docs/src/site/sphinx/conf.py +++ b/docs/src/site/sphinx/conf.py @@ -57,7 +57,7 @@ master_doc = 'index' # General information about the project. project = 'Apache Sqoop' -copyright = '2009-2012 The Apache Software Foundation' +copyright = '2009-2013 The Apache Software Foundation' keep_warnings = True http://git-wip-us.apache.org/repos/asf/sqoop/blob/135f00cf/docs/src/site/sphinx/index.rst ---------------------------------------------------------------------- diff --git a/docs/src/site/sphinx/index.rst b/docs/src/site/sphinx/index.rst index 07f1c31..15ddfbb 100644 --- a/docs/src/site/sphinx/index.rst +++ b/docs/src/site/sphinx/index.rst @@ -60,3 +60,4 @@ Developer Guide - `Building Sqoop2 <BuildingSqoop2.html>`_ - `Development Environment Setup <DevEnv.html>`_ +- `Java Client API Guide <ClientAPI.html>`_
