Added: websites/staging/sqoop/trunk/content/docs/1.99.5/RESTAPI.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.5/RESTAPI.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.5/RESTAPI.html Fri Feb 27 00:59:26 2015 @@ -0,0 +1,1654 @@ + + +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>Sqoop REST API Guide — Apache Sqoop documentation</title> + + <link rel="stylesheet" href="_static/haiku.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + <link rel="stylesheet" href="_static/print.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="_static/theme_extras.js"></script> + <link rel="top" title="Apache Sqoop documentation" href="index.html" /> + </head> + <body> + <div class="header"><img class="rightlogo" src="_static/sqoop-logo.png" alt="Logo"/><h1 class="heading"><a href="index.html"> + <span>Apache Sqoop documentation</span></a></h1> + <h2 class="heading"><span>Sqoop REST API Guide</span></h2> + </div> + <div class="topnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + <div class="content"> + + + <div class="section" id="sqoop-rest-api-guide"> +<h1><a class="toc-backref" href="#id1">Sqoop REST API Guide</a><a class="headerlink" href="#sqoop-rest-api-guide" title="Permalink to this headline">¶</a></h1> +<p>This document will explain how you can use Sqoop REST API to build applications interacting with Sqoop server. +The REST API covers all aspects of managing Sqoop jobs and allows you to build an app in any programming language using HTTP over JSON.</p> +<div class="contents topic" id="table-of-contents"> +<p class="topic-title first">Table of Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#sqoop-rest-api-guide" id="id1">Sqoop REST API Guide</a><ul> +<li><a class="reference internal" href="#initialization" id="id2">Initialization</a></li> +<li><a class="reference internal" href="#understand-connector-driver-link-and-job" id="id3">Understand Connector, Driver, Link and Job</a></li> +<li><a class="reference internal" href="#objects" id="id4">Objects</a><ul> +<li><a class="reference internal" href="#configs-and-inputs" id="id5">Configs and Inputs</a></li> +<li><a class="reference internal" href="#exception-response" id="id6">Exception Response</a></li> +<li><a class="reference internal" href="#config-and-input-validation-status-response" id="id7">Config and Input Validation Status Response</a></li> +<li><a class="reference internal" href="#job-submission-status-response" id="id8">Job Submission Status Response</a></li> +</ul> +</li> +<li><a class="reference internal" href="#header-parameters" id="id9">Header Parameters</a></li> +<li><a class="reference internal" href="#rest-apis" id="id10">REST APIs</a><ul> +<li><a class="reference internal" href="#version-get-get-sqoop-version" id="id11">/version - [GET] - Get Sqoop Version</a></li> +<li><a class="reference internal" href="#v1-connectors-get-get-all-connectors" id="id12">/v1/connectors - [GET] Get all Connectors</a></li> +<li><a class="reference internal" href="#v1-connector-cname-or-v1-connector-cid-get-get-connector" id="id13">/v1/connector/[cname] or /v1/connector/[cid] - [GET] - Get Connector</a></li> +<li><a class="reference internal" href="#v1-driver-get-get-sqoop-driver" id="id14">/v1/driver - [GET]- Get Sqoop Driver</a></li> +<li><a class="reference internal" href="#v1-links-get-get-all-links" id="id15">/v1/links/ - [GET] Get all links</a></li> +<li><a class="reference internal" href="#v1-links-cname-cname-get-get-all-links-by-connector" id="id16">/v1/links?cname=[cname] - [GET] Get all links by Connector</a></li> +<li><a class="reference internal" href="#v1-link-lname-or-v1-link-lid-get-get-link" id="id17">/v1/link/[lname] or /v1/link/[lid] - [GET] - Get Link</a></li> +<li><a class="reference internal" href="#v1-link-post-create-link" id="id18">/v1/link - [POST] - Create Link</a></li> +<li><a class="reference internal" href="#v1-link-lname-or-v1-link-lid-put-update-link" id="id19">/v1/link/[lname] or /v1/link/[lid] - [PUT] - Update Link</a></li> +<li><a class="reference internal" href="#v1-link-lname-or-v1-link-lid-delete-delete-link" id="id20">/v1/link/[lname] or /v1/link/[lid] - [DELETE] - Delete Link</a></li> +<li><a class="reference internal" href="#v1-link-lid-enable-or-v1-link-lname-enable-put-enable-link" id="id21">/v1/link/[lid]/enable or /v1/link/[lname]/enable - [PUT] - Enable Link</a></li> +<li><a class="reference internal" href="#v1-link-lid-disable-put-disable-link" id="id22">/v1/link/[lid]/disable - [PUT] - Disable Link</a></li> +<li><a class="reference internal" href="#v1-jobs-get-get-all-jobs" id="id23">/v1/jobs/ - [GET] Get all jobs</a></li> +<li><a class="reference internal" href="#v1-jobs-cname-cname-get-get-all-jobs-by-connector" id="id24">/v1/jobs?cname=[cname] - [GET] Get all jobs by connector</a></li> +<li><a class="reference internal" href="#v1-job-jname-or-v1-job-jid-get-get-job" id="id25">/v1/job/[jname] or /v1/job/[jid] - [GET] - Get Job</a></li> +<li><a class="reference internal" href="#v1-job-post-create-job" id="id26">/v1/job - [POST] - Create Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-put-update-job" id="id27">/v1/job/[jid] - [PUT] - Update Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-delete-delete-job" id="id28">/v1/job/[jid] - [DELETE] - Delete Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-enable-put-enable-job" id="id29">/v1/job/[jid]/enable - [PUT] - Enable Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-disable-put-disable-job" id="id30">/v1/job/[jid]/disable - [PUT] - Disable Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-start-or-v1-job-jname-start-put-start-job" id="id31">/v1/job/[jid]/start or /v1/job/[jname]/start - [PUT]- Start Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-stop-or-v1-job-jname-stop-put-stop-job" id="id32">/v1/job/[jid]/stop or /v1/job/[jname]/stop - [PUT]- Stop Job</a></li> +<li><a class="reference internal" href="#v1-job-jid-status-or-v1-job-jname-status-get-get-job-status" id="id33">/v1/job/[jid]/status or /v1/job/[jname]/status - [GET]- Get Job Status</a></li> +<li><a class="reference internal" href="#v1-submissions-get-get-all-job-submissions" id="id34">/v1/submissions? - [GET] - Get all job Submissions</a></li> +<li><a class="reference internal" href="#v1-submissions-jname-jname-get-get-submissions-by-job" id="id35">/v1/submissions?jname=[jname] - [GET] - Get Submissions by Job</a></li> +</ul> +</li> +</ul> +</li> +</ul> +</div> +<div class="section" id="initialization"> +<h2><a class="toc-backref" href="#id2">Initialization</a><a class="headerlink" href="#initialization" title="Permalink to this headline">¶</a></h2> +<p>Before continuing further, make sure that the Sqoop server is running.</p> +<p>Then find out the details of the Sqoop server: <tt class="docutils literal"><span class="pre">host</span></tt>, <tt class="docutils literal"><span class="pre">port</span></tt> and <tt class="docutils literal"><span class="pre">webapp</span></tt>, and keep them in mind. Note that the sqoop server is running on Apache Tomcat. To exercise a REST API for Sqoop, you could assemble and send a HTTP request to an url corresponding to that API. Generally, the url contains the <tt class="docutils literal"><span class="pre">host</span></tt> on which the sqoop server is running, the <tt class="docutils literal"><span class="pre">port</span></tt> at which the sqoop server is listening to and <tt class="docutils literal"><span class="pre">webapp</span></tt>, the context path at which the Sqoop server is registered in the Apache Tomcat engine.</p> +<p>Certain requests might need to contain some additional query parameters and post data. These parameters could be given via +the HTTP headers, request body or both. All the content in the HTTP body is in <tt class="docutils literal"><span class="pre">JSON</span></tt> format.</p> +</div> +<div class="section" id="understand-connector-driver-link-and-job"> +<h2><a class="toc-backref" href="#id3">Understand Connector, Driver, Link and Job</a><a class="headerlink" href="#understand-connector-driver-link-and-job" title="Permalink to this headline">¶</a></h2> +<p>To create and run a Sqoop Job, we need to provide config values for connecting to a data source and then processing the data in that data source. Processing might be either reading from or writing to the data source. Thus we have configurable entities such as the <tt class="docutils literal"><span class="pre">From</span></tt> and <tt class="docutils literal"><span class="pre">To</span></tt> parts of the connectors, the driver that each expose configs and one or more inputs within them.</p> +<p>For instance a connector that represents a relational data source such as MySQL will expose config classes for connecting to the database. Some of the relevant inputs are the connection string, driver class, the username and the password to connect to the database. These configs remain the same to read data from any of the tables within that database. Hence they are grouped under <tt class="docutils literal"><span class="pre">LinkConfiguration</span></tt>.</p> +<p>Each connector can support Reading from a data source and/or writing/to a data source it represents. Reading from and writing to a data source are represented by From and To respectively. Specific configurations are required to peform the job of reading from or writing to the data source. These are grouped in the <tt class="docutils literal"><span class="pre">FromJobConfiguration</span></tt> and <tt class="docutils literal"><span class="pre">ToJobConfiguration</span></tt> objects of the connector.</p> +<p>For instance, a connector that represents a relational data source such as MySQL will expose the table name to read from or the SQL query to use while reading data as a FromJobConfiguration. Similarly a connector that represents a data source such as HDFS, will expose the output directory to write to as a ToJobConfiguration.</p> +</div> +<div class="section" id="objects"> +<h2><a class="toc-backref" href="#id4">Objects</a><a class="headerlink" href="#objects" title="Permalink to this headline">¶</a></h2> +<p>This section covers all the objects that might exist in an API request and/or API response.</p> +<div class="section" id="configs-and-inputs"> +<h3><a class="toc-backref" href="#id5">Configs and Inputs</a><a class="headerlink" href="#configs-and-inputs" title="Permalink to this headline">¶</a></h3> +<p>Before creating any link for a connector or a job with associated <tt class="docutils literal"><span class="pre">From</span></tt> and <tt class="docutils literal"><span class="pre">To</span></tt> links, the first thing to do is getting familiar with all the configurations that the connector exposes.</p> +<p>Each config consists of the following information</p> +<table border="1" class="docutils"> +<colgroup> +<col width="24%" /> +<col width="76%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id of this config</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">inputs</span></tt></td> +<td>A array of inputs of this config</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">name</span></tt></td> +<td>The unique name of this config per connector</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">type</span></tt></td> +<td>The type of this config (LINK/ JOB)</td> +</tr> +</tbody> +</table> +<p>A typical config object is showing below:</p> +<div class="highlight-none"><div class="highlight"><pre> { + id:7, + inputs:[ + { + id: 25, + name: "throttlingConfig.numExtractors", + type: "INTEGER", + sensitive: false + }, + { + id: 26, + name: "throttlingConfig.numLoaders", + type: "INTEGER", + sensitive: false + } + ], + name: "throttlingConfig", + type: "JOB" +} +</pre></div> +</div> +<p>Each input object in a config is structured below:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="24%" /> +<col width="76%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id of this input</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">name</span></tt></td> +<td>The unique name of this input per config</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">type</span></tt></td> +<td>The data type of this input field</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">size</span></tt></td> +<td>The length of this input field</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">sensitive</span></tt></td> +<td>Whether this input contain sensitive information</td> +</tr> +</tbody> +</table> +<p>To send a filled config in the request, you should always use config id and input id to map the values to their correspondig names. +For example, the following request contains an input value <tt class="docutils literal"><span class="pre">com.mysql.jdbc.Driver</span></tt> with input id <tt class="docutils literal"><span class="pre">7</span></tt> inside a config with id <tt class="docutils literal"><span class="pre">4</span></tt> that belongs to a link with id <tt class="docutils literal"><span class="pre">3</span></tt></p> +<div class="highlight-none"><div class="highlight"><pre>link: { + id: 3, + enabled: true, + link-config-values: [{ + id: 4, + inputs: [{ + id: 7, + name: "linkConfig.jdbcDriver", + value: "com.mysql.jdbc.Driver", + type: "STRING", + size: 128, + sensitive: false + }, { + id: 8, + name: "linkConfig.connectionString", + value: "jdbc%3Amysql%3A%2F%2Fmysql.ent.cloudera.com%2Fsqoop", + type: "STRING", + size: 128, + sensitive: false + }, + ... + } + } +</pre></div> +</div> +</div> +<div class="section" id="exception-response"> +<h3><a class="toc-backref" href="#id6">Exception Response</a><a class="headerlink" href="#exception-response" title="Permalink to this headline">¶</a></h3> +<p>Each operation on Sqoop server might return an exception in the Http response. Remember to take this into account.The exception code and message could be found in both the header and body of the response.</p> +<p>Please jump to “Header Parameters” section to find how to get exception information from header.</p> +<p>In the body, the exception is expressed in <tt class="docutils literal"><span class="pre">JSON</span></tt> format. An example of the exception is:</p> +<div class="highlight-none"><div class="highlight"><pre>{ + "message":"DERBYREPO_0030:Unable to load specific job metadata from repository - Couldn't find job with id 2", + "stack-trace":[ + { + "file":"DerbyRepositoryHandler.java", + "line":1111, + "class":"org.apache.sqoop.repository.derby.DerbyRepositoryHandler", + "method":"findJob" + }, + { + "file":"JdbcRepository.java", + "line":451, + "class":"org.apache.sqoop.repository.JdbcRepository$16", + "method":"doIt" + }, + { + "file":"JdbcRepository.java", + "line":90, + "class":"org.apache.sqoop.repository.JdbcRepository", + "method":"doWithConnection" + }, + { + "file":"JdbcRepository.java", + "line":61, + "class":"org.apache.sqoop.repository.JdbcRepository", + "method":"doWithConnection" + }, + { + "file":"JdbcRepository.java", + "line":448, + "class":"org.apache.sqoop.repository.JdbcRepository", + "method":"findJob" + }, + { + "file":"JobRequestHandler.java", + "line":238, + "class":"org.apache.sqoop.handler.JobRequestHandler", + "method":"getJobs" + } + ], + "class":"org.apache.sqoop.common.SqoopException" +} +</pre></div> +</div> +</div> +<div class="section" id="config-and-input-validation-status-response"> +<h3><a class="toc-backref" href="#id7">Config and Input Validation Status Response</a><a class="headerlink" href="#config-and-input-validation-status-response" title="Permalink to this headline">¶</a></h3> +<p>The config and the inputs associated with the connectors also provide custom validation rules for the values given to these input fields. Sqoop applies these custom validators and its corresponding valdation logic when config values for the LINK and JOB are posted.</p> +<p>An example of a OK status with the persisted ID:</p> +<div class="highlight-none"><div class="highlight"><pre>{ + "id": 3, + "validation-result": [ + {} + ] +} +</pre></div> +</div> +<p>An example of ERROR status:</p> +<div class="highlight-none"><div class="highlight"><pre>{ + "validation-result": [ + { + "linkConfig": [ + { + "message": "Invalid URI. URI must either be null or a valid URI. Here are a few valid example URIs: hdfs://example.com:8020/, hdfs://example.com/, file:///, file:///tmp, file://localhost/tmp", + "status": "ERROR" + } + ] + } + ] +} +</pre></div> +</div> +</div> +<div class="section" id="job-submission-status-response"> +<h3><a class="toc-backref" href="#id8">Job Submission Status Response</a><a class="headerlink" href="#job-submission-status-response" title="Permalink to this headline">¶</a></h3> +<p>After starting a job, you could look up the running status of it. There could be 7 possible status:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="34%" /> +<col width="66%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Status</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">BOOTING</span></tt></td> +<td>In the middle of submitting the job</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">FAILURE_ON_SUBMIT</span></tt></td> +<td>Unable to submit this job to remote cluster</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">RUNNING</span></tt></td> +<td>The job is running now</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">SUCCEEDED</span></tt></td> +<td>Job finished successfully</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">FAILED</span></tt></td> +<td>Job failed</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">NEVER_EXECUTED</span></tt></td> +<td>The job has never been executed since created</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">UNKNOWN</span></tt></td> +<td>The status is unknown</td> +</tr> +</tbody> +</table> +</div> +</div> +<div class="section" id="header-parameters"> +<h2><a class="toc-backref" href="#id9">Header Parameters</a><a class="headerlink" href="#header-parameters" title="Permalink to this headline">¶</a></h2> +<p>For all the responses, the following parameters in the HTTP message header are available:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="23%" /> +<col width="9%" /> +<col width="68%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Parameter</th> +<th class="head">Required</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">sqoop-error-code</span></tt></td> +<td>false</td> +<td>The error code when some error happen in the server side for this request</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">sqoop-error-message</span></tt></td> +<td>false</td> +<td>The explanation for a error code</td> +</tr> +</tbody> +</table> +<p>So far, there are only these 2 parameters in the header of response message. They only exist when something bad happen in the server. +And they always come along with an exception message in the response body.</p> +</div> +<div class="section" id="rest-apis"> +<h2><a class="toc-backref" href="#id10">REST APIs</a><a class="headerlink" href="#rest-apis" title="Permalink to this headline">¶</a></h2> +<p>The section elaborates all the rest apis that are supported by the Sqoop server.</p> +<p>For all Sqoop requests, the following request parameters will be added automatically. However, this user name is only in simple mode. In Kerberos mode, this user name will be ignored by Sqoop server and user name in UGI which is authenticated by Kerberos server will be used instead.</p> +<table border="1" class="docutils"> +<colgroup> +<col width="32%" /> +<col width="68%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Parameter</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">user.name</span></tt></td> +<td>The name of the user who makes the requests</td> +</tr> +</tbody> +</table> +<div class="section" id="version-get-get-sqoop-version"> +<h3><a class="toc-backref" href="#id11">/version - [GET] - Get Sqoop Version</a><a class="headerlink" href="#version-get-get-sqoop-version" title="Permalink to this headline">¶</a></h3> +<p>Get all the version metadata of Sqoop software in the server side.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Fields of Response:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="26%" /> +<col width="74%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">source-revision</span></tt></td> +<td>The revision number of Sqoop source code</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">api-versions</span></tt></td> +<td>The version of network protocol</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">build-date</span></tt></td> +<td>The Sqoop release date</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">user</span></tt></td> +<td>The user who made the release</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">source-url</span></tt></td> +<td>The url of the source code trunk</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">build-version</span></tt></td> +<td>The version of Sqoop in the server side</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + source-url: "git://vbasavaraj.local/Users/vbasavaraj/Projects/SqoopRefactoring/sqoop2/common", + source-revision: "418c5f637c3f09b94ea7fc3b0a4610831373a25f", + build-version: "2.0.0-SNAPSHOT", + api-versions: [ + "v1" + ], + user: "vbasavaraj", + build-date: "Mon Nov 3 08:18:21 PST 2014" +} +</pre></div> +</div> +</div> +<div class="section" id="v1-connectors-get-get-all-connectors"> +<h3><a class="toc-backref" href="#id12">/v1/connectors - [GET] Get all Connectors</a><a class="headerlink" href="#v1-connectors-get-get-all-connectors" title="Permalink to this headline">¶</a></h3> +<p>Get all the connectors registered in Sqoop</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Example</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + connectors: [{ + id: 1, + link-config: [], + job-config: {}, + name: "hdfs-connector", + class: "org.apache.sqoop.connector.hdfs.HdfsConnector", + all-config-resources: {}, + version: "2.0.0-SNAPSHOT" + }, { + id: 2, + link-config: [], + job-config: {}, + name: "generic-jdbc-connector", + class: "org.apache.sqoop.connector.jdbc.GenericJdbcConnector", + all-config - resources: {}, + version: "2.0.0-SNAPSHOT" + }] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-connector-cname-or-v1-connector-cid-get-get-connector"> +<h3><a class="toc-backref" href="#id13">/v1/connector/[cname] or /v1/connector/[cid] - [GET] - Get Connector</a><a class="headerlink" href="#v1-connector-cname-or-v1-connector-cid-get-get-connector" title="Permalink to this headline">¶</a></h3> +<p>Provide the id or unique name of the connector in the url <tt class="docutils literal"><span class="pre">[cid]</span></tt> or <tt class="docutils literal"><span class="pre">[cname]</span></tt> part.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Fields of Response:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="23%" /> +<col width="77%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id for the connector ( registered as a configurable )</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">job-config</span></tt></td> +<td>Connector job config and inputs for both FROM and TO</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">link-config</span></tt></td> +<td>Connector link config and inputs</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">all-config-resources</span></tt></td> +<td>All config inputs labels and description for the given connector</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">version</span></tt></td> +<td>The build version required for config and input data upgrades</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + connector: { + id: 1, + job-config: { + TO: [{ + id: 3, + inputs: [{ + id: 3, + values: "TEXT_FILE,SEQUENCE_FILE", + name: "toJobConfig.outputFormat", + type: "ENUM", + sensitive: false + }, { + id: 4, + values: "NONE,DEFAULT,DEFLATE,GZIP,BZIP2,LZO,LZ4,SNAPPY,CUSTOM", + name: "toJobConfig.compression", + type: "ENUM", + sensitive: false + }, { + id: 5, + name: "toJobConfig.customCompression", + type: "STRING", + size: 255, + sensitive: false + }, { + id: 6, + name: "toJobConfig.outputDirectory", + type: "STRING", + size: 255, + sensitive: false + }], + name: "toJobConfig", + type: "JOB" + }], + FROM: [{ + id: 2, + inputs: [{ + id: 2, + name: "fromJobConfig.inputDirectory", + type: "STRING", + size: 255, + sensitive: false + }], + name: "fromJobConfig", + type: "JOB" + }] + }, + link-config: [{ + id: 1, + inputs: [{ + id: 1, + name: "linkConfig.uri", + type: "STRING", + size: 255, + sensitive: false + }], + name: "linkConfig", + type: "LINK" + }], + name: "hdfs-connector", + class: "org.apache.sqoop.connector.hdfs.HdfsConnector", + all-config-resources: { + fromJobConfig.label: "From Job configuration", + toJobConfig.ignored.label: "Ignored", + fromJobConfig.help: "Specifies information required to get data from Hadoop ecosystem", + toJobConfig.ignored.help: "This value is ignored", + toJobConfig.label: "ToJob configuration", + toJobConfig.storageType.label: "Storage type", + fromJobConfig.inputDirectory.label: "Input directory", + toJobConfig.outputFormat.label: "Output format", + toJobConfig.outputDirectory.label: "Output directory", + toJobConfig.outputDirectory.help: "Output directory for final data", + toJobConfig.compression.help: "Compression that should be used for the data", + toJobConfig.outputFormat.help: "Format in which data should be serialized", + toJobConfig.customCompression.label: "Custom compression format", + toJobConfig.compression.label: "Compression format", + linkConfig.label: "Link configuration", + toJobConfig.customCompression.help: "Full class name of the custom compression", + toJobConfig.storageType.help: "Target on Hadoop ecosystem where to store data", + linkConfig.help: "Here you supply information necessary to connect to HDFS", + linkConfig.uri.help: "HDFS URI used to connect to HDFS", + linkConfig.uri.label: "HDFS URI", + fromJobConfig.inputDirectory.help: "Directory that should be exported", + toJobConfig.help: "You must supply the information requested in order to get information where you want to store your data." + }, + version: "2.0.0-SNAPSHOT" + } +} +</pre></div> +</div> +</div> +<div class="section" id="v1-driver-get-get-sqoop-driver"> +<h3><a class="toc-backref" href="#id14">/v1/driver - [GET]- Get Sqoop Driver</a><a class="headerlink" href="#v1-driver-get-get-sqoop-driver" title="Permalink to this headline">¶</a></h3> +<p>Driver exposes configurations required for the job execution.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Fields of Response:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="79%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id for the driver ( registered as a configurable )</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">job-config</span></tt></td> +<td>Driver job config and inputs</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">version</span></tt></td> +<td>The build version of the driver</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">all-config-resources</span></tt></td> +<td>Driver exposed config and input labels and description</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + id: 3, + job-config: [{ + id: 7, + inputs: [{ + id: 25, + name: "throttlingConfig.numExtractors", + type: "INTEGER", + sensitive: false + }, { + id: 26, + name: "throttlingConfig.numLoaders", + type: "INTEGER", + sensitive: false + }], + name: "throttlingConfig", + type: "JOB" + }], + all-config-resources: { + throttlingConfig.numExtractors.label: "Extractors", + throttlingConfig.numLoaders.help: "Number of loaders that Sqoop will use", + throttlingConfig.numLoaders.label: "Loaders", + throttlingConfig.label: "Throttling resources", + throttlingConfig.numExtractors.help: "Number of extractors that Sqoop will use", + throttlingConfig.help: "Set throttling boundaries to not overload your systems" + }, + version: "1" +} +</pre></div> +</div> +</div> +<div class="section" id="v1-links-get-get-all-links"> +<h3><a class="toc-backref" href="#id15">/v1/links/ - [GET] Get all links</a><a class="headerlink" href="#v1-links-get-get-all-links" title="Permalink to this headline">¶</a></h3> +<p>Get all the links created in Sqoop</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Example</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + links: [ + { + id: 1, + enabled: true, + update-user: "root", + link-config-values: [], + name: "First Link", + creation-date: 1415309361756, + connector-id: 1, + update-date: 1415309361756, + creation-user: "root" + }, + { + id: 2, + enabled: true, + update-user: "root", + link-config-values: [], + name: "Second Link", + creation-date: 1415309390807, + connector-id: 2, + update-date: 1415309390807, + creation-user: "root" + } + ] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-links-cname-cname-get-get-all-links-by-connector"> +<h3><a class="toc-backref" href="#id16">/v1/links?cname=[cname] - [GET] Get all links by Connector</a><a class="headerlink" href="#v1-links-cname-cname-get-get-all-links-by-connector" title="Permalink to this headline">¶</a></h3> +<p>Get all the links for a given connector identified by <tt class="docutils literal"><span class="pre">[cname]</span></tt> part.</p> +</div> +<div class="section" id="v1-link-lname-or-v1-link-lid-get-get-link"> +<h3><a class="toc-backref" href="#id17">/v1/link/[lname] or /v1/link/[lid] - [GET] - Get Link</a><a class="headerlink" href="#v1-link-lname-or-v1-link-lid-get-get-link" title="Permalink to this headline">¶</a></h3> +<p>Provide the id or unique name of the link in the url <tt class="docutils literal"><span class="pre">[lid]</span></tt> or <tt class="docutils literal"><span class="pre">[lname]</span></tt> part.</p> +<p>Get all the details of the link including the id, name, type and the corresponding config input values for the link</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + link: { + id: 1, + enabled: true, + link-config-values: [{ + id: 1, + inputs: [{ + id: 1, + name: "linkConfig.uri", + value: "hdfs%3A%2F%2Fnamenode%3A8090", + type: "STRING", + size: 255, + sensitive: false + }], + name: "linkConfig", + type: "LINK" + }], + update-user: "root", + name: "First Link", + creation-date: 1415287846371, + connector-id: 1, + update-date: 1415287846371, + creation-user: "root" + } +} +</pre></div> +</div> +</div> +<div class="section" id="v1-link-post-create-link"> +<h3><a class="toc-backref" href="#id18">/v1/link - [POST] - Create Link</a><a class="headerlink" href="#v1-link-post-create-link" title="Permalink to this headline">¶</a></h3> +<p>Create a new link object. Provide values to the link config inputs for the ones that are required.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">POST</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Fields of Request:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="23%" /> +<col width="77%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">link</span></tt></td> +<td>The root of the post data in JSON</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id of the link can be left blank in the post data</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">enabled</span></tt></td> +<td>Whether to enable this link (true/false)</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">update-date</span></tt></td> +<td>The last updated time of this link</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">creation-date</span></tt></td> +<td>The creation time of this link</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">update-user</span></tt></td> +<td>The user who updated this link</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">creation-user</span></tt></td> +<td>The user who created this link</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">name</span></tt></td> +<td>The name of this link</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">link-config-values</span></tt></td> +<td>Config input values for link config for the corresponding connector</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">connector-id</span></tt></td> +<td>The id of the connector used for this link</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>Request Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + link: { + id: -1, + enabled: true, + link-config-values: [{ + id: 1, + inputs: [{ + id: 1, + name: "linkConfig.uri", + value: "hdfs%3A%2F%2Fvbsqoop-1.ent.cloudera.com%3A8020%2Fuser%2Froot%2Fjob1", + type: "STRING", + size: 255, + sensitive: false + }], + name: "testInput", + type: "LINK" + }], + update-user: "root", + name: "testLink", + creation-date: 1415202223048, + connector-id: 1, + update-date: 1415202223048, + creation-user: "root" + } +} +</pre></div> +</div> +<ul class="simple"> +<li>Fields of Response:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="24%" /> +<col width="76%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id assigned for this new created link</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">validation-result</span></tt></td> +<td>The validation status for the link config inputs given in the post data</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>ERROR Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "validation-result": [ + { + "linkConfig": [ + { + "message": "Invalid URI. URI must either be null or a valid URI. Here are a few valid example URIs: hdfs://example.com:8020/, hdfs://example.com/, file:///, file:///tmp, file://localhost/tmp", + "status": "ERROR" + } + ] + } + ] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-link-lname-or-v1-link-lid-put-update-link"> +<h3><a class="toc-backref" href="#id19">/v1/link/[lname] or /v1/link/[lid] - [PUT] - Update Link</a><a class="headerlink" href="#v1-link-lname-or-v1-link-lid-put-update-link" title="Permalink to this headline">¶</a></h3> +<p>Update an existing link object with name [lname] or id [lid]. To make the procedure of filling inputs easier, the general practice +is get the link first and then change some of the values for the inputs.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>OK Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "validation-result": [ + {} + ] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-link-lname-or-v1-link-lid-delete-delete-link"> +<h3><a class="toc-backref" href="#id20">/v1/link/[lname] or /v1/link/[lid] - [DELETE] - Delete Link</a><a class="headerlink" href="#v1-link-lname-or-v1-link-lid-delete-delete-link" title="Permalink to this headline">¶</a></h3> +<p>Delete a link with name [lname] or id [lid]</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">DELETE</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +</ul> +</div> +<div class="section" id="v1-link-lid-enable-or-v1-link-lname-enable-put-enable-link"> +<h3><a class="toc-backref" href="#id21">/v1/link/[lid]/enable or /v1/link/[lname]/enable - [PUT] - Enable Link</a><a class="headerlink" href="#v1-link-lid-enable-or-v1-link-lname-enable-put-enable-link" title="Permalink to this headline">¶</a></h3> +<p>Enable a link with id <tt class="docutils literal"><span class="pre">lid</span></tt> or name <tt class="docutils literal"><span class="pre">lname</span></tt></p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +</ul> +</div> +<div class="section" id="v1-link-lid-disable-put-disable-link"> +<h3><a class="toc-backref" href="#id22">/v1/link/[lid]/disable - [PUT] - Disable Link</a><a class="headerlink" href="#v1-link-lid-disable-put-disable-link" title="Permalink to this headline">¶</a></h3> +<p>Disable a link with id <tt class="docutils literal"><span class="pre">lid</span></tt> or name <tt class="docutils literal"><span class="pre">lname</span></tt></p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +</ul> +</div> +<div class="section" id="v1-jobs-get-get-all-jobs"> +<h3><a class="toc-backref" href="#id23">/v1/jobs/ - [GET] Get all jobs</a><a class="headerlink" href="#v1-jobs-get-get-all-jobs" title="Permalink to this headline">¶</a></h3> +<p>Get all the jobs created in Sqoop</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + jobs: [{ + driver-config-values: [], + enabled: true, + from-connector-id: 1, + update-user: "root", + to-config-values: [], + to-connector-id: 2, + creation-date: 1415310157618, + update-date: 1415310157618, + creation-user: "root", + id: 1, + to-link-id: 2, + from-config-values: [], + name: "First Job", + from-link-id: 1 + },{ + driver-config-values: [], + enabled: true, + from-connector-id: 2, + update-user: "root", + to-config-values: [], + to-connector-id: 1, + creation-date: 1415310650600, + update-date: 1415310650600, + creation-user: "root", + id: 2, + to-link-id: 1, + from-config-values: [], + name: "Second Job", + from-link-id: 2 + }] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-jobs-cname-cname-get-get-all-jobs-by-connector"> +<h3><a class="toc-backref" href="#id24">/v1/jobs?cname=[cname] - [GET] Get all jobs by connector</a><a class="headerlink" href="#v1-jobs-cname-cname-get-get-all-jobs-by-connector" title="Permalink to this headline">¶</a></h3> +<p>Get all the jobs for a given connector identified by <tt class="docutils literal"><span class="pre">[cname]</span></tt> part.</p> +</div> +<div class="section" id="v1-job-jname-or-v1-job-jid-get-get-job"> +<h3><a class="toc-backref" href="#id25">/v1/job/[jname] or /v1/job/[jid] - [GET] - Get Job</a><a class="headerlink" href="#v1-job-jname-or-v1-job-jid-get-get-job" title="Permalink to this headline">¶</a></h3> +<p>Provide the name or the id of the job in the url [jname] +part or [jid] part.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre> { + job: { + driver-config-values: [{ + id: 7, + inputs: [{ + id: 25, + name: "throttlingConfig.numExtractors", + value: "3", + type: "INTEGER", + sensitive: false + }, { + id: 26, + name: "throttlingConfig.numLoaders", + value: "3", + type: "INTEGER", + sensitive: false + }], + name: "throttlingConfig", + type: "JOB" + }], + enabled: true, + from-connector-id: 1, + update-user: "root", + to-config-values: [{ + id: 6, + inputs: [{ + id: 19, + name: "toJobConfig.schemaName", + type: "STRING", + size: 50, + sensitive: false + }, { + id: 20, + name: "toJobConfig.tableName", + value: "text", + type: "STRING", + size: 2000, + sensitive: false + }, { + id: 21, + name: "toJobConfig.sql", + type: "STRING", + size: 50, + sensitive: false + }, { + id: 22, + name: "toJobConfig.columns", + type: "STRING", + size: 50, + sensitive: false + }, { + id: 23, + name: "toJobConfig.stageTableName", + type: "STRING", + size: 2000, + sensitive: false + }, { + id: 24, + name: "toJobConfig.shouldClearStageTable", + type: "BOOLEAN", + sensitive: false + }], + name: "toJobConfig", + type: "JOB" + }], + to-connector-id: 2, + creation-date: 1415310157618, + update-date: 1415310157618, + creation-user: "root", + id: 1, + to-link-id: 2, + from-config-values: [{ + id: 2, + inputs: [{ + id: 2, + name: "fromJobConfig.inputDirectory", + value: "hdfs%3A%2F%2Fvbsqoop-1.ent.cloudera.com%3A8020%2Fuser%2Froot%2Fjob1", + type: "STRING", + size: 255, + sensitive: false + }], + name: "fromJobConfig", + type: "JOB" + }], + name: "First Job", + from-link- id: 1 + } +} +</pre></div> +</div> +</div> +<div class="section" id="v1-job-post-create-job"> +<h3><a class="toc-backref" href="#id26">/v1/job - [POST] - Create Job</a><a class="headerlink" href="#v1-job-post-create-job" title="Permalink to this headline">¶</a></h3> +<p>Create a new job object with the corresponding config values.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">POST</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Fields of Request:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="23%" /> +<col width="77%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">job</span></tt></td> +<td>The root of the post data in JSON</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">from-link-id</span></tt></td> +<td>The id of the from link for the job</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">to-link-id</span></tt></td> +<td>The id of the to link for the job</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">id</span></tt></td> +<td>The id of the link can be left blank in the post data</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">enabled</span></tt></td> +<td>Whether to enable this job (true/false)</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">update-date</span></tt></td> +<td>The last updated time of this job</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">creation-date</span></tt></td> +<td>The creation time of this job</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">update-user</span></tt></td> +<td>The user who updated this job</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">creation-user</span></tt></td> +<td>The uset who creates this job</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">name</span></tt></td> +<td>The name of this job</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">from-config-values</span></tt></td> +<td>Config input values for FROM part of the job</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">to-config-values</span></tt></td> +<td>Config input values for TO part of the job</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">driver-config-values</span></tt></td> +<td>Config input values for driver</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">connector-id</span></tt></td> +<td>The id of the connector used for this link</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>Request Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + job: { + driver-config-values: [ + { + id: 7, + inputs: [ + { + id: 25, + name: "throttlingConfig.numExtractors", + value: "3", + type: "INTEGER", + sensitive: false + }, + { + id: 26, + name: "throttlingConfig.numLoaders", + value: "3", + type: "INTEGER", + sensitive: false + } + ], + name: "throttlingConfig", + type: "JOB" + } + ], + enabled: true, + from-connector-id: 1, + update-user: "root", + to-config-values: [ + { + id: 6, + inputs: [ + { + id: 19, + name: "toJobConfig.schemaName", + type: "STRING", + size: 50, + sensitive: false + }, + { + id: 20, + name: "toJobConfig.tableName", + value: "text", + type: "STRING", + size: 2000, + sensitive: false + }, + { + id: 21, + name: "toJobConfig.sql", + type: "STRING", + size: 50, + sensitive: false + }, + { + id: 22, + name: "toJobConfig.columns", + type: "STRING", + size: 50, + sensitive: false + }, + { + id: 23, + name: "toJobConfig.stageTableName", + type: "STRING", + size: 2000, + sensitive: false + }, + { + id: 24, + name: "toJobConfig.shouldClearStageTable", + type: "BOOLEAN", + sensitive: false + } + ], + name: "toJobConfig", + type: "JOB" + } + ], + to-connector-id: 2, + creation-date: 1415310157618, + update-date: 1415310157618, + creation-user: "root", + id: -1, + to-link-id: 2, + from-config-values: [ + { + id: 2, + inputs: [ + { + id: 2, + name: "fromJobConfig.inputDirectory", + value: "hdfs%3A%2F%2Fvbsqoop-1.ent.cloudera.com%3A8020%2Fuser%2Froot%2Fjob1", + type: "STRING", + size: 255, + sensitive: false + } + ], + name: "fromJobConfig", + type: "JOB" + } + ], + name: "Test Job", + from-link-id: 1 + } + } +</pre></div> +</div> +<ul class="simple"> +<li>Fields of Response:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="23%" /> +<col width="0%" /> +<col width="77%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head" colspan="2">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td colspan="3"><tt class="docutils literal"><span class="pre">id</span></tt> | The id assigned for this new created job</td> +</tr> +<tr class="row-odd"><td colspan="3"><tt class="docutils literal"><span class="pre">validation-result</span></tt> | The validation status for the job config and driver config inputs in the post data</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>ERROR Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "validation-result": [ + { + "linkConfig": [ + { + "message": "Invalid URI. URI must either be null or a valid URI. Here are a few valid example URIs: hdfs://example.com:8020/, hdfs://example.com/, file:///, file:///tmp, file://localhost/tmp", + "status": "ERROR" + } + ] + } + ] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-job-jid-put-update-job"> +<h3><a class="toc-backref" href="#id27">/v1/job/[jid] - [PUT] - Update Job</a><a class="headerlink" href="#v1-job-jid-put-update-job" title="Permalink to this headline">¶</a></h3> +<p>Update an existing job object with id [jid]. To make the procedure of filling inputs easier, the general practice +is get the existing job object first and then change some of the inputs.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +</ul> +<p>The same as Create Job.</p> +<ul class="simple"> +<li>OK Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "validation-result": [ + {} + ] +} +</pre></div> +</div> +</div> +<div class="section" id="v1-job-jid-delete-delete-job"> +<h3><a class="toc-backref" href="#id28">/v1/job/[jid] - [DELETE] - Delete Job</a><a class="headerlink" href="#v1-job-jid-delete-delete-job" title="Permalink to this headline">¶</a></h3> +<p>Delete a job with id <tt class="docutils literal"><span class="pre">jid</span></tt>.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">DELETE</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +</ul> +</div> +<div class="section" id="v1-job-jid-enable-put-enable-job"> +<h3><a class="toc-backref" href="#id29">/v1/job/[jid]/enable - [PUT] - Enable Job</a><a class="headerlink" href="#v1-job-jid-enable-put-enable-job" title="Permalink to this headline">¶</a></h3> +<p>Enable a job with id <tt class="docutils literal"><span class="pre">jid</span></tt>.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +</ul> +</div> +<div class="section" id="v1-job-jid-disable-put-disable-job"> +<h3><a class="toc-backref" href="#id30">/v1/job/[jid]/disable - [PUT] - Disable Job</a><a class="headerlink" href="#v1-job-jid-disable-put-disable-job" title="Permalink to this headline">¶</a></h3> +<p>Disable a job with id <tt class="docutils literal"><span class="pre">jid</span></tt>.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +</ul> +</div> +<div class="section" id="v1-job-jid-start-or-v1-job-jname-start-put-start-job"> +<h3><a class="toc-backref" href="#id31">/v1/job/[jid]/start or /v1/job/[jname]/start - [PUT]- Start Job</a><a class="headerlink" href="#v1-job-jid-start-or-v1-job-jname-start-put-start-job" title="Permalink to this headline">¶</a></h3> +<p>Start a job with name <tt class="docutils literal"><span class="pre">[jname]</span></tt> or with id <tt class="docutils literal"><span class="pre">[jid]</span></tt> to trigger the job execution</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">POST</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">Submission</span> <span class="pre">Record</span></tt></li> +<li>BOOTING Response Example</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "submission": { + "progress": -1, + "last-update-date": 1415312531188, + "external-id": "job_1412137947693_0004", + "status": "BOOTING", + "job": 2, + "creation-date": 1415312531188, + "to-schema": { + "created": 1415312531426, + "name": "HDFS file", + "columns": [] + }, + "external-link": "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/", + "from-schema": { + "created": 1415312531342, + "name": "text", + "columns": [ + { + "name": "id", + "nullable": true, + "unsigned": null, + "type": "FIXED_POINT", + "size": null + }, + { + "name": "txt", + "nullable": true, + "type": "TEXT", + "size": null + } + ] + } + } +} +</pre></div> +</div> +<ul class="simple"> +<li>SUCCEEDED Response Example</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + submission: { + progress: -1, + last-update-date: 1415312809485, + external-id: "job_1412137947693_0004", + status: "SUCCEEDED", + job: 2, + creation-date: 1415312531188, + external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/", + counters: { + org.apache.hadoop.mapreduce.JobCounter: { + SLOTS_MILLIS_MAPS: 373553, + MB_MILLIS_MAPS: 382518272, + TOTAL_LAUNCHED_MAPS: 10, + MILLIS_MAPS: 373553, + VCORES_MILLIS_MAPS: 373553, + OTHER_LOCAL_MAPS: 10 + }, + org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter: { + BYTES_WRITTEN: 0 + }, + org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter: { + BYTES_READ: 0 + }, + org.apache.hadoop.mapreduce.TaskCounter: { + MAP_INPUT_RECORDS: 0, + MERGED_MAP_OUTPUTS: 0, + PHYSICAL_MEMORY_BYTES: 4065599488, + SPILLED_RECORDS: 0, + COMMITTED_HEAP_BYTES: 3439853568, + CPU_MILLISECONDS: 236900, + FAILED_SHUFFLE: 0, + VIRTUAL_MEMORY_BYTES: 15231422464, + SPLIT_RAW_BYTES: 1187, + MAP_OUTPUT_RECORDS: 1000000, + GC_TIME_MILLIS: 7282 + }, + org.apache.hadoop.mapreduce.FileSystemCounter: { + FILE_WRITE_OPS: 0, + FILE_READ_OPS: 0, + FILE_LARGE_READ_OPS: 0, + FILE_BYTES_READ: 0, + HDFS_BYTES_READ: 1187, + FILE_BYTES_WRITTEN: 1191230, + HDFS_LARGE_READ_OPS: 0, + HDFS_WRITE_OPS: 10, + HDFS_READ_OPS: 10, + HDFS_BYTES_WRITTEN: 276389736 + }, + org.apache.sqoop.submission.counter.SqoopCounters: { + ROWS_READ: 1000000 + } + } + } +} +</pre></div> +</div> +<ul class="simple"> +<li>ERROR Response Example</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "submission": { + "progress": -1, + "last-update-date": 1415312390570, + "status": "FAILURE_ON_SUBMIT", + "error-summary": "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run", + "job": 1, + "creation-date": 1415312390570, + "to-schema": { + "created": 1415312390797, + "name": "text", + "columns": [ + { + "name": "id", + "nullable": true, + "unsigned": null, + "type": "FIXED_POINT", + "size": null + }, + { + "name": "txt", + "nullable": true, + "type": "TEXT", + "size": null + } + ] + }, + "from-schema": { + "created": 1415312390778, + "name": "HDFS file", + "columns": [ + ] + }, + "error-details": "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_00" + } +} +</pre></div> +</div> +</div> +<div class="section" id="v1-job-jid-stop-or-v1-job-jname-stop-put-stop-job"> +<h3><a class="toc-backref" href="#id32">/v1/job/[jid]/stop or /v1/job/[jname]/stop - [PUT]- Stop Job</a><a class="headerlink" href="#v1-job-jid-stop-or-v1-job-jname-stop-put-stop-job" title="Permalink to this headline">¶</a></h3> +<p>Stop a job with name <tt class="docutils literal"><span class="pre">[janme]</span></tt> or with id <tt class="docutils literal"><span class="pre">[jid]</span></tt> to abort the running job.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">PUT</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">Submission</span> <span class="pre">Record</span></tt></li> +</ul> +</div> +<div class="section" id="v1-job-jid-status-or-v1-job-jname-status-get-get-job-status"> +<h3><a class="toc-backref" href="#id33">/v1/job/[jid]/status or /v1/job/[jname]/status - [GET]- Get Job Status</a><a class="headerlink" href="#v1-job-jid-status-or-v1-job-jname-status-get-get-job-status" title="Permalink to this headline">¶</a></h3> +<p>Get status of the running job with name <tt class="docutils literal"><span class="pre">[janme]</span></tt> or with id <tt class="docutils literal"><span class="pre">[jid]</span></tt></p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Response Content: <tt class="docutils literal"><span class="pre">Submission</span> <span class="pre">Record</span></tt></li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + "submission": { + "progress": 0.25, + "last-update-date": 1415312603838, + "external-id": "job_1412137947693_0004", + "status": "RUNNING", + "job": 2, + "creation-date": 1415312531188, + "external-link": "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/" + } +} +</pre></div> +</div> +</div> +<div class="section" id="v1-submissions-get-get-all-job-submissions"> +<h3><a class="toc-backref" href="#id34">/v1/submissions? - [GET] - Get all job Submissions</a><a class="headerlink" href="#v1-submissions-get-get-all-job-submissions" title="Permalink to this headline">¶</a></h3> +<p>Get all the submissions for every job started in SQoop</p> +</div> +<div class="section" id="v1-submissions-jname-jname-get-get-submissions-by-job"> +<h3><a class="toc-backref" href="#id35">/v1/submissions?jname=[jname] - [GET] - Get Submissions by Job</a><a class="headerlink" href="#v1-submissions-jname-jname-get-get-submissions-by-job" title="Permalink to this headline">¶</a></h3> +<p>Retrieve all job submissions in the past for the given job. Each submission record will have details such as the status, counters and urls for those submissions.</p> +<p>Provide the name of the job in the url [jname] part.</p> +<ul class="simple"> +<li>Method: <tt class="docutils literal"><span class="pre">GET</span></tt></li> +<li>Format: <tt class="docutils literal"><span class="pre">JSON</span></tt></li> +<li>Request Content: <tt class="docutils literal"><span class="pre">None</span></tt></li> +<li>Fields of Response:</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="23%" /> +<col width="77%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Field</th> +<th class="head">Description</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">progress</span></tt></td> +<td>The progress of the running Sqoop job</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">job</span></tt></td> +<td>The id of the Sqoop job</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">creation-date</span></tt></td> +<td>The submission timestamp</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">last-update-date</span></tt></td> +<td>The timestamp of the last status update</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">status</span></tt></td> +<td>The status of this job submission</td> +</tr> +<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">external-id</span></tt></td> +<td>The job id of Sqoop job running on Hadoop</td> +</tr> +<tr class="row-even"><td><tt class="docutils literal"><span class="pre">external-link</span></tt></td> +<td>The link to track the job status on Hadoop</td> +</tr> +</tbody> +</table> +<ul class="simple"> +<li>Response Example:</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>{ + submissions: [ + { + progress: -1, + last-update-date: 1415312809485, + external-id: "job_1412137947693_0004", + status: "SUCCEEDED", + job: 2, + creation-date: 1415312531188, + external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/", + counters: { + org.apache.hadoop.mapreduce.JobCounter: { + SLOTS_MILLIS_MAPS: 373553, + MB_MILLIS_MAPS: 382518272, + TOTAL_LAUNCHED_MAPS: 10, + MILLIS_MAPS: 373553, + VCORES_MILLIS_MAPS: 373553, + OTHER_LOCAL_MAPS: 10 + }, + org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter: { + BYTES_WRITTEN: 0 + }, + org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter: { + BYTES_READ: 0 + }, + org.apache.hadoop.mapreduce.TaskCounter: { + MAP_INPUT_RECORDS: 0, + MERGED_MAP_OUTPUTS: 0, + PHYSICAL_MEMORY_BYTES: 4065599488, + SPILLED_RECORDS: 0, + COMMITTED_HEAP_BYTES: 3439853568, + CPU_MILLISECONDS: 236900, + FAILED_SHUFFLE: 0, + VIRTUAL_MEMORY_BYTES: 15231422464, + SPLIT_RAW_BYTES: 1187, + MAP_OUTPUT_RECORDS: 1000000, + GC_TIME_MILLIS: 7282 + }, + org.apache.hadoop.mapreduce.FileSystemCounter: { + FILE_WRITE_OPS: 0, + FILE_READ_OPS: 0, + FILE_LARGE_READ_OPS: 0, + FILE_BYTES_READ: 0, + HDFS_BYTES_READ: 1187, + FILE_BYTES_WRITTEN: 1191230, + HDFS_LARGE_READ_OPS: 0, + HDFS_WRITE_OPS: 10, + HDFS_READ_OPS: 10, + HDFS_BYTES_WRITTEN: 276389736 + }, + org.apache.sqoop.submission.counter.SqoopCounters: { + ROWS_READ: 1000000 + } + } + }, + { + progress: -1, + last-update-date: 1415312390570, + status: "FAILURE_ON_SUBMIT", + error-summary: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run", + job: 1, + creation-date: 1415312390570, + error-details: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner...." + } + ] +} +</pre></div> +</div> +</div> +</div> +</div> + + + </div> + <div class="bottomnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + + <div class="footer"> + © Copyright 2009-2013 The Apache Software Foundation. + </div> + </body> +</html> \ No newline at end of file
Added: websites/staging/sqoop/trunk/content/docs/1.99.5/SecurityGuideOnSqoop2.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.5/SecurityGuideOnSqoop2.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.5/SecurityGuideOnSqoop2.html Fri Feb 27 00:59:26 2015 @@ -0,0 +1,205 @@ + + +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>Security Guide On Sqoop 2 — Apache Sqoop documentation</title> + + <link rel="stylesheet" href="_static/haiku.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + <link rel="stylesheet" href="_static/print.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="_static/theme_extras.js"></script> + <link rel="top" title="Apache Sqoop documentation" href="index.html" /> + </head> + <body> + <div class="header"><img class="rightlogo" src="_static/sqoop-logo.png" alt="Logo"/><h1 class="heading"><a href="index.html"> + <span>Apache Sqoop documentation</span></a></h1> + <h2 class="heading"><span>Security Guide On Sqoop 2</span></h2> + </div> + <div class="topnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + <div class="content"> + + + <div class="section" id="security-guide-on-sqoop-2"> +<h1>Security Guide On Sqoop 2<a class="headerlink" href="#security-guide-on-sqoop-2" title="Permalink to this headline">¶</a></h1> +<p>Most Hadoop components, such as HDFS, Yarn, Hive, etc., have security frameworks, which support Simple, Kerberos and LDAP authentication. currently Sqoop 2 provides 2 types of authentication: simple and kerberos. The authentication module is pluggable, so more authentication types can be added.</p> +<div class="section" id="simple-authentication"> +<h2>Simple Authentication<a class="headerlink" href="#simple-authentication" title="Permalink to this headline">¶</a></h2> +<div class="section" id="configuration"> +<h3>Configuration<a class="headerlink" href="#configuration" title="Permalink to this headline">¶</a></h3> +<p>Modify Sqoop configuration file, normally in <Sqoop Folder>/server/config/sqoop.properties.</p> +<div class="highlight-none"><div class="highlight"><pre>org.apache.sqoop.authentication.type=SIMPLE +org.apache.sqoop.authentication.handler=org.apache.sqoop.security.Authentication.SimpleAuthenticationHandler +org.apache.sqoop.anonymous=true +</pre></div> +</div> +<ul class="simple"> +<li>Simple authentication is used by default. Commenting out authentication configuration will yield the use of simple authentication.</li> +</ul> +</div> +<div class="section" id="run-command"> +<h3>Run command<a class="headerlink" href="#run-command" title="Permalink to this headline">¶</a></h3> +<p>Start Sqoop server as usual.</p> +<div class="highlight-none"><div class="highlight"><pre><Sqoop Folder>/bin/sqoop.sh server start +</pre></div> +</div> +<p>Start Sqoop client as usual.</p> +<div class="highlight-none"><div class="highlight"><pre><Sqoop Folder>/bin/sqoop.sh client +</pre></div> +</div> +</div> +</div> +<div class="section" id="kerberos-authentication"> +<h2>Kerberos Authentication<a class="headerlink" href="#kerberos-authentication" title="Permalink to this headline">¶</a></h2> +<p>Kerberos is a computer network authentication protocol which works on the basis of ‘tickets’ to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner. Its designers aimed it primarily at a clientâserver model and it provides mutual authenticationâboth the user and the server verify each other’s identity. Kerberos protocol messages are protected against eavesdropping and replay attacks.</p> +<div class="section" id="dependency"> +<h3>Dependency<a class="headerlink" href="#dependency" title="Permalink to this headline">¶</a></h3> +<p>Set up a KDC server. Skip this step if KDC server exists. It’s difficult to cover every way Kerberos can be setup (ie: there are cross realm setups and multi-trust environments). This section will describe how to setup the sqoop principals with a local deployment of MIT kerberos.</p> +<ul class="simple"> +<li>All components which are Kerberos authenticated need one KDC server. If current Hadoop cluster uses Kerberos authentication, there should be a KDC server.</li> +<li>If there is no KDC server, follow <a class="reference external" href="http://web.mit.edu/kerberos/krb5-devel/doc/admin/install_kdc.html";>http://web.mit.edu/kerberos/krb5-devel/doc/admin/install_kdc.html</a> to set up one.</li> +</ul> +<p>Configure Hadoop cluster to use Kerberos authentication.</p> +<ul class="simple"> +<li>Authentication type should be cluster level. All components must have the same authentication type: use Kerberos or not. In other words, Sqoop with Kerberos authentication could not communicate with other Hadoop components, such as HDFS, Yarn, Hive, etc., without Kerberos authentication, and vice versa.</li> +<li>How to set up a Hadoop cluster with Kerberos authentication is out of the scope of this document. Follow the related links like <a class="reference external" href="https://hadoop.apache.org/docs/r2.5.0/hadoop-project-dist/hadoop-common/SecureMode.html";>https://hadoop.apache.org/docs/r2.5.0/hadoop-project-dist/hadoop-common/SecureMode.html</a></li> +</ul> +<p>Create keytab and principal for Sqoop 2 via kadmin in command line.</p> +<div class="highlight-none"><div class="highlight"><pre>addprinc -randkey HTTP/<FQDN>@<REALM> +addprinc -randkey sqoop/<FQDN>@<REALM> +xst -k /home/kerberos/sqoop.keytab HTTP/<FQDN>@<REALM> +xst -k /home/kerberos/sqoop.keytab sqoop/<FQDN>@<REALM> +</pre></div> +</div> +<ul class="simple"> +<li>The <FQDN> should be replaced by the FQDN of the server, which could be found via âhostname -fâ in command line.</li> +<li>The <REALM> should be replaced by the realm name in krb5.conf file generated when installing the KDC server in the former step.</li> +<li>The principal HTTP/<FQDN>@<REALM> is used in communication between Sqoop client and Sqoop server. Since Sqoop server is an http server, so the HTTP principal is a must during SPNEGO process, and it is case sensitive.</li> +<li>Http request could be sent from other client like browser, wget or curl with SPNEGO support.</li> +<li>The principal sqoop/<FQDN>@<REALM> is used in communication between Sqoop server and Hdfs/Yarn as the credential of Sqoop server.</li> +</ul> +</div> +<div class="section" id="id1"> +<h3>Configuration<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3> +<p>Modify Sqoop configuration file, normally in <Sqoop Folder>/server/config/sqoop.properties.</p> +<div class="highlight-none"><div class="highlight"><pre>org.apache.sqoop.authentication.type=KERBEROS +org.apache.sqoop.authentication.handler=org.apache.sqoop.security.Authentication.KerberosAuthenticationHandler +org.apache.sqoop.authentication.kerberos.principal=sqoop/_HOST@<REALM> +org.apache.sqoop.authentication.kerberos.keytab=/home/kerberos/sqoop.keytab +org.apache.sqoop.authentication.kerberos.http.principal=HTTP/_HOST@<REALM> +org.apache.sqoop.authentication.kerberos.http.keytab=/home/kerberos/sqoop.keytab +org.apache.sqoop.authentication.kerberos.proxyuser=true +</pre></div> +</div> +<ul class="simple"> +<li>When _HOST is used as FQDN in principal, it will be replaced by the real FQDN. <a class="reference external" href="https://issues.apache.org/jira/browse/HADOOP-6632";>https://issues.apache.org/jira/browse/HADOOP-6632</a></li> +<li>If parameter proxyuser is set true, Sqoop server will use proxy user mode (sqoop delegate real client user) to run Yarn job. If false, Sqoop server will use sqoop user to run Yarn job.</li> +</ul> +</div> +<div class="section" id="id2"> +<h3>Run command<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h3> +<p>Set SQOOP2_HOST to FQDN.</p> +<div class="highlight-none"><div class="highlight"><pre>export SQOOP2_HOST=$(hostname -f). +</pre></div> +</div> +<ul class="simple"> +<li>The <FQDN> should be replaced by the FQDN of the server, which could be found via âhostname -fâ in command line.</li> +</ul> +<p>Start Sqoop server using sqoop user.</p> +<div class="highlight-none"><div class="highlight"><pre>sudo âu sqoop <Sqoop Folder>/bin/sqoop.sh server start +</pre></div> +</div> +<p>Run kinit to generate ticket cache.</p> +<div class="highlight-none"><div class="highlight"><pre>kinit HTTP/<FQDN>@<REALM> -kt /home/kerberos/sqoop.keytab +</pre></div> +</div> +<p>Start Sqoop client.</p> +<div class="highlight-none"><div class="highlight"><pre><Sqoop Folder>/bin/sqoop.sh client +</pre></div> +</div> +</div> +<div class="section" id="verify"> +<h3>Verify<a class="headerlink" href="#verify" title="Permalink to this headline">¶</a></h3> +<p>If the Sqoop server has started successfully with Kerberos authentication, the following line will be in <@LOGDIR>/sqoop.log:</p> +<div class="highlight-none"><div class="highlight"><pre>2014-12-04 15:02:58,038 INFO security.KerberosAuthenticationHandler [org.apache.sqoop.security.Authentication.KerberosAuthenticationHandler.secureLogin(KerberosAuthenticationHandler.java:84)] Using Kerberos authentication, principal [sqoop/_h...@hadoop.com] keytab [/home/kerberos/sqoop.keytab] +</pre></div> +</div> +<p>If the Sqoop client was able to communicate with the Sqoop server, the following will be in <Sqoop Folder>/server/log/catalina.out:</p> +<div class="highlight-none"><div class="highlight"><pre>Refreshing Kerberos configuration +Acquire TGT from Cache +Principal is HTTP/<FQDN>@HADOOP.COM +null credentials from Ticket Cache +principal is HTTP/<FQDN>@HADOOP.COM +Will use keytab +Commit Succeeded +</pre></div> +</div> +</div> +</div> +<div class="section" id="customized-authentication"> +<h2>Customized Authentication<a class="headerlink" href="#customized-authentication" title="Permalink to this headline">¶</a></h2> +<p>Users can create their own authentication modules. By performing the following steps:</p> +<ul class="simple"> +<li>Create customized authentication handler extends abstract class AuthenticationHandler.</li> +<li>Implement abstract function doInitialize and secureLogin in AuthenticationHandler.</li> +</ul> +<div class="highlight-none"><div class="highlight"><pre>public class MyAuthenticationHandler extends AuthenticationHandler { + + private static final Logger LOG = Logger.getLogger(MyAuthenticationHandler.class); + + public void doInitialize() { + securityEnabled = true; + } + + public void secureLogin() { + LOG.info("Using customized authentication."); + } +} +</pre></div> +</div> +<ul class="simple"> +<li>Modify configuration org.apache.sqoop.authentication.handler in <Sqoop Folder>/server/config/sqoop.properties and set it to the customized authentication handler class name.</li> +<li>Restart the Sqoop server.</li> +</ul> +</div> +</div> + + + </div> + <div class="bottomnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + + <div class="footer"> + © Copyright 2009-2013 The Apache Software Foundation. + </div> + </body> +</html> \ No newline at end of file
