Added: sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-FTP.html URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-FTP.html?rev=1754350&view=auto ============================================================================== --- sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-FTP.html (added) +++ sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-FTP.html Thu Jul 28 01:16:05 2016 @@ -0,0 +1,335 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>2.2.1. FTP Connector — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../../index.html"/> + <link rel="up" title="2.2. Connectors" href="../Connectors.html"/> + <link rel="next" title="2.2.2. Generic JDBC Connector" href="Connector-GenericJDBC.html"/> + <link rel="prev" title="2.2. Connectors" href="../Connectors.html"/> + + + <script src="../../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../../user.html">2. User Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="../CommandLineClient.html">2.1. Command Line Shell</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="../Connectors.html">2.2. Connectors</a><ul class="current"> +<li class="toctree-l3 current"><a class="current reference internal" href="">2.2.1. FTP Connector</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#usage">2.2.1.1. Usage</a></li> +<li class="toctree-l4"><a class="reference internal" href="#loader">2.2.1.2. Loader</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="Connector-GenericJDBC.html">2.2.2. Generic JDBC Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-HDFS.html">2.2.3. HDFS Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kafka.html">2.2.4. Kafka Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kite.html">2.2.5. Kite Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-SFTP.html">2.2.6. SFTP Connector</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="../Examples.html">2.3. Examples</a></li> +<li class="toctree-l2"><a class="reference internal" href="../Sqoop5MinutesDemo.html">2.4. Sqoop 5 Minutes Demo</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../../index.html">Docs</a> »</li> + + <li><a href="../../user.html">2. User Guide</a> »</li> + + <li><a href="../Connectors.html">2.2. Connectors</a> »</li> + + <li>2.2.1. FTP Connector</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../../_sources/user/connectors/Connector-FTP.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="ftp-connector"> +<h1><a class="toc-backref" href="#id2">2.2.1. FTP Connector</a><a class="headerlink" href="#ftp-connector" title="Permalink to this headline">¶</a></h1> +<p>The FTP connector supports moving data between an FTP server and other supported Sqoop2 connectors.</p> +<p>Currently only the TO direction is supported to write records to an FTP server. A FROM connector is pending (SQOOP-2127).</p> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#ftp-connector" id="id2">FTP Connector</a><ul> +<li><a class="reference internal" href="#usage" id="id3">Usage</a><ul> +<li><a class="reference internal" href="#link-configuration" id="id4"><strong>Link Configuration</strong></a></li> +<li><a class="reference internal" href="#to-job-configuration" id="id5"><strong>TO Job Configuration</strong></a></li> +</ul> +</li> +<li><a class="reference internal" href="#loader" id="id6">Loader</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="usage"> +<h2><a class="toc-backref" href="#id3">2.2.1.1. Usage</a><a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2> +<p>To use the FTP Connector, create a link for the connector and a job that uses the link.</p> +<div class="section" id="link-configuration"> +<h3><a class="toc-backref" href="#id4">2.2.1.1.1. <strong>Link Configuration</strong></a><a class="headerlink" href="#link-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the link configuration include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="7%" /> +<col width="52%" /> +<col width="20%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>FTP server hostname</td> +<td>String</td> +<td>Hostname for the FTP server. +<em>Required</em>.</td> +<td>ftp.example.com</td> +</tr> +<tr class="row-odd"><td>FTP server port</td> +<td>Integer</td> +<td>Port number for the FTP server. Defaults to 21. +<em>Optional</em>.</td> +<td>2100</td> +</tr> +<tr class="row-even"><td>Username</td> +<td>String</td> +<td>The username to provide when connecting to the FTP server. +<em>Required</em>.</td> +<td>sqoop</td> +</tr> +<tr class="row-odd"><td>Password</td> +<td>String</td> +<td>The password to provide when connecting to the FTP server. +<em>Required</em></td> +<td>sqoop</td> +</tr> +</tbody> +</table> +<div class="section" id="notes"> +<h4>2.2.1.1.1.1. <strong>Notes</strong><a class="headerlink" href="#notes" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The FTP connector will attempt to connect to the FTP server as part of the link validation process. If for some reason a connection can not be established, you’ll see a corresponding warning message.</li> +</ol> +</div> +</div> +<div class="section" id="to-job-configuration"> +<h3><a class="toc-backref" href="#id5">2.2.1.1.2. <strong>TO Job Configuration</strong></a><a class="headerlink" href="#to-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the TO direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="20%" /> +<col width="6%" /> +<col width="50%" /> +<col width="24%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Output directory</td> +<td>String</td> +<td>The location on the FTP server that the connector will write files to. +<em>Required</em></td> +<td>uploads</td> +</tr> +</tbody> +</table> +<div class="section" id="id1"> +<h4>2.2.1.1.2.1. <strong>Notes</strong><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The <em>output directory</em> value needs to be an existing directory on the FTP server.</li> +</ol> +</div> +</div> +</div> +<div class="section" id="loader"> +<h2><a class="toc-backref" href="#id6">2.2.1.2. Loader</a><a class="headerlink" href="#loader" title="Permalink to this headline">¶</a></h2> +<p>During the <em>loading</em> phase, the connector will create uniquely named files in the <em>output directory</em> for each partition of data received from the <strong>FROM</strong> connector.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Connector-GenericJDBC.html" class="btn btn-neutral float-right" title="2.2.2. Generic JDBC Connector" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="../Connectors.html" class="btn btn-neutral" title="2.2. Connectors" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file
Added: sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-GenericJDBC.html URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-GenericJDBC.html?rev=1754350&view=auto ============================================================================== --- sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-GenericJDBC.html (added) +++ sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-GenericJDBC.html Thu Jul 28 01:16:05 2016 @@ -0,0 +1,500 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>2.2.2. Generic JDBC Connector — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../../index.html"/> + <link rel="up" title="2.2. Connectors" href="../Connectors.html"/> + <link rel="next" title="2.2.3. HDFS Connector" href="Connector-HDFS.html"/> + <link rel="prev" title="2.2.1. FTP Connector" href="Connector-FTP.html"/> + + + <script src="../../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../../user.html">2. User Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="../CommandLineClient.html">2.1. Command Line Shell</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="../Connectors.html">2.2. Connectors</a><ul class="current"> +<li class="toctree-l3"><a class="reference internal" href="Connector-FTP.html">2.2.1. FTP Connector</a></li> +<li class="toctree-l3 current"><a class="current reference internal" href="">2.2.2. Generic JDBC Connector</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#usage">2.2.2.1. Usage</a></li> +<li class="toctree-l4"><a class="reference internal" href="#partitioner">2.2.2.2. Partitioner</a></li> +<li class="toctree-l4"><a class="reference internal" href="#extractor">2.2.2.3. Extractor</a></li> +<li class="toctree-l4"><a class="reference internal" href="#loader">2.2.2.4. Loader</a></li> +<li class="toctree-l4"><a class="reference internal" href="#destroyers">2.2.2.5. Destroyers</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="Connector-HDFS.html">2.2.3. HDFS Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kafka.html">2.2.4. Kafka Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kite.html">2.2.5. Kite Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-SFTP.html">2.2.6. SFTP Connector</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="../Examples.html">2.3. Examples</a></li> +<li class="toctree-l2"><a class="reference internal" href="../Sqoop5MinutesDemo.html">2.4. Sqoop 5 Minutes Demo</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../../index.html">Docs</a> »</li> + + <li><a href="../../user.html">2. User Guide</a> »</li> + + <li><a href="../Connectors.html">2.2. Connectors</a> »</li> + + <li>2.2.2. Generic JDBC Connector</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../../_sources/user/connectors/Connector-GenericJDBC.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="generic-jdbc-connector"> +<h1><a class="toc-backref" href="#id2">2.2.2. Generic JDBC Connector</a><a class="headerlink" href="#generic-jdbc-connector" title="Permalink to this headline">¶</a></h1> +<p>The Generic JDBC Connector can connect to any data source that adheres to the <strong>JDBC 4</strong> specification.</p> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#generic-jdbc-connector" id="id2">Generic JDBC Connector</a><ul> +<li><a class="reference internal" href="#usage" id="id3">Usage</a><ul> +<li><a class="reference internal" href="#link-configuration" id="id4"><strong>Link Configuration</strong></a></li> +<li><a class="reference internal" href="#from-job-configuration" id="id5"><strong>FROM Job Configuration</strong></a></li> +<li><a class="reference internal" href="#to-job-configuration" id="id6"><strong>TO Job Configuration</strong></a></li> +</ul> +</li> +<li><a class="reference internal" href="#partitioner" id="id7">Partitioner</a></li> +<li><a class="reference internal" href="#extractor" id="id8">Extractor</a></li> +<li><a class="reference internal" href="#loader" id="id9">Loader</a></li> +<li><a class="reference internal" href="#destroyers" id="id10">Destroyers</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="usage"> +<h2><a class="toc-backref" href="#id3">2.2.2.1. Usage</a><a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2> +<p>To use the Generic JDBC Connector, create a link for the connector and a job that uses the link.</p> +<div class="section" id="link-configuration"> +<h3><a class="toc-backref" href="#id4">2.2.2.1.1. <strong>Link Configuration</strong></a><a class="headerlink" href="#link-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the link configuration include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="19%" /> +<col width="6%" /> +<col width="47%" /> +<col width="28%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>JDBC Driver Class</td> +<td>String</td> +<td>The full class name of the JDBC driver. +<em>Required</em> and accessible by the Sqoop server.</td> +<td>com.mysql.jdbc.Driver</td> +</tr> +<tr class="row-odd"><td>JDBC Connection String</td> +<td>String</td> +<td>The JDBC connection string to use when connecting to the data source. +<em>Required</em>. Connectivity upon creation is optional.</td> +<td><a class="reference external" href="jdbc:mysql://localhost/test">jdbc:mysql://localhost/test</a></td> +</tr> +<tr class="row-even"><td>Username</td> +<td>String</td> +<td>The username to provide when connecting to the data source. +<em>Optional</em>. Connectivity upon creation is optional.</td> +<td>sqoop</td> +</tr> +<tr class="row-odd"><td>Password</td> +<td>String</td> +<td>The password to provide when connecting to the data source. +<em>Optional</em>. Connectivity upon creation is optional.</td> +<td>sqoop</td> +</tr> +<tr class="row-even"><td>JDBC Connection Properties</td> +<td>Map</td> +<td>A map of JDBC connection properties to pass to the JDBC driver +<em>Optional</em>.</td> +<td>profileSQL=true&useFastDateParsing=false</td> +</tr> +</tbody> +</table> +</div> +<div class="section" id="from-job-configuration"> +<h3><a class="toc-backref" href="#id5">2.2.2.1.2. <strong>FROM Job Configuration</strong></a><a class="headerlink" href="#from-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the FROM direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="19%" /> +<col width="6%" /> +<col width="47%" /> +<col width="29%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Schema name</td> +<td>String</td> +<td>The schema name the table is part of. +<em>Optional</em></td> +<td>sqoop</td> +</tr> +<tr class="row-odd"><td>Table name</td> +<td>String</td> +<td>The table name to import data from. +<em>Optional</em>. See note below.</td> +<td>test</td> +</tr> +<tr class="row-even"><td>Table SQL statement</td> +<td>String</td> +<td>The SQL statement used to perform a <strong>free form query</strong>. +<em>Optional</em>. See notes below.</td> +<td><tt class="docutils literal"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">test</span> <span class="pre">${CONDITIONS}</span></tt></td> +</tr> +<tr class="row-odd"><td>Table column names</td> +<td>String</td> +<td>Columns to extract from the JDBC data source. +<em>Optional</em> Comma separated list of columns.</td> +<td>col1,col2</td> +</tr> +<tr class="row-even"><td>Partition column name</td> +<td>Map</td> +<td>The column name used to partition the data transfer process. +<em>Optional</em>. Defaults to table’s first column of primary key.</td> +<td>col1</td> +</tr> +<tr class="row-odd"><td>Null value allowed for +the partition column</td> +<td>Boolean</td> +<td>True or false depending on whether NULL values are allowed in data +of the Partition column. <em>Optional</em>.</td> +<td>true</td> +</tr> +<tr class="row-even"><td>Boundary query</td> +<td>String</td> +<td>The query used to define an upper and lower boundary when partitioning. +<em>Optional</em>.</td> +<td> </td> +</tr> +</tbody> +</table> +<div class="section" id="notes"> +<h4>2.2.2.1.2.1. <strong>Notes</strong><a class="headerlink" href="#notes" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li><em>Table name</em> and <em>Table SQL statement</em> are mutually exclusive. If <em>Table name</em> is provided, the <em>Table SQL statement</em> should not be provided. If <em>Table SQL statement</em> is provided then <em>Table name</em> should not be provided.</li> +<li><em>Table column names</em> should be provided only if <em>Table name</em> is provided.</li> +<li>If there are columns with similar names, column aliases are required. For example: <tt class="docutils literal"><span class="pre">SELECT</span> <span class="pre">table1.id</span> <span class="pre">as</span> <span class="pre">"i",</span> <span class="pre">table2.id</span> <span class="pre">as</span> <span class="pre">"j"</span> <span class="pre">FROM</span> <span class="pre">table1</span> <span class="pre">INNER</span> <span class="pre">JOIN</span> <span class="pre">table2</span> <span class="pre">ON</span> <span class="pre">table1.id</span> <span class="pre">=</span> <span class="pre">table2.id</span></tt>.</li> +</ol> +</div> +</div> +<div class="section" id="to-job-configuration"> +<h3><a class="toc-backref" href="#id6">2.2.2.1.3. <strong>TO Job Configuration</strong></a><a class="headerlink" href="#to-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the TO direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="18%" /> +<col width="6%" /> +<col width="46%" /> +<col width="31%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Schema name</td> +<td>String</td> +<td>The schema name the table is part of. +<em>Optional</em></td> +<td>sqoop</td> +</tr> +<tr class="row-odd"><td>Table name</td> +<td>String</td> +<td>The table name to import data from. +<em>Optional</em>. See note below.</td> +<td>test</td> +</tr> +<tr class="row-even"><td>Table SQL statement</td> +<td>String</td> +<td>The SQL statement used to perform a <strong>free form query</strong>. +<em>Optional</em>. See note below.</td> +<td><tt class="docutils literal"><span class="pre">INSERT</span> <span class="pre">INTO</span> <span class="pre">test</span> <span class="pre">(col1,</span> <span class="pre">col2)</span> <span class="pre">VALUES</span> <span class="pre">(?,</span> <span class="pre">?)</span></tt></td> +</tr> +<tr class="row-odd"><td>Table column names</td> +<td>String</td> +<td>Columns to insert into the JDBC data source. +<em>Optional</em> Comma separated list of columns.</td> +<td>col1,col2</td> +</tr> +<tr class="row-even"><td>Stage table name</td> +<td>String</td> +<td>The name of the table used as a <em>staging table</em>. +<em>Optional</em>.</td> +<td>staging</td> +</tr> +<tr class="row-odd"><td>Should clear stage table</td> +<td>Boolean</td> +<td>True or false depending on whether the staging table should be cleared +after the data transfer has finished. <em>Optional</em>.</td> +<td>true</td> +</tr> +</tbody> +</table> +<div class="section" id="id1"> +<h4>2.2.2.1.3.1. <strong>Notes</strong><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li><em>Table name</em> and <em>Table SQL statement</em> are mutually exclusive. If <em>Table name</em> is provided, the <em>Table SQL statement</em> should not be provided. If <em>Table SQL statement</em> is provided then <em>Table name</em> should not be provided.</li> +<li><em>Table column names</em> should be provided only if <em>Table name</em> is provided.</li> +</ol> +</div> +</div> +</div> +<div class="section" id="partitioner"> +<h2><a class="toc-backref" href="#id7">2.2.2.2. Partitioner</a><a class="headerlink" href="#partitioner" title="Permalink to this headline">¶</a></h2> +<p>The Generic JDBC Connector partitioner generates conditions to be used by the extractor. +It varies in how it partitions data transfer based on the partition column data type. +Though, each strategy roughly takes on the following form:</p> +<div class="highlight-none"><div class="highlight"><pre>(upper boundary - lower boundary) / (max partitions) +</pre></div> +</div> +<p>By default, the <em>primary key</em> will be used to partition the data unless otherwise specified.</p> +<p>The following data types are currently supported:</p> +<ol class="arabic simple"> +<li>TINYINT</li> +<li>SMALLINT</li> +<li>INTEGER</li> +<li>BIGINT</li> +<li>REAL</li> +<li>FLOAT</li> +<li>DOUBLE</li> +<li>NUMERIC</li> +<li>DECIMAL</li> +<li>BIT</li> +<li>BOOLEAN</li> +<li>DATE</li> +<li>TIME</li> +<li>TIMESTAMP</li> +<li>CHAR</li> +<li>VARCHAR</li> +<li>LONGVARCHAR</li> +</ol> +</div> +<div class="section" id="extractor"> +<h2><a class="toc-backref" href="#id8">2.2.2.3. Extractor</a><a class="headerlink" href="#extractor" title="Permalink to this headline">¶</a></h2> +<p>During the <em>extraction</em> phase, the JDBC data source is queried using SQL. This SQL will vary based on your configuration.</p> +<ul class="simple"> +<li>If <em>Table name</em> is provided, then the SQL statement generated will take on the form <tt class="docutils literal"><span class="pre">SELECT</span> <span class="pre">*</span> <span class="pre">FROM</span> <span class="pre"><table</span> <span class="pre">name></span></tt>.</li> +<li>If <em>Table name</em> and <em>Columns</em> are provided, then the SQL statement generated will take on the form <tt class="docutils literal"><span class="pre">SELECT</span> <span class="pre"><columns></span> <span class="pre">FROM</span> <span class="pre"><table</span> <span class="pre">name></span></tt>.</li> +<li>If <em>Table SQL statement</em> is provided, then the provided SQL statement will be used.</li> +</ul> +<p>The conditions generated by the <em>partitioner</em> are appended to the end of the SQL query to query a section of data.</p> +<p>The Generic JDBC connector extracts CSV data usable by the <em>CSV Intermediate Data Format</em>.</p> +</div> +<div class="section" id="loader"> +<h2><a class="toc-backref" href="#id9">2.2.2.4. Loader</a><a class="headerlink" href="#loader" title="Permalink to this headline">¶</a></h2> +<p>During the <em>loading</em> phase, the JDBC data source is queried using SQL. This SQL will vary based on your configuration.</p> +<ul class="simple"> +<li>If <em>Table name</em> is provided, then the SQL statement generated will take on the form <tt class="docutils literal"><span class="pre">INSERT</span> <span class="pre">INTO</span> <span class="pre"><table</span> <span class="pre">name></span> <span class="pre">(col1,</span> <span class="pre">col2,</span> <span class="pre">...)</span> <span class="pre">VALUES</span> <span class="pre">(?,?,..)</span></tt>.</li> +<li>If <em>Table name</em> and <em>Columns</em> are provided, then the SQL statement generated will take on the form <tt class="docutils literal"><span class="pre">INSERT</span> <span class="pre">INTO</span> <span class="pre"><table</span> <span class="pre">name></span> <span class="pre">(<columns>)</span> <span class="pre">VALUES</span> <span class="pre">(?,?,..)</span></tt>.</li> +<li>If <em>Table SQL statement</em> is provided, then the provided SQL statement will be used.</li> +</ul> +<p>This connector expects to receive CSV data consumable by the <em>CSV Intermediate Data Format</em>.</p> +</div> +<div class="section" id="destroyers"> +<h2><a class="toc-backref" href="#id10">2.2.2.5. Destroyers</a><a class="headerlink" href="#destroyers" title="Permalink to this headline">¶</a></h2> +<p>The Generic JDBC Connector performs two operations in the destroyer in the TO direction:</p> +<ol class="arabic simple"> +<li>Copy the contents of the staging table to the desired table.</li> +<li>Clear the staging table.</li> +</ol> +<p>No operations are performed in the FROM direction.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Connector-HDFS.html" class="btn btn-neutral float-right" title="2.2.3. HDFS Connector" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Connector-FTP.html" class="btn btn-neutral" title="2.2.1. FTP Connector" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-HDFS.html URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-HDFS.html?rev=1754350&view=auto ============================================================================== --- sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-HDFS.html (added) +++ sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-HDFS.html Thu Jul 28 01:16:05 2016 @@ -0,0 +1,461 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>2.2.3. HDFS Connector — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../../index.html"/> + <link rel="up" title="2.2. Connectors" href="../Connectors.html"/> + <link rel="next" title="2.2.4. Kafka Connector" href="Connector-Kafka.html"/> + <link rel="prev" title="2.2.2. Generic JDBC Connector" href="Connector-GenericJDBC.html"/> + + + <script src="../../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../../user.html">2. User Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="../CommandLineClient.html">2.1. Command Line Shell</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="../Connectors.html">2.2. Connectors</a><ul class="current"> +<li class="toctree-l3"><a class="reference internal" href="Connector-FTP.html">2.2.1. FTP Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-GenericJDBC.html">2.2.2. Generic JDBC Connector</a></li> +<li class="toctree-l3 current"><a class="current reference internal" href="">2.2.3. HDFS Connector</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#usage">2.2.3.1. Usage</a></li> +<li class="toctree-l4"><a class="reference internal" href="#partitioner">2.2.3.2. Partitioner</a></li> +<li class="toctree-l4"><a class="reference internal" href="#extractor">2.2.3.3. Extractor</a></li> +<li class="toctree-l4"><a class="reference internal" href="#loader">2.2.3.4. Loader</a></li> +<li class="toctree-l4"><a class="reference internal" href="#destroyers">2.2.3.5. Destroyers</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kafka.html">2.2.4. Kafka Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kite.html">2.2.5. Kite Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-SFTP.html">2.2.6. SFTP Connector</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="../Examples.html">2.3. Examples</a></li> +<li class="toctree-l2"><a class="reference internal" href="../Sqoop5MinutesDemo.html">2.4. Sqoop 5 Minutes Demo</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../../index.html">Docs</a> »</li> + + <li><a href="../../user.html">2. User Guide</a> »</li> + + <li><a href="../Connectors.html">2.2. Connectors</a> »</li> + + <li>2.2.3. HDFS Connector</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../../_sources/user/connectors/Connector-HDFS.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="hdfs-connector"> +<h1><a class="toc-backref" href="#id3">2.2.3. HDFS Connector</a><a class="headerlink" href="#hdfs-connector" title="Permalink to this headline">¶</a></h1> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#hdfs-connector" id="id3">HDFS Connector</a><ul> +<li><a class="reference internal" href="#usage" id="id4">Usage</a><ul> +<li><a class="reference internal" href="#link-configuration" id="id5"><strong>Link Configuration</strong></a></li> +<li><a class="reference internal" href="#from-job-configuration" id="id6"><strong>FROM Job Configuration</strong></a></li> +<li><a class="reference internal" href="#to-job-configuration" id="id7"><strong>TO Job Configuration</strong></a></li> +</ul> +</li> +<li><a class="reference internal" href="#partitioner" id="id8">Partitioner</a></li> +<li><a class="reference internal" href="#extractor" id="id9">Extractor</a></li> +<li><a class="reference internal" href="#loader" id="id10">Loader</a></li> +<li><a class="reference internal" href="#destroyers" id="id11">Destroyers</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="usage"> +<h2><a class="toc-backref" href="#id4">2.2.3.1. Usage</a><a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2> +<p>To use the HDFS Connector, create a link for the connector and a job that uses the link.</p> +<div class="section" id="link-configuration"> +<h3><a class="toc-backref" href="#id5">2.2.3.1.1. <strong>Link Configuration</strong></a><a class="headerlink" href="#link-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the link configuration include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="7%" /> +<col width="52%" /> +<col width="20%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>URI</td> +<td>String</td> +<td>The URI of the HDFS File System. +<em>Optional</em>. See note below.</td> +<td>hdfs://example.com:8020/</td> +</tr> +<tr class="row-odd"><td>Configuration directory</td> +<td>String</td> +<td>Path to the clusters configuration directory. +<em>Optional</em>.</td> +<td>/etc/conf/hadoop</td> +</tr> +</tbody> +</table> +<div class="section" id="notes"> +<h4>2.2.3.1.1.1. <strong>Notes</strong><a class="headerlink" href="#notes" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The specified URI will override the declared URI in your configuration.</li> +</ol> +</div> +</div> +<div class="section" id="from-job-configuration"> +<h3><a class="toc-backref" href="#id6">2.2.3.1.2. <strong>FROM Job Configuration</strong></a><a class="headerlink" href="#from-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the FROM direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="22%" /> +<col width="7%" /> +<col width="57%" /> +<col width="14%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Input directory</td> +<td>String</td> +<td>The location in HDFS that the connector should look for files in. +<em>Required</em>. See note below.</td> +<td>/tmp/sqoop2/hdfs</td> +</tr> +<tr class="row-odd"><td>Null value</td> +<td>String</td> +<td>The value of NULL in the contents of each file extracted. +<em>Optional</em>. See note below.</td> +<td>N</td> +</tr> +<tr class="row-even"><td>Override null value</td> +<td>Boolean</td> +<td>Tells the connector to replace the specified NULL value. +<em>Optional</em>. See note below.</td> +<td>true</td> +</tr> +</tbody> +</table> +<div class="section" id="id1"> +<h4>2.2.3.1.2.1. <strong>Notes</strong><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>All files in <em>Input directory</em> will be extracted.</li> +<li><em>Null value</em> and <em>override null value</em> should be used in conjunction. If <em>override null value</em> is not set to true, then <em>null value</em> will not be used when extracting data.</li> +</ol> +</div> +</div> +<div class="section" id="to-job-configuration"> +<h3><a class="toc-backref" href="#id7">2.2.3.1.3. <strong>TO Job Configuration</strong></a><a class="headerlink" href="#to-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the TO direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="20%" /> +<col width="6%" /> +<col width="50%" /> +<col width="24%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Output directory</td> +<td>String</td> +<td>The location in HDFS that the connector will load files to. +<em>Optional</em></td> +<td>/tmp/sqoop2/hdfs</td> +</tr> +<tr class="row-odd"><td>Output format</td> +<td>Enum</td> +<td>The format to output data to. +<em>Optional</em>. See note below.</td> +<td>CSV</td> +</tr> +<tr class="row-even"><td>Compression</td> +<td>Enum</td> +<td>Compression class. +<em>Optional</em>. See note below.</td> +<td>GZIP</td> +</tr> +<tr class="row-odd"><td>Custom compression</td> +<td>String</td> +<td>Custom compression class. +<em>Optional</em> Comma separated list of columns.</td> +<td>org.apache.sqoop.SqoopCompression</td> +</tr> +<tr class="row-even"><td>Null value</td> +<td>String</td> +<td>The value of NULL in the contents of each file loaded. +<em>Optional</em>. See note below.</td> +<td>N</td> +</tr> +<tr class="row-odd"><td>Override null value</td> +<td>Boolean</td> +<td>Tells the connector to replace the specified NULL value. +<em>Optional</em>. See note below.</td> +<td>true</td> +</tr> +<tr class="row-even"><td>Append mode</td> +<td>Boolean</td> +<td>Append to an existing output directory. +<em>Optional</em>.</td> +<td>true</td> +</tr> +</tbody> +</table> +<div class="section" id="id2"> +<h4>2.2.3.1.3.1. <strong>Notes</strong><a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li><em>Output format</em> only supports CSV at the moment.</li> +<li><em>Compression</em> supports all Hadoop compression classes.</li> +<li><em>Null value</em> and <em>override null value</em> should be used in conjunction. If <em>override null value</em> is not set to true, then <em>null value</em> will not be used when loading data.</li> +</ol> +</div> +</div> +</div> +<div class="section" id="partitioner"> +<h2><a class="toc-backref" href="#id8">2.2.3.2. Partitioner</a><a class="headerlink" href="#partitioner" title="Permalink to this headline">¶</a></h2> +<p>The HDFS Connector partitioner partitions based on total blocks in all files in the specified input directory. +Blocks will try to be placed in splits based on the <em>node</em> and <em>rack</em> they reside in.</p> +</div> +<div class="section" id="extractor"> +<h2><a class="toc-backref" href="#id9">2.2.3.3. Extractor</a><a class="headerlink" href="#extractor" title="Permalink to this headline">¶</a></h2> +<p>During the <em>extraction</em> phase, the FileSystem API is used to query files from HDFS. The HDFS cluster used is the one defined by:</p> +<ol class="arabic simple"> +<li>The HDFS URI in the link configuration</li> +<li>The Hadoop configuration in the link configuration</li> +<li>The Hadoop configuration used by the execution framework</li> +</ol> +<p>The format of the data must be CSV. The NULL value in the CSV can be chosen via <em>null value</em>. For example:</p> +<div class="highlight-none"><div class="highlight"><pre>1,\N +2,null +3,NULL +</pre></div> +</div> +<p>In the above example, if <em>null value</em> is set to N, then only the first row’s NULL value will be inferred.</p> +</div> +<div class="section" id="loader"> +<h2><a class="toc-backref" href="#id10">2.2.3.4. Loader</a><a class="headerlink" href="#loader" title="Permalink to this headline">¶</a></h2> +<p>During the <em>loading</em> phase, HDFS is written to via the FileSystem API. The number of files created is equal to the number of loads that run. The format of the data currently can only be CSV. The NULL value in the CSV can be chosen via <em>null value</em>. For example:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="67%" /> +<col width="33%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Id</th> +<th class="head">Value</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>1</td> +<td>NULL</td> +</tr> +<tr class="row-odd"><td>2</td> +<td>value</td> +</tr> +</tbody> +</table> +<p>If <em>null value</em> is set to N, then here’s how the data will look like in HDFS:</p> +<div class="highlight-none"><div class="highlight"><pre>1,\N +2,value +</pre></div> +</div> +</div> +<div class="section" id="destroyers"> +<h2><a class="toc-backref" href="#id11">2.2.3.5. Destroyers</a><a class="headerlink" href="#destroyers" title="Permalink to this headline">¶</a></h2> +<p>The HDFS TO destroyer moves all created files to the proper output directory.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Connector-Kafka.html" class="btn btn-neutral float-right" title="2.2.4. Kafka Connector" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Connector-GenericJDBC.html" class="btn btn-neutral" title="2.2.2. Generic JDBC Connector" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kafka.html URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kafka.html?rev=1754350&view=auto ============================================================================== --- sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kafka.html (added) +++ sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kafka.html Thu Jul 28 01:16:05 2016 @@ -0,0 +1,310 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>2.2.4. Kafka Connector — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../../index.html"/> + <link rel="up" title="2.2. Connectors" href="../Connectors.html"/> + <link rel="next" title="2.2.5. Kite Connector" href="Connector-Kite.html"/> + <link rel="prev" title="2.2.3. HDFS Connector" href="Connector-HDFS.html"/> + + + <script src="../../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../../user.html">2. User Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="../CommandLineClient.html">2.1. Command Line Shell</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="../Connectors.html">2.2. Connectors</a><ul class="current"> +<li class="toctree-l3"><a class="reference internal" href="Connector-FTP.html">2.2.1. FTP Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-GenericJDBC.html">2.2.2. Generic JDBC Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-HDFS.html">2.2.3. HDFS Connector</a></li> +<li class="toctree-l3 current"><a class="current reference internal" href="">2.2.4. Kafka Connector</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#usage">2.2.4.1. Usage</a></li> +<li class="toctree-l4"><a class="reference internal" href="#loader">2.2.4.2. Loader</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kite.html">2.2.5. Kite Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-SFTP.html">2.2.6. SFTP Connector</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="../Examples.html">2.3. Examples</a></li> +<li class="toctree-l2"><a class="reference internal" href="../Sqoop5MinutesDemo.html">2.4. Sqoop 5 Minutes Demo</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../../index.html">Docs</a> »</li> + + <li><a href="../../user.html">2. User Guide</a> »</li> + + <li><a href="../Connectors.html">2.2. Connectors</a> »</li> + + <li>2.2.4. Kafka Connector</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../../_sources/user/connectors/Connector-Kafka.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="kafka-connector"> +<h1><a class="toc-backref" href="#id1">2.2.4. Kafka Connector</a><a class="headerlink" href="#kafka-connector" title="Permalink to this headline">¶</a></h1> +<p>Currently, only the TO direction is supported.</p> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#kafka-connector" id="id1">Kafka Connector</a><ul> +<li><a class="reference internal" href="#usage" id="id2">Usage</a><ul> +<li><a class="reference internal" href="#link-configuration" id="id3"><strong>Link Configuration</strong></a></li> +<li><a class="reference internal" href="#to-job-configuration" id="id4"><strong>TO Job Configuration</strong></a></li> +</ul> +</li> +<li><a class="reference internal" href="#loader" id="id5">Loader</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="usage"> +<h2><a class="toc-backref" href="#id2">2.2.4.1. Usage</a><a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2> +<p>To use the Kafka Connector, create a link for the connector and a job that uses the link.</p> +<div class="section" id="link-configuration"> +<h3><a class="toc-backref" href="#id3">2.2.4.1.1. <strong>Link Configuration</strong></a><a class="headerlink" href="#link-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the link configuration include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="17%" /> +<col width="7%" /> +<col width="46%" /> +<col width="29%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Broker list</td> +<td>String</td> +<td>Comma separated list of kafka brokers. +<em>Required</em>.</td> +<td>example.com:10000,example.com:11000</td> +</tr> +<tr class="row-odd"><td>Zookeeper connection</td> +<td>String</td> +<td>Comma separated list of zookeeper servers in your quorum. +<em>Required</em>.</td> +<td>/etc/conf/hadoop</td> +</tr> +</tbody> +</table> +</div> +<div class="section" id="to-job-configuration"> +<h3><a class="toc-backref" href="#id4">2.2.4.1.2. <strong>TO Job Configuration</strong></a><a class="headerlink" href="#to-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the FROM direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="12%" /> +<col width="15%" /> +<col width="56%" /> +<col width="17%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>topic</td> +<td>String</td> +<td>The Kafka topic to transfer to. +<em>Required</em>.</td> +<td>my topic</td> +</tr> +</tbody> +</table> +</div> +</div> +<div class="section" id="loader"> +<h2><a class="toc-backref" href="#id5">2.2.4.2. Loader</a><a class="headerlink" href="#loader" title="Permalink to this headline">¶</a></h2> +<p>During the <em>loading</em> phase, Kafka is written to directly from each loader. The order in which data is loaded into Kafka is not guaranteed.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Connector-Kite.html" class="btn btn-neutral float-right" title="2.2.5. Kite Connector" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Connector-HDFS.html" class="btn btn-neutral" title="2.2.3. HDFS Connector" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kite.html URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kite.html?rev=1754350&view=auto ============================================================================== --- sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kite.html (added) +++ sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-Kite.html Thu Jul 28 01:16:05 2016 @@ -0,0 +1,376 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>2.2.5. Kite Connector — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../../index.html"/> + <link rel="up" title="2.2. Connectors" href="../Connectors.html"/> + <link rel="next" title="2.2.6. SFTP Connector" href="Connector-SFTP.html"/> + <link rel="prev" title="2.2.4. Kafka Connector" href="Connector-Kafka.html"/> + + + <script src="../../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../../user.html">2. User Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="../CommandLineClient.html">2.1. Command Line Shell</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="../Connectors.html">2.2. Connectors</a><ul class="current"> +<li class="toctree-l3"><a class="reference internal" href="Connector-FTP.html">2.2.1. FTP Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-GenericJDBC.html">2.2.2. Generic JDBC Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-HDFS.html">2.2.3. HDFS Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kafka.html">2.2.4. Kafka Connector</a></li> +<li class="toctree-l3 current"><a class="current reference internal" href="">2.2.5. Kite Connector</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#usage">2.2.5.1. Usage</a></li> +<li class="toctree-l4"><a class="reference internal" href="#partitioner">2.2.5.2. Partitioner</a></li> +<li class="toctree-l4"><a class="reference internal" href="#extractor">2.2.5.3. Extractor</a></li> +<li class="toctree-l4"><a class="reference internal" href="#loader">2.2.5.4. Loader</a></li> +<li class="toctree-l4"><a class="reference internal" href="#destroyers">2.2.5.5. Destroyers</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="Connector-SFTP.html">2.2.6. SFTP Connector</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="../Examples.html">2.3. Examples</a></li> +<li class="toctree-l2"><a class="reference internal" href="../Sqoop5MinutesDemo.html">2.4. Sqoop 5 Minutes Demo</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../../index.html">Docs</a> »</li> + + <li><a href="../../user.html">2. User Guide</a> »</li> + + <li><a href="../Connectors.html">2.2. Connectors</a> »</li> + + <li>2.2.5. Kite Connector</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../../_sources/user/connectors/Connector-Kite.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="kite-connector"> +<h1><a class="toc-backref" href="#id3">2.2.5. Kite Connector</a><a class="headerlink" href="#kite-connector" title="Permalink to this headline">¶</a></h1> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#kite-connector" id="id3">Kite Connector</a><ul> +<li><a class="reference internal" href="#usage" id="id4">Usage</a><ul> +<li><a class="reference internal" href="#link-configuration" id="id5"><strong>Link Configuration</strong></a></li> +<li><a class="reference internal" href="#from-job-configuration" id="id6"><strong>FROM Job Configuration</strong></a></li> +<li><a class="reference internal" href="#to-job-configuration" id="id7"><strong>TO Job Configuration</strong></a></li> +</ul> +</li> +<li><a class="reference internal" href="#partitioner" id="id8">Partitioner</a></li> +<li><a class="reference internal" href="#extractor" id="id9">Extractor</a></li> +<li><a class="reference internal" href="#loader" id="id10">Loader</a></li> +<li><a class="reference internal" href="#destroyers" id="id11">Destroyers</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="usage"> +<h2><a class="toc-backref" href="#id4">2.2.5.1. Usage</a><a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2> +<p>To use the Kite Connector, create a link for the connector and a job that uses the link. For more information on Kite, checkout the kite documentation: <a class="reference external" href="http://kitesdk.org/docs/1.0.0/Kite-SDK-Guide.html";>http://kitesdk.org/docs/1.0.0/Kite-SDK-Guide.html</a>.</p> +<div class="section" id="link-configuration"> +<h3><a class="toc-backref" href="#id5">2.2.5.1.1. <strong>Link Configuration</strong></a><a class="headerlink" href="#link-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the link configuration include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="7%" /> +<col width="52%" /> +<col width="20%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>authority</td> +<td>String</td> +<td>The authority of the kite dataset. +<em>Optional</em>. See note below.</td> +<td>hdfs://example.com:8020/</td> +</tr> +</tbody> +</table> +<div class="section" id="notes"> +<h4>2.2.5.1.1.1. <strong>Notes</strong><a class="headerlink" href="#notes" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The authority is useful for specifying Hive metastore or HDFS URI.</li> +</ol> +</div> +</div> +<div class="section" id="from-job-configuration"> +<h3><a class="toc-backref" href="#id6">2.2.5.1.2. <strong>FROM Job Configuration</strong></a><a class="headerlink" href="#from-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the FROM direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="7%" /> +<col width="52%" /> +<col width="20%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>URI</td> +<td>String</td> +<td>The Kite dataset URI to use. +<em>Required</em>. See notes below.</td> +<td>dataset:hdfs:/tmp/ns/ds</td> +</tr> +</tbody> +</table> +<div class="section" id="id1"> +<h4>2.2.5.1.2.1. <strong>Notes</strong><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The URI and the authority from the link configuration will be merged to create a complete dataset URI internally. If the given dataset URI contains authority, the authority from the link configuration will be ignored.</li> +<li>Only <em>hdfs</em> and <em>hive</em> are supported currently.</li> +</ol> +</div> +</div> +<div class="section" id="to-job-configuration"> +<h3><a class="toc-backref" href="#id7">2.2.5.1.3. <strong>TO Job Configuration</strong></a><a class="headerlink" href="#to-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the TO direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="7%" /> +<col width="52%" /> +<col width="20%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>URI</td> +<td>String</td> +<td>The Kite dataset URI to use. +<em>Required</em>. See note below.</td> +<td>dataset:hdfs:/tmp/ns/ds</td> +</tr> +<tr class="row-odd"><td>File format</td> +<td>Enum</td> +<td>The format of the data the kite dataset should write out. +<em>Optional</em>. See note below.</td> +<td>PARQUET</td> +</tr> +</tbody> +</table> +<div class="section" id="id2"> +<h4>2.2.5.1.3.1. <strong>Notes</strong><a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The URI and the authority from the link configuration will be merged to create a complete dataset URI internally. If the given dataset URI contains authority, the authority from the link configuration will be ignored.</li> +<li>Only <em>hdfs</em> and <em>hive</em> are supported currently.</li> +</ol> +</div> +</div> +</div> +<div class="section" id="partitioner"> +<h2><a class="toc-backref" href="#id8">2.2.5.2. Partitioner</a><a class="headerlink" href="#partitioner" title="Permalink to this headline">¶</a></h2> +<p>The kite connector only creates one partition currently.</p> +</div> +<div class="section" id="extractor"> +<h2><a class="toc-backref" href="#id9">2.2.5.3. Extractor</a><a class="headerlink" href="#extractor" title="Permalink to this headline">¶</a></h2> +<p>During the <em>extraction</em> phase, Kite is used to query a dataset. Since there is only one dataset to query, only a single reader is created to read the dataset.</p> +<p><strong>NOTE</strong>: The avro schema kite generates will be slightly different than the original schema. This is because avro identifiers have strict naming requirements.</p> +</div> +<div class="section" id="loader"> +<h2><a class="toc-backref" href="#id10">2.2.5.4. Loader</a><a class="headerlink" href="#loader" title="Permalink to this headline">¶</a></h2> +<p>During the <em>loading</em> phase, Kite is used to write several temporary datasets. The number of temporary datasets is equivalent to the number of <em>loaders</em> that are being used.</p> +</div> +<div class="section" id="destroyers"> +<h2><a class="toc-backref" href="#id11">2.2.5.5. Destroyers</a><a class="headerlink" href="#destroyers" title="Permalink to this headline">¶</a></h2> +<p>The Kite connector TO destroyer merges all the temporary datasets into a single dataset.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Connector-SFTP.html" class="btn btn-neutral float-right" title="2.2.6. SFTP Connector" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Connector-Kafka.html" class="btn btn-neutral" title="2.2.4. Kafka Connector" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-SFTP.html URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-SFTP.html?rev=1754350&view=auto ============================================================================== --- sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-SFTP.html (added) +++ sqoop/site/trunk/content/resources/docs/1.99.7/user/connectors/Connector-SFTP.html Thu Jul 28 01:16:05 2016 @@ -0,0 +1,344 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>2.2.6. SFTP Connector — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../../index.html"/> + <link rel="up" title="2.2. Connectors" href="../Connectors.html"/> + <link rel="next" title="2.3. Examples" href="../Examples.html"/> + <link rel="prev" title="2.2.5. Kite Connector" href="Connector-Kite.html"/> + + + <script src="../../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../../user.html">2. User Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="../CommandLineClient.html">2.1. Command Line Shell</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="../Connectors.html">2.2. Connectors</a><ul class="current"> +<li class="toctree-l3"><a class="reference internal" href="Connector-FTP.html">2.2.1. FTP Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-GenericJDBC.html">2.2.2. Generic JDBC Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-HDFS.html">2.2.3. HDFS Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kafka.html">2.2.4. Kafka Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="Connector-Kite.html">2.2.5. Kite Connector</a></li> +<li class="toctree-l3 current"><a class="current reference internal" href="">2.2.6. SFTP Connector</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#usage">2.2.6.1. Usage</a></li> +<li class="toctree-l4"><a class="reference internal" href="#loader">2.2.6.2. Loader</a></li> +</ul> +</li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="../Examples.html">2.3. Examples</a></li> +<li class="toctree-l2"><a class="reference internal" href="../Sqoop5MinutesDemo.html">2.4. Sqoop 5 Minutes Demo</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../../index.html">Docs</a> »</li> + + <li><a href="../../user.html">2. User Guide</a> »</li> + + <li><a href="../Connectors.html">2.2. Connectors</a> »</li> + + <li>2.2.6. SFTP Connector</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../../_sources/user/connectors/Connector-SFTP.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="sftp-connector"> +<h1><a class="toc-backref" href="#id2">2.2.6. SFTP Connector</a><a class="headerlink" href="#sftp-connector" title="Permalink to this headline">¶</a></h1> +<p>The SFTP connector supports moving data between a Secure File Transfer Protocol (SFTP) server and other supported Sqoop2 connectors.</p> +<p>Currently only the TO direction is supported to write records to an SFTP server. A FROM connector is pending (SQOOP-2218).</p> +<div class="contents topic" id="contents"> +<p class="topic-title first">Contents</p> +<ul class="simple"> +<li><a class="reference internal" href="#sftp-connector" id="id2">SFTP Connector</a><ul> +<li><a class="reference internal" href="#usage" id="id3">Usage</a><ul> +<li><a class="reference internal" href="#link-configuration" id="id4"><strong>Link Configuration</strong></a></li> +<li><a class="reference internal" href="#to-job-configuration" id="id5"><strong>TO Job Configuration</strong></a></li> +</ul> +</li> +<li><a class="reference internal" href="#loader" id="id6">Loader</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="usage"> +<h2><a class="toc-backref" href="#id3">2.2.6.1. Usage</a><a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2> +<p>Before executing a Sqoop2 job with the SFTP connector, set <strong>mapreduce.task.classpath.user.precedence</strong> to true in the Hadoop cluster config, for example:</p> +<div class="highlight-none"><div class="highlight"><pre><property> + <name>mapreduce.task.classpath.user.precedence</name> + <value>true</value> +</property> +</pre></div> +</div> +<p>This is required since the SFTP connector uses the JSch library (<a class="reference external" href="http://www.jcraft.com/jsch/";>http://www.jcraft.com/jsch/</a>) to provide SFTP functionality. Unfortunately Hadoop currently ships with an earlier version of this library which causes an issue with some SFTP servers. Setting this property ensures that the current version of the library packaged with this connector will appear first in the classpath.</p> +<p>To use the SFTP Connector, create a link for the connector and a job that uses the link.</p> +<div class="section" id="link-configuration"> +<h3><a class="toc-backref" href="#id4">2.2.6.1.1. <strong>Link Configuration</strong></a><a class="headerlink" href="#link-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the link configuration include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="21%" /> +<col width="7%" /> +<col width="52%" /> +<col width="20%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>SFTP server hostname</td> +<td>String</td> +<td>Hostname for the SFTP server. +<em>Required</em>.</td> +<td>sftp.example.com</td> +</tr> +<tr class="row-odd"><td>SFTP server port</td> +<td>Integer</td> +<td>Port number for the SFTP server. Defaults to 22. +<em>Optional</em>.</td> +<td>2220</td> +</tr> +<tr class="row-even"><td>Username</td> +<td>String</td> +<td>The username to provide when connecting to the SFTP server. +<em>Required</em>.</td> +<td>sqoop</td> +</tr> +<tr class="row-odd"><td>Password</td> +<td>String</td> +<td>The password to provide when connecting to the SFTP server. +<em>Required</em></td> +<td>sqoop</td> +</tr> +</tbody> +</table> +<div class="section" id="notes"> +<h4>2.2.6.1.1.1. <strong>Notes</strong><a class="headerlink" href="#notes" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The SFTP connector will attempt to connect to the SFTP server as part of the link validation process. If for some reason a connection can not be established, you’ll see a corresponding error message.</li> +<li>Note that during connection, the SFTP connector explictly disables <em>StrictHostKeyChecking</em> to avoid “UnknownHostKey” errors.</li> +</ol> +</div> +</div> +<div class="section" id="to-job-configuration"> +<h3><a class="toc-backref" href="#id5">2.2.6.1.2. <strong>TO Job Configuration</strong></a><a class="headerlink" href="#to-job-configuration" title="Permalink to this headline">¶</a></h3> +<p>Inputs associated with the Job configuration for the TO direction include:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="20%" /> +<col width="6%" /> +<col width="50%" /> +<col width="24%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">Input</th> +<th class="head">Type</th> +<th class="head">Description</th> +<th class="head">Example</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>Output directory</td> +<td>String</td> +<td>The location on the SFTP server that the connector will write files to. +<em>Required</em></td> +<td>uploads</td> +</tr> +</tbody> +</table> +<div class="section" id="id1"> +<h4>2.2.6.1.2.1. <strong>Notes</strong><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h4> +<ol class="arabic simple"> +<li>The <em>output directory</em> value needs to be an existing directory on the SFTP server.</li> +</ol> +</div> +</div> +</div> +<div class="section" id="loader"> +<h2><a class="toc-backref" href="#id6">2.2.6.2. Loader</a><a class="headerlink" href="#loader" title="Permalink to this headline">¶</a></h2> +<p>During the <em>loading</em> phase, the connector will create uniquely named files in the <em>output directory</em> for each partition of data received from the <strong>FROM</strong> connector.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="../Examples.html" class="btn btn-neutral float-right" title="2.3. Examples" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Connector-Kite.html" class="btn btn-neutral" title="2.2.5. Kite Connector" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../../_static/jquery.js"></script> + <script type="text/javascript" src="../../_static/underscore.js"></script> + <script type="text/javascript" src="../../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file
