Added: websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Installation.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Installation.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Installation.html Thu Jul 28 01:17:26 2016 @@ -0,0 +1,358 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>1.1. Installation — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../index.html"/> + <link rel="up" title="1. Admin Guide" href="../admin.html"/> + <link rel="next" title="1.2. Tools" href="Tools.html"/> + <link rel="prev" title="1. Admin Guide" href="../admin.html"/> + + + <script src="../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1 current"><a class="reference internal" href="../admin.html">1. Admin Guide</a><ul class="current"> +<li class="toctree-l2 current"><a class="current reference internal" href="">1.1. Installation</a><ul> +<li class="toctree-l3"><a class="reference internal" href="#server-installation">1.1.1. Server installation</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#hadoop-dependencies">1.1.1.1. Hadoop dependencies</a></li> +<li class="toctree-l4"><a class="reference internal" href="#hadoop-configuration">1.1.1.2. Hadoop configuration</a></li> +<li class="toctree-l4"><a class="reference internal" href="#third-party-jars">1.1.1.3. Third party jars</a></li> +<li class="toctree-l4"><a class="reference internal" href="#configuring-path">1.1.1.4. Configuring <tt class="docutils literal"><span class="pre">PATH</span></tt></a></li> +<li class="toctree-l4"><a class="reference internal" href="#configuring-server">1.1.1.5. Configuring Server</a></li> +<li class="toctree-l4"><a class="reference internal" href="#repository-initialization">1.1.1.6. Repository Initialization</a></li> +<li class="toctree-l4"><a class="reference internal" href="#server-life-cycle">1.1.1.7. Server Life Cycle</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="#client-installation">1.1.2. Client installation</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="Tools.html">1.2. Tools</a></li> +<li class="toctree-l2"><a class="reference internal" href="Upgrade.html">1.3. Upgrade</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../user.html">2. User Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="../admin.html">1. Admin Guide</a> »</li> + + <li>1.1. Installation</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../_sources/admin/Installation.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="installation"> +<h1>1.1. Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h1> +<p>Sqoop ships as one binary package that incorporates two separate parts - client and server.</p> +<ul class="simple"> +<li><strong>Server</strong> You need to install server on single node in your cluster. This node will then serve as an entry point for all Sqoop clients.</li> +<li><strong>Client</strong> Clients can be installed on any number of machines.</li> +</ul> +<div class="section" id="server-installation"> +<h2>1.1.1. Server installation<a class="headerlink" href="#server-installation" title="Permalink to this headline">¶</a></h2> +<p>Copy the Sqoop artifact to the machine where you want to run Sqoop server. The Sqoop server acts as a Hadoop client, therefore Hadoop libraries (Yarn, Mapreduce, and HDFS jar files) and configuration files (<tt class="docutils literal"><span class="pre">core-site.xml</span></tt>, <tt class="docutils literal"><span class="pre">mapreduce-site.xml</span></tt>, ...) must be available on this node. You do not need to run any Hadoop related services - running the server on a “gateway” node is perfectly fine.</p> +<p>You should be able to list a HDFS for example:</p> +<div class="highlight-bash"><div class="highlight"><pre>hadoop dfs -ls +</pre></div> +</div> +<p>Sqoop currently supports Hadoop version 2.6.0 or later. To install the Sqoop server, decompress the tarball (in a location of your choosing) and set the newly created forder as your working directory.</p> +<div class="highlight-bash"><div class="highlight"><pre><span class="c"># Decompress Sqoop distribution tarball</span> +tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz + +<span class="c"># Move decompressed content to any location</span> +mv sqoop-<version>-bin-hadoop<hadoop version>.tar.gz /usr/lib/sqoop + +<span class="c"># Change working directory</span> +<span class="nb">cd</span> /usr/lib/sqoop +</pre></div> +</div> +<div class="section" id="hadoop-dependencies"> +<h3>1.1.1.1. Hadoop dependencies<a class="headerlink" href="#hadoop-dependencies" title="Permalink to this headline">¶</a></h3> +<p>Sqoop server needs following environmental variables pointing at Hadoop libraries - <tt class="docutils literal"><span class="pre">$HADOOP_COMMON_HOME</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_HDFS_HOME</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_MAPRED_HOME</span></tt> and <tt class="docutils literal"><span class="pre">$HADOOP_YARN_HOME</span></tt>. You have to make sure that those variables are defined and pointing to a valid Hadoop installation. Sqoop server will not start if Hadoop libraries can’t be found.</p> +<p>The Sqoop server uses environment variables to find Hadoop libraries. If the environment variable <tt class="docutils literal"><span class="pre">$HADOOP_HOME</span></tt> is set, Sqoop will look for jars in the following locations: <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/common</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/hdfs</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/mapreduce</span></tt> and <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/yarn</span></tt>. You can specify where the Sqoop server should look for the common, hdfs, mapreduce, and yarn jars indepently with the <tt class="docutils literal"><span class="pre">$HADOOP_COMMON_HOME</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_HDFS_HOME</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_MAPRED_HOME</span></tt> and <tt class="docutils literal">< span class="pre">$HADOOP_YARN_HOME</span></tt> environment variables.</p> +<div class="highlight-bash"><div class="highlight"><pre><span class="c"># Export HADOOP_HOME variable</span> +<span class="nb">export </span><span class="nv">HADOOP_HOME</span><span class="o">=</span>/... + +<span class="c"># Or alternatively HADOOP_*_HOME variables</span> +<span class="nb">export </span><span class="nv">HADOOP_COMMON_HOME</span><span class="o">=</span>/... +<span class="nb">export </span><span class="nv">HADOOP_HDFS_HOME</span><span class="o">=</span>/... +<span class="nb">export </span><span class="nv">HADOOP_MAPRED_HOME</span><span class="o">=</span>/... +<span class="nb">export </span><span class="nv">HADOOP_YARN_HOME</span><span class="o">=</span>/... +</pre></div> +</div> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">If the environment <tt class="docutils literal"><span class="pre">$HADOOP_HOME</span></tt> is set, Sqoop will usee the following locations: <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/common</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/hdfs</span></tt>, <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/mapreduce</span></tt> and <tt class="docutils literal"><span class="pre">$HADOOP_HOME/share/hadoop/yarn</span></tt>.</p> +</div> +</div> +<div class="section" id="hadoop-configuration"> +<h3>1.1.1.2. Hadoop configuration<a class="headerlink" href="#hadoop-configuration" title="Permalink to this headline">¶</a></h3> +<p>Sqoop server will need to impersonate users to access HDFS and other resources in or outside of the cluster as the user who started given job rather then user who is running the server. You need to configure Hadoop to explicitly allow this impersonation via so called proxyuser system. You need to create two properties in <tt class="docutils literal"><span class="pre">core-site.xml</span></tt> file - <tt class="docutils literal"><span class="pre">hadoop.proxyuser.$SERVER_USER.hosts</span></tt> and <tt class="docutils literal"><span class="pre">hadoop.proxyuser.$SERVER_USER.groups</span></tt> where <tt class="docutils literal"><span class="pre">$SERVER_USER</span></tt> is the user who will be running Sqoop 2 server. In most scenarios configuring <tt class="docutils literal"><span class="pre">*</span></tt> is sufficient. Please refer to Hadoop documentation for details how to use those properties.</p> +<p>Example fragment that needs to be present in <tt class="docutils literal"><span class="pre">core-site.xml</span></tt> file for case when server is running under <tt class="docutils literal"><span class="pre">sqoop2</span></tt> user:</p> +<div class="highlight-xml"><div class="highlight"><pre><span class="nt"><property></span> + <span class="nt"><name></span>hadoop.proxyuser.sqoop2.hosts<span class="nt"></name></span> + <span class="nt"><value></span>*<span class="nt"></value></span> +<span class="nt"></property></span> +<span class="nt"><property></span> + <span class="nt"><name></span>hadoop.proxyuser.sqoop2.groups<span class="nt"></name></span> + <span class="nt"><value></span>*<span class="nt"></value></span> +<span class="nt"></property></span> +</pre></div> +</div> +<p>If you’re running Sqoop 2 server under a so called system user (user with ID less then <tt class="docutils literal"><span class="pre">min.user.id</span></tt> - 1000 by default), then YARN will by default refuse to run Sqoop 2 jobs. You will need to add the user name who is running Sqoop 2 server (most likely user <tt class="docutils literal"><span class="pre">sqoop2</span></tt>) to a <tt class="docutils literal"><span class="pre">allowed.system.users</span></tt> property of <tt class="docutils literal"><span class="pre">container-executor.cfg</span></tt>. Please refer to YARN documentation for further details.</p> +<p>Example fragment that needs to be present in <tt class="docutils literal"><span class="pre">container-executor.cfg</span></tt> file for case when server is running under <tt class="docutils literal"><span class="pre">sqoop2</span></tt> user:</p> +<div class="highlight-xml"><div class="highlight"><pre>allowed.system.users=sqoop2 +</pre></div> +</div> +</div> +<div class="section" id="third-party-jars"> +<h3>1.1.1.3. Third party jars<a class="headerlink" href="#third-party-jars" title="Permalink to this headline">¶</a></h3> +<p>To propagate any third party jars to Sqoop server classpath, create a directory anywhere on the file system and export it’s location in <tt class="docutils literal"><span class="pre">SQOOP_SERVER_EXTRA_LIB</span></tt> variable.</p> +<div class="highlight-bash"><div class="highlight"><pre><span class="c"># Create directory for extra jars</span> +mkdir -p /var/lib/sqoop2/ + +<span class="c"># Copy all your JDBC drivers to this directory</span> +cp mysql-jdbc*.jar /var/lib/sqoop2/ +cp postgresql-jdbc*.jar /var/lib/sqoop2/ + +<span class="c"># And finally export this directory to SQOOP_SERVER_EXTRA_LIB</span> +<span class="nb">export </span><span class="nv">SQOOP_SERVER_EXTRA_LIB</span><span class="o">=</span>/var/lib/sqoop2/ +</pre></div> +</div> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">Sqoop doesn’t ship with any JDBC drivers due to incompatible licenses. You will need to use this mechanism to install all JDBC drivers that are needed.</p> +</div> +</div> +<div class="section" id="configuring-path"> +<h3>1.1.1.4. Configuring <tt class="docutils literal"><span class="pre">PATH</span></tt><a class="headerlink" href="#configuring-path" title="Permalink to this headline">¶</a></h3> +<p>All user and administrator facing shell commands are stored in <tt class="docutils literal"><span class="pre">bin/</span></tt> directory. It’s recommended to add this directory to your <tt class="docutils literal"><span class="pre">$PATH</span></tt> for easier execution, for example:</p> +<div class="highlight-bash"><div class="highlight"><pre><span class="nv">PATH</span><span class="o">=</span><span class="nv">$PATH</span>:<span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/bin/ +</pre></div> +</div> +<p>The remainder of the Sqoop 2 documentation assumes that the shell commands are in your <tt class="docutils literal"><span class="pre">$PATH</span></tt>.</p> +</div> +<div class="section" id="configuring-server"> +<h3>1.1.1.5. Configuring Server<a class="headerlink" href="#configuring-server" title="Permalink to this headline">¶</a></h3> +<p>Server configuration files are stored in <tt class="docutils literal"><span class="pre">conf</span></tt> directory. File <tt class="docutils literal"><span class="pre">sqoop_bootstrap.properties</span></tt> specifies which configuration provider should be used for loading configuration for rest of Sqoop server. Default value <tt class="docutils literal"><span class="pre">PropertiesConfigurationProvider</span></tt> should be sufficient.</p> +<p>Second configuration file called <tt class="docutils literal"><span class="pre">sqoop.properties</span></tt> contains remaining configuration properties that can affect Sqoop server. The configuration file is very well documented, so check if all configuration properties fits your environment. Default or very little tweaking should be sufficient in most common cases.</p> +</div> +<div class="section" id="repository-initialization"> +<h3>1.1.1.6. Repository Initialization<a class="headerlink" href="#repository-initialization" title="Permalink to this headline">¶</a></h3> +<p>The metadata repository needs to be initialized before starting Sqoop 2 server for the first time. Use <a class="reference internal" href="Tools.html#tool-upgrade"><em>Upgrade tool</em></a> to initialize the repository:</p> +<div class="highlight-bash"><div class="highlight"><pre>sqoop2-tool upgrade +</pre></div> +</div> +<p>You can verify if everything have been configured correctly using <a class="reference internal" href="Tools.html#tool-verify"><em>Verify tool</em></a>:</p> +<div class="highlight-bash"><div class="highlight"><pre>sqoop2-tool verify +... +Verification was successful. +Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly +</pre></div> +</div> +</div> +<div class="section" id="server-life-cycle"> +<h3>1.1.1.7. Server Life Cycle<a class="headerlink" href="#server-life-cycle" title="Permalink to this headline">¶</a></h3> +<p>After installation and configuration you can start Sqoop server with following command:</p> +<div class="highlight-bash"><div class="highlight"><pre>sqoop2-server start +</pre></div> +</div> +<p>You can stop the server using the following command:</p> +<div class="highlight-bash"><div class="highlight"><pre>sqoop2-server stop +</pre></div> +</div> +<p>By default Sqoop server daemon use port <tt class="docutils literal"><span class="pre">12000</span></tt>. You can set <tt class="docutils literal"><span class="pre">org.apache.sqoop.jetty.port</span></tt> in configuration file <tt class="docutils literal"><span class="pre">conf/sqoop.properties</span></tt> to use different port.</p> +</div> +</div> +<div class="section" id="client-installation"> +<h2>1.1.2. Client installation<a class="headerlink" href="#client-installation" title="Permalink to this headline">¶</a></h2> +<p>Just copy Sqoop distribution artifact on target machine and unzip it in desired location. You can start client with following command:</p> +<div class="highlight-bash"><div class="highlight"><pre>sqoop2-shell +</pre></div> +</div> +<p>You can find more documentation for Sqoop shell in <a class="reference internal" href="../user/CommandLineClient.html"><em>Command Line Shell</em></a>.</p> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">Client is not acting as a Hadoop client and thus you do not need to be installed on node with Hadoop libraries and configuration files.</p> +</div> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Tools.html" class="btn btn-neutral float-right" title="1.2. Tools" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="../admin.html" class="btn btn-neutral" title="1. Admin Guide" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file
Added: websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Tools.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Tools.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Tools.html Thu Jul 28 01:17:26 2016 @@ -0,0 +1,431 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>1.2. Tools — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../index.html"/> + <link rel="up" title="1. Admin Guide" href="../admin.html"/> + <link rel="next" title="1.3. Upgrade" href="Upgrade.html"/> + <link rel="prev" title="1.1. Installation" href="Installation.html"/> + + + <script src="../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1 current"><a class="reference internal" href="../admin.html">1. Admin Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="Installation.html">1.1. Installation</a></li> +<li class="toctree-l2 current"><a class="current reference internal" href="">1.2. Tools</a><ul> +<li class="toctree-l3"><a class="reference internal" href="#verify-tool">1.2.1. Verify tool</a></li> +<li class="toctree-l3"><a class="reference internal" href="#upgrade-tool">1.2.2. Upgrade tool</a></li> +<li class="toctree-l3"><a class="reference internal" href="#repositorydump">1.2.3. RepositoryDump</a></li> +<li class="toctree-l3"><a class="reference internal" href="#repositoryload">1.2.4. RepositoryLoad</a></li> +<li class="toctree-l3"><a class="reference internal" href="#repositoryencryption">1.2.5. RepositoryEncryption</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#changing-the-password">1.2.5.1. Changing the Password</a></li> +<li class="toctree-l4"><a class="reference internal" href="#generate-a-new-key-for-the-existing-password">1.2.5.2. Generate a New Key for the Existing Password</a></li> +<li class="toctree-l4"><a class="reference internal" href="#encrypting-an-existing-unencrypted-repository">1.2.5.3. Encrypting an Existing Unencrypted Repository</a></li> +<li class="toctree-l4"><a class="reference internal" href="#decrypting-an-existing-encrypted-repository">1.2.5.4. Decrypting an Existing Encrypted Repository</a></li> +</ul> +</li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="Upgrade.html">1.3. Upgrade</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../user.html">2. User Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="../admin.html">1. Admin Guide</a> »</li> + + <li>1.2. Tools</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../_sources/admin/Tools.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="tools"> +<h1>1.2. Tools<a class="headerlink" href="#tools" title="Permalink to this headline">¶</a></h1> +<p>Tools are server commands that administrators can execute on the Sqoop server machine in order to perform various maintenance tasks. The tool execution will always perform a given task and finish. There are no long running services implemented as tools.</p> +<p>In order to perform the maintenance task each tool is suppose to do, they need to be executed in exactly the same environment as the main Sqoop server. The tool binary will take care of setting up the <tt class="docutils literal"><span class="pre">CLASSPATH</span></tt> and other environmental variables that might be required. However it’s up to the administrator himself to run the tool under the same user as is used for the server. This is usually configured automatically for various Hadoop distributions (such as Apache Bigtop).</p> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">Running tools while the Sqoop Server is also running is not recommended as it might lead to a data corruption and service disruption.</p> +</div> +<p>List of available tools:</p> +<ul class="simple"> +<li>verify</li> +<li>upgrade</li> +</ul> +<p>To run the desired tool, execute binary <tt class="docutils literal"><span class="pre">sqoop2-tool</span></tt> with desired tool name. For example to run <tt class="docutils literal"><span class="pre">verify</span></tt> tool:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool verify +</pre></div> +</div> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">Stop the Sqoop Server before running Sqoop tools. Running tools while Sqoop Server is running can lead to a data corruption and service disruption.</p> +</div> +<div class="section" id="verify-tool"> +<span id="tool-verify"></span><h2>1.2.1. Verify tool<a class="headerlink" href="#verify-tool" title="Permalink to this headline">¶</a></h2> +<p>The verify tool will verify Sqoop server configuration by starting all subsystems with the exception of servlets and tearing them down.</p> +<p>To run the <tt class="docutils literal"><span class="pre">verify</span></tt> tool:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool verify +</pre></div> +</div> +<p>If the verification process succeeds, you should see messages like:</p> +<div class="highlight-none"><div class="highlight"><pre>Verification was successful. +Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly +</pre></div> +</div> +<p>If the verification process will find any inconsistencies, it will print out the following message instead:</p> +<div class="highlight-none"><div class="highlight"><pre>Verification has failed, please check Server logs for further details. +Tool class org.apache.sqoop.tools.tool.VerifyTool has failed. +</pre></div> +</div> +<p>Further details why the verification has failed will be available in the Sqoop server log - same file as the Sqoop Server logs into.</p> +</div> +<div class="section" id="upgrade-tool"> +<span id="tool-upgrade"></span><h2>1.2.2. Upgrade tool<a class="headerlink" href="#upgrade-tool" title="Permalink to this headline">¶</a></h2> +<p>Upgrades all versionable components inside Sqoop2. This includes structural changes inside the repository and stored metadata. +Running this tool on Sqoop deployment that was already upgraded will have no effect.</p> +<p>To run the <tt class="docutils literal"><span class="pre">upgrade</span></tt> tool:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool upgrade +</pre></div> +</div> +<p>Upon successful upgrade you should see following message:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly. +</pre></div> +</div> +<p>Execution failure will show the following message instead:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.UpgradeTool has failed. +</pre></div> +</div> +<p>Further details why the upgrade process has failed will be available in the Sqoop server log - same file as the Sqoop Server logs into.</p> +</div> +<div class="section" id="repositorydump"> +<h2>1.2.3. RepositoryDump<a class="headerlink" href="#repositorydump" title="Permalink to this headline">¶</a></h2> +<p>Writes the user-created contents of the Sqoop repository to a file in JSON format. This includes connections, jobs and submissions.</p> +<p>To run the <tt class="docutils literal"><span class="pre">repositorydump</span></tt> tool:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool repositorydump -o repository.json +</pre></div> +</div> +<p>As an option, the administrator can choose to include sensitive information such as database connection passwords in the file:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool repositorydump -o repository.json --include-sensitive +</pre></div> +</div> +<p>Upon successful execution, you should see the following message:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.RepositoryDumpTool has finished correctly. +</pre></div> +</div> +<p>If repository dump has failed, you will see the following message instead:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.RepositoryDumpTool has failed. +</pre></div> +</div> +<p>Further details why the upgrade process has failed will be available in the Sqoop server log - same file as the Sqoop Server logs into.</p> +</div> +<div class="section" id="repositoryload"> +<h2>1.2.4. RepositoryLoad<a class="headerlink" href="#repositoryload" title="Permalink to this headline">¶</a></h2> +<p>Reads a json formatted file created by RepositoryDump and loads to current Sqoop repository.</p> +<p>To run the <tt class="docutils literal"><span class="pre">repositoryLoad</span></tt> tool:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool repositoryload -i repository.json +</pre></div> +</div> +<p>Upon successful execution, you should see the following message:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.RepositoryLoadTool has finished correctly. +</pre></div> +</div> +<p>If repository load failed you will see the following message instead:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.RepositoryLoadTool has failed. +</pre></div> +</div> +<p>Or an exception. Further details why the upgrade process has failed will be available in the Sqoop server log - same file as the Sqoop Server logs into.</p> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">If the repository dump was created without passwords (default), the connections will not contain a password and the jobs will fail to execute. In that case you’ll need to manually update the connections and set the password.</p> +</div> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">RepositoryLoad tool will always generate new connections, jobs and submissions from the file. Even when an identical objects already exists in repository.</p> +</div> +</div> +<div class="section" id="repositoryencryption"> +<span id="repositoryencryption-tool"></span><h2>1.2.5. RepositoryEncryption<a class="headerlink" href="#repositoryencryption" title="Permalink to this headline">¶</a></h2> +<p>Please see <a class="reference internal" href="../security/RepositoryEncryption.html#repositoryencryption"><em>Repository Encryption</em></a> for more details on repository encryption.</p> +<p>Sometimes we may want to change the password that is used to encrypt our data, generate a new key for our existing password, +encrypt an existing unencrypted repository, or decrypt an existing encrypting repository. Sqoop 2 provides the +Repository Encryption Tool to allow us to do this.</p> +<p>Before using the tool it is important to shut down the Sqoop 2 server.</p> +<p>All changes that the tool makes occur in a single transaction with the repository, which will prevent leaving the +repository in a bad state.</p> +<p>The Repository Encryption Tool is very simple, it uses the exact same configuration specified above (with the exception +of <tt class="docutils literal"><span class="pre">useConf</span></tt>). Configuration prefixed with a “-F” represents the existing repository state, configuration prefixed with +a “-T” represents the desired repository state. If one of these configuration sets is left out that means unencrypted.</p> +<div class="section" id="changing-the-password"> +<h3>1.2.5.1. Changing the Password<a class="headerlink" href="#changing-the-password" title="Permalink to this headline">¶</a></h3> +<p>In order to change the password, we need to specify the current configuration with the existing password and the desired +configuration with the new password. It looks like this:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption \ + -Forg.apache.sqoop.security.repo_encryption.password=old_password \ + -Forg.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256 \ + -Forg.apache.sqoop.security.repo_encryption.cipher_algorithm=AES \ + -Forg.apache.sqoop.security.repo_encryption.cipher_key_size=16 \ + -Forg.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding \ + -Forg.apache.sqoop.security.repo_encryption.initialization_vector_size=16 \ + -Forg.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1 \ + -Forg.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000 \ + -Torg.apache.sqoop.security.repo_encryption.password=new_password \ + -Torg.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256 \ + -Torg.apache.sqoop.security.repo_encryption.cipher_algorithm=AES \ + -Torg.apache.sqoop.security.repo_encryption.cipher_key_size=16 \ + -Torg.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding \ + -Torg.apache.sqoop.security.repo_encryption.initialization_vector_size=16 \ + -Torg.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1 \ + -Torg.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000 +</pre></div> +</div> +</div> +<div class="section" id="generate-a-new-key-for-the-existing-password"> +<h3>1.2.5.2. Generate a New Key for the Existing Password<a class="headerlink" href="#generate-a-new-key-for-the-existing-password" title="Permalink to this headline">¶</a></h3> +<p>Just like with the previous scenario you could copy the same configuration twice like this:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption \ + -Forg.apache.sqoop.security.repo_encryption.password=password \ + -Forg.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256 \ + -Forg.apache.sqoop.security.repo_encryption.cipher_algorithm=AES \ + -Forg.apache.sqoop.security.repo_encryption.cipher_key_size=16 \ + -Forg.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding \ + -Forg.apache.sqoop.security.repo_encryption.initialization_vector_size=16 \ + -Forg.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1 \ + -Forg.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000 \ + -Torg.apache.sqoop.security.repo_encryption.password=password \ + -Torg.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256 \ + -Torg.apache.sqoop.security.repo_encryption.cipher_algorithm=AES \ + -Torg.apache.sqoop.security.repo_encryption.cipher_key_size=16 \ + -Torg.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding \ + -Torg.apache.sqoop.security.repo_encryption.initialization_vector_size=16 \ + -Torg.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1 \ + -Torg.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000 +</pre></div> +</div> +<p>But we do have a shortcut to make this easier:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption -FuseConf -TuseConf +</pre></div> +</div> +<p>The <tt class="docutils literal"><span class="pre">useConf</span></tt> option will read whatever configuration is already in the configured sqoop properties file and apply it +for the specified direction.</p> +</div> +<div class="section" id="encrypting-an-existing-unencrypted-repository"> +<h3>1.2.5.3. Encrypting an Existing Unencrypted Repository<a class="headerlink" href="#encrypting-an-existing-unencrypted-repository" title="Permalink to this headline">¶</a></h3> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption \ + -Torg.apache.sqoop.security.repo_encryption.password=password \ + -Torg.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256 \ + -Torg.apache.sqoop.security.repo_encryption.cipher_algorithm=AES \ + -Torg.apache.sqoop.security.repo_encryption.cipher_key_size=16 \ + -Torg.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding \ + -Torg.apache.sqoop.security.repo_encryption.initialization_vector_size=16 \ + -Torg.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1 \ + -Torg.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000 +</pre></div> +</div> +<p>If the configuration for the encrypted repository has already been written to the sqoop properties file, one can simply +execute:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption -TuseConf +</pre></div> +</div> +</div> +<div class="section" id="decrypting-an-existing-encrypted-repository"> +<h3>1.2.5.4. Decrypting an Existing Encrypted Repository<a class="headerlink" href="#decrypting-an-existing-encrypted-repository" title="Permalink to this headline">¶</a></h3> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption \ + -Forg.apache.sqoop.security.repo_encryption.password=password \ + -Forg.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256 \ + -Forg.apache.sqoop.security.repo_encryption.cipher_algorithm=AES \ + -Forg.apache.sqoop.security.repo_encryption.cipher_key_size=16 \ + -Forg.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding \ + -Forg.apache.sqoop.security.repo_encryption.initialization_vector_size=16 \ + -Forg.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1 \ + -Forg.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000 +</pre></div> +</div> +<p>If the configuration for the encrypted repository has not yet been removed from the sqoop properties file, one can simply +execute:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop.sh tool repositoryencryption -FuseConf +</pre></div> +</div> +</div> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="Upgrade.html" class="btn btn-neutral float-right" title="1.3. Upgrade" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Installation.html" class="btn btn-neutral" title="1.1. Installation" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Upgrade.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Upgrade.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.7/admin/Upgrade.html Thu Jul 28 01:17:26 2016 @@ -0,0 +1,276 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>1.3. Upgrade — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../index.html"/> + <link rel="up" title="1. Admin Guide" href="../admin.html"/> + <link rel="next" title="2. User Guide" href="../user.html"/> + <link rel="prev" title="1.2. Tools" href="Tools.html"/> + + + <script src="../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1 current"><a class="reference internal" href="../admin.html">1. Admin Guide</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="Installation.html">1.1. Installation</a></li> +<li class="toctree-l2"><a class="reference internal" href="Tools.html">1.2. Tools</a></li> +<li class="toctree-l2 current"><a class="current reference internal" href="">1.3. Upgrade</a><ul> +<li class="toctree-l3"><a class="reference internal" href="#upgrading-server">1.3.1. Upgrading Server</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#upgrading-server-using-upgrade-tool">1.3.1.1. Upgrading Server using upgrade tool</a></li> +<li class="toctree-l4"><a class="reference internal" href="#upgrading-server-on-start-up">1.3.1.2. Upgrading Server on start-up</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="#upgrading-client">1.3.2. Upgrading Client</a></li> +</ul> +</li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../user.html">2. User Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../dev.html">3. Developer Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="../admin.html">1. Admin Guide</a> »</li> + + <li>1.3. Upgrade</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../_sources/admin/Upgrade.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="upgrade"> +<h1>1.3. Upgrade<a class="headerlink" href="#upgrade" title="Permalink to this headline">¶</a></h1> +<p>This page describes procedure that you need to take in order to upgrade Sqoop from one release to a higher release. Upgrading both client and server component will be discussed separately.</p> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">Only updates from one Sqoop 2 release to another are covered, starting with upgrades from version 1.99.2. This guide do not contain general information how to upgrade from Sqoop 1 to Sqoop 2.</p> +</div> +<div class="section" id="upgrading-server"> +<h2>1.3.1. Upgrading Server<a class="headerlink" href="#upgrading-server" title="Permalink to this headline">¶</a></h2> +<p>As Sqoop server is using a database repository for persisting sqoop entities such as the connector, driver, links and jobs the repository schema might need to be updated as part of the server upgrade. In addition the configs and inputs described by the various connectors and the driver may also change with a new server version and might need a data upgrade.</p> +<p>There are two ways how to upgrade Sqoop entities in the repository, you can either execute upgrade tool or configure the sqoop server to perform all necessary upgrades on start up.</p> +<p>It’s strongly advised to back up the repository before moving on to next steps. Backup instructions will vary depending on the repository implementation. For example, using MySQL as a repository will require a different back procedure than Apache Derby. Please follow the repositories’ backup procedure.</p> +<div class="section" id="upgrading-server-using-upgrade-tool"> +<h3>1.3.1.1. Upgrading Server using upgrade tool<a class="headerlink" href="#upgrading-server-using-upgrade-tool" title="Permalink to this headline">¶</a></h3> +<p>Preferred upgrade path is to explicitly run the <a class="reference internal" href="Tools.html#tool-upgrade"><em>Upgrade tool</em></a>. First step is to however shutdown the server as having both the server and upgrade utility accessing the same repository might corrupt it:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-server stop +</pre></div> +</div> +<p>When the server has been successfully stopped, you can update the server bits and simply run the upgrade tool:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-tool upgrade +</pre></div> +</div> +<p>You should see that the upgrade process has been successful:</p> +<div class="highlight-none"><div class="highlight"><pre>Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly. +</pre></div> +</div> +<p>In case of any failure, please take a look into <a class="reference internal" href="Tools.html#tool-upgrade"><em>Upgrade tool</em></a> documentation page.</p> +</div> +<div class="section" id="upgrading-server-on-start-up"> +<h3>1.3.1.2. Upgrading Server on start-up<a class="headerlink" href="#upgrading-server-on-start-up" title="Permalink to this headline">¶</a></h3> +<p>The capability of performing the upgrade has been built-in to the server, however is disabled by default to avoid any unintentional changes to the repository. You can start the repository schema upgrade procedure by stopping the server:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-server stop +</pre></div> +</div> +<p>Before starting the server again you will need to enable the auto-upgrade feature that will perform all necessary changes during Sqoop Server start up.</p> +<p>You need to set the following property in configuration file <tt class="docutils literal"><span class="pre">sqoop.properties</span></tt> for the repository schema upgrade.</p> +<div class="highlight-none"><div class="highlight"><pre>org.apache.sqoop.repository.schema.immutable=false +</pre></div> +</div> +<p>You need to set the following property in configuration file <tt class="docutils literal"><span class="pre">sqoop.properties</span></tt> for the connector config data upgrade.</p> +<div class="highlight-none"><div class="highlight"><pre>org.apache.sqoop.connector.autoupgrade=true +</pre></div> +</div> +<p>You need to set the following property in configuration file <tt class="docutils literal"><span class="pre">sqoop.properties</span></tt> for the driver config data upgrade.</p> +<div class="highlight-none"><div class="highlight"><pre>org.apache.sqoop.driver.autoupgrade=true +</pre></div> +</div> +<p>When all properties are set, start the sqoop server using the following command:</p> +<div class="highlight-none"><div class="highlight"><pre>sqoop2-server start +</pre></div> +</div> +<p>All required actions will be performed automatically during the server bootstrap. It’s strongly advised to set all three properties to their original values once the server has been successfully started and the upgrade has completed</p> +</div> +</div> +<div class="section" id="upgrading-client"> +<h2>1.3.2. Upgrading Client<a class="headerlink" href="#upgrading-client" title="Permalink to this headline">¶</a></h2> +<p>Client do not require any manual steps during upgrade. Replacing the binaries with updated version is sufficient.</p> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="../user.html" class="btn btn-neutral float-right" title="2. User Guide" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="Tools.html" class="btn btn-neutral" title="1.2. Tools" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: websites/staging/sqoop/trunk/content/docs/1.99.7/dev.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.7/dev.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.7/dev.html Thu Jul 28 01:17:26 2016 @@ -0,0 +1,358 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>3. Developer Guide — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="index.html"/> + <link rel="next" title="3.1. Building Sqoop2 from source code" href="dev/BuildingSqoop2.html"/> + <link rel="prev" title="2.4. Sqoop 5 Minutes Demo" href="user/Sqoop5MinutesDemo.html"/> + + + <script src="_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="admin.html">1. Admin Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="user.html">2. User Guide</a></li> +<li class="toctree-l1 current"><a class="current reference internal" href="">3. Developer Guide</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/BuildingSqoop2.html">3.1. Building Sqoop2 from source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html">3.2. Sqoop Java Client API Guide</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ConnectorDevelopment.html">3.3. Sqoop 2 Connector Development</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/DevEnv.html">3.4. Sqoop 2 Development Environment Setup</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/RESTAPI.html">3.5. Sqoop REST API Guide</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/Repository.html">3.6. Repository</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="index.html">Docs</a> »</li> + + <li>3. Developer Guide</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="_sources/dev.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="developer-guide"> +<h1>3. Developer Guide<a class="headerlink" href="#developer-guide" title="Permalink to this headline">¶</a></h1> +<div class="toctree-wrapper compound"> +<ul> +<li class="toctree-l1"><a class="reference internal" href="dev/BuildingSqoop2.html">3.1. Building Sqoop2 from source code</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/BuildingSqoop2.html#downloading-source-code">3.1.1. Downloading source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/BuildingSqoop2.html#building-project">3.1.2. Building project</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/BuildingSqoop2.html#running-tests">3.1.3. Running tests</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="dev/ClientAPI.html">3.2. Sqoop Java Client API Guide</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#workflow">3.2.1. Workflow</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#project-dependencies">3.2.2. Project Dependencies</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#initialization">3.2.3. Initialization</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#link">3.2.4. Link</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/ClientAPI.html#save-link">3.2.4.1. Save Link</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#job">3.2.5. Job</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/ClientAPI.html#save-job">3.2.5.1. Save Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ClientAPI.html#list-of-status-codes">3.2.5.2. List of status codes</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ClientAPI.html#view-error-or-warning-valdiation-message">3.2.5.3. View Error or Warning valdiation message</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ClientAPI.html#updating-link-and-job">3.2.5.4. Updating link and job</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#job-start">3.2.6. Job Start</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ClientAPI.html#display-config-and-input-names-for-connector">3.2.7. Display Config and Input Names For Connector</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="dev/ConnectorDevelopment.html">3.3. Sqoop 2 Connector Development</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/ConnectorDevelopment.html#what-is-a-sqoop-connector">3.3.1. What is a Sqoop Connector?</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#when-do-we-add-a-new-connector">3.3.1.1. When do we add a new connector?</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/ConnectorDevelopment.html#connector-implementation">3.3.2. Connector Implementation</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#from">3.3.2.1. From</a><ul> +<li class="toctree-l4"><a class="reference internal" href="dev/ConnectorDevelopment.html#initializer-and-destroyer">3.3.2.1.1. Initializer and Destroyer</a></li> +<li class="toctree-l4"><a class="reference internal" href="dev/ConnectorDevelopment.html#partitioner">3.3.2.1.2. Partitioner</a></li> +<li class="toctree-l4"><a class="reference internal" href="dev/ConnectorDevelopment.html#extractor">3.3.2.1.3. Extractor</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#to">3.3.2.2. To</a><ul> +<li class="toctree-l4"><a class="reference internal" href="dev/ConnectorDevelopment.html#id1">3.3.2.2.1. Initializer and Destroyer</a></li> +<li class="toctree-l4"><a class="reference internal" href="dev/ConnectorDevelopment.html#loader">3.3.2.2.2. Loader</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#sqoop-connector-identifier-sqoopconnector-properties">3.3.2.3. Sqoop Connector Identifier : sqoopconnector.properties</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#sqoop-connector-build-time-dependencies">3.3.2.4. Sqoop Connector Build-time Dependencies</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#sqoop-connector-build">3.3.2.5. Sqoop Connector Build</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/ConnectorDevelopment.html#configurables">3.3.3. Configurables</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#configurable-registration">3.3.3.1. Configurable registration</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#configurations">3.3.3.2. Configurations</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#configs-and-inputs">3.3.3.3. Configs and Inputs</a><ul> +<li class="toctree-l4"><a class="reference internal" href="dev/ConnectorDevelopment.html#empty-configuration">3.3.3.3.1. Empty Configuration</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#configuration-resourcebundle">3.3.3.4. Configuration ResourceBundle</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/ConnectorDevelopment.html#validations-for-configs-and-inputs">3.3.3.5. Validations for Configs and Inputs</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/ConnectorDevelopment.html#loading-external-connectors">3.3.4. Loading External Connectors</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/ConnectorDevelopment.html#sqoop-2-mapreduce-job-execution-lifecycle-with-connector-api">3.3.5. Sqoop 2 MapReduce Job Execution Lifecycle with Connector API</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="dev/DevEnv.html">3.4. Sqoop 2 Development Environment Setup</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/DevEnv.html#system-requirement">3.4.1. System Requirement</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/DevEnv.html#java">3.4.1.1. Java</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/DevEnv.html#maven">3.4.1.2. Maven</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/DevEnv.html#eclipse-setup">3.4.2. Eclipse Setup</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="dev/RESTAPI.html">3.5. Sqoop REST API Guide</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/RESTAPI.html#initialization">3.5.1. Initialization</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/RESTAPI.html#understand-connector-driver-link-and-job">3.5.2. Understand Connector, Driver, Link and Job</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/RESTAPI.html#objects">3.5.3. Objects</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#configs-and-inputs">3.5.3.1. Configs and Inputs</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#exception-response">3.5.3.2. Exception Response</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#config-and-input-validation-status-response">3.5.3.3. Config and Input Validation Status Response</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#job-submission-status-response">3.5.3.4. Job Submission Status Response</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="dev/RESTAPI.html#header-parameters">3.5.4. Header Parameters</a></li> +<li class="toctree-l2"><a class="reference internal" href="dev/RESTAPI.html#rest-apis">3.5.5. REST APIs</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#version-get-get-sqoop-version">3.5.5.1. /version - [GET] - Get Sqoop Version</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-connectors-get-get-all-connectors">3.5.5.2. /v1/connectors - [GET] Get all Connectors</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-connector-cname-get-get-connector">3.5.5.3. /v1/connector/[cname] - [GET] - Get Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-driver-get-get-sqoop-driver">3.5.5.4. /v1/driver - [GET]- Get Sqoop Driver</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-links-get-get-all-links">3.5.5.5. /v1/links/ - [GET] Get all links</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-links-cname-cname-get-get-all-links-by-connector">3.5.5.6. /v1/links?cname=[cname] - [GET] Get all links by Connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-link-lname-get-get-link">3.5.5.7. /v1/link/[lname] - [GET] - Get Link</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-link-post-create-link">3.5.5.8. /v1/link - [POST] - Create Link</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-link-lname-put-update-link">3.5.5.9. /v1/link/[lname] - [PUT] - Update Link</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-link-lname-delete-delete-link">3.5.5.10. /v1/link/[lname] - [DELETE] - Delete Link</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-link-lname-enable-put-enable-link">3.5.5.11. /v1/link/[lname]/enable - [PUT] - Enable Link</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-link-lname-disable-put-disable-link">3.5.5.12. /v1/link/[lname]/disable - [PUT] - Disable Link</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-jobs-get-get-all-jobs">3.5.5.13. /v1/jobs/ - [GET] Get all jobs</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-jobs-cname-cname-get-get-all-jobs-by-connector">3.5.5.14. /v1/jobs?cname=[cname] - [GET] Get all jobs by connector</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-get-get-job">3.5.5.15. /v1/job/[jname] - [GET] - Get Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-post-create-job">3.5.5.16. /v1/job - [POST] - Create Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-put-update-job">3.5.5.17. /v1/job/[jname] - [PUT] - Update Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-delete-delete-job">3.5.5.18. /v1/job/[jname] - [DELETE] - Delete Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-enable-put-enable-job">3.5.5.19. /v1/job/[jname]/enable - [PUT] - Enable Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-disable-put-disable-job">3.5.5.20. /v1/job/[jname]/disable - [PUT] - Disable Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-start-put-start-job">3.5.5.21. /v1/job/[jname]/start - [PUT]- Start Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-stop-put-stop-job">3.5.5.22. /v1/job/[jname]/stop - [PUT]- Stop Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-job-jname-status-get-get-job-status">3.5.5.23. /v1/job/[jname]/status - [GET]- Get Job Status</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-submissions-get-get-all-job-submissions">3.5.5.24. /v1/submissions? - [GET] - Get all job Submissions</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-submissions-jname-jname-get-get-submissions-by-job">3.5.5.25. /v1/submissions?jname=[jname] - [GET] - Get Submissions by Job</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-roles-create-post-create-role">3.5.5.26. /v1/authorization/roles/create - [POST] - Create Role</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-role-role-name-delete-delete-role">3.5.5.27. /v1/authorization/role/[role-name] - [DELETE] - Delete Role</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-roles-principal-type-principal-type-principal-name-principal-name-get-get-all-roles-by-principal">3.5.5.28. /v1/authorization/roles?principal_type=[principal-type]&principal_name=[principal-name] - [GET] Get all Roles by Principal</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-principals-role-name-rname-get-get-all-principals-by-role">3.5.5.29. /v1/authorization/principals?role_name=[rname] - [GET] Get all Principals by Role</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-roles-grant-put-grant-a-role-to-a-principal">3.5.5.30. /v1/authorization/roles/grant - [PUT] - Grant a Role to a Principal</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-roles-revoke-put-revoke-a-role-from-a-principal">3.5.5.31. /v1/authorization/roles/revoke - [PUT] - Revoke a Role from a Principal</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-privileges-grant-put-grant-a-privilege-to-a-principal">3.5.5.32. /v1/authorization/privileges/grant - [PUT] - Grant a Privilege to a Principal</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-privileges-revoke-put-revoke-a-privilege-to-a-principal">3.5.5.33. /v1/authorization/privileges/revoke - [PUT] - Revoke a Privilege to a Principal</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/RESTAPI.html#v1-authorization-privilieges-principal-type-principal-type-principal-name-principal-name-resource-type-resource-type-resource-name-resource-name-get-get-all-roles-by-principal-and-resource">3.5.5.34. /v1/authorization/privilieges?principal_type=[principal-type]&principal_name=[principal-name]&resource_type=[resource-type]&resource_name=[resource-name] - [GET] Get all Roles by Principal (and Resource)</a></li> +</ul> +</li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="dev/Repository.html">3.6. Repository</a><ul> +<li class="toctree-l2"><a class="reference internal" href="dev/Repository.html#sqoop-schema">3.6.1. Sqoop Schema</a><ul> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-system">3.6.1.1. SQ_SYSTEM</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-direction">3.6.1.2. SQ_DIRECTION</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-configurable">3.6.1.3. SQ_CONFIGURABLE</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-connector-directions">3.6.1.4. SQ_CONNECTOR_DIRECTIONS</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-config">3.6.1.5. SQ_CONFIG</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-config-directions">3.6.1.6. SQ_CONFIG_DIRECTIONS</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-input">3.6.1.7. SQ_INPUT</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-link">3.6.1.8. SQ_LINK</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-job">3.6.1.9. SQ_JOB</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-link-input">3.6.1.10. SQ_LINK_INPUT</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-job-input">3.6.1.11. SQ_JOB_INPUT</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-submission">3.6.1.12. SQ_SUBMISSION</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-counter-group">3.6.1.13. SQ_COUNTER_GROUP</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-counter">3.6.1.14. SQ_COUNTER</a></li> +<li class="toctree-l3"><a class="reference internal" href="dev/Repository.html#sq-counter-submission">3.6.1.15. SQ_COUNTER_SUBMISSION</a></li> +</ul> +</li> +</ul> +</li> +</ul> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="dev/BuildingSqoop2.html" class="btn btn-neutral float-right" title="3.1. Building Sqoop2 from source code" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="user/Sqoop5MinutesDemo.html" class="btn btn-neutral" title="2.4. Sqoop 5 Minutes Demo" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: websites/staging/sqoop/trunk/content/docs/1.99.7/dev/BuildingSqoop2.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.7/dev/BuildingSqoop2.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.7/dev/BuildingSqoop2.html Thu Jul 28 01:17:26 2016 @@ -0,0 +1,261 @@ + + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>3.1. Building Sqoop2 from source code — Apache Sqoop documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + + + + + + <link rel="top" title="Apache Sqoop documentation" href="../index.html"/> + <link rel="up" title="3. Developer Guide" href="../dev.html"/> + <link rel="next" title="3.2. Sqoop Java Client API Guide" href="ClientAPI.html"/> + <link rel="prev" title="3. Developer Guide" href="../dev.html"/> + + + <script src="../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../index.html" class="icon icon-home"> Apache Sqoop + + + + + <img src="../_static/sqoop-logo.png" class="logo" /> + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../admin.html">1. Admin Guide</a></li> +<li class="toctree-l1"><a class="reference internal" href="../user.html">2. User Guide</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="../dev.html">3. Developer Guide</a><ul class="current"> +<li class="toctree-l2 current"><a class="current reference internal" href="">3.1. Building Sqoop2 from source code</a><ul> +<li class="toctree-l3"><a class="reference internal" href="#downloading-source-code">3.1.1. Downloading source code</a></li> +<li class="toctree-l3"><a class="reference internal" href="#building-project">3.1.2. Building project</a></li> +<li class="toctree-l3"><a class="reference internal" href="#running-tests">3.1.3. Running tests</a></li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="ClientAPI.html">3.2. Sqoop Java Client API Guide</a></li> +<li class="toctree-l2"><a class="reference internal" href="ConnectorDevelopment.html">3.3. Sqoop 2 Connector Development</a></li> +<li class="toctree-l2"><a class="reference internal" href="DevEnv.html">3.4. Sqoop 2 Development Environment Setup</a></li> +<li class="toctree-l2"><a class="reference internal" href="RESTAPI.html">3.5. Sqoop REST API Guide</a></li> +<li class="toctree-l2"><a class="reference internal" href="Repository.html">3.6. Repository</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../security.html">4. Security Guide</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Apache Sqoop</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="../dev.html">3. Developer Guide</a> »</li> + + <li>3.1. Building Sqoop2 from source code</li> + <li class="wy-breadcrumbs-aside"> + + + <a href="../_sources/dev/BuildingSqoop2.txt" rel="nofollow"> View page source</a> + + + </li> + </ul> + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="building-sqoop2-from-source-code"> +<h1>3.1. Building Sqoop2 from source code<a class="headerlink" href="#building-sqoop2-from-source-code" title="Permalink to this headline">¶</a></h1> +<p>This guide will show you how to build Sqoop2 from source code. Sqoop is using <a class="reference external" href="http://maven.apache.org/";>maven</a> as build system. You you will need to use at least version 3.0 as older versions will not work correctly. All other dependencies will be downloaded by maven automatically. With exception of special JDBC drivers that are needed only for advanced integration tests.</p> +<div class="section" id="downloading-source-code"> +<h2>3.1.1. Downloading source code<a class="headerlink" href="#downloading-source-code" title="Permalink to this headline">¶</a></h2> +<p>Sqoop project is using git as a revision control system hosted at Apache Software Foundation. You can clone entire repository using following command:</p> +<div class="highlight-none"><div class="highlight"><pre>git clone https://git-wip-us.apache.org/repos/asf/sqoop.git sqoop2 +</pre></div> +</div> +<p>Sqoop2 is currently developed in special branch <tt class="docutils literal"><span class="pre">sqoop2</span></tt> that you need to check out after clone:</p> +<div class="highlight-none"><div class="highlight"><pre>cd sqoop2 +git checkout sqoop2 +</pre></div> +</div> +</div> +<div class="section" id="building-project"> +<h2>3.1.2. Building project<a class="headerlink" href="#building-project" title="Permalink to this headline">¶</a></h2> +<p>You can use usual maven targets like <tt class="docutils literal"><span class="pre">compile</span></tt> or <tt class="docutils literal"><span class="pre">package</span></tt> to build the project. Sqoop supports one major Hadoop revision at the moment - 2.x. As compiled code for one Hadoop major version can’t be used on another, you must compile Sqoop against appropriate Hadoop version.</p> +<div class="highlight-none"><div class="highlight"><pre>mvn compile +</pre></div> +</div> +<p>Maven target <tt class="docutils literal"><span class="pre">package</span></tt> can be used to create Sqoop packages similar to the ones that are officially available for download. Sqoop will build only source tarball by default. You need to specify <tt class="docutils literal"><span class="pre">-Pbinary</span></tt> to build binary distribution.</p> +<div class="highlight-none"><div class="highlight"><pre>mvn package -Pbinary +</pre></div> +</div> +</div> +<div class="section" id="running-tests"> +<h2>3.1.3. Running tests<a class="headerlink" href="#running-tests" title="Permalink to this headline">¶</a></h2> +<p>Sqoop supports two different sets of tests. First smaller and much faster set is called <strong>unit tests</strong> and will be executed on maven target <tt class="docutils literal"><span class="pre">test</span></tt>. Second larger set of <strong>integration tests</strong> will be executed on maven target <tt class="docutils literal"><span class="pre">integration-test</span></tt>. Please note that integration tests might require manual steps for installing various JDBC drivers into your local maven cache.</p> +<p>Example for running unit tests:</p> +<div class="highlight-none"><div class="highlight"><pre>mvn test +</pre></div> +</div> +<p>Example for running integration tests:</p> +<div class="highlight-none"><div class="highlight"><pre>mvn integration-test +</pre></div> +</div> +<p>For the <strong>unit tests</strong>, there are two helpful profiles: <strong>fast</strong> and <strong>slow</strong>. The <strong>fast</strong> unit tests do not start or use any services. The <strong>slow</strong> unit tests, may start services or use an external service (ie. MySQL).</p> +<div class="highlight-none"><div class="highlight"><pre>mvn test -Pfast,hadoop200 +mvn test -Pslow,hadoop200 +</pre></div> +</div> +</div> +</div> + + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="ClientAPI.html" class="btn btn-neutral float-right" title="3.2. Sqoop Java Client API Guide" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="../dev.html" class="btn btn-neutral" title="3. Developer Guide" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2009-2016 The Apache Software Foundation. + + </p> + </div> + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file
