Added: websites/staging/sqoop/trunk/content/docs/1.99.3/DevEnv.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.3/DevEnv.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.3/DevEnv.html Wed Nov 6 22:20:15 2013 @@ -0,0 +1,94 @@ + + +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>Sqoop 2 Development Environment Setup — Apache Sqoop documentation</title> + + <link rel="stylesheet" href="_static/haiku.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + <link rel="stylesheet" href="_static/print.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="_static/theme_extras.js"></script> + <link rel="top" title="Apache Sqoop documentation" href="index.html" /> + </head> + <body> + <div class="header"><img class="rightlogo" src="_static/sqoop-logo.png" alt="Logo"/><h1 class="heading"><a href="index.html"> + <span>Apache Sqoop documentation</span></a></h1> + <h2 class="heading"><span>Sqoop 2 Development Environment Setup</span></h2> + </div> + <div class="topnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + <div class="content"> + + + <div class="section" id="sqoop-2-development-environment-setup"> +<h1>Sqoop 2 Development Environment Setup<a class="headerlink" href="#sqoop-2-development-environment-setup" title="Permalink to this headline">¶</a></h1> +<p>This document describes you how to setup development environment for Sqoop 2.</p> +<div class="section" id="system-requirement"> +<h2>System Requirement<a class="headerlink" href="#system-requirement" title="Permalink to this headline">¶</a></h2> +<div class="section" id="java"> +<h3>Java<a class="headerlink" href="#java" title="Permalink to this headline">¶</a></h3> +<p>Sqoop written in Java and using version 1.6. You can <a class="reference external" href="http://www.oracle.com/technetwork/java/javase/downloads/index.html";>download java</a> and install. Locate JAVA_HOME to installed directroy, e.g. export JAVA_HOME=/usr/lib/jvm/jdk1.6.0_32.</p> +</div> +<div class="section" id="maven"> +<h3>Maven<a class="headerlink" href="#maven" title="Permalink to this headline">¶</a></h3> +<p>Sqoop uses Maven 3 for building the project. Download <a class="reference external" href="http://maven.apache.org/download.cgi";>Maven</a> and its Installation instructions given in <a class="reference external" href="http://maven.apache.org/download.cgi#Maven_Documentation";>link</a>.</p> +</div> +</div> +<div class="section" id="eclipse-setup"> +<h2>Eclipse Setup<a class="headerlink" href="#eclipse-setup" title="Permalink to this headline">¶</a></h2> +<p>Steps for downloading source code is given in <a class="reference external" href="BuildingSqoop2.html">Building Sqoop2</a></p> +<p>Sqoop 2 project has multiple modules where one module is depend on another module for e.g. sqoop 2 client module has sqoop 2 common module dependency. Follow below step for creating eclipse’s project and classpath for each module.</p> +<div class="highlight-none"><div class="highlight"><pre>//Install all package into local maven repository +mvn clean install -DskipTests + +//Adding M2_REPO variable to eclipse workspace +mvn eclipse:configure-workspace -Declipse.workspace=<path-to-eclipse-workspace-dir-for-sqoop-2> + +//Eclipse project creation with optional parameters +mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true +</pre></div> +</div> +<p>Alternatively, for manually adding M2_REPO classpath variable as maven repository path in eclipse-> window-> Java ->Classpath Variables ->Click “New” ->In new dialog box, input Name as M2_REPO and Path as $HOME/.m2/repository ->click Ok.</p> +<p>On successful execution of above maven commands, Then import the sqoop project modules into eclipse-> File -> Import ->General ->Existing Projects into Workspace-> Click Next-> Browse Sqoop 2 directory ($HOME/git/sqoop2) ->Click Ok ->Import dialog shows multiple projects (sqoop-client, sqoop-common, etc.) -> Select all modules -> click Finish.</p> +</div> +</div> + + + </div> + <div class="bottomnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + + <div class="footer"> + © Copyright 2009-2013 The Apache Software Foundation. + </div> + </body> +</html> \ No newline at end of file
Added: websites/staging/sqoop/trunk/content/docs/1.99.3/Installation.html ============================================================================== --- websites/staging/sqoop/trunk/content/docs/1.99.3/Installation.html (added) +++ websites/staging/sqoop/trunk/content/docs/1.99.3/Installation.html Wed Nov 6 22:20:15 2013 @@ -0,0 +1,116 @@ + + +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> + + +<html xmlns="http://www.w3.org/1999/xhtml";> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + + <title>Installation — Apache Sqoop documentation</title> + + <link rel="stylesheet" href="_static/haiku.css" type="text/css" /> + <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> + <link rel="stylesheet" href="_static/print.css" type="text/css" /> + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT: '', + VERSION: '', + COLLAPSE_INDEX: false, + FILE_SUFFIX: '.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="_static/theme_extras.js"></script> + <link rel="top" title="Apache Sqoop documentation" href="index.html" /> + </head> + <body> + <div class="header"><img class="rightlogo" src="_static/sqoop-logo.png" alt="Logo"/><h1 class="heading"><a href="index.html"> + <span>Apache Sqoop documentation</span></a></h1> + <h2 class="heading"><span>Installation</span></h2> + </div> + <div class="topnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + <div class="content"> + + + <div class="section" id="installation"> +<h1>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h1> +<p>Sqoop ships as one binary package however it’s compound from two separate parts - client and server. You need to install server on single node in your cluster. This node will then serve as an entry point for all connecting Sqoop clients. Server acts as a mapreduce client and therefore Hadoop must be installed and configured on machine hosting Sqoop server. Clients can be installed on any arbitrary number of machines. Client is not acting as a mapreduce client and thus you do not need to install Hadoop on nodes that will act only as a Sqoop client.</p> +<div class="section" id="server-installation"> +<h2>Server installation<a class="headerlink" href="#server-installation" title="Permalink to this headline">¶</a></h2> +<p>Copy Sqoop artifact on machine where you want to run Sqoop server. This machine must have installed and configured Hadoop. You don’t need to run any Hadoop related services there, however the machine must be able to act as an Hadoop client. You should be able to list a HDFS for example:</p> +<div class="highlight-none"><div class="highlight"><pre>hadoop dfs -ls +</pre></div> +</div> +<p>Sqoop server supports multiple Hadoop versions. However as Hadoop major versions are not compatible with each other, Sqoop have multiple binary artefacts - one for each supported major version of Hadoop. You need to make sure that you’re using appropriated binary artifact for your specific Hadoop version. To install Sqoop server decompress appropriate distribution artifact in location at your convenience and change your working directory to this folder.</p> +<div class="highlight-none"><div class="highlight"><pre># Decompress Sqoop distribution tarball +tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz + +# Move decompressed content to any location +mv sqoop-<version>-bin-hadoop<hadoop version>.tar.gz /usr/lib/sqoop + +# Change working directory +cd /usr/lib/sqoop +</pre></div> +</div> +<div class="section" id="installing-dependencies"> +<h3>Installing Dependencies<a class="headerlink" href="#installing-dependencies" title="Permalink to this headline">¶</a></h3> +<p>Hadoop libraries must be available on node where you are planning to run Sqoop server with proper configuration for major services - <tt class="docutils literal"><span class="pre">NameNode</span></tt> and either <tt class="docutils literal"><span class="pre">JobTracker</span></tt> or <tt class="docutils literal"><span class="pre">ResourceManager</span></tt> depending whether you are running Hadoop 1 or 2. There is no need to run any Hadoop service on the same node as Sqoop server, just the libraries and configuration must be available.</p> +<p>Path to Hadoop libraries is stored in file <tt class="docutils literal"><span class="pre">catalina.properties</span></tt> inside directory <tt class="docutils literal"><span class="pre">server/conf</span></tt>. You need to change property called <tt class="docutils literal"><span class="pre">common.loader</span></tt> to contain all directories with your Hadoop libraries. The default expected locations are <tt class="docutils literal"><span class="pre">/usr/lib/hadoop</span></tt> and <tt class="docutils literal"><span class="pre">/usr/lib/hadoop/lib/</span></tt>. Please check out the comments in the file for further description how to configure different locations.</p> +<p>Lastly you might need to install JDBC drivers that are not bundled with Sqoop because of incompatible licenses. You can add any arbitrary Java jar file to Sqoop server by copying it into <tt class="docutils literal"><span class="pre">lib/</span></tt> directory. You can create this directory if it do not exists already.</p> +</div> +<div class="section" id="configuring-server"> +<h3>Configuring Server<a class="headerlink" href="#configuring-server" title="Permalink to this headline">¶</a></h3> +<p>Before starting server you should revise configuration to match your specific environment. Server configuration files are stored in <tt class="docutils literal"><span class="pre">server/config</span></tt> directory of distributed artifact along side with other configuration files of Tomcat.</p> +<p>File <tt class="docutils literal"><span class="pre">sqoop_bootstrap.properties</span></tt> specifies which configuration provider should be used for loading configuration for rest of Sqoop server. Default value <tt class="docutils literal"><span class="pre">PropertiesConfigurationProvider</span></tt> should be sufficient.</p> +<p>Second configuration file <tt class="docutils literal"><span class="pre">sqoop.properties</span></tt> contains remaining configuration properties that can affect Sqoop server. File is very well documented, so check if all configuration properties fits your environment. Default or very little tweaking should be sufficient most common cases.</p> +</div> +<div class="section" id="server-life-cycle"> +<h3>Server Life Cycle<a class="headerlink" href="#server-life-cycle" title="Permalink to this headline">¶</a></h3> +<p>After installation and configuration you can start Sqoop server with following command:</p> +<div class="highlight-none"><div class="highlight"><pre>./bin/sqoop.sh server start +</pre></div> +</div> +<p>Similarly you can stop server using following command:</p> +<div class="highlight-none"><div class="highlight"><pre>./bin/sqoop.sh server stop +</pre></div> +</div> +<p>By default Sqoop server daemons use ports 12000 and 12001. You can set <tt class="docutils literal"><span class="pre">SQOOP_HTTP_PORT</span></tt> and <tt class="docutils literal"><span class="pre">SQOOP_ADMIN_PORT</span></tt> in configuration file <tt class="docutils literal"><span class="pre">server/bin/setenv.sh</span></tt> to use different ports.</p> +</div> +</div> +<div class="section" id="client-installation"> +<h2>Client installation<a class="headerlink" href="#client-installation" title="Permalink to this headline">¶</a></h2> +<p>Client do not need extra installation and configuration steps. Just copy Sqoop distribution artifact on target machine and unzip it in desired location. You can start client with following command:</p> +<div class="highlight-none"><div class="highlight"><pre>bin/sqoop.sh client +</pre></div> +</div> +<p>You can find more documentation to Sqoop client in <a class="reference external" href="CommandLineClient.html">Command Line Client</a> section.</p> +</div> +</div> + + + </div> + <div class="bottomnav"> + + <p> + <a class="uplink" href="index.html">Contents</a> + </p> + + </div> + + <div class="footer"> + © Copyright 2009-2013 The Apache Software Foundation. + </div> + </body> +</html> \ No newline at end of file
