Author: buildbot
Date: Thu Sep 4 20:37:10 2014
New Revision: 921257
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/developing/building.html
websites/staging/slider/trunk/content/developing/windows.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Thu Sep 4 20:37:10 2014
@@ -1 +1 @@
-1622109
+1622558
Modified: websites/staging/slider/trunk/content/developing/building.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/building.html (original)
+++ websites/staging/slider/trunk/content/developing/building.html Thu Sep 4
20:37:10 2014
@@ -179,8 +179,8 @@ Latest release: <strong>0.40-incubating<
<p>The network on the development system must be functional, with hostname
lookup
of the local host working. Tests will fail without this. For maven builds to
work, remote network access is often a pre-requisite.</p>
-<h3 id="java-7">Java 7</h3>
-<p>Slider is built on Java 7 -please have a JDK for Java 7 or 8 set up</p>
+<h3 id="java">Java</h3>
+<p>Slider is built on Java 6 or later</p>
<h3 id="python">Python</h3>
<p>Slider requires python 2.7</p>
<h3 id="maven">Maven</h3>
@@ -204,13 +204,18 @@ This is absolutely critical to prevent J
</pre></div>
-<p>Slider code base includes unit tests and functional tests. By default
functional tests are not run as these tests are developed to be run against
live hadoop clusters and require some manual setup. You can run them based as
described at <a href="/developing/functional_tests.html">functional test</a>.
</p>
+<p>Slider code base includes unit tests and functional tests.
+By default functional tests are not run as these tests are developed to be run
+against live hadoop clusters and require some manual setup.
+You can run them based as described at <a
href="/developing/functional_tests.html">functional test</a>. </p>
<h3 id="create-slider-package">Create Slider Package</h3>
<div class="codehilite"><pre> <span class="n">mvn</span> <span
class="n">clean</span> <span class="n">site</span><span class="p">:</span><span
class="n">site</span> <span class="n">site</span><span class="p">:</span><span
class="n">stage</span> <span class="n">package</span> <span
class="o">-</span><span class="n">DskipTests</span>
</pre></div>
-<p><strong><em>At this point you are ready to use the Slider toolset. The
build instructions below are optional and are proven to be useful when
debugging deep into the hadoop code base.</em></strong></p>
+<p><strong><em>At this point you are ready to use the Slider toolset.
+The build instructions below are optional and are proven to
+be useful when debugging deep into the hadoop code base.</em></strong></p>
<hr />
<h2 id="building-a-compatible-hadoop-version">Building a compatible Hadoop
version</h2>
<p>Slider is built against Hadoop 2 -you can download and install
@@ -218,25 +223,23 @@ a copy from the <a href="http://hadoop.a
<p>During development, its convenient (but not mandatory)
to have a local version of Hadoop -so that we can find and fix bugs/add
features in
Hadoop as well in Slider.</p>
-<p>To build and install locally, check out apache svn/github, branch
<code>release-2.4.0</code>,
+<p>To build and install locally, check out apache svn/github, branch
<code>branch-2</code>,
and create a branch off that tag</p>
<div class="codehilite"><pre><span class="n">git</span> <span
class="n">clone</span> <span class="n">git</span><span class="p">:</span><span
class="o">//</span><span class="n">git</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span class="n">org</span><span
class="o">/</span><span class="n">hadoop</span><span class="o">-</span><span
class="n">common</span><span class="p">.</span><span class="n">git</span>
<span class="n">cd</span> <span class="n">hadoop</span><span
class="o">-</span><span class="n">common</span>
-<span class="n">git</span> <span class="n">remote</span> <span
class="n">rename</span> <span class="n">origin</span> <span
class="n">apache</span>
-<span class="n">git</span> <span class="n">fetch</span> <span
class="o">--</span><span class="n">tags</span> <span class="n">apache</span>
-<span class="n">git</span> <span class="n">checkout</span> <span
class="n">release</span><span class="o">-</span>2<span class="p">.</span>4<span
class="p">.</span>0 <span class="o">--</span>
-<span class="n">git</span> <span class="n">checkout</span> <span
class="o">-</span><span class="n">b</span> <span class="n">release</span><span
class="o">-</span>2<span class="p">.</span>4<span class="p">.</span>0
-</pre></div>
+<span class="n">git</span> <span class="n">fetch</span> <span
class="o">--</span><span class="n">tags</span> <span class="n">origin</span>
+<span class="n">git</span> <span class="n">checkout</span> <span
class="o">-</span><span class="n">b</span> <span class="n">origin</span><span
class="o">/</span><span class="n">branch</span><span class="o">-</span>2
-<p>For the scripts below, set the <code>HADOOP_VERSION</code> variable to the
version</p>
-<div class="codehilite"><pre><span class="n">export</span> <span
class="n">HADOOP_VERSION</span><span class="p">=</span>2<span
class="p">.</span>4<span class="p">.</span>0
+<span class="n">git</span> <span class="n">checkout</span> <span
class="o">-</span><span class="n">b</span> <span class="n">branch</span><span
class="o">-</span>2
+<span class="n">export</span> <span class="n">HADOOP_VERSION</span><span
class="p">=</span>2<span class="p">.</span>6<span class="p">.</span>0<span
class="o">-</span><span class="n">SNAPSHOT</span>
</pre></div>
-<p>or, for building against a pre-release version of Hadoop 2.4</p>
-<div class="codehilite"><pre><span class="n">git</span> <span
class="n">checkout</span> <span class="n">branch</span><span class="o">-</span>2
-<span class="n">export</span> <span class="n">HADOOP_VERSION</span><span
class="p">=</span>2<span class="p">.</span>4<span class="p">.</span>0<span
class="o">-</span><span class="n">SNAPSHOT</span>
+<p>(to build against a release, check out that specific release and create
+a branch off it:)</p>
+<div class="codehilite"><pre><span class="n">git</span> <span
class="n">checkout</span> <span class="n">release</span><span
class="o">-</span>2<span class="p">.</span>6<span class="p">.</span>0 <span
class="o">--</span>
+<span class="n">git</span> <span class="n">checkout</span> <span
class="o">-</span><span class="n">b</span> <span class="n">release</span><span
class="o">-</span>2<span class="p">.</span>6<span class="p">.</span>0
</pre></div>
@@ -254,10 +257,9 @@ and create a branch off that tag</p>
</pre></div>
-<p>Then expand this</p>
+<p>Then clean expand this</p>
<div class="codehilite"><pre><span class="n">pushd</span> <span
class="n">hadoop</span><span class="o">-</span><span class="n">dist</span><span
class="o">/</span><span class="n">target</span><span class="o">/</span>
-<span class="n">gunzip</span> <span class="n">hadoop</span><span
class="o">-</span>$<span class="n">HADOOP_VERSION</span><span
class="p">.</span><span class="n">tar</span><span class="p">.</span><span
class="n">gz</span>
-<span class="n">tar</span> <span class="o">-</span><span class="n">xvf</span>
<span class="n">hadoop</span><span class="o">-</span>$<span
class="n">HADOOP_VERSION</span><span class="p">.</span><span
class="n">tar</span>
+<span class="n">tar</span> <span class="o">-</span><span class="n">xvzf</span>
<span class="n">hadoop</span><span class="o">-</span>$<span
class="n">HADOOP_VERSION</span><span class="p">.</span><span
class="n">tar</span><span class="p">.</span><span class="n">gz</span>
<span class="n">popd</span>
</pre></div>
@@ -540,7 +542,8 @@ either stream for logging</p>
</li>
</ol>
<p>Example:</p>
-<div class="codehilite"><pre><span class="n">target</span><span
class="o">/</span><span class="n">TestLiveRegionService</span><span
class="o">/</span><span class="n">TestLiveRegionService</span><span
class="o">-</span><span class="n">logDir</span><span class="o">-</span><span
class="n">nm</span><span class="o">-</span>1<span class="n">_0</span><span
class="o">/</span><span class="n">application_1376095770244_0001</span><span
class="o">/</span><span
class="n">container_1376095770244_0001_01_000001</span><span
class="o">/</span><span class="n">out</span><span class="p">.</span><span
class="n">txt</span>
+<div class="codehilite"><pre><span class="n">target</span><span
class="o">/</span><span class="n">TestLiveRegionService</span><span
class="o">/</span><span class="n">TestLiveRegionService</span><span
class="o">-</span><span class="n">logDir</span><span class="o">-</span><span
class="n">nm</span><span class="o">-</span>1<span class="n">_0</span><span
class="o">/</span>
+ <span class="n">application_1376095770244_0001</span><span
class="o">/</span><span
class="n">container_1376095770244_0001_01_000001</span><span
class="o">/</span><span class="n">out</span><span class="p">.</span><span
class="n">txt</span>
</pre></div>
Modified: websites/staging/slider/trunk/content/developing/windows.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/windows.html (original)
+++ websites/staging/slider/trunk/content/developing/windows.html Thu Sep 4
20:37:10 2014
@@ -186,20 +186,35 @@ Latest release: <strong>0.40-incubating<
</ul>
<h2 id="virtual-box">Virtual Box</h2>
<ol>
-<li>Ask for lots of RAM</li>
+<li>Ask for lots of RAM, CPU. You can vary CPU and limits as you go, and
+increase them for test runs/full screen work.</li>
<li>2D acceleration of video</li>
<li>Critical: Enable PAE under "processor". this avoids random hangs.</li>
</ol>
+<h3 id="os-features">OS features</h3>
+<p>In the server management app, go to "manage features", where you can add</p>
+<ul>
+<li>Telnet server: lets you telnet in.</li>
+<li>Maybe: desktop experience</li>
+</ul>
+<h3 id="os-settings">OS settings</h3>
+<p>Firewall setup is a mystery. You need to convince the OS that your Ether
network
+is "private", then edit the firewall to open it up.</p>
<h2 id="dev-tools">Dev tools</h2>
<ol>
-<li>Bitvise SSH server; add your private key to the account you create.</li>
+<li>Bitvise SSH server; add your private key to the account you create.
+ (even with this installed, you will still be mostly working from the windows
+ command prompt on the machine, as it has the env variables set up)</li>
<li>Git</li>
<li>Java 7</li>
-<li>Everything Hadoop's <code>BUILDING.TXT</code> needs. This includes the gnu
tools, protoc,
+<li>Python 2,7</li>
+<li>OpenSSL</li>
+<li>Everything Hadoop's <code>BUILDING.TXT</code> needs. This includes the
cygwin tools, protoc,
python and more. If you cannot build Hadoop from the <code>Windows SDK Command
Prompt</code>
then don't proceed.</li>
<li>PFE32 is still one of the best plain text editors for development around,
even just for viewing logs & test results.</li>
+<li>...JEdit is fairly nimble too, though not as lightweight.</li>
<li>Chrome can be used a browser instead of IE if you want âbut you should
still
view all web pages in IE to verify they look OK.</li>
</ol>
@@ -217,6 +232,130 @@ view all web pages in IE to verify they
</ol>
</li>
</ol>
+<h2 id="windows">Windows</h2>
+<ol>
+<li>Follow the Hadoop <code>BUILDING.TXT</code> instructions. </li>
+<li>install cygwin64 with it's <code>sh</code> command.</li>
+<li>Get the cygwin64 bin dir on the PATH (remember, ";" as separators)</li>
+<li>find where the <code>CL</code> C/C++ compiler is and make sure it is on
the PATH</li>
+<li>get the (native) <a href="http://www.cmake.org/";>CMake 2.8+</a> command on
the path,
+not the cygwin one.</li>
+</ol>
+<p>Things to install in cygwin
+<em> sh
+</em> ls
+<em> rm
+</em> cp
+<em> gzip
+</em> tar
+<em> awk
+</em> grep
+* xargs</p>
+<h2 id="avoid">Avoid</h2>
+<ul>
+<li>having cygwin <code>cmake</code> on your path; it must be the native
cmake</li>
+</ul>
+<h3 id="env-variables">env variables</h3>
+<div class="codehilite"><pre><span class="n">CYGWIN</span><span
class="p">=</span><span class="n">nodosfilewarning</span>
+<span class="n">PATH</span> <span class="p">=</span> <span
class="n">cmake</span> <span class="n">bin</span><span class="p">,</span> <span
class="n">cygwin</span> <span class="n">bin</span><span class="p">,</span>
<span class="n">python</span> <span class="n">bin</span><span
class="p">,</span> <span class="n">hadoop_home</span> <span
class="n">bin</span> <span class="p">(</span><span class="n">see</span> <span
class="n">below</span><span class="p">)</span>
+</pre></div>
+
+
+<p>example:</p>
+<div class="codehilite"><pre><span class="n">C</span><span
class="p">:</span><span class="o">\</span><span class="n">bin</span><span
class="o">\</span><span class="n">cygwin64</span><span class="o">\</span><span
class="n">bin</span><span class="p">;</span><span class="n">C</span><span
class="p">:</span><span class="o">\</span><span class="n">bin</span><span
class="o">\</span><span class="n">Python27</span><span class="o">\</span><span
class="p">;</span><span class="n">C</span><span class="p">:</span><span
class="o">\</span><span class="n">Windows</span><span class="o">\</span><span
class="n">system32</span><span class="p">;</span><span class="n">C</span><span
class="p">:</span><span class="o">\</span><span class="n">Windows</span><span
class="p">;</span>
+<span class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">Windows</span><span class="o">\</span><span
class="n">System32</span><span class="o">\</span><span
class="n">Wbem</span><span class="p">;</span><span class="n">C</span><span
class="p">:</span><span class="o">\</span><span class="n">Windows</span><span
class="o">\</span><span class="n">System32</span><span class="o">\</span><span
class="n">WindowsPowerShell</span><span class="o">\</span><span
class="n">v1</span><span class="p">.</span>0<span class="o">\</span><span
class="p">;</span>
+<span class="n">c</span><span class="p">:</span><span class="o">\</span><span
class="n">java</span><span class="o">\</span><span class="n">jdk7</span><span
class="o">\</span><span class="n">bin</span><span class="p">;</span><span
class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">Program</span> <span class="n">Files</span><span
class="o">\</span><span class="n">Microsoft</span> <span
class="n">Windows</span> <span class="n">Performance</span> <span
class="n">Toolkit</span><span class="o">\</span><span class="p">;</span>
+<span class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">apps</span><span class="o">\</span><span class="n">maven</span><span
class="o">\</span><span class="n">bin</span><span class="p">;</span><span
class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">apps</span><span class="o">\</span><span class="n">Git</span><span
class="o">\</span><span class="n">cmd</span><span class="p">;</span><span
class="n">c</span><span class="p">:</span><span class="o">\</span><span
class="n">bin</span><span class="p">;</span>
+<span class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">Program</span> <span class="n">Files</span> <span
class="p">(</span><span class="n">x86</span><span class="p">)</span><span
class="o">\</span><span class="n">MSBuild</span><span class="o">\</span>12<span
class="p">.</span>0<span class="o">\</span><span class="n">Bin</span><span
class="p">;</span>
+<span class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">Program</span> <span class="n">Files</span> <span
class="p">(</span><span class="n">x86</span><span class="p">)</span><span
class="o">\</span><span class="n">Microsoft</span> <span
class="n">Visual</span> <span class="n">Studio</span> 10<span
class="p">.</span>0<span class="o">\</span><span class="n">VC</span><span
class="o">\</span><span class="n">bin</span><span class="p">;</span>
+<span class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">bin</span><span class="o">\</span><span class="n">cmake</span><span
class="o">-</span>2<span class="p">.</span>8<span class="o">\</span><span
class="n">bin</span><span class="p">;</span><span class="n">C</span><span
class="p">:</span><span class="o">\</span><span class="n">bin</span><span
class="o">\</span><span class="n">hadoop</span><span class="o">\</span><span
class="n">bin</span><span class="p">;</span><span class="n">C</span><span
class="p">:</span><span class="o">\</span><span class="n">bin</span><span
class="o">\</span><span class="n">OpenSSL</span><span class="o">\</span><span
class="n">bin</span>
+</pre></div>
+
+
+<h3 id="commands-to-test-cli-setup">Commands to test CLI setup</h3>
+<div class="codehilite"><pre><span class="n">cmake</span> <span
class="o">--</span><span class="n">version</span>
+<span class="n">openssl</span> <span class="n">version</span>
+<span class="n">python</span> <span class="o">--</span><span
class="n">version</span>
+<span class="n">git</span> <span class="o">--</span><span
class="n">version</span>
+<span class="n">tar</span> <span class="o">--</span><span
class="n">version</span>
+
+<span class="n">java</span> <span class="o">-</span><span
class="n">version</span>
+<span class="n">mvn</span> <span class="o">--</span><span
class="n">version</span>
+
+<span class="n">msbuild</span> <span class="o">/</span><span
class="n">version</span>
+<span class="n">cl</span> <span class="o">/</span><span class="n">help</span>
+</pre></div>
+
+
+<p>and for hadoop, once installed</p>
+<p>winutils sysinfo
+hadoop version
+hadoop checknative </p>
+<h3 id="sliders-test-for-windows-support">Slider's test for windows
support</h3>
+<p>You can test for windows support with the slider-core test</p>
+<div class="codehilite"><pre><span class="n">mvn</span> <span
class="n">test</span> <span class="o">-</span><span
class="n">Dtest</span><span class="p">=</span><span
class="n">TestWindowsSupport</span>
+</pre></div>
+
+
+<p>This looks for the arguments and other valid settings</p>
+<h2 id="building-hadoop">Building Hadoop</h2>
+<ol>
+<li>Find the "Windows SDK 7.1 command prompt" using the search field in the
+windows 8 screen</li>
+<li>Run this, and "pin" it to the task bar. This will be the primary console
+for building hadoop.</li>
+<li>Right mouse button on the icon in the top-left corner to bring up the
"defaults"
+dialog.</li>
+<li>Choose whatever colors and fonts you want</li>
+<li>In layout, select a wider window size than 80 (if you want)</li>
+<li>in "screen buffer", set the no. of lines to 999.</li>
+<li>close the window and re-open it to get the new settings</li>
+</ol>
+<p>From a Windows SDK command prompt:</p>
+<div class="codehilite"><pre><span class="n">set</span> <span
class="n">HADOOP_VERSION</span><span class="p">=</span>2<span
class="p">.</span>6<span class="p">.</span>0<span class="o">-</span><span
class="n">SNAPSHOT</span>
+</pre></div>
+
+
+<p>or</p>
+<div class="codehilite"><pre><span class="n">set</span> <span
class="n">HADOOP_VERSION</span><span class="p">=</span>3<span
class="p">.</span>0<span class="p">.</span>0<span class="o">-</span><span
class="n">SNAPSHOT</span>
+</pre></div>
+
+
+<p>in the root directory of the hadoop repository</p>
+<div class="codehilite"><pre><span class="n">mvn</span> <span
class="n">clean</span>
+<span class="n">mvn</span> <span class="n">install</span> <span
class="n">package</span> <span class="o">-</span><span class="n">Pdist</span>
<span class="o">-</span><span class="n">Pnative</span><span
class="o">-</span><span class="n">win</span> <span class="o">-</span><span
class="n">Dtar</span> <span class="o">-</span><span class="n">DskipTests</span>
<span class="o">-</span><span class="n">Dmaven</span><span
class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span
class="n">skip</span><span class="p">=</span><span class="n">true</span>
+</pre></div>
+
+
+<p>(use two commands as sometimes clean can fail with some recursive
directory-in-use
+exception)</p>
+<div class="codehilite"><pre><span class="n">pushd</span> <span
class="n">hadoop</span><span class="o">-</span><span class="n">dist</span><span
class="o">\</span><span class="n">target</span>
+<span class="n">tar</span> <span class="o">-</span><span class="n">xvzf</span>
<span class="n">hadoop</span><span class="o">-</span><span
class="c">%HADOOP_VERSION%.tar.gz</span>
+<span class="n">rm</span> <span class="o">-</span><span class="n">rf</span>
<span class="n">c</span><span class="p">:</span><span class="o">\</span><span
class="n">bin</span><span class="o">\</span><span class="n">hadoop</span>
+<span class="n">cp</span> <span class="o">-</span><span class="n">r</span>
<span class="n">hadoop</span><span class="o">-</span><span
class="c">%HADOOP_VERSION% c:\bin\hadoop</span>
+<span class="n">popd</span>
+</pre></div>
+
+
+<p>Hadoop is now installed into <code>c:\bin\%HADOOP_VERSION%</code></p>
+<p>Set up the env variables by hand</p>
+<div class="codehilite"><pre> <span class="n">set</span> <span
class="n">HADOOP_HOME</span><span class="p">=</span><span
class="n">C</span><span class="p">:</span><span class="o">\</span><span
class="n">bin</span><span class="o">\</span><span class="n">hadoop</span>
+ <span class="n">set</span> <span class="n">PATH</span><span
class="p">=</span><span class="n">C</span><span class="p">:</span><span
class="o">\</span><span class="n">bin</span><span class="o">\</span><span
class="n">hadoop</span><span class="o">\</span><span class="n">bin</span><span
class="p">;</span><span class="c">%PATH%</span>
+</pre></div>
+
+
+<p>Verify the version is as expected</p>
+<div class="codehilite"><pre> <span class="n">hadoop</span> <span
class="n">version</span>
+</pre></div>
+
+
+<p>Verify that the binaries are there</p>
+<div class="codehilite"><pre> <span class="n">hadoop</span> <span
class="n">checknative</span> <span class="o">-</span><span class="n">a</span>
+</pre></div>
+
+
+<p>This should report the hadoop binaries, even if it warns about missing
+ compression libraries</p>
</div>
<div id="footer">
