Author: lewismc
Date: Wed Apr 22 16:35:25 2015
New Revision: 1675408
URL: http://svn.apache.org/r1675408
Log:
NUTCH-1996 Make protocol-selenium README part of plugin
Added:
nutch/trunk/src/plugin/protocol-selenium/README.md
Modified:
nutch/trunk/CHANGES.txt
Modified: nutch/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/nutch/trunk/CHANGES.txt?rev=1675408&r1=1675407&r2=1675408&view=diff
==============================================================================
--- nutch/trunk/CHANGES.txt (original)
+++ nutch/trunk/CHANGES.txt Wed Apr 22 16:35:25 2015
@@ -2,6 +2,8 @@ Nutch Change Log
Nutch Current Development 1.10-SNAPSHOT
+* NUTCH-1996 Make protocol-selenium README part of plugin (lewismc)
+
* NUTCH-1990 Use URI.normalise() in BasicURLNormalizer (snagel, jnioche)
* NUTCH-1973 Job Administration end point for the REST service (Sujen Shah via
mattmann)
Added: nutch/trunk/src/plugin/protocol-selenium/README.md
URL:
http://svn.apache.org/viewvc/nutch/trunk/src/plugin/protocol-selenium/README.md?rev=1675408&view=auto
==============================================================================
--- nutch/trunk/src/plugin/protocol-selenium/README.md (added)
+++ nutch/trunk/src/plugin/protocol-selenium/README.md Wed Apr 22 16:35:25 2015
@@ -0,0 +1,64 @@
+Nutch Selenium
+==============
+
+This plugin allows you to fetch Javascript pages using
[Selenium](http://www.seleniumhq.org/), while relying on the rest of the
awesome Nutch stack!
+
+The underlying code is based on the nutch-htmlunit plugin, which was in turn
based on nutch-httpclient.
+
+# IMPORTANT NOTES:
+
+ * A version of this plugin which relies on the Selenium Hub/Node system can
be found here:
[nutch-selenium-grid-plugin](https://github.com/momer/nutch-selenium-grid-plugin)
+
+# Installation (tested on Ubuntu 14.0x)
+
+## Part 1: Setting up Selenium
+
+ * Ensure that you have Firefox installed
+```
+# More info about the package @
[launchpad](https://launchpad.net/ubuntu/trusty/+source/firefox)
+
+sudo apt-get install firefox
+```
+ * Install Xvfb and its associates
+```
+sudo apt-get install xorg synaptic xvfb gtk2-engines-pixbuf xfonts-cyrillic
xfonts-100dpi \
+ xfonts-75dpi xfonts-base xfonts-scalable freeglut3-dev dbus-x11 openbox
x11-xserver-utils \
+ libxrender1 cabextract
+```
+ * Set a display for Xvfb, so that firefox believes a display is connected
+```
+sudo /usr/bin/Xvfb :11 -screen 0 1024x768x24 &
+sudo export DISPLAY=:11
+```
+## Part 2: Installing plugin for Nutch (where NUTCH_HOME is the root of your
nutch install)
+
+ * Ensure that the plugin will be used as the protocol parser in your config
+```
+<!-- NUTCH_HOME/conf/nutch-site.xml -->
+
+<configuration>
+ ...
+ <property>
+ <name>plugin.includes</name>
+
<value>protocol-selenium|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</value>
+ <description>Regular expression naming plugin directory names to
+ include. Any plugin not matching this expression is excluded.
+ In any case you need at least include the nutch-extensionpoints plugin. By
+ default Nutch includes crawling just HTML and plain text via HTTP,
+ and basic indexing and search plugins. In order to use HTTPS please enable
+ protocol-httpclient, but be aware of possible intermittent problems with
the
+ underlying commons-httpclient library.
+ </description>
+ </property>
+```
+ * Compile nutch
+```
+ant runtime
+```
+
+ * Start your web crawl (Ensure that you followed the above steps and have
started your xvfb display as shown above)
+```
+NUTCH_HOME/runtime/local/bin/crawl [-i|--index] [-D \"key=value\"] <Seed Dir>
<Crawl Dir> <Num Rounds>
+```
+
+
