This is an automated email from the ASF dual-hosted git repository.
lewismc pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nutch.git
The following commit(s) were added to refs/heads/master by this push:
new 7d7f58bbb NUTCH-3198 Yetus: pay down historical style debt (authors,
whitespace, markdown) (#979)
7d7f58bbb is described below
commit 7d7f58bbb22cbbfef9d8c5812025307c8d071dd7
Author: Lewis John McGibbney <[email protected]>
AuthorDate: Sat Oct 17 20:20:18 2026 -0700
NUTCH-3198 Yetus: pay down historical style debt (authors, whitespace,
markdown) (#979)
---
.github/pull_request_template.md | 17 +++--
.markdownlintignore | 1 +
THREAT_MODEL.md | 7 +++
docker/README.md | 8 ++-
src/plugin/exchange-jexl/README.md | 15 +++--
src/plugin/index-links/README.md | 6 +-
src/plugin/indexer-cloudsearch/README.md | 27 +++++---
src/plugin/indexer-csv/README.md | 11 +++-
src/plugin/indexer-dummy/README.md | 11 +++-
src/plugin/indexer-elastic/README.md | 12 ++--
src/plugin/indexer-elastic/howto_upgrade_es.md | 8 ++-
src/plugin/indexer-opensearch-1x/README.md | 22 ++++---
.../howto_upgrade_opensearch.md | 10 +--
src/plugin/indexer-rabbit/README.md | 18 +++---
src/plugin/indexer-solr/README.md | 17 +++--
src/plugin/indexer-solr/howto_upgrade_solr.md | 10 +--
src/plugin/lib-selenium/README.md | 6 +-
src/plugin/lib-selenium/howto_upgrade_selenium.md | 16 +++--
src/plugin/protocol-interactiveselenium/README.md | 15 +++--
src/plugin/protocol-okhttp/howto_upgrade_okhttp.md | 4 +-
src/plugin/protocol-selenium/README.md | 73 ++++++++++++----------
src/plugin/urlfilter-fast/README.md | 4 +-
src/plugin/urlfilter-ignoreexempt/README.md | 9 +--
23 files changed, 209 insertions(+), 118 deletions(-)
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index 5172e9bde..2fd233e34 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -1,17 +1,22 @@
+# Pull request checklist
+
Thanks for your contribution to [Apache Nutch](https://nutch.apache.org/)!
Your help is appreciated!
Before opening the pull request, please verify that
-* there is an open issue on the [Nutch issue
tracker](https://issues.apache.org/jira/projects/NUTCH) which describes the
problem or the improvement. We cannot accept pull requests without an issue
because the change wouldn't be listed in the release notes.
+
+* there is an open issue on the [Nutch issue
tracker](https://issues.apache.org/jira/projects/NUTCH) which describes the
problem or the improvement.
+ We cannot accept pull requests without an issue because the change wouldn't
be listed in the release notes.
* the issue ID (`NUTCH-XXXX`)
- - is referenced in the title of the pull request
- - and placed in front of your commit messages surrounded by square brackets
(`[NUTCH-XXXX] Issue or pull request title`)
+ * is referenced in the title of the pull request
+ * and placed in front of your commit messages surrounded by square brackets
(`[NUTCH-XXXX] Issue or pull request title`)
* commits are squashed into a single one (or few commits for larger changes)
* Java source code follows [Nutch Eclipse Code Formatting
rules](https://github.com/apache/nutch/blob/master/eclipse-codeformat.xml)
* Nutch is successfully built and unit tests pass by running `ant clean
runtime test`
-* there should be no conflicts when merging the pull request branch into the
*recent* master branch. If there are conflicts, please try to rebase the pull
request branch on top of a freshly pulled master branch.
+* there should be no conflicts when merging the pull request branch into the
*recent* master branch.
+ If there are conflicts, please try to rebase the pull request branch on top
of a freshly pulled master branch.
* if new dependencies are added,
- - are these dependencies licensed in a way that is compatible for inclusion
under [ASF 2.0](https://www.apache.org/legal/resolved.html#category-a)?
- - are `LICENSE-binary` and `NOTICE-binary` updated accordingly?
+ * are these dependencies licensed in a way that is compatible for inclusion
under [ASF 2.0](https://www.apache.org/legal/resolved.html#category-a)?
+ * are `LICENSE-binary` and `NOTICE-binary` updated accordingly?
We will be able to faster integrate your pull request if these
conditions are met. If you have any questions how to fix your problem
diff --git a/.markdownlintignore b/.markdownlintignore
new file mode 100644
index 000000000..9fc9e21de
--- /dev/null
+++ b/.markdownlintignore
@@ -0,0 +1 @@
+CHANGES.md
diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md
index d905b806a..538cdb059 100644
--- a/THREAT_MODEL.md
+++ b/THREAT_MODEL.md
@@ -100,6 +100,7 @@ not about preventing Nutch from reaching or parsing hostile
content.
## §4 Trust boundaries and data flow
Two boundaries matter, and they are different from a typical service:
+
1. **The fetch boundary (primary).** Everything Nutch fetches from the web is
untrusted and crosses into the parser/store. The trust transition is
"bytes-from-the-internet → parsed structures." *(inferred)*
@@ -110,6 +111,7 @@ Data flow: seed URLs (operator) → fetch (untrusted content)
→ parse (Tika/pl
on untrusted bytes) → URL extraction + filtering → crawl DB → index to backend.
**Reachability preconditions per family:**
+
- A fetcher/parser finding is in-model when it is reachable **from
attacker-controlled fetched content** (a hostile page/feed/redirect) — that
is
the core in-model surface.
@@ -162,6 +164,7 @@ needs a §8 line *(inferred — §14)*.
## §7 Adversary model
**In scope** *(inferred — §14)*:
+
- **Malicious web-content supplier** — anyone who controls a page Nutch
fetches:
serves hostile HTML/XML/feeds/redirects/headers/`robots.txt` to crash the
parser, exhaust resources, exfiltrate via the parser (XXE), or poison the
crawl
@@ -173,6 +176,7 @@ needs a §8 line *(inferred — §14)*.
cannot (assumed) reach the operator's config, host, or backends.
**Explicitly out of scope:**
+
- The **operator** and anyone who controls seeds/filters/plugins/host — already
trusted.
- Anyone reaching a `nutch-server` or crawl store that the operator exposed
@@ -210,6 +214,7 @@ enumerate committed properties.)*
- **No protection for deployments outside a trusted environment.**
*(documented)*
**False-friend properties:**
+
- **URL filters are a crawl-scoping tool, not a security boundary against a
hostile operator or a guarantee against SSRF to internal services** — they
reduce reachable URLs but are operator-configured and regex-based.
*(inferred)*
@@ -217,6 +222,7 @@ enumerate committed properties.)*
the operator's scope allowed it. *(inferred)*
**Well-known attack classes left to the operator / inherent to crawling:**
+
- **XXE** in XML/feed parsing of hostile content (Tika/parsers) — a real
in-model concern (parser must be hardened).
- **ReDoS** in URL-filter / normalizer regex fed URLs from hostile pages.
@@ -295,6 +301,7 @@ Grouped in waves; each states a **proposed answer** to
confirm/correct/strike.
Every *(inferred)* tag above maps to one of these.
**Wave 1 — scope & adversary (these shape everything):**
+
1. **Trusted-environment posture / `nutch-server`.** Proposed: the supported
posture is "trusted environment only"; an exposed no-auth `nutch-server` is
`OUT-OF-MODEL: non-default-build`. Correct? (→ §5a, §3, §11a)
diff --git a/docker/README.md b/docker/README.md
index 1c3f57df6..899269d94 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -24,7 +24,7 @@ Current configuration of this image consists of components:
* Default runtime user `nutch` (not root)
* `NUTCH_HOME=/opt/nutch/runtime/local` (sources under `/opt/nutch`)
-## Base Image
+## Base Image
* [alpine:3.19](https://hub.docker.com/_/alpine/tags)
@@ -32,7 +32,7 @@ Current configuration of this image consists of components:
You may need to alias docker to "docker --tls" if you see errors such as:
-```
+```text
2015/04/07 09:19:56 Post
http://192.168.59.103:2376/v1.14/containers/create?name=NutchContainer:
malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"
```
@@ -65,11 +65,13 @@ plugins, and keep the crawl process on the image default
user `nutch` so the
filesystem cannot be abused to drop unexpected JARs or `plugin.xml` files into
that tree. Use `docker run --user root ...` only for privileged debugging.
-User-defined JEXL in configuration (for example `index.jexl.filter`, generator
expressions, and `hostdb.filter.expression`) is evaluated in a **sandboxed**
engine by default. The property `nutch.jexl.disable.sandbox` disables that
protection and must not be set in untrusted environments.
+User-defined JEXL in configuration (for example `index.jexl.filter`, generator
expressions, and `hostdb.filter.expression`) is evaluated in a **sandboxed**
engine by default.
+The property `nutch.jexl.disable.sandbox` disables that protection and must
not be set in untrusted environments.
## Usage
If not already running, start docker
+
```bash
boot2docker up
$(boot2docker shellinit | grep export)
diff --git a/src/plugin/exchange-jexl/README.md
b/src/plugin/exchange-jexl/README.md
index 6d6ad51c1..5049e4c42 100644
--- a/src/plugin/exchange-jexl/README.md
+++ b/src/plugin/exchange-jexl/README.md
@@ -40,9 +40,12 @@ The **exchange-jexl plugin** must be configured in the
exchanges.xml file, inclu
Each `<exchange>` element has two mandatory attributes:
-* `<exchange_id>` is a unique identification for each configuration. It is
used by Nutch to distinguish each one, even when they are for the same exchange
implementation and this ID allows to have multiple instances for the same
exchange, but with different configurations.
+* `<exchange_id>` is a unique identification for each configuration.
+ It is used by Nutch to distinguish each one, even when they are for the same
exchange implementation and this ID allows to have multiple instances for the
same exchange,
+ but with different configurations.
-* `org.apache.nutch.exchange.jexl.JexlExchange` corresponds to the canonical
name of the class that implements the Exchange extension point. This value must
not be modified for the **exchange-jexl plugin**.
+* `org.apache.nutch.exchange.jexl.JexlExchange` corresponds to the canonical
name of the class that implements the Exchange extension point.
+ This value must not be modified for the **exchange-jexl plugin**.
## Writers section
@@ -52,7 +55,9 @@ The `<writers>` element is independent for each configuration
and contains a lis
The `<params>` element is where the parameters that the exchange needs are
specified. Each parameter has the form `<param name="<name>" value="<value>"/>`.
-The unique parameter needed by the **exchange-jexl plugin** has the `<name>`
**expr** and the `<value>` is a JEXL expression used to validate each document.
The variable **doc** can be used on the expressions and represents the document
itself. For example, the expression `doc.getFieldValue('host')=='example.org'`
will match the documents where the **host** field has the value **example.org**.
+The unique parameter needed by the **exchange-jexl plugin** has the `<name>`
**expr** and the `<value>` is a JEXL expression used to validate each document.
+The variable **doc** can be used on the expressions and represents the
document itself.
+For example, the expression `doc.getFieldValue('host')=='example.org'` will
match the documents where the **host** field has the value **example.org**.
## Use case 1
@@ -78,4 +83,6 @@ The unique parameter needed by the **exchange-jexl plugin**
has the `<name>` **e
</exchanges>
```
-According to this example, the documents which the value of **host** field is
**example.org** will be sent to **indexer_solr_1** and **indexer_rabbit_1**.
The rest of documents where **host** is different to **example.org** do not
match with **exchange_jexl_1** exchange and will be sent where the default
exchange says; in this case to **indexer_dummy_1**.
\ No newline at end of file
+According to this example, the documents which the value of **host** field is
**example.org** will be sent to **indexer_solr_1** and **indexer_rabbit_1**.
+The rest of documents where **host** is different to **example.org** do not
match with **exchange_jexl_1** exchange and will be sent where the default
exchange says;
+in this case to **indexer_dummy_1**.
diff --git a/src/plugin/index-links/README.md b/src/plugin/index-links/README.md
index ac0f071f4..094e07bef 100644
--- a/src/plugin/index-links/README.md
+++ b/src/plugin/index-links/README.md
@@ -57,14 +57,14 @@ examples of default values for the Solr schema.
* The `inlink`/`outlink` fields have to be multivalued, because normally a
given URL will have multiple inlinks and outlinks.
-```
+```xml
<fieldType name="string" class="solr.StrField" sortMissingLast="true"
omitNorms="true"/>
```
The field configuration could look like:
-```
+```xml
<field name="inlinks" type="multiValuedString" stored="true" indexed="true"
multiValued="true"/>
<field name="outlinks" type="multiValuedString" stored="true" indexed="true"
multiValued="true"/>
-```
\ No newline at end of file
+```
diff --git a/src/plugin/indexer-cloudsearch/README.md
b/src/plugin/indexer-cloudsearch/README.md
index 9865b13eb..9571416f8 100644
--- a/src/plugin/indexer-cloudsearch/README.md
+++ b/src/plugin/indexer-cloudsearch/README.md
@@ -20,7 +20,8 @@ AWS CloudSearch plugin for Nutch
See [http://aws.amazon.com/cloudsearch/] for information on AWS CloudSearch.
-**indexer-cloudsearch plugin** is used for sending documents from one or more
segments to Amazon CloudSearch. The configuration for the index writers is on
**conf/index-writers.xml** file, included in the official Nutch distribution
and it's as follow:
+**indexer-cloudsearch plugin** is used for sending documents from one or more
segments to Amazon CloudSearch.
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.cloudsearch.CloudSearchIndexWriter">
@@ -35,9 +36,12 @@ See [http://aws.amazon.com/cloudsearch/] for information on
AWS CloudSearch.
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.cloudsearch.CloudSearchIndexWriter`
corresponds to the canonical name of the class that implements the IndexWriter
extension point. This value should not be modified for the
**indexer-cloudsearch plugin**.
+* `org.apache.nutch.indexwriter.cloudsearch.CloudSearchIndexWriter`
corresponds to the canonical name of the class that implements the IndexWriter
extension point.
+ This value should not be modified for the **indexer-cloudsearch plugin**.
## Mapping
@@ -49,8 +53,8 @@ Each parameter has the form `<param name="<name>"
value="<value>"/>` and the par
Parameter Name | Description | Default value
--|--|--
-endpoint | Endpoint where service requests should be submitted. |
-region | Region name. |
+endpoint | Endpoint where service requests should be submitted. | <!-- none -->
+region | Region name. | <!-- none -->
batch.dump | **true** to store the JSON representation of the documents to a
local temp dir. The files has the prefix "CloudSearch_" e.g.
`/tmp/CloudSearch_4822180575734804454.json`. This temp file can be used as a
template when defining the fields in the domain creation. | false
batch.maxSize | Maximum number of documents to send as a batch to CloudSearch.
| -1
@@ -58,13 +62,18 @@ batch.maxSize | Maximum number of documents to send as a
batch to CloudSearch. |
This can be done using the web console
[https://eu-west-1.console.aws.amazon.com/cloudsearch/home?region=eu-west-1#].
You can use the temp file generated above to bootstrap the field definition.
-You can also create the domain using the AWS CLI
[http://docs.aws.amazon.com/cloudsearch/latest/developerguide/creating-domains.html]
and the `createCSDomain.sh` example script provided. This script is merely as
starting point which you should further improve and fine tune.
+You can also create the domain using the AWS CLI
[http://docs.aws.amazon.com/cloudsearch/latest/developerguide/creating-domains.html]
and the `createCSDomain.sh` example script provided.
+This script is merely as starting point which you should further improve and
fine tune.
Note that the creation of the domain can take some time. Once it is complete,
note the document endpoint, or alternatively verify the region and domain name.
-> The CloudSearchIndexWriter will log any errors while sending the batches to
CloudSearch and will resume the process without breaking. This means that you
might not get all the documents in the index. You should check the log files
for errors. Using small batch sizes will limit the number of documents skipped
in case of error.
-
-> Any fields not defined in the CloudSearch domain will be ignored by the
CloudSearchIndexWriter. Again, the logs will contain a trace of any field names
skipped.
+> The CloudSearchIndexWriter will log any errors while sending the batches to
CloudSearch and will resume the process without breaking.
+> This means that you might not get all the documents in the index.
+> You should check the log files for errors.
+> Using small batch sizes will limit the number of documents skipped in case
of error.
+>
+> Any fields not defined in the CloudSearch domain will be ignored by the
CloudSearchIndexWriter.
+> Again, the logs will contain a trace of any field names skipped.
diff --git a/src/plugin/indexer-csv/README.md b/src/plugin/indexer-csv/README.md
index 612bc015d..3fe85188e 100644
--- a/src/plugin/indexer-csv/README.md
+++ b/src/plugin/indexer-csv/README.md
@@ -18,7 +18,9 @@
indexer-csv plugin for Nutch
============================
-**indexer-csv plugin** is used for writing documents to a CSV file. It does
not work in distributed mode, the output is written to the local filesystem,
not to HDFS, see
[NUTCH-1541](https://issues.apache.org/jira/browse/NUTCH-1541). The
configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
+**indexer-csv plugin** is used for writing documents to a CSV file.
+It does not work in distributed mode, the output is written to the local
filesystem, not to HDFS, see
[NUTCH-1541](https://issues.apache.org/jira/browse/NUTCH-1541).
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.csv.CSVIndexWriter">
@@ -33,9 +35,12 @@ indexer-csv plugin for Nutch
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.csv.CSVIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
This value should not be modified for the **indexer-csv plugin**.
+* `org.apache.nutch.indexwriter.csv.CSVIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
+ This value should not be modified for the **indexer-csv plugin**.
## Mapping
diff --git a/src/plugin/indexer-dummy/README.md
b/src/plugin/indexer-dummy/README.md
index a35880588..f8840a7a9 100644
--- a/src/plugin/indexer-dummy/README.md
+++ b/src/plugin/indexer-dummy/README.md
@@ -18,7 +18,9 @@
indexer-dummy plugin for Nutch
==============================
-**indexer-dummy plugin** is used for writing "action"\t"url"\n lines to a
plain text file for debugging purposes. It does not work in distributed mode,
the output is written to the local filesystem, not to HDFS. The configuration
for the index writers is on **conf/index-writers.xml** file, included in the
official Nutch distribution and it's as follow:
+**indexer-dummy plugin** is used for writing "action"\t"url"\n lines to a
plain text file for debugging purposes.
+It does not work in distributed mode, the output is written to the local
filesystem, not to HDFS.
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.dummy.DummyIndexWriter">
@@ -33,9 +35,12 @@ indexer-dummy plugin for Nutch
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.dummy.DummyIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
This value should not be modified for the **indexer-dummy plugin**.
+* `org.apache.nutch.indexwriter.dummy.DummyIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
+ This value should not be modified for the **indexer-dummy plugin**.
## Mapping
diff --git a/src/plugin/indexer-elastic/README.md
b/src/plugin/indexer-elastic/README.md
index bf7a5e736..c6c871bb9 100644
--- a/src/plugin/indexer-elastic/README.md
+++ b/src/plugin/indexer-elastic/README.md
@@ -18,7 +18,8 @@
indexer-elastic plugin for Nutch
================================
-**indexer-elastic plugin** is used for sending documents from one or more
segments to an Elasticsearch server. The configuration for the index writers is
on **conf/index-writers.xml** file, included in the official Nutch distribution
and it's as follow:
+**indexer-elastic plugin** is used for sending documents from one or more
segments to an Elasticsearch server.
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.elastic.ElasticIndexWriter">
@@ -33,9 +34,12 @@ indexer-elastic plugin for Nutch
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.elastic.ElasticIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
This value should not be modified for the **indexer-elastic plugin**.
+* `org.apache.nutch.indexwriter.elastic.ElasticIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
+ This value should not be modified for the **indexer-elastic plugin**.
## Mapping
@@ -47,7 +51,7 @@ Each parameter has the form `<param name="<name>"
value="<value>"/>` and the par
Parameter Name | Description | Default value
--|--|--
-host | Comma-separated list of hostnames to send documents to using
[TransportClient](https://static.javadoc.io/org.elasticsearch/elasticsearch/5.3.0/org/elasticsearch/client/transport/TransportClient.html).
Either host and port must be defined. |
+host | Comma-separated list of hostnames to send documents to using
[TransportClient](https://static.javadoc.io/org.elasticsearch/elasticsearch/5.3.0/org/elasticsearch/client/transport/TransportClient.html).
Either host and port must be defined. | <!-- none -->
port | The port to connect to using
[TransportClient](https://static.javadoc.io/org.elasticsearch/elasticsearch/5.3.0/org/elasticsearch/client/transport/TransportClient.html).
| 9300
scheme | The scheme (http or https) to connect to elastic server. | http
index | Default index to send documents to. | nutch
diff --git a/src/plugin/indexer-elastic/howto_upgrade_es.md
b/src/plugin/indexer-elastic/howto_upgrade_es.md
index ca58639d1..7254e2a07 100644
--- a/src/plugin/indexer-elastic/howto_upgrade_es.md
+++ b/src/plugin/indexer-elastic/howto_upgrade_es.md
@@ -15,6 +15,8 @@
limitations under the License.
-->
+# How to Upgrade Elasticsearch in Nutch
+
1. Upgrade Elasticsearch dependency in src/plugin/indexer-elastic/ivy.xml
2. Upgrade the Elasticsearch specific dependencies in
src/plugin/indexer-elastic/plugin.xml
@@ -29,7 +31,7 @@
<!-- end of Elastic Rest Client dependencies -->
with the output of the command above.
-4. (Optionally) remove overlapping dependencies between indexer-elastic and
Nutch core dependencies:
+3. (Optionally) remove overlapping dependencies between indexer-elastic and
Nutch core dependencies:
- check for libs present both in
build/lib
and
@@ -40,11 +42,11 @@
- but it should be made sure that the library versions in ivy/ivy.xml
correspond to
those required by Tika
-5. Remove the locally "installed" dependencies in
src/plugin/indexer-elastic/lib/:
+4. Remove the locally "installed" dependencies in
src/plugin/indexer-elastic/lib/:
$ rm -rf lib/
-6. Build Nutch and run all unit tests:
+5. Build Nutch and run all unit tests:
$ cd ../../../
$ ant clean runtime test
diff --git a/src/plugin/indexer-opensearch-1x/README.md
b/src/plugin/indexer-opensearch-1x/README.md
index a04d68625..d9468daa9 100644
--- a/src/plugin/indexer-opensearch-1x/README.md
+++ b/src/plugin/indexer-opensearch-1x/README.md
@@ -18,7 +18,8 @@
indexer-opensearch1x plugin for Nutch
================================
-**indexer-opensearch1x plugin** is used for sending documents from one or more
segments to an OpenSearch server. The configuration for the index writers is on
**conf/index-writers.xml** file, included in the official Nutch distribution
and it's as follow:
+**indexer-opensearch1x plugin** is used for sending documents from one or more
segments to an OpenSearch server.
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.opensearch1x.OpenSearch1xIndexWriter">
@@ -33,9 +34,12 @@ indexer-opensearch1x plugin for Nutch
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.opensearch1x.OpenSearch1x.IndexWriter`
corresponds to the canonical name of the class that implements the IndexWriter
extension point. This value should not be modified for the
**indexer-opensearch1x plugin**.
+* `org.apache.nutch.indexwriter.opensearch1x.OpenSearch1x.IndexWriter`
corresponds to the canonical name of the class that implements the IndexWriter
extension point.
+ This value should not be modified for the **indexer-opensearch1x plugin**.
## Mapping
@@ -47,17 +51,17 @@ Each parameter has the form `<param name="<name>"
value="<value>"/>` and the par
Parameter Name | Description | Default value
--|--|--
-host | Comma-separated list of hostnames to send documents to using
[TransportClient](https://static.javadoc.io/org.opensearch/opensearch/1.3.8/org/opensearch/client/transport/TransportClient.html).
Either host and port must be defined. |
+host | Comma-separated list of hostnames to send documents to using
[TransportClient](https://static.javadoc.io/org.opensearch/opensearch/1.3.8/org/opensearch/client/transport/TransportClient.html).
Either host and port must be defined. | <!-- none -->
port | The port to connect to using
[TransportClient](https://static.javadoc.io/org.opensearch/opensearch/1.3.8/org/opensearch/client/transport/TransportClient.html).
| 9300
scheme | The scheme (http or https) to connect to OpenSearch server. | https
index | Default index to send documents to. | nutch
username | Username for auth credentials | admin
password | Password for auth credentials | admin
-trust.store.path | Path to the trust store |
-trust.store.password | Password for trust store |
+trust.store.path | Path to the trust store | <!-- none -->
+trust.store.password | Password for trust store | <!-- none -->
trust.store.type | Type of trust store | JKS
-key.store.path | Path to the key store |
-key.store.password | Password for the key and the key store |
+key.store.path | Path to the key store | <!-- none -->
+key.store.password | Password for the key and the key store | <!-- none -->
key.store.type | Type of key store | JKS
max.bulk.docs | Maximum size of the bulk in number of documents. | 250
max.bulk.size | Maximum size of the bulk in bytes. | 2500500
@@ -80,4 +84,4 @@ If users do not specify at least 1-way TLS (trust-store),
this indexer logs a wa
This index writer was built to be as close as possible to Nutch's existing
indexer-elastic code. We
therefore chose to use the to-be-deprecated-in-3.x
`opensearch-rest-high-level-client`.
We should plan to migrate to the `java client` for 2.x, whenever the
BulkProcessor has been added.
-See the discussion on
[NUTCH-2920](https://issues.apache.org/jira/projects/NUTCH/issues/NUTCH-2920).
\ No newline at end of file
+See the discussion on
[NUTCH-2920](https://issues.apache.org/jira/projects/NUTCH/issues/NUTCH-2920).
diff --git a/src/plugin/indexer-opensearch-1x/howto_upgrade_opensearch.md
b/src/plugin/indexer-opensearch-1x/howto_upgrade_opensearch.md
index 4bf4c5c60..2750fa9a9 100644
--- a/src/plugin/indexer-opensearch-1x/howto_upgrade_opensearch.md
+++ b/src/plugin/indexer-opensearch-1x/howto_upgrade_opensearch.md
@@ -15,6 +15,8 @@
limitations under the License.
-->
+# How to Upgrade OpenSearch in Nutch
+
1. Upgrade OpenSearch dependency in src/plugin/indexer-opensearch-1x/ivy.xml
2. Upgrade the OpenSearch specific dependencies in
src/plugin/indexer-opensearch-1x/plugin.xml
@@ -29,7 +31,7 @@
<!-- end of OpenSearch Rest Client dependencies -->
with the output of the command above.
-4. (Optionally) remove overlapping dependencies between indexer-opensearch-1x
and Nutch core dependencies:
+3. (Optionally) remove overlapping dependencies between indexer-opensearch-1x
and Nutch core dependencies:
- check for libs present both in
build/lib
and
@@ -40,11 +42,11 @@
- but it should be made sure that the library versions in ivy/ivy.xml
correspond to
those required by Tika
-5. Remove the locally "installed" dependencies in
src/plugin/indexer-opensearch-1x/lib/:
+4. Remove the locally "installed" dependencies in
src/plugin/indexer-opensearch-1x/lib/:
$ rm -rf lib/
-6. Build Nutch and run all unit tests:
+5. Build Nutch and run all unit tests:
$ cd ../../../
- $ ant clean runtime test
\ No newline at end of file
+ $ ant clean runtime test
diff --git a/src/plugin/indexer-rabbit/README.md
b/src/plugin/indexer-rabbit/README.md
index deef6b564..954d5f7b2 100644
--- a/src/plugin/indexer-rabbit/README.md
+++ b/src/plugin/indexer-rabbit/README.md
@@ -18,7 +18,8 @@
indexer-rabbit plugin for Nutch
===============================
-**indexer-rabbit plugin** is used for sending documents from one or more
segments to a RabbitMQ server. The configuration for the index writers is on
**conf/index-writers.xml** file, included in the official Nutch distribution
and it's as follow:
+**indexer-rabbit plugin** is used for sending documents from one or more
segments to a RabbitMQ server.
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.rabbit.RabbitIndexWriter">
@@ -33,9 +34,12 @@ indexer-rabbit plugin for Nutch
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.rabbit.RabbitIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
This value should not be modified for the **indexer-rabbit plugin**.
+* `org.apache.nutch.indexwriter.rabbit.RabbitIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
+ This value should not be modified for the **indexer-rabbit plugin**.
## Mapping
@@ -49,13 +53,13 @@ Parameter Name | Description | Default value
--|--|--
server.uri | URI with connection parameters in the form
`amqp://<hostname>:<port>/<virtualHost>` or
`amqp://<username>:<password>@<hostname>:<port>/<virtualHost>`<br>Where:<ul><li>`<username>`
is the username for RabbitMQ server.</li><li>`<password>` is the password for
RabbitMQ server.</li><li>`<hostname>` is where the RabbitMQ server is
running.</li><li>`<port>` is where the RabbitMQ server is
listening.</li><li>`<virtualHost>` is where the exchange is and the user has
access.</li></ul [...]
binding | Whether the relationship between an exchange and a queue is created
automatically.<br>**NOTE:** Binding between exchanges is not supported. | false
-binding.arguments | Arguments used in binding. It must have the form
`key1=value1,key2=value2`. This value is only used when the exchange's type is
headers and the value of binding property is **true**. In other cases is
ignored. |
-exchange.name | Name for the exchange where the messages will be sent. |
+binding.arguments | Arguments used in binding. It must have the form
`key1=value1,key2=value2`. This value is only used when the exchange's type is
headers and the value of binding property is **true**. In other cases is
ignored. | <!-- none -->
+exchange.name | Name for the exchange where the messages will be sent. | <!--
none -->
exchange.options | Options used when the exchange is created. Only used when
the value of `binding` property is **true**. It must have the form
`type=<type>,durable=<durable>`<br>Where:<ul><li>`<type>` is **direct**,
**topic**, **headers** or **fanout**</li><li>`<durable>` is **true** or
**false** | type=direct,durable=true</li></ul>
queue.name | Name of the queue used to create the binding. Only used when the
value of `binding` property is **true**. | nutch.queue
queue.options | Options used when the queue is created. Only used when the
value of `binding` property is **true**. It must have the form
`durable=<durable>,exclusive=<exclusive>,auto-delete=<auto-delete>,arguments=<arguments>`<br>Where:<ul><li>`<durable>`
is **true** or **false**</li><li>`<exclusive>` is **true** or
**false**</li><li>`<auto-delete>` is **true** or
**false**</li><li>`<arguments>` must be the form `key1:value1;key2:value2` |
durable=true,exclusive=false,auto-delete=false [...]
routingkey | The routing key used to route messages in the exchange. It only
makes sense when the exchange type is **topic** or **direct**. | Value of
`queue.name` property
commit.mode | **single** if a message contains only one document. In this
case, a header with the action (write, update or delete) will be added.
**multiple** if a message contains all documents. | multiple
commit.size | Amount of documents to send into each message if the value of
`commit.mode` property is **multiple**. In **single** mode this value
represents the amount of messages to be sent. | 250
-headers.static | Headers to add to each message. It must have the form
`key1=value1,key2=value2`. |
-headers.dynamic | Document's fields to add as headers to each message. It must
have the form `field1,field2`. Only used when the value of `commit.mode`
property is **single**. |
+headers.static | Headers to add to each message. It must have the form
`key1=value1,key2=value2`. | <!-- none -->
+headers.dynamic | Document's fields to add as headers to each message. It must
have the form `field1,field2`. Only used when the value of `commit.mode`
property is **single**. | <!-- none -->
diff --git a/src/plugin/indexer-solr/README.md
b/src/plugin/indexer-solr/README.md
index 9b8560f8c..cae6d7033 100644
--- a/src/plugin/indexer-solr/README.md
+++ b/src/plugin/indexer-solr/README.md
@@ -18,7 +18,8 @@
indexer-solr plugin for Nutch
=============================
-**indexer-solr plugin** is used for sending documents from one or more
segments to a Solr server. The configuration for the index writers is on
**conf/index-writers.xml** file, included in the official Nutch distribution
and it's as follow:
+**indexer-solr plugin** is used for sending documents from one or more
segments to a Solr server.
+The configuration for the index writers is on **conf/index-writers.xml** file,
included in the official Nutch distribution and it's as follow:
```xml
<writer id="<writer_id>"
class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
@@ -33,9 +34,12 @@ indexer-solr plugin for Nutch
Each `<writer>` element has two mandatory attributes:
-* `<writer_id>` is a unique identification for each configuration. This
feature allows Nutch to distinguish each configuration, even when they are for
the same index writer. In addition, it allows to have multiple instances for
the same index writer, but with different configurations.
+* `<writer_id>` is a unique identification for each configuration.
+ This feature allows Nutch to distinguish each configuration, even when they
are for the same index writer.
+ In addition, it allows to have multiple instances for the same index writer,
but with different configurations.
-* `org.apache.nutch.indexwriter.solr.SolrIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
This value should not be modified for the **indexer-solr plugin**.
+* `org.apache.nutch.indexwriter.solr.SolrIndexWriter` corresponds to the
canonical name of the class that implements the IndexWriter extension point.
+ This value should not be modified for the **indexer-solr plugin**.
## Mapping
@@ -49,8 +53,8 @@ Parameter Name | Description | Default value
--|--|--
type | Specifies the
[SolrClient](https://lucene.apache.org/solr/5_5_0/solr-solrj/org/apache/solr/client/solrj/SolrClient.html)
implementation to use. This is a string value of one of the following
**cloud** or **http**. The values represent
[CloudSolrServer](https://lucene.apache.org/solr/5_5_0/solr-solrj/org/apache/solr/client/solrj/impl/CloudSolrServer.html)
or
[HttpSolrServer](https://lucene.apache.org/solr/5_5_0/solr-solrj/org/apache/solr/client/solrj/impl/HttpSolrServer.html)
respe [...]
url | Defines the fully qualified URL of Solr into which data should be
indexed. Multiple URL can be provided using comma as a delimiter. When the
value of type property is **cloud**, the URL should not include any collections
or cores; just the root Solr path. | http://localhost:8983/solr/nutch
-collection | The collection used in requests. Only used when the value of type
property is **cloud**. |
-weight.field | Field's name where the weight of the documents will be written.
If it is empty no field will be used. |
+collection | The collection used in requests. Only used when the value of type
property is **cloud**. | <!-- none -->
+weight.field | Field's name where the weight of the documents will be written.
If it is empty no field will be used. | <!-- none -->
commitSize | Defines the number of documents to send to Solr in a single
update batch. Decrease when handling very large documents to prevent Nutch from
running out of memory.<br>**Note**: It does not explicitly trigger a server
side commit. | 1000
auth | Whether to enable HTTP basic authentication for communicating with
Solr. Use the `username` and `password` properties to configure your
credentials. | false
username | The username of Solr server. | username
@@ -58,4 +62,5 @@ password | The password of Solr server. | password
## schema.xml
-In the distribution of the indexer-solr plugin there is a schema.xml file
available. Nutch does not use this file, but it is provided to Solr users as a
reference/guide to facilitate the configuration of Solr.
+In the distribution of the indexer-solr plugin there is a schema.xml file
available.
+Nutch does not use this file, but it is provided to Solr users as a
reference/guide to facilitate the configuration of Solr.
diff --git a/src/plugin/indexer-solr/howto_upgrade_solr.md
b/src/plugin/indexer-solr/howto_upgrade_solr.md
index c7bdba5f3..3aed435c9 100644
--- a/src/plugin/indexer-solr/howto_upgrade_solr.md
+++ b/src/plugin/indexer-solr/howto_upgrade_solr.md
@@ -15,6 +15,8 @@
limitations under the License.
-->
+# How to Upgrade Solr in Nutch
+
1. Upgrade Solr dependency in src/plugin/indexer-solr/ivy.xml
2. Upgrade the Solr specific dependencies in src/plugin/indexer-solr/plugin.xml
@@ -29,7 +31,7 @@
<!-- end of Solr dependencies -->
with the output of the command above.
-4. (Optionally) remove overlapping dependencies between indexer-solr and Nutch
core dependencies:
+3. (Optionally) remove overlapping dependencies between indexer-solr and Nutch
core dependencies:
- check for libs present both in
build/lib
and
@@ -40,11 +42,11 @@
- but it should be made sure that the library versions in ivy/ivy.xml
correspond to
those required by Tika
-5. Remove the locally "installed" dependencies in src/plugin/indexer-solr/lib/:
+4. Remove the locally "installed" dependencies in src/plugin/indexer-solr/lib/:
$ rm -rf lib/
-6. Build Nutch and run all unit tests:
+5. Build Nutch and run all unit tests:
$ cd ../../../
- $ ant clean runtime test
\ No newline at end of file
+ $ ant clean runtime test
diff --git a/src/plugin/lib-selenium/README.md
b/src/plugin/lib-selenium/README.md
index a3f519950..0aefa1998 100644
--- a/src/plugin/lib-selenium/README.md
+++ b/src/plugin/lib-selenium/README.md
@@ -16,9 +16,13 @@
-->
# Updates
+
* The use of phantomjs has been deprecated. Check
[Wikipedia](https://en.wikipedia.org/wiki/PhantomJS) for more info.
* The updated code for Safari webriver is under development as starting Safari
10 on OS X El Capitan and macOS Sierra, Safari comes bundled with a new driver
implementation.
-* Opera is now based on ChromeDriver and has been adapted by Opera that
enables programmatic automation of Chromium-based Opera products but hasn't
been updated since April 5, 2017. We have suspended its support and removed
from the code.([link](https://github.com/operasoftware/operachromiumdriver))
+* Opera is now based on ChromeDriver and has been adapted by Opera that
+ enables programmatic automation of Chromium-based Opera products but hasn't
+ been updated since April 5, 2017. We have suspended its support and removed
+ from the code.([link](https://github.com/operasoftware/operachromiumdriver))
* Headless mode has been added for Chrome and Firefox. Set
`selenium.enable.headless` to `true` in nutch-default.xml or nutch-site.xml to
use it.
diff --git a/src/plugin/lib-selenium/howto_upgrade_selenium.md
b/src/plugin/lib-selenium/howto_upgrade_selenium.md
index a14a346b1..b59fd4100 100644
--- a/src/plugin/lib-selenium/howto_upgrade_selenium.md
+++ b/src/plugin/lib-selenium/howto_upgrade_selenium.md
@@ -15,15 +15,19 @@
limitations under the License.
-->
+# How to upgrade Selenium
+
1. Upgrade various driver versions dependency in
`src/plugin/lib-selenium/ivy.xml`
2. Upgrade Selenium's own dependencies in `src/plugin/lib-selenium/plugin.xml`
To get a list of dependencies and their versions execute:
+
+ ```bash
+ ant -f ./build-ivy.xml
+ ls lib | sed 's/^/ <library name="/g' | sed 's/$/">\n <export
name="*"\/>\n <\/library>/g'
```
- $ ant -f ./build-ivy.xml
- $ ls lib | sed 's/^/ <library name="/g' | sed 's/$/">\n <export
name="*"\/>\n <\/library>/g'
- ```
+
Note that all dependent libraries are exported for a "library" plugin
`lib-selenium`.
N.B. The above Regex + Sed commands may not work if you are using MacOSX's
Sed. In this instance you can install GNU Sed as follows
@@ -44,7 +48,7 @@
5. Build Nutch and run all unit tests:
- ```
- $ cd ../../../
- $ ant clean runtime test
+ ```bash
+ cd ../../../
+ ant clean runtime test
```
diff --git a/src/plugin/protocol-interactiveselenium/README.md
b/src/plugin/protocol-interactiveselenium/README.md
index b94b7466d..a60b1b158 100644
--- a/src/plugin/protocol-interactiveselenium/README.md
+++ b/src/plugin/protocol-interactiveselenium/README.md
@@ -25,7 +25,8 @@ This protocol plugin allows you to fetch and interact with
pages using [Selenium
You will need to have [Selenium](https://www.selenium.dev/) and a compatible
version of Firefox installed to use this plugin.
Set the protocol to be used in your Nutch configuration files.
-```
+
+```xml
<!-- NUTCH_HOME/conf/nutch-site.xml -->
<configuration>
@@ -39,9 +40,12 @@ Set the protocol to be used in your Nutch configuration
files.
# Custom Handlers
-Only basic functionality is included in the DefaultHandler that comes with the
plugin. If you want additional functionality you can implement custom handlers
by implementing the InteractiveSeleniumHandler interface in the plugin package.
Be sure to also update the plugin config to include your new handler.
+Only basic functionality is included in the DefaultHandler that comes with the
+plugin. If you want additional functionality you can implement custom handlers
+by implementing the InteractiveSeleniumHandler interface in the plugin package.
+Be sure to also update the plugin config to include your new handler.
-```
+```xml
<!-- NUTCH_HOME/conf/nutch-site.xml -->
<property>
<name>interactiveselenium.handlers</name>
@@ -52,4 +56,7 @@ Only basic functionality is included in the DefaultHandler
that comes with the p
# Handler Info
-Handlers are called in the order that they're specified in the configuration.
A "clean" driver is used for each handler so multiple handlers won't interfere
with each other. Page content is appended together from each handler and
returned for the request.
+Handlers are called in the order that they're specified in the configuration.
+A "clean" driver is used for each handler so multiple handlers won't interfere
+with each other. Page content is appended together from each handler and
+returned for the request.
diff --git a/src/plugin/protocol-okhttp/howto_upgrade_okhttp.md
b/src/plugin/protocol-okhttp/howto_upgrade_okhttp.md
index 16ae70d71..20be9254d 100644
--- a/src/plugin/protocol-okhttp/howto_upgrade_okhttp.md
+++ b/src/plugin/protocol-okhttp/howto_upgrade_okhttp.md
@@ -15,6 +15,8 @@
limitations under the License.
-->
+# How to Upgrade OkHttp in Nutch
+
1. Upgrade OkHttp dependency in src/plugin/protocol-okhttp/ivy.xml
2. Upgrade OkHttp's own dependencies in src/plugin/protocol-okhttp/plugin.xml
@@ -41,4 +43,4 @@
$ ant test-plugin -Dplugin=protocol-okhttp
-4. (optionally but recommended) Run a test crawl using protocol-okhttp
\ No newline at end of file
+4. (optionally but recommended) Run a test crawl using protocol-okhttp
diff --git a/src/plugin/protocol-selenium/README.md
b/src/plugin/protocol-selenium/README.md
index dc7f277d6..9fbfe29d4 100644
--- a/src/plugin/protocol-selenium/README.md
+++ b/src/plugin/protocol-selenium/README.md
@@ -26,71 +26,76 @@ The underlying code is based on the nutch-htmlunit plugin,
which was in turn bas
There are essentially two ways in which Nutch can be used with Selenium.
- * Locally (on each node) as a self contained process, or
- * via the RemoteWebDriver which connects to
[Selenium-Grid](https://selenium.dev/documentation/en/grid/). A grid consists
of a single hub, and one or more nodes.
+* Locally (on each node) as a self contained process, or
+* via the RemoteWebDriver which connects to
[Selenium-Grid](https://selenium.dev/documentation/en/grid/). A grid consists
of a single hub, and one or more nodes.
# Installation
-## Part 1:
+## Part 1
### A) Setting up Selenium (local mode)
- * Ensure that you have your preferred browser installed. Currently Chrome,
Safari, Opera, PhantomJS and Firefox are supported. Here there example of
installing Firefox is provided. More info about the package @
[launchpad](https://launchpad.net/ubuntu/trusty/+source/firefox)
-```
+* Ensure that you have your preferred browser installed. Currently Chrome,
+ Safari, Opera, PhantomJS and Firefox are supported. Here there example of
+ installing Firefox is provided. More info about the package @
+ [launchpad](https://launchpad.net/ubuntu/trusty/+source/firefox)
+
+```bash
sudo apt-get install firefox
```
- * Install Xvfb and its associates
+* Install Xvfb and its associates
This step is not necessary for the PhantomJs browser and may not be needed for
all browsers.
-```
+```bash
sudo apt-get install xorg synaptic xvfb gtk2-engines-pixbuf xfonts-cyrillic
xfonts-100dpi \
xfonts-75dpi xfonts-base xfonts-scalable freeglut3-dev dbus-x11 openbox
x11-xserver-utils \
libxrender1 cabextract
```
- * Set a display for Xvfb, so that firefox believes a display is connected
+* Set a display for Xvfb, so that firefox believes a display is connected
-```
+```bash
sudo /usr/bin/Xvfb :11 -screen 0 1024x768x24 &
sudo export DISPLAY=:11
```
+
### B) Setting up a Selenium Grid
Using the Selenium Grid will allow you to parallelize the job by facilitating
access of several instances of browsers whether on one machine or on several
machines.
Note that grid facilitates heterogeneity with regards to browser types used.
However, these steps have been tested using a homogeneous Selenium Grid with
Firefox and PhantomJS browsers.
- * Download the [Selenium Standalone Server](https://selenium.dev/downloads/)
and follow the installation instructions.
+* Download the [Selenium Standalone Server](https://selenium.dev/downloads/)
and follow the installation instructions.
- * Some important configurations to note while setting up the selenium-hub and
the selenium-nodes are:
- * For the hub:
- - maxSession (how many browser sessions to allow on the grid at a time)
- - browserTimeout (how long to wait before timing out a browser session.
This is dependent on the interactivity to be completed on the page)
+* Some important configurations to note while setting up the selenium-hub and
the selenium-nodes are:
+ * For the hub:
+ * maxSession (how many browser sessions to allow on the grid at a time)
+ * browserTimeout (how long to wait before timing out a browser session.
This is dependent on the interactivity to be completed on the page)
- * For the nodes:
- - browserName=<browser>, maxInstances (the max number of instances of
the same version browser to allow per a system)
- - browserName=<browser>, maxSession (the max number of sessions of any
type of browser/version to allow per a system)
+ * For the nodes:
+ * browserName=<browser>, maxInstances (the max number of instances of the
same version browser to allow per a system)
+ * browserName=<browser>, maxSession (the max number of sessions of any
type of browser/version to allow per a system)
- * Go headless with your selenium Grid installation. There are different ways
to this. See [this resource](https://elementalselenium.com/tips/38-headless)
for further details.
+* Go headless with your selenium Grid installation. There are different ways
to this. See [this resource](https://elementalselenium.com/tips/38-headless)
for further details.
- * For Nutch efficiency, and optimization of the grid, consider editing the
following configs in **nutch-site.xml**
- - fetcher.threads.per.queue (change value to the value of the maxSession
config on the hub)
- - fetcher.threads.fetch (change value to the value of the maxSession
config on the hub)
- - fetcher.server.delay (As multiple threads may be accessing a single
server at a time, consider changing this value to 4-5 seconds for politeness)
- - fetcher.server.min.delay (As multiple threads may be accessing a single
server at a time, consider changing this values to 4-5 seconds for politeness)
- - Ensure all configs for the hub mentioned in Part 2 are appropriately set.
+* For Nutch efficiency, and optimization of the grid, consider editing the
following configs in **nutch-site.xml**
+ * fetcher.threads.per.queue (change value to the value of the maxSession
config on the hub)
+ * fetcher.threads.fetch (change value to the value of the maxSession config
on the hub)
+ * fetcher.server.delay (As multiple threads may be accessing a single server
at a time, consider changing this value to 4-5 seconds for politeness)
+ * fetcher.server.min.delay (As multiple threads may be accessing a single
server at a time, consider changing this values to 4-5 seconds for politeness)
+ * Ensure all configs for the hub mentioned in Part 2 are appropriately set.
- * To activate the full selenium grid, edit
**$NUTCH_HOME/runtime/local/bin/crawl** script:
- - numThreads = maxSession on nodes * num of nodes
+* To activate the full selenium grid, edit
**$NUTCH_HOME/runtime/local/bin/crawl** script:
+ * numThreads = maxSession on nodes * num of nodes
## Part 2: Installing plugin for Nutch (where NUTCH_HOME is the root of your
nutch install)
- * Ensure that the plugin will be used as the protocol parser in your config
+* Ensure that the plugin will be used as the protocol parser in your config
-```
+```xml
<!-- NUTCH_HOME/conf/nutch-site.xml -->
<configuration>
@@ -111,7 +116,7 @@ Note that grid facilitates heterogeneity with regards to
browser types used. How
* Then ensure that you have the correct configuration set within the following
configuration options
-```
+```xml
<!-- protocol-selenium plugin properties -->
<property>
@@ -202,15 +207,17 @@ Note that grid facilitates heterogeneity with regards to
browser types used. How
</description>
</property>
```
- * If you've selected 'remote' value for the 'selenium.driver' property,
ensure that you've configured
+
+* If you've selected 'remote' value for the 'selenium.driver' property, ensure
that you've configured
the additional properties based on your [Selenium-Grid
installation](https://selenium.dev/documentation/en/grid/setting_up_your_own_grid/).
- * Compile nutch
-```
+* Compile nutch
+
+```bash
ant runtime
```
- * Start your web crawl (Ensure that you followed the above steps and have
started your xvfb display as shown above)
+* Start your web crawl (Ensure that you followed the above steps and have
started your xvfb display as shown above)
## Part 3: Common Pitfalls
diff --git a/src/plugin/urlfilter-fast/README.md
b/src/plugin/urlfilter-fast/README.md
index 7b6955f07..18d2eec6a 100644
--- a/src/plugin/urlfilter-fast/README.md
+++ b/src/plugin/urlfilter-fast/README.md
@@ -15,13 +15,15 @@
limitations under the License.
-->
+# Fast URL Filter
+
Filters URLs based on a file of regular expressions using host/domains
matching first. The default policy is to accept a URL if no matches
are found.
Rule Format:
-```
+```text
Host www.example.org
DenyPath /path/to/be/excluded
DenyPath /some/other/path/excluded
diff --git a/src/plugin/urlfilter-ignoreexempt/README.md
b/src/plugin/urlfilter-ignoreexempt/README.md
index a4b9b58b4..e9c7bc715 100644
--- a/src/plugin/urlfilter-ignoreexempt/README.md
+++ b/src/plugin/urlfilter-ignoreexempt/README.md
@@ -22,6 +22,7 @@ This is useful when focused crawl is setup but some resources
like static files
# How to enable ?
Add `urlfilter-ignoreexempt` value to `plugin.includes` property
+
```xml
<property>
<name>plugin.includes</name>
@@ -33,7 +34,7 @@ Add `urlfilter-ignoreexempt` value to `plugin.includes`
property
open `conf/db-ignore-external-exemptions.txt` and add the regex rules.
-## Format :
+## Format
The format is same same as `regex-urlfilter.txt`.
Each non-comment, non-blank line contains a regular expression
@@ -41,16 +42,16 @@ prefixed by '+' or '-'. The first matching pattern in the
file
determines whether a URL is exempted or ignored. If no pattern
matches, the URL is ignored.
-## Example :
+## Example
To exempt urls ending with image extensions, use this rule
`+(?i)\.(jpg|png|gif)$`
-## Testing the Rules :
+## Testing the Rules
After enabling the plugin and adding your rules to
`conf/db-ignore-external-exemptions.txt`, run:
`bin/nutch plugin urlfilter-ignoreexempt
org.apache.nutch.urlfilter.ignoreexempt.ExemptionUrlFilter http://yoururl.here`
-This should print `true` for urls which are accepted by configured rules.
\ No newline at end of file
+This should print `true` for urls which are accepted by configured rules.