[
https://issues.apache.org/jira/browse/NUTCH-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380958#comment-15380958
]
Hudson commented on NUTCH-2287:
-------------------------------
FAILURE: Integrated in Nutch-trunk #3384 (See
[https://builds.apache.org/job/Nutch-trunk/3384/])
NUTCH-2287 Indexer-elastic plugin should use Elasticsearch BulkProcessor
(jnaegele: rev 0fff24acff99df1ce9d3f9c9b1a2c674f9c8e325)
* src/plugin/indexer-elastic/build.xml
*
src/plugin/indexer-elastic/src/test/org/apache/nutch/indexwriter/elastic/TestElasticIndexWriter.java
*
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
* conf/nutch-default.xml
* src/plugin/indexer-elastic/plugin.xml
* src/plugin/indexer-elastic/src/test/conf/nutch-site-test.xml
* src/plugin/build.xml
* build.xml
*
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticConstants.java
NUTCH-2287 Indexer-elastic plugin should use Elasticsearch BulkProcessor
(lewis.mcgibbney: rev 80afa3134d7e8de07fcfbb03dba1c51fb7c7dce2)
*
src/plugin/indexer-elastic/src/test/org/apache/nutch/indexwriter/elastic/TestElasticIndexWriter.java
* src/plugin/indexer-elastic/plugin.xml
* src/plugin/indexer-elastic/build.xml
* conf/nutch-default.xml
* src/plugin/build.xml
*
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
*
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticConstants.java
* build.xml
* src/plugin/indexer-elastic/src/test/conf/nutch-site-test.xml
> Indexer-elastic plugin should use Elasticsearch BulkProcessor and
> BackoffPolicy
> -------------------------------------------------------------------------------
>
> Key: NUTCH-2287
> URL: https://issues.apache.org/jira/browse/NUTCH-2287
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, plugin
> Affects Versions: 1.12
> Reporter: Joseph Naegele
> Assignee: Lewis John McGibbney
> Fix For: 1.13
>
>
> Elasticsearch's API (since at least v2.0) includes the {{BulkProcessor}},
> which automatically handles flushing bulk requests given a max doc count
> and/or max bulk size. It also now (I believe since 2.2.0) offers a
> {{BackoffPolicy}} option, allowing the BulkProcessor/Client to retry bulk
> requests when the Elasticsearch cluster is saturated. Using the
> {{BulkProcessor}} was originally suggested
> [here|https://issues.apache.org/jira/browse/NUTCH-1527?focusedCommentId=13666616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13666616].
> Refactoring the {{indexer-elastic}} plugin to use the {{BulkProcessor}} will
> greatly simplify the existing plugin at the cost of slightly less debug
> logging. Additionally, it will allow the plugin to handle cluster saturation
> gracefully (rather than raising a RuntimeException and killing the reduce
> task), by using a configurable "exponential back-off policy".
> https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.3/java-docs-bulk-processor.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
