[
https://issues.apache.org/jira/browse/NUTCH-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541483#comment-17541483
]
Hudson commented on NUTCH-2950:
-------------------------------
SUCCESS: Integrated in Jenkins build Nutch ยป Nutch-trunk #76 (See
[https://ci-builds.apache.org/job/Nutch/job/Nutch-trunk/76/])
NUTCH-2950 Improve performance of UpdateHostDb (snagel:
[https://github.com/apache/nutch/commit/70b2d5e556d937a90018e156c6147e525bfee04b])
* (edit) src/java/org/apache/nutch/util/URLUtil.java
* (edit) src/java/org/apache/nutch/hostdb/UpdateHostDbMapper.java
NUTCH-2950 Improve performance of UpdateHostDb (snagel:
[https://github.com/apache/nutch/commit/5a6ac3bb1ccbd49c556ae5c98e686948d0171634])
* (edit) src/java/org/apache/nutch/crawl/Generator.java
* (edit) src/java/org/apache/nutch/hostdb/HostDatum.java
* (edit) src/java/org/apache/nutch/hostdb/ReadHostDb.java
* (edit) src/java/org/apache/nutch/hostdb/UpdateHostDbReducer.java
NUTCH-2950 Improve performance of UpdateHostDb (snagel:
[https://github.com/apache/nutch/commit/417dee6d1ef642586388ebf5b57f8b1bda18e2d9])
* (edit) src/java/org/apache/nutch/util/URLUtil.java
* (edit) src/java/org/apache/nutch/hostdb/UpdateHostDbMapper.java
NUTCH-2950 Improve performance of UpdateHostDb (snagel:
[https://github.com/apache/nutch/commit/5086958be704022de2253d5311da1cc23fc19b37])
* (edit) src/java/org/apache/nutch/hostdb/UpdateHostDbMapper.java
* (edit) src/java/org/apache/nutch/util/URLUtil.java
NUTCH-2950 Improve performance of UpdateHostDb (snagel:
[https://github.com/apache/nutch/commit/947e67bef075f7e3c87d6b55f5004b0cd6402385])
* (edit) src/java/org/apache/nutch/util/URLUtil.java
> UpdateHostDb: performance improvements
> --------------------------------------
>
> Key: NUTCH-2950
> URL: https://issues.apache.org/jira/browse/NUTCH-2950
> Project: Nutch
> Issue Type: Improvement
> Components: hostdb
> Affects Versions: 1.18
> Reporter: Sebastian Nagel
> Assignee: Sebastian Nagel
> Priority: Major
> Fix For: 1.19
>
>
> This issue addresses a couple of performance improvements when creating the
> HostDb:
> - avoid needless conversions between hostname and URL
> - improvements of HostDb serialization (write and read)
> - parametrize logging and log less on level INFO
> - do not create DNS resolver threads if DNS look-ups are not requested by
> command-line options
> A patch/PR is ready. Depending on the chosen command-line options, a 10-20%
> speed-up should be visible if DNS look-ups, normalization and filtering are
> off.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
