[
https://issues.apache.org/jira/browse/NUTCH-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18060147#comment-18060147
]
ASF GitHub Bot commented on NUTCH-2455:
---------------------------------------
lewismc commented on PR #888:
URL: https://github.com/apache/nutch/pull/888#issuecomment-3941515717
Hi @sebastian-nagel this PR ended up massive!
I addressed the conflict and propose a solution for the performance
regression you identified (~20% longer runtime) when generating fetch lists
from large CrawlDbs without HostDb configured. The regression was caused by:
1. `HostDatum` serialization overhead: Every `SelectorEntry` was serializing
a full `HostDatum` object, even when HostDb was not used
2. FloatTextPair comparison overhead: Using `FloatTextPair` composite keys
requires object deserialization during sorting, whereas `FloatWritable` uses
optimized raw byte comparison
In summary the changes are as follows
**Conditional HostDatum Serialization**
* Added `hasHostDatum` flag to `SelectorEntry` to make `HostDatum`
serialization optional
* When HostDb is not used, only a single boolean byte is written instead of
the full `HostDatum` structure
**Dual-Path Job Configuration**
* Created separate MapReduce components for each code path:
* With HostDb: `SelectorMapperWithHostDb`, `SelectorReducerWithHostDb`,
`SelectorWithHostDb` using `FloatTextPair` keys and `ScoreHostKeyComparator`
for secondary sorting
* Without HostDb: `SelectorMapper`,`SelectorReducer,` Selector using
`FloatWritable` keys and `DecreasingFloatComparator` for optimized raw byte
comparison
* Modified `the generate()` method to conditionally configure the job based
on whether HostDb is provided
If you could please test and profile again it would be greatly appreciated.
When `generate.hostdb` is not configured, the Generator uses the original
optimized code path. No changes to configuration properties or command-line
interface. Existing Generator workflows remain unaffected which is very handy.
> Use secondary sorting for memory-efficient HostDb integration in Generator
> --------------------------------------------------------------------------
>
> Key: NUTCH-2455
> URL: https://issues.apache.org/jira/browse/NUTCH-2455
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Affects Versions: 1.13
> Reporter: Markus Jelsma
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.23
>
> Attachments: NUTCH-2455.patch
>
>
> Citing Sebastian at NUTCH-2420:
> ??The correct solution would be to use <host,score> pairs as keys in the
> Selector job, with a partitioner and secondary sorting so that all keys with
> same host end up in the same call of the reducer. If values can also hold a
> HostDb entry and the sort comparator guarantees that the HostDb entry
> (entries if partitioned by domain or IP) comes in front of all CrawlDb
> entries. But that would be a substantial improvement...??
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
