lewismc commented on PR #888: URL: https://github.com/apache/nutch/pull/888#issuecomment-6568116492
Hi @sebastian-nagel, I measured the follow-up that hashes the extracted host directly and drops the HostDb partition cache. ## What changed `SelectorWithHostDb` now sends both the HostDb hostname and the host parsed from each CrawlDb URL through `URLPartitioner.getPartitionForRoot`. Those CrawlDb rows no longer run `URLNormalizers.SCOPE_PARTITION`. If `generate.hostdb` is set and `partition.url.mode` is not `byHost`, the selector logs an error and still partitions by host. The job configuration is left unchanged. The no-HostDb job is still the `FloatWritable` selector. ## Same 4 GB setup Same 50M CrawlDb and 1M-host HostDb, local mode, Java 17, the default 4 GB heap, and eight reducers. Three runs of each, median of `Generator: finished`. Every run exited 0. The cache build and the new build were measured in the same session. Profiles went to the same `nutch.generator` service on the `profiles-prod-008` data source of my Grafana Cloud stack. Note: apologies, I actually cannot add you there after all as it is a staff account. Sorry! The cache rows are `variant=nutch-2455-hostcache`, `git_sha=d4cb33ba8-hostcache`. The new rows are `variant=nutch-2455-hostroot`, `git_sha=d4cb33ba8-hostroot`. The window is 2026-11-02T01:35:00Z through 2026-11-02T02:45:00Z. **50,000,000 URLs, 1,000,000 hosts** | Path | Partition cache | Extracted-host hash | Peak RSS | | --- | ---: | ---: | ---: | | HostDb | 6 min 11 s (371122 ms) | 6 min 5 s (365067 ms, −1.6%) | 4613 MiB (was 4450 MiB) | | No HostDb | 7 min 27 s (446711 ms, prior session) | 7 min 25 s (445408 ms, −0.3%) | 4434 MiB | HostDb off still logs `using standard generation (no HostDb)`. HostDb on still logs `using HostDb integration with secondary sorting`. The individual HostDb times were 371122, 378037, and 368031 ms for the cache, and 364062, 374074, and 365067 ms for the extracted-host hash. The no-HostDb times were 433360, 490506, and 445408 ms. ## What the profiles show On the cache profile, `SelectorWithHostDb.getPartition` was 1.22% of allocation, 17.9 GB. `rawHost` was 0.65% of that, and `URLPartitioner.getPartition` through `URLNormalizers.normalize` was 0.47%. On the new profile those two frames are gone. Partitioning plus `rawHost` is 0.65% of allocation, 9.46 GB. CPU is still local filesystem I/O: kernel `write` about 36%, `read` about 20%, and `fstat64` about 14%. `SelectorReducerWithHostDb.reduce` is about 19%, almost all of it under the fetchlist write. `getPartition` and `URLNormalizers` do not appear in the top CPU nodes of either profile. The wall-clock gain is the three-run median. The profile difference is the removal of that small normalization slice, which is too small by itself to account for the 1.6%. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
