lewismc commented on PR #888: URL: https://github.com/apache/nutch/pull/888#issuecomment-6401157672
Hi @sebastian-nagel, a follow-up on the 50M HostDb profile from the runs above. ## What a updated profiling exercise showed On each 50M HostDb run, almost all of the CPU was local filesystem I/O in the spill and merge. Inside Generator, `SelectorWithHostDb.getPartition` was about 4% of CPU. Nearly all of that was `URLNormalizers.normalize`, and that same frame was a large share of allocation. `MapOutputBuffer.compare` was about 1.3%. A raw-byte `ScoreHostKeyComparator` would not have moved the 50M time, so I left the comparator as it is. The CrawlDb side was calling `URLPartitioner` once per URL. On this fixture that is about 50 URLs per host, and the default normalizers do not change the host. The new code keeps a host-to-partition map on `SelectorWithHostDb`. The first CrawlDb URL for a host still goes through `URLPartitioner`. Later URLs for that same raw host reuse the partition. HostDb rows still partition by hostname hash. The no-HostDb job is still the `FloatWritable` selector. For the new run, the profiles are on the same `nutch.generator` service, labeled `git_sha=6d9a64e63-hostcache`. In that profile, `getPartition` and `URLNormalizers.normalize` no longer show up in the summary. The reducer is still mostly I/O. ## Same 4 GB setup I performed a new run with the same CrawlDbs, local mode, Java 17, and the default 4 GB heap as the tables above. Three runs, median of `Generator: finished`. Every run exited 0. I did not change the heap. The comparison point is the NUTCH-2455 column already posted (`ee6eecd68`). **1,000,000 URLs, 200,000 hosts** | Path | Previous NUTCH-2455 | With the partition cache | Peak RSS | | --- | ---: | ---: | ---: | | No HostDb | 17.7 s (17685 ms) | 17.7 s (17683 ms, −0.01%) | 760 MiB (was 769 MiB) | | HostDb | 18.6 s (18636 ms) | 16.6 s (16634 ms, −10.7%) | 893 MiB (was 793 MiB) | **50,000,000 URLs, 1,000,000 hosts** | Path | Previous NUTCH-2455 | With the partition cache | Peak RSS | | --- | ---: | ---: | ---: | | No HostDb | 7 min 36 s (455533 ms) | 7 min 27 s (446711 ms, −1.9%) | 3803 MiB (was 3975 MiB) | | HostDb | 7 min 51 s (471043 ms) | 6 min 23 s (383300 ms, −18.6%) | 3695 MiB (was 3935 MiB) | The log lines are unchanged: HostDb off still says `using standard generation (no HostDb)`, and HostDb on still says `using HostDb integration with secondary sorting`. At 50M the process still sits on the 4 GB heap because the sort dominates. The 1M HostDb RSS median is higher than before. The 50M HostDb RSS median is lower. ## One limit The cache key is the raw host, taken before `URLNormalizers.SCOPE_PARTITION`. The default normalizers leave that host alone, and `testHostDbPartitionCacheMatchesUrlPartitioner` checks the cached partition against `URLPartitioner` for those URLs, including a userinfo URL and a mixed-case host. A custom normalizer that rewrites the host from the path can pin a later URL to the first URL's partition. Any testng you can do, particularly in distributed mode, would be greatly appreciated. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
