lewismc commented on PR #888: URL: https://github.com/apache/nutch/pull/888#issuecomment-6401201509
Hi @sebastian-nagel, the note above gives the timings. This is how the partition cache works. ## Why `getPartition` was on the profile With HostDb **on**, the selector job uses `SelectorWithHostDb`. For a CrawlDb row the key's hostname is empty, so `getPartition` calls `URLPartitioner`. That normalizes the URL with `URLNormalizers.SCOPE_PARTITION`, parses it, and hashes the host (or domain, or IP). It does that once per CrawlDb record. On the 50M fixture that is about 50 URLs per host, and the default partition rules do not change the host, so those 50 calls return the same partition. On that profile `SelectorWithHostDb.getPartition` was about 4% of CPU, almost all of it `URLNormalizers.normalize`, and about 22% of allocation. HostDb rows are separate. Their key carries the hostname, and `getPartition` hashes that hostname with `partition.url.seed`. Those rows are not cached. ## What the cache stores `SelectorWithHostDb` keeps a `HashMap` from raw host to the partition `URLPartitioner` already returned. The raw host is read from the URL bytes: find `://`, drop userinfo up to `@`, take the host up to `/`, `?`, `#`, or `:`, keep an IPv6 `[...]` intact, and lowercase it. `http://user:[email protected]:80/p2` and `http://h00000.bench.invalid/p0` share the key `h00000.bench.invalid`. A URL with no scheme or no host returns null from that scan. Those URLs call `URLPartitioner` every time and are not stored. On a CrawlDb row: 1. If this map task's reduce-task count changed, the map is cleared. `setConf` clears it too. 2. If the raw host is already in the map, return the stored int. 3. Otherwise call `partitioner.getPartition(url, score, numReduceTasks)` and store that int under the raw host. The first URL of a host still pays for normalization. Later URLs for that host reuse the int. The cache does not hash the host itself, so the stored value is whatever `URLPartitioner` decided for that first URL, including `partition.url.mode` and `partition.url.seed`. `Selector`, the no-HostDb partitioner, is unchanged. It still calls `URLPartitioner` for every URL, and its key is still `FloatWritable`. ## When the stored int is wrong The cache assumes every later URL with the same raw host would get the same partition from `URLPartitioner`. That holds for the default `SCOPE_PARTITION` rules, which do not rewrite the host from the path. `testHostDbPartitionCacheMatchesUrlPartitioner` checks the first call and the cache hit against `URLPartitioner` for the fixture URL shapes: a second path, `index.html`, a query, userinfo with a port, and a mixed-case host. A custom normalizer that rewrites the host from the path breaks that assumption. `http://example.com/a` and `http://example.com/b` share a cache key. If the rule sends `/b` to a different host, the second URL still gets the partition stored for `/a`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
