rzo1 commented on PR #2122: URL: https://github.com/apache/stormcrawler/pull/2122#issuecomment-5590049347
Had a quick chat with @jnioche about this. Rather than canonicalising the host at the fetcher and partitioner call sites, it probably belongs in `AbstractStatusUpdaterBolt`, so there's one place every URL passes through instead of scattering it. `AbstractStatusUpdaterBolt.execute` (`core/src/main/java/org/apache/stormcrawler/persistence/AbstractStatusUpdaterBolt.java:146`) is the only write path into the status store — all seven subclasses inherit it, none override `execute`. The fetch path only reads from the store (`AggregationSpout` in the archetype `crawler.flux`), and injection goes `FileSpout → URLFilterBolt → StatusUpdaterBolt`. So if the URL is normalised there, it's normalised everywhere downstream, and both spellings stop reaching the backend, which was Julien's point. Two things to get right: - Only do it when the status is `DISCOVERED`. That flag already exists at `:151` as `potentiallyNew`. `FETCHED`/`FETCH_ERROR`/`REDIRECTION` updates carry URLs that came out of the store, so re-normalising them is wasted work on the hot path. Redirect targets arrive as `DISCOVERED` via the outlink path, so they're still covered. - Put it **before** the dedup cache lookup at `:157`. That cache is keyed on the raw URL, so `http://exampl%65.org/` and `http://example.org/` currently miss each other there too. Same reason it needs to sit ahead of the `AS_IS_NEXTFETCHDATE` early return at `:175`. One thing for the release notes: `getDocumentID` hashes the URL, so URLs already stored in raw form get re-discovered under the normalised form as a second record. Bounded, but it should sit behind a config flag defaulting to off for one release. That would leave this PR much smaller — possibly just the `byIP`/`byDomain` canonical-host handling in the partitioner, if anything. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
