rzo1 commented on PR #2122: URL: https://github.com/apache/stormcrawler/pull/2122#issuecomment-5589828482
> Shouldn't that be handled as part of the filtering / normalisation? For the storage side, yes, and we don't do it: `BasicURLNormalizer` lowercases the host and can do IDN→ASCII (`BasicURLNormalizer.java:132-152`), but `unescapePath` only ever runs on the file part, so the host keeps its percent-escapes. Both spellings end up in the backend as separate records today. That's worth fixing there, and the trailing-dot stripping in `getCanonicalHost` would fit better in the normaliser than in a fetcher helper. It doesn't cover this PR though, since the two are on different paths. `URLFilters` only get instantiated in `StatusEmitterBolt`, `URLFilterBolt` and the `LinkParseFilter`s, i.e. the outlink emit path. Nothing normalises URLs on the way into the fetcher, and seeds injected straight into the status store never see a normaliser unless the injection topology has a `URLFilterBolt` in it. Normalisation is configurable too, and I'd rather the per-host delay and the robots.txt fetch count not depend on which filters someone enabled. `byIP` and `byDomain` need a canonical host to resolve/split from either way. Does it make sense? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
