abhinav-phi commented on PR #2122: URL: https://github.com/apache/stormcrawler/pull/2122#issuecomment-5609284696
Thanks for the second round. The partitioner path is now covered, all in this push: - \URLPartitionerBolt\ and \URLPartitioner.getPartition\ derive host/domain/IP keys from \URLUtil.getCanonicalHost\ (\a0ced65\), so percent-escaped, case and trailing-dot aliases share one key in every mode. An explicitly supplied \ip\ metadata value stays authoritative in both. - \URLPartitionerBoltTest\ (\cb77a2d8\) exercises the bolt end to end: same key for \xample.org\ / \xampl%65.org\ / \xample.org.\ / \EXAMPLE.org\, and the URL itself is emitted unchanged. \URLPartitionerTest\ covers byHost/byDomain/byIP plus the provided-IP case. - The fetcher side (\FetcherBolt.FetchItem.create\, \SimpleFetcherBolt\, robots cache key) was already canonicalised; the status-updater normalisation (\5509a2a0\) stays behind \status.updater.normalise.hosts\ (default off) as discussed. - IDN stays out of scope and is recorded as #2145 (also noted in the description). \mvn -pl core verify -DCI_ENV=true\ is green on the branch (0 checkstyle). Please re-review when you get a moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
