abhinav-phi commented on issue #2145: URL: https://github.com/apache/stormcrawler/issues/2145#issuecomment-6220022937
no one is on it, please go ahead. i opened this from the #2122 review and have not started anything on it since, so it is yours. `URLUtil.getCanonicalHost` is the right place. that is the single point both the politeness key and the robots.txt cache key pass through, so collapsing there fixes both at once instead of letting the two call sites drift apart again, which is what went wrong the first time. two things worth pinning down before you write it: `IDN.toASCII` is more lenient than a strict resolver, it accepts some labels that a real lookup would not, and it throws IllegalArgumentException on others. catching per label and keeping the raw lowercased host on failure is what the issue describes and it matches what okhttp settles on at connect time, so that shape is fine. also add a case for the host that is already punycoded. `xn--exampl-gva.org` going in should come back identical, so a host that was already stored in ascii form stays stable when it is read back and re keyed. thanks for writing the plan out first, that is the right way to take an issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
