sebastian-nagel commented on PR #2006: URL: https://github.com/apache/stormcrawler/pull/2006#issuecomment-5054548792
Thanks, @HarshDevelops! The implementation looks straightforward and solid. However, it might need a rework... Not your fault, but because the web has changed since the implementation of Nutch's SpellCheckedMetadata. Also the way it's used in Nutch has changed or will change, cf. [NUTCH-2563](https://issues.apache.org/jira/browse/NUTCH-2563), [NUTCH-3002](https://issues.apache.org/jira/browse/NUTCH-3002)/[NUTCH-3122](https://issues.apache.org/jira/browse/NUTCH-3122) and [this discussion](https://github.com/apache/nutch/pull/869#discussion_r2507085852). > outside of Metadata itself. We have comparable utility classes in `o.a.s.util.*`, this would be the right place for it I think. Definitely, that was even the case in Nutch, although not from the beginning on (apache/nutch@bbbc29d and apache/nutch@f510e19). In addition, approximate header extraction shouldn't be the default. Some context and explanations: - Nutch's SpellCheckedMetadata class is there since the 2006, that is the times of HTTP/1.1, with HTTP/1.0 or 0.9 still frequently used. These are text-based protocols. When entire HTTP messages are hand-written (maybe as templates), typos could easily slip in. - Nowadays, HTTP/2 is dominant. As a binary protocol, it's much less likely that header names are mispelled. Headers in HTTP/2 are lower-case, so case-insensitive header processing is mandatory. - My impression is, that spellchecking headers unconditionally can introduce more harm than benefits. Below is one example, a WARC capture of a HTTP 302 redirect. The redirect requires the `Location` header. In addition, the response contains an `X-Location` header. The spell checker would map it also to `Location`. - Standard header names prefixed by `X-` or another prefix are quite common, often they are added by a proxy or the front-end server, rewriting the original response headers. - Would be better to do the approximate header lookup / mapping, only if an expected standard header is not found. In case of a HTTP redirect, if the `Location` header is missing. ``` WARC/1.0 WARC-Type: response WARC-Date: 2023-03-26T14:32:42Z WARC-Record-ID: <urn:uuid:5543a3db-2cff-43bc-af20-bdc5fd66d517> Content-Length: 879 Content-Type: application/http; msgtype=response WARC-Warcinfo-ID: <urn:uuid:4d3666d6-9165-4654-a45f-59bd4e135488> WARC-Concurrent-To: <urn:uuid:1a58671a-0c2c-4d50-8bfd-ba9b7302810f> WARC-IP-Address: 54.162.128.250 WARC-Target-URI: https://legacy.latam.coderhouse.com/blog/gente-pongan-moderadores WARC-Payload-Digest: sha1:ZYUB2QWIWMHTIA6667UBTITTQMBDP3QH WARC-Block-Digest: sha1:GDQFDDKZHNZJT6XQVOTKYG4C3MKV63QR WARC-Identified-Payload-Type: text/html HTTP/1.1 302 Found Connection: keep-alive Date: Sun, 26 Mar 2023 14:32:42 GMT Content-Type: text/html; charset=utf-8 Content-Length: 160 X-Dns-Prefetch-Control: off Expect-Ct: max-age=0 Strict-Transport-Security: max-age=15552000; includeSubDomains X-Download-Options: noopen X-Content-Type-Options: nosniff X-Permitted-Cross-Domain-Policies: none Referrer-Policy: no-referrer X-Xss-Protection: 0 Access-Control-Allow-Origin: * Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept Cache-Control: public, max-age=3600 Location: https://latam.coderhouse.com/blog/gente-pongan-moderadores Vary: Accept, Accept-Encoding Server: Coderhouse-IT X-Location: cached Via: 1.1 vegur <p>Found. Redirecting to <a href="https://latam.coderhouse.com/blog/gente-pongan-moderadores";>https://latam.coderhouse.com/blog/gente-pongan-moderadores</a></p> ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
