jnioche opened a new issue, #2266: URL: https://github.com/apache/stormcrawler/issues/2266
See discussion on https://github.com/apache/stormcrawler/pull/2236/#issuecomment-6250049212 ### Context After #2211, core depends only on `org.apache.httpcomponents.core5:httpcore5`. It uses it for HTTP header and status constants and for query-string parsing in `BasicURLNormalizer`. A whole library for that is not worth the dependency, and HttpComponents types in our code mean we can't change these classes in a 4.x minor release without breaking user code. 4.0.0 is the time to remove it. ### What's affected **1. `HttpHeaders` constants** - **Main code:** `FeedParserBolt`, `JSoupParserBolt`, `SimpleFetcherBolt`, `SiteMapParserBolt`, `FetchOutcomes`, `AdaptiveScheduler`, `HttpRobotRulesParser`, `FileResponse`, okhttp `HttpProtocol`, `CharsetIdentification`, plus `ParserBolt` (tika) and `WARCRecordFormat` (warc). - **Tests:** `FeedParserBoltTest`, `JSoupParserBoltTest`, `SiteMapParserBoltTest`, `AdaptiveSchedulerTest`, `CharsetIdentificationTest`, plus the tika and warc tests. - **Constants used:** `ACCEPT`, `ACCEPT_LANGUAGE`, `AUTHORIZATION`, `CONTENT_LENGTH`, `CONTENT_TYPE`, `ETAG`, `IF_MODIFIED_SINCE`, `IF_NONE_MATCH`, `LAST_MODIFIED`, `LOCATION`, `PROXY_AUTHORIZATION`, `USER_AGENT`. **2. `HttpStatus` constants** - Used in `FileResponse`. - **Constants used:** `SC_OK`, `SC_MULTIPLE_CHOICES`, `SC_BAD_REQUEST`, `SC_UNAUTHORIZED`, `SC_FORBIDDEN`, `SC_NOT_FOUND`, `SC_METHOD_FAILURE`, `SC_INTERNAL_SERVER_ERROR`. **3. `NameValuePair` and `URLEncodedUtils` in `BasicURLNormalizer`** - These sort the query parameters and remove the filtered ones. - The code calls `URLEncodedUtils.parse(query, UTF_8, '&', ';')` and `URLEncodedUtils.format(pairs, UTF_8)`. That class is deprecated in httpcore5. ### Actions 1. Add StormCrawler-owned `HttpHeaders` and `HttpStatus` classes in core, for example under `org.apache.stormcrawler.protocol`. Each holds the constants listed above as plain `public static final` fields, with the same names and values as HttpCore 5. Switching to them is then an import change only. 2. Update all the imports above in core, tika and warc, main and test code. 3. Replace `URLEncodedUtils` and `NameValuePair` in `BasicURLNormalizer` with a small in-house parser and formatter. The output must stay exactly the same. 4. Remove `httpcore5` from `core/pom.xml`, and the `httpcore5.version` property from the root `pom.xml` if nothing else uses it. 5. Remove the leftover `<exclude>org/apache/http/**</exclude>` from the archetype's shade configuration. 6. Update the "Migrating to 4.0.0" docs. Core no longer brings in any HttpComponents classes, so users who relied on them transitively must declare the dependency themselves. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
