jnioche commented on code in PR #2318: URL: https://github.com/apache/stormcrawler/pull/2318#discussion_r4553003692
########## docs/src/main/asciidoc/configuration.adoc: ########## @@ -215,7 +218,7 @@ implementation. | http.proxy.rotation | ROUND_ROBIN | MultiProxyManager proxy rotation scheme when that manager is selected. | http.proxy.type | HTTP | SingleProxyManager component proxy type. | http.proxy.user | - | SingleProxyManager component proxy username, used with `http.proxy.pass`. -| http.retry.on.connection.failure | true | (OkHttp only) Retry fetching on connection failure. See link:https://square.github.io/okhttp/5.x/okhttp/okhttp3/-ok-http-client/-builder/retry-on-connection-failure.html[OkHttp docs]. +| http.retry.on.connection.failure | true | (OkHttp only) Retry fetching on connection failure. See link:https://lysine.dev/okhttp/5.x/okhttp/okhttp3/-ok-http-client/-builder/retry-on-connection-failure.html[OkHttp docs]. Review Comment: point to https://okhttp.org/okhttp-connection-recovery instead? ########## docs/src/main/asciidoc/overview.adoc: ########## @@ -43,11 +43,11 @@ Here is a short list of provided features: ** Parse Filters for post-fetch content filtering * Built-in support for crawl metrics and monitoring * Configurable politeness policies (e.g., crawl delay, user agent management) -* Robust HTTP fetcher based on link:https://hc.apache.org/[Apache HttpComponents] or link:https://square.github.io/okhttp/[OkHttp]. +* HTTP fetcher based on link:https://lysine.dev/okhttp/[OkHttp]. Review Comment: https://okhttp.org/ instead? ########## docs/src/main/asciidoc/migrating.adoc: ########## @@ -6,49 +6,400 @@ https://www.apache.org/licenses/LICENSE-2.0 //// == Migrating to 4.0.0 -Both overloads of `CookieConverter.getCookies(...)` now return immutable +This section lists what a crawler upgraded from 3.7.0, the last 3.x release, +must change or check to keep working as expected. That covers its code, its +configuration, its stored data and the tools which read its output. Each entry +says who is affected and what to do. For other changes, see the release +notes. When upgrading from an earlier 3.x release, also read the release notes +of the versions in between. + +The changes are grouped by the work they require. +<<Before upgrading>> applies to every crawler. +<<Crawler configuration and behaviour>> +applies to crawlers which use the keys or modules named in its entries. +<<Custom Java code>> applies to projects with their own Java +classes. + +=== Before upgrading + +==== Upgrade steps + +. If a crawl using the SQL module may need a rollback to 3.x, back up its status +table first (see <<Stored data>>). +. Upgrade the Storm cluster and its JDK, and update the project, as described +in <<Java and Storm>> and <<Project changes>>. +. Fix the build of custom classes, as described in +<<Custom Java code>>. +. Review the configuration against +<<Crawler configuration and behaviour>>. +A topology built by hand with a querying spout also needs the status stream +connection described in <<Topologies>>. +. Start the topology. These fail at startup: an invalid IP filter rule, Tika +configuration or jsoup XPath expression, and a `FastURLFilter` rule without a +type. Credentials set with an empty `http.basicauth.hosts` or +`http.custom.headers.hosts` are not sent, and a warning is logged. +. Check the first crawl. The changes below leave the topology running but +change what it fetches, parses or stores. Check a representative set of URLs +for their fetch status, extracted content and stored keys. +* robots.txt redirects to another host, and robots.txt files over 512 KiB +* hosts with invalid certificates or non-public IP addresses +* cookies without a `Domain` attribute +* sitemaps without `isSitemap` in their metadata +* Tika metadata keys, embedded documents and the MIME type whitelist +* custom JSoupFilters, which now also run on Tika output +* stored URL forms and host keys +* `HostURLFilter` with `ignoreOutsideDomain: true` +* custom URL filters which throw exceptions +* overrides of the removed `addHeadersToRequest(Request.Builder, Metadata)` + +==== Java and Storm + +StormCrawler 4.0 requires Java 25 and Apache Storm 3.1.0, so the Storm cluster +and its JDK must be upgraded too. The changes to the cluster are described in +the release notes of +link:https://storm.apache.org/2026/07/22/storm300-released.html[Storm 3.0.0] and +link:https://storm.apache.org/2026/09/12/storm310-released.html[Storm 3.1.0]. + +==== Project changes + +Projects generated from a 3.x archetype need these changes in their `pom.xml`: + +* Set `stormcrawler.version` to `4.0.0` and `storm.version` to `3.1.0`. Review Comment: we are on 3.2.0 now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
