abhinav-phi commented on PR #2124: URL: https://github.com/apache/stormcrawler/pull/2124#issuecomment-5574691508
Thanks — pushed a902982f: - **Host component:** `file://evil.example.com/etc/passwd` is now refused outright (403) instead of the host being silently ignored — a URL with a non-empty, non-`localhost` host never serves. Pinned by `fileProtocolRejectsHostComponents`. - **TOCTOU window:** a comment documents the accepted check-vs-read window (canonicalise and check are two operations; a symlink swapped in between would be followed), with the rationale that the scheme is now opt-in and root-confined so the operator enabling it accepts the worker user's read rights as the boundary. - **Assumptions check:** confirmed — nothing in the archetypes or the ProtocolFactory assumes `file` is in `protocols`; the only remaining `http,https,file` reference is in this PR's own test asserting it is *not* the default. **Combined upgrade note** (carrying it here as the most breaking PR of the batch — will move to the release notes page on approval): > **Upgrading to 4.0.0 — changed defaults.** Six changes in this release alter the behaviour of existing topologies: > 1. **File scheme off by default** (this PR): `protocols` no longer contains `file`, and file reads require `file.protocol.root`. Topologies crawling local corpora must set both. > 2. **TLS certificates are validated** (#2126): `http.trust.everything` defaults to `false`. Hosts with self-signed certificates need `http.trust.everything: true`, and credentials (basic auth, credential headers, cookies) additionally need `http.credentials.allow.insecure: true` to be sent to them — credentials are also withheld from cleartext http:// requests now. > 3. **The fetcher stays off private address space** (#2127): `http.filter.ipaddress.exclude` ships enabled (loopback, RFC1918, link-local, CGNAT, ULA). Crawling an intranet requires overriding it explicitly (`http.filter.ipaddress.exclude: ""`). > 4. **Sitemap content sniffing is off** (#2125): sitemaps are only recognised via `isSitemap=true` metadata unless `sitemap.sniffContent: true`. Strict URL checking (`sitemap.strict`) is available but also defaults to false. > 5. **Spouts refuse unexpected schemes** (#2123): stored URLs whose scheme is not in `protocols` are emitted as `Status.ERROR` and removed from the store. > 6. **Bad IP filter rules stop the topology** (#2119): a rule which is neither a keyword, a CIDR block nor an address now fails configuration instead of being ignored. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
