This is an automated email from the ASF dual-hosted git repository. rzo1 pushed a commit to branch security-model-expansion in repository https://gitbox.apache.org/repos/asf/stormcrawler-site.git
commit c88328d713189f08c14cd6034748dabc00f771e4 Author: Richard Zowalla <[email protected]> AuthorDate: Tue Aug 18 19:51:17 2026 +0200 Expand security model with network-facing threat model The security page covered trusted configuration, trusted Storm cluster access and multi-tenancy, but said nothing about the side of the system that actually faces the network. This adds: * a statement that crawled content is untrusted by design, and that consequences of the crawler crawling what it was configured to crawl are not in themselves vulnerabilities; * operator responsibilities with the relevant configuration keys and their shipped defaults: URL filtering (urlfilters.config.file ships commented out, archetypes supply default-regex-filters.txt), protocol schemes (protocols ships as "http,https,file"), egress restriction (http.filter.ipaddress.include/exclude, OkHttp only) and resource limits (http.content.limit defaults to -1, archetypes set 65536); * transport and credential handling: the rationale and the consequence of http.trust.everything defaulting to true, the lack of host scoping on http.basicauth credentials and cookie replay under http.use.cookies; * a short summary of when the project issues an advisory and when it treats a change as hardening, referencing ASF policy. The reporting process, trusted configuration, Storm cluster and known vulnerabilities sections are unchanged. security/index.html was also the only file in the repository still using CRLF line terminators, so it is converted to LF along with the edit. --- security/index.html | 168 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 106 insertions(+), 62 deletions(-) diff --git a/security/index.html b/security/index.html index 8156457..080dcd8 100644 --- a/security/index.html +++ b/security/index.html @@ -1,62 +1,106 @@ ---- -layout: default -slug: security -title: Reporting Security Problems to Apache StormCrawler ---- - -<div class="row row-col"> - <h1>Security</h1> - <h2>Reporting New Security Problems with Apache StormCrawler</h2> - <p>The Apache Software Foundation takes a very active stance in eliminating security problems and denial of service attacks against its products.</p> - <p>We strongly encourage people to report security problems privately using the security mailing list of the <a href="https://www.apache.org/security/";>ASF Security Team</a> before disclosing them in a public forum.</p> - <p>Please note that the security mailing list should only be used for reporting undisclosed security vulnerabilities and managing the process of fixing such vulnerabilities. We cannot accept regular bug reports or other queries at this address. All mail sent to this address that does not relate to an undisclosed security problem in our source code will be ignored.</p> - <p>The private security mailing address is: <a class="externalLink" href="mailto:[email protected]";>[email protected]</a></p> - - <h2>Threat Model and Security Considerations</h2> - <p>StormCrawler is designed to operate in trusted environments as part of a distributed Apache Storm® cluster. This document outlines the threat model and key security assumptions to help users understand the secure use and deployment of StormCrawler.</p> - - <h3>Trusted Configuration</h3> - <p>The configuration file used by StormCrawler is loaded during topology submission and is treated as a trusted source. It does not involve any user-supplied input at runtime.</p> - <p>If an attacker is able to modify this file, they would already have full access to the system, including:</p> - <ul> - <li>The ability to alter behavior of the topology</li> - <li>Access to credentials and other secrets</li> - <li>Arbitrary control over job execution</li> - </ul> - <p>Securing the configuration file and the environment in which topologies are submitted is essential. However, modification of the file implies full system compromise and is out of scope for runtime protections.</p> - - <h3>Apache Storm® Cluster Security</h3> - <p>StormCrawler runs on an Apache Storm® cluster, which is designed to allow users to:</p> - <ul> - <li>Submit topologies</li> - <li>Execute custom, user-defined code</li> - </ul> - <p>This model inherently trusts cluster users and assumes they are authorized.</p> - - <h4>Security Recommendations:</h4> - <ul> - <li>Access to the Apache Storm® cluster must be strictly restricted to trusted users</li> - <li>Underlying systems should not store secrets or hold elevated privileges beyond those assigned to the authorized users</li> - <li>Avoid deploying StormCrawler in multi-tenant environments without strong isolation guarantees</li> - </ul> - - <h3>Summary</h3> - <p>StormCrawler's security model assumes a trusted deployment environment. Users should:</p> - <ul> - <li>Secure configuration files and deployment infrastructure</li> - <li>Restrict Apache Storm® cluster access</li> - <li>Follow best practices for secret and privilege management</li> - </ul> - - <h2>Asking Questions About Known Security Problems</h2> - <p>Questions about:</p> - <ul> - <li>if a vulnerability applies to your particular application</li> - <li>obtaining further information on a published vulnerability</li> - <li>availability of patches and/or new releases</li> - </ul> - <p>should be addressed to the <a href="https://lists.apache.org/[email protected]";>dev mailing list</a>.</p> - - <h2>Known Security Vulnerabilities</h2> - <p>No known security vulnerability yet.</p> -</div> +--- +layout: default +slug: security +title: Reporting Security Problems to Apache StormCrawler +--- + +<div class="row row-col"> + <h1>Security</h1> + <h2>Reporting New Security Problems with Apache StormCrawler</h2> + <p>The Apache Software Foundation takes a very active stance in eliminating security problems and denial of service attacks against its products.</p> + <p>We strongly encourage people to report security problems privately using the security mailing list of the <a href="https://www.apache.org/security/";>ASF Security Team</a> before disclosing them in a public forum.</p> + <p>Please note that the security mailing list should only be used for reporting undisclosed security vulnerabilities and managing the process of fixing such vulnerabilities. We cannot accept regular bug reports or other queries at this address. All mail sent to this address that does not relate to an undisclosed security problem in our source code will be ignored.</p> + <p>The private security mailing address is: <a class="externalLink" href="mailto:[email protected]";>[email protected]</a></p> + + <h2>Threat Model and Security Considerations</h2> + <p>StormCrawler is designed to operate in trusted environments as part of a distributed Apache Storm® cluster. This document outlines the threat model and key security assumptions to help users understand the secure use and deployment of StormCrawler.</p> + + <h3>Crawled Content Is Untrusted by Design</h3> + <p>StormCrawler is a library for broad web crawling. It fetches URLs it was told to fetch, including URLs discovered in previously fetched pages, and it parses the bytes that come back. Both the URLs and the bytes are chosen by the operators of the sites being crawled, which means attacker-controlled input is not an edge case for a crawler: it is the normal operating condition. Every component that handles a response body, a response header, a redirect target or an extracted outlink is [...] + <p>It follows that consequences of the crawler crawling what it was configured to crawl are not, in themselves, vulnerabilities. If a topology is pointed at a host, fetches a resource from it and stores the result, that is the software working as intended, regardless of what the remote host chose to serve. What the project does treat as a security problem is content crossing a boundary it was never meant to cross: crawled bytes influencing the crawler's own configuration or control data [...] + <p>This is the frame for the rest of this section. The controls described below exist so that operators can define where that boundary lies for their deployment. Choosing not to configure them widens the boundary rather than removing it.</p> + <p>Malformed or hostile responses that make a crawl slow, stall a worker or exhaust its memory are bugs the project fixes. Such effects are confined to the crawl the operator chose to run, and the project treats them as robustness rather than as a compromise of the deployment.</p> + + <h3>Trusted Configuration</h3> + <p>The configuration file used by StormCrawler is loaded during topology submission and is treated as a trusted source. It does not involve any user-supplied input at runtime.</p> + <p>If an attacker is able to modify this file, they would already have full access to the system, including:</p> + <ul> + <li>The ability to alter behavior of the topology</li> + <li>Access to credentials and other secrets</li> + <li>Arbitrary control over job execution</li> + </ul> + <p>Securing the configuration file and the environment in which topologies are submitted is essential. However, modification of the file implies full system compromise and is out of scope for runtime protections.</p> + + <h3>Apache Storm® Cluster Security</h3> + <p>StormCrawler runs on an Apache Storm® cluster, which is designed to allow users to:</p> + <ul> + <li>Submit topologies</li> + <li>Execute custom, user-defined code</li> + </ul> + <p>This model inherently trusts cluster users and assumes they are authorized.</p> + + <h4>Security Recommendations:</h4> + <ul> + <li>Access to the Apache Storm® cluster must be strictly restricted to trusted users</li> + <li>Underlying systems should not store secrets or hold elevated privileges beyond those assigned to the authorized users</li> + <li>Avoid deploying StormCrawler in multi-tenant environments without strong isolation guarantees</li> + </ul> + + <h3>Operator Responsibilities</h3> + <p>StormCrawler is a library, not a finished crawler. Several controls that determine what a crawl is allowed to reach ship disabled or permissive, because the library cannot know the scope of the crawl a given operator intends to run. Configuring them is the operator's responsibility.</p> + + <h4>URL Filtering</h4> + <p>The library ships no URL filters at all: <code>urlfilters.config.file</code> is present but commented out in <code>crawler-default.yaml</code>, so a topology built directly on the library will follow every outlink it discovers, unbounded, unless the operator supplies a filter configuration. Projects generated from the StormCrawler Maven archetypes do not have this problem: they set <code>urlfilters.config.file</code> and ship a <code>default-regex-filters.txt</code> as a starting point.</p> + <p>A topology running with no URL filtering at all is not a supported configuration. Operators should define the intended crawl scope explicitly, in terms of hosts, domains and URL patterns, and should treat the filter configuration as a security control rather than a tuning parameter.</p> + + <h4>Protocol Schemes</h4> + <p>The <code>protocols</code> key currently ships as <code>"http,https,file"</code>, which registers the <code>file</code> protocol implementation alongside the HTTP ones. Because outlinks are extracted from crawled pages, an enabled <code>file</code> scheme gives crawled content a path to the local filesystem of the worker unless URL filtering prevents it. This is the combination that matters: the file scheme and the absence of URL filters together, not either one alone.</p> + <p>Non-web schemes should be enabled deliberately, for deployments that actually crawl local files, and should be paired with URL filters that constrain which paths may be reached. Operators who do not crawl local content should reduce <code>protocols</code> to <code>"http,https"</code>.</p> + + <h4>Egress Restriction</h4> + <p>Regular-expression URL filters operate on the URL string and cannot know what a hostname resolves to. A hostname under the control of a crawled site can resolve to a loopback, link-local or private address, which no pattern on the URL can detect. <code>http.filter.ipaddress.include</code> and <code>http.filter.ipaddress.exclude</code> address this: they are applied after DNS resolution, at connection time, and are the only control in the library that can act on the resolved address. [...] + <p>Operators crawling the public web should set an exclude rule covering loopback and site-local ranges at minimum; operators crawling a known internal estate should prefer an explicit include rule. Note that this filtering is implemented in the OkHttp protocol only. Topologies using other protocol implementations, including the browser-based ones, need to obtain the same restriction from the network layer around the workers.</p> + + <h4>Resource Limits</h4> + <p><code>http.content.limit</code> defaults to <code>-1</code>, meaning no limit: the size of a fetched document is decided by the remote server. The archetype configurations set it to <code>65536</code>. Since response bodies are attacker-chosen, an unbounded limit lets a crawled host decide how much memory a worker allocates for a single fetch.</p> + <p>Operators should set a finite <code>http.content.limit</code> appropriate to the content they intend to index. The related <code>http.robots.content.limit</code> defaults to the same value and deserves the same treatment.</p> + + <h3>Transport and Credentials</h3> + <p><code>http.trust.everything</code> currently defaults to <code>true</code>. With that setting, the OkHttp protocol accepts any server certificate and performs no hostname verification. The rationale is that broad crawling encounters expired, self-signed and otherwise misconfigured certificates constantly, that a crawl aborting on them would be of limited use, and that the fetched bytes are treated as hostile in any case, so certificate validity adds little to how the response is hand [...] + <p>That rationale covers the content of an ordinary anonymous crawl. It does not cover anything the crawler sends. With certificate and hostname verification disabled, the crawler cannot tell which host it is actually talking to, so any credential, custom header or replayed cookie attached to a request crosses a channel whose far end was never authenticated. Operators who crawl anything authenticated must weigh this, and should set <code>http.trust.everything</code> to <code>false</code [...] + <p>Credentials configured through <code>http.basicauth.user</code> and <code>http.basicauth.password</code> are turned into a fixed <code>Authorization</code> header that is attached to every request the protocol instance makes, with no host scoping. If such a topology follows an outlink to a third-party host, the credentials go with it. Operators crawling authenticated sites should scope credentials by origin themselves, by running a separate topology per origin with URL filters confin [...] + <p>Cookie replay is a related risk. <code>http.use.cookies</code> is <code>false</code> by default, and enabling it causes cookies stored from a response to be sent with requests for links found in that response. Operators enabling it on an authenticated crawl should apply the same per-origin separation as for Basic authentication.</p> + <p>Cookie scoping as implemented is not a security boundary. Operators should confine authenticated crawls by topology rather than relying on which cookies are sent where.</p> + + <h3>Advisories and Hardening</h3> + <p>StormCrawler follows the <a href="https://www.apache.org/security/";>ASF Security Team</a>'s policy on when an advisory is issued.</p> + <ul> + <li>We issue an advisory for a vulnerability in a released artifact. This includes low-severity issues, issues that affect only non-default but valid configurations, and issues the project found itself rather than receiving from an external reporter.</li> + <li>We treat improvements to default configuration values and added safeguards as security hardening. These ship in normal releases, described in the release notes, without an advisory.</li> + <li>Vulnerabilities in dependencies follow the ASF's dependency guidance. We update affected dependencies in normal releases and do not usually issue our own advisory for them.</li> + </ul> + <p>As the ASF policy notes, what counts as a vulnerability depends in part on what a project states as expected behaviour, which is why this page describes the crawler's threat model rather than only its reporting process.</p> + + <h3>Summary</h3> + <p>StormCrawler's security model assumes a trusted deployment environment. Users should:</p> + <ul> + <li>Secure configuration files and deployment infrastructure</li> + <li>Restrict Apache Storm® cluster access</li> + <li>Follow best practices for secret and privilege management</li> + <li>Define the crawl scope explicitly with URL filters, and enable only the protocol schemes the crawl needs</li> + <li>Restrict egress by resolved IP address and set a finite content limit</li> + <li>Scope any credentials used for authenticated crawls to a topology confined to their origin</li> + </ul> + + <h2>Asking Questions About Known Security Problems</h2> + <p>Questions about:</p> + <ul> + <li>if a vulnerability applies to your particular application</li> + <li>obtaining further information on a published vulnerability</li> + <li>availability of patches and/or new releases</li> + </ul> + <p>should be addressed to the <a href="https://lists.apache.org/[email protected]";>dev mailing list</a>.</p> + + <h2>Known Security Vulnerabilities</h2> + <p>No known security vulnerability yet.</p> +</div>
