rzo1 opened a new issue, #2086:
URL: https://github.com/apache/stormcrawler/issues/2086
## What happens
`IPFilterRules.parseIPRules()` understands three keywords: `localhost` and
`loopback` map to `InetAddress::isLoopbackAddress`, and `sitelocal` maps to
`InetAddress::isSiteLocalAddress`. For IPv4, `isSiteLocalAddress` covers 10/8,
172.16/12 and 192.168/16 only. Addresses in 169.254.0.0/16, 100.64.0.0/10 and
0.0.0.0/8, and IPv6 addresses in fd00::/8 and fe80::/10, match none of the
keywords, so the example shipped in `crawler-default.yaml`,
`"localhost,sitelocal"`, still allows them. Anything else is parsed as a CIDR
block or a single address, and a value that parses as neither is dropped with a
`LOG.error`, so a typo in an exclude list removes a rule the operator thought
was in force.
## Where
`core/src/main/java/org/apache/stormcrawler/protocol/IPFilterRules.java:111-129`,
keys `http.filter.ipaddress.include` and `http.filter.ipaddress.exclude`. The
example is at `core/src/main/resources/crawler-default.yaml:162` and the same
keyword list is in the class javadoc at lines 45-52.
```java
case "sitelocal":
rules.add(InetAddress::isSiteLocalAddress);
break;
```
## Why it matters
An operator who turns the filter on and copies the shipped example gets
loopback and RFC1918 blocked and nothing else. On a cloud worker the link-local
range is the one that carries an unauthenticated instance metadata service, and
it is reachable from the worker whatever DNS name the crawled link used,
because the check runs on the resolved socket address. The ranges can be
written today as plain CIDR rules, so this is about the vocabulary and the
example being incomplete rather than a missing capability. The interceptor is
also skipped when a proxy is configured, as its javadoc notes.
## Reproduction
Save as
`core/src/test/java/org/apache/stormcrawler/protocol/IPFilterRulesKeywordsTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.protocol;
import static org.junit.jupiter.api.Assertions.assertFalse;
import com.google.common.net.InetAddresses;
import java.net.InetAddress;
import java.util.HashMap;
import java.util.Map;
import org.junit.jupiter.api.Test;
/** The rule keywords and the example shipped in crawler-default.yaml. */
class IPFilterRulesKeywordsTest {
private static InetAddress ip(String address) {
return InetAddresses.forString(address);
}
private static IPFilterRules exclude(Object exclude) {
Map<String, Object> conf = new HashMap<>();
conf.put(IPFilterRules.EXCLUDE_RULES_KEY, exclude);
return new IPFilterRules(conf);
}
/** The example from crawler-default.yaml. */
@Test
void documentedExampleBlocksTheNonRoutableRanges() {
IPFilterRules r = exclude("localhost,sitelocal");
assertFalse(r.accept(ip("169.254.169.254")), "IPv4 link-local");
assertFalse(r.accept(ip("100.64.0.1")), "CGNAT");
assertFalse(r.accept(ip("fd00::1")), "IPv6 unique local");
assertFalse(r.accept(ip("fe80::1")), "IPv6 link-local");
}
/** There is no keyword for link-local addresses. */
@Test
void linkLocalKeywordIsSupported() {
IPFilterRules r = exclude("linklocal");
assertFalse(r.accept(ip("169.254.169.254")));
assertFalse(r.accept(ip("fe80::1")));
}
}
```
Run it:
```
mvn -pl core test -Dtest=IPFilterRulesKeywordsTest
```
Both tests fail on main. It sits next to the existing `IPFilterRulesTest`.
```
[main] ERROR org.apache.stormcrawler.protocol.IPFilterRules - Failed to
parse linklocal as CIDR,
ignoring it while configuring IP rules (http.filter.ipaddress.exclude)
[ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
[ERROR]
IPFilterRulesKeywordsTest.documentedExampleBlocksTheNonRoutableRanges:45
IPv4 link-local ==> expected: <false> but was: <true>
[ERROR] IPFilterRulesKeywordsTest.linkLocalKeywordIsSupported:55 expected:
<false> but was: <true>
```
## Suggested fix
Add a `linklocal` keyword mapping to `InetAddress::isLinkLocalAddress` in
`IPFilterRules.parseIPRules`, and consider `anylocal` and `multicast` alongside
it. Update the class javadoc and the commented example in
`crawler-default.yaml` to
`"loopback,sitelocal,linklocal,100.64.0.0/10,fd00::/8"`. Change the
unparseable-rule branch to throw instead of logging, so a mistyped exclude rule
stops the topology rather than silently widening what it will connect to; that
is a behaviour change for anyone who has a bad rule in their config today, and
the existing `invalidRuleIsIgnored` test in `IPFilterRulesTest` needs to be
updated with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
