rzo1 commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280320379
##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
public final String filterPathRepeat(String urlToFilter) {
// check whether a path element is repeated N times
String[] paths = urlToFilter.split("/");
- if (paths.length <= 4) return urlToFilter;
+
+ final int minimumPathLength = 5;
Review Comment:
Either a constant or a configurable options (if it makes sense to be able to
configure it), imho.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
