rzo1 opened a new issue, #2026: URL: https://github.com/apache/stormcrawler/issues/2026
`RobotRulesParser.setConf()` never adds `http.agent.name` to `agentNames` when `http.robots.agents` is non-empty: - if the advertised name is listed first, `index++` skips it and the loop appends only `agents[1..]`, so the name is dropped entirely; - if it is not listed first, the code logs *"Agent we advertise (X) not listed first"* and appends the configured list, still without adding the name. `agentNames` is only ever written in the empty-list branch and in that loop, which contradicts the in-code comment: *"If both are present, our agent-string should be the first one we advertise to robots-parsing."* ### Impact With `http.agent.name: mybot` and `http.robots.agents: mybot,mybot-uk`, the match set handed to crawler-commons is `[mybot-uk]`. A `User-agent: mybot` group in a robots.txt does not match and the parser falls back to `*`. Rules addressed specifically to the crawler are ignored, which is a compliance issue rather than a cosmetic one. ### Reproducer With a robots.txt of: ``` User-agent: mybot Disallow: /restricted/ User-agent: * Disallow: / ``` and `http.agent.name: mybot`, `http.robots.agents: mybot,mybot-uk`, `/index.html` is reported as disallowed because the wildcard group is selected. Nothing in `core/src/test` currently references `http.robots.agents`. ### Fix Add `agentNames.add(agentName)` before the append loop. `agentNames` is a `LinkedHashSet`, so the existing `index++` can stay and no duplicate is introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
