rzo1 opened a new pull request, #2027: URL: https://github.com/apache/stormcrawler/pull/2027
Closes #2026. `RobotRulesParser.setConf()` never adds `http.agent.name` to `agentNames` when `http.robots.agents` is non-empty: - if the advertised name is listed first, `index++` skips it and the loop appends only `agents[1..]`, so the name is dropped entirely; - if it is not listed first, the code logs *"Agent we advertise (X) not listed first"* and appends the configured list, still without adding the name. `agentNames` is only written in the empty-list branch and in that loop, which contradicts the in-code comment: *"If both are present, our agent-string should be the first one we advertise to robots-parsing."* **Impact.** With `http.agent.name: mybot` and `http.robots.agents: mybot,mybot-uk`, the match set handed to crawler-commons is `[mybot-uk]`. A `User-agent: mybot` group in a robots.txt does not match and the parser falls back to `*`. Rules addressed specifically to the crawler are ignored, which is a compliance issue rather than a cosmetic one. **Fix.** Add `agentNames.add(agentName)` before the append loop. `agentNames` is a `LinkedHashSet`, so the existing `index++` stays as-is and no duplicate is introduced. Two commits: the failing reproducer first, then the fix - happy to squash before merge. ### For all changes - [x] Is there a issue associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are trying to resolve? - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? - two commits, red reproducer then fix - [ ] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`? - not run yet ### For code changes - [ ] Have you ensured that the full suite of tests is executed via `mvn clean verify`? - only the `protocol` robots tests were run locally - [x] Have you written or updated unit tests to verify your changes? - new `RobotRulesParserAgentNamesTest` (4 tests, 3 of which fail without the fix) - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? - no new dependencies - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file? - n/a - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file? - n/a -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
