dpol1 opened a new pull request, #2228: URL: https://github.com/apache/stormcrawler/pull/2228
With `http.proxy.manager: org.apache.stormcrawler.proxy.MultiProxyManager` and `http.proxy.rotation` left at its default `ROUND_ROBIN`, some fetches fail when the rotation wraps around the end of the proxy list. The URL gets the status `FETCH_ERROR` with `fetch.exception: java.lang.ArrayIndexOutOfBoundsException` in its metadata. When it happens on a robots.txt fetch, the log shows `Couldn't get robots.txt for ...` and the host gets allow-all rules until the robots error cache expires (1 hour by default). The fetcher threads share the proxy manager of the protocol instance they use. `getRoundRobin()` checks the index and increments it in two separate steps. Two threads can both pass the check on the last index, and the second one then reads past the end of the proxy array. The new test in `MultiProxyManagerTest` calls `getProxy()` from 8 threads. It failed in 3 runs out of 3 on main with `Index 3 out of bounds for length 3`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
