rzo1 opened a new pull request, #2167: URL: https://github.com/apache/stormcrawler/pull/2167
`HostURLFilter` kept the last source URL, host and domain in three plain instance fields and compared by identity. `StatusEmitterBolt` builds one `URLFilters` per bolt and `FetcherBolt` runs `fetcher.threads.number` threads through it, so those fields were written and read concurrently with no synchronisation, and the three writes were not atomic with each other. A thread could see its own URL in `previousSourceUrl` next to another thread's host in `previousSourceHost`, and compare the candidate against the wrong origin — dropping a same-host URL or admitting one outside the source host. The cache only saved one `PaidLevelDomain.getPLD` call, so it is removed rather than synchronised. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
