dpol1 commented on PR #2050: URL: https://github.com/apache/stormcrawler/pull/2050#issuecomment-5316331605
Thanks — the concerns from my previous comment are addressed: temporary redirects are excluded and the ERROR/FETCHED behaviour is now pinned. One remaining observation about the resulting contract: the PR is described as deleting permanent redirects, but a meta refresh carries no permanence signal. Could we narrow this feature to HTTP 301/308 only and put it behind `status.updater.delete.redirections` (default false)? Deletion should also require redirect following to be enabled; otherwise `redirections.allowed=false` can suppress the target while deleting the source. A characterization test should also pin REDIRECTION without `fetch.statusCode` as no deletion. Daily re-emission would still affect topologies opting in. I'd keep that as a separate scheduler/idempotency follow-up rather than consider it solved by the default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
