rzo1 opened a new pull request, #2168: URL: https://github.com/apache/stormcrawler/pull/2168
The protocols read the headers to add to an outgoing request from `<protocol.md.prefix>set-header`. The fetcher bolts merge every response header into the tuple metadata under that same prefix, lowercased, so a server returning a header named `set-header` writes exactly that key. Where the operator persists or transfers `protocol.` keys — internals.adoc recommends it for `protocol.etag` and `protocol.set-cookie`, and the lists accept wildcards — the forged value is carried into the next fetch, and the crawled site chooses headers sent afterwards, including on requests to other hosts. The merge now drops the key the way it already drops `robots.crawl.delay`, then puts back the value configured for the URL. `SET_HEADER_BY_REQUEST` becomes public so the bolts can name it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
