dpol1 commented on issue #2305: URL: https://github.com/apache/stormcrawler/issues/2305#issuecomment-6691823155
Thanks @rzo1, makes sense to wait for it and use the same config. That limit applies in the parser though, and here `http.content.limit` has already cut the response in the protocol before it gets there. So we'd still need the fetch to allow more for sitemaps. One SC key could set both `setMaxBytes` and the per-URL `http.content.limit`. I can look at it once crawler-commons is released. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
