dpol1 opened a new pull request, #2157: URL: https://github.com/apache/stormcrawler/pull/2157
Closes #2092. A page that opens a `<meta charset="` declaration and never closes it made `getCharsetFromMeta` retry with a window ten bytes larger, once per ten bytes of body, until the stack overflowed. The detection ran outside the parser bolt's try block, so the error took the worker down, and the replayed tuple took the restarted worker down again. Two changes: - `CharsetIdentification`: when the declaration is cut by the detection window, the window is extended once, by 64 bytes, room for any registered charset name. A declaration still open after that is broken content, as before. - `JSoupParserBolt`: charset detection now runs inside the existing try, so a failure there is a parse error for that URL. Tests: the open declaration at 20 KB and 400 KB on a 1 MB stack, the same with full-content detection, a declaration cut by the window that is still read (#870), and an error injected into the detection checked against the status stream. ### For all changes - [x] Is there a issue associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are trying to resolve? - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? (two commits, one per change) - [x] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`? ### For code changes - [x] Have you ensured that the full suite of tests is executed via `mvn clean verify`? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
