nicoscandolo opened a new issue, #2250:
URL: https://github.com/apache/stormcrawler/issues/2250
## What happens
The fetcher stores the response headers under `protocol.md.prefix`
(`protocol.` in `crawler-default.yaml`). `JSoupParserBolt` reads the mime type
with that prefix, but it hands the metadata as is to `CharsetIdentification`,
which reads `metadata.getFirstValue(HttpHeaders.CONTENT_TYPE)` without the
prefix. In a shipped topology the charset declared by the server is therefore
never found.
With `detect.charset.fast: false` (the default), `getCharset` uses the
declared charset when the HTTP header and the meta charset agree. Since the
header is never seen they can never agree, so the meta charset is only passed
to ICU as a hint and ICU guesses from the first `detect.charset.maxlength`
bytes. When those bytes are plain ASCII, as with a long head, ICU returns
ISO-8859-1. A page served with `Content-Type: text/html; charset=UTF-8` and
`<meta charset="utf-8">` is then decoded as Latin-1, and every non-ASCII
character is stored as two or three wrong characters, e.g. `café` becomes
`café`. With `detect.charset.fast: true` the header would take precedence, but
it is not seen either.
## Where
`core/src/main/java/org/apache/stormcrawler/util/CharsetIdentification.java:137`
```java
return
getCharsetFromContentType(metadata.getFirstValue(HttpHeaders.CONTENT_TYPE));
```
`core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:288-294`
passes `metadata` without the prefix, while line 230 reads the same header with
it:
```java
String mimeType =
metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMetadataPrefix);
```
This is the same pattern as #2100 for `SiteMapParserBolt`.
## How to reproduce
`protocol.md.prefix: "protocol."`, `detect.charset.maxlength: 2048`,
metadata `protocol.Content-Type: text/html; charset=UTF-8`, and a UTF-8 page
that starts with `<meta charset="utf-8">` followed by more than 2048 bytes of
ASCII. The text emitted by `JSoupParserBolt` is `Whatâs new: café ...` instead
of `What’s new: café ...`. The test in the PR reproduces it.
## Fix
Read the content type under the protocol prefix: add overloads of
`getCharset` and `getCharsetFast` that take the prefix, and pass
`protocolMetadataPrefix` from `JSoupParserBolt`. The existing signatures keep
their behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
