corgy-w opened a new pull request, #13011: URL: https://github.com/apache/seatunnel/pull/13011
### What The HTTP source now hands the response body to the format deserializer as **UTF-8 bytes** instead of encoding it with the JVM default charset. ### Why `HttpSourceReader` collects the raw response body as a `String` (`contentData`, optionally extracted through `content_json` / `json_field`) and converts it to `byte[]` for the format deserializer: ```java // before deserializationCollector.collect(contentData.getBytes(), output); ``` `String.getBytes()` uses `Charset.defaultCharset()`, but every format deserializer on the other side of this boundary (JSON/Jackson, CSV, Debezium JSON, text) decodes UTF-8. On a JVM whose default charset is not UTF-8, the two disagree and non-ASCII data is lost: | JVM default charset | `"中文值".getBytes()` (measured on JDK 8) | effect downstream | | --- | --- | --- | | UTF-8 (Linux/macOS default) | `[-28, -72, -83, ...]` | correct — this is why it went unnoticed | | GBK | `[-42, -48, -50, -60, -42, -75]` | decoded as UTF-8 → mojibake / parse error | | Cp1252 (Windows JDK 8/11 default) | `[63, 63, 63]` = `???` | **irreversible data loss** | The rest of this connector is already explicit about UTF-8 (`HttpClientProvider` / `HttpHelper` decode responses with `"UTF-8"`), and the repository has 66 other `getBytes(StandardCharsets.UTF_8)` call sites, so the missing charset here is an oversight rather than a convention. ### Fix One line plus the import: ```java deserializationCollector.collect(contentData.getBytes(StandardCharsets.UTF_8), output); ``` No behaviour change on UTF-8 JVMs — the bytes produced are identical. ### How verified - New test in the existing `HttpSourceReaderInternalPollNextTest` (no new test class): it stubs the response body with non-ASCII content, injects a mock `DeserializationSchema`, captures the `byte[]` actually handed to the deserializer with `ArgumentCaptor`, and asserts that decoding it as UTF-8 returns the original body. A mock schema is used on purpose: `SimpleTextDeserializationSchema` would re-decode with the same default charset and hide the problem. - `mvn -o -pl seatunnel-connectors-v2/connector-http/connector-http-base test -Dtest=HttpSourceReaderInternalPollNextTest` → `Tests run: 4, Failures: 0, Errors: 0`. - `./mvnw -o -q -pl ... spotless:apply` → no reformatting needed. **Test limitation (worth knowing before merge):** this project pins `-Dfile.encoding=UTF-8` for surefire (`pom.xml:174`), so the new test passes on the pre-fix code as well — under the pinned charset the old and new code produce the same bytes. The regression is only observable on a JVM with a non-UTF-8 default, for example: ```bash java -Dfile.encoding=Cp1252 ... # JDK 8 honours it; "中文值".getBytes() -> [63, 63, 63] ``` (NB: on JDK 11+ `file.encoding` cannot be switched at launch, and `Charset.defaultCharset()` is fixed, so the reproduction needs JDK 8 or a real Windows JVM.) I kept the test because it pins the contract at the boundary, but I am explicitly not claiming it fails before the fix. If you would rather have no test than a non-failing one, say so and I will drop it and leave the one-line fix. ### Follow-ups (same defect, not touched here) - `connector-graphql/.../GraphQLUtil.java:148,152,157` — `data.getBytes()` for the same deserializer boundary. - `connector-starrocks/.../client/HttpHelper.java:83` — `new ByteArrayEntity(postBody.getBytes())` for the stream-load request body; this one writes the mis-encoded bytes into the target table, so it is the worst of the three. Happy to send those as separate PRs if this one lands. ### Compatibility No option name, default value, or public API change. Behaviour on UTF-8 JVMs is byte-for-byte identical, so existing jobs are unaffected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
