leehaut opened a new pull request, #9054: URL: https://github.com/apache/hop/pull/9054
Fix https://github.com/apache/hop/issues/9032 ## Summary Text File Output encodes each value, separator, enclosure and newline with `String.getBytes(encoding)`. Java charsets that emit a byte order mark (`UTF-16`, `x-UTF-16LE-BOM`, `X-UTF-32BE-BOM`, `X-UTF-32LE-BOM`, and aliases such as `UnicodeBig` and `UnicodeLittle`) prefix every one of those calls, so readers keep `U+FEFF` between fields. Fragments are now encoded with the matching charset that does not write a mark (`UTF-16` content is `UTF-16BE`, and so on). The mark is written once, when a new file is opened or when appending to an empty file. Appending to a file that already has content does not write another mark. Non-string fields, which `ValueMetaBase` still encodes with the file charset, have that one leading mark removed. `ISO-2022-KR`, `x-ISO-2022-CN-CNS` and `x-ISO-2022-CN-GB` can leave a fragment in shift-out. A separator encoded on its own is then read as a double-byte character. A fragment that ends shifted out is closed with shift-in. ## Test plan - [x] `TextFileOutputEncodingTest` round-trips every JVM charset that can encode the sample, and checks that BOM charsets have exactly one leading mark - [x] Covers aliases (`UnicodeBig`, `unicode`, `UnicodeLittle`, `UTF-32BE-BOM`, `UTF-32LE-BOM`), a CJK field (including the ISO-2022 charsets above), append, header, DOS line endings, an integer field, a literal `U+FEFF`, and right-padding - [x] In Hop GUI, run a Data Grid of `V001;Maria` / `V002;Jose` / `V003;Ana` through Text File Output with encoding `x-UTF-16LE-BOM`, no header, separator `;`. The file should read back as those three lines, with a single `FF FE` at the start -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
