jtao15 opened a new issue, #16607: URL: https://github.com/apache/pinot/issues/16607
Decoding a byte array to String without specifying a charset is unsafe [here](https://github.com/apache/pinot/blob/058f7aae2f9cb09caaf9c6f882b1a2f73c3f3fae/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/util/ValueReader.java#L66) because it depends on the JVM’s default charset. If the default is US-ASCII, valid UTF-8 sequences for non-ASCII characters are decoded as the Unicode replacement character **U+FFFD** (UTF-8 bytes **EF BF BD**). This corrupts data if the segment is rebuilt in minion tasks (e.g. PurgeTask), and also inflates the segment size if the same segment is processed multiple times. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
