andygrove opened a new pull request, #7119: URL: https://github.com/apache/datafusion-comet/pull/7119
## Which issue does this PR close? Part of #7118. ## Rationale for this change Since #6786, Comet converts a Text scan on a broadcast join's build side to Arrow by default. Text files are where invalid UTF-8 most often shows up, for example a file written in Latin-1, and the conversion replaces those bytes with `U+FFFD`. A query that ran entirely in Spark on 1.1.0 can now return different `hex`, `length` or join results. The compatibility guide says invalid UTF-8 "does not occur for text read from Parquet or produced by string expressions", which reads as if a query over Parquet and Text files is unaffected. #7118 also asks whether to turn the conversion off by default for 1.2.0. This PR documents the current behavior either way. If the default changes, the sentence about the default changes with it. ## What changes are included in this PR? - `compatibility/index.md`: name Text files in another encoding and strings cast from binary as sources of invalid UTF-8, say that a Text scan on a broadcast build side is converted by default, and give the config that keeps it on Spark. - `datasources.md`: in the paragraph about the build-side conversion, link to the invalid UTF-8 section and say that a Text file in another encoding can give different results. ## How are these changes tested? Documentation only. Prettier passes. The behavior described is the one reproduced in #7118. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
