hassaanch23 commented on issue #12047: URL: https://github.com/apache/arrow-rs/issues/12047#issuecomment-6692962013
For step 1, #11144 is that standalone fix, for #11140. It has been open since September, and I rebased it onto `main` last week. It takes the same approach as #12064: - The array readers turn on `set_validate_utf8` based on the Arrow type they build: `Utf8`, `LargeUtf8`, `Utf8View`, or a dictionary of one of those. - The dictionary decoder takes its value type from whether it validates. String dictionaries over unannotated columns, `Dictionary<_, Utf8View>` included, now get string values. The tests read valid and invalid data into each string type, over five kinds of pages: dictionary, a dictionary that falls back to PLAIN, PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY. They also cover a `JSON` column, which is read as `Utf8` but was never validated. @kita-renji tested it against 165 cases, and @Rich-T-kid is reviewing it now. If that works for everyone, it could land as step 1, and the `InvalidUtf8` option could then build on the same flag. cc @andygrove, since #12064 makes the same change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
