alamb commented on issue #27065: URL: https://github.com/apache/datafusion/issues/27065#issuecomment-6632076560
I reproduced this and confirmed it is a regression (relative to every released version), so I have updated the title to match the other regression issues. I used `datafusion-cli` so the same file could be read by each released binary. The file is a `list<binary>` column whose second element is the single byte `0xff`, written on `main` with: ```sql COPY (SELECT make_array(X'6f6e65', X'ff') AS s) TO 'nested_binary.parquet'; -- DESCRIBE 'nested_binary.parquet' -> s: List(Binary) ``` and read back through a table schema that declares the elements as strings: ```sql CREATE EXTERNAL TABLE t (s VARCHAR[]) STORED AS PARQUET LOCATION 'nested_binary.parquet'; SELECT s, arrow_typeof(s) FROM t; SELECT octet_length(s[2]) FROM t; ``` | Version | Result | |---|---| | 53.1.0 | `Arrow error: Invalid argument error: Encountered non UTF-8 data: invalid utf-8 sequence of 1 bytes from index 3` | | 54.1.0 | same error | | 55.0.0 | same error | | 55.1.0 | same error | | `main` (release build, 181a87af61) | succeeds: returns `[one, �]` typed `List(Utf8View)`, and `octet_length(s[2])` is `1`, i.e. the invalid byte came through as a string unvalidated | So every released version rejects the file and `main` (and `branch-56`, which also contains 3a6a18cd81 / #25193) returns an unvalidated string array. #25193 is not in any released tag yet, so this only affects 56.0.0 if it ships as is. One more data point on the "same gap exists for top-level fields" note: a top-level `BINARY` column holding `0xff`, read through `CREATE EXTERNAL TABLE t2 (b VARCHAR)`, makes `datafusion-cli` exit with a segfault (exit code 139) on `SELECT b FROM t2`, on every version I tried from 53.1.0 through `main`, with `schema_force_view_types` either on or off. (`SELECT count(*)` works.) That part is pre-existing rather than a regression, but it shows the unvalidated-string path is already reachable at the top level and worth closing at the same time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
