sunchao opened a new pull request, #7718:
URL: https://github.com/apache/datafusion-comet/pull/7718

   ## Which issue does this PR close?
   
   Part of #7032 and the ingress policy in #4764. #7106 covers nested values. 
#6246 handles top-level columns with `spark.sql.parquet.binaryAsString=true`. 
This PR covers top-level columns that reach Comet's schema adapter as binary, 
including reads with an explicit string schema and the setting off.
   
   ## Rationale for this change
   
   A file can store an unannotated Parquet `BINARY` column named `value` with 
an `ARROW:schema` hint of `Dictionary(Int32, LargeBinary)`. For 
`spark.read.schema("value string").parquet(path)`, DataFusion passes the 
dictionary through to Comet. The scalar cast delegates its `LargeBinary` values 
to Arrow, which returns NULL for malformed UTF-8 in safe mode and an error in 
strict mode. For the bytes `66 80 6F`, Spark returns a non-null string that 
renders as `f�o`. The same conversion gap affects `LargeBinary` and 
`BinaryView` when the file field is named `VALUE` and Comet resolves it 
case-insensitively after DataFusion's exact-name check. [DataFusion 
coercion](https://github.com/apache/datafusion/blob/55.2.0/datafusion/datasource-parquet/src/schema_coercion.rs#L83-L110),
 [Arrow 
cast](https://github.com/apache/arrow-rs/blob/59.3.2/arrow-cast/src/cast/string.rs#L345-L375)
   
   ## What changes are included in this PR?
   
   Route these file-to-string casts through the existing Parquet conversion 
used for nested values. It applies JVM-compatible replacement decoding, 
including one U+FFFD for an encoded surrogate. Spark keeps the original bytes 
internally, so `hex(value)` yields `66806F` in Spark and `66EFBFBD6F` in Comet 
under the [documented 
policy](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html#strings-with-non-utf-8-bytes).
   
   The dictionary path decodes values before expanding the keys. Handle an 
empty dictionary as an all-null batch before normalizing its keys, and return 
an Arrow error if replacement bytes exceed the string builder's offset limit. 
The Parquet guide describes the affected reads and the batch size limit tracked 
in #7254.
   
   For a same-name plain `BINARY` field, DataFusion can request `Utf8` directly 
from the reader before the adapter runs. That unchecked path is still open 
under #7032. Map-key identity is tracked in #7360.
   
   ## How are these changes tested?
   
   - `ParquetReadSuite` extends the dictionary fixture to read 
`Dictionary(Int32, LargeBinary)` with an explicit string schema and 
`binaryAsString=false`. It compares collected strings with Spark, requires a 
`CometNativeScanExec`, and checks `hex(value)` and `value IS NULL` for a 
repeated malformed value, valid bytes and a genuine null.
   - A Rust Parquet round trip exercises case-insensitive resolution for 
`LargeBinary` and `BinaryView`, an exact-name `Dictionary(Int8, LargeBinary)`, 
and an all-null dictionary. It checks `café`, NULL and `41 ED A0 80 42` → `A�B`.
   
   Checked Rust and Scala formatting and `git diff --check` locally. Test 
execution is delegated to hosted CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to