rich7420 commented on code in PR #6246:
URL: https://github.com/apache/datafusion-comet/pull/6246#discussion_r4471220471
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -64,6 +69,178 @@ pub(crate) struct CometParquetScanOptions {
pub prefetch_next_file_metadata: bool,
}
+/// Keep top-level Spark string fields as binary while arrow-rs decodes
Parquet. arrow-rs only
+/// validates UTF-8 for columns annotated as strings, so decoding an
unannotated BINARY column that
+/// `spark.sql.parquet.binaryAsString=true` types as a string straight into
Arrow Utf8 would admit
+/// invalid UTF-8. A projection added at the scan boundary converts the binary
arrays to valid Utf8
+/// with Spark-compatible replacement semantics.
+fn binary_as_string_read_schema(schema: &SchemaRef) -> SchemaRef {
+ Arc::new(Schema::new_with_metadata(
+ schema
+ .fields()
+ .iter()
+ .map(binary_as_string_read_field)
+ .collect::<Vec<_>>(),
+ schema.metadata().clone(),
+ ))
+}
+
+/// The field `binary_as_string_read_schema` reads a top-level `field` as.
Spark's StringType is
+/// Arrow Utf8.
+fn binary_as_string_read_field(field: &FieldRef) -> FieldRef {
+ if is_binary_as_string_read(field.data_type()) {
+ Arc::new(field.as_ref().clone().with_data_type(DataType::Binary))
Review Comment:
Please preserve Spark’s rejection of `FIXED_LEN_BYTE_ARRAY(3)` as `STRING`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
