rich7420 commented on code in PR #6246:
URL: https://github.com/apache/datafusion-comet/pull/6246#discussion_r4502866398
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -198,10 +376,34 @@ pub(crate) fn init_datasource_exec(
.flat_map(|s| s.fields().iter())
.map(|f| Arc::new(Field::new(f.name(), f.data_type().clone(),
f.is_nullable())))
.collect();
- let table_schema = TableSchema::builder(base_schema)
- .with_table_partition_cols(partition_fields)
+ let output_table_schema = TableSchema::builder(Arc::clone(&base_schema))
+ .with_table_partition_cols(partition_fields.clone())
+ .build();
+ // A scan that projects no string column has nothing to read as Binary, so
it keeps the
+ // projection-indices path below.
+ let binary_as_string = binary_as_string
+ && required_schema
+ .fields()
+ .iter()
+ .any(|field| is_binary_as_string_read(field.data_type()));
+ let read_base_schema = if binary_as_string {
+ binary_as_string_read_schema(&base_schema)
Review Comment:
@manuzhang thanks for the update.
Dictionary string hints still slip through here and panic on malformed bytes
in debug builds. Let's handle both shapes in one small helper and test PLAIN
and dictionary-encoded pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
