devinjdangelo commented on code in PR #9431:
URL: https://github.com/apache/arrow-datafusion/pull/9431#discussion_r1510086655
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -157,6 +157,22 @@ impl FileScanConfig {
})
}
+ /// Projects only file schema, ignoring partition columns
+ pub(crate) fn projected_file_schema(&self) -> SchemaRef {
Review Comment:
This method looks good, but it would be nice if we could leverage
`file_column_projection_indices` (which CsvOpener uses) so we aren't
duplicating the logic to exclude the partition columns.
I think in general we could make Csv, Json, and Arrow file opening /
configuring more consistent. We can cut follow on tickets for this.
##########
datafusion/sqllogictest/test_files/ddl.slt:
##########
@@ -490,20 +490,28 @@ DROP TABLE csv_with_timestamps
statement ok
CREATE EXTERNAL TABLE csv_with_timestamps (
- name VARCHAR,
- ts TIMESTAMP,
c_date DATE,
+ name VARCHAR,
Review Comment:
Are the changes to this test related / required? I think it would be better
to leave this test as-is and add a new one if required so we can validate that
we haven't inadvertently changed how CSV partitioned tables are read.
##########
datafusion/sqllogictest/test_files/insert_to_external.slt:
##########
@@ -195,9 +195,15 @@ INSERT INTO partitioned_insert_test_json values (1, 2),
(3, 4), (5, 6), (1, 2),
----
6
-# Issue open for this error:
https://github.com/apache/arrow-datafusion/issues/7816
-query error DataFusion error: Arrow error: Json error: Encountered unmasked
nulls in non\-nullable StructArray child: Field \{ name: "a", data_type: Utf8,
nullable: false, dict_id: 0, dict_is_ordered: false, metadata: \{\} \}
+query TT
select * from partitioned_insert_test_json order by a,b
Review Comment:
:partying_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
