parthchandra opened a new pull request, #2668: URL: https://github.com/apache/iceberg-rust/pull/2668
## Which issue does this PR close? - part of #https://github.com/apache/iceberg-rust/issues/2607. ## What changes are included in this PR? Implements the _partition metadata column for table scans. This is a struct column whose type is the union of all partition fields across all partition specs (handling partition evolution). Each row gets the partition values for its data file. - Adds compute_unified_partition_type() to compute the union of partition fields across all specs (equivalent to Java's Partitioning.partitionType()) - Adds PartitionColumnConstant and build_partition_column_constant() for pre-computing the struct values per file - Adds ColumnSource::AddStructConstant variant to RecordBatchTransformer for materializing struct columns - Threads the unified partition type through scan planning and populates the constant in FileScanTask - Pipeline detects RESERVED_FIELD_ID_PARTITION in projected fields and injects the struct constant ## Are these changes tested? Because we do not have write support yet, I made the corresponding change to comet and then tested by adding tests in Comet which uses iceberg-java to write files and then iceberg-rust to read them back. https://github.com/parthchandra/datafusion-comet/blob/iceberg-metadata-columns/spark/src/test/resources/sql-tests/iceberg/metadata_column_partition.sql -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
