NoahKusaba opened a new issue, #2974: URL: https://github.com/apache/iceberg-rust/issues/2974
### Apache Iceberg Rust version None ### Describe the bug Datafusion ExecutionPlan must declare the schema() it produces as part of the Trait method. Currently IcebergTableProvider::insert_into() passes it's Arrow table-schema to IcebergWriteExec for building PlanProperties; however, the operator output is a single data_files: Utf8 column. This hasn't been an issue as execute() returns a RecordBatchStreamAdapter with self.result_schema == (make_result_schema()) => The correct data_files schema. IcebergCommitExec builds it's own schema, and it locates the column by name on the runtime batch, so single-process inserts aren't affected. It breaks where the declared schema is consumed independently of the batches such as: physical-plan serialization for distributed execution. The parameter can't be set correctly by any caller: the output shape is fixed and doesn't depend on the input. ### To Reproduce Pass an IcebergTable + ExecutionPlan + TableSchema to IcebergWriteExec. Read .schema() and notice it returns the original TableSchema ### Expected behavior IcebergWriteExec::schema() returns the schema it actually emits: a single non-nullable data_files: Utf8 column, matching what execute() streams and what IcebergCommitExec reads. Since the output shape is fixed and independent of the input, the schema: ArrowSchemaRef parameter should be removed from IcebergWriteExec::new() and PlanProperties derived from make_result_schema() , which is already used by result_schema . ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
