CTTY commented on code in PR #1511:
URL: https://github.com/apache/iceberg-rust/pull/1511#discussion_r2223962611
##########
crates/iceberg/src/arrow/value.rs:
##########
@@ -440,10 +440,12 @@ impl PartnerAccessor<ArrayRef> for ArrowArrayAccessor {
Ok(schema_partner)
}
+ // todo generate field_pos in datafusion instead of passing to here
Review Comment:
This method is used when using `ParquetWriter` to write `RecordBatch`. When
it's counting nan values, it will need to walk through both `RecordBatch`'s
schema and Iceberg schema in a partner fashion:
https://github.com/apache/iceberg-rust/blob/9787140165a15afaf50fc4742484d22c2230bd60/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L528
Basically the call stack is `NanValueCountVisitor::compute` ->
`visit_struct_with_partner` -> `ArrowArrayAccessor::field_partner` ->
`get_field_id`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
