hareshkh commented on code in PR #3423:
URL: https://github.com/apache/iceberg-rust/pull/3423#discussion_r4253129029
##########
crates/iceberg/src/writer/file_writer/parquet_writer.rs:
##########
@@ -605,13 +605,44 @@ fn resolve_writer_properties(
.build())
}
+/// Fails if a column that `schema` marks required holds a null.
+///
+/// The Arrow schema the file is written with comes from `schema`, so required
columns are
+/// written as Parquet `REQUIRED`, and a null in one would be written as
whatever value lies
+/// beneath it, typically zero or empty. A batch's own schema may still
declare such a column
+/// nullable, and a dictionary column may hold a null value even when the
column is
+/// declared non-nullable, so this counts the logical nulls of every required
column.
+///
+/// The Parquet writer pairs the schema's fields with the batch's columns by
position and
+/// ignores any extra trailing columns, so this pairs them the same way.
+fn check_required_columns(schema: &Schema, batch: &arrow_array::RecordBatch)
-> Result<()> {
Review Comment:
Similar to
https://sourcegraph.com/r/github.com/apache/datafusion/-/blob/datafusion/physical-plan/src/execution_plan.rs?L1956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
