sunchao commented on code in PR #7100:
URL: https://github.com/apache/datafusion-comet/pull/7100#discussion_r4315153683
##########
native/core/src/execution/operators/iceberg_write.rs:
##########
@@ -1488,6 +1488,22 @@ fn parse_iceberg_schema(json: &str) ->
DFResult<IcebergSchemaRef> {
let schema: IcebergSchema = serde_json::from_str(json).map_err(|e| {
DataFusionError::Internal(format!("Failed to parse iceberg schema
JSON: {e}"))
})?;
+ // iceberg-java writes no column for a top-level `unknown` field.
+ let fields = schema.as_struct().fields();
+ let kept: Vec<_> = fields
+ .iter()
+ .filter(|f| *f.field_type != Type::Primitive(PrimitiveType::Unknown))
Review Comment:
[P2] Preserve or decline `unknown` partition sources before pruning them.
Iceberg permits a v3 schema `(id INT, u unknown)` with an identity partition on
`u`. With native writes enabled and a native child, the widened gate admits
this schema, but this filter removes `u` from the schema subsequently passed to
`PartitionValueCalculator::try_new`. Its field-ID projector then fails with
`Field not found`, aborting an insert that previously fell back successfully to
iceberg-java. Spark writes the rows under `u=null`. Could the gate decline
specs referencing a pruned field, or could partition calculation retain the
logical source while pruning only the Parquet input? Please include a
regression partitioned on `u` itself.
Evidence: Created a v3 table using `new Schema(optional(1, "id",
IntegerType.get()), optional(2, "u", UnknownType.get()))` and
`PartitionSpec.builderFor(schema).identity("u").build()`, with
`write.distribution-mode=none` and `write.spark.fanout.enabled=true`. Stock
Spark 4.1.3/Iceberg 1.11 completed `INSERT INTO review.default.t VALUES (1,
NULL), (2, NULL)`, returned `[[1,null], [2,null]]`, and produced files under
`u=null/`. Java API probes also accepted the spec on Iceberg 1.10. A Rust probe
compiled against this checkout's dependencies used the exact
`parse_iceberg_schema` implementation and included the actual
`iceberg_partition_value.rs`: calculator construction returned `Ok(())` with
the original schema and `Err(Unexpected => Field not found; field_id: 2)` after
pruning. `run_write_task` passes that pruned schema into the same calculator
through `PartitionSplitter`. Before this PR, `UNKNOWN` in
`UnsupportedWriteTypeIds` kept this write on iceberg-java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
