Visorgood opened a new issue, #7622:
URL: https://github.com/apache/datafusion-comet/issues/7622
### Describe the bug
In the split Iceberg write plan, both write nodes are printed with Spark's
`!` prefix, which marks a node whose input attributes are missing:
```
IcebergCommit IcebergBatchWrite(table=local.db.dst, format=PARQUET), ...
+- AdaptiveSparkPlan isFinalPlan=false
+- !CometIcebergWrite [iceberg_commit_message#NN], ...
```
Both nodes take `output` (the `iceberg_commit_message` attribute) as a
constructor argument but don't override `producedAttributes`. Spark therefore
counts that attribute in `references`, and since no child produces it,
`missingInput` is non-empty. The write itself runs correctly; the plan only
looks broken. Spark's own `AppendData` node (Comet disabled) has no `!`.
This is the same class of issue as #2214 (`CometHashAggregate`) and #2789
(`CometWindowExec`).
### Steps to reproduce
Comet `main` at `4de670439`, Spark 4.1.3, Iceberg 1.11.0, `spark-shell` with
a Hadoop catalog named `local` and `spark.comet.write.iceberg.enabled=true`:
```scala
spark.sql("CREATE TABLE local.db.src (id INT, amount INT) USING iceberg")
spark.sql("INSERT INTO local.db.src VALUES (1, 10), (2, 20)")
spark.sql("CREATE TABLE local.db.dst (id INT, amount INT) USING iceberg")
spark.sql("CREATE TABLE local.db.dst_orc (id INT, amount INT) USING iceberg
TBLPROPERTIES ('write.format.default'='orc')")
// native write
spark.sql("EXPLAIN INSERT INTO local.db.dst SELECT * FROM local.db.src")
// ORC is not written natively, so the write stays on iceberg-java
spark.sql("EXPLAIN INSERT INTO local.db.dst_orc SELECT * FROM local.db.src")
```
Native write:
```
== Physical Plan ==
IcebergCommit IcebergBatchWrite(table=local.db.dst, format=PARQUET),
IcebergWrite(table=local.db.dst, format=PARQUET),
org.apache.comet.iceberg.IcebergWriteStrategy$$Lambda/0x00000070021c8ba8@1b766d44
+- !CometIcebergWrite [iceberg_commit_message#303],
/tmp/iceberg-warehouse/db/dst/data, ICEBERG_WRITER_UNPARTITIONED
+- CometIcebergNativeScan [id#292, amount#293],
/tmp/iceberg-warehouse/db/src/metadata/v2.metadata.json,
IcebergScan(table=local.db.src, schemaId=0, snapshotId=7721505751242735037,
branch=null, filters=, runtimeFilters=, groupedBy=), 2
```
Fallback to iceberg-java:
```
== Physical Plan ==
IcebergCommit IcebergBatchWrite(table=local.db.dst_orc, format=ORC),
IcebergWrite(table=local.db.dst_orc, format=ORC),
org.apache.comet.iceberg.IcebergWriteStrategy$$Lambda/0x00000070021c8ba8@4ba48bc7
+- !IcebergWrite IcebergBatchWrite(table=local.db.dst_orc, format=ORC),
[iceberg_commit_message#322], PlainIcebergWrite
+- *(1) CometColumnarToRow
+- CometIcebergNativeScan [id#311, amount#312],
/tmp/iceberg-warehouse/db/src/metadata/v2.metadata.json,
IcebergScan(table=local.db.src, schemaId=0, snapshotId=7721505751242735037,
branch=null, filters=, runtimeFilters=, groupedBy=), 2
```
### Expected behavior
No `!` on `IcebergWrite` and `CometIcebergWrite`: the commit-message
attribute is produced by the node itself, so `missingInput` should be empty.
### Additional context
Proposed fix: `override def producedAttributes: AttributeSet = outputSet` in
`IcebergWriteExec` and `CometIcebergWriteExec`, as `CometProjectExec` already
does, plus a test that asserts `missingInput.isEmpty` for both nodes in an
existing `CometIcebergWriteActionSuite` write.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
