zhangfengcdt opened a new pull request, #7536: URL: https://github.com/apache/datafusion-comet/pull/7536
## Which issue does this PR close? Closes #7239. ## Rationale for this change A native Iceberg write over a Spark-to-Arrow conversion wrapped the conversion's `executeColumnar` batches in a reader that closes each batch once native has it. The conversion reuses its vectors for the next batch, and closing a struct vector drops its children, so the second batch of a partition fails with `no more field nodes`. Native shuffle had the same bug, fixed in #6689 by reading the child's Arrow stream. On `main` the symptom is hidden since #7325, which copies each batch into fresh vectors. The release branches have neither change, so the bug is live there. ## What changes are included in this PR? `CometIcebergWriteExec` reads a `CometNativeArrowSource` child through `doExecuteAsArrowStream()`, as #6689 does for shuffle. Any other columnar child is wrapped into one Arrow stream per partition, which is what the operator did per task before; the scheduler test probe stays on that path. ## How are these changes tested? A test in `CometIcebergWriteActionSuite` writes a struct column from an RDD-backed DataFrame with `spark.comet.batchSize=2`. With the batch reader as it was before #7325 it fails with `no more field nodes for field a`, and passes with this change. `CometIcebergWriteActionSuite` and `CometIcebergWriteDetectionSuite` pass on Spark 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
