andygrove opened a new pull request, #4928:
URL: https://github.com/apache/datafusion-comet/pull/4928
## Summary
Suppress a spurious `WriteFilesExec is not supported` fallback reason
emitted during native Parquet writes.
Spark's `V1Writes` rule wraps native writes as `DataWritingCommandExec(cmd,
WriteFilesExec(child))`. `CometExecRule` converts the outer
`DataWritingCommandExec` to `CometNativeWriteExec` and unwraps the inner
`WriteFilesExec` inside `convertToComet`, so the write runs natively end-to-end.
However, `transformUp` visits `WriteFilesExec` before its parent. No case in
the outer `match` handles it, so it falls through to the generic
`${op.nodeName} is not supported` tag at the bottom of `convertNode`. The
`WriteFilesExec` instance is then discarded when the parent converts, so the
executed plan looks correct — but with
`spark.comet.logFallbackReasons.enabled=true` the log emits a misleading
warning:
```
Comet cannot accelerate WriteFilesExec because: WriteFilesExec is not
supported
```
Add `WriteFilesExec` to the list of operators that should never be tagged
with a generic fallback reason (alongside `CometPlan`, `AQEShuffleReadExec`,
`BroadcastExchangeExec`, `BroadcastQueryStageExec`, `AdaptiveSparkPlanExec`,
`ExecutedCommandExec`, `V2CommandExec`).
## Test plan
- [ ] Enable `spark.comet.logFallbackReasons.enabled=true` and a native
Parquet write via `df.write.parquet(...)`; confirm the `WriteFilesExec is not
supported` warning no longer appears.
- [ ] Confirm existing `CometParquetWriterSuite` tests still pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
