peterxcli opened a new issue, #4992: URL: https://github.com/apache/datafusion-comet/issues/4992
### What is the problem the feature request solves? Comet currently skips native plan rewriting for all Spark Structured Streaming queries: https://github.com/apache/datafusion-comet/blob/8a8fe44753cd2fbacdc01fb4328398b966604823/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala#L561-L564 As a result, Iceberg streaming reads cannot use Comet’s existing native Iceberg scan. This would benefit event-driven pipelines where a main Iceberg event table collects user activity and Spark Structured Streaming continuously updates downstream silver or gold tables. Reference: https://iceberg.apache.org/docs/latest/spark-structured-streaming/ ### Describe the potential solution A reasonable initial scope is native Iceberg micro-batch reads: - Recognize supported Iceberg `MicroBatchScanExec` plans. - Reuse Iceberg’s already-planned, offset-bounded file tasks for each micro-batch. - Rewrite only the supported scan leaf to Comet’s existing native Iceberg scan. - Preserve Spark fallback for unsupported formats or Iceberg features. - Keep checkpoints, offsets, stateful operators, and streaming sink commits managed by Spark and Iceberg. - Add tests covering multiple snapshots, checkpoint restart, progress metrics, and native-plan assertions. Native stateless operators could be enabled around the scan separately once the scan and restart semantics are validated. ### Additional context This is an initial proposal. More investigation is needed around Spark-version-specific streaming plan and progress-reporting APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
