alexprosak opened a new issue, #13823: URL: https://github.com/apache/iceberg/issues/13823
### Feature Request / Improvement When using Iceberg as a source in a streaming query with `Trigger.AvailableNow`, it falls back to single batch execution behavior (a.k.a. `Trigger.Once`) and ignoring rate limits to process everything in one go via the [AvailableNowDataStreamWrapper](https://github.com/apache/spark/blob/7f3c7047ae4cc86fca1f9fa82d8bc46073dc833d/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AvailableNowDataStreamWrapper.scala#L72), which always returns `ReadLimit.AllAvailable` for the limit: Reading with a rate limit + `Trigger.AvailableNow` ``` spark.readStream .format("iceberg") .option("streaming-max-files-per-micro-batch", "2") .load(f"${sourceDatabase}.${sourceTable}") ... .trigger(Trigger.AvailableNow()) ``` Will end up ignoring the rate limit and process one large batch: ``` AvailableNowMicroBatchStreamWrapper - The read limit MaxFiles: 2 is ignored because source org.apache.iceberg.spark.source.SparkMicroBatchStream@4918755f does not support running Trigger.AvailableNow queries. ``` Ref. [SPARK-45178](https://issues.apache.org/jira/browse/SPARK-45178) ## Proposal Proposal is to change `SparkMicroBatchStream` to implement the `SupportsTriggerAvailableNow` interface, so rate limits can be applied on AvailableNow and query execution can split into smaller batches. AvailableNow has a lot of benefits for incremental processing and is used quite frequently with Delta, adding this support would be a big step up. ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
