kbendick opened a new issue, #5538:
URL: https://github.com/apache/iceberg/issues/5538
### Feature Request / Improvement
Currently, the SparkReadConf can be used with `start-snapshot-id` and
`end-snapshot-id` options to read just a specific chunk of the table.
These options are not respected when `SparkMicroBatchStream` is initialized,
i.e. using the following will silently ignore these options.
```scala
val df = spark
.readStream
.format("iceberg")
.option("start-snapshot-id", startId)
.option("end-snapshot-id", endId)
.table("rauls_table")
```
We should implement support for more than jsut timestamp based incremental
reads in `SparkMicrobatchStream`.
This was reported on Slack and I'm opening an issue for it. Please feel free
to work on this if you have time 👍
### Query engine
Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
