cccs-jc commented on issue #8921: URL: https://github.com/apache/iceberg/issues/8921#issuecomment-1780023262
ya seems pretty clear that it should be picking up the offset from the checkpoint when it exists. could this come into the picture. There are two checks about the timestamp here https://github.com/apache/iceberg/blob/aa891acf20040d15e7ca59dc503adb3c1e4325b8/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L326 and we do set rate limits on rows and on files. public Offset latestOffset(Offset startOffset, ReadLimit limit) { I have only tested with 1.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
