singhpk234 commented on issue #8921: URL: https://github.com/apache/iceberg/issues/8921#issuecomment-1780012237
I am not entirely sure why this would happen, i was under the impression that initialOffset() is used from checkpoint file and the way it's calculated is same as you stated above : https://github.com/apache/iceberg/blob/aa891acf20040d15e7ca59dc503adb3c1e4325b8/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L112 https://github.com/apache/iceberg/blob/aa891acf20040d15e7ca59dc503adb3c1e4325b8/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L458 Is there some caching that would be coming into picture for snapshot not found ? i think we have something like `SparkCachedTableCatalog` can you try making the ttl zero, just making a wild guess here., this might require deep-dive if we have a crisp steps of repro. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
