SreeramGarlapati edited a comment on pull request #4473: URL: https://github.com/apache/iceberg/pull/4473#issuecomment-1086892995
@RussellSpitzer - thanks a lot for your review; The challenge with unit testing this - is that - sparkSession is holding on to the state of the stream - internally. So, I need to author a unit test which spans across 2 sparkSessions; am hitting failures when trying to create 2 spark sessions. Is there any precedence in the codebase for this pattern that you could kindly point me to. I have put the fix before the public for visibility - before the test - so ppl. can port it as needed. This bug is essentially a time bomb in the code and is a very critical one. Let me work on the test and get back. putting the problem being explained above aside - the code itself has coverage in this unittest - `testResumingStreamReadFromCheckpoint` - using which I was able to debug thru & fix the code - where I need to use `file.createOrOverwrite()` instead of the existing `file.create()`. cc: @rajarshisarkar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
