Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3801#issuecomment-68684233
Pushed some commits addressing most of the feedback, but I'm still
struggling to remove that last `Thread.sleep(1000)`. I think that the problem
here is that the writing of the checkpoint is asynchronous and without the
sleep, we wind up in a state where batch 3 has started processing but has not
finished, and the StreamingContext shuts down before a snapshot including batch
3's file info is written. I plan to dig into this tomorrow to see whether this
is actually the case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
