Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67268086
Alright, I think I fixed up a few problems in my first version of the
SPARK-1600 fix, so hopefully it works now.
For those who are interested in the details:
- We had a "time-travel" issue when restarting from the checkpoint because
I didn't copy over the old ManualClock value to the new StreamingContext.
- My modified FileInputDStream didn't handle `clock` properly during
recovery (the field shouldn't have been `@transient`, so I made it into a `def`
instead).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
