Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/7600#issuecomment-126213051
>LGTM There are still an issues that occurred to me in the last moment, but
I am inclined to merge this as is to unblock other PRs - RateController should
not be serialized with checkpoints. Does not make sense to recover the
RateController. Can you make a PR for this?
I forgot to mention this. I spent some time on it, but it looked like the
gains were not worth it. The problem is deserialization, and the specific
`RateEstimator` instance. At deserialization time there's no `ssc` available
yet (it's `null`), so there's no easy way to get to a `SparkConf` in order to
get the configured estimator. We could work around that by attaching the
controller at a later point, during `start` or some other lifecycle event, but
it seemed to complicate matters quite a bit, for little gain that I can see.
Thanks for not blocking the PR over this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
