Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/7600#issuecomment-126091255
LGTM There are still an issues that occurred to me in the last moment, but
I am inclined to merge this as is to unblock other PRs - RateController should
not be serialized with checkpoints. Does not make sense to recover the
RateController. Can you make a PR for this?
Regarding flakiness, it often times occurs only in Jenkins as things can
run very slowly in Jenkins, and causes all this Thread.sleep related flakiness
to occur. But your current method without background thread make it less flaky,
but still does not guarantee. I thought about the test. The least flaky way to
test that multiple rate update reach the receiver, would be to send one rate
update, wait for it to be received, then send another update with different
rate, then wait for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
