GitHub user dragos opened a pull request:
https://github.com/apache/spark/pull/7648
[SPARK-8979] Add a PID based rate estimator
Based on #7600
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/typesafehub/spark topic/streaming-bp/pid
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7648.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7648
----
commit 4721c7d9394b7917b2be8fb7df5e4eb1c31d68df
Author: François Garillot <[email protected]>
Date: 2015-07-13T11:26:57Z
[SPARK-8975][Streaming] Add a mechanism to send a new rate from the driver
to the block generator
commit d15de422b973a020d5aa9035016c1274262631fb
Author: François Garillot <[email protected]>
Date: 2015-07-15T11:14:31Z
[SPARK-8975][Streaming] Adds Ratelimiter unit tests w.r.t.
spark.streaming.receiver.maxRate
commit 6369b305b87f812e0bbb6a37e2a1d338c1e1cb26
Author: Iulian Dragos <[email protected]>
Date: 2015-07-17T09:20:38Z
Merge pull request #15 from huitseeker/SPARK-8975
[SPARK-8975][Streaming] Adds a mechanism to send a new rate from the driver
to the block generator
commit cd1397d141eda98ded62491c0f2d90a2b47e56c5
Author: Iulian Dragos <[email protected]>
Date: 2015-07-17T14:38:10Z
Add a test for the propagation of a new rate limit from driver to receivers.
commit 261a05128ec1e1e055c62a6afd44fef39fb711c1
Author: Iulian Dragos <[email protected]>
Date: 2015-07-20T12:28:46Z
- removed field to hold the current rate limit in rate limiter
- made rate limit a Long and default to Long.MaxValue (consequence of the
above)
- removed custom `waitUntil` and replaced it by `eventually`
commit 0c51959c9315f63bc80a7ff5b716f48f907b1152
Author: Iulian Dragos <[email protected]>
Date: 2015-07-20T14:40:01Z
Added a few tests that measure the receiverâs rate.
As I mentioned before, I donât think this is a great idea:
- such tests are flaky (original test in ReceiverSuite was ignored for
that reason)
- Guavaâs code has its own test suite, so we can assume it implements
`setRate` correctly
I noticed one flaky failure in about 10 runs on my machine (receiver got
1 message less than the lower bound, which is within 5% of the nominal
rate).
commit 210f495fff34f25caaba41a8db720c1e3a63fa95
Author: Iulian Dragos <[email protected]>
Date: 2015-07-21T12:29:04Z
Revert "Added a few tests that measure the receiverâs rate."
This reverts commit 0c51959c9315f63bc80a7ff5b716f48f907b1152.
commit 162d9e598040b5b2bad36fa0f7139a99df95e79d
Author: Iulian Dragos <[email protected]>
Date: 2015-07-21T13:06:13Z
Use Reflection for accessing truly private `executor` method and
use the listener bus to know when receivers have registered (`onStart`
is called before receivers have registered, leading to flaky behavior).
commit 8941cf91b03aa7835a78bc756bee1f32cb7bb1d8
Author: Iulian Dragos <[email protected]>
Date: 2015-07-22T14:45:57Z
Renames and other nitpicks.
commit d32ca3697ab18ba8db9905c81d57559bf8472195
Author: François Garillot <[email protected]>
Date: 2015-07-14T11:11:01Z
[SPARK-8977][Streaming] Defines the RateEstimator interface, and implements
the ReceiverRateController
commit 34a389dda362467879bb8c87dd047e41a1a931ca
Author: Iulian Dragos <[email protected]>
Date: 2015-07-22T21:13:24Z
Various style changes and a first test for the rate controller.
commit 238cfc698e47bfc43e00d511208a4ba1eac83e35
Author: Iulian Dragos <[email protected]>
Date: 2015-07-23T10:16:10Z
Merge remote-tracking branch 'upstream/master' into
topic/streaming-bp/rate-controller
commit b425d32808c5325b8e1bc73159a08b47b8c5a03a
Author: Iulian Dragos <[email protected]>
Date: 2015-07-23T14:30:49Z
Removed DeveloperAPI, removed rateEstimator field, removed Noop rate
estimator, changed logic for initialising rate estimator.
commit e57c66b945a9cd3364f75c8ff7c1855031059dd4
Author: Iulian Dragos <[email protected]>
Date: 2015-07-23T16:36:22Z
Added a couple of tests for the full scenario from driver to receivers,
with several rate updates.
commit 715437af0ddb00c90739f818aab2b56934084b93
Author: Iulian Dragos <[email protected]>
Date: 2015-07-24T14:26:09Z
Review comments and added a `reset` call in ReceiverTrackerTest.
commit e9fb45ebfce956005ca2830bc6fbc100f090ed53
Author: Iulian Dragos <[email protected]>
Date: 2015-07-24T15:34:08Z
- Add a test for checkpointing
- fixed serialization for RateController.executionContext
commit d3bfbcd1ce8438cf941d03cce092d031df24358f
Author: François Garillot <[email protected]>
Date: 2015-07-14T11:55:29Z
[SPARK-8979][Streaming] Implements a PIDRateEstimator
commit 70d875b14c944ac2aab1b5f8dfad06f2e6b281b8
Author: Iulian Dragos <[email protected]>
Date: 2015-07-24T17:02:26Z
Update to latest version of the code, various style and name improvements.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
