GitHub user dmcguire81 opened a pull request:
https://github.com/apache/spark/pull/5559
[SPARK-6985][streaming] Receiver maxRate over 1000 causes a
StackOverflowError
A simple truncation in integer division (on rates over 1000 messages /
second) causes the existing implementation to sleep for 0 milliseconds, then
call itself recursively; this causes what is essentially an infinite recursion,
since the base case of the calculated amount of time having elapsed can't be
reached before available stack space is exhausted. A fix to this truncation
error is included in this patch.
However, even with the defect patched, the accuracy of the existing
implementation is abysmal (the error bounds of the original test were
effectively [-30%, +10%], although this fact was obscured by hard-coded error
margins); as such, when the error bounds were tightened down to [-5%, +5%], the
existing implementation failed to meet the new, tightened, requirements.
Therefore, an industry-vetted solution (from Guava) was used to get the adapted
tests to pass.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dmcguire81/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5559.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5559
----
commit d6e107977e791ca70d7b118739e7160f4dec1867
Author: David McGuire <[email protected]>
Date: 2015-04-17T16:43:57Z
Stack overflow error in RateLimiter on rates over 1000/s
commit 24b1bc02ecd458d5b34f32fe51a221cad621b26d
Author: David McGuire <[email protected]>
Date: 2015-04-17T17:00:07Z
Fix truncation in integer division causing infinite recursion
commit 38f3ca8a4fb7845aebd855ccaa73339fba5ca091
Author: David McGuire <[email protected]>
Date: 2015-04-17T17:05:30Z
Ratchet down the error rate to +/- 5%; tests fail
commit 27947177802c340e6e465f23006b592b6deb43b5
Author: David McGuire <[email protected]>
Date: 2015-04-17T17:11:02Z
Replace the RateLimiter with the Guava implementation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
