GitHub user proflin opened a pull request:
https://github.com/apache/spark/pull/11470
[SPARK-13618][STREAMING][WEB-UI][WIP] Make Streaming web UI page display
rate-limit lines on statistics graph
## What changes were proposed in this pull request?
This PR makes Streaming web UI display rate-limit lines in the statistics
graph.
Specifically, this PR:
1. adds in `RateLimiter` a data structure keeping history of rate limit
changes, so that calculating the upper bound of how many records we can receive
in a block interval is possible;
2. adds the `numRecordsLimit` information into the path from
`BlockGenerator` generates a `Block` to the `ReceivedBlockInfo` (so that
`numRecordsLimit` can be transferred on wire to the driver side's
`ReceivedBlockTracker`);
3. makes changes in `StreamingJobProgressListener` and related places, so
that the aggregated `numRecordsLimit` information for every batch can be
calculated;
4. makes changes in `StreamingPage` and related places, so two or more
lines can be drawn on a single statistics graph.
## How was this patch tested?
- units tests
- manually checked UI(see below)
## Screenshots
### without back pressure
### with back pressure
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/proflin/spark
display-rate-limit-on-streaming-web-ui
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11470.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11470
----
commit a4e0739b941b61a67a3cf78f4d00223aa20b0e93
Author: proflin <[email protected]>
Date: 2016-03-02T12:40:23Z
Adds in `RateLimiter` a data structure keeping history of rate limit changes
commit 239eab9f70bbf3de432bf19d5b2259dce9818a53
Author: proflin <[email protected]>
Date: 2016-03-02T12:45:52Z
Add `numRecordLimit` information into the `BlockGenerator` ->
`ReceivedBlockInfo` path
commit 54afb2e6c0a26fb155ba93265813bbe546258b9d
Author: proflin <[email protected]>
Date: 2016-03-02T12:52:52Z
Enables `StreamingJobProgressListener` calculate the aggregated
`numRecordsLimit` information for every batch
commit 7aba06393484c822c25f03d8911cf07d2e0258a9
Author: proflin <[email protected]>
Date: 2016-03-02T12:54:10Z
Display the rate limit lines on `StreamingPage`
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
