GitHub user saturday-shi opened a pull request:
https://github.com/apache/spark/pull/16253
[SPARK-18537][Web UI] Add a REST api to serve spark streaming information
## What changes were proposed in this pull request?
This PR is an inheritance from #16000, and is a completion of #15904.
**Description**
> 1. implement a package(org.apache.spark.streaming.status.api.v1) that
serve the same purpose as org.apache.spark.status.api.v1
> 1. register the api path through StreamingPage
> 1. retrive the streaming informateion through StreamingJobProgressListener
>
> this api should cover exceptly the same amount of information as you can
get from the web interface
> the implementation is base on the current REST implementation of
spark-core
> and will be available for running applications only
>
> https://issues.apache.org/jira/browse/SPARK-18537
## How was this patch tested?
Local test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/saturday-shi/spark SPARK-18537
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16253.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16253
----
commit 680a59aab2f8b3624f9f39fbc78bb5cbd7ec3bac
Author: Chan Chor Pang <[email protected]>
Date: 2016-10-26T05:36:44Z
compile ok, try to test
commit 04f9e9c914b58c14c98845b5529472333b348590
Author: Chan Chor Pang <[email protected]>
Date: 2016-10-26T07:39:40Z
add path /streamingapi
commit da20ce37b88770e2cb997ed48976ba2542305e6b
Author: Chan Chor Pang <[email protected]>
Date: 2016-10-28T05:42:40Z
need attach to some where
commit 3468d40336091201108169b168bd612c0f5fcf77
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-02T02:10:03Z
no writer yet
commit 525ae5fe25a47ed3884c2bf15143dcb932581f48
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-02T05:13:48Z
not work, may be the data need to be in Iterator form
commit f854767cb5b3bb4a303d3418b426b73d02599c25
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-08T06:05:24Z
remove unuse file
commit 34c8b3b90a9b045e3d6b6ac86d270edff9ed24a3
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-02T06:18:17Z
package name didnt change in the copy process
commit 170b18904f041dfeb271f54fdb408ad2f575a2ee
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-07T04:35:43Z
try to get the real info
commit 2f51c59a37f994c6bee2dd65d1517b32e7d9776d
Author: saturday_s <[email protected]>
Date: 2016-11-14T09:51:02Z
Refactor to fit scalastyle.
commit 76324b7c6f8849bef7d45363d925fd95efbbedcf
Author: saturday_s <[email protected]>
Date: 2016-11-16T04:40:24Z
Try to get startTime.
commit 68d734f07b43b44127ae5f698db39d671aaa59c1
Author: saturday_s <[email protected]>
Date: 2016-11-16T04:53:41Z
Change api path prefix.
commit ccfe0f5f28db73bb300d43c32d40d6e0e596c77c
Author: saturday_s <[email protected]>
Date: 2016-11-16T09:13:59Z
Implement statistics api.
commit 2d1e88440902c5212f43746f5c0b7f282b7a6243
Author: saturday_s <[email protected]>
Date: 2016-11-17T02:59:58Z
Implement receivers api.
commit 0d9f6b9667ef774f5a8c868a453e3d68b66a6702
Author: saturday_s <[email protected]>
Date: 2016-11-17T04:46:13Z
Fix last-error-info format.
commit 8088fa5bad4c0e15bb14abfc0ee7475ba4ad138b
Author: saturday_s <[email protected]>
Date: 2016-11-17T05:08:30Z
Implement one-receiver api.
commit f1da6b1f2856b761696ae9d767836af6417e4f43
Author: saturday_s <[email protected]>
Date: 2016-11-17T05:21:39Z
Fix access level issue of `ErrorWrapper`.
commit 4d8138191f1529137e4c1e858998bd78477ca739
Author: saturday_s <[email protected]>
Date: 2016-11-18T01:30:30Z
Synchronize to listener when getting info from it.
commit 17cb832cedb4b2cfeff5e501a9f71378b3402cee
Author: saturday_s <[email protected]>
Date: 2016-11-18T05:30:15Z
Implement batch(es) api.
commit 137e8fb7de34b39b218939b371062e225adc958e
Author: saturday_s <[email protected]>
Date: 2016-11-18T06:55:42Z
Remove details of outputOpsInfo from batchInfo.
commit 08f33522251ff20b14af15952ac918cbcfada551
Author: saturday_s <[email protected]>
Date: 2016-11-18T08:35:55Z
Implement outputOpsInfo api.
commit 477e71de47bbde642a9222729c73b7dd52318529
Author: saturday_s <[email protected]>
Date: 2016-11-18T09:37:04Z
Try another approach to get outputOpsInfo.
commit 7ddac2929343ad60f733166d25ed485fa3976cc0
Author: saturday_s <[email protected]>
Date: 2016-11-21T02:03:55Z
Try another more approach to get outputOpsInfo.
commit e0fe970fa64fc87de277a7f63f39423608cfef52
Author: saturday_s <[email protected]>
Date: 2016-11-21T02:41:25Z
Continue trying to get outputOpsInfo(jobIds).
commit 35963312dcf722b98cd3b0dabff97d398ccd020c
Author: saturday_s <[email protected]>
Date: 2016-11-21T04:14:38Z
Fix outputOpsInfo and jobIds issue.
commit 9760492cb826c7552c453e3c55a1098455eaa0bc
Author: saturday_s <[email protected]>
Date: 2016-11-21T04:35:55Z
Fix syntax error.
commit 65b39078d54408d8ac1ee608a21e49a978e7415d
Author: saturday_s <[email protected]>
Date: 2016-11-21T05:23:31Z
Consolidate the param check logic into one place.
commit 0080bccf75862d9643ead1c98bf600287227f40d
Author: saturday_s <[email protected]>
Date: 2016-11-21T05:28:37Z
Use the same regex-pattern with `ApiRootResource`.
commit 32b1d867506269ecd83beb390d7d7ce8c9ef78fc
Author: saturday_s <[email protected]>
Date: 2016-11-21T05:39:53Z
Remove unused codes/files.
commit 23337d3dc712c59b7da4e42dccf3552ae6bc2901
Author: saturday_s <[email protected]>
Date: 2016-11-21T06:14:16Z
Remove unused codes.
commit 22e75e49f6a9f5597fb03cbd2688bff109408d69
Author: saturday_s <[email protected]>
Date: 2016-11-21T08:54:24Z
Rename `UIRootFromServletContext` to `StreamingUIRootFromServletContext`.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
