GitHub user ChorPangChan opened a pull request:
https://github.com/apache/spark/pull/15974
[SPARK-18537] Add a REST api to spark streaming
## What changes were proposed in this pull request?
1. implement a package(org.apache.spark.streaming.status.api.v1) that serve
the same purpose as org.apache.spark.status.api.v1
1. register the api path through StreamingPage
1. retrive the streaming informateion through StreamingJobProgressListener
this api should cover exceptly the same amount of information as you can
get from the web interface
the implementation is base on the current REST pmplementation of spark-core
and will be available for running applications only
https://issues.apache.org/jira/browse/SPARK-18537
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ChorPangChan/spark stream-api-dev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15974.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15974
----
commit 4ac2fd0767345e1ac31a15d726a1d5542c885d43
Author: Chan Chor Pang <[email protected]>
Date: 2016-10-26T05:36:44Z
compile ok, try to test
commit 328e77d4a1e248f5fb2dee707113da21a04fa302
Author: Chan Chor Pang <[email protected]>
Date: 2016-10-26T07:39:40Z
add path /streamingapi
commit 0ce5054a35b0f34ee499d7b1e6c12d310450cff0
Author: Chan Chor Pang <[email protected]>
Date: 2016-10-28T05:42:40Z
need attach to some where
commit 2de6b4adcf24e5c40956e1b69b8f7dc7fe796fdc
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-08T06:05:24Z
remove unuse file
commit f1301b03abf26ce7ea338176c66bbb72c16d86d7
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-02T02:10:03Z
no writer yet
commit 25147cdc4e7bfa0e428d4b7a3d23ace3a3d8c95a
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-02T05:13:48Z
not work, may be the data need to be in Iterator form
commit fd5afced470b9c14c348f30996ddd20081798794
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-02T06:18:17Z
package name didnt change in the copy process
commit dbb317fb0f394e6afa11ce9f162e11858f1d601e
Author: Chan Chor Pang <[email protected]>
Date: 2016-11-07T04:35:43Z
try to get the real info
commit 9f488bc8028b7fcff33937399782988bf0383a16
Author: saturday_s <[email protected]>
Date: 2016-11-14T09:51:02Z
Refactor to fit scalastyle.
commit 77f1d189fe74c02e7a31611c66b90c23bab8f2a5
Author: saturday_s <[email protected]>
Date: 2016-11-16T04:40:24Z
Try to get startTime.
commit bc16ed8af007753f96151db5ad8bd3329788700c
Author: saturday_s <[email protected]>
Date: 2016-11-16T04:53:41Z
Change api path prefix.
commit 6174419bfff201623dfa7525f518586c9271e2aa
Author: saturday_s <[email protected]>
Date: 2016-11-16T09:13:59Z
Implement statistics api.
commit dea1e62607e35dee579fdb1fc050c631f41291ad
Author: saturday_s <[email protected]>
Date: 2016-11-17T02:59:58Z
Implement receivers api.
commit db4e78fa02f40aff61b994e92dda07b92f58af43
Author: saturday_s <[email protected]>
Date: 2016-11-17T04:46:13Z
Fix last-error-info format.
commit 08fb3c8623e2289bd6689ed5a8652184a7022350
Author: saturday_s <[email protected]>
Date: 2016-11-17T05:08:30Z
Implement one-receiver api.
commit 083d2477eeb8a0823fc3f676b2d3ebb5b6629fcb
Author: saturday_s <[email protected]>
Date: 2016-11-17T05:21:39Z
Fix access level issue of `ErrorWrapper`.
commit ecccd94356ada9dcff220dfe6b7275247a6f40e2
Author: saturday_s <[email protected]>
Date: 2016-11-18T01:30:30Z
Synchronize to listener when getting info from it.
commit 1fe8c24096d6a8b84e6c5c870cffbaa401ddba40
Author: saturday_s <[email protected]>
Date: 2016-11-18T05:30:15Z
Implement batch(es) api.
commit 9758774c450dcaf8802d09a0f03251e78bdfd73b
Author: saturday_s <[email protected]>
Date: 2016-11-18T06:55:42Z
Remove details of outputOpsInfo from batchInfo.
commit 53e36293e1a4a881f574018511ea37c068a4afc4
Author: saturday_s <[email protected]>
Date: 2016-11-18T08:35:55Z
Implement outputOpsInfo api.
commit 402299de15966927358770901420f5311aa5d92b
Author: saturday_s <[email protected]>
Date: 2016-11-18T09:37:04Z
Try another approach to get outputOpsInfo.
commit 7511416b03497088a3dc98b865ff5a9c4a889d6a
Author: saturday_s <[email protected]>
Date: 2016-11-21T02:03:55Z
Try another more approach to get outputOpsInfo.
commit d8d847483f9c12cc83c9546936f2f64a676502d0
Author: saturday_s <[email protected]>
Date: 2016-11-21T02:41:25Z
Continue trying to get outputOpsInfo(jobIds).
commit 3ff94954ff97a7d514f971412569804913f73596
Author: saturday_s <[email protected]>
Date: 2016-11-21T04:14:38Z
Fix outputOpsInfo and jobIds issue.
commit 8279fd0289518f0d2fea3890fd4d11d1fd0d258a
Author: saturday_s <[email protected]>
Date: 2016-11-21T04:35:55Z
Fix syntax error.
commit 6a58642660950725334df0d1dc86511c5723f694
Author: saturday_s <[email protected]>
Date: 2016-11-21T05:23:31Z
Consolidate the param check logic into one place.
commit 80588f2319c29e31d64fea29906d141c2740af5c
Author: saturday_s <[email protected]>
Date: 2016-11-21T05:28:37Z
Use the same regex-pattern with `ApiRootResource`.
commit aa18e7cc32692e58051580348b4f2a71e9ef6854
Author: saturday_s <[email protected]>
Date: 2016-11-21T05:39:53Z
Remove unused codes/files.
commit 1e253e81096ec97d28861aadac87cfbdaeef5e3d
Author: saturday_s <[email protected]>
Date: 2016-11-21T06:14:16Z
Remove unused codes.
commit 278383aea374e5cd12643d96a7777c8dc95b3a24
Author: saturday_s <[email protected]>
Date: 2016-11-21T06:20:42Z
Enable `SecurityFilter`.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
