While we're at it, adding endpoints that get results by jobGroup (cf. SparkContext#setJobGroup) instead of just for a single Job would also be very useful to some of us.
On Thu, Sep 17, 2015 at 7:30 AM, Imran Rashid <iras...@cloudera.com> wrote: > Hi Kevin, > > I think it would be great if you added this. It never got added in the > first place b/c the original PR was already pretty bloated, and just never > got back to this. I agree with Reynold -- you shouldn't need to increase > the version for just adding new endpoints (or even adding new fields to > existing endpoints). See the guarantees we make here: > > http://spark.apache.org/docs/latest/monitoring.html#rest-api > > (Though if you think we should make different guarantees around versions, > that would be worth discussing as well.) > > Can you file a jira, and we move discussion there? Please cc me, and > maybe also Josh Rosen (I'm not sure if he has cycles now but he's been very > helpful on these issues in the past). > > thanks, > Imran > > > On Wed, Sep 16, 2015 at 9:10 PM, Kevin Chen <kc...@palantir.com> wrote: > >> Just wanted to bring this email up again in case there were any thoughts. >> Having all the information from the web UI accessible through a supported >> json API is very important to us; are there any objections to us adding a >> v2 API to Spark? >> >> Thanks! >> >> From: Kevin Chen <kc...@palantir.com> >> Date: Friday, September 11, 2015 at 11:30 AM >> To: "dev@spark.apache.org" <dev@spark.apache.org> >> Cc: Matt Cheah <mch...@palantir.com>, Mingyu Kim <m...@palantir.com> >> Subject: New Spark json endpoints >> >> Hello Spark Devs, >> >> I noticed that [SPARK-3454], which introduces new json endpoints at >> /api/v1/[path] for information previously only shown on the web UI, does >> not expose several useful properties about Spark jobs that are exposed on >> the web UI and on the unofficial /json endpoint. >> >> Specific examples include the maximum number of allotted cores per >> application, amount of memory allotted to each slave, and number of cores >> used by each worker. These are provided at ‘app.cores, app.memoryperslave, >> and worker.coresused’ in the /json endpoint, and also all appear on the web >> UI page. >> >> Is there any specific reason that these fields are not exposed in the >> public API? If not, would it be reasonable to add them to the json blobs, >> possibly in a future /api/v2 API? >> >> Thank you, >> Kevin Chen >> >> >
