Thanks for the update, moving this thread into the new Apache dev list
rather than the old Cloudera Google Group
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: [email protected]
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
From: Janki Akhani <[email protected]>
To: Alex Bozarth <[email protected]>
Cc: Livy Development <[email protected]>
Date: 07/17/2017 02:31 PM
Subject: Re: Codahale metrics for Livy
Hi Alex,
Thank you for providing this detail. I looked into Scalatra and it
basically uses Dropwizard metrics. So I will be going ahead with this
option. I will be adding user level and system level metrics as mentioned
in previous email. I have not yet decided on whether to add new API or add
metrics in same end point. I will provide an update soon for the same.
Thanks,
Janki Akhani
On Thu, Jun 1, 2017 at 2:54 PM, Alex Bozarth <[email protected]> wrote:
Thanks for the update Janki,
The current /metrics is an auto-generated html page with links to json or
text formatted metrics and is very minimal. After some testing I decided
to leave it alone and outside the ui. The metrics you wish to add would
be useful in a Metric UI tab though. Are you planning on adding a new API
that returns the metrics in json format similar to the current Livy API?
The UI gets it's data from the API via js ajax calls, not direct from the
code, so json formated metrics would be the easiest to use in an eventual
Metrics UI tab. I haven't done too much reading into the available
metrics servlets in Scalatra (Coda Hale), but there are probably
extendable servlets already available.
For the Web UI code, only the first PR has been merged so far, but you
can follow my progress via the umbrella JIRA here:
https://issues.cloudera.org/browse/LIVY-87 I have a few open PRs
currently and code ready for the next few.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: [email protected]
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
Inactive hide details for Janki Akhani ---06/01/2017 02:38:15 AM---Hi
Alex, It's good to know about Livy Web UI. My plan is to Janki Akhani
---06/01/2017 02:38:15 AM---Hi Alex, It's good to know about Livy Web UI.
My plan is to collect System level
From: Janki Akhani <[email protected]>
To: Alex Bozarth <[email protected]>
Cc: Livy Development <[email protected]>
Date: 06/01/2017 02:38 AM
Subject: Re: Codahale metrics for Livy
Hi Alex,
It's good to know about Livy Web UI. My plan is to collect System level
metrics as well as below mentioned user level metrics.
Unique users using livy at any given time
# of active sessions by each users
Total number of concurrent connections to livy
Queue, Driver and executor memory requested in eash session,
queue usage etc
Basically at any given point in time, we would like to know who are
active users, sessions, maximum number of active sessions, requested
resources, # of exceptions and 400s and 500s returned in response.
As you mentioned, you are planning to add /metrics UI page. Are you
planning to show collected metrics in graph or json format?
I was thinking of using Dropwizard to collect System and user level
metrics in livy.
Please share your thoughts for the same. I have not yet checked UI and
MetricsAdminServet. I will have a look at it.
Thanks,
Janki Akhani
On Thu, Jun 1, 2017 at 2:18 AM, Alex Bozarth <[email protected]> wrote:
Hi Janki,
I'm currently working on adding a Livy Web UI and would love to
potentially include any metrics and monitoring you add in the UI. I
was wondering what new metrics you were looking to add? Livy
already has the MetricsAdminServet from Coda Hale mounted at / (or
for 0.4, /metrics if the UI is enabled). Currently I have no plans
to include those metrics in the UI though. Adding any Coda Hale
metrics should be straight forward since the servlet mounts are
included with Scalatra, which Livy uses.
Alex Bozarth
Spark Technology Center
On Wednesday, May 31, 2017 at 1:33:12 AM UTC-7, Janki Akhani wrote:
Hi,
We have started Livy at LinkedIn. It is in poc stage right now. We
are thinking of adding Metrics and Monitoring for it. It will be
useful to add Codahale metrics to livy. Can you please provide your
thoughts for the same? I am looking into possible solutions for the
same. I will share those detail soon.
--
Thanks & Regards,
Janki Akhani
Data Analytics and Infrastructure
+91 96200 51508
[email protected]
linkedin.com/in/jakhani
--
Thanks & Regards,
Janki Akhani
Data Analytics and Infrastructure
+91 96200 51508
[email protected]
linkedin.com/in/jakhani
--
Thanks & Regards,
Janki Akhani
Data Analytics and Infrastructure
+91 96200 51508
[email protected]
linkedin.com/in/jakhani
