[jira] [Commented] (SPARK-26357) Expose executors' procfs metrics to Metrics system

2019-03-01 Thread Reza Safi (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782046#comment-16782046
 ] 

Reza Safi commented on SPARK-26357:
---

I think the primary usecase is for those users who are depending on codahale / 
dropwizard metrics.

> Expose executors' procfs metrics to Metrics system
> --
>
> Key: SPARK-26357
> URL: https://issues.apache.org/jira/browse/SPARK-26357
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Reza Safi
>Priority: Major
>
> It will be good to make the prosfs metrics from SPARK-24958 visible to 
> Metrics system. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26357) Expose executors' procfs metrics to Metrics system

2019-03-01 Thread Sean Owen (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782039#comment-16782039
 ] 

Sean Owen commented on SPARK-26357:
---

What is the use case for this?

> Expose executors' procfs metrics to Metrics system
> --
>
> Key: SPARK-26357
> URL: https://issues.apache.org/jira/browse/SPARK-26357
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Reza Safi
>Priority: Major
>
> It will be good to make the prosfs metrics from SPARK-24958 visible to 
> Metrics system. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26357) Expose executors' procfs metrics to Metrics system

2018-12-12 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719801#comment-16719801
 ] 

ASF GitHub Bot commented on SPARK-26357:


rezasafi opened a new pull request #23306: [SPARK-26357][Core] Expose 
executors' procfs metrics to Metrics system
URL: https://github.com/apache/spark/pull/23306
 
 
   This change exposes the executors' proscfs metrics that were introduced in 
SPARK-24958 to Metrics system. To do this a new metric source is defined and a 
new config is also added. Using the configs user can choose whether they want 
to see procfs metrics using REST API and Spark UI or to see them through 
Metrics system. To avoid overhead user can't choose to report these metrics to 
both UI and Metrics system.
   This was tested manually and I verified that procfs metrics are reporting in 
Metrics system using Console sink:
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMRSSMemory
value = 558977024
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMVMemory
value = 4888707072
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherRSSMemory
value = 0
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherVMemory
value = 0
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonRSSMemory
value = 33546240
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonVMemory
value = 401543168
   
   
   And they got updated as well:
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMRSSMemory
value = 659648512
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMVMemory
value = 4965343232
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherRSSMemory
value = 0
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherVMemory
value = 0
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonRSSMemory
value = 33546240
   
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonVMemory
value = 401543168


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Expose executors' procfs metrics to Metrics system
> --
>
> Key: SPARK-26357
> URL: https://issues.apache.org/jira/browse/SPARK-26357
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Reza Safi
>Priority: Major
>
> It will be good to make the prosfs metrics from SPARK-24958 visible to 
> Metrics system. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26357) Expose executors' procfs metrics to Metrics system

2018-12-12 Thread Reza Safi (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719788#comment-16719788
 ] 

Reza Safi commented on SPARK-26357:
---

I will send a pr soon

> Expose executors' procfs metrics to Metrics system
> --
>
> Key: SPARK-26357
> URL: https://issues.apache.org/jira/browse/SPARK-26357
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Reza Safi
>Priority: Major
>
> It will be good to make the prosfs metrics from SPARK-24958 visible to 
> Metrics system. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org