[
https://issues.apache.org/jira/browse/SPARK-26357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719801#comment-16719801
]
ASF GitHub Bot commented on SPARK-26357:
----------------------------------------
rezasafi opened a new pull request #23306: [SPARK-26357][Core] Expose
executors' procfs metrics to Metrics system
URL: https://github.com/apache/spark/pull/23306
This change exposes the executors' proscfs metrics that were introduced in
SPARK-24958 to Metrics system. To do this a new metric source is defined and a
new config is also added. Using the configs user can choose whether they want
to see procfs metrics using REST API and Spark UI or to see them through
Metrics system. To avoid overhead user can't choose to report these metrics to
both UI and Metrics system.
This was tested manually and I verified that procfs metrics are reporting in
Metrics system using Console sink:
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMRSSMemory
value = 558977024
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMVMemory
value = 4888707072
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherRSSMemory
value = 0
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherVMemory
value = 0
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonRSSMemory
value = 33546240
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonVMemory
value = 401543168
And they got updated as well:
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMRSSMemory
value = 659648512
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMVMemory
value = 4965343232
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherRSSMemory
value = 0
application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherVMemory
value = 0
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonRSSMemory
value = 33546240
application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonVMemory
value = 401543168
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Expose executors' procfs metrics to Metrics system
> --------------------------------------------------
>
> Key: SPARK-26357
> URL: https://issues.apache.org/jira/browse/SPARK-26357
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Reza Safi
> Priority: Major
>
> It will be good to make the prosfs metrics from SPARK-24958 visible to
> Metrics system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
