[ https://issues.apache.org/jira/browse/SPARK-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269738#comment-15269738 ]
Alex Bozarth commented on SPARK-13269: -------------------------------------- Hey [~andrewor14], I was interested in this and took a look at the two examples you gave and am a bit confused at what exactly you actually want. You can currently see the used memory, used disk space, and active task count for each executor by calling /applications/[app-id]/executors or (in code) getting the ExecutorSummary class for each executor and checking activeTask, memoryUsed, and diskUsed. Are these numbers different from what you were interested in surfacing? I was unsure of how those example related to JobProgressListener as well. > Expose more executor stats in stable status API > ----------------------------------------------- > > Key: SPARK-13269 > URL: https://issues.apache.org/jira/browse/SPARK-13269 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Reporter: Andrew Or > > Currently the stable status API is quite limited; it exposes only a small > subset of the things exposed by JobProgressListener. It is useful for very > high level querying but falls short when the developer wants to build an > application on top of Spark with more integration. > In this issue I propose that we expose at least two things: > - Which executors are running tasks, and > - Which executors cached how much in memory and on disk > The goal is not to expose exactly these two things, but to expose something > that would allow the developer to learn about them. These concepts are very > much fundamental in Spark's design so there's almost no chance that they will > go away in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org