Andrew Or created SPARK-13269:
---------------------------------
Summary: Expose more executor stats in stable status API
Key: SPARK-13269
URL: https://issues.apache.org/jira/browse/SPARK-13269
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: Andrew Or
Currently the stable status API is quite limited; it exposes only a small
subset of the things exposed by JobProgressListener. It is useful for very high
level querying but falls short when the developer wants to build an application
on top of Spark with more integration.
In this issue I propose that we expose at least two things:
- Which executors are running tasks, and
- Which executors cached how much in memory and on disk
The goal is not to expose exactly these two things, but to expose something
that would allow the developer to learn about them. These concepts are very
much fundamental in Spark's design so there's almost no chance that they will
go away in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
