[
https://issues.apache.org/jira/browse/SPARK-20659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354628#comment-16354628
]
Marcelo Vanzin commented on SPARK-20659:
----------------------------------------
I haven't really looked in detail at this stuff, so I don't really have many
answers, but...
bq. In the new method of SparkStatusTracker is it fine just to give back an
array of RDDStorageInfo from the store?
That type does not seem to be tracking the same thing as {{StorageStatus}}.
There doesn't seem to be an immediate equivalent type in the REST API. So I'd
keep it simple - expose a few extra things like memory metrics in
{{SparkExecutorInfo}}, for example, and wait until someone asks for more
information about individual RDDs to take action.
[~joshrosen] added the original {{getExecutorStorageStatus}} method so he might
have more background on what it was trying to expose.
bq. And for the metrics to pass BlockManagerInfo array via the
BlockManagerMasterEndpoint
Not sure what the question is. But the same data currently needed by that code
path should be kept, maybe just using a different type (or a simplified,
private version of {{StorageStatus}}). You have to find the callers of that RPC
and what they need to figure out what needs to happen here.
> Remove StorageStatus, or make it private.
> -----------------------------------------
>
> Key: SPARK-20659
> URL: https://issues.apache.org/jira/browse/SPARK-20659
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Marcelo Vanzin
> Priority: Major
>
> With the work being done in SPARK-18085, StorageStatus is not used anymore by
> the UI. It's still used in a couple of other places, though:
> - {{SparkContext.getExecutorStorageStatus}}
> - {{BlockManagerSource}} (a metrics source)
> Both could be changed to use the REST API types; the first one could be
> replaced with a new method in {{SparkStatusTracker}}, which I also think is a
> better place for it anyway.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
