[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo
[
https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752734#comment-16752734
]
Marcelo Vanzin commented on SPARK-23411:
Hmm, removing this method is a little tricky because the REPL tests use it, and
need a public API for this, which wouldn't exist... I'll fix the test
separately.
> Deprecate SparkContext.getRDDStorageInfo
>
>
> Key: SPARK-23411
> URL: https://issues.apache.org/jira/browse/SPARK-23411
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> This is another developer API in SparkContext that is better moved to a more
> monitoring-minded place such as {{SparkStatusTracker}}. The same information
> is also already available through the REST API.
> Moreover, it exposes {{RDDInfo}}, which is a mutable internal Spark type, and
> exposing that to applications is kinda sketchy. (It can't be made private,
> though, since it's exposed indirectly in events posted to the listener bus.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo
[
https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752661#comment-16752661
]
Marcelo Vanzin commented on SPARK-23411:
I saw the test for this method fail (see e.g.
https://amplab.cs.berkeley.edu/jenkins/user/vanzin/my-views/view/Spark/job/spark-master-test-sbt-hadoop-2.7/5437/testReport/junit/org.apache.spark/SparkContextInfoSuite/getRDDStorageInfo_only_reports_on_RDDs_that_actually_persist_data/)
so I'll just go ahead and remove it in 3.0.
> Deprecate SparkContext.getRDDStorageInfo
>
>
> Key: SPARK-23411
> URL: https://issues.apache.org/jira/browse/SPARK-23411
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> This is another developer API in SparkContext that is better moved to a more
> monitoring-minded place such as {{SparkStatusTracker}}. The same information
> is also already available through the REST API.
> Moreover, it exposes {{RDDInfo}}, which is a mutable internal Spark type, and
> exposing that to applications is kinda sketchy. (It can't be made private,
> though, since it's exposed indirectly in events posted to the listener bus.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo
[
https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363037#comment-16363037
]
Marcelo Vanzin commented on SPARK-23411:
Argh, copied the wrong method name. Fixed.
> Deprecate SparkContext.getRDDStorageInfo
>
>
> Key: SPARK-23411
> URL: https://issues.apache.org/jira/browse/SPARK-23411
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> This is another developer API in SparkContext that is better moved to a more
> monitoring-minded place such as {{SparkStatusTracker}}. The same information
> is also already available through the REST API.
> Moreover, it exposes {{RDDInfo}}, which is a mutable internal Spark type, and
> exposing that to applications is kinda sketchy. (It can't be made private,
> though, since it's exposed indirectly in events posted to the listener bus.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
