[
https://issues.apache.org/jira/browse/SPARK-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323473#comment-14323473
]
Josh Rosen commented on SPARK-1418:
-----------------------------------
This issue's description is now a little confusing, since
{{_get_unmangled_rdd}} has been removed from the Python MLlib bindings. Does
anyone know if this issue is still relevant? If so, could we update its
description? Otherwise, let's close this.
> Python MLlib's _get_unmangled_rdd should uncache RDDs when training is done
> ---------------------------------------------------------------------------
>
> Key: SPARK-1418
> URL: https://issues.apache.org/jira/browse/SPARK-1418
> Project: Spark
> Issue Type: Improvement
> Components: MLlib, PySpark
> Reporter: Matei Zaharia
>
> Right now when PySpark converts a Python RDD of NumPy vectors to a Java one,
> it caches the Java one, since many of the algorithms are iterative. We should
> call unpersist() at the end of the algorithm though to free cache space. In
> addition it may be good to persist the Java RDD with
> StorageLevel.MEMORY_AND_DISK instead of going back through the NumPy
> conversion.. it will almost certainly be faster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
