[
https://issues.apache.org/jira/browse/SPARK-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng closed SPARK-7360.
--------------------------------
Resolution: Done
Fix Version/s: 1.4.0
Assignee: Xiangrui Meng (was: Nicholas Chammas)
Target Version/s: 1.4.0 (was: 1.5.0)
I marked this as "done". [~davies] mentioned that we need to serialize a lot of
classes without useMemo, which hurts performance. So turning useMemo off
globally is not a good option.
> Compare Pyrolite performance affected by useMemo
> ------------------------------------------------
>
> Key: SPARK-7360
> URL: https://issues.apache.org/jira/browse/SPARK-7360
> Project: Spark
> Issue Type: Task
> Components: PySpark
> Reporter: Xiangrui Meng
> Assignee: Xiangrui Meng
> Fix For: 1.4.0
>
>
> As discussed in SPARK-6288, disabling useMemo shows significant performance
> on some ML tasks. We should test whether this is true across PySpark, and
> consider patch Pyrolite for Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
