Julien Diener created SPARK-16069:
-------------------------------------
Summary: rdd.map(identity).cache very slow
Key: SPARK-16069
URL: https://issues.apache.org/jira/browse/SPARK-16069
Project: Spark
Issue Type: Question
Components: Spark Core
Affects Versions: 1.6.0
Environment: ubuntu
Reporter: Julien Diener
Fix For: 1.6.0
I found out that when using .map( identity ).cache on a rdd, it become very
slow if the items are big. While it is pretty much instantaneous otherwise.
I posted the question on SO but did not get an answer:
http://stackoverflow.com/q/37859386/1206998
Basically, from an in-memory cached rdd with big item content,
`map(identity).cache` is very slow:
profile( rdd.count ) // around 12 ms
profile( rdd.map(identity).count ) // same
profile( rdd.cache.count ) // same
profile( rdd.map(identity).cache.count ) // 5700 ms !!!
While, if the rdd content is little, this is very fast. I don't understand why
this would take time. In my understanding, in-memory cache should "simply" keep
a reference to the data, no copy, no serialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
