[
https://issues.apache.org/jira/browse/SPARK-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095494#comment-15095494
]
Sun Rui edited comment on SPARK-12635 at 1/13/16 2:35 AM:
----------------------------------------------------------
[~dselivanov] PySpark uses pickle and CloudPickle on python side and
net.razorvine.pickle on JVM side for data serialization/deserialization between
Python and JVM. While there lacks a library similar to net.razorvine.pickle
which can deserialize from and serialize to R serialization format. So
currently, SparkR depends on ReadBin()/writeBin() on R side and Java
DataInputStream/DataOutputStream for serialization/deserialization between R
and JVM, based on the fact that for simple types like integer, double, byte
array, they share the same format.
For collect(), the serialization/deserialization happens along with the
communication via socket. I suspect there are much communication overhead
occurring during many socket reads/writes. Maybe we can change the behavior in
batch way, that is, serialize part of the collection result into a buffer in
memory and transfer it back. Would you interested in doing a prototype and see
if there is any performance improvement?
Another idea would be introduce something like net.razorvine.pickle, but that
sounds a lot of effort.
was (Author: sunrui):
[~dselivanov] PySpark uses pickle and CloudPickle on python side and
net.razorvine.pickle on JVM side for data serialization/deserialization between
Python and JVM. While there lacks a library similar to net.razorvine.pickle
which can deserialize from and serialize to R serialization format. So
currently, SparkR depends on ReadBin()/writeBin() on R side and
DataInputStream/DataOutputStream for serialization/deserialization between R
and JVM, based on the fact that for simple types like integer, double, array
byte, they shares the same format.
For collect(), the serialization/deserialization happens along with the
communication via socket. I suspect there are much communication overhead
occurring during many socket reads/writes. Maybe we can change the behavior in
batch way, that is, serialize part of the collection result into a buffer in
memory and transfer it back. Would you interested in doing a prototype and see
if there is any performance improvement?
Another idea would be introduce something like net.razorvine.pickle, but that
sounds a lot of effort.
> More efficient (column batch) serialization for Python/R
> --------------------------------------------------------
>
> Key: SPARK-12635
> URL: https://issues.apache.org/jira/browse/SPARK-12635
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, SparkR, SQL
> Reporter: Reynold Xin
>
> Serialization between Scala / Python / R is pretty slow. Python and R both
> work pretty well with column batch interface (e.g. numpy arrays). Technically
> we should be able to just pass column batches around with minimal
> serialization (maybe even zero copy memory).
> Note that this depends on some internal refactoring to use a column batch
> interface in Spark SQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
