[ https://issues.apache.org/jira/browse/SPARK-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550012#comment-15550012 ]
holdenk commented on SPARK-15369: --------------------------------- Certainly we can investigate speeding up the serialization between the JVM and Python as well. I think Wes has some interesting ideas around using Arrow for something like this (although last I looked the JVM side was maybe a bit far away from being usable). I'll keep following along with Arrow & related projects as well :) The Jython limitations are fairly restrictive its true, but the performance improvement can be pretty large as well so it might be a reasonable trade-off for those cases (and also if we eventually no longer have the overhead of JVM/Python communication be a dominating factor for so many use cases we can just remove the Jython APIs since the same code should work in regular python). > Investigate selectively using Jython for parts of PySpark > --------------------------------------------------------- > > Key: SPARK-15369 > URL: https://issues.apache.org/jira/browse/SPARK-15369 > Project: Spark > Issue Type: Improvement > Components: PySpark > Reporter: holdenk > Priority: Minor > > Transferring data from the JVM to the Python executor can be a substantial > bottleneck. While Jython is not suitable for all UDFs or map functions, it > may be suitable for some simple ones. We should investigate the option of > using Jython to accelerate these small functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org