[
https://issues.apache.org/jira/browse/SPARK-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707428#comment-15707428
]
Marius Van Niekerk edited comment on SPARK-15369 at 11/30/16 3:49 AM:
----------------------------------------------------------------------
Oh yeah, once we have a pip installable spark it should be pretty easy testing
this with some docker pieces with travis.
Basic idea is to convert the benchmarks into an integration test.
Feel free to open issues on that project.
was (Author: mariusvniekerk):
Oh yeah, once we have a pip installable spark it should be pretty easy testing
this with some docker pieces with travis.
Basic idea is to convert the benchmarks into an integration test.
> Investigate selectively using Jython for parts of PySpark
> ---------------------------------------------------------
>
> Key: SPARK-15369
> URL: https://issues.apache.org/jira/browse/SPARK-15369
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Reporter: holdenk
> Priority: Minor
>
> Transferring data from the JVM to the Python executor can be a substantial
> bottleneck. While Jython is not suitable for all UDFs or map functions, it
> may be suitable for some simple ones. We should investigate the option of
> using Jython to accelerate these small functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
