holdenk commented on SPARK-15369:

I can understand the hesitancy to adopt this long term - I wish we could 
explore exposing this as a developer API instead but if that isn't something of 
interest I'll take a look at adding this as a spark package (or adding to one 
of the meta-spark packages like Bahir). In the meantime I'll do some more 
poking with Arrow and see if its getting to a point where we could selectively 
start using it in PySpark.

> Investigate selectively using Jython for parts of PySpark
> ---------------------------------------------------------
>                 Key: SPARK-15369
>                 URL: https://issues.apache.org/jira/browse/SPARK-15369
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>            Reporter: holdenk
>            Priority: Minor
> Transferring data from the JVM to the Python executor can be a substantial 
> bottleneck. While Jython is not suitable for all UDFs or map functions, it 
> may be suitable for some simple ones. We should investigate the option of 
> using Jython to accelerate these small functions.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to