[
https://issues.apache.org/jira/browse/TOREE-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gino Bustelo closed TOREE-309.
------------------------------
> pyspark_runner needs to be upgraded for Spark 1.5, 1.6
> ------------------------------------------------------
>
> Key: TOREE-309
> URL: https://issues.apache.org/jira/browse/TOREE-309
> Project: TOREE
> Issue Type: Bug
> Affects Versions: 0.1.0
> Reporter: Eric Chang
> Fix For: 0.1.0
>
>
> pyspark_runner.py includes version-specific switches for Spark 1.4 that
> should be applied to spark 1.5/1.6:
> {code}
> if sparkVersion.startswith("1.4"):
> gateway = JavaGateway(client, auto_convert = True)
> else:
> gateway = JavaGateway(client)
> ...
> elif sparkVersion.startswith("1.4"):
> java_import(gateway.jvm, "org.apache.spark.sql.*")
> java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
> {code}
> I have a branch on my fork that extends the 1.4 logic to 1.5, 1.6. Not sure
> if packaging is going to change significantly in 2.x so I didn't extend
> beyond 1.6:
> https://github.com/apache/incubator-toree/compare/master...ericchang:pyspark-runner-1.5-fix
> Let me know if it's ok for me to issue the PR against the main repo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
