Eric Chang created TOREE-309:
--------------------------------
Summary: pyspark_runner needs to be upgraded for Spark 1.5, 1.6
Key: TOREE-309
URL: https://issues.apache.org/jira/browse/TOREE-309
Project: TOREE
Issue Type: Bug
Affects Versions: 0.1.0
Reporter: Eric Chang
pyspark_runner.py includes version-specific switches for Spark 1.4 that should
be applied to spark 1.5/1.6:
{code}
if sparkVersion.startswith("1.4"):
gateway = JavaGateway(client, auto_convert = True)
else:
gateway = JavaGateway(client)
...
elif sparkVersion.startswith("1.4"):
java_import(gateway.jvm, "org.apache.spark.sql.*")
java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
{code}
I have a branch on my fork that extends the 1.4 logic to 1.5, 1.6. Not sure if
packaging is going to change significantly in 2.x so I didn't extend beyond
1.6:
https://github.com/apache/incubator-toree/compare/master...ericchang:pyspark-runner-1.5-fix
Let me know if it's ok for me to issue the PR against the main repo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
