GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/4779
[SPARK-6027][SPARK-5546] Fixed --jar not working for KafkaUtils and
improved error message
The problem with SPARK-6027 in short is that JARs like the
kafka-assembly.jar does not work in python as the added JAR is not visible in
the classloader used by Py4J. Py4J uses Class.forName(), which does not uses
the systemclassloader, but the JARs are only visible in the Thread's
contextclassloader. So this back uses the context class loader to create the
KafkaUtils dstream object.
Also improves the error message. This PR assumes that #4754 is going to
make --packages work for pyspark (which is what is referred to in the error
message as 1. ).
@davies
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark kafka-python-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4779.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4779
----
commit 7b88be8e2898068e9dd2fba14a96e377e9ed219c
Author: Tathagata Das <[email protected]>
Date: 2015-02-26T03:59:29Z
Fixed --jar not working for KafkaUtils and improved error message
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
