Github user boneill42 commented on the pull request:
https://github.com/apache/spark/pull/12209#issuecomment-207042981
That was my fear as well. Right now, we are adding the
kinesis-asl-assembly jar via spark-defaults.conf.
```
spark.driver.extraClassPath
/Users/brianoneill/tools/spark/spark-streaming-kinesis-asl-assembly_2.10-1.6.1-monetate.jar
spark.executor.extraClassPath
/Users/brianoneill/tools/spark/spark-streaming-kinesis-asl-assembly_2.10-1.6.1-monetate.jar
```
That appears to work, but it could be luck. It could be that some of the
2.5.0 and 2.6.1 classes are compatible, or a 2.6.1-specific class is getting
loaded first. I'm not familiar enough with how Spark constructs the
classpaths.
Note also that we are running this from PySpark.
Here is a gist with the example that fails w/ KCL 1.4.0, but works with 1.6:
https://gist.github.com/boneill42/020dde814346c6b4ad0ba28406c3ea10
If you add the assembly to the classpath as above, you can run this gist
with:
```
python dump.py boneill-2 my-stream https://kinesis.us-east-1.amazonaws.com
us-east-1
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
