Repository: spark Updated Branches: refs/heads/master 81bf87084 -> 81e3bfc16
[SPARK-14421] Upgrades protobuf dependency to 2.6.1 for the new version of KCL, and⦠## What changes were proposed in this pull request? When running with Kinesis Consumer Library (KCL), against a stream that contains aggregated data, the KCL needs access to protobuf to de-aggregate the records. Without this patch, that results in the following error message: ``` Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList ``` This PR upgrades the protobuf dependency within the kinesis-asl-assembly, and relocates that package (as not to conflict with Spark's use of 2.5.0), which fixes the above CNFE. ## How was this patch tested? Used kinesis word count example against a stream containing aggregated data. See: SPARK-14421 Author: Brian O'Neill <[email protected]> Closes #13054 from boneill42/protobuf-relocation-for-kcl. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/81e3bfc1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/81e3bfc1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/81e3bfc1 Branch: refs/heads/master Commit: 81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba Parents: 81bf870 Author: Brian O'Neill <[email protected]> Authored: Thu May 12 20:10:33 2016 +0100 Committer: Sean Owen <[email protected]> Committed: Thu May 12 20:10:33 2016 +0100 ---------------------------------------------------------------------- external/kinesis-asl-assembly/pom.xml | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/81e3bfc1/external/kinesis-asl-assembly/pom.xml ---------------------------------------------------------------------- diff --git a/external/kinesis-asl-assembly/pom.xml b/external/kinesis-asl-assembly/pom.xml index e057b78..6fb88eb 100644 --- a/external/kinesis-asl-assembly/pom.xml +++ b/external/kinesis-asl-assembly/pom.xml @@ -63,7 +63,12 @@ <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> - <scope>provided</scope> + <version>2.6.1</version> + <!-- + We are being explicit about version here and overriding the + spark default of 2.5.0 because KCL appears to have introduced + a dependency on protobuf 2.6.1 somewhere between KCL 1.4.0 and 1.6.1. + --> </dependency> <dependency> <groupId>org.glassfish.jersey.core</groupId> @@ -147,6 +152,15 @@ <include>*:*</include> </includes> </artifactSet> + <relocations> + <relocation> + <pattern>com.google.protobuf</pattern> + <shadedPattern>kinesis.protobuf</shadedPattern> + <includes> + <include>com.google.protobuf.**</include> + </includes> + </relocation> + </relocations> <filters> <filter> <artifact>*:*</artifact> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
