Repository: spark Updated Branches: refs/heads/branch-1.5 4c176d95d -> 7a49b6048
[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk Successfully ran kinesis demo on a live, aws hosted kinesis stream against master and 1.6 branches. For reasons I don't entirely understand it required a manual merge to 1.5 which I did as shown here: https://github.com/BrianLondon/spark/commit/075c22e89bc99d5e99be21f40e0d72154a1e23a2 The demo ran successfully on the 1.5 branch as well. According to `mvn dependency:tree` it is still pulling a fairly old version of the aws-java-sdk (1.9.37), but this appears to have fixed the kinesis regression in 1.5.2. Author: BrianLondon <br...@seatgeek.com> Closes #10492 from BrianLondon/remove-only. (cherry picked from commit ff89975543b153d0d235c0cac615d45b34aa8fe7) Signed-off-by: Sean Owen <so...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7a49b604 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7a49b604 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7a49b604 Branch: refs/heads/branch-1.5 Commit: 7a49b60489b3c161d3f94f018a1417ec2c4b4df5 Parents: 4c176d9 Author: BrianLondon <br...@seatgeek.com> Authored: Tue Jan 5 23:15:07 2016 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Tue Jan 5 23:19:17 2016 +0000 ---------------------------------------------------------------------- extras/kinesis-asl/pom.xml | 5 ----- .../org/apache/spark/streaming/kinesis/KinesisReceiver.scala | 1 + pom.xml | 1 - 3 files changed, 1 insertion(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/7a49b604/extras/kinesis-asl/pom.xml ---------------------------------------------------------------------- diff --git a/extras/kinesis-asl/pom.xml b/extras/kinesis-asl/pom.xml index 0b1421b..24d4264 100644 --- a/extras/kinesis-asl/pom.xml +++ b/extras/kinesis-asl/pom.xml @@ -60,11 +60,6 @@ <version>${aws.kinesis.client.version}</version> </dependency> <dependency> - <groupId>com.amazonaws</groupId> - <artifactId>aws-java-sdk</artifactId> - <version>${aws.java.sdk.version}</version> - </dependency> - <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-core</artifactId> <scope>test</scope> http://git-wip-us.apache.org/repos/asf/spark/blob/7a49b604/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala ---------------------------------------------------------------------- diff --git a/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala b/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala index 22324e8..2c2dbfd 100644 --- a/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala +++ b/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala @@ -178,6 +178,7 @@ private[kinesis] class KinesisReceiver( workerThread.setName(s"Kinesis Receiver ${streamId}") workerThread.setDaemon(true) workerThread.start() + logInfo(s"Started receiver with workerId $workerId") } http://git-wip-us.apache.org/repos/asf/spark/blob/7a49b604/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 3c745c2..627886f 100644 --- a/pom.xml +++ b/pom.xml @@ -151,7 +151,6 @@ <avro.version>1.7.7</avro.version> <avro.mapred.classifier>hadoop2</avro.mapred.classifier> <jets3t.version>0.7.1</jets3t.version> - <aws.java.sdk.version>1.9.16</aws.java.sdk.version> <aws.kinesis.client.version>1.3.0</aws.kinesis.client.version> <!-- org.apache.httpcomponents/httpclient--> <commons.httpclient.version>4.3.2</commons.httpclient.version> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
