This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 3aad111 [SPARK-29677][DSTREAMS] amazon-kinesis-client 1.12.0
3aad111 is described below
commit 3aad111fbe2a873a539e4c8c44179d9573405f8c
Author: Eric Meisel <[email protected]>
AuthorDate: Sat Nov 2 16:42:49 2019 -0500
[SPARK-29677][DSTREAMS] amazon-kinesis-client 1.12.0
### What changes were proposed in this pull request?
Upgrading the amazon-kinesis-client dependency to 1.12.0.
### Why are the changes needed?
The current amazon-kinesis-client version is 1.8.10. This version depends
on the use of `describeStream`, which has a hard limit on an AWS account (10
reqs / second). Versions 1.9.0 and up leverage `listShards`, which has no such
limit. For large customers, this can be a major problem.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Existing tests
Closes #26333 from etspaceman/kclUpgrade.
Authored-by: Eric Meisel <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit be022d9aeeb6b39f7b51427d848600e15a100c30)
Signed-off-by: Sean Owen <[email protected]>
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index f79e388..aad9844 100644
--- a/pom.xml
+++ b/pom.xml
@@ -142,7 +142,7 @@
<codahale.metrics.version>3.1.5</codahale.metrics.version>
<avro.version>1.8.2</avro.version>
<avro.mapred.classifier>hadoop2</avro.mapred.classifier>
- <aws.kinesis.client.version>1.8.10</aws.kinesis.client.version>
+ <aws.kinesis.client.version>1.12.0</aws.kinesis.client.version>
<!-- Should be consistent with Kinesis client dependency -->
<aws.java.sdk.version>1.11.271</aws.java.sdk.version>
<!-- the producer is used in tests -->
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
