[
https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-25361:
---------------------------------
Target Version/s: (was: 3.0.0)
> Support for Kinesis Client Library 2.0
> --------------------------------------
>
> Key: SPARK-25361
> URL: https://issues.apache.org/jira/browse/SPARK-25361
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 2.3.1
> Reporter: Cory Locklear
> Priority: Minor
> Labels: aws, http2, kinesis, spark, streaming, structured
>
> Amazon has recently release KCL 2.0 which provides a HTTP/2 data retrieval
> API for Kinesis. This API, along with the new enhanced fan-out features,
> promises better data throughput and faster delivery of records to consumers,
> specifically in multi-consumer environments.
> [https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/]
> My organization is very interested in getting support for these features into
> Spark; is anyone already working on this? I'm happy to give it a go myself -
> in fact, I'm currently attempting to create my own Spark package for this
> functionality. Assuming that goes well, it's my intention to port it back to
> core Spark.
> If no one is already working on this, would anyone have any opinions on
> whether this should be an inplace upgrade for the existing implementation, or
> should this be a completely separate streaming source (kinesis2, for lack of
> better name)?
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
