[
https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-25361.
----------------------------------
Resolution: Incomplete
> Support for Kinesis Client Library 2.0
> --------------------------------------
>
> Key: SPARK-25361
> URL: https://issues.apache.org/jira/browse/SPARK-25361
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 2.3.1
> Reporter: Cory Locklear
> Priority: Minor
> Labels: aws, bulk-closed, http2, kinesis, spark, streaming,
> structured
>
> Amazon has recently released version 2.0 of the KCL which provides a HTTP/2
> data retrieval API for Kinesis. This API, along with the new enhanced
> fan-out features, promises better data throughput and faster delivery of
> records to consumers, specifically in multi-consumer environments.
> [https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/]
> My organization is very interested in getting support for these features into
> Spark; is anyone already working on this? I'm happy to give it a go myself -
> in fact, I'm currently attempting to create my own Spark package for this
> functionality. Assuming that goes well, it's my intention to port it back to
> core Spark.
> If no one is already working on this, would anyone have any opinions on
> whether this should be an inplace upgrade for the existing implementation, or
> should this be a completely separate streaming source (kinesis2, for lack of
> better name)?
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
