[
https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cory Locklear updated SPARK-25361:
----------------------------------
Description:
Amazon has recently released version 2.0 of the KCL which provides a HTTP/2
data retrieval API for Kinesis. This API, along with the new enhanced fan-out
features, promises better data throughput and faster delivery of records to
consumers, specifically in multi-consumer environments.
[https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/]
My organization is very interested in getting support for these features into
Spark; is anyone already working on this? I'm happy to give it a go myself -
in fact, I'm currently attempting to create my own Spark package for this
functionality. Assuming that goes well, it's my intention to port it back to
core Spark.
If no one is already working on this, would anyone have any opinions on whether
this should be an inplace upgrade for the existing implementation, or should
this be a completely separate streaming source (kinesis2, for lack of better
name)?
Thanks!
was:
Amazon has recently release KCL 2.0 which provides a HTTP/2 data retrieval API
for Kinesis. This API, along with the new enhanced fan-out features, promises
better data throughput and faster delivery of records to consumers,
specifically in multi-consumer environments.
[https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/]
My organization is very interested in getting support for these features into
Spark; is anyone already working on this? I'm happy to give it a go myself -
in fact, I'm currently attempting to create my own Spark package for this
functionality. Assuming that goes well, it's my intention to port it back to
core Spark.
If no one is already working on this, would anyone have any opinions on whether
this should be an inplace upgrade for the existing implementation, or should
this be a completely separate streaming source (kinesis2, for lack of better
name)?
Thanks!
> Support for Kinesis Client Library 2.0
> --------------------------------------
>
> Key: SPARK-25361
> URL: https://issues.apache.org/jira/browse/SPARK-25361
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 2.3.1
> Reporter: Cory Locklear
> Priority: Minor
> Labels: aws, http2, kinesis, spark, streaming, structured
>
> Amazon has recently released version 2.0 of the KCL which provides a HTTP/2
> data retrieval API for Kinesis. This API, along with the new enhanced
> fan-out features, promises better data throughput and faster delivery of
> records to consumers, specifically in multi-consumer environments.
> [https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/]
> My organization is very interested in getting support for these features into
> Spark; is anyone already working on this? I'm happy to give it a go myself -
> in fact, I'm currently attempting to create my own Spark package for this
> functionality. Assuming that goes well, it's my intention to port it back to
> core Spark.
> If no one is already working on this, would anyone have any opinions on
> whether this should be an inplace upgrade for the existing implementation, or
> should this be a completely separate streaming source (kinesis2, for lack of
> better name)?
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
