[ https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606685#comment-16606685 ]
Hyukjin Kwon commented on SPARK-25361: -------------------------------------- (please avoid to set the target version which is usually reserved for committers) > Support for Kinesis Client Library 2.0 > -------------------------------------- > > Key: SPARK-25361 > URL: https://issues.apache.org/jira/browse/SPARK-25361 > Project: Spark > Issue Type: Improvement > Components: DStreams > Affects Versions: 2.3.1 > Reporter: Cory Locklear > Priority: Minor > Labels: aws, http2, kinesis, spark, streaming, structured > > Amazon has recently release KCL 2.0 which provides a HTTP/2 data retrieval > API for Kinesis. This API, along with the new enhanced fan-out features, > promises better data throughput and faster delivery of records to consumers, > specifically in multi-consumer environments. > [https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/] > My organization is very interested in getting support for these features into > Spark; is anyone already working on this? I'm happy to give it a go myself - > in fact, I'm currently attempting to create my own Spark package for this > functionality. Assuming that goes well, it's my intention to port it back to > core Spark. > If no one is already working on this, would anyone have any opinions on > whether this should be an inplace upgrade for the existing implementation, or > should this be a completely separate streaming source (kinesis2, for lack of > better name)? > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org