Re: How To subscribe a Kinesis Stream using enhance fanout?
Hi all, Just to give a quick update: there will be contributors from the AWS Kinesis team working on contributing enhanced fan out support to the connector. You can follow the progress here: https://issues.apache.org/jira/browse/FLINK-17688 Cheers, Gordon On Fri, May 15, 2020 at 5:55 PM orionemail wrote: > Hi, > > We also recently needed this functionality, unfortunately we were unable > to implement it ourselves so changed our plan accordingly. > > However we very much see the benefit for this feature and would be > interested in following the JIRA ticket. > > Thanks > > > > ‐‐‐ Original Message ‐‐‐ > On Thursday, 14 May 2020 11:34, Xiaolong Wang > wrote: > > Thanks, I'll check it out. > > On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai > wrote: > >> Hi Xiaolong, >> >> You are right, the way the Kinesis connector is implemented / the way the >> AWS APIs are used, does not allow it to consume Kinesis streams with >> enhanced fan-out enabled consumers [1]. >> Could you open a JIRA ticket for this? >> As far as I can tell, this could be a valuable contribution to the >> connector for Kinesis users who require dedicated throughput isolated from >> other running consumers. >> >> Cheers, >> Gordon >> >> [1] >> https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html >> >> On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang < >> xiaolong.w...@smartnews.com> wrote: >> >>> Hello Flink Community! >>> >>> I'm currently coding on a project relying on AWS Kinesis. With the >>> provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the >>> message. >>> >>> But as the main stream is used among several other teams, I was >>> required to use the enhance fanout of Kinesis. I checked the connector code >>> and found no implementations. >>> >>> Has this issue occurred to anyone before? >>> >>> Thanks for your help. >>> >> >
Re: How To subscribe a Kinesis Stream using enhance fanout?
Hi, We also recently needed this functionality, unfortunately we were unable to implement it ourselves so changed our plan accordingly. However we very much see the benefit for this feature and would be interested in following the JIRA ticket. Thanks ‐‐‐ Original Message ‐‐‐ On Thursday, 14 May 2020 11:34, Xiaolong Wang wrote: > Thanks, I'll check it out. > > On Thu, May 14, 2020 at 6:26 PM Tzu-Li (Gordon) Tai > wrote: > >> Hi Xiaolong, >> >> You are right, the way the Kinesis connector is implemented / the way the >> AWS APIs are used, does not allow it to consume Kinesis streams with >> enhanced fan-out enabled consumers [1]. >> Could you open a JIRA ticket for this? >> As far as I can tell, this could be a valuable contribution to the connector >> for Kinesis users who require dedicated throughput isolated from other >> running consumers. >> >> Cheers, >> Gordon >> >> [1] >> https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html >> >> On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang >> wrote: >> >>> Hello Flink Community! >>> >>> I'm currently coding on a project relying on AWS Kinesis. With the >>> provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the >>> message. >>> >>> But as the main stream is used among several other teams, I was >>> required to use the enhance fanout of Kinesis. I checked the connector code >>> and found no implementations. >>> >>> Has this issue occurred to anyone before? >>> >>> Thanks for your help.
Re: How To subscribe a Kinesis Stream using enhance fanout?
Hi Xiaolong, You are right, the way the Kinesis connector is implemented / the way the AWS APIs are used, does not allow it to consume Kinesis streams with enhanced fan-out enabled consumers [1]. Could you open a JIRA ticket for this? As far as I can tell, this could be a valuable contribution to the connector for Kinesis users who require dedicated throughput isolated from other running consumers. Cheers, Gordon [1] https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html On Wed, May 13, 2020 at 1:44 PM Xiaolong Wang wrote: > Hello Flink Community! > > I'm currently coding on a project relying on AWS Kinesis. With the > provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the > message. > > But as the main stream is used among several other teams, I was > required to use the enhance fanout of Kinesis. I checked the connector code > and found no implementations. > > Has this issue occurred to anyone before? > > Thanks for your help. >
How To subscribe a Kinesis Stream using enhance fanout?
Hello Flink Community! I'm currently coding on a project relying on AWS Kinesis. With the provided connector (flink-connector-kinesis_2.11;1.10.0), I can consume the message. But as the main stream is used among several other teams, I was required to use the enhance fanout of Kinesis. I checked the connector code and found no implementations. Has this issue occurred to anyone before? Thanks for your help.
