Regarding licensing issue - let me add some details. It was a problem with old version of KPL that is used in KinesisIO.Write under the hood. So, that is why KinesisIO was put into separate package, aside from other AWS IO connectors. Now it’s not an issue anymore because KPL moved to Apache License.
I’d also add that a while ago we decided to move all AWS related IO connectors to AWS SDK v2, so, potentially, v1 connectors will be deprecated soon. The only issue with that, for now, is again KinesisIO.Write since KPL uses AWS SDK v1internally. Though we could keep using "v1 version" of KPL if it will be possible to move KinesisIO.Write user API to v2 (?) because we don’t want to keep v1 deps and change API in the future. > On 3 Sep 2020, at 16:54, Jonothan Farr <[email protected]> wrote: > > Hi Mani. The connector that’s in io/kinesis is based on version 1 of the AWS > SDK for Java and the one in amazon-web-services2 is based on the version 2 > SDK. My understanding is that there were some licensing issues preventing it > from being bundled together with the other AWS IOs initially but those have > since been resolved. For now the two connectors are functionally equivalent > but the v1 is deprecated so you should use v2 instead. > >> On Sep 3, 2020, at 1:45 AM, Sunny, Mani Kolbe <[email protected]> wrote: >> >> >> Hi Jonothan, >> >> That is good news! I was of the impression, it is not supported or have to >> enable some flag. By the way, what is the difference between KinesisIO and >> aws V2. Is aws V2 is a set of classes to support AWS related connectors and >> KinesisIO is referring back to them? >> >> I mean for read/write from Kinesis, KinesisIO is the still the way to go? >> >> Regards, >> Mani >> >> From: Jonothan Farr <[email protected]> >> Sent: Thursday, September 3, 2020 1:02 AM >> To: [email protected] >> Subject: Re: KinesisIO - aggregation >> >> CAUTION: This email originated from outside of D&B. Please do not click >> links or open attachments unless you recognize the sender and know the >> content is safe. >> >> KinesisIO works fine for me with aggegated records. Here's where >> deaggregate() is called in v1: >> >> https://github.com/apache/beam/blob/9f4d54a2e60ec45150437c0b050f4c73cca91f36/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java#L170 >> >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F9f4d54a2e60ec45150437c0b050f4c73cca91f36%2Fsdks%2Fjava%2Fio%2Fkinesis%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fsdk%2Fio%2Fkinesis%2FSimplifiedKinesisClient.java%23L170&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515965488&sdata=wLQ2ou1IJOgNW%2FJ3JNgS1WeWnYyQnE5CSEparVAGAOE%3D&reserved=0> >> >> And v2: >> >> https://github.com/apache/beam/blob/9f4d54a2e60ec45150437c0b050f4c73cca91f36/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java#L181 >> >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F9f4d54a2e60ec45150437c0b050f4c73cca91f36%2Fsdks%2Fjava%2Fio%2Famazon-web-services2%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fsdk%2Fio%2Faws2%2Fkinesis%2FSimplifiedKinesisClient.java%23L181&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515975481&sdata=hqEy9YHc70qy74KFG0CNU3%2BJIcTxpW2jZF7xjpagLFo%3D&reserved=0> >> >> Were you having problems getting it to work? >> >> On Wed, Sep 2, 2020 at 8:22 AM Alexey Romanenko <[email protected] >> <mailto:[email protected]>> wrote: >> Yes, it’s not supported for now, but, at the first sight, it seems that we >> just need to call UserRecord.deaggregate() in GetKinesisRecordsResult in >> case if record is aggregated. >> >> >> On 2 Sep 2020, at 14:30, Sunny, Mani Kolbe <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Alexey, >> >> I am looking for reading Kinesis stream with that has aggregated record. >> From your reply, I take that it is currently not supported? Could that be >> solved by adding an uncompression function on the pipeline ? >> >> Regards, >> Mani >> >> From: Alexey Romanenko <[email protected] >> <mailto:[email protected]>> >> Sent: Tuesday, September 1, 2020 6:04 PM >> To: [email protected] <mailto:[email protected]> >> Subject: Re: KinesisIO - aggregation >> >> CAUTION: This email originated from outside of D&B. Please do not click >> links or open attachments unless you recognize the sender and know the >> content is safe. >> >> Hello Mani, >> >> For Write part it should be already supported since KinesisIO uses KPL to >> write records under the hood. So, it’s just a question of proper >> configuration [1][2][3] >> For Read part, since it’s based on AWS API, it’s more complicated and we >> need to add a support for this explicitly. >> >> [1] >> https://github.com/apache/beam/blob/afa59c463ca9c5d42a9ab13402a72c8b95240fa5/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java#L657 >> >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2Fafa59c463ca9c5d42a9ab13402a72c8b95240fa5%2Fsdks%2Fjava%2Fio%2Fkinesis%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fsdk%2Fio%2Fkinesis%2FKinesisIO.java%23L657&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515975481&sdata=lXWOD%2BvVO6e6AkOmoMjVDIitGnLfNnoOeunxAIMwrbw%3D&reserved=0> >> [2] https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fstreams%2Flatest%2Fdev%2Fkinesis-kpl-config.html&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515985474&sdata=%2FuJFYGCd3wrMW2u7Lxkm9uR9jMQv0ud8JMijSTj4Yps%3D&reserved=0> >> [3] >> https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties >> >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fawslabs%2Famazon-kinesis-producer%2Fblob%2Fmaster%2Fjava%2Famazon-kinesis-producer-sample%2Fdefault_config.properties&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515985474&sdata=vULfC%2B4HbDgUeAc94rurIIvxm6AbX8RX7lfxOok2yKk%3D&reserved=0> >> >> >> On 31 Aug 2020, at 16:12, Sunny, Mani Kolbe <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello, >> >> Does Beam have plans to support Kinesis records with aggregation[1]? I see >> some code [2] in the repo related to that. Is it planned for any near future >> releases? >> >> Regards, >> Mani >> >> >> [1] >> https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation >> >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fstreams%2Flatest%2Fdev%2Fkinesis-kpl-concepts.html%23kinesis-kpl-concepts-aggretation&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515995467&sdata=UpJo6P8fkm632IyTaeLrityfGRSdevN6ND%2FfD2MvCIM%3D&reserved=0> >> [2] >> https://github.com/apache/beam/blob/22822b8c611f4d8888d3b2039e4f7e7d98fcab39/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java#L181 >> >> <https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F22822b8c611f4d8888d3b2039e4f7e7d98fcab39%2Fsdks%2Fjava%2Fio%2Famazon-web-services2%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fsdk%2Fio%2Faws2%2Fkinesis%2FSimplifiedKinesisClient.java%23L181&data=02%7C01%7CSunnyM%40dnb.com%7C9ed5ba8b092c4643e59e08d84f9ca5e0%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637346881515995467&sdata=Z55XBOaVjlRrJtpEWI7ck7PlGx7SpABaiTm3HAVp%2Bec%3D&reserved=0>
