Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hey Priya, Having local dockerized integration testing would help devs to run the integration tests when they don't have AWS service credentials. You could have a look to the DynamoDB [1] or Kinesis [2] connector integration tests. DynamoDBContainer extends interface from Testcontainers [3].

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Thanks Danny/ Muhammet for the feedback. Updated the FLIP accordingly. I am not familiar with docker based integration testing but if that is what we have used for other AWS services, I can follow the same here. If you can share any existing sample code link that would be very helpful. Thanks P

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Thanks for the FLIP Priya. I have seen many custom implementations of SQS sinks, so having an Apache supported connector will be great. I agree on the points about implementation detail in the FLIP. There is still too much code really, we will review that in the PR. The FLIP is mainly to agree we

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hey Priya, Thanks for the FLIP and driving it! One question from my side on test plan: End to end integration tests that hit the real Amazon SQS service. These tests will be enabled when credentials are defined. Would it make sense to use here docker based integration testing using Testcont

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Thanks Samrat, that sounds great!. Updated "Yes" for Table support in FLIP. On 4/11/24, 10:18 PM, "Samrat Deb" mailto:decordea...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and kno

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hi Priya, Thank you for the FLIP, SQS connector will be a great addition for AWS connectors. I agree with Ahmed, it would be better to leave implementation details of the sink out of the FLIP and focus on how users of this connector will be interacting with it, mainly available configuration optio

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hi priya , I can help you with adding table api support and add patch for it . Let me know if that works for you to add it as part of this flip . Thank you Bests, Samrat On Fri, 12 Apr 2024 at 3:11 AM, Dhingra, Priya wrote: > Hi Ahmed, > > Thanks for reviewing it. I have updated the FLIP aga

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hi Ahmed, Thanks for reviewing it. I have updated the FLIP again. On your question of concerns on adding Table API support right now, the only reason I don't want to commit for it is that I am really not sure about the effort it involves and how to do/test it since I never worked with Table API

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Thanks Priya the FLIP now looks much better, I would move out some of the implementation details . Just highlight how the writer actually uses the client (the subitRequestEntries part) and how the Sink itself exposes its API to users (the Sink & builder part), other parts should not be part of the

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

On 4/9/24, 3:57 PM, "Dhingra, Priya" mailto:dhipr...@amazon.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi Ahmed and Samrat, Thanks a lot f

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

 Hi Ahmed and Samrat, Thanks a lot for all the feedbacks, this is my first ever contribution to apache Flink, hence I was bit unaware about few things but updated all of them as per your suggestions, thanks again for all the support here, much appreciated!! 1)  I am not sure why we need to su

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hi Priya, Thank you for the FLIP. sqs connector would be a great addition to the flink connector aws. +1 for all the queries raised by Ahmed. Adding to Ahmed's queries, I have a few more: 1. Different connectors provide different types of fault tolerant guarantees[1]. What type of fault toleran

Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hi Dhingra, thanks for raising the FLIP I am in favour of this addition in general. I have a couple of comments/questions on the FLIP. - I am not sure why we need to suppress warnings in the sink example in the FLIP? - You provided the sink example as it is the Public Interface, however the actual

[DISCUSS] FLIP-438: Amazon SQS Sink Connector

Hi Dev, I would like to start a discussion about FLIP-438: Amazon SQS Sink Connector. This FLIP is proposing to add support for AWS SQS sink in flink-connector-aws repo. For more details, see FLIP-438. Loo