[
https://issues.apache.org/jira/browse/BEAM-8542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajo Thomas updated BEAM-8542:
-----------------------------
Description:
- While working with SNS IO for one of my work-related projects, I found that
the IO uses synchronous publishes during writes. I had a simple mock pipeline
where I was reading from a kinesis stream and publishing it to SNS using Beam's
SNS IO. For comparison, I also had a lamdba which did the same using
asynchronous publishes but was about 5x faster. Changing the SNS IO to use
async publishes would improve publish latencies.
- SNS IO also has some retry logic which isn't required as SNS clients can
handle retries. The retry logic in the SNS client is user-configurable and
therefore, an explicit retry logic in SNS IO is not required
I have a working version of the IO with these changes, will create a PR linking
this ticket to it once I get some feedback here.
was:
- While working with SNS IO for one of my work-related projects, I found that
the IO uses synchronous publishes during writes. I had a simple mock pipeline
where I was reading from a kinesis stream and publishing it to SNS using Beam's
SNS IO. For comparison, I also had a lamdba which did the same using
asynchronous publishes but was about 5x faster. Changing the SNS IO to use
async publishes would improve publish latencies.
- SNS IO also has some retry logic which isn't required as SNS clients can
handle retries. The retry logic in the SNS client is user-configurable and
therefore, an explicit retry logic in SNS IO is not required
> Add async write to AWS SNS IO & remove retry logic
> --------------------------------------------------
>
> Key: BEAM-8542
> URL: https://issues.apache.org/jira/browse/BEAM-8542
> Project: Beam
> Issue Type: Improvement
> Components: io-java-aws
> Reporter: Ajo Thomas
> Assignee: Ajo Thomas
> Priority: Minor
>
> - While working with SNS IO for one of my work-related projects, I found that
> the IO uses synchronous publishes during writes. I had a simple mock pipeline
> where I was reading from a kinesis stream and publishing it to SNS using
> Beam's SNS IO. For comparison, I also had a lamdba which did the same using
> asynchronous publishes but was about 5x faster. Changing the SNS IO to use
> async publishes would improve publish latencies.
> - SNS IO also has some retry logic which isn't required as SNS clients can
> handle retries. The retry logic in the SNS client is user-configurable and
> therefore, an explicit retry logic in SNS IO is not required
> I have a working version of the IO with these changes, will create a PR
> linking this ticket to it once I get some feedback here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
