[
https://issues.apache.org/jira/browse/BEAM-13175?focusedWorklogId=700547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-700547
]
ASF GitHub Bot logged work on BEAM-13175:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Dec/21 12:08
Start Date: 23/Dec/21 12:08
Worklog Time Spent: 10m
Work Description: mosche edited a comment on pull request #16077:
URL: https://github.com/apache/beam/pull/16077#issuecomment-1000260579
> Also, did you have a chance to run it against a real Kinesis instance and
maybe compare the performance between this version and KPL's one (for AWS SDK
v1)?
I did quite some runs against real Kinesis using the integration test (using
`--useLocalstack=false`). But I'll try and spend some time comparing
performance.
Though, the key issue will be the partitioner here. I'm not sure how to make
it a meaningful & fair comparison. For instance, when using unique ids as
partition key the new writer won't aggregate anything (except for some
collisions on the effective hash key). On the other hand, using the provided
random partitioner with the correct number of shards, it will aggregate as much
as possible. The new writer certainly pushes a bit more responsibility to the
user to think about partitioning upfront :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 700547)
Time Spent: 2h 40m (was: 2.5h)
> Implement missing KinesisIO.Write for aws2
> ------------------------------------------
>
> Key: BEAM-13175
> URL: https://issues.apache.org/jira/browse/BEAM-13175
> Project: Beam
> Issue Type: New Feature
> Components: io-java-aws
> Reporter: Moritz Mack
> Assignee: Moritz Mack
> Priority: P2
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The Kinesis Producer library KPL isn't available for aws2. Hence, we cannot
> trivially port the old KinesisIO.Write over.
> But at the same time KPL also doesn't align with the ideas behind SDFs. So
> it's a good opportunity to implement it properly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
