mosche commented on pull request #16077: URL: https://github.com/apache/beam/pull/16077#issuecomment-1000260579
> Also, did you have a chance to run it against a real Kinesis instance and maybe compare the performance between this version and KPL's one (for AWS SDK v1)? I did quite some runs against real Kinesis using the integration test (using `--useLocalstack=false`). But I'll try and spend some time comparing performance. Though, the key issue will be the partitioner here. I'm not sure how to make it a meaningful & fair comparison. For instance, when using unique ids as partition key the new writer won't aggregate anything (except for some collisions on the effective hash key). On the other hand, using the provided random partitioner with the correct number of shards, it will aggregate as much as possible. The new writer certainly pushes a bit more responsibility to the user to think about partitioning upfront :/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
