psolomin commented on PR #23540: URL: https://github.com/apache/beam/pull/23540#issuecomment-1275253817
@mosche many thanks for the detailed feedback 👍 This was my first attempt. I basically tried to pull some pieces from #9899 to get myself more familiar with this codebase, and then after basic tests with real Kinesis stream found out this approach doesn't seem to work when I consider resharding. > Overall I'm honestly a bit torn and I'm wondering about your experience working on the IO? First time I am putting my hands on into the guts of a Beam IO, but not the first time dealing with Kinesis consumers overall. > pretty close and mostly misses the resharding stuff My impression was that I needed to re-think this approach - namely, avoid overloading `ShardReadersPool` and use a new `ShardSubscribersPool` as you suggested. > Is it worth it to step back from the current implementation and think through how a pure subscriber based implementation would look like without the existing luggage? I felt the same and that's actually the thing I'm busy now with. Once I have something tangible which covers re-sharding cleanly, may I ping you again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
