mdedetrich commented on PR #226: URL: https://github.com/apache/incubator-pekko-connectors/pull/226#issuecomment-1686584088
> Then if a user wants to consume from more than 16 kinesis streams (i.e creates more than 16 KinesisSchedulerSource), then the thread pool is fully utilized, creating a deadlock. That's why I opted for creating a thread just for running the kinesis Scheduler, as in the example [documentation](https://docs.aws.amazon.com/streams/latest/dev/kcl2-standard-consumer-java-example.html) from AWS. Maybe I misunderstand, but I don't think this would create a deadlock. Deadlocks are in fact caused in the converse situation, i.e. when you don't deal with blocking code correctly (i.e. either by not using `Future#blocking` or some `IODispatcher`. Actually our resident @jrudolph stated as such here https://discuss.lightbend.com/t/blocking-io-dispatcher-without-a-batchingexecutor/5954/6 The only case I can think of when having a legitimate deadlock is if you have multiple blocking IO operators that share the same `IODispatcher` and are combined into a single stream. In this case if you have more than 16 sources you can create a deadlock since the different blocking IO sources will block the `IODispatcher` at exactly the same time. If this is the case then you just need to use multiple `IODispatchers`. https://stackoverflow.com/a/71268911 is a good explanation of the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
