We cannot compensate blocked threads with new ones forever. At some point we will have to limit new threads which will bring us to the same problem, no?
I do like the idea, however, of users creating their own thread pools and utilizing them for execution, as proposed by Vladimir. D. On Mon, Feb 13, 2017 at 6:37 AM, Alexey Goncharuk < [email protected]> wrote: > Vladimir, > > I like the idea of compensating threads with new ones. However, I do not > see why users should notify Ignite when a new distributed operation is > started because Ignite already is the task executor. We can intercept > (virtually) all such calls and adjust pool size automatically. No need to > introduce ManagedBlocker. > > --AG > > 2017-02-11 23:43 GMT+03:00 Vladimir Ozerov <[email protected]>: > > > Igniters, > > > > We have one well-known limitation: one distributed opeartion cannot be > > executed from within another if they both end up in the same pool, > because > > it will lead to starvation and hang. Most notable example - running data > > streamer from compute task. > > > > Users with complex logic will have the same issue if they want to execute > > one job from another. I think it could be resolved with introduction of > > custom executors and routing computing jobs to them. > > > > Config: > > ExecutorConfiguration execCfg = new > > ExecutorConfiguration().setName("myExecutor").setThreads(8); > > > > Starting MR: > > ignite.compute().withExecutor("myExecutor").execute(...); > > > > Another improvement we can apply here is managed blockers, similar to > > ForkJoinPool.ManagedBlocker [1]. If user is to execute distributed > > operation, he can infrom us about it, and we will compensate blocked > thread > > with a new one. > > > > Thoughts? > > > > [1] > > https://docs.oracle.com/javase/8/docs/api/java/util/ > > concurrent/ForkJoinPool.ManagedBlocker.html > > >
