Vladimir, I like the idea of compensating threads with new ones. However, I do not see why users should notify Ignite when a new distributed operation is started because Ignite already is the task executor. We can intercept (virtually) all such calls and adjust pool size automatically. No need to introduce ManagedBlocker.
--AG 2017-02-11 23:43 GMT+03:00 Vladimir Ozerov <[email protected]>: > Igniters, > > We have one well-known limitation: one distributed opeartion cannot be > executed from within another if they both end up in the same pool, because > it will lead to starvation and hang. Most notable example - running data > streamer from compute task. > > Users with complex logic will have the same issue if they want to execute > one job from another. I think it could be resolved with introduction of > custom executors and routing computing jobs to them. > > Config: > ExecutorConfiguration execCfg = new > ExecutorConfiguration().setName("myExecutor").setThreads(8); > > Starting MR: > ignite.compute().withExecutor("myExecutor").execute(...); > > Another improvement we can apply here is managed blockers, similar to > ForkJoinPool.ManagedBlocker [1]. If user is to execute distributed > operation, he can infrom us about it, and we will compensate blocked thread > with a new one. > > Thoughts? > > [1] > https://docs.oracle.com/javase/8/docs/api/java/util/ > concurrent/ForkJoinPool.ManagedBlocker.html >
