Guys, I feel very uneasy about proliferation of thread pools in Ignite. I suggest we take step back. Most of the users (including yours truly) have no idea how to correctly configure the existing thread pools, what those thread pools are for (like, management thread pool, or marshaller cache thread pool, or async callback thread pool, or peer class loading thread pool, or - my personal favorite - utility cache thread pool, just to name a few), and why they should worry about them altogether. In reality, there should probably be only 2 parameters exposed at the configuration level: the size of Ignite's internal (or "system") thread pool, and the size of the user thread pool. Or alternatively, one number could indicate the total number of thread available to Ignite (hard upper bound) and the other would be a ratio of system threads to user threads. Either way, it's Ignite's internal business how to manage the thread pools given the constraints. For example, Ignite may choose to allocate the system threads across twenty thread pools, or maybe just one -- the users do not care. What users do care about is the size of the user thread pool because it's the one that executes their code. I know it's not the case in the current version, but going forward, Ignite must ensure that all user code (including all sorts of listeners and callbacks) is executed on the user threads. In any case, it's the user who is ultimately in charge of figuring out the correct thread pool sizes, and I believe having just 2 knobs to turn vs. 7 or 8 as is the case today, would make things a lot simpler and reduce the guess work. Speaking of query execution, it feels natural to have it executed on the user thread pool, as the query is a user-triggered action executing a user's code (even though the code is not Java in this case, but something that looks rather like SQL). Regards, Andrey _____________________________ From: Dmitriy Setrakyan <dsetrak...@apache.org<mailto:dsetrak...@apache.org>> Sent: Monday, October 24, 2016 9:23 AM Subject: Re: Create separate thread pool for query execution To: <dev@ignite.apache.org<mailto:dev@ignite.apache.org>>
I think this makes sense. Do we have a ticket filed? On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <yzhda...@apache.org<mailto:yzhda...@apache.org>> wrote: > > As far as the query thread pool, how many threads should be in it by > > default? What happens is the query is run from compute task - will the > > execution be shifted from the public to the query thread pool? > > Pool management should be the same as for other pools. > > Remote executions should be executed in query pool. Local should run > synchronously. > > --Yakov > > 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan > <dsetrak...@apache.org<mailto:dsetrak...@apache.org>>: > > > Sergey, I think we already have a general approach with 3 or 4 thread > pools > > we have today. > > > > As far as the query thread pool, how many threads should be in it by > > default? What happens is the query is run from compute task - will the > > execution be shifted from the public to the query thread pool? > > > > D. > > > > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov > > <skoz...@gridgain.com<mailto:skoz...@gridgain.com>> > > wrote: > > > > > Hi > > > > > > It seems we need a set of dedicated pools for various purposes. Could > we > > > design a general apporach for this like following: > > > - define dedicated pools in the grid configuration > > > - run a query/compute/etc in the particular pool > > > > > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov > > > <voze...@gridgain.com<mailto:voze...@gridgain.com> > > > > > wrote: > > > > > > > Romain, > > > > We do not pre-start threads in our pools on Ignite start. Moreover, > > idle > > > > threads are stopped after some timeout. > > > > > > > > 24 окт. 2016 г. 8:57 пользователь "Gilles, Romain" < > > > > romain.gil...@misys.com<mailto:romain.gil...@misys.com>> > > > > написал: > > > > > > > > Hi igniters, > > > > I'm not against to create a thread pool for each features but I have > > some > > > > difficulty to minimize the number of threads required to start > > ignite... > > > If > > > > you add too many thread pools does this will not increase the number > of > > > > threads at startup? > > > > > > > > Thanks. > > > > > > > > Romain > > > > > > > > > > > > ________________________________ > > > > De: Dmitriy Setrakyan > > > > <dsetrak...@apache.org<mailto:dsetrak...@apache.org>> > > > > Envoye: 23 oct. 2016 23:00 > > > > A: dev@ignite.apache.org<mailto:dev@ignite.apache.org> > > > > Objet: Re: Create separate thread pool for query execution > > > > > > > > How about long running compute? Do we need a separate thread pool for > > it > > > as > > > > well? > > > > > > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin < > > > sergi.vlady...@gmail.com<mailto:sergi.vlady...@gmail.com>> > > > > wrote: > > > > > > > > > +1 > > > > > > > > > > Sergi > > > > > > > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov > > > > > <voze...@gridgain.com<mailto:voze...@gridgain.com>>: > > > > > > > > > > > Igniters, > > > > > > > > > > > > Currently if several long-running queries are submitted, it may > > stall > > > > all > > > > > > cache operations because all theads in the system pool will be > busy > > > > with > > > > > > queries. > > > > > > > > > > > > I think it makes sense to move queries execution into separate > > > > dedicated > > > > > > thread pool. As we have timeouts for core pool threads now, it > > should > > > > not > > > > > > affect memory consumption or startup time anyhow. Thoughts? > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > "Misys" is the trade name of the Misys group of companies. This email > > and > > > > any attachments have been scanned for known viruses using multiple > > > > scanners. This email message is intended for the named recipient > only. > > It > > > > may be privileged and/or confidential. If you are not the named > > recipient > > > > of this email please notify us immediately and do not copy it or use > it > > > for > > > > any purpose, nor disclose its contents to any other person. This > email > > > does > > > > not constitute the commencement of legal relations between you and > > Misys. > > > > Please refer to the executed contract between you and the relevant > > member > > > > of the Misys group for the identity of the contracting party with > which > > > you > > > > are dealing. > > > > > > > > > > > > > > > > -- > > > Sergey Kozlov > > > GridGain Systems > > > www.gridgain.com<http://www.gridgain.com> > > > > > >
