Hey guys, Firstly, I'd like to recap the conversation a little bit so that we have a shared understanding.
1. We don't want another environmental variable to control daemon parameters. 2. (Quoting Adam) We want to have a config file in the project directory somewhere (whatever that ends up being) that can declare the minimum heap and permspace requirements of the build, and maybe other JVM args that are required to launch the build. Secondly, I'd like to share my humble feedback on some of the things mentioned on the thread :) It would be cool if we approach the problem incrementally. That is, first we make it possible to configure the daemon vm parameters. Then, if we concur that it is important at this point, we can discuss a different story, that is what should be the default daemon vm args. IMHO, this discussion should include: 1. consideration around backwards compatibility 2. compatibility in the context of tooling api - slightly tricky due to the tooling api compatibility contract and the way tooling api is used, e.g. under the hood of the IDE plugins. Cheers! On Wed, Dec 7, 2011 at 7:57 PM, Kris De Volder <[email protected]> wrote: > > What kind of management does it need to do? i.e. what would very good > management look like? > > I've already given some thoughts on this to Adam in the Gradle issue > tracker issue about 'runaway daemon'. > > But this is a hard question. I don't have the answer, just some random > thoughts :-) > > What I definitely don't want: > - multiple daemons parking themselves in memory practically forever. > > So one thing it needs to do is limit the number of daemons that are > allowed to exist concurrently. > I think the 'kill' logic should be more and more aggressive based on # of > daemons already in use. > And it would be nice if there was some API controlled way to set a hard > limit on the # of daemons allowed > to exist at the same time. > > The 'daemon manager' should be able to manage all Gradle daemons, across > gradle versions > (a common thing in my use has been multiple daemons get started because of > different gradle > versions). Sometimes one of those daemons is 'by accident' because gradle > version set wrong... > and only used once. But it sticks around for what seems forever. > > Also it has been a problem that daemons would get 'stuck' or something. > And then sit there > 'defunct' while another daemon got started in its place. This way, I've > seen 10+ concurrent daemons on my machine. > So daemons should have a robust 'self destruct'. If something is wrong, or > if they haven't been used in some time, they should just commit suicide. > And this mechanism should be robust and not itself fail or get stuck > somehow. > > Another useful thing would be that I could somehow 'own my daemon' and > kill it when I know I am not > going to be using it again (e.g. if the IDE is shutdown, it's 'associated' > daemon probably should also > go away as it probably won't be used anymore). > > > > Another thing to consider: as you say, indeed the JVM never gives > > > memory back to the OS. The only way to really give it back is to > > > terminate the JVM (i.e. daemon). So I'd suggest not having the > > > daemon stick around for too long, unless it is really being used > > > quite frequently… > > > > At the moment, the default idle time is 3 hours. This will be user > > configurable though. > > I exaggerate, but 3 minutes would be better :-) > I see we are somewhat far apart on the meaning of 'frequently'. > > > > > > My compromise would be a combination of two strategies: > > > - be generous with max memory limits (so things work without much > > > manual tweaking) > > > - be conservative and 'kill' unused daemons of fairly quickly (so > > > unused resources are returned to OS) > > > > This sounds right to me, now we just need to agree on what the idle > > timeout should be :) > > Tricky :-). My feeling is... something that can be measured in minutes, > not hours :-) > > Maybe 10 minutes? Maybe 5 minutes, maybe 15 minutes (that's long!)... > 3 hours in my world... is practically equivalent to 'forever'. You might > as well have no timeout. > > Just put in to perspective: how much do you gain from not having to pay > the startup > cost once every 3 hours? Is that worth permanently sacrificing +1Gb of > memory? > > If it is tweakable somehow via the tooling API it is less important what > the > exact value would be, because I can tweak it from the IDE if I don't like > its default setting. > > > > > If we are too conservative we almost completely negate the user > > experience benefit of using the daemon. > > > > The same is true if you keep the daemon there for too long. Because it > degrades overall performance (memory given to the daemon does hurt > performance because it isn't available for other processes and OS level > caches). > > I'm more worried about overall performance degradation than speeding up > the occasional gradle command. > > This is especially the case *if* you reach the 'swap tipping point', where > it makes the difference between a usable or unusable system. Paying the > startup cost for Gradle degrades the experience, but doesn't really make my > machine unusable. > > This problem is worse for me, because as a Tooling API user, I do not have > the option of not using the daemon. A commandline user always has this > option if the daemon doesn't work for them. > > > > I believe, it just isn't worth the price to keep a daemon parked in > > > precious memory for extended periods of time, if it is not being > > > used. Rather than counting on paging it out, its much better to > > > shut it down. If it is paged out, it won't give you great > > > performance next time you try to use it anyway. > > > > It would be interesting to test if paging in is cheaper than > > bootstrapping Groovy or not. I'm not suggesting we do this. > > Testing a bit to find out doesn't hurt :-) > > Also keep in mind, I don't think the JVM is ever really 100% 'idle'. So > even when not doing anything, it is probably still touching some of the > memory in its heap every so often. So try to factor in a form of 'constant > drag' on the system just for having a parked daemon. > > > > > Shutting it down is costly though in terms of user experience so > > that's not without consequence. This is a fine balancing act. > > Keeping it there may be more costly in certain cases (if it causes paging). > Also... shutting it down, in itself doesn't really hurt user experience. > > It is the startup that hurts, not the shutdown. > > So the shutdown only hurts assuming that I'm actually going to be using > the same daemon again, soon. > > > No problem, no offence taken. I just had to clarify what I was > > saying. Your input is very valuable and I think we are increasing > > the understanding here. Although, personally I don't think I have a > > good idea on these defaults yet. > > That's good. Glad to hear that. I think it is good to keep an open mind > about this. And > also it should point to the importance of allowing the user to override > the defaults. You do your best to set sensible defaults, but accept that > they may not be right for every use case. > > I'm speaking as a Tooling API client here... and for me, there currently > is no way to tweak any of these defaults. > > Thanks for the interesting discussion :-) > > Kris > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > -- Szczepan Faber Principal engineer@gradleware Lead@mockito
