On 09/12/2011, at 12:36 PM, Adam Murdoch wrote: > > On 09/12/2011, at 1:52 AM, Szczepan Faber wrote: > >> Hey guys, >> >> Firstly, I'd like to recap the conversation a little bit so that we have a >> shared understanding. >> >> 1. We don't want another environmental variable to control daemon parameters. >> 2. (Quoting Adam) We want to have a config file in the project directory >> somewhere (whatever that ends up being) that can declare the minimum heap >> and permspace requirements of the build, and maybe other JVM args that are >> required to launch the build. >> >> Secondly, I'd like to share my humble feedback on some of the things >> mentioned on the thread :) >> >> It would be cool if we approach the problem incrementally. That is, first we >> make it possible to configure the daemon vm parameters. Then, if we concur >> that it is important at this point, we can discuss a different story, that >> is what should be the default daemon vm args. IMHO, this discussion should >> include: 1. consideration around backwards compatibility 2. compatibility in >> the context of tooling api - slightly tricky due to the tooling api >> compatibility contract and the way tooling api is used, e.g. under the hood >> of the IDE plugins. > > Here's the plan for milestone-7: > > There will be 3 properties available that affect how Gradle runs the build: > > * org.gradle.jvmargs - the JVM args required for the build. > * org.gradle.daemon - whether the daemon is enabled or disabled by default. > * org.gradle.daemon.idletimeout - for tweaking purposes. > > There will be a few places you can specify these, in increasing order of > precedence: > > * In the 'gradle.properties' in the root directory of a build. This will > allow you to enable the daemon by default for a particular build, or to > declare the default JVM args to use for the build. > > * In '~/.gradle/gradle.properties'. This will allow the user to override > these settings, or apply some settings globally. So, for example, on your CI > machine, you can disable the daemon for all build, but still enable it for > all your projects on a dev machine. > > * As a system property, via the -D command-line option. This will allow the > user to override the above, or tweak for a particular invocation. > > * Using the --daemon/--no-daemon command-line options. > > Some things to note about this in milestone-7: > > This is very much experimental, so may change completely or (not very likely) > disappear in milestone-8. > > There are a couple of restrictions: org.gradle.jvmargs will be honoured only > by the tooling API and when you are running with the daemon. And > org.gradle.daemon is ignored by the tooling API. It will always use the > daemon. > > Also, we will still use the old default jvm args of -Xmx1024m > -XX:MaxPermSize=256m if you don't specify anything. > > For milestone-8, the plan is: > > 1. Come up with a solution to allow org.gradle.jvmargs to be honoured when > you're not using the daemon. > 2. If this is all working ok for milestone-7, lock it down as an official > interface. > 3. Provide a way for the tooling API to override and/or configure these > values.
4. Allow you to specify the required gradle version/distribution/installation for the build, and deprecate gradle/wrapper/gradle-wrapper.properties 5. Allow you to specify the required java version/java home for the build. > > For (milestone-8 + 1): > > 1. Come up with some better defaults. > > >> >> Cheers! >> >> On Wed, Dec 7, 2011 at 7:57 PM, Kris De Volder <[email protected]> wrote: >> > What kind of management does it need to do? i.e. what would very good >> > management look like? >> >> I've already given some thoughts on this to Adam in the Gradle issue tracker >> issue about 'runaway daemon'. >> >> But this is a hard question. I don't have the answer, just some random >> thoughts :-) >> >> What I definitely don't want: >> - multiple daemons parking themselves in memory practically forever. >> >> So one thing it needs to do is limit the number of daemons that are allowed >> to exist concurrently. >> I think the 'kill' logic should be more and more aggressive based on # of >> daemons already in use. >> And it would be nice if there was some API controlled way to set a hard >> limit on the # of daemons allowed >> to exist at the same time. >> >> The 'daemon manager' should be able to manage all Gradle daemons, across >> gradle versions >> (a common thing in my use has been multiple daemons get started because of >> different gradle >> versions). Sometimes one of those daemons is 'by accident' because gradle >> version set wrong... >> and only used once. But it sticks around for what seems forever. >> >> Also it has been a problem that daemons would get 'stuck' or something. And >> then sit there >> 'defunct' while another daemon got started in its place. This way, I've seen >> 10+ concurrent daemons on my machine. >> So daemons should have a robust 'self destruct'. If something is wrong, or >> if they haven't been used in some time, they should just commit suicide. And >> this mechanism should be robust and not itself fail or get stuck somehow. >> >> Another useful thing would be that I could somehow 'own my daemon' and kill >> it when I know I am not >> going to be using it again (e.g. if the IDE is shutdown, it's 'associated' >> daemon probably should also >> go away as it probably won't be used anymore). >> >> > > Another thing to consider: as you say, indeed the JVM never gives >> > > memory back to the OS. The only way to really give it back is to >> > > terminate the JVM (i.e. daemon). So I'd suggest not having the >> > > daemon stick around for too long, unless it is really being used >> > > quite frequently… >> > >> > At the moment, the default idle time is 3 hours. This will be user >> > configurable though. >> >> I exaggerate, but 3 minutes would be better :-) >> I see we are somewhat far apart on the meaning of 'frequently'. >> >> > >> > > My compromise would be a combination of two strategies: >> > > - be generous with max memory limits (so things work without much >> > > manual tweaking) >> > > - be conservative and 'kill' unused daemons of fairly quickly (so >> > > unused resources are returned to OS) >> > >> > This sounds right to me, now we just need to agree on what the idle >> > timeout should be :) >> >> Tricky :-). My feeling is... something that can be measured in minutes, not >> hours :-) >> >> Maybe 10 minutes? Maybe 5 minutes, maybe 15 minutes (that's long!)... >> 3 hours in my world... is practically equivalent to 'forever'. You might as >> well have no timeout. >> >> Just put in to perspective: how much do you gain from not having to pay the >> startup >> cost once every 3 hours? Is that worth permanently sacrificing +1Gb of >> memory? >> >> If it is tweakable somehow via the tooling API it is less important what the >> exact value would be, because I can tweak it from the IDE if I don't like >> its default setting. >> >> > >> > If we are too conservative we almost completely negate the user >> > experience benefit of using the daemon. >> > >> >> The same is true if you keep the daemon there for too long. Because it >> degrades overall performance (memory given to the daemon does hurt >> performance because it isn't available for other processes and OS level >> caches). >> >> I'm more worried about overall performance degradation than speeding up the >> occasional gradle command. >> >> This is especially the case *if* you reach the 'swap tipping point', where >> it makes the difference between a usable or unusable system. Paying the >> startup cost for Gradle degrades the experience, but doesn't really make my >> machine unusable. >> >> This problem is worse for me, because as a Tooling API user, I do not have >> the option of not using the daemon. A commandline user always has this >> option if the daemon doesn't work for them. >> >> > > I believe, it just isn't worth the price to keep a daemon parked in >> > > precious memory for extended periods of time, if it is not being >> > > used. Rather than counting on paging it out, its much better to >> > > shut it down. If it is paged out, it won't give you great >> > > performance next time you try to use it anyway. >> > >> > It would be interesting to test if paging in is cheaper than >> > bootstrapping Groovy or not. I'm not suggesting we do this. >> >> Testing a bit to find out doesn't hurt :-) >> >> Also keep in mind, I don't think the JVM is ever really 100% 'idle'. So even >> when not doing anything, it is probably still touching some of the memory in >> its heap every so often. So try to factor in a form of 'constant drag' on >> the system just for having a parked daemon. >> >> > >> > Shutting it down is costly though in terms of user experience so >> > that's not without consequence. This is a fine balancing act. >> >> Keeping it there may be more costly in certain cases (if it causes paging). >> Also... shutting it down, in itself doesn't really hurt user experience. >> >> It is the startup that hurts, not the shutdown. >> >> So the shutdown only hurts assuming that I'm actually going to be using >> the same daemon again, soon. >> >> > No problem, no offence taken. I just had to clarify what I was >> > saying. Your input is very valuable and I think we are increasing >> > the understanding here. Although, personally I don't think I have a >> > good idea on these defaults yet. >> >> That's good. Glad to hear that. I think it is good to keep an open mind >> about this. And >> also it should point to the importance of allowing the user to override >> the defaults. You do your best to set sensible defaults, but accept that >> they may not be right for every use case. >> >> I'm speaking as a Tooling API client here... and for me, there currently >> is no way to tweak any of these defaults. >> >> Thanks for the interesting discussion :-) >> >> Kris >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> >> >> >> >> -- >> Szczepan Faber >> Principal engineer@gradleware >> Lead@mockito > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
