On 30 Jun 2014, at 12:21 pm, Daniel Lacasse <daniel.lacass...@gmail.com> wrote:
> Thanks Adam for your insight. I agree that option #2 would probably be a > really good start. Unfortunately, for someone who is pretty new to the code > base, I would rather start with #1 by exposing n on the compile task, set it > to one by default and have the logic there to fork compiler tag. I would > stress that it's a temporary fix and it will be stream line later with #2 and > #3. It would also be much easier for me to contribute #1 in a reasonable > time, say for 2.1, and take more time to implement a full solution. How does > that sound? Sounds good to me. > > If you strongly prefer #2 as a start point, could you point me to a couple > place where changes should be applied so I start looking into it? > > > On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <adam.murd...@gradleware.com> > wrote: > > On 30 Jun 2014, at 1:51 am, Daniel Lacasse <daniel.lacass...@gmail.com> wrote: > >> I have been using the native extension for Gradle inside an actual project >> for a couple months. The biggest limitation I'm seeing is the inflexibility >> of the compile task when compared to other build tool in the industry. The >> --parallel switch works great for concurrent execution of tasks. When it >> comes to the compile task, all files are compiled one after the other. The >> native compiler are quite slow especially when it comes to compiling C++ >> templates. As a comparison, my dev box is mostly idle while compiling with >> Gradle as oppose to a fairly important load when compiling Visual Studio. >> >> The main use case of this feature is the speed up of the compilation process >> for the native extension. To highlight how this feature is important, I will >> point out that some company where I previously worked at use system like >> Incredibuild to perform parallel distributed compilation. Even with such >> system, the compilation was still pretty time consuming. In it's present >> form, Gradle is not suitable in term of speed for those scenario. I talked >> to a couple Gradleware engineers during Gradle Summit 2014 and some insane >> features are planed to address this problem. Unfortunately, a quicker >> solution is needed in order to speed up the adoption of Gradle as a native >> build tool. I also want this feature to be in accordance to the long term >> Gradle road map. >> >> I would like to start the discussion for contributing this into Gradle. > > That would be great. > >> From my limiting knowledge of Gradle here are a couple open issues I have. I >> hope some brilliant minds from Gradleware can share there wisdom on where to >> move forward with this feature. >> >> Open issues >> - What is the current road map for such feature. > > I think there are 3 potential steps we could take: > > 1. The compilation tasks do something specific, where they fork n concurrent > compilations, and n is just some setting on the compilation tasks. > > 2. Then, we introduce some general service that tasks can use for > coordinating concurrent work. This would be integrated with > —parallel-threads. The compilation tasks, the test tasks, and the task > executor would all use this service to ensure that an appropriate amount of > concurrent work happens at any given time. This would be a public service > that any task implementation could use. > > 3. Then later, we add more capabilities to this so that the work can treated > more like tasks - with their own up-to-date checks, dependencies, and so on. > > We could start with #1 and later extract #2, or we could jump straight to #2. > In some ways, it might be nice to start with #2. > > >> - This could also be used in any language which require compilation such >> as Java. >> - Allowing custom implementation of this feature could allow a company to >> plug Gradle in there current Incredibuild infrastructure or any other >> distributed framework. >> - How this feature fits with the --parallel flag? > > For every build, there should be two settings that you can tweak: > > 1. The maximum amount of parallel work that can be performed by Gradle. > 2. Whether or not tasks should be executed in parallel. > > That is, Gradle will be able to do stuff in parallel even if the tasks aren’t > executed concurrently. It already does this with test execution. > > I would change —parallel-threads to control the maximum work but to not > enable parallel task execution. It would default to some ‘reasonable’ value - > the number of cores, say, at least to start with. > > >> - How the number of parallel compilation unit will be configure aka number >> of files that can be compiled in parallel? > > As above. > >> - Should this feature be always on by default or have a toggle flag? >> > > > Always on, I think. Why would you turn it off? > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > CTO Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > > > > > > -- > Daniel -- Adam Murdoch Gradle Co-founder http://www.gradle.org CTO Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
