Thanks Adam for your insight. I agree that option #2 would probably be a really good start. Unfortunately, for someone who is pretty new to the code base, I would rather start with #1 by exposing n on the compile task, set it to one by default and have the logic there to fork compiler tag. I would stress that it's a temporary fix and it will be stream line later with #2 and #3. It would also be much easier for me to contribute #1 in a reasonable time, say for 2.1, and take more time to implement a full solution. How does that sound?
If you strongly prefer #2 as a start point, could you point me to a couple place where changes should be applied so I start looking into it? On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <adam.murd...@gradleware.com> wrote: > > On 30 Jun 2014, at 1:51 am, Daniel Lacasse <daniel.lacass...@gmail.com> > wrote: > > I have been using the native extension for Gradle inside an actual project > for a couple months. The biggest limitation I'm seeing is the inflexibility > of the compile task when compared to other build tool in the industry. The > --parallel switch works great for concurrent execution of tasks. When it > comes to the compile task, all files are compiled one after the other. The > native compiler are quite slow especially when it comes to compiling C++ > templates. As a comparison, my dev box is mostly idle while compiling with > Gradle as oppose to a fairly important load when compiling Visual Studio. > > The main use case of this feature is the speed up of the compilation > process for the native extension. To highlight how this feature is > important, I will point out that some company where I previously worked at > use system like Incredibuild to perform parallel distributed compilation. > Even with such system, the compilation was still pretty time consuming. In > it's present form, Gradle is not suitable in term of speed for those > scenario. I talked to a couple Gradleware engineers during Gradle Summit > 2014 and some insane features are planed to address this problem. > Unfortunately, a quicker solution is needed in order to speed up the > adoption of Gradle as a native build tool. I also want this feature to be > in accordance to the long term Gradle road map. > > I would like to start the discussion for contributing this into Gradle. > > > That would be great. > > From my limiting knowledge of Gradle here are a couple open issues I have. > I hope some brilliant minds from Gradleware can share there wisdom on where > to move forward with this feature. > > Open issues > - What is the current road map for such feature. > > > I think there are 3 potential steps we could take: > > 1. The compilation tasks do something specific, where they fork n > concurrent compilations, and n is just some setting on the compilation > tasks. > > 2. Then, we introduce some general service that tasks can use for > coordinating concurrent work. This would be integrated with > —parallel-threads. The compilation tasks, the test tasks, and the task > executor would all use this service to ensure that an appropriate amount of > concurrent work happens at any given time. This would be a public service > that any task implementation could use. > > 3. Then later, we add more capabilities to this so that the work can > treated more like tasks - with their own up-to-date checks, dependencies, > and so on. > > We could start with #1 and later extract #2, or we could jump straight to > #2. In some ways, it might be nice to start with #2. > > > - This could also be used in any language which require compilation > such as Java. > - Allowing custom implementation of this feature could allow a company > to plug Gradle in there current Incredibuild infrastructure or any other > distributed framework. > - How this feature fits with the --parallel flag? > > > For every build, there should be two settings that you can tweak: > > 1. The maximum amount of parallel work that can be performed by Gradle. > 2. Whether or not tasks should be executed in parallel. > > That is, Gradle will be able to do stuff in parallel even if the tasks > aren’t executed concurrently. It already does this with test execution. > > I would change —parallel-threads to control the maximum work but to not > enable parallel task execution. It would default to some ‘reasonable’ value > - the number of cores, say, at least to start with. > > > - How the number of parallel compilation unit will be configure aka > number of files that can be compiled in parallel? > > > As above. > > - Should this feature be always on by default or have a toggle flag? > > > Always on, I think. Why would you turn it off? > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > CTO Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > > > -- Daniel
