I did some research on how to implement option #1 and here is the
implementation:
- In
org.gradle.nativebinaries.language.c.tasks.AbstractNativeCompilerTask:
- Add option int maximumNumberOfCompilerFork = 1
- 1 is to prevent any side effect. It will mainly behave just like it
was previously.
- I'm open to suggestion on the chosen name.
- No @Input on the new option
- In org.gradle.nativebinaries.toolchain.internal.NativeCompileSpec:
- set/getMaximumNumberOfCompilerFork
- In
org.gradle.nativebinaries.language.internal.AbstractNativeCompilerSpec:
- set/getMaximumNumberOfCompilerFork implementation
- In
org.gradle.nativebinaries.toolchain.internal.{gcc|msvcpp}.NativeCompilerSpec:
- Add modification to method WorkResult execute(T spec)
- Use a ThreadPoolExecutor with maximumPoolSize set to
maximumNumberOfCompileFork and corePoolSize set to 1
- Submit anonymous Callable<WorkResult> class to the
ThreadPoolExecutor
- Use the get method on each Future returned by Submit to join all
the work together.
This cover pretty much the option #1 for parallel compilation in the native
extension. The 2 open issues would be:
- Is maximumNumberOfCompilerFork a good choice for the option name? If
not, what would be the appropriate name?
- How can we test this new feature? The only way I can think of right
now is to compile 2 file with maximumNumberOfCompileFork set to 1 and a
second time with the option set to 2 and compare the execution time. This
heavily depend on timing which will probably be non deterministic.
On Sun, Jun 29, 2014 at 11:02 PM, Adam Murdoch <adam.murd...@gradleware.com>
wrote:
>
> On 30 Jun 2014, at 12:21 pm, Daniel Lacasse <daniel.lacass...@gmail.com>
> wrote:
>
> Thanks Adam for your insight. I agree that option #2 would probably be a
> really good start. Unfortunately, for someone who is pretty new to the code
> base, I would rather start with #1 by exposing n on the compile task, set
> it to one by default and have the logic there to fork compiler tag. I would
> stress that it's a temporary fix and it will be stream line later with #2
> and #3. It would also be much easier for me to contribute #1 in a
> reasonable time, say for 2.1, and take more time to implement a full
> solution. How does that sound?
>
>
> Sounds good to me.
>
>
> If you strongly prefer #2 as a start point, could you point me to a couple
> place where changes should be applied so I start looking into it?
>
>
> On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <adam.murd...@gradleware.com
> > wrote:
>
>>
>> On 30 Jun 2014, at 1:51 am, Daniel Lacasse <daniel.lacass...@gmail.com>
>> wrote:
>>
>> I have been using the native extension for Gradle inside an actual
>> project for a couple months. The biggest limitation I'm seeing is the
>> inflexibility of the compile task when compared to other build tool in the
>> industry. The --parallel switch works great for concurrent execution of
>> tasks. When it comes to the compile task, all files are compiled one after
>> the other. The native compiler are quite slow especially when it comes to
>> compiling C++ templates. As a comparison, my dev box is mostly idle while
>> compiling with Gradle as oppose to a fairly important load when compiling
>> Visual Studio.
>>
>> The main use case of this feature is the speed up of the compilation
>> process for the native extension. To highlight how this feature is
>> important, I will point out that some company where I previously worked at
>> use system like Incredibuild to perform parallel distributed compilation.
>> Even with such system, the compilation was still pretty time consuming. In
>> it's present form, Gradle is not suitable in term of speed for those
>> scenario. I talked to a couple Gradleware engineers during Gradle Summit
>> 2014 and some insane features are planed to address this problem.
>> Unfortunately, a quicker solution is needed in order to speed up the
>> adoption of Gradle as a native build tool. I also want this feature to be
>> in accordance to the long term Gradle road map.
>>
>> I would like to start the discussion for contributing this into Gradle.
>>
>>
>> That would be great.
>>
>> From my limiting knowledge of Gradle here are a couple open issues I
>> have. I hope some brilliant minds from Gradleware can share there wisdom on
>> where to move forward with this feature.
>>
>> Open issues
>> - What is the current road map for such feature.
>>
>>
>> I think there are 3 potential steps we could take:
>>
>> 1. The compilation tasks do something specific, where they fork n
>> concurrent compilations, and n is just some setting on the compilation
>> tasks.
>>
>> 2. Then, we introduce some general service that tasks can use for
>> coordinating concurrent work. This would be integrated with
>> —parallel-threads. The compilation tasks, the test tasks, and the task
>> executor would all use this service to ensure that an appropriate amount of
>> concurrent work happens at any given time. This would be a public service
>> that any task implementation could use.
>>
>> 3. Then later, we add more capabilities to this so that the work can
>> treated more like tasks - with their own up-to-date checks, dependencies,
>> and so on.
>>
>> We could start with #1 and later extract #2, or we could jump straight to
>> #2. In some ways, it might be nice to start with #2.
>>
>>
>> - This could also be used in any language which require compilation
>> such as Java.
>> - Allowing custom implementation of this feature could allow a company
>> to plug Gradle in there current Incredibuild infrastructure or any other
>> distributed framework.
>> - How this feature fits with the --parallel flag?
>>
>>
>> For every build, there should be two settings that you can tweak:
>>
>> 1. The maximum amount of parallel work that can be performed by Gradle.
>> 2. Whether or not tasks should be executed in parallel.
>>
>> That is, Gradle will be able to do stuff in parallel even if the tasks
>> aren’t executed concurrently. It already does this with test execution.
>>
>> I would change —parallel-threads to control the maximum work but to not
>> enable parallel task execution. It would default to some ‘reasonable’ value
>> - the number of cores, say, at least to start with.
>>
>>
>> - How the number of parallel compilation unit will be configure aka
>> number of files that can be compiled in parallel?
>>
>>
>> As above.
>>
>> - Should this feature be always on by default or have a toggle flag?
>>
>>
>> Always on, I think. Why would you turn it off?
>>
>>
>> --
>> Adam Murdoch
>> Gradle Co-founder
>> http://www.gradle.org
>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>> http://www.gradleware.com
>>
>>
>>
>>
>
>
> --
> Daniel
>
>
>
> --
> Adam Murdoch
> Gradle Co-founder
> http://www.gradle.org
> CTO Gradleware Inc. - Gradle Training, Support, Consulting
> http://www.gradleware.com
>
>
>
>
--
Daniel
