On Mon, Jun 30, 2014 at 7:24 PM, Daniel Lacasse <daniel.lacass...@gmail.com>
wrote:
> I did some research on how to implement option #1 and here is the
> implementation:
>
> - In
> org.gradle.nativebinaries.language.c.tasks.AbstractNativeCompilerTask:
> - Add option int maximumNumberOfCompilerFork = 1
> - 1 is to prevent any side effect. It will mainly behave just like
> it was previously.
> - I'm open to suggestion on the chosen name.
> - No @Input on the new option
> - In org.gradle.nativebinaries.toolchain.internal.NativeCompileSpec:
> - set/getMaximumNumberOfCompilerFork
> - In
> org.gradle.nativebinaries.language.internal.AbstractNativeCompilerSpec:
> - set/getMaximumNumberOfCompilerFork implementation
> - In
>
> org.gradle.nativebinaries.toolchain.internal.{gcc|msvcpp}.NativeCompilerSpec:
> - Add modification to method WorkResult execute(T spec)
> - Use a ThreadPoolExecutor with maximumPoolSize set to
> maximumNumberOfCompileFork and corePoolSize set to 1
> - Submit anonymous Callable<WorkResult> class to the
> ThreadPoolExecutor
> - Use the get method on each Future returned by Submit to join all
> the work together.
>
> This cover pretty much the option #1 for parallel compilation in the
> native extension. The 2 open issues would be:
>
> - Is maximumNumberOfCompilerFork a good choice for the option name? If
> not, what would be the appropriate name?
>
> Let's avoid adding a configuration option for this at this stage. What we
really want is to make this configurable at a much higher level, and
auto-detect the optimal setting.
For now, let's use the value of 'parallel-threads' for this setting. Even
though this is a bit coarse grained, it takes us in the right direction. So
instead of passing the value from Task->CompileSpec->NativeCompiler, we
would access the value from StartParameter#getParallelThreadCount.
I think the easiest way to do this would be to add a method to
ExecActionFactory that provided an AsyncExecAction, which would be similar
to ExecAction, but something like:
public interface AsyncExecAction extends ExecSpec {
Future<ExecResult> submit();
}
You could then modify
org.gradle.nativebinaries.toolchain.internal.CommandLineTool to use async
actions, and to present some sort of submit/collect API. This would mean
that the parallel functionality would be available everywhere that
CommandLineTool is used, which means both VisualCpp and Gcc based tool
chains.
How does this sound? While it may seem a bit more involved, the design will
be largely the same. You'll still need a ThreadPoolExecuter and code to
submit/collect, but this design takes your suggested solution and moves
most of the code out of the NativeCompiler implementations into the
CommandLineTool and ExecActionFactory. The nice thing about this is that
it makes this feature more generally available, and makes is simpler for us
to have a global setting for parallelisation. (The funny thing is that this
is looking more like Option #2!)
*So native compilation would happen in parallel whenever --parallel and/or
--parallel-threads is used*. As Adam suggested, we should really separate
these 2 options so that --parallel is a separate option to enable parallel
project execution, while parallel-threads is a general purpose setting that
defaults to a reasonable value.
(Currently StartParameter#getParallelThreadCount defaults to zero if
neither --parallel or --parallel-threads is specified).
>
> - How can we test this new feature? The only way I can think of right
> now is to compile 2 file with maximumNumberOfCompileFork set to 1 and a
> second time with the option set to 2 and compare the execution time. This
> heavily depend on timing which will probably be non deterministic.
>
> No, timing-sensitive tests are not the way to go. We could certainly unit
test the AsyncExecAction stuff, and also integration test with a few 'sleep
1 second' actions and ensure that they don't take much more than 1 second
in total. I guess we could integration test parallel native compilation by
using a 'sleep exe' in place of the actual compiler... I'll think about
this a little more.
Thanks for pushing to get this done. You're right that we really need
parallel compilation to be a serious C++ build tool.
Daz
>
>
>
>
> On Sun, Jun 29, 2014 at 11:02 PM, Adam Murdoch <
> adam.murd...@gradleware.com> wrote:
>
>>
>> On 30 Jun 2014, at 12:21 pm, Daniel Lacasse <daniel.lacass...@gmail.com>
>> wrote:
>>
>> Thanks Adam for your insight. I agree that option #2 would probably be a
>> really good start. Unfortunately, for someone who is pretty new to the code
>> base, I would rather start with #1 by exposing n on the compile task, set
>> it to one by default and have the logic there to fork compiler tag. I would
>> stress that it's a temporary fix and it will be stream line later with #2
>> and #3. It would also be much easier for me to contribute #1 in a
>> reasonable time, say for 2.1, and take more time to implement a full
>> solution. How does that sound?
>>
>>
>> Sounds good to me.
>>
>>
>> If you strongly prefer #2 as a start point, could you point me to a
>> couple place where changes should be applied so I start looking into it?
>>
>>
>> On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <
>> adam.murd...@gradleware.com> wrote:
>>
>>>
>>> On 30 Jun 2014, at 1:51 am, Daniel Lacasse <daniel.lacass...@gmail.com>
>>> wrote:
>>>
>>> I have been using the native extension for Gradle inside an actual
>>> project for a couple months. The biggest limitation I'm seeing is the
>>> inflexibility of the compile task when compared to other build tool in the
>>> industry. The --parallel switch works great for concurrent execution of
>>> tasks. When it comes to the compile task, all files are compiled one after
>>> the other. The native compiler are quite slow especially when it comes to
>>> compiling C++ templates. As a comparison, my dev box is mostly idle while
>>> compiling with Gradle as oppose to a fairly important load when compiling
>>> Visual Studio.
>>>
>>> The main use case of this feature is the speed up of the compilation
>>> process for the native extension. To highlight how this feature is
>>> important, I will point out that some company where I previously worked at
>>> use system like Incredibuild to perform parallel distributed compilation.
>>> Even with such system, the compilation was still pretty time consuming. In
>>> it's present form, Gradle is not suitable in term of speed for those
>>> scenario. I talked to a couple Gradleware engineers during Gradle Summit
>>> 2014 and some insane features are planed to address this problem.
>>> Unfortunately, a quicker solution is needed in order to speed up the
>>> adoption of Gradle as a native build tool. I also want this feature to be
>>> in accordance to the long term Gradle road map.
>>>
>>> I would like to start the discussion for contributing this into Gradle.
>>>
>>>
>>> That would be great.
>>>
>>> From my limiting knowledge of Gradle here are a couple open issues I
>>> have. I hope some brilliant minds from Gradleware can share there wisdom on
>>> where to move forward with this feature.
>>>
>>> Open issues
>>> - What is the current road map for such feature.
>>>
>>>
>>> I think there are 3 potential steps we could take:
>>>
>>> 1. The compilation tasks do something specific, where they fork n
>>> concurrent compilations, and n is just some setting on the compilation
>>> tasks.
>>>
>>> 2. Then, we introduce some general service that tasks can use for
>>> coordinating concurrent work. This would be integrated with
>>> —parallel-threads. The compilation tasks, the test tasks, and the task
>>> executor would all use this service to ensure that an appropriate amount of
>>> concurrent work happens at any given time. This would be a public service
>>> that any task implementation could use.
>>>
>>> 3. Then later, we add more capabilities to this so that the work can
>>> treated more like tasks - with their own up-to-date checks, dependencies,
>>> and so on.
>>>
>>> We could start with #1 and later extract #2, or we could jump straight
>>> to #2. In some ways, it might be nice to start with #2.
>>>
>>>
>>> - This could also be used in any language which require compilation
>>> such as Java.
>>> - Allowing custom implementation of this feature could allow a
>>> company to plug Gradle in there current Incredibuild infrastructure or any
>>> other distributed framework.
>>> - How this feature fits with the --parallel flag?
>>>
>>>
>>> For every build, there should be two settings that you can tweak:
>>>
>>> 1. The maximum amount of parallel work that can be performed by Gradle.
>>> 2. Whether or not tasks should be executed in parallel.
>>>
>>> That is, Gradle will be able to do stuff in parallel even if the tasks
>>> aren’t executed concurrently. It already does this with test execution.
>>>
>>> I would change —parallel-threads to control the maximum work but to not
>>> enable parallel task execution. It would default to some ‘reasonable’ value
>>> - the number of cores, say, at least to start with.
>>>
>>>
>>> - How the number of parallel compilation unit will be configure aka
>>> number of files that can be compiled in parallel?
>>>
>>>
>>> As above.
>>>
>>> - Should this feature be always on by default or have a toggle flag?
>>>
>>>
>>> Always on, I think. Why would you turn it off?
>>>
>>>
>>> --
>>> Adam Murdoch
>>> Gradle Co-founder
>>> http://www.gradle.org
>>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>>> http://www.gradleware.com
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Daniel
>>
>>
>>
>> --
>> Adam Murdoch
>> Gradle Co-founder
>> http://www.gradle.org
>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>> http://www.gradleware.com
>>
>>
>>
>>
>
>
> --
> Daniel
>
--
Darrell (Daz) DeBoer
Principal Software Engineer, *Gradleware <http://gradleware.com>*
Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA:
http://www.gradlesummit.com
