On Wed, Jul 2, 2014 at 7:36 PM, Daniel Lacasse <daniel.lacass...@gmail.com>
wrote:
> Sounds like a better plan as it would add more building blocks to gradle.
>
> Correct me if I'm wrong. They would probably be a global ExecutorService
> to avoid recreating it at ever task and share the threads between everyone
> who use it.
Yes. ExecActionFactory is currently implemented by DefaultFileOperations,
which is created per-project. So if you just make the change there you
could have an ExecutorService per project. We could further improve this if
we like by adding something to BuildScopeServices that is shared by all
project instances.
For now, I'd keep the implementation of ExecActionFactory.createAsyncAction
nice and simple, because Adam is sure to have some ideas on how this could
be better structured, when he's back from holidays.
The problem I see - which could be a temporary thing until option #3 is
> implemented - is parallel-threads will be the number of parallel task
> executed at the same time as the number of compiler fork. For someone using
> the parallel flags, it may be non-intuitive. It basically boils down to how
> do we share the parallel-threads number between parallel task execution and
> number of compiler fork? In my mind parallel-threads is the maximum number
> of threads that Gradle should be allowed to create for his parallel
> business.
>
Yes, we'd be overloading the 'parallel-threads' setting, so that you'd have
both N projects being built in parallel, as well as N files being compiled
in parallel. But the end goal is to use a single setting to control all
parallelism, and this is a step in that direction.
> Thanks again Daz and I will move in the direction you are suggesting.
>
> ---
> Daniel
> ------------------------------
> On Wednesday, July 2, 2014 09:34 AM, Daz DeBoer <
> darrell.deb...@gradleware.com> wrote:
>
> On Mon, Jun 30, 2014 at 7:24 PM, Daniel Lacasse <
> daniel.lacass...@gmail.com> wrote:
>
>> I did some research on how to implement option #1 and here is the
>> implementation:
>>
>> - In
>> org.gradle.nativebinaries.language.c.tasks.AbstractNativeCompilerTask:
>> - Add option int maximumNumberOfCompilerFork = 1
>> - 1 is to prevent any side effect. It will mainly behave just like
>> it was previously.
>> - I'm open to suggestion on the chosen name.
>> - No @Input on the new option
>> - In org.gradle.nativebinaries.toolchain.internal.NativeCompileSpec:
>> - set/getMaximumNumberOfCompilerFork
>> - In
>> org.gradle.nativebinaries.language.internal.AbstractNativeCompilerSpec:
>> - set/getMaximumNumberOfCompilerFork implementation
>> - In
>>
>> org.gradle.nativebinaries.toolchain.internal.{gcc|msvcpp}.NativeCompilerSpec:
>> - Add modification to method WorkResult execute(T spec)
>> - Use a ThreadPoolExecutor with maximumPoolSize set to
>> maximumNumberOfCompileFork and corePoolSize set to 1
>> - Submit anonymous Callable<WorkResult> class to the
>> ThreadPoolExecutor
>> - Use the get method on each Future returned by Submit to join all
>> the work together.
>>
>> This cover pretty much the option #1 for parallel compilation in the
>> native extension. The 2 open issues would be:
>>
>> - Is maximumNumberOfCompilerFork a good choice for the option name?
>> If not, what would be the appropriate name?
>>
>> Let's avoid adding a configuration option for this at this stage. What
> we really want is to make this configurable at a much higher level, and
> auto-detect the optimal setting.
>
> For now, let's use the value of 'parallel-threads' for this setting.
> Even though this is a bit coarse grained, it takes us in the right
> direction. So instead of passing the value from
> Task->CompileSpec->NativeCompiler, we would access the value from
> StartParameter#getParallelThreadCount.
>
> I think the easiest way to do this would be to add a method to
> ExecActionFactory that provided an AsyncExecAction, which would be similar
> to ExecAction, but something like:
>
> public interface AsyncExecAction extends ExecSpec {
> Future<ExecResult> submit();
> }
>
> You could then modify
> org.gradle.nativebinaries.toolchain.internal.CommandLineTool to use async
> actions, and to present some sort of submit/collect API. This would mean
> that the parallel functionality would be available everywhere that
> CommandLineTool is used, which means both VisualCpp and Gcc based tool
> chains.
>
> How does this sound? While it may seem a bit more involved, the design
> will be largely the same. You'll still need a ThreadPoolExecuter and code
> to submit/collect, but this design takes your suggested solution and moves
> most of the code out of the NativeCompiler implementations into the
> CommandLineTool and ExecActionFactory. The nice thing about this is that
> it makes this feature more generally available, and makes is simpler for us
> to have a global setting for parallelisation. (The funny thing is that this
> is looking more like Option #2!)
>
> *So native compilation would happen in parallel whenever --parallel
> and/or --parallel-threads is used*. As Adam suggested, we should really
> separate these 2 options so that --parallel is a separate option to enable
> parallel project execution, while parallel-threads is a general purpose
> setting that defaults to a reasonable value.
> (Currently StartParameter#getParallelThreadCount defaults to zero if
> neither --parallel or --parallel-threads is specified).
>
>>
>> - How can we test this new feature? The only way I can think of right
>> now is to compile 2 file with maximumNumberOfCompileFork set to 1 and a
>> second time with the option set to 2 and compare the execution time. This
>> heavily depend on timing which will probably be non deterministic.
>>
>> No, timing-sensitive tests are not the way to go. We could certainly
> unit test the AsyncExecAction stuff, and also integration test with a few
> 'sleep 1 second' actions and ensure that they don't take much more than 1
> second in total. I guess we could integration test parallel native
> compilation by using a 'sleep exe' in place of the actual compiler... I'll
> think about this a little more.
>
> Thanks for pushing to get this done. You're right that we really need
> parallel compilation to be a serious C++ build tool.
>
> Daz
>
>>
>>
>>
>>
>> On Sun, Jun 29, 2014 at 11:02 PM, Adam Murdoch <
>> adam.murd...@gradleware.com> wrote:
>>
>>>
>>> On 30 Jun 2014, at 12:21 pm, Daniel Lacasse <
>>> daniel.lacass...@gmail.com> wrote:
>>>
>>> Thanks Adam for your insight. I agree that option #2 would probably be
>>> a really good start. Unfortunately, for someone who is pretty new to the
>>> code base, I would rather start with #1 by exposing n on the compile task,
>>> set it to one by default and have the logic there to fork compiler tag. I
>>> would stress that it's a temporary fix and it will be stream line later
>>> with #2 and #3. It would also be much easier for me to contribute #1 in a
>>> reasonable time, say for 2.1, and take more time to implement a full
>>> solution. How does that sound?
>>>
>>>
>>> Sounds good to me.
>>>
>>>
>>> If you strongly prefer #2 as a start point, could you point me to a
>>> couple place where changes should be applied so I start looking into it?
>>>
>>>
>>> On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <
>>> adam.murd...@gradleware.com> wrote:
>>>
>>>>
>>>> On 30 Jun 2014, at 1:51 am, Daniel Lacasse <
>>>> daniel.lacass...@gmail.com> wrote:
>>>>
>>>> I have been using the native extension for Gradle inside an actual
>>>> project for a couple months. The biggest limitation I'm seeing is the
>>>> inflexibility of the compile task when compared to other build tool in the
>>>> industry. The --parallel switch works great for concurrent execution of
>>>> tasks. When it comes to the compile task, all files are compiled one after
>>>> the other. The native compiler are quite slow especially when it comes to
>>>> compiling C++ templates. As a comparison, my dev box is mostly idle while
>>>> compiling with Gradle as oppose to a fairly important load when compiling
>>>> Visual Studio.
>>>>
>>>> The main use case of this feature is the speed up of the compilation
>>>> process for the native extension. To highlight how this feature is
>>>> important, I will point out that some company where I previously worked at
>>>> use system like Incredibuild to perform parallel distributed compilation.
>>>> Even with such system, the compilation was still pretty time consuming. In
>>>> it's present form, Gradle is not suitable in term of speed for those
>>>> scenario. I talked to a couple Gradleware engineers during Gradle Summit
>>>> 2014 and some insane features are planed to address this problem.
>>>> Unfortunately, a quicker solution is needed in order to speed up the
>>>> adoption of Gradle as a native build tool. I also want this feature to be
>>>> in accordance to the long term Gradle road map.
>>>>
>>>> I would like to start the discussion for contributing this into
>>>> Gradle.
>>>>
>>>>
>>>> That would be great.
>>>>
>>>> From my limiting knowledge of Gradle here are a couple open issues I
>>>> have. I hope some brilliant minds from Gradleware can share there wisdom on
>>>> where to move forward with this feature.
>>>>
>>>> Open issues
>>>> - What is the current road map for such feature.
>>>>
>>>>
>>>> I think there are 3 potential steps we could take:
>>>>
>>>> 1. The compilation tasks do something specific, where they fork n
>>>> concurrent compilations, and n is just some setting on the compilation
>>>> tasks.
>>>>
>>>> 2. Then, we introduce some general service that tasks can use for
>>>> coordinating concurrent work. This would be integrated with
>>>> —parallel-threads. The compilation tasks, the test tasks, and the task
>>>> executor would all use this service to ensure that an appropriate amount of
>>>> concurrent work happens at any given time. This would be a public service
>>>> that any task implementation could use.
>>>>
>>>> 3. Then later, we add more capabilities to this so that the work can
>>>> treated more like tasks - with their own up-to-date checks, dependencies,
>>>> and so on.
>>>>
>>>> We could start with #1 and later extract #2, or we could jump
>>>> straight to #2. In some ways, it might be nice to start with #2.
>>>>
>>>>
>>>> - This could also be used in any language which require
>>>> compilation such as Java.
>>>> - Allowing custom implementation of this feature could allow a
>>>> company to plug Gradle in there current Incredibuild infrastructure or any
>>>> other distributed framework.
>>>> - How this feature fits with the --parallel flag?
>>>>
>>>>
>>>> For every build, there should be two settings that you can tweak:
>>>>
>>>> 1. The maximum amount of parallel work that can be performed by
>>>> Gradle.
>>>> 2. Whether or not tasks should be executed in parallel.
>>>>
>>>> That is, Gradle will be able to do stuff in parallel even if the
>>>> tasks aren’t executed concurrently. It already does this with test
>>>> execution.
>>>>
>>>> I would change —parallel-threads to control the maximum work but to
>>>> not enable parallel task execution. It would default to some ‘reasonable’
>>>> value - the number of cores, say, at least to start with.
>>>>
>>>>
>>>> - How the number of parallel compilation unit will be configure aka
>>>> number of files that can be compiled in parallel?
>>>>
>>>>
>>>> As above.
>>>>
>>>> - Should this feature be always on by default or have a toggle
>>>> flag?
>>>>
>>>>
>>>> Always on, I think. Why would you turn it off?
>>>>
>>>>
>>>> --
>>>> Adam Murdoch
>>>> Gradle Co-founder
>>>> http://www.gradle.org
>>>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>>>> http://www.gradleware.com
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Daniel
>>>
>>>
>>>
>>> --
>>> Adam Murdoch
>>> Gradle Co-founder
>>> http://www.gradle.org
>>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>>> http://www.gradleware.com
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Daniel
>>
>
>
>
> --
> Darrell (Daz) DeBoer
> Principal Software Engineer, *Gradleware <http://gradleware.com>*
>
> Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA:
> http://www.gradlesummit.com
>
>
--
Darrell (Daz) DeBoer
Principal Software Engineer, *Gradleware <http://gradleware.com>*
Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA:
http://www.gradlesummit.com
