I haven’t been following this thread closely so this may not be relevant, but I just wanted to point out that we have the start of a spec for process management. Might be some overlap.
https://github.com/gradle/gradle/blob/master/design-docs/process-management.md On 6 July 2014 at 6:50:15 am, Daz DeBoer (darrell.deb...@gradleware.com) wrote: I'm pretty sure we don't want to expose a raw ExecutorService. We are pretty careful about exposing Gradle APIs that we have control over, rather than APIs that we don't have control over. One of the benefits of providing a specific API for parallel external processes is that Gradle could automatically buffer the input/output, provide progress reporting, etc. I haven't put a lot of thought into what our Gradle-wide parallel mechanism will look like, but if we start with an AsyncExecAction (or similar) we can later change AsyncExecAction to wrap the new API. If you want to create a completely separate service for this (instead of extending ExecActionFactory) then that would be fine, too. Daz On Sat, Jul 5, 2014 at 1:20 PM, Daniel Lacasse <daniel.lacass...@gmail.com> wrote: I was thinking about this Daz. Would it be more flexible, extensible and powerful if we would just inject a singleton whenever someone request it of a ExecutorService that is configured according to the parallel-threads count? Creating an AsyncExecAction is more code to maintain and support as opposed to letting the developers choose to just wrap an normal ExecAction into a Callable and submitting it to the ExecutorService. The pro is we get closer to option #2 and #3 for a Gradle-wide parallel mechanism. The con is do we really want to expose the actual ExecutorService? I realise that we should explore this a bit more. What are your thoughts on this? --- Daniel On Thursday, July 3, 2014 11:09 PM, Daz DeBoer <darrell.deb...@gradleware.com> wrote: On Wed, Jul 2, 2014 at 7:36 PM, Daniel Lacasse <daniel.lacass...@gmail.com> wrote: Sounds like a better plan as it would add more building blocks to gradle. Correct me if I'm wrong. They would probably be a global ExecutorService to avoid recreating it at ever task and share the threads between everyone who use it. Yes. ExecActionFactory is currently implemented by DefaultFileOperations, which is created per-project. So if you just make the change there you could have an ExecutorService per project. We could further improve this if we like by adding something to BuildScopeServices that is shared by all project instances. For now, I'd keep the implementation of ExecActionFactory.createAsyncAction nice and simple, because Adam is sure to have some ideas on how this could be better structured, when he's back from holidays. The problem I see - which could be a temporary thing until option #3 is implemented - is parallel-threads will be the number of parallel task executed at the same time as the number of compiler fork. For someone using the parallel flags, it may be non-intuitive. It basically boils down to how do we share the parallel-threads number between parallel task execution and number of compiler fork? In my mind parallel-threads is the maximum number of threads that Gradle should be allowed to create for his parallel business. Yes, we'd be overloading the 'parallel-threads' setting, so that you'd have both N projects being built in parallel, as well as N files being compiled in parallel. But the end goal is to use a single setting to control all parallelism, and this is a step in that direction. Thanks again Daz and I will move in the direction you are suggesting. --- Daniel On Wednesday, July 2, 2014 09:34 AM, Daz DeBoer < darrell.deb...@gradleware.com> wrote: On Mon, Jun 30, 2014 at 7:24 PM, Daniel Lacasse <daniel.lacass...@gmail.com> wrote: I did some research on how to implement option #1 and here is the implementation: In org.gradle.nativebinaries.language.c.tasks.AbstractNativeCompilerTask: Add option int maximumNumberOfCompilerFork = 1 1 is to prevent any side effect. It will mainly behave just like it was previously. I'm open to suggestion on the chosen name. No @Input on the new option In org.gradle.nativebinaries.toolchain.internal.NativeCompileSpec: set/getMaximumNumberOfCompilerFork In org.gradle.nativebinaries.language.internal.AbstractNativeCompilerSpec: set/getMaximumNumberOfCompilerFork implementation In org.gradle.nativebinaries.toolchain.internal.{gcc|msvcpp}.NativeCompilerSpec: Add modification to method WorkResult execute(T spec) Use a ThreadPoolExecutor with maximumPoolSize set to maximumNumberOfCompileFork and corePoolSize set to 1 Submit anonymous Callable<WorkResult> class to the ThreadPoolExecutor Use the get method on each Future returned by Submit to join all the work together. This cover pretty much the option #1 for parallel compilation in the native extension. The 2 open issues would be: Is maximumNumberOfCompilerFork a good choice for the option name? If not, what would be the appropriate name? Let's avoid adding a configuration option for this at this stage. What we really want is to make this configurable at a much higher level, and auto-detect the optimal setting. For now, let's use the value of 'parallel-threads' for this setting. Even though this is a bit coarse grained, it takes us in the right direction. So instead of passing the value from Task->CompileSpec->NativeCompiler, we would access the value from StartParameter#getParallelThreadCount. I think the easiest way to do this would be to add a method to ExecActionFactory that provided an AsyncExecAction, which would be similar to ExecAction, but something like: public interface AsyncExecAction extends ExecSpec { Future<ExecResult> submit(); } You could then modify org.gradle.nativebinaries.toolchain.internal.CommandLineTool to use async actions, and to present some sort of submit/collect API. This would mean that the parallel functionality would be available everywhere that CommandLineTool is used, which means both VisualCpp and Gcc based tool chains. How does this sound? While it may seem a bit more involved, the design will be largely the same. You'll still need a ThreadPoolExecuter and code to submit/collect, but this design takes your suggested solution and moves most of the code out of the NativeCompiler implementations into the CommandLineTool and ExecActionFactory. The nice thing about this is that it makes this feature more generally available, and makes is simpler for us to have a global setting for parallelisation. (The funny thing is that this is looking more like Option #2!) So native compilation would happen in parallel whenever --parallel and/or --parallel-threads is used. As Adam suggested, we should really separate these 2 options so that --parallel is a separate option to enable parallel project execution, while parallel-threads is a general purpose setting that defaults to a reasonable value. (Currently StartParameter#getParallelThreadCount defaults to zero if neither --parallel or --parallel-threads is specified). How can we test this new feature? The only way I can think of right now is to compile 2 file with maximumNumberOfCompileFork set to 1 and a second time with the option set to 2 and compare the execution time. This heavily depend on timing which will probably be non deterministic. No, timing-sensitive tests are not the way to go. We could certainly unit test the AsyncExecAction stuff, and also integration test with a few 'sleep 1 second' actions and ensure that they don't take much more than 1 second in total. I guess we could integration test parallel native compilation by using a 'sleep exe' in place of the actual compiler... I'll think about this a little more. Thanks for pushing to get this done. You're right that we really need parallel compilation to be a serious C++ build tool. Daz On Sun, Jun 29, 2014 at 11:02 PM, Adam Murdoch <adam.murd...@gradleware.com> wrote: On 30 Jun 2014, at 12:21 pm, Daniel Lacasse < daniel.lacass...@gmail.com> wrote: Thanks Adam for your insight. I agree that option #2 would probably be a really good start. Unfortunately, for someone who is pretty new to the code base, I would rather start with #1 by exposing n on the compile task, set it to one by default and have the logic there to fork compiler tag. I would stress that it's a temporary fix and it will be stream line later with #2 and #3. It would also be much easier for me to contribute #1 in a reasonable time, say for 2.1, and take more time to implement a full solution. How does that sound? Sounds good to me. If you strongly prefer #2 as a start point, could you point me to a couple place where changes should be applied so I start looking into it? On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <adam.murd...@gradleware.com> wrote: On 30 Jun 2014, at 1:51 am, Daniel Lacasse < daniel.lacass...@gmail.com> wrote: I have been using the native extension for Gradle inside an actual project for a couple months. The biggest limitation I'm seeing is the inflexibility of the compile task when compared to other build tool in the industry. The --parallel switch works great for concurrent execution of tasks. When it comes to the compile task, all files are compiled one after the other. The native compiler are quite slow especially when it comes to compiling C++ templates. As a comparison, my dev box is mostly idle while compiling with Gradle as oppose to a fairly important load when compiling Visual Studio. The main use case of this feature is the speed up of the compilation process for the native extension. To highlight how this feature is important, I will point out that some company where I previously worked at use system like Incredibuild to perform parallel distributed compilation. Even with such system, the compilation was still pretty time consuming. In it's present form, Gradle is not suitable in term of speed for those scenario. I talked to a couple Gradleware engineers during Gradle Summit 2014 and some insane features are planed to address this problem. Unfortunately, a quicker solution is needed in order to speed up the adoption of Gradle as a native build tool. I also want this feature to be in accordance to the long term Gradle road map. I would like to start the discussion for contributing this into Gradle. That would be great. From my limiting knowledge of Gradle here are a couple open issues I have. I hope some brilliant minds from Gradleware can share there wisdom on where to move forward with this feature. Open issues - What is the current road map for such feature. I think there are 3 potential steps we could take: 1. The compilation tasks do something specific, where they fork n concurrent compilations, and n is just some setting on the compilation tasks. 2. Then, we introduce some general service that tasks can use for coordinating concurrent work. This would be integrated with —parallel-threads. The compilation tasks, the test tasks, and the task executor would all use this service to ensure that an appropriate amount of concurrent work happens at any given time. This would be a public service that any task implementation could use. 3. Then later, we add more capabilities to this so that the work can treated more like tasks - with their own up-to-date checks, dependencies, and so on. We could start with #1 and later extract #2, or we could jump straight to #2. In some ways, it might be nice to start with #2. - This could also be used in any language which require compilation such as Java. - Allowing custom implementation of this feature could allow a company to plug Gradle in there current Incredibuild infrastructure or any other distributed framework. - How this feature fits with the --parallel flag? For every build, there should be two settings that you can tweak: 1. The maximum amount of parallel work that can be performed by Gradle. 2. Whether or not tasks should be executed in parallel. That is, Gradle will be able to do stuff in parallel even if the tasks aren’t executed concurrently. It already does this with test execution. I would change —parallel-threads to control the maximum work but to not enable parallel task execution. It would default to some ‘reasonable’ value - the number of cores, say, at least to start with. - How the number of parallel compilation unit will be configure aka number of files that can be compiled in parallel? As above. - Should this feature be always on by default or have a toggle flag? Always on, I think. Why would you turn it off? -- Adam Murdoch Gradle Co-founder http://www.gradle.org CTO Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com -- Daniel -- Adam Murdoch Gradle Co-founder http://www.gradle.org CTO Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com -- Daniel -- Darrell (Daz) DeBoer Principal Software Engineer, Gradleware Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com -- Darrell (Daz) DeBoer Principal Software Engineer, Gradleware Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com -- Darrell (Daz) DeBoer Principal Software Engineer, Gradleware Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA: http://www.gradlesummit.com — Luke Daley http://www.gradleware.com
