Re: [webkit-dev] Building WebKit's GTK+ back-end using distcc timing
Hi,
I think that this waning being displayed at the beginning of the
build, is related to the distcc-based build slowdown:
make[1]: warning: -jN forced in submake: disabling jobserver mode.
-Ilyes
On Thu, Mar 10, 2011 at 4:48 PM, Ilyes Gouta ilyes.go...@gmail.com wrote:
Hi,
additional reasons why it is taking so much time now, but I would point
to pre-processing and linking as possible causes for the distcc master
now doing more work.
I would rule out the linking phase as being the bottleneck: this is
happening at the end of the process. I've eventually timed it and it's
not the one to blame. From observation I can tell that {preprocessing,
compiling C++ code, other Makefile non-visible actions} is what's
taking more time to process. And the result is very drastic: the
distcc master (the client which initiated the build) is failing to
load properly the distcc slaves, which are very capable machines, with
code to build - these aren't properly exploited anymore.
Especially when you do not have enough memory and the system start swapping.
Have you checked that?
Well, distcc master itself isn't participating in the build, so no
compiler invocation and stuff... DISTCC_HOSTS contains only the remote
machines, not the localhost. No, I won't say that it's due to RAM
shortage and swapping. It's more like CPU bound than memory.
I thought that preprocessing and the # of source code files won't be
the reason, because even with the
pre-9161f3277b15aae8d15a929d5362e28cad7e01c0 (almost 1 year ago) code
base, preprocessing existed and the count of WebKit's source files is
already important. Then it must be the build system (configure.ac, the
.m4 macros and Makefiles) that has changed and caused this slowdown.
I'm still puzzled! :)
-Ilyes
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Building WebKit's GTK+ back-end using distcc timing
On Thu, Mar 10, 2011 at 11:55:15AM +0100, Ilyes Gouta wrote: Hi, [...] Could some one enlighten us about what have changed in the build system of the Gtk+ port of WebKit that would have an impact on the total build time? Could it be that configure.ac has changed so that the final generated GNUMakefile is less optimal and spends more time doing something else, when going through the CXX files to compile them? I think after that commit (if I checked it correctly it was 1 year ago) a lot things have changed and I do not think there is just one patch to blame, probably just the additions of new files to the compilation after that commit could be increasing the time. I've also noticed compilation is slower since 1 year ago, but I do not think it has been just one patch the reason. Anyway the main change lately in the compilation system has been this: http://trac.webkit.org/changeset/75362 But not sure if that could cause such a big impact in your distcc compilation, you can try to check it. I hope this helps. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Building WebKit's GTK+ back-end using distcc timing
Hi Alejandro, I don't think that the slowdown is due to the addition of new C++/code. Even building with most of the WebKit options turned off, actually takes that much of time. By just observing how the files are dispatched to the distcc servers, how these are loaded by the client machine (which initiate the build), almost could tell that this latter is doing more work (slower, more preprocessor work?) for dispatching source code, than a pre-9161f3277b15aae8d15a929d5362e28cad7e01c0 commit code base. In our setup, the client machine which initiates the distributed build, doesn't contribute to it, i.e doesn't compile code, but just serves the source code. Anyway the main change lately in the compilation system has been this: http://trac.webkit.org/changeset/75362 Thanks for the reference! I'll check it out. -Ilyes On Thu, Mar 10, 2011 at 12:46 PM, Alejandro Garcia Castro a...@igalia.com wrote: On Thu, Mar 10, 2011 at 11:55:15AM +0100, Ilyes Gouta wrote: Hi, [...] Could some one enlighten us about what have changed in the build system of the Gtk+ port of WebKit that would have an impact on the total build time? Could it be that configure.ac has changed so that the final generated GNUMakefile is less optimal and spends more time doing something else, when going through the CXX files to compile them? I think after that commit (if I checked it correctly it was 1 year ago) a lot things have changed and I do not think there is just one patch to blame, probably just the additions of new files to the compilation after that commit could be increasing the time. I've also noticed compilation is slower since 1 year ago, but I do not think it has been just one patch the reason. Anyway the main change lately in the compilation system has been this: http://trac.webkit.org/changeset/75362 But not sure if that could cause such a big impact in your distcc compilation, you can try to check it. I hope this helps. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Building WebKit's GTK+ back-end using distcc timing
Hi Ilyes, On Thu, 10 Mar 2011 13:51:48 +0100 Ilyes Gouta ilyes.go...@gmail.com wrote: I don't think that the slowdown is due to the addition of new C++/code. Even building with most of the WebKit options turned off, actually takes that much of time. By just observing how the files are dispatched to the distcc servers, how these are loaded by the client machine (which initiate the build), almost could tell that this latter is doing more work (slower, more preprocessor work?) for dispatching source code, than a pre-9161f3277b15aae8d15a929d5362e28cad7e01c0 commit code base. In our setup, the client machine which initiates the distributed build, doesn't contribute to it, i.e doesn't compile code, but just serves the source code. As far as I know, distcc master process pre-processes the code before sending it to the slaves. Having more (or bigger) files in the tree could be the cause of your master machine having to do more work before offloading the compilation to the slaves. Last, but not least, more code means more work when linking libraries and programs, and this step is also done by the distcc build master. There could me some other additional reasons why it is taking so much time now, but I would point to pre-processing and linking as possible causes for the distcc master now doing more work. Hope this helps, or provides some hint to help you troubleshoot your builds. Cheers, -- Adrian Perez ape...@igalia.com Igalia - Free Software Engineering signature.asc Description: PGP signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Building WebKit's GTK+ back-end using distcc timing
As far as I know, distcc master process pre-processes the code before sending it to the slaves. Having more (or bigger) files in the tree could be the cause of your master machine having to do more work before offloading the compilation to the slaves. Last, but not least, more code means more work when linking libraries and programs, and this step is also done by the distcc build master. There could me some other additional reasons why it is taking so much time now, but I would point to pre-processing and linking as possible causes for the distcc master now doing more work. Especially when you do not have enough memory and the system start swapping. Have you checked that? ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
