Re: [webkit-dev] [webkit meeting notes] build systems
2010/4/21 Kevin Ollivier kev...@theolliviers.com Hi Marc-Antoine, On Apr 18, 2010, at 10:47 AM, Marc-Antoine Ruel wrote: 2010/4/17 Kevin Ollivier kev...@theolliviers.com Hi Marc-Antoine, On Apr 17, 2010, at 11:26 AM, Marc-Antoine Ruel wrote: Like this? http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi - It is currently not JSON compliant. It's in fact a python file but that can be fixed: s/'//g and removing the extra commas *should* be sufficient. BTW, there is no way to make it JSON-compliant after all. Sorry for the misleading comment. mac, qt, gtk, wx, symbian and even some JSC files are listed. I'm 100% sure the lists aren't complete and I'll need to fix them along the way, especially JSC stuff which *is* a show stopper. .gypi files are header files. They can be imported by other .gypi or a .gyp file. Each .gyp file will generate a .sln, .xcodeproj, Makefile or SConstruct, depending on the selected build system. Each target inside a .gyp will generate a .vcproj or .scons. The regex logic lives in inside sources entries inside a target. You can have a list of file names, 'include' or 'exclude' with an array of regexp. Please take a look at the very bottom of this file for an example: http://trac.webkit.org/browser/trunk/WebCore/WebCore.gyp/WebCore.gyp May I ask why it is being done this way, that is, having one really large list of build files and using include / exclude regexs to trim the list? i.e. why is there not a WebCoreCommon.gypi, WebCoreWinCommon.gypi, WebCoreChromium.gypi, etc.? Are you guys using scripts to create WebCore.gypi? We do that for chromium, it already to support different platforms so I guess since there was already regexp done, these were simply reused. A good question, cc'ing people more knowledgeable than me. Anyway, the fact that this file is actually Python (I had forgotten the format was JSON-like rather than straight JSON) makes things even better, as we only really need to handle export now, and not parse some import file list. i.e. for WebKitTools/Scripts/update-sources-list.py, getWebCoreFilesDict() basically becomes a very small script that execfile's the gypi file, then we run whatever filtering mechanism on it (waf has a list of ports we could use to do the filtering that I could probably move into WebKitTools/Scripts, if we don't have a pre-made Chromium solution here), and then passes the final source list along to generateWebCoreSourcesGTKandQT to generate the sources / includes for those platforms, and actually this solution should work for Android.mk and possibly jam too, as their syntax looks largely similar. Then we'd add some XML parsing code to grab the node for common file groups and update them for the MSVC projects, and then I think that mostly leaves XCode, which I think would be pretty similar in nature. For MSVC and XCode, it makes more sense to use the native gyp support since it makes really clean projects. FYI, VS2010 is not supported for now. Yes, if all MSVC and XCode projects are planning to switch over anyway, it's probably not worth bothering with. Though if some choose not to, we could actually re-use the chunk of Gyp code that creates the file lists and just inject that into the project file. Whatever works for people and is implemented. -- M-A ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
On Wed, Apr 21, 2010 at 9:23 AM, Marc-Antoine Ruel mar...@chromium.org wrote: May I ask why it is being done this way, that is, having one really large list of build files and using include / exclude regexs to trim the list? i.e. why is there not a WebCoreCommon.gypi, WebCoreWinCommon.gypi, WebCoreChromium.gypi, etc.? Are you guys using scripts to create WebCore.gypi? We did this to optimize for writes (http://diveintomark.org/archives/2007/12/28/receipts). With one file, it's a no-brainer to add new files to the build. The maintainers of the ports can then use regexps and explicit lists to exclude builds as needed. It's worked pretty well for us. :DG ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
2010/4/17 Kevin Ollivier kev...@theolliviers.com Hi Marc-Antoine, On Apr 17, 2010, at 11:26 AM, Marc-Antoine Ruel wrote: 2010/4/17 Kevin Ollivier kev...@theolliviers.com Hi Maciej, On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote: On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now. If I had to do it myself, I would just skip any ports that don't use DerivedSources.make. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development. Converting, say, a JSON list of files to some XML-based output format would not be difficult at all (and I Like this? http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi - It is currently not JSON compliant. It's in fact a python file but that can be fixed: s/'//g and removing the extra commas *should* be sufficient. - It is currently chromium specific, that's what I meant by de-chromification of the gyp files. It's mainly adding more stuff and fixing the regexp if I'm not mistaken. I don't mind if it doesn't become the golden file, the goal is just to hopefully reduce the number of build system, nothing more. Yes, precisely why I mentioned JSON and later gyp, though I don't know if Chromium-specific is the right word here. It even has wx port files in it, which I don't think are built by Chromium. :) I suppose you somehow filter out which ones Chromium should build after the fact? If so, where does that logic reside? mac, qt, gtk, wx, symbian and even some JSC files are listed. I'm 100% sure the lists aren't complete and I'll need to fix them along the way, especially JSC stuff which *is* a show stopper. .gypi files are header files. They can be imported by other .gypi or a .gyp file. Each .gyp file will generate a .sln, .xcodeproj, Makefile or SConstruct, depending on the selected build system. Each target inside a .gyp will generate a .vcproj or .scons. The regex logic lives in inside sources entries inside a target. You can have a list of file names, 'include' or 'exclude' with an array of regexp. Please take a look at the very bottom of this file for an example: http://trac.webkit.org/browser/trunk/WebCore/WebCore.gyp/WebCore.gyp The main gyp caveat is some of its shortcut notation like /, ! and +++. Please see the documentationhttp://code.google.com/p/gyp/wiki/InputFormatReferencefor more info if interested. WebCore.gyp already brings a lot of conditions which can make the whole thing messy as the number of platforms increases but these can be moved off individual .gypi files to keep the whole thing bearable. In the end, it may even not make sense to share the gyp files accross platform if they don't share enough build logic, it's no big deal as long as people can add and remove files easily. BTW, I don't why WebCore.gypihttp://trac.webkit.org/browser/trunk/WebCore/WebCore.gypilists the idl, svg idl, source files, webinspector and images as individual lists since they all have
Re: [webkit-dev] [webkit meeting notes] build systems
Hi Maciej, On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote: On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now. If I had to do it myself, I would just skip any ports that don't use DerivedSources.make. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development. Converting, say, a JSON list of files to some XML-based output format would not be difficult at all (and I believe we can automatically convert the XCode project files from binary to plist and back, though IIRC there might be some UUID handling issues to consider), so we could patch the IDE files much like we patch the other ports. As for the other cases, such as additions / removals of things shared by multiple ports and the derived sources problem (that one probably would indeed need some 'template' to insert into project files), I think these could be added over time if we feel it'd bring a large benefit, but I think even just covering the common case of cross-platform source file maintenance is already a huge win for the project. I maintained Bakefile projects for years, and I'd say 80-90% of the time when a change broke my build, it was one of these common source file additions / removals. And it usually happened several times every week. I personally think the way to look at it is to start by solving the simple stuff that could be solved quickly, as in my experience that makes it far more likely to actually get done. If, say, gyp - Gtk / Qt / MSVC (/ XCode?) build file lists could be done in a weekend of hacking and some devs are interested in working on it, why not? Once we start solving problems like that, I suspect we end up with something closer in complexity to Gyp or CMake. True, but I think the real problem that we're not addressing in this discussion is that different ports have different sets of requirements, meaning their own evaluation process would lead them to choose different tools. If we want a 'one size fits all' build system, the first step is getting each port to come together and consolidate the requirements, and there will most likely need to be some compromises involved as some ports may have to agree to move to a tool that's not really as well suited for their project as the one they're using now. For example, a primary reason tools like Gyp and Bakefile exist is that for some people, lack of a 100% native IDE-based build system is a deal breaker. For others, like myself, I want low maintenance, completely cross-platform, highly automated and highly scriptable, which are actually features the IDE projects don't fare very well in. So one man's feature is another man's drawback. There are also factors besides features that are important as well. I think it's also important to remember that from each port's perspective, one potentially big factor in build system choice is also making users comfortable with contributing. For GTK, for
Re: [webkit-dev] [webkit meeting notes] build systems
2010/4/17 Kevin Ollivier kev...@theolliviers.com Hi Maciej, On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote: On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now. If I had to do it myself, I would just skip any ports that don't use DerivedSources.make. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development. Converting, say, a JSON list of files to some XML-based output format would not be difficult at all (and I Like this? http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi - It is currently not JSON compliant. It's in fact a python file but that can be fixed: s/'//g and removing the extra commas *should* be sufficient. - It is currently chromium specific, that's what I meant by de-chromification of the gyp files. It's mainly adding more stuff and fixing the regexp if I'm not mistaken. I don't mind if it doesn't become the golden file, the goal is just to hopefully reduce the number of build system, nothing more. believe we can automatically convert the XCode project files from binary to plist and back, though IIRC there might be some UUID handling issues to consider), so we could patch the IDE files much like we patch the other ports. As for the other cases, such as additions / removals of things shared by multiple ports and the derived sources problem (that one probably would indeed need some 'template' to insert into project files), I think these could be added over time if we feel it'd bring a large benefit, but I think even just covering the common case of cross-platform source file maintenance is already a huge win for the project. I maintained Bakefile projects for years, and I'd say 80-90% of the time when a change broke my build, it was one of these common source file additions / removals. And it usually happened several times every week. I personally think the way to look at it is to start by solving the simple stuff that could be solved quickly, as in my experience that makes it far more likely to actually get done. If, say, gyp - Gtk / Qt / MSVC (/ XCode?) build file lists could be done in a weekend of hacking and some devs are interested in working on it, why not? Once we start solving problems like that, I suspect we end up with something closer in complexity to Gyp or CMake. True, but I think the real problem that we're not addressing in this discussion is that different ports have different sets of requirements, meaning their own evaluation process would lead them to choose different tools. If we want a 'one size fits all' build system, the first step is getting each port to come together and I don't think anyone wants that. It's just too involved. consolidate the requirements, and there will most likely need to be some compromises involved as some ports may have to agree to move to a tool that's not really as well suited for their project as the one they're using now. For example, a
Re: [webkit-dev] [webkit meeting notes] build systems
On Apr 17, 2010, at 11:11 AM, Kevin Ollivier wrote: True, but I think the real problem that we're not addressing in this discussion is that different ports have different sets of requirements, meaning their own evaluation process would lead them to choose different tools. If we want a 'one size fits all' build system, the first step is getting each port to come together and consolidate the requirements, and there will most likely need to be some compromises involved as some ports may have to agree to move to a tool that's not really as well suited for their project as the one they're using now. It's true that different ports have different requirements. So far, this has resulted in every port choosing a different build system. While in some ways this may be convenient for individual ports, it creates negative externalities for the project as a whole. That being said, we're not trying to convert to one true build system right now, merely seeing if we can reduce the number by having more ports share a build system. We're even considering changing the build system for Apple's ports (in fact the Apple Windows port may be one of the first experiments). For example, a primary reason tools like Gyp and Bakefile exist is that for some people, lack of a 100% native IDE-based build system is a deal breaker. For others, like myself, I want low maintenance, completely cross-platform, highly automated and highly scriptable, which are actually features the IDE projects don't fare very well in. So one man's feature is another man's drawback. There are also factors besides features that are important as well. I think it's also important to remember that from each port's perspective, one potentially big factor in build system choice is also making users comfortable with contributing. For GTK, for example, that may mean that using autotools, the build system most likely to be familiar to potential contributors, is in part a way of making contributing a little less intimidating on a big project like WebKit. Similar with qmake, XCode, etc. I think that a big part of build system decision making is based not necessarily around features, but around being in the developers' comfort zones. So my gut impression is that it's going to be very difficult to find an existing tool that solves all the issues like this for most / all ports in a way that makes everyone happy. As the saying goes, sometimes the perfect is indeed the enemy of the good. I personally think a quick and simple solution along the lines of what Nikolas and I were talking about maybe the only short term improvement that is viable. Everything else is looking more long- term, and requires both a lot of effort and a lot of collaboration among ports. To the point that, as a practical matter, I'm not sure if the system would save more time than it would take to develop. That's not to say such a system won't be developed, but absent some sponsorship of the project or some highly motivated hackers, I don't know how we plan on getting there. I just think this subject has come up numerous times, and each time the discussion ended without any improvements being made, so I worry that so long as we wait for the perfect system to come along, we're not going to build the good system that can make life better today. I would put that the other way around. Gyp exists today and knows how to generate an Xcode project. The templating solution that can generate an Xcode project while seeming less intrusive to Bakefile and Automake users is completely hypothetical. Anyone who wants to is welcome to try to code it, but my expectation going in is that it would evolve to be as complex as Gyp. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Hi Marc-Antoine, On Apr 17, 2010, at 11:26 AM, Marc-Antoine Ruel wrote: 2010/4/17 Kevin Ollivier kev...@theolliviers.com Hi Maciej, On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote: On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now. If I had to do it myself, I would just skip any ports that don't use DerivedSources.make. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development. Converting, say, a JSON list of files to some XML-based output format would not be difficult at all (and I Like this? http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi - It is currently not JSON compliant. It's in fact a python file but that can be fixed: s/'//g and removing the extra commas should be sufficient. - It is currently chromium specific, that's what I meant by de-chromification of the gyp files. It's mainly adding more stuff and fixing the regexp if I'm not mistaken. I don't mind if it doesn't become the golden file, the goal is just to hopefully reduce the number of build system, nothing more. Yes, precisely why I mentioned JSON and later gyp, though I don't know if Chromium-specific is the right word here. It even has wx port files in it, which I don't think are built by Chromium. :) I suppose you somehow filter out which ones Chromium should build after the fact? If so, where does that logic reside? Anyway, the fact that this file is actually Python (I had forgotten the format was JSON-like rather than straight JSON) makes things even better, as we only really need to handle export now, and not parse some import file list. i.e. for WebKitTools/Scripts/update-sources-list.py, getWebCoreFilesDict() basically becomes a very small script that execfile's the gypi file, then we run whatever filtering mechanism on it (waf has a list of ports we could use to do the filtering that I could probably move into WebKitTools/Scripts, if we don't have a pre-made Chromium solution here), and then passes the final source list along to generateWebCoreSourcesGTKandQT to generate the sources / includes for those platforms, and actually this solution should work for Android.mk and possibly jam too, as their syntax looks largely similar. Then we'd add some XML parsing code to grab the node for common file groups and update them for the MSVC projects, and then I think that mostly leaves XCode, which I think would be pretty similar in nature. As long as people are willing to test out this solution with their build systems and help with any debugging, I would be willing to help out in hacking it together, though I can't promise anything in the way of time since this is not an immediate concern for me personally. Thanks, Kevin___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Adam On Friday 16 April 2010 12:42:01 am Peter Kasting wrote: On Thu, Apr 15, 2010 at 9:15 PM, Kevin Ollivier kev...@theolliviers.comwrote: Anyway, my $0.02 is that, in terms of immediate bang for the buck, we're probably better off trying to synchronize the build systems automatically in some way. My perception is that Qt developers will want to use qmake, GTK will want to use autotools, etc. and while some build systems could certainly be merged (e.g. as you say the Win and Mac projects for WebKit itself could be gyp-generated), I don't think we'll ever really narrow it down to one master system for a variety of reasons. I do, however, think we may be able to narrow the 'build system updating' process down to one step / one script that updates all ports simultaneously, and do so without too much effort. If I'm not mistaking you, that seems to be the route Evan is discussing, with his concrete proposal being to investigate the feasibility of making gyp be the mechanism by which various different build systems can be generated from a single place (rather than seven). Or is this not what you were saying? Since most build systems out there have their data stored in either XML or plain text, converting a build file list from one build system's data format to another is probably not more than a few hours of Python hacking, if that. Indeed, we already have MSVC/XCode/makefile generators for gyp, I would assume that given those, a generator for e.g. qmake wouldn't be hard. I really think someone should seriously consider investing some time and resources into improving this issue though, updating a build system doesn't take long but updating 7-10 build systems on almost a daily basis probably adds up to some pretty significant amount of time and energy that could probably be spent on better things. I think that is the shared viewpoint that has led to this discussion. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $ $SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. Of course this is some kind of meta-meta build system, but we could also agree to move the template files in a WebCore/build-templates folder and run a update-project-files script after adding/remove files, that generates the build files, like WebCore.pro and commit them. This way only the person who added/removed a file from build would need to run that script, instead of every one, this would effectively hide the meta-meta build system. If we find an acceptable solution for just adding/removing cross- platform buildable files that it would already make me happy :-) Looking for comments! Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Hi Peter, On Apr 15, 2010, at 9:42 PM, Peter Kasting wrote: On Thu, Apr 15, 2010 at 9:15 PM, Kevin Ollivier kev...@theolliviers.com wrote: Anyway, my $0.02 is that, in terms of immediate bang for the buck, we're probably better off trying to synchronize the build systems automatically in some way. My perception is that Qt developers will want to use qmake, GTK will want to use autotools, etc. and while some build systems could certainly be merged (e.g. as you say the Win and Mac projects for WebKit itself could be gyp-generated), I don't think we'll ever really narrow it down to one master system for a variety of reasons. I do, however, think we may be able to narrow the 'build system updating' process down to one step / one script that updates all ports simultaneously, and do so without too much effort. If I'm not mistaking you, that seems to be the route Evan is discussing, with his concrete proposal being to investigate the feasibility of making gyp be the mechanism by which various different build systems can be generated from a single place (rather than seven). Or is this not what you were saying? I was talking about something far more limited, like converting common build file lists from one format to another. Since most build systems out there have their data stored in either XML or plain text, converting a build file list from one build system's data format to another is probably not more than a few hours of Python hacking, if that. Indeed, we already have MSVC/XCode/makefile generators for gyp, I would assume that given those, a generator for e.g. qmake wouldn't be hard. Perhaps, but in any case, I think the first step there is for the Gyp developers to try implementing support and see how it goes. However, from that perspective, until Gyp has support for those formats, isn't a discussion about switching for other ports a bit premature on the WebKit side? i.e. that would be next steps for Gyp, not next steps for WebKit. I'd think Gyp would implement support for those formats and then come here and ask people to consider a switch. Obviously there is something to discuss on the WebKit / Win and WebKit / Mac side since Gyp can already handle those, but at least for the other ports I was thinking of something less involved than a build system migration of that nature (at least, in the sense of moving to it being auto-generated). My suggestion was about leaving the actual build system untouched but having it include a file (or having a script automatically replace a group of source files in the project file) with a generated list of common build files. More like a script that generates a patch for each build system when changes are made to common build files than auto-generating the whole project file(s). This way the ports can continue to work with and maintain their own build system, but when a common build file is added or removed, a script is run that makes a patch for all build systems. I think that would cover a vast majority of the manual hassle involved in managing the cross-platform build files. Regards, Kevin I really think someone should seriously consider investing some time and resources into improving this issue though, updating a build system doesn't take long but updating 7-10 build systems on almost a daily basis probably adds up to some pretty significant amount of time and energy that could probably be spent on better things. I think that is the shared viewpoint that has led to this discussion. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Hi Nikolas, On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. Of course this is some kind of meta-meta build system, but we could also agree to move the template files in a WebCore/build-templates folder and run a update-project-files script after adding/remove files, that generates the build files, like WebCore.pro and commit them. This way only the person who added/removed a file from build would need to run that script, instead of every one, this would effectively hide the meta-meta build system. If we find an acceptable solution for just adding/removing cross-platform buildable files that it would already make me happy :-) Correct me if I'm wrong, but I thought it would actually be possible to have the values for HEADER_LIST and SOURCE_LIST set in a separate file, and that Qt and GTK build systems could then just include that file. This way we could avoid using templates and such. Just add the included file to the tree (e.g. WebCore/sources.inc) and have the Qt and GTK projects include WebCore/sources.inc and reference those variables. As I mentioned earlier, I actually coded up something using this approach a long time ago back when wx was still using Bakefile, but I guess it never caught the attention of projects like Qt as I never got any feedback on it. (Or maybe the idea was just so obviously flawed somehow. :) Here's the script: http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/update-sources-list.py It would probably not take long at all to have getWebCoreFilesDict pull from a gyp project, XCode project, or whatever we wanted to use as the master file list that should be updated. Thoughts? Thanks, Kevin Looking for comments! Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
This seems like just a more restricted version of gyp to me. And one that isn't sufficiently powerful, e.g., what if a platform decides to disable SVG and doesn't want to compile SVG files? You end up adding some sort of syntax for that. In the end, you have something that's 90% of the way to a gyp file and now we have yet another build system. Whether the build file generation happens when you sync versus when the person adding the file makes the change is also flexible with gyp. We could very well use gyp to generate the build files for each platform and check that in. I don't actually think that's a good idea though. The risk of build files and gyp files getting out of sync is not worth it. Ojan On Fri, Apr 16, 2010 at 8:09 AM, Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. Of course this is some kind of meta-meta build system, but we could also agree to move the template files in a WebCore/build-templates folder and run a update-project-files script after adding/remove files, that generates the build files, like WebCore.pro and commit them. This way only the person who added/removed a file from build would need to run that script, instead of every one, this would effectively hide the meta-meta build system. If we find an acceptable solution for just adding/removing cross-platform buildable files that it would already make me happy :-) Looking for comments! Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
On Fri, Apr 16, 2010 at 8:42 AM, Kevin Ollivier kev...@theolliviers.comwrote: Perhaps, but in any case, I think the first step there is for the Gyp developers to try implementing support and see how it goes. However, from that perspective, until Gyp has support for those formats, isn't a discussion about switching for other ports a bit premature on the WebKit side? i.e. that would be next steps for Gyp, not next steps for WebKit. I'd think Gyp would implement support for those formats and then come here and ask people to consider a switch. Which is basically what we're proposing here. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Please guys, No need to speculate here on what I proposed at the session since Evan somehow left some details out and let me reinforce some points. For those who missed the session, please understand that what's Peter said isn't exactly what we agreed on. Here's some data points: - No plan to generate qtmake or makefile.am or whatever else - No plan to force anyone to switch - No plan (for now) to get any automatic file list for the checkout (some build system do that) - No plan for any template based project generation - Making a new fully functional backend to gyp *is* involved. It's a big deal. There's no use in a 95% done build system. The rough proposal is: - De-chromify most .gyp and .gypi files currently living in webkit.org, split the files off as necessary in the process to untangle conditions. - With coordination with aroben, look at implementing a faithful copy of apple's win port, with a potential switch over *if it works out*. No guarantee. - Some people expressed interest into generating the file list to their build system and having the build system import the generated project files. That wouldn't be a complete backend implementation. Obviously, complete backend implementation at http://code.google.com/p/gyp are welcome as long as they have unit tests. What you can do *only if you want*: - Extract the file list of your port project into a single simple file and have your build system import this file. I'm not talking about pushing anyone's back and I have *no intention* to. So unless you want to actively submit patches towards the proposal, please let this thread die, it's an unproductive use of my time. I don't plan to do that in the very short term as I have other things on my plate, I'll definitely do this incrementally. M-A 2010/4/16 Peter Kasting pkast...@google.com On Fri, Apr 16, 2010 at 8:42 AM, Kevin Ollivier kev...@theolliviers.comwrote: Perhaps, but in any case, I think the first step there is for the Gyp developers to try implementing support and see how it goes. However, from that perspective, until Gyp has support for those formats, isn't a discussion about switching for other ports a bit premature on the WebKit side? i.e. that would be next steps for Gyp, not next steps for WebKit. I'd think Gyp would implement support for those formats and then come here and ask people to consider a switch. Which is basically what we're proposing here. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- M-A ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Am 16.04.2010 um 18:26 schrieb Ojan Vafai: This seems like just a more restricted version of gyp to me. And one that isn't sufficiently powerful, e.g., what if a platform decides to disable SVG and doesn't want to compile SVG files? You end up adding some sort of syntax for that. In the end, you have something that's 90% of the way to a gyp file and now we have yet another build system. Ermm, we have ENABLE(SVG) blocks around all SVG related files. That was not a good example, as those platforms may just compile the SVG* files, they are just empty. So no need to add any syntax for that. That applies to all other features as well - we _always_ compile all files, not depending on any setting. Cheers, Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Am 16.04.2010 um 19:14 schrieb Marc-Antoine Ruel: Please guys, No need to speculate here on what I proposed at the session since Evan somehow left some details out and let me reinforce some points. For those who missed the session, please understand that what's Peter said isn't exactly what we agreed on. What you can do only if you want: - Extract the file list of your port project into a single simple file and have your build system import this file. I'm not talking about pushing anyone's back and I have no intention to. So unless you want to actively submit patches towards the proposal, please let this thread die, it's an unproductive use of my time. I don't plan to do that in the very short term as I have other things on my plate, I'll definitely do this incrementally. This thread is not only about what you may have proposed in the meeting. Please reread the other mails of the thread. I'm really a bit astonished, reading statments like your last sentences above. It's not very polite to state you don't care about other opinions and call that an unproductive use of your time. Cheers, Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now. If I had to do it myself, I would just skip any ports that don't use DerivedSources.make. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $ $SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development. Once we start solving problems like that, I suspect we end up with something closer in complexity to Gyp or CMake. Regards, Maciej Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
2010/4/16 Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de Am 16.04.2010 um 19:14 schrieb Marc-Antoine Ruel: Please guys, No need to speculate here on what I proposed at the session since Evan somehow left some details out and let me reinforce some points. For those who missed the session, please understand that what's Peter said isn't exactly what we agreed on. What you can do *only if you want*: - Extract the file list of your port project into a single simple file and have your build system import this file. I'm not talking about pushing anyone's back and I have *no intention* to. So unless you want to actively submit patches towards the proposal, please let this thread die, it's an unproductive use of my time. I don't plan to do that in the very short term as I have other things on my plate, I'll definitely do this incrementally. This thread is not only about what you may have proposed in the meeting. Please reread the other mails of the thread. I'm really a bit astonished, reading statments like your last sentences above. It's not very polite to state you don't care about other opinions and call that an unproductive use of your time. Cheers, Niko Sorry, I didn't mean to be blunt but we did wear the PM hat and decide AI at the session. Chromium folks did look at CMake in the past, I'll add comments related to this build system to the other thread. -- M-A ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
Am 16.04.2010 um 18:14 schrieb Kevin Ollivier: Hi Nikolas, On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote: Am 16.04.2010 um 16:44 schrieb Adam Treat: I am very skeptical that it is feasible to write a gyp generator that would output QMake files. There is a log of magic in those QMake files. My sense is that it would not be trivial by any means. Plus, I don't like the idea of a meta-meta generators. Seems way to mickey- mouse to me. Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well. What we really need to resolve is adding/removing files from compilation, that's the most common task that has to be done in 5+ build systems at the moment. So I have a new proposal: 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..) 2) Keep all existing Makefile.am, WebCore.pro etc files as templates, ie. WebCore.pro.template, with a special variable somewhere marking the $$HEADER_LIST$$ and the $ $SOURCE_LIST$$ 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only needs to insert the file list with the correct syntax in the correct places 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template) I think we'll never find a consensus on a single build system, there are too many different needs. I only care about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files. Of course this is some kind of meta-meta build system, but we could also agree to move the template files in a WebCore/build-templates folder and run a update-project-files script after adding/remove files, that generates the build files, like WebCore.pro and commit them. This way only the person who added/removed a file from build would need to run that script, instead of every one, this would effectively hide the meta-meta build system. If we find an acceptable solution for just adding/removing cross- platform buildable files that it would already make me happy :-) Correct me if I'm wrong, but I thought it would actually be possible to have the values for HEADER_LIST and SOURCE_LIST set in a separate file, and that Qt and GTK build systems could then just include that file. This way we could avoid using templates and such. Just add the included file to the tree (e.g. WebCore/sources.inc) and have the Qt and GTK projects include WebCore/sources.inc and reference those variables. As I mentioned earlier, I actually coded up something using this approach a long time ago back when wx was still using Bakefile, but I guess it never caught the attention of projects like Qt as I never got any feedback on it. (Or maybe the idea was just so obviously flawed somehow. :) Here's the script: http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/update-sources-list.py It would probably not take long at all to have getWebCoreFilesDict pull from a gyp project, XCode project, or whatever we wanted to use as the master file list that should be updated. Hi Kevin, I think that solution could work, though when reading through the other mails, especially Maciejs mail regarding the generated-source files, this solution probably is just too simple. I also take back my proposal, I didn't have certain things in mind when writing it, especially not the generated sources.. We do need something more sophisticated. Have a nice day, Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] [webkit meeting notes] build systems
On Wed, Apr 14, 2010 at 4:16 PM, Evan Martin e...@chromium.org wrote: On Tuesday some people gathered to discuss the build-system bingo within WebKit and what we can do to reduce (if not unify) the number of project files you need to modify when you rename a file. I took the stage as a representative of Chromium's gyp project (I am responsible for its Linux Makefile generator). Here's a quick list of the different ports I'm aware of and their build systems. (Sorry for the non-monospaced tabbing, hope it comes out ok...) mac xcode win vcproj qt qmake (cross-platform output) gtk autotools android Android.mk (I heard rumors Android isn't actually upstream yet, though (?)) chrome gyp (cross-platform output) haiku jam (?) brew waf (?) derivative builds: rim hacked version of qt (qmake) efl qmake actually we were using automake to share with gtk, but it is becoming a mess and we're trying to help clean it. as it might not happen we are almost done with a cmake version... adding even more entropy :-P sony autotools (like gtk) wincairo win + vsprops wince like wincairo (different platforms: use a tree of vcproj) air hacked win (?) Nobody is happy with the current situation where you must edit seven files to avoid breaking someone's build; the question is what concrete steps can improve it. Everyone present seemed at least mildly positive on migrating to other systems, though everyone is also understandably skeptical of how much work would be involved. We (Googlers) built gyp to deal with this exact problem -- building Chrome (including WebKit) on Win/Mac/Linux while still producing natural build files for the existing IDEs. (On Linux, we generate non-recursive Makefiles that get dependencies correct.) It would take a non-zero amount of work to use it for building things that are not Chrome, as in one interpretation gyp's input data is kind of just a glorified ls -R in JSON syntax, but it has worked well for us. In general, the more complicated your platform gets (like the many varieties of WinCE out there) the more complicated your build system needs are, so I expect those would be the most work to change. (For example, the RIM build generates Makefiles on Windows but doesn't use Visual Studio or the MS compiler, which is pretty esoteric from a build system feature checklist perspective. On the other hand, they said it takes 7 minutes to finish generation on a fast computer, so they might be more interested than others in alternatives.) But particularly for the relatively simple builds (like the win/mac ports) it is likely that gyp would be adequate without much work. Perhaps they could serve as a proving ground for gyp. Concretely, I'd propose: - strongly encourage newer ports to hook on to an existing build, since new build systems add overhead for everyone - chromium/gyp developers should prove gyp by making it build a non-chromium port; that includes subtasks of - getting the gyp+jsc build working again (it has worked in the past) - see how close to the existing mac/win build files we can get with gyp In particular, if we can generate build files like the Apple Windows WebKit port, there are other projects like the Windows Cairo build and the WinCE work that could hopefully benefit with minor tweaking. From there, it would probably fall on other builds (qmake, autotools) to see if they can benefit from gyp's shared file lists. From my brief tinkering with the GTK build I think I could generate Makefile.am's pretty easily; I lack experience with qmake. But I also think it would be a good thing to simply reduce the number of build systems EFL port is fine with whatever builds. As I just finished writing the CMake based on autotools from GTK with quick look at Qt's and others, I can give some useful input and review: 1. generators need to be normalized, really. At least input/output files need conversions to help them to be made by automatic rules. Some scripts take input and output as parameters, some assumes files are in the current directory and generates others in the current directory. Others reads such information from .in makefiles and can generate file names that are impossible to guess from the command itself without reading the actual input files! :-( DerivedSources... but also generated sources elsewhere... needs to normalize it as well. 2. while common parts are pretty straightforward to share, the number of customization options makes the system a mess. That was one of the biggest problems of sharing autotools with GTK, the if OPTION then SOMETHING endif became huge. Most of the SOMETHING is quite similar, so if we enforce naming schemes we can have general rules that does something like: Option$PORT.xyz, and it will work for all ports that supports it. Having a configuration system that shares is
Re: [webkit-dev] [webkit meeting notes] build systems
Hi Evan, On Apr 14, 2010, at 5:16 PM, Evan Martin wrote: On Tuesday some people gathered to discuss the build-system bingo within WebKit and what we can do to reduce (if not unify) the number of project files you need to modify when you rename a file. I took the stage as a representative of Chromium's gyp project (I am responsible for its Linux Makefile generator). Here's a quick list of the different ports I'm aware of and their build systems. (Sorry for the non-monospaced tabbing, hope it comes out ok...) macxcode win vcproj qtqmake (cross-platform output) gtk autotools androidAndroid.mk (I heard rumors Android isn't actually upstream yet, though (?)) chromegyp (cross-platform output) haiku jam (?) brewwaf (?) wx uses waf, not sure about brew. derivative builds: rim hacked version of qt (qmake) efl qmake sonyautotools (like gtk) wincairo win + vsprops wince like wincairo (different platforms: use a tree of vcproj) air hacked win (?) Nobody is happy with the current situation where you must edit seven files to avoid breaking someone's build; the question is what concrete steps can improve it. Everyone present seemed at least mildly positive on migrating to other systems, though everyone is also understandably skeptical of how much work would be involved. [snip] Concretely, I'd propose: - strongly encourage newer ports to hook on to an existing build, since new build systems add overhead for everyone - chromium/gyp developers should prove gyp by making it build a non-chromium port; that includes subtasks of - getting the gyp+jsc build working again (it has worked in the past) - see how close to the existing mac/win build files we can get with gyp In particular, if we can generate build files like the Apple Windows WebKit port, there are other projects like the Windows Cairo build and the WinCE work that could hopefully benefit with minor tweaking. From there, it would probably fall on other builds (qmake, autotools) to see if they can benefit from gyp's shared file lists. From my brief tinkering with the GTK build I think I could generate Makefile.am's pretty easily; I lack experience with qmake. But I also think it would be a good thing to simply reduce the number of build systems FWIW, for the wx build, I have waf calculate a list of sources to build by passing in a set of directories (also used for setting includes) and rules about feature defines and port info. So a sample rule list might be compile any source file in any of the following root directories, and also compile any source file in any wx and curl subdirectory, along with any source file in a cf subdirectory when building on Mac. Thanks to the highly structured nature of the WebKit build system, this works surprisingly well. Compiling a source file list (of even 1200+ files) actually adds almost no time to the build but has saved me many hours of patching build file lists and/or sharing that maintenance work with some other port. 90% of the build fixes I have to commit these days are new platform methods that need a port implementation (or at least a stub). Since waf is really just a Python API, this is all done in straight Python, and outside of the addition of new directories, I almost never need to update the build system, which is probably one reason most contributors hardly ever touch wx or notice it anymore, even though we're still out there chugging along at filling the implementation holes. ;-) Anyway, my $0.02 is that, in terms of immediate bang for the buck, we're probably better off trying to synchronize the build systems automatically in some way. My perception is that Qt developers will want to use qmake, GTK will want to use autotools, etc. and while some build systems could certainly be merged (e.g. as you say the Win and Mac projects for WebKit itself could be gyp-generated), I don't think we'll ever really narrow it down to one master system for a variety of reasons. I do, however, think we may be able to narrow the 'build system updating' process down to one step / one script that updates all ports simultaneously, and do so without too much effort. Since most build systems out there have their data stored in either XML or plain text, converting a build file list from one build system's data format to another is probably not more than a few hours of Python hacking, if that. I even long ago made a start on such an approach in WebKitTools/Scripts/update-sourc es-list.py, that used the old Bakefile XML to grab the list of common WebCore sources, but I bet it could be updated to pull from the .gyp file lists in no time. I really think someone should seriously consider investing some time and resources into improving this issue though, updating a build system doesn't take long but updating 7-10 build systems on almost a
Re: [webkit-dev] [webkit meeting notes] build systems
On Thu, Apr 15, 2010 at 9:15 PM, Kevin Ollivier kev...@theolliviers.comwrote: Anyway, my $0.02 is that, in terms of immediate bang for the buck, we're probably better off trying to synchronize the build systems automatically in some way. My perception is that Qt developers will want to use qmake, GTK will want to use autotools, etc. and while some build systems could certainly be merged (e.g. as you say the Win and Mac projects for WebKit itself could be gyp-generated), I don't think we'll ever really narrow it down to one master system for a variety of reasons. I do, however, think we may be able to narrow the 'build system updating' process down to one step / one script that updates all ports simultaneously, and do so without too much effort. If I'm not mistaking you, that seems to be the route Evan is discussing, with his concrete proposal being to investigate the feasibility of making gyp be the mechanism by which various different build systems can be generated from a single place (rather than seven). Or is this not what you were saying? Since most build systems out there have their data stored in either XML or plain text, converting a build file list from one build system's data format to another is probably not more than a few hours of Python hacking, if that. Indeed, we already have MSVC/XCode/makefile generators for gyp, I would assume that given those, a generator for e.g. qmake wouldn't be hard. I really think someone should seriously consider investing some time and resources into improving this issue though, updating a build system doesn't take long but updating 7-10 build systems on almost a daily basis probably adds up to some pretty significant amount of time and energy that could probably be spent on better things. I think that is the shared viewpoint that has led to this discussion. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
