Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
If you're willing to give it a shot, then that sounds like a fine idea. - Maciej On Jul 15, 2009, at 10:51 PM, Ryan Leavengood wrote: On Mon, Jul 13, 2009 at 4:56 PM, Maciej Stachowiakm...@apple.com wrote: One belated comment on this topic. It would be neat if some port agreed to be the guinea pig to see if gyp could plausibly work for more than Google's ports. The Wx port probably has the lowest resources of any complete port in the tree, so they might not be the best choice of experimental subject, particularly if for them the process required writing a new gyp back end and if they are not yet entirely comfortable going the gyp route. I would need to discuss it with my student, but what about the brand new Haiku port being the gyp guinea pig? For those who don't know, I am mentoring a student in the Google Summer of Code for the Haiku operating system (http://www.haiku-os.org) and we are working on a native Haiku web browser with WebKit as the rendering engine. I don't know if our port is any better of a choice than the Wx port, since the resources are also small (just two of us for now) and we aren't even in the WebKit tree yet, but I think we still might be a good choice because: 1) We obviously don't yet have a production browser using our port so breakage isn't an issue. Plus only my student (Maxime Simon) and I are working on it. 2) I have decent experience with build systems and think I could handle working with gyp and writing a new back end. 3) Haiku generally uses Jam for building and we would like our port to do the same. Rather than adding Yet Another Build System to WebKit, we could use gyp and write a Jam backend for it. This can therefore serve as a test of gyp for another platform as well as for another backend. I would rather not have to maintain a Jamfile for WebKit if I can avoid it, and I certainly don't want to burden the other WebKit developers with having to maintain it for what is now (and may forever be) a tiny port. Though we certainly hope Haiku's popularity increases in the future (it hasn't even had a first release anyhow, so there is plenty of room to grow.) Anyhow, I'd be interested in hearing what other people think. -- Regards, Ryan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Mon, Jul 13, 2009 at 4:56 PM, Maciej Stachowiakm...@apple.com wrote: One belated comment on this topic. It would be neat if some port agreed to be the guinea pig to see if gyp could plausibly work for more than Google's ports. The Wx port probably has the lowest resources of any complete port in the tree, so they might not be the best choice of experimental subject, particularly if for them the process required writing a new gyp back end and if they are not yet entirely comfortable going the gyp route. I would need to discuss it with my student, but what about the brand new Haiku port being the gyp guinea pig? For those who don't know, I am mentoring a student in the Google Summer of Code for the Haiku operating system (http://www.haiku-os.org) and we are working on a native Haiku web browser with WebKit as the rendering engine. I don't know if our port is any better of a choice than the Wx port, since the resources are also small (just two of us for now) and we aren't even in the WebKit tree yet, but I think we still might be a good choice because: 1) We obviously don't yet have a production browser using our port so breakage isn't an issue. Plus only my student (Maxime Simon) and I are working on it. 2) I have decent experience with build systems and think I could handle working with gyp and writing a new back end. 3) Haiku generally uses Jam for building and we would like our port to do the same. Rather than adding Yet Another Build System to WebKit, we could use gyp and write a Jam backend for it. This can therefore serve as a test of gyp for another platform as well as for another backend. I would rather not have to maintain a Jamfile for WebKit if I can avoid it, and I certainly don't want to burden the other WebKit developers with having to maintain it for what is now (and may forever be) a tiny port. Though we certainly hope Haiku's popularity increases in the future (it hasn't even had a first release anyhow, so there is plenty of room to grow.) Anyhow, I'd be interested in hearing what other people think. -- Regards, Ryan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi Peter, On Jul 13, 2009, at 2:03 PM, Peter Kasting wrote: On Mon, Jul 13, 2009 at 1:56 PM, Maciej Stachowiak m...@apple.com wrote: One belated comment on this topic. It would be neat if some port agreed to be the guinea pig to see if gyp could plausibly work for more than Google's ports. The Wx port probably has the lowest resources of any complete port in the tree, so they might not be the best choice of experimental subject, particularly if for them the process required writing a new gyp back end and if they are not yet entirely comfortable going the gyp route. Another note, based on some #chromium conversations: if someone passionate made CMake (or any other tool) into something compelling enough to work better for Chromium than gyp does (or at least to work close-to-as-well), and that tool was more plausible for other ports in the WK tree to adopt, we wouldn't be opposed in principle to using it. The potential benefits of a shared build system are clear, and we're not trying to tell people that system has to be gyp; we're just probably not prepared to be the ones to go determine other ports' needs and decide on the Build System To Rule Them All. If no one wants to do this, but other ports do want to try gyp, we can lend them a hand in checking it out too. Whatever makes things at least a little easier. After reading Mark's comments, one idea that has been stewing around in my head is seeing if it would be possible to have waf optionally call the GYP GenerateOutput methods directly, instead of doing a build, by converting the final, computed compiler/linker settings into a GYP-friendly series of lists/dicts. If that could be made to work, it would offer the best of both worlds. On one hand, we have a fast and scalable build system that can be extended to do pretty much anything you can code up in Python, and on the other, you can still use IDE projects if that's what you're comfortable with. Of course, the big question is if waf would have the same limitations as SCons in regards to doing this, but I think it's at least worth exploring. I'd be interested to know what limitations you guys ran into when trying to use SCons for this sort of thing. Of course, there will always be some things that would be hard to do with the IDE projects, but it may be possible for us to make use of waf there too, by using command line arguments to have it, for example, only generate the derived sources, which we could call from the IDE projects. Thoughts? Thanks, Kevin PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Tue, Jul 14, 2009 at 9:24 AM, Kevin Ollivier kev...@theolliviers.comwrote: Of course, the big question is if waf would have the same limitations as SCons in regards to doing this, but I think it's at least worth exploring. I'd be interested to know what limitations you guys ran into when trying to use SCons for this sort of thing. Of course, there will always be some things that would be hard to do with the IDE projects, but it may be possible for us to make use of waf there too, by using command line arguments to have it, for example, only generate the derived sources, which we could call from the IDE projects. I don't really know enough about either waf or SCons to comment. My impression as a naive observer was that SCons had serious speed issues with our build (especially the overhead for a null build). PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Tue, Jul 14, 2009 at 10:27 AM, Peter Kasting pkast...@google.com wrote: On Tue, Jul 14, 2009 at 9:24 AM, Kevin Ollivier kev...@theolliviers.comwrote: Of course, the big question is if waf would have the same limitations as SCons in regards to doing this, but I think it's at least worth exploring. I'd be interested to know what limitations you guys ran into when trying to use SCons for this sort of thing. Of course, there will always be some things that would be hard to do with the IDE projects, but it may be possible for us to make use of waf there too, by using command line arguments to have it, for example, only generate the derived sources, which we could call from the IDE projects. I don't really know enough about either waf or SCons to comment. My impression as a naive observer was that SCons had serious speed issues with our build (especially the overhead for a null build). PK In addition to slow startup time, the SCons system meant that we had to give up full IDE integration. That was the main deal breaker, which led us to look for a xcode/vcproj/etc. generation solution. -Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Kevin Ollivier wrote: I'd be interested to know what limitations you guys ran into when trying to use SCons for this sort of thing. The SCons input language is too free-form to be easily shoehorned into things like Xcode and Visual Studio projects. SCons is flexible enough that mapping its input language, and all of the logic that its input might contain, to just about anything else would have been an incredibly difficult task. For anything more involved than compile these files and produce a library or executable, with SCons, we would have needed to produce project files that weren't strictly native. The project files would have had to have contained instructions to call SCons to perform the actual build. We sought a greater level of native tool integration. Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi, Kevin Ollivier wrote: I'd be interested to know what limitations you guys ran into when trying to use SCons for this sort of thing. On Tue, Jul 14, 2009 at 11:04 AM, Mark Mentovaim...@chromium.org wrote: The SCons input language is too free-form to be easily shoehorned into things like Xcode and Visual Studio projects. The thing that worries me about this latest push to add GYP is that there was much Google enthusiasm for the SCons stuff six months or so ago, and SCons stuff started landing in the tree, and now that's all been tossed away (apparently for performance reasons). Do we feel that GYP is finally the right tool? -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Tue, Jul 14, 2009 at 11:20 AM, Brent Fulgham bfulg...@gmail.com wrote: Do we feel that GYP is finally the right tool? Well, unlike SCons we're actually building all our ports on top of GYP. We never actually expanded SCons to all platforms or addressed the problems we had with it. But I can't speak for eternity. If some other tool comes up that's even better then I imagine we'd switch to that. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi Peter, On Jul 14, 2009, at 10:27 AM, Peter Kasting wrote: On Tue, Jul 14, 2009 at 9:24 AM, Kevin Ollivier kev...@theolliviers.com wrote: Of course, the big question is if waf would have the same limitations as SCons in regards to doing this, but I think it's at least worth exploring. I'd be interested to know what limitations you guys ran into when trying to use SCons for this sort of thing. Of course, there will always be some things that would be hard to do with the IDE projects, but it may be possible for us to make use of waf there too, by using command line arguments to have it, for example, only generate the derived sources, which we could call from the IDE projects. I don't really know enough about either waf or SCons to comment. My impression as a naive observer was that SCons had serious speed issues with our build (especially the overhead for a null build). Yeah, I hit the same problems as well, and after some profiling and discussions on the SCons mailing list, came to the conclusion that SCons simply had architectural issues that made it a poor choice for a large project like WebKit. The overhead for waf is minimal, though, and quite possibly even less than make on Unix systems. Some benchmarks done have shown it to be significantly faster than make, in fact, which is my subjective opinion as well. My Windows builds also feel noticeably faster, but then again I was using .vc makefiles, which meant I couldn't do parallel builds there. ;-) I just haven't had time to really sit down and benchmark. Thanks, Kevin PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Brent Fulgham wrote: The thing that worries me about this latest push to add GYP is that there was much Google enthusiasm for the SCons stuff six months or so ago, and SCons stuff started landing in the tree, and now that's all been tossed away (apparently for performance reasons). Do we feel that GYP is finally the right tool? It's fair to say that it's always been the Chromium team's desire to move at least some chunk of our WebKit build system upstream into the WebKit repository. Conceptually, the instructions for building WebKit belong with WebKit. What you're seeing with Dimitri's recent checkin of .gypi files is a step in this direction. Given this, you're quite right to pick up on the build system du jour nature of the discussion. A year ago, the Chromium team was expecting SCons files to become the master description of our build, but for various reasons including performance, that didn't really work out. We were gung-ho about SCons, but our enthusiasm may have been premature: at least for Chromium, it turned out to not be the right solution right now. The difference between the SCons-driven and GYP-driven builds of Chromium is that with the latter, we really were able to unify our disparate build systems. With GYP, we've got a single build system instead of three (or four), and we were able to do that while meeting the other requirements peculiar to our project. GYP may just be another build system du jour, but cette jour, it's actually carrying Chromium on all of its platforms. It's also important to keep in mind that as long as Chromium is using GYP for its build, we (the Chromium team) are committed to maintaining .gyp/.gypi files checked into the WebKit repository at least as well as they need to be maintained to support our use. Speaking as someone heavily involved with GYP's development, I'd be thrilled if it worked for other projects too; speaking as a Chromium team member, it'd be helpful if people working on other WebKit ports maintained the .gyp/.gypi files as they add, move, and remove files in the WebKit tree, even if no port other than Chromium is actually using GYP. To be honest, though, I don't think that WebKit (or any other project) should choose a build system hastily, and it's entirely conceivable that what works best for Chromium might not work best for WebKit. Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
will be replaced by waf soon) - Qt Qmake - GTK GNUMakefile - Google's gyp (added recently) Are any of the other ports going to switch to generating their build files using gyp? Dave - Original Message From: Kevin Ollivier kev...@theolliviers.com To: Dimitri Glazkov dglaz...@chromium.org Cc: Mark Mentovai m...@chromium.org; WebKit Development webkit-dev@lists.webkit.org Sent: Friday, July 10, 2009 8:52:57 AM Subject: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family) Hi Dimitri and all, Congrats on getting this into WebKit! Actually, I'm in the middle of a build system switch as well - to waf, a re-write of scons that removed many of the performance issues related to searching and calculating dependencies, and which has added some nice features as well (such as .app bundle building). I haven't completed the switch so I can't do preliminary benchmarks, but I'm pretty sure it's actually as fast or faster than make on *nix/Mac. (And BTW, it will probably make Apple devs happy to hear that I'm no longer using the horrid build-wxwebkit bash script to manage the build, but instead have integrated everything into build-webkit finally!) The main reason I bring this up, though, is because I think this sort of thing shows that we're unlikely to centralize our build systems any time soon, and I feel a bit sorry for the core devs who, as they accept new ports, are probably finding it more and more tedious, if not difficult, to make sure all the projects get updated by a change to the common parts of the build. They've been very helpful in terms of trying to keep the ports in shape when they make changes, and I feel like I'd like to do what I can to make it easier and faster to keep the other ports in sync. At one point, I started on a script (located in WebKitTools/Scripts/update-sources-list.py) whose idea was to take the list of common sources from one file, and make changes to MSVC, Qt, GTK, etc. build systems, so that WebKit devs need only add the file once, run the script, and commit the results. I got it as far as theoretically generating the Qt and GTK file lists, but I don't think anyone on those ports tried integrating it into their build system, and I sort of moved on to other things. Unfortunately, right now I'm really swamped (my build system rewrite was prompted by WebKit exceeding internal, hardcoded, Bakefile limits, not by choice), but if a common location for the source files list could be decided upon, I really think a script would be a simple matter to write up (even in Perl :P ), and I think it would probably save developers time and reduce build breakages as well, which I think would add up to a lot of saved time for a lot of people over the long term. The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. So, does anyone think this would be a bad idea, or have any alternate suggestions on how to improve things? If not, is anyone willing to take the ball and run with it? I'd be willing to invest some more time in it, but my ability to commit to it over the next couple weeks at least would be very limited. Thanks, Kevin On Jul 9, 2009, at 9:23 PM, Dimitri Glazkov wrote: Dear WebKiteurs, In our persisting quest to be more like a common WebKit port, we have added Chromium build files to the tree this afternoon. These files are WebCore/WebCore.gypi and JavaScriptCore/JavaScriptCore.gypi and they are the GYP include files. As you may know, we use GYP (http://code.google.com/p/gyp) for generating MSVC, XCode, Scons, and even Make projects for Chromium. We are rather fond of GYP. Perhaps it is because it allows us to maintain one set of project files for all three Chromium platforms; or maybe because it lets us to do things like WebCore.gypi, where we can just mindlessly add all project files to the list and then use various neat GYP filtering facilities to narrow them down to sets that are relevant for specific builds; or maybe because it easifies creating cross-platform and cross-build-system targets, actions, and rules; or maybe because we just love saying Gyp! I don't truthfully know. What I do know is that starting now, we'd love for you to remember WebCore.gypi and JavaScriptCore.gypi when you are adding or removing files from WebCore or JavaScriptCore. Thanks to the power of GYP, you don't have worry whether this file will be used by Chromium: the rule is that if there's a project file change, it applies to the *.gypi files. The format is very simple and intuitive, a simple Python/JSONey list+dict. Thank
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Mon, Jul 13, 2009 at 1:56 PM, Maciej Stachowiak m...@apple.com wrote: One belated comment on this topic. It would be neat if some port agreed to be the guinea pig to see if gyp could plausibly work for more than Google's ports. The Wx port probably has the lowest resources of any complete port in the tree, so they might not be the best choice of experimental subject, particularly if for them the process required writing a new gyp back end and if they are not yet entirely comfortable going the gyp route. Another note, based on some #chromium conversations: if someone passionate made CMake (or any other tool) into something compelling enough to work better for Chromium than gyp does (or at least to work close-to-as-well), and that tool was more plausible for other ports in the WK tree to adopt, we wouldn't be opposed in principle to using it. The potential benefits of a shared build system are clear, and we're not trying to tell people that system has to be gyp; we're just probably not prepared to be the ones to go determine other ports' needs and decide on the Build System To Rule Them All. If no one wants to do this, but other ports do want to try gyp, we can lend them a hand in checking it out too. Whatever makes things at least a little easier. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Jul 13, 2009, at 2:03 PM, Peter Kasting wrote: On Mon, Jul 13, 2009 at 1:56 PM, Maciej Stachowiak m...@apple.com wrote: One belated comment on this topic. It would be neat if some port agreed to be the guinea pig to see if gyp could plausibly work for more than Google's ports. The Wx port probably has the lowest resources of any complete port in the tree, so they might not be the best choice of experimental subject, particularly if for them the process required writing a new gyp back end and if they are not yet entirely comfortable going the gyp route. Another note, based on some #chromium conversations: if someone passionate made CMake (or any other tool) into something compelling enough to work better for Chromium than gyp does (or at least to work close-to-as-well), and that tool was more plausible for other ports in the WK tree to adopt, we wouldn't be opposed in principle to using it. The potential benefits of a shared build system are clear, and we're not trying to tell people that system has to be gyp; we're just probably not prepared to be the ones to go determine other ports' needs and decide on the Build System To Rule Them All. If no one wants to do this, but other ports do want to try gyp, we can lend them a hand in checking it out too. Whatever makes things at least a little easier. I haven't yet reviewed all the issues, but I personally find gyp to be a more plausible choice than CMake based on what I know. I will see how Apple's build system folks feel about the matter. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Maciej Stachowiak wrote: Another note, based on some #chromium conversations: if someone passionate made CMake (or any other tool) into something compelling enough to work better for Chromium than gyp does (or at least to work close-to-as-well), and that tool was more plausible for other ports in the WK tree to adopt, we wouldn't be opposed in principle to using it. The potential benefits of a shared build system are clear, and we're not trying to tell people that system has to be gyp; we're just probably not prepared to be the ones to go determine other ports' needs and decide on the Build System To Rule Them All. If no one wants to do this, but other ports do want to try gyp, we can lend them a hand in checking it out too. Whatever makes things at least a little easier. I haven't yet reviewed all the issues, but I personally find gyp to be a more plausible choice than CMake based on what I know. I will see how Apple's build system folks feel about the matter. The other obvious choice to start with is qmake, since it's already used by the Qt port. Have you got a list of issues with that similar to the list for cmake? Joe ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi Jeremy, On Jul 10, 2009, at 6:54 PM, Jeremy Orlow wrote: On Fri, Jul 10, 2009 at 6:26 PM, Kevin Ollivier kev...@theolliviers.com wrote: Hi Jeremy, On Jul 10, 2009, at 3:03 PM, Jeremy Orlow wrote: [snip] Your argument makes sense if WebKit is only built for one platform/ build-system. Unfortunately it's not. So the question is whether it's easier to maintain lots of different build files or whether it's easier to maintain one file that's perhaps a bit more abstract + the tool that interprets it. I agree that working directly in the project file for your platform is easier IF you're only developing for one platform. But once you start maintaining more than one project file, I think GYP is a big win. While we hope that others will update our GYPI file when they add/ remove files, our build depends on it...so we'll definitely be keeping it in sync. So I think the question then becomes whether it's easier for you to maintain your new build format, or whether it's easier to make it a target for GYP. I honestly don't know what the answer is, but I think it's worth taking a closer look at GYP. Actually, the big question in regards to having GYP reduce overall project maintenance is whether or not the other ports will adopt GYP. If the answer is yes, then it would be more compelling for wx to do so as well, assuming of course that someone implements a waf backend so that we can. :-) If the answer is no, though, then GYP is not reducing the amount of project maintenance work for any port other than Chromium, in which case there will still be 6 build systems (still 5 even if wx were to switch) and the problem I originally posed in this thread will still be an issue. In that case, the only way to really reduce the maintenance work of adding / removing files would again be to adopt a script like the one I suggested earlier. You're right. The burden of updating the GYPI file is less than the others, but it's still another file you need to change. But note that the maintainers of each project/platform/etc define which files to exclude. This is different from the other build systems where the person updating the file needs to decide whether it should go into the build or whether it should only be built with certain flags. Speaking of which, with waf / Python I've actually almost completely automated the generation of the list of include dirs for my build projects based on the source files, so that I'm not maintaining them by hand anymore except for a few exceptions. And thinking about it, I bet I can even mostly automate the list of source files too, by having it grab all the .cpp files in the common dirs and special subdirs like curl and wx, then having some include / exclude filters to deal with a few special cases. :-) The question will be the performance hit, but at least with the includes it's not even noticeable, and I could always look into caching and changing only when you do an svn up or svn add/remove. I suggest you take a look at GYP. Much of what you're talking about it already does. Of course, GYP (and Bakefile, too) supports a wildcard file matching and exclusion mechanism, but that's not really as flexible as what I'm talking about. I'm talking about something that calculates and then traverses a list of subdirectories where even the set of subdirectories traversed are conditional upon variable values set earlier in the build. It's certainly possible what I'm asking for could be made to work, but in the end if I did get it going I'm pretty sure that I would be very unhappy with the syntax because, quite simply, a data format is not really designed to support recursion, looping and the like. It's, well, designed for storing data, which is why even conditionals are expressed in GYP using nested lists and property=value syntax. So the bottom line for me is not really that I couldn't figure out a way to use GYP, as after all I've been using something like it for years, rather that when I need more flexibility and more control, the way to get that using a format like Bakefile or GYP does not feel natural or intuitive; it's more like trying to squeeze a square peg into a round hole. Even the question of where should the change go - extend GYP, throw it somewhere in build-webkit, write a build command that spits its output to GYP, etc. can become a difficult question. As for waf, the only question is whether the change should go in the common portion of the build system, or in a project-specific part, and refactoring as needs change is simple and straightforward, meaning I'm more likely to just do it rather than settle for 'good enough' just because refactoring would require extending the project generator tool, etc. GYP's primary goal is to generate IDE project files, because that's what apparently many ports want. Since IDE projects have many of
[webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi Dimitri and all, Congrats on getting this into WebKit! Actually, I'm in the middle of a build system switch as well - to waf, a re-write of scons that removed many of the performance issues related to searching and calculating dependencies, and which has added some nice features as well (such as .app bundle building). I haven't completed the switch so I can't do preliminary benchmarks, but I'm pretty sure it's actually as fast or faster than make on *nix/Mac. (And BTW, it will probably make Apple devs happy to hear that I'm no longer using the horrid build-wxwebkit bash script to manage the build, but instead have integrated everything into build-webkit finally!) The main reason I bring this up, though, is because I think this sort of thing shows that we're unlikely to centralize our build systems any time soon, and I feel a bit sorry for the core devs who, as they accept new ports, are probably finding it more and more tedious, if not difficult, to make sure all the projects get updated by a change to the common parts of the build. They've been very helpful in terms of trying to keep the ports in shape when they make changes, and I feel like I'd like to do what I can to make it easier and faster to keep the other ports in sync. At one point, I started on a script (located in WebKitTools/Scripts/ update-sources-list.py) whose idea was to take the list of common sources from one file, and make changes to MSVC, Qt, GTK, etc. build systems, so that WebKit devs need only add the file once, run the script, and commit the results. I got it as far as theoretically generating the Qt and GTK file lists, but I don't think anyone on those ports tried integrating it into their build system, and I sort of moved on to other things. Unfortunately, right now I'm really swamped (my build system rewrite was prompted by WebKit exceeding internal, hardcoded, Bakefile limits, not by choice), but if a common location for the source files list could be decided upon, I really think a script would be a simple matter to write up (even in Perl :P ), and I think it would probably save developers time and reduce build breakages as well, which I think would add up to a lot of saved time for a lot of people over the long term. The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. So, does anyone think this would be a bad idea, or have any alternate suggestions on how to improve things? If not, is anyone willing to take the ball and run with it? I'd be willing to invest some more time in it, but my ability to commit to it over the next couple weeks at least would be very limited. Thanks, Kevin On Jul 9, 2009, at 9:23 PM, Dimitri Glazkov wrote: Dear WebKiteurs, In our persisting quest to be more like a common WebKit port, we have added Chromium build files to the tree this afternoon. These files are WebCore/WebCore.gypi and JavaScriptCore/JavaScriptCore.gypi and they are the GYP include files. As you may know, we use GYP (http://code.google.com/p/gyp) for generating MSVC, XCode, Scons, and even Make projects for Chromium. We are rather fond of GYP. Perhaps it is because it allows us to maintain one set of project files for all three Chromium platforms; or maybe because it lets us to do things like WebCore.gypi, where we can just mindlessly add all project files to the list and then use various neat GYP filtering facilities to narrow them down to sets that are relevant for specific builds; or maybe because it easifies creating cross-platform and cross-build-system targets, actions, and rules; or maybe because we just love saying Gyp! I don't truthfully know. What I do know is that starting now, we'd love for you to remember WebCore.gypi and JavaScriptCore.gypi when you are adding or removing files from WebCore or JavaScriptCore. Thanks to the power of GYP, you don't have worry whether this file will be used by Chromium: the rule is that if there's a project file change, it applies to the *.gypi files. The format is very simple and intuitive, a simple Python/JSONey list+dict. Thank you for your attention, men and women of WebKit. :DG ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
At one point, I started on a script (located in WebKitTools/Scripts/update-sources-list.py) whose idea was to take the list of common sources from one file, and make changes to MSVC, Qt, GTK, etc. build systems, so that WebKit devs need only add the file once, run the script, and commit the results. I got it as far as theoretically generating the Qt and GTK file lists, but I don't think anyone on those ports tried integrating it into their build system, and I sort of moved on to other things. I should add that I haven't felt enough pain (maybe my threshold is too high?) to attempt to write a script that updates all of the build systems when adding a new source file. Instead of regenerating parts of the build files, I though such a script should simply do in-place edits of each of the build files. My current thinking was that if you could provide an example source file that would be in the same group as the new source file, then you could embed less knowledge about each build format in the tool and just make it smart enough to add new lines for the source file for each build file. (For example, on Xcode project files, the only extra thing you'd need to do when adding a new file is to generate an ID for the new file(s) that didn't already exist in the current project file.) I think this would probably handle the most common cases. Dave - Original Message From: Kevin Ollivier kev...@theolliviers.com To: Dimitri Glazkov dglaz...@chromium.org Cc: Mark Mentovai m...@chromium.org; WebKit Development webkit-dev@lists.webkit.org Sent: Friday, July 10, 2009 8:52:57 AM Subject: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family) Hi Dimitri and all, Congrats on getting this into WebKit! Actually, I'm in the middle of a build system switch as well - to waf, a re-write of scons that removed many of the performance issues related to searching and calculating dependencies, and which has added some nice features as well (such as .app bundle building). I haven't completed the switch so I can't do preliminary benchmarks, but I'm pretty sure it's actually as fast or faster than make on *nix/Mac. (And BTW, it will probably make Apple devs happy to hear that I'm no longer using the horrid build-wxwebkit bash script to manage the build, but instead have integrated everything into build-webkit finally!) The main reason I bring this up, though, is because I think this sort of thing shows that we're unlikely to centralize our build systems any time soon, and I feel a bit sorry for the core devs who, as they accept new ports, are probably finding it more and more tedious, if not difficult, to make sure all the projects get updated by a change to the common parts of the build. They've been very helpful in terms of trying to keep the ports in shape when they make changes, and I feel like I'd like to do what I can to make it easier and faster to keep the other ports in sync. At one point, I started on a script (located in WebKitTools/Scripts/update-sources-list.py) whose idea was to take the list of common sources from one file, and make changes to MSVC, Qt, GTK, etc. build systems, so that WebKit devs need only add the file once, run the script, and commit the results. I got it as far as theoretically generating the Qt and GTK file lists, but I don't think anyone on those ports tried integrating it into their build system, and I sort of moved on to other things. Unfortunately, right now I'm really swamped (my build system rewrite was prompted by WebKit exceeding internal, hardcoded, Bakefile limits, not by choice), but if a common location for the source files list could be decided upon, I really think a script would be a simple matter to write up (even in Perl :P ), and I think it would probably save developers time and reduce build breakages as well, which I think would add up to a lot of saved time for a lot of people over the long term. The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. So, does anyone think this would be a bad idea, or have any alternate suggestions on how to improve things? If not, is anyone willing to take the ball and run with it? I'd be willing to invest some more time in it, but my ability to commit to it over the next couple weeks at least would be very limited. Thanks, Kevin On Jul 9, 2009, at 9:23 PM, Dimitri Glazkov wrote: Dear WebKiteurs, In our persisting quest to be more like a common WebKit port, we have added Chromium build
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Fri, Jul 10, 2009 at 11:06 AM, David Kilzerddkil...@webkit.org wrote: So, does anyone think this would be a bad idea, or have any alternate suggestions on how to improve things? What about adding support for waf to gyp? +1. I think GYP took the right step back from project files to a meta-format from which you could generate project files, and we have a pretty good record at Chromium to show that this was a good idea. The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. GYP generates very nice XCode projects -- they are always properly sorted, logically structured, and clean. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. I count 6 build systems in use currently (SCons support was added and removed within the last year): - Apple's Xcode - Apple's vcproj (also used by at least one other Windows port) - wx Bakefile (which will be replaced by waf soon) - Qt Qmake - GTK GNUMakefile - Google's gyp (added recently) Are any of the other ports going to switch to generating their build files using gyp? While I have no cycles to do it, I certainly hope that somebody would try. Switching to GYP for our port was an incredible experience. No more vcproj-muddling, needing to have a Mac to edit XCode files, and trying to understand how various build systems work. This approach also encourages you to unify the thinking about targets/actions/rules, and GYP has already solved this problem for pretty much all known platforms. I highly encourage you guys to check out Chromium and see it in action. The generation occurs silently at each update (which is something that could be hooked in to update-webkit script), it's accurate, and very seamless. :DG ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Fri, Jul 10, 2009 at 2:20 PM, Dimitri Glazkovdglaz...@chromium.org wrote: If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. GYP generates very nice XCode projects -- they are always properly sorted, logically structured, and clean. I think the difference worth highlighting here is that GYP doesn't *edit* Xcode projects. That is explicitly a non-goal. It generates new ones from scatch every time (but is smart enough not to touch anything if nothing would change). As a result, it doesn't need to know about all the various ways a project file could change from state A to state B. In Chromium, we don't have any Xcode files checked in, they're generated (again, from scratch, replacing whatever was there before) on every checkout. If you want to make local changes during the course of your development, that's fine (and helpful to test small tweaks), but the only canonical representation of the Xcode project file is in GYP. -- Mike Pinkerton Mac Weenie pinker...@google.com ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
To add to this, GYP also only generates files for _your_ platform on checkout. Even to the degree that on Windows, it would generate VS2008 projects if you had it installed, and VS2005 otherwise. To be precise, the selection is not automatic (you need an env variable) to avoid dual-VS installs ambiguity. :DG On Fri, Jul 10, 2009 at 11:32 AM, Mike Pinkertonpinker...@chromium.org wrote: On Fri, Jul 10, 2009 at 2:20 PM, Dimitri Glazkovdglaz...@chromium.org wrote: If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. GYP generates very nice XCode projects -- they are always properly sorted, logically structured, and clean. I think the difference worth highlighting here is that GYP doesn't *edit* Xcode projects. That is explicitly a non-goal. It generates new ones from scatch every time (but is smart enough not to touch anything if nothing would change). As a result, it doesn't need to know about all the various ways a project file could change from state A to state B. In Chromium, we don't have any Xcode files checked in, they're generated (again, from scratch, replacing whatever was there before) on every checkout. If you want to make local changes during the course of your development, that's fine (and helpful to test small tweaks), but the only canonical representation of the Xcode project file is in GYP. -- Mike Pinkerton Mac Weenie pinker...@google.com ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Having used both Chromium's .gyp solution and WebKit's .update-a-bunch-of-random-files solution, I'd like to add a huge +1 to anyone interested in doing this for their own port. Especially since the lack of trybots for webkit means that it's prohibitively difficult for people to actually test their build changes cross-platform currently. Although maybe I'm just extra-sensitive to this since dimich and I spent a chunk of time last night diagnosing and fixing some gtk test bustage due to missing files from the build config. -atw On Fri, Jul 10, 2009 at 11:37 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: To add to this, GYP also only generates files for _your_ platform on checkout. Even to the degree that on Windows, it would generate VS2008 projects if you had it installed, and VS2005 otherwise. To be precise, the selection is not automatic (you need an env variable) to avoid dual-VS installs ambiguity. :DG On Fri, Jul 10, 2009 at 11:32 AM, Mike Pinkertonpinker...@chromium.org wrote: On Fri, Jul 10, 2009 at 2:20 PM, Dimitri Glazkovdglaz...@chromium.org wrote: If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. GYP generates very nice XCode projects -- they are always properly sorted, logically structured, and clean. I think the difference worth highlighting here is that GYP doesn't *edit* Xcode projects. That is explicitly a non-goal. It generates new ones from scatch every time (but is smart enough not to touch anything if nothing would change). As a result, it doesn't need to know about all the various ways a project file could change from state A to state B. In Chromium, we don't have any Xcode files checked in, they're generated (again, from scratch, replacing whatever was there before) on every checkout. If you want to make local changes during the course of your development, that's fine (and helpful to test small tweaks), but the only canonical representation of the Xcode project file is in GYP. -- Mike Pinkerton Mac Weenie pinker...@google.com ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi David, On Jul 10, 2009, at 11:06 AM, David Kilzer wrote: So, does anyone think this would be a bad idea, or have any alternate suggestions on how to improve things? What about adding support for waf to gyp? If we're talking about Chromium, waf is much faster for building WebKit than scons is, so yeah, I would say adding the ability to use it instead of scons is a no-brainer. As for wx, while I'm open to the idea of using gyp to help others out or simplify matters for the core devs, for us I'm pretty sure it will mostly just be one more added layer to our build system. I've been using Bakefile for years now and while I thought the idea was great at first, I've found it mostly a pain to work with. Some of that is due to Bakefile itself, of course (like the latest bug...), but a lot of it is just the general approach and issues inherent with adding one more tool to the toolchain. Even though there's some sort of meta-tool generating the build projects, it only reduces, not removes, the need to understand and deal with the native project formats it outputs. Also, since the meta-tool can have bugs or missing features too, there's an additional point of failure added, and also added work in terms of maintaining the build system. Plus, any time you want to add a feature, you should look into how to implement it in the various formats that gyp supports to avoid hacks, a process which can sometimes require pretty creative solutions for less capable build tools. And there's also the fact that, if wx uses gyp, people are going to start asking for support for their favorite project format, which is just one more thing to deal with. In short, I think I've just come to be soured on the approach in general. I think I've come to agree with the point that the waf site makes about domain-specific languages not being a good idea for build systems, and when you use a tool like Bakefile or gyp, you not only need to become familiar with that tool's project file language, but also to some level with the formats of the project files that it outputs in order to debug or add new capabilities. With waf, all I ever really need to know is Python, which I already know, and the waf APIs, which I'm learning but for the most part have been pretty simple and straightforward to work with. The bottom line for me is that anything that has me spending less time on maintaining the build system and more on the actual product is a good thing in my book. ;-) Thanks, Kevin The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. I count 6 build systems in use currently (SCons support was added and removed within the last year): - Apple's Xcode - Apple's vcproj (also used by at least one other Windows port) - wx Bakefile (which will be replaced by waf soon) - Qt Qmake - GTK GNUMakefile - Google's gyp (added recently) Are any of the other ports going to switch to generating their build files using gyp? Dave - Original Message From: Kevin Ollivier kev...@theolliviers.com To: Dimitri Glazkov dglaz...@chromium.org Cc: Mark Mentovai m...@chromium.org; WebKit Development webkit-dev@lists.webkit.org Sent: Friday, July 10, 2009 8:52:57 AM Subject: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family) Hi Dimitri and all, Congrats on getting this into WebKit! Actually, I'm in the middle of a build system switch as well - to waf, a re-write of scons that removed many of the performance issues related to searching and calculating dependencies, and which has added some nice features as well (such as .app bundle building). I haven't completed the switch so I can't do preliminary benchmarks, but I'm pretty sure it's actually as fast or faster than make on *nix/Mac. (And BTW, it will probably make Apple devs happy to hear that I'm no longer using the horrid build-wxwebkit bash script to manage the build, but instead have integrated everything into build-webkit finally!) The main reason I bring this up, though, is because I think this sort of thing shows that we're unlikely to centralize our build systems any time soon, and I feel a bit sorry for the core devs who, as they accept new ports, are probably finding it more and more tedious, if not difficult
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Fri, Jul 10, 2009 at 1:14 PM, Kevin Ollivier kev...@theolliviers.comwrote: Hi David, On Jul 10, 2009, at 11:06 AM, David Kilzer wrote: So, does anyone think this would be a bad idea, or have any alternate suggestions on how to improve things? What about adding support for waf to gyp? If we're talking about Chromium, waf is much faster for building WebKit than scons is, so yeah, I would say adding the ability to use it instead of scons is a no-brainer. As for wx, while I'm open to the idea of using gyp to help others out or simplify matters for the core devs, for us I'm pretty sure it will mostly just be one more added layer to our build system. I've been using Bakefile for years now and while I thought the idea was great at first, I've found it mostly a pain to work with. Some of that is due to Bakefile itself, of course (like the latest bug...), but a lot of it is just the general approach and issues inherent with adding one more tool to the toolchain. Even though there's some sort of meta-tool generating the build projects, it only reduces, not removes, the need to understand and deal with the native project formats it outputs. Also, since the meta-tool can have bugs or missing features too, there's an additional point of failure added, and also added work in terms of maintaining the build system. Plus, any time you want to add a feature, you should look into how to implement it in the various formats that gyp supports to avoid hacks, a process which can sometimes require pretty creative solutions for less capable build tools. And there's also the fact that, if wx uses gyp, people are going to start asking for support for their favorite project format, which is just one more thing to deal with. In short, I think I've just come to be soured on the approach in general. I think I've come to agree with the point that the waf site makes about domain-specific languages not being a good idea for build systems, and when you use a tool like Bakefile or gyp, you not only need to become familiar with that tool's project file language, but also to some level with the formats of the project files that it outputs in order to debug or add new capabilities. With waf, all I ever really need to know is Python, which I already know, and the waf APIs, which I'm learning but for the most part have been pretty simple and straightforward to work with. The bottom line for me is that anything that has me spending less time on maintaining the build system and more on the actual product is a good thing in my book. ;-) Your argument makes sense if WebKit is only built for one platform/build-system. Unfortunately it's not. So the question is whether it's easier to maintain lots of different build files or whether it's easier to maintain one file that's perhaps a bit more abstract + the tool that interprets it. I agree that working directly in the project file for your platform is easier IF you're only developing for one platform. But once you start maintaining more than one project file, I think GYP is a big win. While we hope that others will update our GYPI file when they add/remove files, our build depends on it...so we'll definitely be keeping it in sync. So I think the question then becomes whether it's easier for you to maintain your new build format, or whether it's easier to make it a target for GYP. I honestly don't know what the answer is, but I think it's worth taking a closer look at GYP. J Thanks, Kevin The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. I count 6 build systems in use currently (SCons support was added and removed within the last year): - Apple's Xcode - Apple's vcproj (also used by at least one other Windows port) - wx Bakefile (which will be replaced by waf soon) - Qt Qmake - GTK GNUMakefile - Google's gyp (added recently) Are any of the other ports going to switch to generating their build files using gyp? Dave - Original Message From: Kevin Ollivier kev...@theolliviers.com To: Dimitri Glazkov dglaz...@chromium.org Cc: Mark Mentovai m...@chromium.org; WebKit Development webkit-dev@lists.webkit.org Sent: Friday, July 10, 2009 8:52:57 AM Subject: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family) Hi Dimitri
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
Hi Jeremy, On Jul 10, 2009, at 3:03 PM, Jeremy Orlow wrote: [snip] Your argument makes sense if WebKit is only built for one platform/ build-system. Unfortunately it's not. So the question is whether it's easier to maintain lots of different build files or whether it's easier to maintain one file that's perhaps a bit more abstract + the tool that interprets it. I agree that working directly in the project file for your platform is easier IF you're only developing for one platform. But once you start maintaining more than one project file, I think GYP is a big win. While we hope that others will update our GYPI file when they add/ remove files, our build depends on it...so we'll definitely be keeping it in sync. So I think the question then becomes whether it's easier for you to maintain your new build format, or whether it's easier to make it a target for GYP. I honestly don't know what the answer is, but I think it's worth taking a closer look at GYP. Actually, the big question in regards to having GYP reduce overall project maintenance is whether or not the other ports will adopt GYP. If the answer is yes, then it would be more compelling for wx to do so as well, assuming of course that someone implements a waf backend so that we can. :-) If the answer is no, though, then GYP is not reducing the amount of project maintenance work for any port other than Chromium, in which case there will still be 6 build systems (still 5 even if wx were to switch) and the problem I originally posed in this thread will still be an issue. In that case, the only way to really reduce the maintenance work of adding / removing files would again be to adopt a script like the one I suggested earlier. Speaking of which, with waf / Python I've actually almost completely automated the generation of the list of include dirs for my build projects based on the source files, so that I'm not maintaining them by hand anymore except for a few exceptions. And thinking about it, I bet I can even mostly automate the list of source files too, by having it grab all the .cpp files in the common dirs and special subdirs like curl and wx, then having some include / exclude filters to deal with a few special cases. :-) The question will be the performance hit, but at least with the includes it's not even noticeable, and I could always look into caching and changing only when you do an svn up or svn add/remove. That's the sort of flexibility and ability to quickly experiment and automate that scripts offer, and I suspect I will really miss that if I switch back to something like Bakefile / GYP. Thanks, Kevin J Thanks, Kevin The only format I'm not sure if we could automate reliably would be the XCode format (at least, on non-Mac machines), because IIUC we'd need some sort of parser for it, but Apple is the only port maintaining those directly IIUC, as now Chromium will be using GYP to update their XCode projects. If you hand-edit Xcode project files enough times you start to understand them, but you also may go insane in the process. I don't know that you have to write a full parser for it, but there is some non-trivial, minimal structure you have to understand to update the file properly. So even if we couldn't solve the XCode issue, that would drop it to updating two locations tops. I count 6 build systems in use currently (SCons support was added and removed within the last year): - Apple's Xcode - Apple's vcproj (also used by at least one other Windows port) - wx Bakefile (which will be replaced by waf soon) - Qt Qmake - GTK GNUMakefile - Google's gyp (added recently) Are any of the other ports going to switch to generating their build files using gyp? Dave - Original Message From: Kevin Ollivier kev...@theolliviers.com To: Dimitri Glazkov dglaz...@chromium.org Cc: Mark Mentovai m...@chromium.org; WebKit Development webkit-dev@lists.webkit.org Sent: Friday, July 10, 2009 8:52:57 AM Subject: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family) Hi Dimitri and all, Congrats on getting this into WebKit! Actually, I'm in the middle of a build system switch as well - to waf, a re-write of scons that removed many of the performance issues related to searching and calculating dependencies, and which has added some nice features as well (such as .app bundle building). I haven't completed the switch so I can't do preliminary benchmarks, but I'm pretty sure it's actually as fast or faster than make on *nix/Mac. (And BTW, it will probably make Apple devs happy to hear that I'm no longer using the horrid build-wxwebkit bash script to manage the build, but instead have integrated everything into build-webkit finally!) The main reason I bring this up, though, is because I think this sort of thing shows that we're unlikely to centralize our build
Re: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)
On Fri, Jul 10, 2009 at 6:26 PM, Kevin Ollivier kev...@theolliviers.comwrote: Hi Jeremy, On Jul 10, 2009, at 3:03 PM, Jeremy Orlow wrote: [snip] Your argument makes sense if WebKit is only built for one platform/build-system. Unfortunately it's not. So the question is whether it's easier to maintain lots of different build files or whether it's easier to maintain one file that's perhaps a bit more abstract + the tool that interprets it. I agree that working directly in the project file for your platform is easier IF you're only developing for one platform. But once you start maintaining more than one project file, I think GYP is a big win. While we hope that others will update our GYPI file when they add/remove files, our build depends on it...so we'll definitely be keeping it in sync. So I think the question then becomes whether it's easier for you to maintain your new build format, or whether it's easier to make it a target for GYP. I honestly don't know what the answer is, but I think it's worth taking a closer look at GYP. Actually, the big question in regards to having GYP reduce overall project maintenance is whether or not the other ports will adopt GYP. If the answer is yes, then it would be more compelling for wx to do so as well, assuming of course that someone implements a waf backend so that we can. :-) If the answer is no, though, then GYP is not reducing the amount of project maintenance work for any port other than Chromium, in which case there will still be 6 build systems (still 5 even if wx were to switch) and the problem I originally posed in this thread will still be an issue. In that case, the only way to really reduce the maintenance work of adding / removing files would again be to adopt a script like the one I suggested earlier. You're right. The burden of updating the GYPI file is less than the others, but it's still another file you need to change. But note that the maintainers of each project/platform/etc define which files to exclude. This is different from the other build systems where the person updating the file needs to decide whether it should go into the build or whether it should only be built with certain flags. Speaking of which, with waf / Python I've actually almost completely automated the generation of the list of include dirs for my build projects based on the source files, so that I'm not maintaining them by hand anymore except for a few exceptions. And thinking about it, I bet I can even mostly automate the list of source files too, by having it grab all the .cpp files in the common dirs and special subdirs like curl and wx, then having some include / exclude filters to deal with a few special cases. :-) The question will be the performance hit, but at least with the includes it's not even noticeable, and I could always look into caching and changing only when you do an svn up or svn add/remove. I suggest you take a look at GYP. Much of what you're talking about it already does. That's the sort of flexibility and ability to quickly experiment and automate that scripts offer, and I suspect I will really miss that if I switch back to something like Bakefile / GYP. Of course you're going to be able to experiment easier with something you wrote yourself. That said, the generator for scons is only 780 lines of code including a couple hundred of templates...so it doesn't seem like it'd be _that_ hard to figure it out. I'm not necessarily saying you guys should be using GYP, but I really think you should take a closer look at it. J ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
