[webkit-dev] Towards a unified build system
As we've discussed off-and-on for a while, one of the larger pain-points in working on WebKit today is the large number of build systems. Simple things, like adding, removing, or renaming a file take a lot more effort than they should. There are a number of approaches we could take to addressing this problem. One of the major constraints is that folks would like to use familiar IDEs (e.g., Xcode, Visual Studio) to build and debug WebKit. That means we'll need some sort of meta-build system that can generate port-specific IDE projects. Looking around the web, there are a handful of systems that could do the job. Dimitri, Eric, and I have started experimenting with one such system, GYP. There are various trade-offs between GYP and other systems, such as CMake, that we've discussed on this list and at the WebKit meeting last year. Ultimately, I don't think it really matters which system we use. If someone else would like to take over the project, I'd be happy to let them choose whichever system they'd like to use. Our current plan is as follows: 1) Bring up a build of the Apple-Mac port using GYP. At this stage, we'll have two ways of building Apple-Mac, the existing xcodeproj build system and the fancy-pants GYP-generated xcodeproj build system. 2) Replace the manually created xcodeproj files with GYP generated xcodeproj files. Ideally, contributors wouldn't even know when this step occurred. They can continue to use the generated project files in exactly the same way they use the manually created project files today. This step will help us work out any kinks in the new project files. 3) Remove the xcodeproj files from svn.webkit.org and integrate the generation of xcodeproj files with the WebKit build / update scripts. At this point, contributors will notice that something has changed because there'll be one less build system to worry about keeping up to date! Once we finish these steps, we can repeat them for other build systems, systematically reducing the number of build systems we need to maintain. Currently, we have a fully working build of JavaScriptGlue and a mostly working build of JavaScriptCore using this system. Some folks have asked why we're bothering to convert JavaScriptGlue at all. Working with JavaScriptGlue has been helpful because it's like a reduced test case for WebCore. We might not end up actually changing its build system in the end (because it's only used by Apple-Mac and is deprecated), but bringing it up has taught us a lot about how GYP works, especially for WebKit. As we get further along towards these milestones, I'll send updates to webkit-dev, especially if contributors need to change what they're doing. If you'd like to follow along in more detail, you're welcome to CC yourself on the meta bug: https://bugs.webkit.org/show_bug.cgi?id=55018 Hopefully we'll be able to make this transition as smooth as possible. I'm sure there will be some rough spots, and I'd like to thank you all in advance for bearing with us. Thanks, Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
Looking around the web, there are a handful of systems that could do the job. Dimitri, Eric, and I have started experimenting with one such system, GYP. There are various trade-offs between GYP and other systems, such as CMake, that we've discussed on this list and at the WebKit meeting last year. Ultimately, I don't think it really matters which system we use. If someone else would like to take over the project, I'd be happy to let them choose whichever system they'd like to use. At the moment only Chromium is using GYP. CMake is used by EFL and WinCE. AFAIK RIM is using CMake and the Clutter port are using it too. If you look around in the web I can't find any non-google project using GYP. CMake is used in many projects (e.g. KDE) and has a bigger community around. I already finished most of the parts for the WinApple port based on CMake BTW: CMake has WinCE support, GYP not! So I really prefer CMake over GYP. If the Mac port is the most important one I can try to port it to CMake. Please let me know where I can help to make CMake our meta build system. - Patrick ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
Reducing the number of build systems is a face-meltingly worthy goal. I have one small question: On Mar 1, 2011, at 4:21 AM, Adam Barth wrote: 3) Remove the xcodeproj files from svn.webkit.org and integrate the generation of xcodeproj files with the WebKit build / update scripts. At this point, contributors will notice that something has changed because there'll be one less build system to worry about keeping up to date! Won't contributors who don't use update-webkit/build-webkit also notice something has changed, because they will no longer have any project files? (I ask this as a contributor who doesn't use update-webkit/build-webkit.) -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Mar 1, 2011, at 5:18 AM, Adam Roben wrote: Reducing the number of build systems is a face-meltingly worthy goal. I have one small question: On Mar 1, 2011, at 4:21 AM, Adam Barth wrote: 3) Remove the xcodeproj files from svn.webkit.org and integrate the generation of xcodeproj files with the WebKit build / update scripts. At this point, contributors will notice that something has changed because there'll be one less build system to worry about keeping up to date! Won't contributors who don't use update-webkit/build-webkit also notice something has changed, because they will no longer have any project files? (I ask this as a contributor who doesn't use update-webkit/build-webkit.) Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 5:18 AM, Adam Roben aro...@apple.com wrote: Reducing the number of build systems is a face-meltingly worthy goal. I have one small question: On Mar 1, 2011, at 4:21 AM, Adam Barth wrote: 3) Remove the xcodeproj files from svn.webkit.org and integrate the generation of xcodeproj files with the WebKit build / update scripts. At this point, contributors will notice that something has changed because there'll be one less build system to worry about keeping up to date! Won't contributors who don't use update-webkit/build-webkit also notice something has changed, because they will no longer have any project files? (I ask this as a contributor who doesn't use update-webkit/build-webkit.) Yes. Step 3 will certainly be visible to contributors. As we get closer to being able to switch over, we'll probably want to discuss the best way to approach that. One possibility is to have something like a ./configure script, as is common in many open-source projects. On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
Am 01.03.2011 um 18:33 schrieb Maciej Stachowiak: On Mar 1, 2011, at 5:18 AM, Adam Roben wrote: Reducing the number of build systems is a face-meltingly worthy goal. I have one small question: On Mar 1, 2011, at 4:21 AM, Adam Barth wrote: 3) Remove the xcodeproj files from svn.webkit.org and integrate the generation of xcodeproj files with the WebKit build / update scripts. At this point, contributors will notice that something has changed because there'll be one less build system to worry about keeping up to date! Won't contributors who don't use update-webkit/build-webkit also notice something has changed, because they will no longer have any project files? (I ask this as a contributor who doesn't use update-webkit/build-webkit.) Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. What about checking in the generated Xcodeproj? Would that be an option? Cheers, Niko ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 11:08 AM, Nikolas Zimmermann zimmerm...@physik.rwth-aachen.de wrote: Am 01.03.2011 um 18:33 schrieb Maciej Stachowiak: On Mar 1, 2011, at 5:18 AM, Adam Roben wrote: Reducing the number of build systems is a face-meltingly worthy goal. I have one small question: On Mar 1, 2011, at 4:21 AM, Adam Barth wrote: 3) Remove the xcodeproj files from svn.webkit.org and integrate the generation of xcodeproj files with the WebKit build / update scripts. At this point, contributors will notice that something has changed because there'll be one less build system to worry about keeping up to date! Won't contributors who don't use update-webkit/build-webkit also notice something has changed, because they will no longer have any project files? (I ask this as a contributor who doesn't use update-webkit/build-webkit.) Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. What about checking in the generated Xcodeproj? Would that be an option? We could quickly find ourselves in the position of generating build files for many different ports every time we submit patches, which could add several minutes to the submit process (which is even worse if you lose a race with another committer and have to update/rebase and try again). Generating on checkout seems like the best answer--if feasible. J ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works. (For those who have mentioned changing how submission to the central build system works, that is largely beyond the Safari team's control, so while it is possible in theory, it is probably not a practical option.) Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 1:13 PM, Jeremy Orlow jor...@chromium.org wrote: We could quickly find ourselves in the position of generating build files for many different ports every time we submit patches, which could add several minutes to the submit process The alternative is to get everyone to spend this time :). (Seems better to submit them with the change from this perspective.) (which is even worse if you lose a race with another committer and have to update/rebase and try again). I did a log (git whatchanged --diff-filter=AD --pretty=oneline --abbrev-commit) and it appeared that there were about ~4 check-ins per day that added or deleted files, so this race wouldn't be hit very often. dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. We can certainly use a Makefile to kick off the build. The GYP system itself is just a handful of BSD-licensed Python scripts, so I wouldn't expect any trouble with checking them into svn.webkit.org. I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works. It would be helpful to understand why that's the case. We can certain improve the xcodeproj backend until it generates project files that work with Apple's internal build system. The current xcode backend is only 1000 lines of Python. We could re-write it from scratch if needed. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 1:59 PM, David Levin le...@google.com wrote: On Tue, Mar 1, 2011 at 1:13 PM, Jeremy Orlow jor...@chromium.org wrote: We could quickly find ourselves in the position of generating build files for many different ports every time we submit patches, which could add several minutes to the submit process The alternative is to get everyone to spend this time :). (Seems better to submit them with the change from this perspective.) True. (which is even worse if you lose a race with another committer and have to update/rebase and try again). I did a log (git whatchanged --diff-filter=AD --pretty=oneline --abbrev-commit) and it appeared that there were about ~4 check-ins per day that added or deleted files, so this race wouldn't be hit very often. Good point. I was just thinking in terms of the times I get a ChangeLog conflict, but we should hit a conflict on build files much less often. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On 2011-03-01, at 14:06, Adam Barth wrote: On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. We can certainly use a Makefile to kick off the build. The GYP system itself is just a handful of BSD-licensed Python scripts, so I wouldn't expect any trouble with checking them into svn.webkit.org. I don't think driving the build via a Makefile is a desirable option. XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary. It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project. I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works. It would be helpful to understand why that's the case. We can certain improve the xcodeproj backend until it generates project files that work with Apple's internal build system. The current xcode backend is only 1000 lines of Python. We could re-write it from scratch if needed. The initial issues that I observed when looking at JavaScriptCore was the Xcode project is generated in to a subdirectory. This prevents xcodebuild installsrc from working correctly (only the files in directories below the Xcode project appear to be installed). I made some quick hacks to work around that and then ran in to the issue that the configurations in the generated JavaScriptCore.xcodeproj are not set up correctly (Production is no longer the default configuration, Production is based on DebugRelease.xcconfig rather than Base.xcconfig, etc.). There's also the fact that the generated Xcode project doesn't build the same content as the existing Xcode project (No jsc, testapi, minidom). At this point I gave up and stopped looking at it. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 2:18 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-01, at 14:06, Adam Barth wrote: On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. We can certainly use a Makefile to kick off the build. The GYP system itself is just a handful of BSD-licensed Python scripts, so I wouldn't expect any trouble with checking them into svn.webkit.org. I don't think driving the build via a Makefile is a desirable option. XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary. It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project. This issue relates only to the how and when to generate the projects phase. It seems we have at least two feasible solutions (using Makefiles and checking in generated projects), so it's mostly a matter of thinking of more clever solutions and making desirability trade-offs. I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works. It would be helpful to understand why that's the case. We can certain improve the xcodeproj backend until it generates project files that work with Apple's internal build system. The current xcode backend is only 1000 lines of Python. We could re-write it from scratch if needed. The initial issues that I observed when looking at JavaScriptCore was the Xcode project is generated in to a subdirectory. This prevents xcodebuild installsrc from working correctly (only the files in directories below the Xcode project appear to be installed). Correct. We're working in a subdirectory for the time begin to avoid disrupting other folks. Moving from a subdirectory to the main directory is easy once we're ready. I made some quick hacks to work around that and then ran in to the issue that the configurations in the generated JavaScriptCore.xcodeproj are not set up correctly (Production is no longer the default configuration, Production is based on DebugRelease.xcconfig rather than Base.xcconfig, etc.). There's also the fact that the generated Xcode project doesn't build the same content as the existing Xcode project (No jsc, testapi, minidom). At this point I gave up and stopped looking at it. Thanks for looking at the projects in detail. The issues you're running into are all things that we know we need to fix. We're still in the early experimental stages of step 1, which is why the generated projects aren't complete yet. At this point, I'm just happy they build anything at all! None of these issues appear insurmountable. Thanks, Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Towards a unified build system
On 2011-03-01, at 14:29, Adam Barth wrote: On Tue, Mar 1, 2011 at 2:18 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-01, at 14:06, Adam Barth wrote: On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. We can certainly use a Makefile to kick off the build. The GYP system itself is just a handful of BSD-licensed Python scripts, so I wouldn't expect any trouble with checking them into svn.webkit.org. I don't think driving the build via a Makefile is a desirable option. XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary. It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project. This issue relates only to the how and when to generate the projects phase. It seems we have at least two feasible solutions (using Makefiles and checking in generated projects), so it's mostly a matter of thinking of more clever solutions and making desirability trade-offs. Given that you've listed the Makefile as a feasible solution after what I've said above I can only assume that I misunderstand how you intend for it to be used. Can you expand on how you see that working? I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works. It would be helpful to understand why that's the case. We can certain improve the xcodeproj backend until it generates project files that work with Apple's internal build system. The current xcode backend is only 1000 lines of Python. We could re-write it from scratch if needed. The initial issues that I observed when looking at JavaScriptCore was the Xcode project is generated in to a subdirectory. This prevents xcodebuild installsrc from working correctly (only the files in directories below the Xcode project appear to be installed). Correct. We're working in a subdirectory for the time begin to avoid disrupting other folks. Moving from a subdirectory to the main directory is easy once we're ready. Until this particular issue is addressed it's going to be very difficult for anyone at Apple to evaluate the correctness of the generated projects. I made some quick hacks to work around that and then ran in to the issue that the configurations in the generated JavaScriptCore.xcodeproj are not set up correctly (Production is no longer the default configuration, Production is based on DebugRelease.xcconfig rather than Base.xcconfig, etc.). There's also the fact that the generated Xcode project doesn't build the same content as the existing Xcode project (No jsc, testapi, minidom). At this point I gave up and stopped looking at it. Thanks for looking at the projects in detail. The issues you're running into are all things that we know we need to fix. We're still in the early experimental stages of step 1, which is why the generated projects aren't complete yet. At this point, I'm just happy they build anything at all! One other thing that I just remembered is that the built framework doesn't include any headers! I suspected that the issues may be known, but it was suggested that I mention the issues I observed in case they were things that you weren't aware of. There are other issues that appear to be less specific to the particular JavaScriptCore project and more specific to gyp in general: 1) The generated Xcode project includes a number of configuration settings that are redundant given those
Re: [webkit-dev] Towards a unified build system
On Tue, Mar 1, 2011 at 2:46 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-01, at 14:29, Adam Barth wrote: On Tue, Mar 1, 2011 at 2:18 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-01, at 14:06, Adam Barth wrote: On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. We can certainly use a Makefile to kick off the build. The GYP system itself is just a handful of BSD-licensed Python scripts, so I wouldn't expect any trouble with checking them into svn.webkit.org. I don't think driving the build via a Makefile is a desirable option. XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary. It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project. This issue relates only to the how and when to generate the projects phase. It seems we have at least two feasible solutions (using Makefiles and checking in generated projects), so it's mostly a matter of thinking of more clever solutions and making desirability trade-offs. Given that you've listed the Makefile as a feasible solution after what I've said above I can only assume that I misunderstand how you intend for it to be used. Can you expand on how you see that working? Oh, you wrote that it might not be desirable, not that it was infeasible. I'd classify the status quo as undesirable as well. At some point, we should have a higher bandwidth discussion about how to make this work. I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works. It would be helpful to understand why that's the case. We can certain improve the xcodeproj backend until it generates project files that work with Apple's internal build system. The current xcode backend is only 1000 lines of Python. We could re-write it from scratch if needed. The initial issues that I observed when looking at JavaScriptCore was the Xcode project is generated in to a subdirectory. This prevents xcodebuild installsrc from working correctly (only the files in directories below the Xcode project appear to be installed). Correct. We're working in a subdirectory for the time begin to avoid disrupting other folks. Moving from a subdirectory to the main directory is easy once we're ready. Until this particular issue is addressed it's going to be very difficult for anyone at Apple to evaluate the correctness of the generated projects. I made some quick hacks to work around that and then ran in to the issue that the configurations in the generated JavaScriptCore.xcodeproj are not set up correctly (Production is no longer the default configuration, Production is based on DebugRelease.xcconfig rather than Base.xcconfig, etc.). There's also the fact that the generated Xcode project doesn't build the same content as the existing Xcode project (No jsc, testapi, minidom). At this point I gave up and stopped looking at it. Thanks for looking at the projects in detail. The issues you're running into are all things that we know we need to fix. We're still in the early experimental stages of step 1, which is why the generated projects aren't complete yet. At this point, I'm just happy they build anything at all! One other thing that I just remembered is that the built framework doesn't include any headers! Yes. That's actually the bug I'm working on right now. Mark Mentovai recently added support to GYP for framework
Re: [webkit-dev] Towards a unified build system
On 2011-03-01, at 15:04, Adam Barth wrote: On Tue, Mar 1, 2011 at 2:46 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-01, at 14:29, Adam Barth wrote: On Tue, Mar 1, 2011 at 2:18 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-01, at 14:06, Adam Barth wrote: On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2011, at 11:07 AM, Adam Barth wrote: On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak m...@apple.com wrote: Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards. I don't fully understand the constraints of Apple's central build system. It might be worthwhile talking over the constraints with someone who's an expert. If the requirements are that the entry point to the build system is an xcodeproj, we can create an xcodeproj with an action that creates the other xcodeproj files and then defers to the generated projects as dependencies. I haven't tested this approach, but I suspect we'll find something that satisfies the Apple-internal constraints. It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X. We can certainly use a Makefile to kick off the build. The GYP system itself is just a handful of BSD-licensed Python scripts, so I wouldn't expect any trouble with checking them into svn.webkit.org. I don't think driving the build via a Makefile is a desirable option. XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary. It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project. This issue relates only to the how and when to generate the projects phase. It seems we have at least two feasible solutions (using Makefiles and checking in generated projects), so it's mostly a matter of thinking of more clever solutions and making desirability trade-offs. Given that you've listed the Makefile as a feasible solution after what I've said above I can only assume that I misunderstand how you intend for it to be used. Can you expand on how you see that working? Oh, you wrote that it might not be desirable, not that it was infeasible. I'd classify the status quo as undesirable as well. At some point, we should have a higher bandwidth discussion about how to make this work. It was something of an understatement when I said it was not a desirable option. It's clearly not ok to make it impossible to use sometimes necessary features of the build system. I'll try and be clearer in the future. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
