Re: [webkit-dev] Build system update
On Fri, Mar 25, 2011 at 10:59 AM, Maciej Stachowiak m...@apple.com wrote: In addition to your comments, I also find the gyp syntax somewhat unpleasant. In particular, in .gypi lists of files to compile, ever entry is double-quoted, comma-separated, line-separated, and then grouped in multiple levels of braces. This is noisier than any of our current formats except the XML-based ones. It seems like a newline-separated list, possibly indented, should be sufficient. For .gyp files themselves, I admit I don't have a clear understanding of what they are actually doing, so it's hard for me to say if the syntax they use is good or not. I think in general the trick of making the files essentially Python code is (presumably) handy for Python experts but not easy to read for anyone else. It's just python directory literals, which is virtually identical to json. Nico But I am assuming syntax-level things can be fixed once we have things working for more ports, and doesn't necessarily need to be a showstopper to initial adoption. Perhaps after successfully deploying gyp for one additional port, the right next steps are to make sure the experience is smooth and enjoyable for WebKit hackers, before propagating it further. Regards, Maciej On Mar 24, 2011, at 9:49 PM, Brent Fulgham wrote: Hi Dimitri, LONG screed follows... On Mar 24, 2011, at 9:24 AM, Dimitri Glazkov wrote: \With the gyp conversion at this stage, we now have a possible solution to this problem. Given that there aren't any other viable alternatives in the present, please consider the most productive way of contributing: filing well-formulated bugs, blocking bug 55018 (https://bugs.webkit.org/showdependencytree.cgi?id=55018hide_resolved=1). We can then discuss these specific problems and adjust the shape of the solution accordingly. While I applaud the idea of a unified build system, my own experience trying to get gyp working has been a uniformly frustrating experience. Imagine for a moment that rather than being a Chromium developer, long schooled in the use of gyp and the various build components needed to support it, you are a lowly external developer who just wishes to build WebKit. Out of curiosity, I took a stab at playing with gyp again this evening: 1. The top Google hit for gyp build system takes me to an exhaustive specification of the gyp input file syntax format. If I click on the Project Home link, I find no information on downloading or installing the system. Presumably I can download the source and somehow generate the tool. 2. If I go to the User Documentation page (http://code.google.com/p/gyp/wiki/GypUserDocumentation), I am instructed on how to draft my own gyp files to build things. But why would I care when I can't even try the software out on an existing project, like WebKit? Once I download the gyp sources and 'build' them using the included setup.py, it's not clear what to do with them: link:Source brent$ gyp . Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp ./gyp Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp help Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ I kind of hate gyp! Compare that to the native Xcode projects, CMake files, or normal GNU Makefiles. These other solutions almost always JUST WORK, because if you can build any software on your machine, you must have used one of these options. I don't use gyp for anything, I don't have it installed, and its not clear how to go about using it. Please don't further raise the barrier of entry to new WebKit developers by adding yet another obscure build requirement to the system. :-( Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev
Re: [webkit-dev] Build system update
Hi Brent, I definitely agree that gyp is rather undocumented and kind of hard to use. It's nowhere near the level of documentation of CMake, let alone Xcode or GNU makefiles. Hopefully we can fix this in the near future. That said, I'd be kind of surprised if cmake was already installed on your system, or scons, or xcode, or nmake, or just about anything except for make on Linux mac ;) -- Dirk On Thu, Mar 24, 2011 at 9:49 PM, Brent Fulgham bfulg...@gmail.com wrote: Hi Dimitri, LONG screed follows... On Mar 24, 2011, at 9:24 AM, Dimitri Glazkov wrote: \With the gyp conversion at this stage, we now have a possible solution to this problem. Given that there aren't any other viable alternatives in the present, please consider the most productive way of contributing: filing well-formulated bugs, blocking bug 55018 (https://bugs.webkit.org/showdependencytree.cgi?id=55018hide_resolved=1). We can then discuss these specific problems and adjust the shape of the solution accordingly. While I applaud the idea of a unified build system, my own experience trying to get gyp working has been a uniformly frustrating experience. Imagine for a moment that rather than being a Chromium developer, long schooled in the use of gyp and the various build components needed to support it, you are a lowly external developer who just wishes to build WebKit. Out of curiosity, I took a stab at playing with gyp again this evening: 1. The top Google hit for gyp build system takes me to an exhaustive specification of the gyp input file syntax format. If I click on the Project Home link, I find no information on downloading or installing the system. Presumably I can download the source and somehow generate the tool. 2. If I go to the User Documentation page (http://code.google.com/p/gyp/wiki/GypUserDocumentation), I am instructed on how to draft my own gyp files to build things. But why would I care when I can't even try the software out on an existing project, like WebKit? Once I download the gyp sources and 'build' them using the included setup.py, it's not clear what to do with them: link:Source brent$ gyp . Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp ./gyp Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp help Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ I kind of hate gyp! Compare that to the native Xcode projects, CMake files, or normal GNU Makefiles. These other solutions almost always JUST WORK, because if you can build any software on your machine, you must have used one of these options. I don't use gyp for anything, I don't have it installed, and its not clear how to go about using it. Please don't further raise the barrier of entry to new WebKit developers by adding yet another obscure build requirement to the system. :-( Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
In addition to your comments, I also find the gyp syntax somewhat unpleasant. In particular, in .gypi lists of files to compile, ever entry is double-quoted, comma-separated, line-separated, and then grouped in multiple levels of braces. This is noisier than any of our current formats except the XML-based ones. It seems like a newline-separated list, possibly indented, should be sufficient. For .gyp files themselves, I admit I don't have a clear understanding of what they are actually doing, so it's hard for me to say if the syntax they use is good or not. I think in general the trick of making the files essentially Python code is (presumably) handy for Python experts but not easy to read for anyone else. But I am assuming syntax-level things can be fixed once we have things working for more ports, and doesn't necessarily need to be a showstopper to initial adoption. Perhaps after successfully deploying gyp for one additional port, the right next steps are to make sure the experience is smooth and enjoyable for WebKit hackers, before propagating it further. Regards, Maciej On Mar 24, 2011, at 9:49 PM, Brent Fulgham wrote: Hi Dimitri, LONG screed follows... On Mar 24, 2011, at 9:24 AM, Dimitri Glazkov wrote: \With the gyp conversion at this stage, we now have a possible solution to this problem. Given that there aren't any other viable alternatives in the present, please consider the most productive way of contributing: filing well-formulated bugs, blocking bug 55018 (https://bugs.webkit.org/showdependencytree.cgi?id=55018hide_resolved=1). We can then discuss these specific problems and adjust the shape of the solution accordingly. While I applaud the idea of a unified build system, my own experience trying to get gyp working has been a uniformly frustrating experience. Imagine for a moment that rather than being a Chromium developer, long schooled in the use of gyp and the various build components needed to support it, you are a lowly external developer who just wishes to build WebKit. Out of curiosity, I took a stab at playing with gyp again this evening: 1. The top Google hit for gyp build system takes me to an exhaustive specification of the gyp input file syntax format. If I click on the Project Home link, I find no information on downloading or installing the system. Presumably I can download the source and somehow generate the tool. 2. If I go to the User Documentation page (http://code.google.com/p/gyp/wiki/GypUserDocumentation), I am instructed on how to draft my own gyp files to build things. But why would I care when I can't even try the software out on an existing project, like WebKit? Once I download the gyp sources and 'build' them using the included setup.py, it's not clear what to do with them: link:Source brent$ gyp . Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp ./gyp Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp help Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ I kind of hate gyp! Compare that to the native Xcode projects, CMake files, or normal GNU Makefiles. These other solutions almost always JUST WORK, because if you can build any software on your machine, you must have used one of these options. I don't use gyp for anything, I don't have it installed, and its not clear how to go about using it. Please don't further raise the barrier of entry to new WebKit developers by adding yet another obscure build requirement to the system. :-( Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 8:46 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 23, 2011, at 3:33 AM, Adam Barth wrote: From my perspective, approach (2) is more desirable than checking in generated project files because approach (2) encapsulates Apple-internal build process to Apple folks, more specifically to the Apple folks who interact with the Apple-internal build system. Checking in generated project files, on the other hand, imposes a maintenance burden on all WebKit contributors. I believe Apple submissions generally happen with greater frequency than the rate at which new files are added to the project. Furthermore: When files are added to the project, the patch submitted must already run the tool to regenerate projects, and is already going to submit a patch, so the maintenance burden of the Xcode projects being checked in is low. But having to regenerate project files and then check them in on a branch adds extra steps, doing things that are not done in the normal course of development, and therefore may have bitrotted. Thanks, that's helpful information. Looking at http://trac.webkit.org/log/trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj, it looks like files are added to the project on the order of once a day. I don't think you are going to get Apple folks enthusiastic about switching to a build system that creates significantly more work for us, on the basis that it saves everyone else a small amount of work. For that matter, slowing down the pace of Apple engineers' development would be a bad thing for the project overall, not just for Apple. I certainly have no desire to slow down the pace of development for anyone. My overarching goal here is to increase developer productivity for everyone by removing the need to learn and correctly modify N build systems just to add, remove, or move a file. From this discussion, it sounds like folks are generally comfortable with replacing the existing xcodeproj files with ones generated by GYP. Before taking that step, I'm going to spend some more time polishing the GYP-generated xcodeproj files, hopefully addressing some of Mark's concerns. (I'm also told there are some scheduling issues, which I'm happy to accomodate.) I'm slightly skeptical that checking in generated files will scale as we unify more of the build systems, but we can cross that bridge when we come to it. Thanks everyone, Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Mar 24, 2011, at 12:28 AM, Adam Barth wrote: On Wed, Mar 23, 2011 at 8:46 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 23, 2011, at 3:33 AM, Adam Barth wrote: From my perspective, approach (2) is more desirable than checking in generated project files because approach (2) encapsulates Apple-internal build process to Apple folks, more specifically to the Apple folks who interact with the Apple-internal build system. Checking in generated project files, on the other hand, imposes a maintenance burden on all WebKit contributors. I believe Apple submissions generally happen with greater frequency than the rate at which new files are added to the project. Furthermore: When files are added to the project, the patch submitted must already run the tool to regenerate projects, and is already going to submit a patch, so the maintenance burden of the Xcode projects being checked in is low. But having to regenerate project files and then check them in on a branch adds extra steps, doing things that are not done in the normal course of development, and therefore may have bitrotted. Thanks, that's helpful information. Looking at http://trac.webkit.org/log/trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj, it looks like files are added to the project on the order of once a day. I slightly underestimated how often files are added to the project (no wonder our build has gotten so slow!), but the rate is similar. (Side note: it looks like 382 files have been added to the WebCore build over the past year, a ~25% growth rate - if this rate continues, we are in big trouble.) I don't think you are going to get Apple folks enthusiastic about switching to a build system that creates significantly more work for us, on the basis that it saves everyone else a small amount of work. For that matter, slowing down the pace of Apple engineers' development would be a bad thing for the project overall, not just for Apple. I certainly have no desire to slow down the pace of development for anyone. My overarching goal here is to increase developer productivity for everyone by removing the need to learn and correctly modify N build systems just to add, remove, or move a file. From this discussion, it sounds like folks are generally comfortable with replacing the existing xcodeproj files with ones generated by GYP. Before taking that step, I'm going to spend some more time polishing the GYP-generated xcodeproj files, hopefully addressing some of Mark's concerns. (I'm also told there are some scheduling issues, which I'm happy to accomodate.) I'm slightly skeptical that checking in generated files will scale as we unify more of the build systems, but we can cross that bridge when we come to it. Other ports probably do not have so many strange requirements driven by an external build system, so I'm not sure it will be as necessary. That being said the average WebKit developer updates or builds much more often than he or she changes a project file, so it may be the right tradeoff to save people the most total time. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
First, let me warm the tone of this thread a bit with an earnest salute to Adam for trying to solve the problem by actually doing _something_, rather than talking about it, as we've done for years now. My own contributions to the matter are small and insignificant in comparison. To recall, we do have a real and present problem. Adding files to the all the build systems (and _always_ missing at least one) sucks balls. Needing to use XCode to add new files to WebKit is silly. On the other hand, we have an otherwise well-established and rather smooth workflow and we don't want to give that up, either. With the gyp conversion at this stage, we now have a possible solution to this problem. Given that there aren't any other viable alternatives in the present, please consider the most productive way of contributing: filing well-formulated bugs, blocking bug 55018 (https://bugs.webkit.org/showdependencytree.cgi?id=55018hide_resolved=1). We can then discuss these specific problems and adjust the shape of the solution accordingly. :DG On Thu, Mar 24, 2011 at 1:30 AM, Maciej Stachowiak m...@apple.com wrote: On Mar 24, 2011, at 12:28 AM, Adam Barth wrote: On Wed, Mar 23, 2011 at 8:46 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 23, 2011, at 3:33 AM, Adam Barth wrote: From my perspective, approach (2) is more desirable than checking in generated project files because approach (2) encapsulates Apple-internal build process to Apple folks, more specifically to the Apple folks who interact with the Apple-internal build system. Checking in generated project files, on the other hand, imposes a maintenance burden on all WebKit contributors. I believe Apple submissions generally happen with greater frequency than the rate at which new files are added to the project. Furthermore: When files are added to the project, the patch submitted must already run the tool to regenerate projects, and is already going to submit a patch, so the maintenance burden of the Xcode projects being checked in is low. But having to regenerate project files and then check them in on a branch adds extra steps, doing things that are not done in the normal course of development, and therefore may have bitrotted. Thanks, that's helpful information. Looking at http://trac.webkit.org/log/trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj, it looks like files are added to the project on the order of once a day. I slightly underestimated how often files are added to the project (no wonder our build has gotten so slow!), but the rate is similar. (Side note: it looks like 382 files have been added to the WebCore build over the past year, a ~25% growth rate - if this rate continues, we are in big trouble.) I don't think you are going to get Apple folks enthusiastic about switching to a build system that creates significantly more work for us, on the basis that it saves everyone else a small amount of work. For that matter, slowing down the pace of Apple engineers' development would be a bad thing for the project overall, not just for Apple. I certainly have no desire to slow down the pace of development for anyone. My overarching goal here is to increase developer productivity for everyone by removing the need to learn and correctly modify N build systems just to add, remove, or move a file. From this discussion, it sounds like folks are generally comfortable with replacing the existing xcodeproj files with ones generated by GYP. Before taking that step, I'm going to spend some more time polishing the GYP-generated xcodeproj files, hopefully addressing some of Mark's concerns. (I'm also told there are some scheduling issues, which I'm happy to accomodate.) I'm slightly skeptical that checking in generated files will scale as we unify more of the build systems, but we can cross that bridge when we come to it. Other ports probably do not have so many strange requirements driven by an external build system, so I'm not sure it will be as necessary. That being said the average WebKit developer updates or builds much more often than he or she changes a project file, so it may be the right tradeoff to save people the most total time. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
Hi Dimitri, LONG screed follows... On Mar 24, 2011, at 9:24 AM, Dimitri Glazkov wrote: \With the gyp conversion at this stage, we now have a possible solution to this problem. Given that there aren't any other viable alternatives in the present, please consider the most productive way of contributing: filing well-formulated bugs, blocking bug 55018 (https://bugs.webkit.org/showdependencytree.cgi?id=55018hide_resolved=1). We can then discuss these specific problems and adjust the shape of the solution accordingly. While I applaud the idea of a unified build system, my own experience trying to get gyp working has been a uniformly frustrating experience. Imagine for a moment that rather than being a Chromium developer, long schooled in the use of gyp and the various build components needed to support it, you are a lowly external developer who just wishes to build WebKit. Out of curiosity, I took a stab at playing with gyp again this evening: 1. The top Google hit for gyp build system takes me to an exhaustive specification of the gyp input file syntax format. If I click on the Project Home link, I find no information on downloading or installing the system. Presumably I can download the source and somehow generate the tool. 2. If I go to the User Documentation page (http://code.google.com/p/gyp/wiki/GypUserDocumentation), I am instructed on how to draft my own gyp files to build things. But why would I care when I can't even try the software out on an existing project, like WebKit? Once I download the gyp sources and 'build' them using the included setup.py, it's not clear what to do with them: link:Source brent$ gyp . Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp ./gyp Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ gyp help Traceback (most recent call last): File /usr/local/bin/gyp, line 18, in module sys.exit(gyp.main(sys.argv[1:])) File /Library/Python/2.6/site-packages/gyp/__init__.py, line 374, in main '--depth as a workaround.' Exception: Could not automatically locate src directory. This is a temporary Chromium feature that will be removed. Use --depth as a workaround. link:Source brent$ I kind of hate gyp! Compare that to the native Xcode projects, CMake files, or normal GNU Makefiles. These other solutions almost always JUST WORK, because if you can build any software on your machine, you must have used one of these options. I don't use gyp for anything, I don't have it installed, and its not clear how to go about using it. Please don't further raise the barrier of entry to new WebKit developers by adding yet another obscure build requirement to the system. :-( Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On 2011-03-23, at 03:33, Adam Barth wrote: On Wed, Mar 23, 2011 at 12:22 AM, Mark Rowe mr...@apple.com wrote: On 2011-03-22, at 23:50, Adam Barth wrote: On Tue, Mar 22, 2011 at 8:14 PM, Mark Rowe mr...@apple.com wrote: Product names for targets are redundantly declared in the Xcode project when they're already defined in the .xcconfig files. My plan for this issue is to remove the product names from the xcconfig files once they're not needed by the generated xcodeproj files. That seems backwards. JavaScriptCore.xcconfig defines how to create JavaScriptCore.framework. The name of the framework (e.g., PRODUCT_NAME) is an important part of that, much like the install path, Info.plist file, etc. I don't see how we benefit from that information being spread around. The information about how to create JavaScriptCore.framework is already spread around to a certain extent. For example, JavaScriptCore.xcconfig does not contain the list of files that need to be built or information about the dependencies. I'm having a hard time seeing what the practical difference is between specifying the product name in the xcodeproj file or the xcconfig file. What's the argument for changing where it is specified? Is it simply because gyp doesn't know better? One area that I could use some help from one or more Apple folks is making sure that this build system integrates properly with Apple's internal build system. Based on my (incomplete!) understanding of Apple's internal build system, there are at least two options: 1) Apple's internal build system consumes Source in its entirety and builds a master WebKit.xcodeproj (or a Makefile), which abstracts further details about the build, such as how subsequent xcodeproj are created or how many libraries WebKit is factored into (e.g., letting us extract WTF from JavaScriptCore). Making that particular change is completely unrelated to GYP. My understanding is that it would solve the Apple internal build system integration issue because the Apple internal build system would transfer control to the master WebKit.xcodeproj, which could then invoke GYP to generate the remaining xcodeproj files. I'm not particularly keen on a solution that involves manually invoking xcodebuild to build projects. It becomes very complicated to ensure that the correct settings, including any overridden settings, make it down to the nested invocations of xcodebuild. I'm not sure I've correctly communicated how I envision this approach working. If approach (2) doesn't work out for whatever reason, I can build a mockup of how this would work, which will be more concrete. You'll need to provide more detail then because I cannot see how this would work in a manner that doesn't cause the problems I have noted. 2) For each submission to Apple's internal build system, we create a branch, generate xcodeproj files, and check them into the branch. Apple's internal build system can then svn export the branch and see xcodeproj files, as today. While this is certainly technically feasible, it would add a huge amount of overhead to the process of performing a submission. How often do you submit WebKit to the Apple internal build system? If that's sensitive information, I'm just looking for an order of magnitude (e.g., every revision, every hour, every day, every week). As a reference, the Chromium project creates one branch per day for daily builds. I don't see how the frequency is relevant. The relevancy arises from the observation that the overhead is proportional to the frequency. If we only submit WebKit to Apple's internal build system once a week, then this approach won't cause too much branch proliferation. If, on the other hand, we're submitting every revision, then we're talking about doubling or tripling the revision burn rate, which might not be desirable. The rate at which we use SVN revisions or the number of branches isn't of particular concern to me. The simple fact is that your proposal turns a single, trivial operation (create a tag) in to a sequence of more complicated operations (create a branch, check it out, run a script to generate a bunch of files, test that it builds, commit the new files, create a tag). What used to be a single command-line operation that took a second to run is now a series of operations that takes 5+ minutes to complete. In any case, I'm glad we've found a technically feasible solution. We've had at least one technically feasible solution from day zip: check in the generated project files. From my perspective, approach (2) is more desirable than checking in generated project files because approach (2) encapsulates Apple-internal build process to Apple folks, more specifically to the Apple folks who interact with the Apple-internal build system. Checking in generated project files, on the other hand, imposes a maintenance burden on all WebKit
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 3:33 AM, Adam Barth aba...@webkit.org wrote: On Wed, Mar 23, 2011 at 12:22 AM, Mark Rowe mr...@apple.com wrote: In any case, I'm glad we've found a technically feasible solution. We've had at least one technically feasible solution from day zip: check in the generated project files. From my perspective, approach (2) is more desirable than checking in generated project files because approach (2) encapsulates Apple-internal build process to Apple folks, more specifically to the Apple folks who interact with the Apple-internal build system. Checking in generated project files, on the other hand, imposes a maintenance burden on all WebKit contributors. Living with the chromium system of generating the files on the fly, I always find it bothersome when the slowest step (by far) of my sync is generating these project files. So I personally prefer checking in the generated files (and letting this delay be on the person making the change -- rather than distributing this generation step to everyone). (As Mark mentions) this seems to also have the benefit of disrupting people's workflow the least. dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 1:33 PM, David Levin le...@google.com wrote: On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth aba...@webkit.org wrote: $ time git svn rebase [... update my working copy from changes during lunch (four revisions) ...] real 1m10.316s user 0m8.194s sys 0m16.400s $ time ./Tools/Scripts/generate-project-files real 0m4.339s user 0m3.888s sys 0m0.269s which is about 6.1% overhead on an almost trivial update. We can also reduce this overhead to zero in many cases by detecting that we don't need to re-generate the project files if the inputs to the generation process haven't changed. It looks like you are using a slow mechanism for syncing. :) TLDR version: 8% overhead for updating 191 revisions. 49% overhead for trivial sync (3 revisions) -- in addition to the added complexity (when I sync or switch branches in git). Here's my results for updating 191 revisions (81605 through 81796). $ time git fetch time git svn rebase fetch, svn rebase times real 0m21.071s, 0m36.195s user 0m2.271s, 0m3.205s sys 0m0.497s, 0m9.428s total 57 seconds for a nontrivial update. $ time ./Tools/Scripts/generate-project-files real 0m4.642s user 0m4.243s sys 0m0.322s An 8% overhead on a non-trivial sync. For a trivial update of 3 revisions: real 0m3.490s, 0m6.017s user 0m0.932s, 0m2.266s sys 0m0.184s, 0m2.824s For a total of 9.5 seconds. The generate project files step remained the same, so this adds a 49% overhead for a trivial sync. It also adds time (and complications) whenever I switch branches in git. dave PS For chromium, my experience is $ time gclient runhooks --force real 0m42.459s user 0m29.236s sys 0m2.543s I don't know enough about chromium build system and gyp to know why it is an order of magnitude slower, but this overhead is noticeable and annoying there. I love the idea of a one project file system, but I have concerns about project file generation on sync becoming the norm in WebKit. fwiw, I checked how many gyp files were in chromium and it appeared within 20%. There seem to be four approaches to improving this situation: 1) Check in the generated project files. In this approach, only one person pays the cost of generating the project files. 2) Make generate-project-files run faster. I haven't looked at what's taking 4 seconds, but it seems entirely possible that we can improve the performance. 3) Avoid running generate-project-files when not necessary. This approach would improve the performance of the trivial sync, for example. 4) Run generate-project-files at time other than sync. For example, we could run generate-project-files as part of build-webkit, which already takes more more than 4 seconds, leading to a lower percentage overhead. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On 2011-03-23, at 13:49, Adam Barth wrote: On Wed, Mar 23, 2011 at 1:33 PM, David Levin le...@google.com wrote: On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth aba...@webkit.org wrote: $ time git svn rebase [... update my working copy from changes during lunch (four revisions) ...] real1m10.316s user0m8.194s sys 0m16.400s $ time ./Tools/Scripts/generate-project-files real0m4.339s user0m3.888s sys 0m0.269s which is about 6.1% overhead on an almost trivial update. We can also reduce this overhead to zero in many cases by detecting that we don't need to re-generate the project files if the inputs to the generation process haven't changed. It looks like you are using a slow mechanism for syncing. :) TLDR version: 8% overhead for updating 191 revisions. 49% overhead for trivial sync (3 revisions) -- in addition to the added complexity (when I sync or switch branches in git). Here's my results for updating 191 revisions (81605 through 81796). $ time git fetch time git svn rebase fetch, svn rebase times real 0m21.071s, 0m36.195s user 0m2.271s, 0m3.205s sys 0m0.497s, 0m9.428s total 57 seconds for a nontrivial update. $ time ./Tools/Scripts/generate-project-files real 0m4.642s user 0m4.243s sys 0m0.322s An 8% overhead on a non-trivial sync. For a trivial update of 3 revisions: real 0m3.490s, 0m6.017s user 0m0.932s, 0m2.266s sys 0m0.184s, 0m2.824s For a total of 9.5 seconds. The generate project files step remained the same, so this adds a 49% overhead for a trivial sync. It also adds time (and complications) whenever I switch branches in git. dave PS For chromium, my experience is $ time gclient runhooks --force real 0m42.459s user 0m29.236s sys 0m2.543s I don't know enough about chromium build system and gyp to know why it is an order of magnitude slower, but this overhead is noticeable and annoying there. I love the idea of a one project file system, but I have concerns about project file generation on sync becoming the norm in WebKit. fwiw, I checked how many gyp files were in chromium and it appeared within 20%. There seem to be four approaches to improving this situation: 1) Check in the generated project files. In this approach, only one person pays the cost of generating the project files. 2) Make generate-project-files run faster. I haven't looked at what's taking 4 seconds, but it seems entirely possible that we can improve the performance. 3) Avoid running generate-project-files when not necessary. This approach would improve the performance of the trivial sync, for example. 4) Run generate-project-files at time other than sync. For example, we could run generate-project-files as part of build-webkit, which already takes more more than 4 seconds, leading to a lower percentage overhead. People build much more frequently than the update, so 4 would have to be done in conjunction with 2 and/or 3. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 1:58 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-23, at 13:49, Adam Barth wrote: On Wed, Mar 23, 2011 at 1:33 PM, David Levin le...@google.com wrote: On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth aba...@webkit.org wrote: $ time git svn rebase [... update my working copy from changes during lunch (four revisions) ...] real 1m10.316s user 0m8.194s sys 0m16.400s $ time ./Tools/Scripts/generate-project-files real 0m4.339s user 0m3.888s sys 0m0.269s which is about 6.1% overhead on an almost trivial update. We can also reduce this overhead to zero in many cases by detecting that we don't need to re-generate the project files if the inputs to the generation process haven't changed. It looks like you are using a slow mechanism for syncing. :) TLDR version: 8% overhead for updating 191 revisions. 49% overhead for trivial sync (3 revisions) -- in addition to the added complexity (when I sync or switch branches in git). Here's my results for updating 191 revisions (81605 through 81796). $ time git fetch time git svn rebase fetch, svn rebase times real 0m21.071s, 0m36.195s user 0m2.271s, 0m3.205s sys 0m0.497s, 0m9.428s total 57 seconds for a nontrivial update. $ time ./Tools/Scripts/generate-project-files real 0m4.642s user 0m4.243s sys 0m0.322s An 8% overhead on a non-trivial sync. For a trivial update of 3 revisions: real 0m3.490s, 0m6.017s user 0m0.932s, 0m2.266s sys 0m0.184s, 0m2.824s For a total of 9.5 seconds. The generate project files step remained the same, so this adds a 49% overhead for a trivial sync. It also adds time (and complications) whenever I switch branches in git. dave PS For chromium, my experience is $ time gclient runhooks --force real 0m42.459s user 0m29.236s sys 0m2.543s I don't know enough about chromium build system and gyp to know why it is an order of magnitude slower, but this overhead is noticeable and annoying there. I love the idea of a one project file system, but I have concerns about project file generation on sync becoming the norm in WebKit. fwiw, I checked how many gyp files were in chromium and it appeared within 20%. There seem to be four approaches to improving this situation: 1) Check in the generated project files. In this approach, only one person pays the cost of generating the project files. 2) Make generate-project-files run faster. I haven't looked at what's taking 4 seconds, but it seems entirely possible that we can improve the performance. 3) Avoid running generate-project-files when not necessary. This approach would improve the performance of the trivial sync, for example. 4) Run generate-project-files at time other than sync. For example, we could run generate-project-files as part of build-webkit, which already takes more more than 4 seconds, leading to a lower percentage overhead. People build much more frequently than the update, so 4 would have to be done in conjunction with 2 and/or 3. Indeed. I suspect (2) and (3) are worth doing regardless. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 2:08 PM, Adam Barth aba...@webkit.org wrote: Indeed. I suspect (2) and (3) are worth doing regardless. AFAIK, gyp currently always regenerates everything and then compares the new versions to the old to see if it actually needs to touch the files on disk. This seems safe but slow. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 2:13 PM, Peter Kasting pkast...@chromium.org wrote: On Wed, Mar 23, 2011 at 2:08 PM, Adam Barth aba...@webkit.org wrote: Indeed. I suspect (2) and (3) are worth doing regardless. AFAIK, gyp currently always regenerates everything and then compares the new versions to the old to see if it actually needs to touch the files on disk. This seems safe but slow. My plan is to use a make-like system to stat the input files and compare the last-modified date with the output files. That approach seems to work well for other aspects of the build system. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Wed, Mar 23, 2011 at 1:33 PM, David Levin le...@google.com wrote: On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth aba...@webkit.org wrote: $ time git svn rebase [... update my working copy from changes during lunch (four revisions) ...] real 1m10.316s user 0m8.194s sys 0m16.400s $ time ./Tools/Scripts/generate-project-files real 0m4.339s user 0m3.888s sys 0m0.269s which is about 6.1% overhead on an almost trivial update. We can also reduce this overhead to zero in many cases by detecting that we don't need to re-generate the project files if the inputs to the generation process haven't changed. It looks like you are using a slow mechanism for syncing. :) TLDR version: 8% overhead for updating 191 revisions. 49% overhead for trivial sync (3 revisions) -- in addition to the added complexity (when I sync or switch branches in git). Here's my results for updating 191 revisions (81605 through 81796). $ time git fetch time git svn rebase fetch, svn rebase times real 0m21.071s, 0m36.195s user 0m2.271s, 0m3.205s sys 0m0.497s, 0m9.428s total 57 seconds for a nontrivial update. $ time ./Tools/Scripts/generate-project-files real 0m4.642s user 0m4.243s sys 0m0.322s An 8% overhead on a non-trivial sync. For a trivial update of 3 revisions: real 0m3.490s, 0m6.017s user 0m0.932s, 0m2.266s sys 0m0.184s, 0m2.824s For a total of 9.5 seconds. The generate project files step remained the same, so this adds a 49% overhead for a trivial sync. It also adds time (and complications) whenever I switch branches in git. dave PS For chromium, my experience is $ time gclient runhooks --force real 0m42.459s user 0m29.236s sys 0m2.543s I don't know enough about chromium build system and gyp to know why it is an order of magnitude slower, but this overhead is noticeable and annoying there. I love the idea of a one project file system, but I have concerns about project file generation on sync becoming the norm in WebKit. fwiw, I checked how many gyp files were in chromium and it appeared within 20%. FWIW, with Chromium, the regular 'gclient sync' regenerates the project files even if it didn't update anything, which is a step that could probably be eliminated. Also, as far as I know, there's been little effort to profile and optimize gyp, so I bet we could probably speed things up a fair amount. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Mar 23, 2011, at 3:33 AM, Adam Barth wrote: While this is certainly technically feasible, it would add a huge amount of overhead to the process of performing a submission. How often do you submit WebKit to the Apple internal build system? If that's sensitive information, I'm just looking for an order of magnitude (e.g., every revision, every hour, every day, every week). As a reference, the Chromium project creates one branch per day for daily builds. I don't see how the frequency is relevant. The relevancy arises from the observation that the overhead is proportional to the frequency. If we only submit WebKit to Apple's internal build system once a week, then this approach won't cause too much branch proliferation. If, on the other hand, we're submitting every revision, then we're talking about doubling or tripling the revision burn rate, which might not be desirable. In any case, I'm glad we've found a technically feasible solution. We've had at least one technically feasible solution from day zip: check in the generated project files. From my perspective, approach (2) is more desirable than checking in generated project files because approach (2) encapsulates Apple-internal build process to Apple folks, more specifically to the Apple folks who interact with the Apple-internal build system. Checking in generated project files, on the other hand, imposes a maintenance burden on all WebKit contributors. I believe Apple submissions generally happen with greater frequency than the rate at which new files are added to the project. Furthermore: When files are added to the project, the patch submitted must already run the tool to regenerate projects, and is already going to submit a patch, so the maintenance burden of the Xcode projects being checked in is low. But having to regenerate project files and then check them in on a branch adds extra steps, doing things that are not done in the normal course of development, and therefore may have bitrotted. I don't think you are going to get Apple folks enthusiastic about switching to a build system that creates significantly more work for us, on the basis that it saves everyone else a small amount of work. For that matter, slowing down the pace of Apple engineers' development would be a bad thing for the project overall, not just for Apple. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On 2011-03-22, at 19:16, Adam Barth wrote:
WebKit-folk,
With a bunch of help from Dimitri and Eric, we now have a functioning
GYP-based build for the Apple Mac port. There are still a couple bugs
we need to fix before this build system is ready for production
(they're filed as blocking Bug 55018, if you're curious). If you'd
like to try out the GYP-based build, you can invoke it (on a Mac) as
follows:
./Tools/Scripts/build-webkit --gyp
Is this expected to work? Running the command quoted above results in a build
failure in JavaScriptCore for me (jsc is unable to find UString.h).
If you just want to look at the generated project files, you can
generate them using this command:
./Tools/Scripts/generate-project-files
That script will create
Source/JavaScriptCore/gyp/JavaScriptCore.xcodeproj and
Source/WebCore/gyp/WebCore.xcodeproj (as well as a JavaScriptGlue
project file, which you should feel free to ignore).
Is it useful for me to file bugs about these generated projects?
A few issues I noticed while skimming the Xcode project files…
The source is all hidden two levels deep in the hierarchy (inside a group named
..?).
There's an oddly-named BUILT_PRODUCTS_DIR group.
There's a mysterious ForwardingHeaders group.
The .xcconfig files are in the Source group.
The script build phases are unnecessarily verbose ('Action …' or 'Postbuild
…' where only '…' is meaningful).
Product names for targets are redundantly declared in the Xcode project when
they're already defined in the .xcconfig files.
There are mysterious extra settings in the Xcode project that serve no apparent
purpose (EXECUTABLE_PREFIX, INTERMEDIATE_DIR, SHARED_INTERMEDIATE_DIR,
WRAPPER_PREFIX, ALWAYS_SEARCH_USER_PATHS?)
The default configuration is Debug rather than Production.
Updating Info.plist is done via a separate aggregate target rather than simply
being a script phase.
The main advantage of this build system over the existing Apple Mac
build system is that the list of files is shared between this build
system and the Chromium build system. That means one fewer location
to update when adding, removing, or renaming a file. Over time, we
should be able to share more between the build systems (because they
are written in a common language) and to share the list of files
between more ports.
One area that I could use some help from one or more Apple folks is
making sure that this build system integrates properly with Apple's
internal build system. Based on my (incomplete!) understanding of
Apple's internal build system, there are at least two options:
1) Apple's internal build system consumes Source in its entirety and
builds a master WebKit.xcodeproj (or a Makefile), which abstracts
further details about the build, such as how subsequent xcodeproj are
created or how many libraries WebKit is factored into (e.g., letting
us extract WTF from JavaScriptCore).
Making that particular change is completely unrelated to GYP.
2) For each submission to Apple's internal build system, we create a
branch, generate xcodeproj files, and check them into the branch.
Apple's internal build system can then svn export the branch and see
xcodeproj files, as today.
While this is certainly technically feasible, it would add a huge amount of
overhead to the process of performing a submission.
- Mark
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On Tue, Mar 22, 2011 at 8:14 PM, Mark Rowe mr...@apple.com wrote:
On 2011-03-22, at 19:16, Adam Barth wrote:
WebKit-folk,
With a bunch of help from Dimitri and Eric, we now have a functioning
GYP-based build for the Apple Mac port. There are still a couple bugs
we need to fix before this build system is ready for production
(they're filed as blocking Bug 55018, if you're curious). If you'd
like to try out the GYP-based build, you can invoke it (on a Mac) as
follows:
./Tools/Scripts/build-webkit --gyp
Is this expected to work? Running the command quoted above results in a
build failure in JavaScriptCore for me (jsc is unable to find UString.h).
Interesting. I just did a clean build (rm -rf WebKitBuild) with ToT
and was unable to reproduce. But I'd *love* to see the logs if you
still have them.
Given that this all started working a couple hours ago, I would expect
it's all pretty broken yet.
If you just want to look at the generated project files, you can
generate them using this command:
./Tools/Scripts/generate-project-files
That script will create
Source/JavaScriptCore/gyp/JavaScriptCore.xcodeproj and
Source/WebCore/gyp/WebCore.xcodeproj (as well as a JavaScriptGlue
project file, which you should feel free to ignore).
Is it useful for me to file bugs about these generated projects?
Eventually we will need to file bugs to fix all the issues folks find,
but for now it's probably easiest for Adam or I to file them as we go
based on your email just now. Of course you should feel welcome to
file any bugs and block https://bugs.webkit.org/show_bug.cgi?id=55018
when you do.
A few issues I noticed while skimming the Xcode project files…
The source is all hidden two levels deep in the hierarchy (inside a group
named ..?).
There's an oddly-named BUILT_PRODUCTS_DIR group.
There's a mysterious ForwardingHeaders group.
The .xcconfig files are in the Source group.
The script build phases are unnecessarily verbose ('Action …' or 'Postbuild
…' where only '…' is meaningful).
Product names for targets are redundantly declared in the Xcode project when
they're already defined in the .xcconfig files.
There are mysterious extra settings in the Xcode project that serve no
apparent purpose (EXECUTABLE_PREFIX, INTERMEDIATE_DIR,
SHARED_INTERMEDIATE_DIR, WRAPPER_PREFIX, ALWAYS_SEARCH_USER_PATHS?)
The default configuration is Debug rather than Production.
Updating Info.plist is done via a separate aggregate target rather than
simply being a script phase.
I'll let Adam reply to these inline.
The main advantage of this build system over the existing Apple Mac
build system is that the list of files is shared between this build
system and the Chromium build system. That means one fewer location
to update when adding, removing, or renaming a file. Over time, we
should be able to share more between the build systems (because they
are written in a common language) and to share the list of files
between more ports.
One area that I could use some help from one or more Apple folks is
making sure that this build system integrates properly with Apple's
internal build system. Based on my (incomplete!) understanding of
Apple's internal build system, there are at least two options:
1) Apple's internal build system consumes Source in its entirety and
builds a master WebKit.xcodeproj (or a Makefile), which abstracts
further details about the build, such as how subsequent xcodeproj are
created or how many libraries WebKit is factored into (e.g., letting
us extract WTF from JavaScriptCore).
Making that particular change is completely unrelated to GYP.
2) For each submission to Apple's internal build system, we create a
branch, generate xcodeproj files, and check them into the branch.
Apple's internal build system can then svn export the branch and see
xcodeproj files, as today.
While this is certainly technically feasible, it would add a huge amount of
overhead to the process of performing a submission.
Thanks for your feedback!
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Build system update
On 2011-03-22, at 21:28, Eric Seidel wrote: On Tue, Mar 22, 2011 at 8:14 PM, Mark Rowe mr...@apple.com wrote: On 2011-03-22, at 19:16, Adam Barth wrote: WebKit-folk, With a bunch of help from Dimitri and Eric, we now have a functioning GYP-based build for the Apple Mac port. There are still a couple bugs we need to fix before this build system is ready for production (they're filed as blocking Bug 55018, if you're curious). If you'd like to try out the GYP-based build, you can invoke it (on a Mac) as follows: ./Tools/Scripts/build-webkit --gyp Is this expected to work? Running the command quoted above results in a build failure in JavaScriptCore for me (jsc is unable to find UString.h). Interesting. I just did a clean build (rm -rf WebKitBuild) with ToT and was unable to reproduce. But I'd *love* to see the logs if you still have them. The logs aren't accessible to me right now. My suspicion is that it may be related to building on a case-sensitive filesystem. It should be relatively easy to test this theory using a case-sensitive read/write disk image. - Mark ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
