[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 4:56 AM, Mark Mentovaim...@chromium.org wrote: An objdir-ish solution would make sense, except the native build systems we use don't really work in terms of objdirs, so we'd just wind up generating a parallel directory structure with nothing but xcode projects, Makefiles, and other similar junk. That's right. (This is how autoconf projects work when you do an out of tree build.) Plus, it means that we can't supply a pre-gypped srcdir tarball, which can frustrate the snot out of casual developers. Sure you can; just include the build directory in the tarball. But gyp does the equivalent of 'configure', so its output depends on the system you're running it on, and it might not always be a good idea to use pre-gyp'd tarballs. I think this already bit somebody. I'm not arguing for out-of-tree builds anymore, since y'all feel so strongly about it, but if we ever decide to support cross-compiling, $objdir might start making more sense. Also, we'd have to do something like http://www.python.org/dev/peps/pep-0304/ or do 'make clean' to keep those pesky .pyc files from causing the same problems, anyway. - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Wed, Jul 22, 2009 at 9:56 PM, Mark Mentovai m...@chromium.org wrote: Dan Kegel wrote: No. Generating files is the point of gyp. Nothing says they have to be in the source tree. Out of all of the proposals out there, the only ones that make any sense to my software-developin' mind are the ones that keep the build files in the source tree. I find it much easier to type make or xcodebuild or whatever without having to keep track of where I am. Dumping all of our build files into a single directory without any real association to the source they build sucks. An objdir-ish solution would make sense, except the native build systems we use don't really work in terms of objdirs, so we'd just wind up generating a parallel directory structure with nothing but xcode projects, Makefiles, and other similar junk. Plus, it means that we can't supply a pre-gypped srcdir tarball, which can frustrate the snot out of casual developers. Maybe this could be an optional mode? Perhaps we could use this mode on the build slaves? Or... maybe for the benefit of the build slaves, GYP could write out a log of all generated files, and then as a first step, the build slaves could delete all generated files? My point is that maybe we can solve a large part of the pain here with something that addresses the needs of the build slaves only. -Darin The only change that makes any amount of sense to me would be to move all GYP-generated files one directory below where they are now, by introducing new build-file directories. That'd ease the svn:ignore burden. But those files are still in the source tree. One of the intentional properties of our current GYP setup is that the generated files are as good as checked-in. GYP generation looks and feels just like another file in the working copy being updated. I think that this transparency is a really good thing. You've pointed out a legitimate flaw, so perhaps we should focus on other solutions to that problem. There are ways to fix this that don't involve moving files around. Mark --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 7:26 AM, Darin Fisher da...@chromium.org wrote: On Wed, Jul 22, 2009 at 9:56 PM, Mark Mentovai m...@chromium.org wrote: Dan Kegel wrote: No. Generating files is the point of gyp. Nothing says they have to be in the source tree. Out of all of the proposals out there, the only ones that make any sense to my software-developin' mind are the ones that keep the build files in the source tree. I find it much easier to type make or xcodebuild or whatever without having to keep track of where I am. Dumping all of our build files into a single directory without any real association to the source they build sucks. An objdir-ish solution would make sense, except the native build systems we use don't really work in terms of objdirs, so we'd just wind up generating a parallel directory structure with nothing but xcode projects, Makefiles, and other similar junk. Plus, it means that we can't supply a pre-gypped srcdir tarball, which can frustrate the snot out of casual developers. Maybe this could be an optional mode? Perhaps we could use this mode on the build slaves? I forgot to mention that I would personally much prefer this mode than having generated files in my checked out source tree. I guess I'm very accustomed to an objdir solution from my Mozilla days (autoconf). Perhaps this objdir could even be married to the current output dirs used by the builds (xcodebuild, sconsbuild, chrome/Debug, etc.). -Darin Or... maybe for the benefit of the build slaves, GYP could write out a log of all generated files, and then as a first step, the build slaves could delete all generated files? My point is that maybe we can solve a large part of the pain here with something that addresses the needs of the build slaves only. -Darin The only change that makes any amount of sense to me would be to move all GYP-generated files one directory below where they are now, by introducing new build-file directories. That'd ease the svn:ignore burden. But those files are still in the source tree. One of the intentional properties of our current GYP setup is that the generated files are as good as checked-in. GYP generation looks and feels just like another file in the working copy being updated. I think that this transparency is a really good thing. You've pointed out a legitimate flaw, so perhaps we should focus on other solutions to that problem. There are ways to fix this that don't involve moving files around. Mark --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Wed, Jul 22, 2009 at 9:15 PM, Mark Larson (Google) m...@chromium.orgwrote: On Wed, Jul 22, 2009 at 20:27, Dan Kegel d...@kegel.com wrote: Stop me if you've heard this one before. Today, a new directory was added to the source tree, and shortly thereafter was reverted. Should have been no problem, but... because the new directory contained a gyp file, a file was generated in that directory, and svn couldn't delete the directory when the revert landed. This caused a build breakage, and I gather from nsylvain's comments that this wasn't the first time this has happened. At some point soon, it'd be good to teach gyp not to generate files in the source tree. Or maybe teach gclient how to deal forcefully with directories with no files under version control. Generating files in the source tree is kinda the point of gyp. gclient has nothing to do with this case. svn update src/ was trying to add a directory called src/bleh, but src/bleh already existed, so svn update failed. The only thing gclient might want to do, is clobber your tree, and try again. We used to have stuff like that, but they are all disabled AFAIK, I lost too many important files because gclient wanted to be nice and clean up my machine. What we currently do, on the buildbot side, is clobber the source tree when something goes wrong with gclient. In this case it mostly worked, except on about 8 machines, where a file was locked. (No idea what it was, procexp was not helpful, closing all apps either. I had to reboot the machine) To answer dirk : Well, it needs manual cleanup. Ask a trooper to connect to each machine and fix them. (Or ask on irc, at this point about half of the team knows the passwords to the machines :| ). I guess we should formalize it more : When in doubt, find someone to reboot the machine. It does not need any special skills, the machine will auto login and restart all the buildbot stuff automatically. Nicolas --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 2:28 PM, Darin Fisherda...@chromium.org wrote: An objdir-ish solution would make sense, except the native build systems we use don't really work in terms of objdirs, so we'd just wind up generating a parallel directory structure with nothing but xcode projects, Makefiles, and other similar junk. Plus, it means that we can't supply a pre-gypped srcdir tarball, which can frustrate the snot out of casual developers. Maybe this could be an optional mode? Perhaps we could use this mode on the build slaves? I forgot to mention that I would personally much prefer this mode than having generated files in my checked out source tree. I guess I'm very accustomed to an objdir solution from my Mozilla days (autoconf). Perhaps this objdir could even be married to the current output dirs used by the builds (xcodebuild, sconsbuild, chrome/Debug, etc.). Indeed. Here's my christmas wish: I'd like gyp and chrome to support cross-compilation, so that I could (on my Linux box) kick off distcc-accelerated builds for all three platforms, each one going into a separate objdir. It's not as farfetched as it sounds; Mac can already be built on Linux, I hear, and a Windows build on Linux is within reach (just have to remove the last bits of ATL and apply some elbow grease, I think). That desire is part of what makes me want gyp output to go into objdir -- because gyp's output is going to depend on which platform you're building for. - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 8:53 AM, Evan Martin e...@chromium.org wrote: On Thu, Jul 23, 2009 at 7:47 AM, Nicolas Sylvainnsylv...@chromium.org wrote: gclient has nothing to do with this case. svn update src/ was trying to add a directory called src/bleh, but src/bleh already existed, so svn update failed. Sorry to back-seat drive, but can't you do something like svn status | xargs rm -rf before syncing? svn status does not show files that are svn:ignore'd, like all generated files should be. But, that's a good point, the directory might not be ignored, so that can work. We already have code to do that and we do it on the try bots. This is really slow though, can take 1-2 minutes to run before each sync. (In git it's a builtin: git clean -f to delete all files that aren't officially part of the repository.) The only thing gclient might want to do, is clobber your tree, and try again. We used to have stuff like that, but they are all disabled AFAIK, I lost too many important files because gclient wanted to be nice and clean up my machine. I don't understand this paragraph, which might explain why my above proposal is dumb. An example that bit me in the past: You have a DEPS file that fetches src/blah. you add new code in src/blah, but before you commit, someone deletes the line src/blah in the DEPS file. At the time the default behavior was : rm -rf src/blah, which killed all my new code in there. Maybe the non-versionned files are important. We can't kill them. (We could have a mode for buildbot only that is more aggressive though, since we are not likely to have important non-versionned files there). Nicolas --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 12:00 PM, Dan Kegel d...@kegel.com wrote: On Thu, Jul 23, 2009 at 2:28 PM, Darin Fisherda...@chromium.org wrote: An objdir-ish solution would make sense, except the native build systems we use don't really work in terms of objdirs, so we'd just wind up generating a parallel directory structure with nothing but xcode projects, Makefiles, and other similar junk. Plus, it means that we can't supply a pre-gypped srcdir tarball, which can frustrate the snot out of casual developers. Maybe this could be an optional mode? Perhaps we could use this mode on the build slaves? I forgot to mention that I would personally much prefer this mode than having generated files in my checked out source tree. I guess I'm very accustomed to an objdir solution from my Mozilla days (autoconf). Perhaps this objdir could even be married to the current output dirs used by the builds (xcodebuild, sconsbuild, chrome/Debug, etc.). Indeed. Here's my christmas wish: I'd like gyp and chrome to support cross-compilation, so that I could (on my Linux box) kick off distcc-accelerated builds for all three platforms, each one going into a separate objdir. It's not as farfetched as it sounds; Mac can already be built on Linux, I hear, and a Windows build on Linux is within reach (just have to remove the last bits of ATL and apply some elbow grease, I think). A mac build still needs a Mac to drive it. We simply did the work to distribute the compiling to linux also. But the build is driven by xcode, and you need a bunch of other mac tools for the scripts to be able to invoke. Window is probably in a similar boat, you'd need the tools for compiling resources, etc. That desire is part of what makes me want gyp output to go into objdir -- because gyp's output is going to depend on which platform you're building for. At the moment, each platform is generating a native for a different build system. So the xcode project can live on disk next to the windows solution, and the makefile/scons files too. As far as object dirs go, i believe all three now already write the output into a different directory (supporting both debug and release in there). So the configure world need of different output dirs isn't the exact same situation. TVL - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 9:07 AM, Thomas Van Lententhoma...@chromium.org wrote: Here's my christmas wish: I'd like gyp and chrome to support cross-compilation, so that I could (on my Linux box) kick off distcc-accelerated builds for all three platforms, each one going into a separate objdir. It's not as farfetched as it sounds; Mac can already be built on Linux, I hear, and a Windows build on Linux is within reach (just have to remove the last bits of ATL and apply some elbow grease, I think). A mac build still needs a Mac to drive it. We simply did the work to distribute the compiling to linux also. But the build is driven by xcode, and you need a bunch of other mac tools for the scripts to be able to invoke. OK then, let's make a Mac drive all three builds :-) Window is probably in a similar boat, you'd need the tools for compiling resources, etc. Those are all in. mingw32 is quite complete. At the moment, each platform is generating a native for a different build system. But in the cross-compiling scenario, we'd have gyp generate makefiles for everything. - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
The other thing to remember is the buildbot scripts and a bunch of build scripts on all platforms are full of assumptions about the relationships between projects and the tree. :( TVL On Thu, Jul 23, 2009 at 12:11 PM, Dan Kegel d...@kegel.com wrote: On Thu, Jul 23, 2009 at 9:07 AM, Thomas Van Lententhoma...@chromium.org wrote: Here's my christmas wish: I'd like gyp and chrome to support cross-compilation, so that I could (on my Linux box) kick off distcc-accelerated builds for all three platforms, each one going into a separate objdir. It's not as farfetched as it sounds; Mac can already be built on Linux, I hear, and a Windows build on Linux is within reach (just have to remove the last bits of ATL and apply some elbow grease, I think). A mac build still needs a Mac to drive it. We simply did the work to distribute the compiling to linux also. But the build is driven by xcode, and you need a bunch of other mac tools for the scripts to be able to invoke. OK then, let's make a Mac drive all three builds :-) Window is probably in a similar boat, you'd need the tools for compiling resources, etc. Those are all in. mingw32 is quite complete. At the moment, each platform is generating a native for a different build system. But in the cross-compiling scenario, we'd have gyp generate makefiles for everything. - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 9:15 AM, Thomas Van Lententhoma...@chromium.org wrote: The other thing to remember is the buildbot scripts and a bunch of build scripts on all platforms are full of assumptions about the relationships between projects and the tree. :( Well, yes. I didn't say it would be easy. I don't even think it's especially likely, but I'd kind of like to avoid changes that make it harder, and I'd like to encourage other people to keep it in mind as they hack on the build machinery. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 8:53 AM, Evan Martin e...@chromium.org wrote: On Thu, Jul 23, 2009 at 7:47 AM, Nicolas Sylvainnsylv...@chromium.org wrote: gclient has nothing to do with this case. svn update src/ was trying to add a directory called src/bleh, but src/bleh already existed, so svn update failed. Sorry to back-seat drive, but can't you do something like svn status | xargs rm -rf before syncing? (In git it's a builtin: git clean -f to delete all files that aren't officially part of the repository.) I think this could work provided we set svn:ignore properly for directories that gclient inserted into src/. -Darin The only thing gclient might want to do, is clobber your tree, and try again. We used to have stuff like that, but they are all disabled AFAIK, I lost too many important files because gclient wanted to be nice and clean up my machine. I don't understand this paragraph, which might explain why my above proposal is dumb. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 9:01 AM, Nicolas Sylvainnsylv...@chromium.org wrote: On Thu, Jul 23, 2009 at 8:53 AM, Evan Martin e...@chromium.org wrote: On Thu, Jul 23, 2009 at 7:47 AM, Nicolas Sylvainnsylv...@chromium.org wrote: gclient has nothing to do with this case. svn update src/ was trying to add a directory called src/bleh, but src/bleh already existed, so svn update failed. Sorry to back-seat drive, but can't you do something like svn status | xargs rm -rf before syncing? svn status does not show files that are svn:ignore'd, like all generated files should be. Hmm. In creating a clean client, would it make sense to delete all files which exist and are svn:ignore'd? Just as a matter of course? -scott --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Wed, Jul 22, 2009 at 20:27, Dan Kegel d...@kegel.com wrote: Stop me if you've heard this one before. Today, a new directory was added to the source tree, and shortly thereafter was reverted. Should have been no problem, but... because the new directory contained a gyp file, a file was generated in that directory, and svn couldn't delete the directory when the revert landed. This caused a build breakage, and I gather from nsylvain's comments that this wasn't the first time this has happened. At some point soon, it'd be good to teach gyp not to generate files in the source tree. Or maybe teach gclient how to deal forcefully with directories with no files under version control. Generating files in the source tree is kinda the point of gyp. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
Being the person who perpetrated this crime, if someone could even tell me how to fix it, that would be an improvement. It seems like nsylvain is the only one with the appropriate mojo (at least in the evenings...) -- Dirk On Wed, Jul 22, 2009 at 8:27 PM, Dan Kegeld...@kegel.com wrote: Stop me if you've heard this one before. Today, a new directory was added to the source tree, and shortly thereafter was reverted. Should have been no problem, but... because the new directory contained a gyp file, a file was generated in that directory, and svn couldn't delete the directory when the revert landed. This caused a build breakage, and I gather from nsylvain's comments that this wasn't the first time this has happened. At some point soon, it'd be good to teach gyp not to generate files in the source tree. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Thu, Jul 23, 2009 at 4:15 AM, Mark Larson (Google)m...@chromium.org wrote: Should have been no problem, but... because the new directory contained a gyp file, a file was generated in that directory, and svn couldn't delete the directory when the revert landed. This caused a build breakage, and I gather from nsylvain's comments that this wasn't the first time this has happened. At some point soon, it'd be good to teach gyp not to generate files in the source tree. Or maybe teach gclient how to deal forcefully with directories with no files under version control. That's kind of, um, forceful. svn doesn't do that for a reason... Generating files in the source tree is kinda the point of gyp. No. Generating files is the point of gyp. Nothing says they have to be in the source tree. - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Generating files in source tree considered harmful
On Wed, Jul 22, 2009 at 21:26, Dan Kegel d...@kegel.com wrote: On Thu, Jul 23, 2009 at 4:15 AM, Mark Larson (Google)m...@chromium.org wrote: Should have been no problem, but... because the new directory contained a gyp file, a file was generated in that directory, and svn couldn't delete the directory when the revert landed. This caused a build breakage, and I gather from nsylvain's comments that this wasn't the first time this has happened. At some point soon, it'd be good to teach gyp not to generate files in the source tree. Or maybe teach gclient how to deal forcefully with directories with no files under version control. That's kind of, um, forceful. svn doesn't do that for a reason... I was perhaps a bit too flip. I don't advocate automatic destruction of these directories, but it seems gclient could offer a mode/flag to clean them up. The build slaves will never have directories with un-versioned changes and could run gclient with this flag all the time. Generating files in the source tree is kinda the point of gyp. No. Generating files is the point of gyp. Nothing says they have to be in the source tree. That is true. The real Mark has already responded to this better than I could. --other Mark - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
