Re: Problem with 64-bit only compiler build
On Wed, 2020-08-12 at 15:05 -0500, Segher Boessenkool wrote: > > As usual I've built my own version of GCC, and then I check it into > > Git so that all builds can use this one canonical compiler > > regardless of operating system, etc. > > There's your problem. Git is not an archiver. Git does not track > directories at all, including empty directories. Git is a "stupid > content tracker", and empty directories are contentless ;-) Yes, I'm aware of Git's shortcomings in this area but... the advantages of using it over other alternatives are simply too great to overcome, at least in my environment. It provides branching (which I need, since older versions of our software need to build with older versions of these tools), it allows for tracking of changes, reversion, it provides a single interface across all platforms including multiple GNU/Linux distributions, Windows, and MacOS (and possibly others in the future), without users having to learn new tools. And it's available both online and offline for developers working remotely, which is critical. Yes, it's not as efficient as it could be for binary files, and it has this "issue" with empty directories (which is not really a big deal, most of the time). But it supports symbolic links, tracks executable permissions, handles moved files and created/deleted directories, etc. Also, an installation of a package like GCC has some binary files, yes, but it also has a LOT of text files. Combined with worktrees so we can share the same repo across different builds on the same system, to reduce the overhead of cloning, it works very well for the most part. At some point this may become too unwieldy and I'll have to look more carefully into git-lfs or similar. But that day has not arrived. > > Of course, I can force the lib directory to exist by adding a dummy > > file there and I will do so. However, it seems to me that it would > > be better if GCC would apply a simplifying function to pathnames to > > reduce all the "/.." elements, before it starts looking for > > directories, to avoid failing if some unnecessary element doesn't > > exist. > > It would be nice if this could be done (where possible -- see Jakub's > reply) before install already, so people looking at your install do > not have to look at ../../../../.. and remove five path components by > hand. There shouldn't be a problem doing it at file open time as you > say (unless we wand to support users putting .. in the paths > themselves, instead of just our make system doing that :-) ) I confess I didn't really follow all of that :) I'll file a bug tonight/tomorrow, then possible solutions can be discussed there.
Re: Problem with 64-bit only compiler build
On Tue, Aug 11, 2020 at 08:01:55PM -0400, Paul Smith wrote: > This is a kind of esoteric problem, but all the more annoying for that. :-) > As usual I've built my own version of GCC, and then I check it into Git > so that all builds can use this one canonical compiler regardless of > operating system, etc. There's your problem. Git is not an archiver. Git does not track directories at all, including empty directories. Git is a "stupid content tracker", and empty directories are contentless ;-) > Of course, I can force the lib directory to exist by adding a dummy > file there and I will do so. However, it seems to me that it would be > better if GCC would apply a simplifying function to pathnames to reduce > all the "/.." elements, before it starts looking for directories, to > avoid failing if some unnecessary element doesn't exist. This > shouldn't be too difficult since the path will be either the same > length as, or shorter than, the original so there's no need to > reallocate memory etc.: it can be updated in place. It would be nice if this could be done (where possible -- see Jakub's reply) before install already, so people looking at your install do not have to look at ../../../../.. and remove five path components by hand. There shouldn't be a problem doing it at file open time as you say (unless we wand to support users putting .. in the paths themselves, instead of just our make system doing that :-) ) Segher
Re: Problem with 64-bit only compiler build
On Wed, 12 Aug 2020 at 17:43, Paul Smith wrote: > However, the lib directory is empty in my build. What lives in your > version of lib? All the runtime libs, but I think that's because mingw doesn't use the lib/lib64 split. $ ls -1 ~/gcc/mingw/x86_64-w64-mingw32/lib/ libatomic-1.dll libatomic.a libatomic.dll.a libatomic.la libgcc_s.a libgcc_s_seh-1.dll libgomp-1.dll libgomp.a libgomp.dll.a libgomp.la libgomp.spec libquadmath-0.dll libquadmath.a libquadmath.dll.a libquadmath.la libssp-0.dll libssp.a libssp.dll.a libssp.la libssp_nonshared.a libssp_nonshared.la libstdc++-6.dll libstdc++.a libstdc++.dll.a libstdc++.dll.a-gdb.py libstdc++.la libstdc++fs.a libstdc++fs.la libsupc++.a libsupc++.la
Re: Problem with 64-bit only compiler build
On Wed, 2020-08-12 at 16:53 +0100, Jonathan Wakely wrote: > On Wed, 12 Aug 2020 at 14:33, Paul Smith > wrote: > > > I'm not talking about PREFIX/lib, though. As can be seen from my > > question I'm talking about PREFIX///lib. This is > > where GCC keeps its own internal libraries, > > Not by default, it isn't. I'm not sure what directory that is, but > none of my builds have it. > > Is this a cross-compiler? Mine have PREFIX//lib instead, and > it's not empty (for a 64-bit --disable-multlib build) Sorry, you're right, the above is part of the PREFIX I provided; my mistake. The path is indeed PREFIX//lib. However, the lib directory is empty in my build. What lives in your version of lib? $ cd $ ls -1 bin/ include/ lib/ lib64/ libexec/ share/ x86_64-unknown-linux-gnu/ $ ls -1 x86_64-unknown-linux-gnu/ bin/ include/ lib/ lib64/ $ ls -1 x86_64-unknown-linux-gnu/lib/ $ I am invoking configure like this (with some flags etc. removed for simplicity): .../gcc-10.2.0/configure \ --prefix= --with-local-prefix= \ --target=x86_64-unknown-linux-gnu --with-sysroot= \ --enable-languages=c,c++ --disable-nls --disable-werror \ --disable-multilib --disable-libgomp --disable-libvtv \ --disable-libstdcxx-dual-abi --with-build-config=bootstrap-lto \ --enable-bootstrap --with-gnu-as --with-gnu-ld --enable-gold so yes, it's a "cross compiler" because I'm setting --target even though the target is really the same architecture. I want to ensure that the entire build both of GCC, and also the binaries built with this GCC, uses only the sysroot and not any local content. Maybe that's overkill, I dunno... it works! :)
Re: Problem with 64-bit only compiler build
On Wed, 12 Aug 2020 at 14:33, Paul Smith wrote: > > I'm not talking about PREFIX/lib, though. > > As can be seen from my question I'm talking about > PREFIX///lib. This is where GCC keeps its own > internal libraries, Not by default, it isn't. I'm not sure what directory that is, but none of my builds have it. Is this a cross-compiler? Mine have PREFIX//lib instead, and it's not empty (for a 64-bit --disable-multlib build)
Re: Problem with 64-bit only compiler build
On Wed, 2020-08-12 at 15:37 +0200, Jakub Jelinek wrote: > The important thing is that GCC wants to be relocatable, so most > paths are not hardcoded into the compiler, but depend on where the > gcc driver actually is. One can then just move the whole gcc tree > somewhere else and it should still work. For sure! I take full advantage of this: as mentioned I check GCC into Git and users are free to clone that repo into whatever directory they want, anywhere on their filesystem, and it works. If we consider the internal paths GCC computes to be constructed in two parts: / then the part that I'm talking about is the . > > It appears that what happens is that the frontend computes the path > > to the "lib" (32bit) directory, then when it wants to look for > > 64bit libraries instead of re-computing the path from scratch with > > "lib64", it just adds on a "../lib64" to the 32bit path. > > That is because that is the way OS multilib directories are > specified, it is defined by adding `gcc -print-multi-os-directory > $CFLAGS` to the normal paths. Well, perhaps the right thing to do is simplify the results of this without requiring that the original exists.
Re: ** POTENTIAL FRAUD ALERT - RED HAT ** Re: Problem with 64-bit only compiler build
On Wed, Aug 12, 2020 at 09:33:05AM -0400, Paul Smith wrote: > As someone who takes a lot of advantage of the flexibility that tools > like GCC provide I'm wary of reducing that flexibility. On the other > hand I'm not sure that breaking up the internal structure of GCC's > installation via symlinks in random ways is something that needs to be > supported. Anyway, the user can always replace the "lib64" directory > with a symlink as well. The important thing is that GCC wants to be relocatable, so most paths are not hardcoded into the compiler, but depend on where the gcc driver actually is. One can then just move the whole gcc tree somewhere else and it should still work. > It appears that what happens is that the frontend computes the path to > the "lib" (32bit) directory, then when it wants to look for 64bit > libraries instead of re-computing the path from scratch with "lib64", > it just adds on a "../lib64" to the 32bit path. That is because that is the way OS multilib directories are specified, it is defined by adding `gcc -print-multi-os-directory $CFLAGS` to the normal paths. Jakub
Re: Problem with 64-bit only compiler build
On Wed, 2020-08-12 at 15:08 +0200, Jakub Jelinek wrote: > > I think it's worth adding this to bugzilla. Depending on the > > existence of empty directories seems less than ideal. > > But canonicalizing the paths without taking the filesystem state into > account will significantly change the behavior in some setups (e.g. > in presence of symlinks) and I'm not sure it is desirable. You're right, about paths that the user provides on the command line. Those should not be modified. However, I'm talking about paths that GCC computes on its own. These are not paths taken from GCC's command line, they are paths computed by the front-end from scratch to be passed to collect2. I agree it would need to be considered carefully. As someone who takes a lot of advantage of the flexibility that tools like GCC provide I'm wary of reducing that flexibility. On the other hand I'm not sure that breaking up the internal structure of GCC's installation via symlinks in random ways is something that needs to be supported. Anyway, the user can always replace the "lib64" directory with a symlink as well. > The lib paths usually are not empty, the typical installation is > together with other programs/libraries. And even the PREFIX/lib path > should have gcc subdirectory with it with lots of files in there. I'm not talking about PREFIX/lib, though. As can be seen from my question I'm talking about PREFIX///lib. This is where GCC keeps its own internal libraries, and that directory will always be empty if you don't build a 32bit compiler. I don't think it's common at all for people to be dropping non-GCC libraries into these internal GCC directories (just from a packaging standpoint it would be a nightmare to maintain). It appears that what happens is that the frontend computes the path to the "lib" (32bit) directory, then when it wants to look for 64bit libraries instead of re-computing the path from scratch with "lib64", it just adds on a "../lib64" to the 32bit path. So maybe the right answer is to build the lib64 path using the same algorithm used to build the "lib" path, except with "lib64", rather than adding a "../lib64" to the end of the "lib" path. I'll be the first to admit I've not looked at the frontend code at all so I could be completely wrong about how it works.
Re: Problem with 64-bit only compiler build
On Wed, Aug 12, 2020 at 01:08:53PM +0100, Jonathan Wakely via Gcc wrote: > > Oddly, I looked through the gnulib library and didn't find any > > appropriate module for this. It seems like there should be one. > > C++17 provides std::filesystem::weakly_canonical for that. It doesn't > help GCC or gnulib though. > > POSIX realpath and GNU canonicalize_file_name do the canonicalization, > but require every path component to exist. > > I think it's worth adding this to bugzilla. Depending on the existence > of empty directories seems less than ideal. But canonicalizing the paths without taking the filesystem state into account will significantly change the behavior in some setups (e.g. in presence of symlinks) and I'm not sure it is desirable. The lib paths usually are not empty, the typical installation is together with other programs/libraries. And even the PREFIX/lib path should have gcc subdirectory with it with lots of files in there. Jakub
Re: Problem with 64-bit only compiler build
On Wed, 12 Aug 2020 at 01:02, Paul Smith wrote: > > This is a kind of esoteric problem, but all the more annoying for that. > > As usual I've built my own version of GCC, and then I check it into Git > so that all builds can use this one canonical compiler regardless of > operating system, etc. > > After being checked into Git, the compiler started failing to link with > an error that it could not find libgcc_s.so, even though it exists in > exactly the same place (relatively) as it was before. > > After some debugging and strace-ing, I discovered that in the failure > mode the front-end didn't provide this flag to collect2, which is where > libgcc_s.so lives: > > -L/.../cc/unknown/bin/../lib/gcc/x86_64-unknown-linux-gnu/10.2.0/../../../../x86_64-unknown-linux-gnu/lib/../lib64/. > > The problem is this: when I built this compiler I only built the 64bit > version of it, not the 32bit version. As a result, the lib/. > directories are empty: only the lib64 directories have any content. > > When I check this into Git, of course all empty directories are > ignored. So while the /.../cc/unknown/x86_64-unknown-linux-gnu/lib64 > directory, where 64bit libgcc_s.so lives, does exist, the > /.../cc/unknown/x86_64-unknown-linux-gnu/lib directory doesn't exist, > which means the above path cannot be resolved. > > Of course, I can force the lib directory to exist by adding a dummy > file there and I will do so. However, it seems to me that it would be > better if GCC would apply a simplifying function to pathnames to reduce > all the "/.." elements, before it starts looking for directories, to > avoid failing if some unnecessary element doesn't exist. This > shouldn't be too difficult since the path will be either the same > length as, or shorter than, the original so there's no need to > reallocate memory etc.: it can be updated in place. > > Oddly, I looked through the gnulib library and didn't find any > appropriate module for this. It seems like there should be one. C++17 provides std::filesystem::weakly_canonical for that. It doesn't help GCC or gnulib though. POSIX realpath and GNU canonicalize_file_name do the canonicalization, but require every path component to exist. I think it's worth adding this to bugzilla. Depending on the existence of empty directories seems less than ideal.
