[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #20 from John Soo --- I think that in order to really rid gcc of the E2BIG problem on linux, COLLECT_*_OPTIONS will have to be deprecated and removed. This is particularly a problem when executing spec files, it seems.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #19 from John Soo --- I verified the proposed patch sent argv through @file, but COLLECT_GCC_OPTIONS still caused E2BIG. In the failing execve, COLLECT_GCC_OPTIONS was 134227 characters long.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #18 from John Soo --- And actually the proposed patch is not conservative enough, because the size of the strings in argv/env also matter.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #17 from John Soo --- Created attachment 55910 --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=55910=edit libiberty, Unix: pass argv over ARG_MAX through an @file This does not handle environ being too large, but it is an adaptation of the argv fix in pex-win32.c.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #16 from John Soo --- It is actually somewhat likely that ARG_MAX will be hit when running on linux because it is hit when the stack can't contain enough pointers to contain argv and environ (see exec.c in the kernel https://github.com/torvalds/linux/blob/master/fs/exec.c#L509). The bad news is that response files can only mitigate the problem since environ must also be small enough to fit into an execv* call. Is there a reasonable way to keep only the env vars required to exec subprocesses?
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #15 from John Soo --- Just for some context on what limit is hit: under man sysconf you will find a description of ARG_MAX (https://www.man7.org/linux/man-pages/man3/sysconf.3.html) > ARG_MAX - _SC_ARG_MAX > The maximum length of the arguments to the exec(3) family > of functions. Must not be less than _POSIX_ARG_MAX > (4096). Exceeding ARG_MAX will result in E2BIG (https://man7.org/linux/man-pages/man3/errno.3.html). > E2BIG Argument list too long (POSIX.1-2001).
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #14 from John Soo --- > Here though it seems that you are dealing with another sort of limit which is > much larger (I have seen 128K being mentioned on the GH page).If this > somehow corrupts the command line, it wouldn't help if that command line went > into a response file because it would still be wrong.To my knowledge, > Linux-based systems don't have a command line length limitation, so I can't > see how a response file approach would be useful at the point where the > subprocess is spawned.Whether something similar can be used at an earlier > point to save it from the 128K limit, whatever it is, is unknown to me. It is a much larger limit (ARG_MAX resulting in E2BIG), but it is fundamentally the same problem. I think we should assume that the command line is correct and still respect ARG_MAX on linux/unix systems, too. It seems to me that the temporary response file is the best way to do this.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #13 from Costas Argyris --- (In reply to John Soo from comment #12) > I think the general problem in that issue is that ARG_MAX is not respected > when the driver (or any subprocess) execs things on linux. I think that it > is not the same as the original issue here, though. > > > I don't know if its going to be helpful to see that patch as a guide > > Do you think using response files like in pex-win32.c in pex-unix.c could > help? I tried this out and it seems like this may not solve all the ARG_MAX > problems. I really don't know because the problem might be happening even before it gets to the point where the new subprocess is spawned.Then, passing the command line through a response file wouldn't help with anything, if the problem has already happened by that point. The Windows patch above just checks if the subprocess is about to be spawned with a command line > 32K and if that is true it simply puts it in a temporary response file and calls the subprocess with @rsp as the only arg, instead of the large command line which would break it.But obviously this makes the assumption that the large command line is correct (nothing has been dropped or otherwise messed up). Here though it seems that you are dealing with another sort of limit which is much larger (I have seen 128K being mentioned on the GH page).If this somehow corrupts the command line, it wouldn't help if that command line went into a response file because it would still be wrong.To my knowledge, Linux-based systems don't have a command line length limitation, so I can't see how a response file approach would be useful at the point where the subprocess is spawned.Whether something similar can be used at an earlier point to save it from the 128K limit, whatever it is, is unknown to me.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #12 from John Soo --- I think the general problem in that issue is that ARG_MAX is not respected when the driver (or any subprocess) execs things on linux. I think that it is not the same as the original issue here, though. > I don't know if its going to be helpful to see that patch as a guide Do you think using response files like in pex-win32.c in pex-unix.c could help? I tried this out and it seems like this may not solve all the ARG_MAX problems.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #11 from Costas Argyris --- (In reply to John Soo from comment #10) > I'm also not sure > https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff; > h=180ebb8a24d24fc5b105f2257d6216f6dfde62df fixes the collect bug because > collect uses collect_execute instead of the pex_* exec functions. That patch was only meant to deal with the 32K command line length limitation on Windows hosts, for programs spawned with the pex functions (to my knowledge, this is how the driver spawns its sub-programs like the compiler, assembler and linker). In this PR though the problem appears to be something else which I am not familiar with, so I don't know if its going to be helpful to see that patch as a guide. I only put the reference in the hopes that it solved this as well because it looked similar, but it doesn't look like it.You may need to come up with a completely new solution for all I know.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #10 from John Soo --- I'm also not sure https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=180ebb8a24d24fc5b105f2257d6216f6dfde62df fixes the collect bug because collect uses collect_execute instead of the pex_* exec functions.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #9 from John Soo --- Would a patch for unix doing something similar to https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=180ebb8a24d24fc5b105f2257d6216f6dfde62df be accepted? I am happy to start working on something like it but I have no gcc contributions yet and would like to know ahead of time if it is desired.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #8 from John Soo --- > Also, it is typically Windows that suffers from this limitation of command > line length. Ok that may be true but I am effected by this on linux as are quite a few others in this issue https://github.com/NixOS/nixpkgs/issues/41340
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #7 from Costas Argyris --- (In reply to John Soo from comment #6) > This is not a Windows-only bug, so I don't think it is fixed. Althought it is not mentioned explicitly in the title of this PR, the original reporter did describe it as Windows-specific problem: "GCC does not use response files for %D and %I handling (see do_spec_1), so if you have a project which contrains a lot of Includes or library directories cc1 or collect2 respectively will overflow the commandline limit on Windows and error out." Also, it is typically Windows that suffers from this limitation of command line length.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 John Soo changed: What|Removed |Added CC||john.soo+gcc-bugzilla@arist ||a.com --- Comment #6 from John Soo --- This is not a Windows-only bug, so I don't think it is fixed.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 Costas Argyris changed: What|Removed |Added CC||costas.argyris at gmail dot com --- Comment #5 from Costas Argyris --- Very likely fixed by https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=180ebb8a24d24fc5b105f2257d6216f6dfde62df
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 Jonathan Wakely changed: What|Removed |Added Last reconfirmed|2018-07-02 00:00:00 |2021-5-19 --- Comment #4 from Jonathan Wakely --- This is still a problem today: https://github.com/envoyproxy/envoy/issues/16196
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 --- Comment #3 from Tamar Christina --- Hmm according to 45749 with `HAVE_GNU_LD` on it should work for the ld call. I will give that a try. But the environment variable one is still an issue as far as I can tell.
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 Tamar Christina changed: What|Removed |Added Status|RESOLVED|NEW Last reconfirmed||2018-07-02 Resolution|FIXED |--- Ever confirmed|0 |1 --- Comment #2 from Tamar Christina --- It seems that this is not fully fixed yet. There are a number of other issues here. Given a simple response file /d/t/temp cat foo test.c -L/tmp/temp -L/bar/temp/rr -ldata_boo and calling gcc @foo -save-temps -### shows a few problems. At first glance it looks like the correct thing was done, the options are split into two new response files and passed along to collect2 and the commandline for collect2 no longer has the paths expanded in them. /d/t/temp cat /tmp/ccIP9us9 test.o -ldata_boo /d/t/temp cat /tmp/ccSls8Kg -L/tmp/temp -L/bar/temp/rr However, annoyingly, they have been expanded and placed into the `COLLECT_GCC_OPTIONS` environment variable COLLECT_GCC_OPTIONS='-L/tmp/temp' '-L/bar/temp/rr' '-save-temps' '-mlittle-endian' '-mabi=lp64' Which means they're getting passed twice.. And finally it seems that when collect2 makes the call to ld, it places everything on one giant commandline, undoing all the work of creating the response files and exploding the path. collect2 should honor the response files in it's output!
[Bug driver/86030] specs file processing does not create response files for input directories
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030 Tamar Christina changed: What|Removed |Added Status|UNCONFIRMED |RESOLVED Resolution|--- |FIXED --- Comment #1 from Tamar Christina --- Fixed by https://gcc.gnu.org/ml/gcc-patches/2018-04/msg01172.html and r261474. Thanks Eric.
