RE: flex: exec failed?
No longer a Cygwin issue (repros on Linux), but here's why if anyone interested:
#include
#include
int main()
{
signal(SIGPIPE, SIG_IGN); Python
putenv(M4=foo);
system(flex 1.l);
return 0;
}
Without ignoring sigpipe, flex dies reading the pipe of a failed-to-exec child,
before it gets around to unlink(lex.yy.c).
If you put instrumentation (printf) in the fgets/fputs loop in
filter_tee_header in flex,
you find the sh variant stops without exiting the loop, whereas the python
variant
continues on, and deletes lex.yy.c.
I'll patch my gmp/configure to not set M4 like this.
I've reported this to gcc and gmp (gcc hits this
because it configures gmp with cpu=none; usually building
gmp sets M4=m4, which is ok, though it really it should leave it alone.).
- Jay
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: flex: exec failed?
Date: Thu, 31 Jul 2008 12:45:45 +
Ok, it works the same on Cygwin and Linux/x86, but Python/NTAMD64 works.
Somehow Python/Posix related, portably.
For anyone interested:
$ echo %% conftest.l
$ echo %% conftest.l
$ rm lex.yy.c
$ export M4=foo
$ flex conftest.l
flex: fatal internal error, exec failed
lex.yy.c exists
$ rm lex.yy.c
$ python -c import os; os.system(\flex conftest.l\)
flex: fatal internal error, exec failed
flex: 2 error writing output file lex.yy.c (I added the 2 to disambiguate 4
identical error messages)
lex.yy.c does not exists -- make configure-gmp fails
relevant code in flex:
filter_tee_header:
if ((to_cfd = dup (1)) == -1)
flexfatal (_(dup(1) failed));
to_c = fdopen (to_cfd, w);
...
fflush (to_c);
if (ferror (to_c))
lerrsf (_(2 error writing output file %s),
outfilename ? outfilename : );
else if (fclose (to_c))
lerrsf (_(error closing output file %s),
outfilename ? outfilename : );
while (wait (0) 0) ;
again, I added the 2.
Digging deeper...
- Jay
[snip snip snip]
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
RE: flex: exec failed?
Arg.. simple repro is this Python code, with the appropriate gcc source tree: #! /usr/bin/env python import os import sys def Run(Directory, Command): sys.stderr.flush() sys.stdout.flush() print(cd + Directory ++ Command) PreviousDirectory = os.getcwd() os.chdir(Directory) if os.name == nt: Command = sh -c \ + Command + \ sys.stderr.flush() sys.stdout.flush() ExitCode = os.system(Command) os.chdir(PreviousDirectory) sys.stderr.flush() sys.stdout.flush() if ExitCode != 0: sys.exit(ExitCode) if not os.path.isdir(./obj): Run(., mkdir ./obj) if not os.path.isfile(./obj/Makefile): Run(./obj, ../gcc/configure) Run(., rm -rf ./obj/gmp) Run(./obj, make configure-gmp) Repros in Python, not outside of Python? Why? That is the question. Other tidbits: flex actually fails to exec m4 either way, just that one way it manages to create lex.yy.c first. It fails to exec m4 because gmp's configure has set M4=m4-not-needed. This is almost certainly a simple bug in gmp's configure. A fix would be to probe for external programs (yacc, lex), earlier, before it sets M4. Or unset M4 after it is done with it. But there is still something amiss, maybe Cygwin specific, maybe not. Oh, and whether or not gmp sets M4 varies somewhat on build/host/target. Usually, configuring gmp on its own, probably it is set. But within gcc, generally not, because gcc configures it with processor=none for reasons I don't understand.. I guess it'd be nice if flex printed what it was trying to run -- getenv(M4) and not just m4. I'll follow up separately, at least gcc and gmp.. Still not completely solved. I do have a quick repro though, you can strip down the configure script a bunch and it repros in just a few seconds. Going through the strace logsbut need to read flex also for when it creates the output vs. running m4.. - Jay From: [EMAIL PROTECTED] To: cygwin@cygwin.com Subject: RE: flex: exec failed? Date: Sat, 26 Jul 2008 12:40:01 + anecdotes... I've seen this quite a few times now, like 10. Always this same error -- always configure unable to determine the (f)lex output, I think always configuring gmp. (Maybe binutils sometimes, due to the flex source directory, understood, red herring.) I've done a lot else successfully, configure/built gcc/binutils/gmp/mpfr, this is the one failure I've seen a few of. That is -- lots of stuff works, I'm not messing things up in some large way, but this fails often. I'm not using tee any longer, but still running all of this within Python (os.system). I don't usually check config.log, but it always the same configure test. Peeling one layer, either cd'ing in cmd or bash to gcc toplevel and doing make configure-gmp or two layers, cd'ing to gcc/gmp and whatever/configure whatever, has a much much higher success rate. And then I can run my stuff, which is incremental enough, and so I skip past this. I stopped installing Cygwin at the root, though I've seen it both work and not-work that way. I am usually on XP/AMD64, and maybe every repro is on that. I do notice comments about thread unsafety in Cygwin's fork/exec code. My Python is single threaded (I was getting intermittent errors about unable to allocate locks). I'll have to put strace in there and focus in on it, instead of continuing to workaround. ..Jay From: [EMAIL PROTECTED] To: cygwin@cygwin.com Subject: FW: flex: exec failed? Date: Wed, 16 Jul 2008 18:55:06 + It failed again without the time/tee. Time to try with -disable-bootstrap, and if it still failed, reboot, and... From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: flex: exec failed? Date: Wed, 16 Jul 2008 18:53:23 + anyone familiar with this: flex: fatal internal error, exec failed flex: error writing output file lex.yy.c It comes while configuring gmp, in a merged (Cygnus) gcc tree. I've been able to build this several times, various configurations. (various host/target and options, mostly build=cygwin, but on some other systems too, unrelated..) What I am doing differently this time is two things. 1) I'm no longer -disable-bootstrap. I am running under a wrapper .py file. (It goes through multiple configurations.) That isn't changed, but: 2) previously: ./build.py now: time (sh -c ./build.py | tee 1.log) or time (./build.py | tee 1.log) I have tried both. config.log nearby: configure:32814: checking for flex configure:32840: result: flex configure:32853: checking for yywrap in -lfl configure:32883: gcc -o conftest.exe -g -fkeep-inline-functions -DNO_ASM conftest.c -lfl5 configure:32889: $? = 0 configure:32893: test -z || test ! -s conftest.err configure:32896: $? = 0 configure:32899: test -s conftest.exe configure:32902: $? = 0 configure:32915: result: yes configure:32993: checking lex output file root configure:33004: flex conftest.l flex
RE: flex: exec failed?
Ok, it works the same on Cygwin and Linux/x86, but Python/NTAMD64 works. Somehow Python/Posix related, portably. For anyone interested: $ echo %% conftest.l $ echo %% conftest.l $ rm lex.yy.c $ export M4=foo $ flex conftest.l flex: fatal internal error, exec failed lex.yy.c exists $ rm lex.yy.c $ python -c import os; os.system(\flex conftest.l\) flex: fatal internal error, exec failed flex: 2 error writing output file lex.yy.c (I added the 2 to disambiguate 4 identical error messages) lex.yy.c does not exists -- make configure-gmp fails relevant code in flex: filter_tee_header: if ((to_cfd = dup (1)) == -1) flexfatal (_(dup(1) failed)); to_c = fdopen (to_cfd, w); ... fflush (to_c); if (ferror (to_c)) lerrsf (_(2 error writing output file %s), outfilename ? outfilename : ); else if (fclose (to_c)) lerrsf (_(error closing output file %s), outfilename ? outfilename : ); while (wait (0) 0) ; again, I added the 2. Digging deeper... - Jay [snip snip snip] -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: flex: exec failed?
anecdotes...
I've seen this quite a few times now, like 10.
Always this same error -- always configure unable to determine the (f)lex
output, I think always configuring gmp.
(Maybe binutils sometimes, due to the flex source directory, understood, red
herring.)
I've done a lot else successfully, configure/built gcc/binutils/gmp/mpfr,
this is the one failure I've seen a few of.
That is -- lots of stuff works, I'm not messing things up in some large way,
but this fails often.
I'm not using tee any longer, but still running all of this within Python
(os.system).
I don't usually check config.log, but it always the same configure test.
Peeling one layer, either cd'ing in cmd or bash to gcc toplevel and doing make
configure-gmp or two layers, cd'ing to gcc/gmp and whatever/configure
whatever, has a much much higher success rate. And then I can run my stuff,
which is incremental enough, and so I skip past this.
I stopped installing Cygwin at the root, though I've seen it both work and
not-work that way.
I am usually on XP/AMD64, and maybe every repro is on that.
I do notice comments about thread unsafety in Cygwin's fork/exec code.
My Python is single threaded (I was getting intermittent errors about unable to
allocate locks).
I'll have to put strace in there and focus in on it, instead of continuing to
workaround.
..Jay
From: [EMAIL PROTECTED]
To: cygwin@cygwin.com
Subject: FW: flex: exec failed?
Date: Wed, 16 Jul 2008 18:55:06 +
It failed again without the time/tee. Time to try with -disable-bootstrap,
and if it still failed, reboot, and...
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: flex: exec failed?
Date: Wed, 16 Jul 2008 18:53:23 +
anyone familiar with this:
flex: fatal internal error, exec failed
flex: error writing output file lex.yy.c
It comes while configuring gmp, in a merged (Cygnus) gcc tree.
I've been able to build this several times, various configurations.
(various host/target and options, mostly build=cygwin, but
on some other systems too, unrelated..)
What I am doing differently this time is two things.
1) I'm no longer -disable-bootstrap.
I am running under a wrapper .py file.
(It goes through multiple configurations.)
That isn't changed, but:
2) previously:
./build.py
now:
time (sh -c ./build.py | tee 1.log)
or
time (./build.py | tee 1.log)
I have tried both.
config.log nearby:
configure:32814: checking for flex
configure:32840: result: flex
configure:32853: checking for yywrap in -lfl
configure:32883: gcc -o conftest.exe -g -fkeep-inline-functions -DNO_ASM
conftest.c -lfl5
configure:32889: $? = 0
configure:32893: test -z
|| test ! -s conftest.err
configure:32896: $? = 0
configure:32899: test -s conftest.exe
configure:32902: $? = 0
configure:32915: result: yes
configure:32993: checking lex output file root
configure:33004: flex conftest.l
flex: fatal internal error, exec failed
flex: error writing output file lex.yy.c
configure:33007: $? = 1
configure:33014: error: cannot find output from flex; giving up
shell code nearby (configure):
if test x$LEX != x:; then
echo $as_me:$LINENO: checking lex output file root5
echo $ECHO_N checking lex output file root... $ECHO_C6
if test ${ac_cv_prog_lex_root+set} = set; then
echo $ECHO_N (cached) $ECHO_C6
else
# The minimal lex program is just a single line: %%. But some broken lexes
# (Solaris, I think it was) want two %% lines, so accommodate them.
catconftest.l 5
(eval $LEX conftest.l) 25
ac_status=$?
echo $as_me:$LINENO: \$? = $ac_status5
(exit $ac_status); }
if test -f lex.yy.c; then
ac_cv_prog_lex_root=lex.yy
elif test -f lexyy.c; then
ac_cv_prog_lex_root=lexyy
else
{ { echo $as_me:$LINENO: error: cannot find output from $LEX; giving up5
echo $as_me: error: cannot find output from $LEX; giving up2;}
{ (exit 1); exit 1; }; }
fi
fi
flex code nearby:
/** Fork and exec entire filter chain.
* @param chain The head of the chain.
* @return true on success.
*/
bool filter_apply_chain (struct filter * chain)
{
int pid, pipes[2];
/* Tricky recursion, since we want to begin the chain
* at the END. Why? Because we need all the forked processes
* to be children of the main flex process.
*/
if (chain)
filter_apply_chain (chain-next);
else
return true;
/* Now we are the right-most unprocessed link in the chain.
*/
fflush (stdout);
fflush (stderr);
if (pipe (pipes) == -1)
flexerror (_(pipe failed));
if ((pid = fork ()) == -1)
flexerror (_(fork failed));
if (pid == 0) {
/* child */
/* We need stdin (the FILE* stdin) to connect to this new pipe.
* There is no portable way to set stdin to a new file descriptor,
* as stdin is not an lvalue on some systems (BSD).
* So we dup the new pipe onto the stdin descriptor and use a no-op fseek
* to sync the stream. This is a Hail Mary situation. It seems to work.
*/
close (pipes[1]);
if (dup2 (pipes[0], fileno (stdin)) == -1)
flexfatal
Re: flex: exec failed? (root install/mount)
On Jul 16 19:33, Jay wrote: configure:10596: /obj/gcc.5/i686-pc-cygwin/i686-pc-cygwin/build-i686-pc-cygwin/flex/flex conftest.l strace: error creating process D:\\obj/gcc.5/i686-pc-cygwin/i686-pc-cygwin/build-i686-pc-cygwin/flex/flex, (error 3) which indicates the problem is: $ mount d:\bin on /usr/bin type system (binmode) d:\lib on /usr/lib type system (binmode) d: on / type system (binmode) c: on /cygdrive/c type system (binmode,noumount) d: is being interpeted, as it is on Win32, as the current directory on drive d, and not, as intended here, as the root of d. This is never the case in Cygwin. The above error message occurs in strace, which is not a Cygwin application, but a native Win32 application. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: flex: exec failed? (root install/mount)
[J] jay [C] corinna Subject: Re: flex: exec failed? (root install/mount) [J] configure:10596: /obj/gcc.5/i686-pc-cygwin/i686-pc-cygwin/build-i686-pc-cygwin/flex/flex conftest.l [J] strace: error creating process D:\\obj/gcc.5/i686-pc-cygwin/i686-pc-cygwin/build-i686-pc-cygwin/flex/flex, (error 3) [J] [J] which indicates the problem is: [J] $ mount [J] d:\bin on /usr/bin type system (binmode) [J] d:\lib on /usr/lib type system (binmode) [J] d: on / type system (binmode) [J] c: on /cygdrive/c type system (binmode,noumount) [J] [J] d: is being interpeted, as it is on Win32, as the current directory on drive d, [J] and not, as intended here, as the root of d. [C] This is never the case in Cygwin. The above error message occurs in [C] strace, which is not a Cygwin application, but a native Win32 [C] application. Thanks Corinna. Good catch. I realize what you say, but on the other hand, the mount stuff kind of lives in the transition zone between Win32 and the rest of Cygwin. Look, those are Win32 paths on the left, and the theory I gave is reasonable, even if untrue. :) However, I did later notice another problem. After believing my theory, and reinstalling to c:\cygwin, the error kept occuring. (Oh, and yes, I stuck strace into the configure file). Finally I remembered, that after the original problem, and noticing that the gcc tree is apparently setup to build a bunch of stuff all together, not just gcc, binutils, gmp, mpfr, but a bunch more, including flex, I stuck the flex source in there as well. Figuring I'd maybe build and debug it.Well, it appears the gcc's support for integrating flex is actually not working. Configuring binutils decides to run a flex that doesn't yet exist, if the source is there. Looking in . and looking where configure thinks flex will be, while perhaps different by a path element or two, to the very lazy/casual/careless/Jay observer -- same thing. SO, I've deleted the flex source, and reinstalled to c:\cygwin, and I'll punt this for now. I have symlinks /c = /cygdrive/c, /obj = /c/obj, etc. At some point I should go back to c:\ (or d:\, whatever), keep flex source out, and see what happens. I still think my theory is close to correct, but I probably polluted the repro case before the later debugging, oops. I'm sure I didn't put the flex code there until after I first had some problem. Thanks, - Jay -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: flex: exec failed? (root install/mount)
As a follow-up and clarification to my previous post, what I would like to be able to do is to pipe output to a printer, e.g. ls -l | lpr This doesn't work for me as I explained previously. Using notepad /P I'll bet it doesn't! From the lpr manpage: No formatting is done -- data is sent raw. This means your printer has to understand *raw* text, which is what the 'ls' command generates. Many printers can do this, but U'll probably have to play with driver setup to make it work. BTW, I didn't see this thread on gmane yet, so I just started it over. I apologize for messing up threading, but I just *had* to reply. Click to see huge collection of designer watches. http://thirdpartyoffers.juno.com/TGL2131/fc/Ioyw6iie2atUHtI1aAGPjGD6RZew9zmJ8AOq3k8bX2bmvK7k16TTx4/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: flex: exec failed? (root install/mount)
configure:10596: /obj/gcc.5/i686-pc-cygwin/i686-pc-cygwin/build-i686-pc-cygwin/flex/flex conftest.l strace: error creating process D:\\obj/gcc.5/i686-pc-cygwin/i686-pc-cygwin/build-i686-pc-cygwin/flex/flex, (error 3) which indicates the problem is: $ mount d:\bin on /usr/bin type system (binmode) d:\lib on /usr/lib type system (binmode) d: on / type system (binmode) c: on /cygdrive/c type system (binmode,noumount) d: is being interpeted, as it is on Win32, as the current directory on drive d, and not, as intended here, as the root of d. I tried unmounting it and explicitly mounting d:\, d:\\, d:/, but they still all got treated as d:. Tried editing the registry, same thing. Used d:\foo also to make sure what I was doing was being seen by the code. Another way to achieve what I want is for d:\cygwin to be an NTFS junction do d:\, but then I have circularities in my file system, which I'd rather avoid. I know installing to the root is discouraged, but..agreed this is a bug? Should be fixed? It is easy to fix? I think so, on all counts. I'm sure I can provide a patch. The /cygdrive mounts must deal with this specially?? - Jay From: [EMAIL PROTECTED] To: cygwin@cygwin.com Subject: FW: flex: exec failed? Date: Wed, 16 Jul 2008 18:55:06 + From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: flex: exec failed? Date: Wed, 16 Jul 2008 18:53:23 + anyone familiar with this: flex: fatal internal error, exec failed flex: error writing output file lex.yy.c -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
