Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
Greetings, Corinna Vinschen! That has nothing to do with Cygwin. This is just how the environment is handled in general. Child processes inherit it from their parent processes, regardless whether on Windows or Unix. Windows have means of notifying running processes about changes in global environment. It isn't really reliable, though. Only works, when you're editing environment from the GUI, AFAIK. -- WBR, Andrey Repin (anrdae...@freemail.ru) 11.05.2012, 12:55 Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
The cygwin DLL can't intercede here. It's not some superior process controlling I/O. It's just a DLL used by two programs. cat is writing to the stdout that it inherited from the shell. Good point, I had not thought this through enough. In this case, bash.exe is asking CYGWIN1.DLL for a pipe, and then setting up the pipe between the two processes exactly as you outline. Again maybe I'm naive, but to do what I mention: 1. Cygwin would need an additional function with a parameter to specify byte pipes (maybe you already added this, I have not checked the recent source code very much). 2. Bash and other shells would need a platform-specific patch to do the checks I described: if piping to a non-Cygwin program, then use the special Cygwin function for making byte pipes. But that would require patching upstream shell with a platform-specific change. Any other code in packages that might pipe data to a non-Cygwin process would need patching too (although, I cannot at the moment think of anything besides a shell). The bright side is that it's a local solution to a local problem - no global variable that might adversely affect other code. [cat] doesn't know (and shouldn't know) what's on the other end of the pipe and that it should be changing the mode of the pipe. Agree 100%; the previous message wasn't meant to imply otherwise. 1.7.15 will contain a new CYGWIN option pipe_byte which will force non- tty, non-fifo pipes to use byte mode. That's about as far as we're willing to modify Cygwin for this apocalyptic problem. Thanks - I guess that will address the immediate issue in practice, although it's a global solution to a local problem - though I don't see a way around it without patching the shell itself. Forgive me for asking perhaps a dumb question, but I was reading the page at http://cygwin.com/cygwin-ug-net/setup-env.html and wasn't too sure what was meant by the sentence All Windows environment variables are imported when Cygwin starts - which I assume includes parsing and using CYGWIN option? Is it: (1) the variables are processed the first time CYGWIN1.DLL is loaded into a process, and then never used again thereafter, or (2) every time a process linking with CYGWIN1.DLL loads - even if CYGWIN1.DLL is concurrently loaded by someone else? I would assume it is #2 Or to phrase another way: does Cygwin start once when the first process linking to it is loaded, or does it start every time a process loads? (I realize Cygwin has extensive IPC and shared memory mechanisms, some of which is probably done only on first load of first process to initialize shared memory, which is what prompted this question.) -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
Hi James, 1. Cygwin would need an additional function with a parameter to specify byte pipes (maybe you already added this, I have not checked the recent source code very much). You just missed the announcement: On 2012-05-10 08:28, Corinna Vinschen wrote: [snip] I just released 1.7.15. This is a bugfix release. Only one new feature has been added. [snip] - Add CYGWIN=pipe_byte option to force opening of pipes in byte mode rather than message mode. That's probably just for you. Cheers, Otto -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Thu, May 10, 2012 at 12:28 PM, James Johnston wrote: Forgive me for asking perhaps a dumb question, but I was reading the page at http://cygwin.com/cygwin-ug-net/setup-env.html and wasn't too sure what was meant by the sentence All Windows environment variables are imported when Cygwin starts - which I assume includes parsing and using CYGWIN option? Is it: (1) the variables are processed the first time CYGWIN1.DLL is loaded into a process, and then never used again thereafter, or (2) every time a process linking with CYGWIN1.DLL loads - even if CYGWIN1.DLL is concurrently loaded by someone else? I would assume it is #2 Or to phrase another way: does Cygwin start once when the first process linking to it is loaded, or does it start every time a process loads? (I realize Cygwin has extensive IPC and shared memory mechanisms, some of which is probably done only on first load of first process to initialize shared memory, which is what prompted this question.) IIRC Cygwin processes the Windows environment upon initial parent process load and stores it internally for each parent process. Child processes inherit the environment of the parent. So if I start to shell command windows both look the same on start-up but if I modify one of the environments in one command window the other is unaffected. If I start a process in the changed environment then the subsequent process will have the changed environment. HTH, -- Earnie -- https://sites.google.com/site/earnieboyd -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On May 10 13:57, Earnie Boyd wrote: On Thu, May 10, 2012 at 12:28 PM, James Johnston wrote: Forgive me for asking perhaps a dumb question, but I was reading the page at http://cygwin.com/cygwin-ug-net/setup-env.html and wasn't too sure what was meant by the sentence All Windows environment variables are imported when Cygwin starts - which I assume includes parsing and using CYGWIN option? Is it: (1) the variables are processed the first time CYGWIN1.DLL is loaded into a process, and then never used again thereafter, or (2) every time a process linking with CYGWIN1.DLL loads - even if CYGWIN1.DLL is concurrently loaded by someone else? I would assume it is #2 Or to phrase another way: does Cygwin start once when the first process linking to it is loaded, or does it start every time a process loads? (I realize Cygwin has extensive IPC and shared memory mechanisms, some of which is probably done only on first load of first process to initialize shared memory, which is what prompted this question.) IIRC Cygwin processes the Windows environment upon initial parent process load and stores it internally for each parent process. Child processes inherit the environment of the parent. So if I start to shell command windows both look the same on start-up but if I modify one of the environments in one command window the other is unaffected. If I start a process in the changed environment then the subsequent process will have the changed environment. That has nothing to do with Cygwin. This is just how the environment is handled in general. Child processes inherit it from their parent processes, regardless whether on Windows or Unix. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On 5/9/12 1:11 PM, Christopher Faylor wrote: Please stop thinking that there is a simple solution that you can intuit without fully understanding how Cygwin works. Indeed. _Everything_ is more complicated than it looks. 1.7.15 will contain a new CYGWIN option pipe_byte which will force non-tty, non-fifo pipes to use byte mode. That's about as far as we're willing to modify Cygwin for this apocalyptic problem. What's the downside to using pipe_byte? signature.asc Description: OpenPGP digital signature
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
James Johnston wrote: I can't say with 100% certainty, but I would bet with 90+% confidence that this is a bug in MS's libraries -- they cheat and use a null/0 byte read as a signal for end of file rather than sending out-of-band information that *nix supports. From examining .NET Framework source, it seems clear to me they did not plan on message pipe inputs. Win32 ReadFile API normally uses a zero byte read to indicate end-of-file; I think it's not cheating. Pipes present a special case. I'm not sure how MS's runtime libraries could send out-of-band information; the operating system does not really support this as far as I know? It is cheating because they didn't follow the standard. It is perfectly legal to write 0 bytes. Of course, these libraries are very widely used... --- MSWindows is widely used and is known to be one of the buggiest OS's on the planet. So what's your point? Essentially you need a 'shim' layer between a POSIX compliant subsystem to NON-POSIX compliant programs. I'm sure that in the case that my assumptions are true, you wouldn't want to deliberate put something in cygwin that would make it less POSIX compliant when it is only to support external-to-cygwin, NON-POSIX compliant programs... Note -- I use programs between cygwin and Windows 'alot', so I want these things to work as well. Maybe I'm naïve and this is harder than it looks, but couldn't Cygwin determine if the program being piped to links with CYGWIN1.DLL (or similar detection technique), and then use byte or message pipes accordingly? Example: --- Because (1) it wouldn't be reliable. 2, cygwin could be delay-loaded so again back to 1... and have you contacted Microsoft and asked them to fix the problem? Who did you contact and what did they say? # Byte pipe used because Win32Program.exe does not link with CYGWIN1.DLLL cat Testfile.txt | ./Win32Program.exe # Message pipe used because grep links with CYGWIN1.DLL cat Testfile.txt | grep Hello # Message pipe still used because the program we are calling links with CYGWIN1.DLL ./Win32Program.exe | grep Hello If that is still not POSIX compliant enough for the sending program (e.g. cat in this example), then I suppose Cygwin could automatically insert a message-to-byte pipe shim like the author proposes. That seems pretty kludgy though. Cygwin wouldn't insert a special shim for an app using a buggy library. Or rather it wouldn't be the place of cygwin to do so. It would be the responsibility of whoever needs to run some sort of adaptation with **THEIR** program... (and perhaps whoever they pay to do I don't speak for the Cygwin project - the above is purely my point of view. I was suggesting a way for you to create a workaround for your program -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
Daniel Colascione wrote: On 4/27/12 10:27 PM, Christopher Faylor wrote: The above comment shows an and relationship here. Message type pipes more closely mimic Linux (UNIX) pipe behavior AND are definitely required for ptys. Yeah, but because message pipes break other programs. Cygwin has only been using message pipes since 23-Oct-11 (1.134), and it got along fine before that, ICANON aside. If Cygwin programs don't actually _need_ message pipes in the general case, Cygwin should use byte pipes instead. But Daniel -- if the point is to make cygwin behave like *nix, wouldn't it be proper to use the method that most closely emulates it's behavior? I can't say with 100% certainty, but I would bet with 90+% confidence that this is a bug in MS's libraries -- they cheat and use a null/0 byte read as a signal for end of file rather than sending out-of-band information that *nix supports. A suggestion for a workaround: Write a simple C program as a pipe for stdin -- that buffers the I/O by 'line' (or xx bytes). Then you can have it NOT do writes of 0 bytes to MS progs. You could even do it as a wrapper avoidmsbug normal progline with it's arguments avoid would open the appropriate pipes and do the buffering to any buggy MS lib... Essentially you need a 'shim' layer between a POSIX compliant subsystem to NON-POSIX compliant programs. I'm sure that in the case that my assumptions are true, you wouldn't want to deliberate put something in cygwin that would make it less POSIX compliant when it is only to support external-to-cygwin, NON-POSIX compliant programs... Note -- I use programs between cygwin and Windows 'alot', so I want these things to work as well. Another approach in using the wrapper... Just have your real program be named xxx.bin.exe, and let xxx be a link to avoidmsbug.exe. That prog could look at it's name, automatically append the .bin to it's name to execute it... Then you could have any number of cygwin programs with transparent wrappers around them designed to deal with calling programs that mishandle pipe I/O...? -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
-Original Message- Sent: Wednesday, May 09, 2012 19:00 Subject: Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping) I can't say with 100% certainty, but I would bet with 90+% confidence that this is a bug in MS's libraries -- they cheat and use a null/0 byte read as a signal for end of file rather than sending out-of-band information that *nix supports. From examining .NET Framework source, it seems clear to me they did not plan on message pipe inputs. Win32 ReadFile API normally uses a zero byte read to indicate end-of-file; I think it's not cheating. Pipes present a special case. I'm not sure how MS's runtime libraries could send out-of-band information; the operating system does not really support this as far as I know? Of course, these libraries are very widely used... Essentially you need a 'shim' layer between a POSIX compliant subsystem to NON-POSIX compliant programs. I'm sure that in the case that my assumptions are true, you wouldn't want to deliberate put something in cygwin that would make it less POSIX compliant when it is only to support external-to-cygwin, NON-POSIX compliant programs... Note -- I use programs between cygwin and Windows 'alot', so I want these things to work as well. Maybe I'm naïve and this is harder than it looks, but couldn't Cygwin determine if the program being piped to links with CYGWIN1.DLL (or similar detection technique), and then use byte or message pipes accordingly? Example: # Byte pipe used because Win32Program.exe does not link with CYGWIN1.DLLL cat Testfile.txt | ./Win32Program.exe # Message pipe used because grep links with CYGWIN1.DLL cat Testfile.txt | grep Hello # Message pipe still used because the program we are calling links with CYGWIN1.DLL ./Win32Program.exe | grep Hello If that is still not POSIX compliant enough for the sending program (e.g. cat in this example), then I suppose Cygwin could automatically insert a message-to-byte pipe shim like the author proposes. That seems pretty kludgy though. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Wed, May 09, 2012 at 07:21:34PM -, James Johnston wrote: On Wed, May 09, 2012 at 11:59:58AM -0700, Linda Walsh, the dead discussion resurrector wrote: I can't say with 100% certainty, but I would bet with 90+% confidence that this is a bug in MS's libraries -- they cheat and use a null/0 byte read as a signal for end of file rather than sending out-of-band information that *nix supports. Maybe I'm na??ve and this is harder than it looks, but couldn't Cygwin determine if the program being piped to links with CYGWIN1.DLL (or similar detection technique), and then use byte or message pipes accordingly? Example: You're being naive. Of course this is harder than it looks. Or, if it really is so easy, why haven't you submitted a patch or at least a proof of concept? # Byte pipe used because Win32Program.exe does not link with CYGWIN1.DLLL cat Testfile.txt | ./Win32Program.exe In the above, the series of events are likely something like: 1) Shell creates a pipe 2) Shell starts cat with its stdout set to the output side of the pipe, first closing the input side. 3) Shell starts ./Win32Program.exe with its input set to the input side of the pipe, first closing the output side. 4) Shell closes pipe. 5) Processes execute. The cygwin DLL can't intercede here. It's not some superior process controlling I/O. It's just a DLL used by two programs. cat is writing to the stdout that it inherited from the shell. It doesn't know (and shouldn't know) what's on the other end of the pipe and that it should be changing the mode of the pipe. Please stop thinking that there is a simple solution that you can intuit without fully understanding how Cygwin works. 1.7.15 will contain a new CYGWIN option pipe_byte which will force non-tty, non-fifo pipes to use byte mode. That's about as far as we're willing to modify Cygwin for this apocalyptic problem. cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On 4/27/12 10:27 PM, Christopher Faylor wrote: The above comment shows an and relationship here. Message type pipes more closely mimic Linux (UNIX) pipe behavior AND are definitely required for ptys. Yeah, but because message pipes break other programs. Cygwin has only been using message pipes since 23-Oct-11 (1.134), and it got along fine before that, ICANON aside. If Cygwin programs don't actually _need_ message pipes in the general case, Cygwin should use byte pipes instead. signature.asc Description: OpenPGP digital signature
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On 26 April 2012 22:18, James Johnston jam...@motionview3d.com wrote: I have run into an issue with Cygwin. This is arguably not a bug in Cygwin, but in other platform runtime libraries. Never-the-less, the symptoms occur only with Cygwin and not the Windows command prompt. So, from a practical standpoint, Cygwin is broken. This is almost certainly the same issue mentioned in the 1.7.10-1.7.13 : output from .NET programs does not get through pipeline to a visual c++ program thread started by cygwin AT alanhowells.e4ward.com last week. It's also related to the issue I raised over a month ago, titled Can't reliably redirect standard output from C# program in recent Cygwin. Thank you! It is good to see someone else reproduce this! snip I am providing several sample programs that can be compiled in Visual C# 2008 / .NET Framework 3.5, or Visual C++ 2008. They should demonstrate the problem 100% of the time on any system, because appropriate Sleep() calls are made. Of course, why didn't I think of using Sleep?? When adding sleep to my test programs, the test where it failed on the first attempt reliably fails. I think I hinted at an issue with the VC++ runtime as I proved that using the Win32 API calls succeeded, but unfortunately didn't follow that through. I tested with VC++ 2010 runtime and that still has the issue. snip Christopher Faylor says Nope, it won't always be that because I get what's expected. I built the C++ files using mingw g++. Although I actually expected the reader to honor the null byte, it did not. Perhaps you are using a different version of Windows than I am or a different runtime. But the point by James is that it is the VC++ runtimes (and .NET runtimes) that think that a null write is EOF. Other runtimes may not (e.g. mingw). I agree with James that the runtimes are probably buggy BUT I also agree that cygwin should be able to have a handle these scenarios. I have also found a workaround, by piping through a cygwin command, you lose the null writes, a good one to use is cat so: $ ./SenderC | cat | ./ReceiverCPP $ Got line Hello world! $ Got end-of-file snip Unfortunately, this isn't a real workaround unless you remember to use it all the time. Alan -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
-Original Message- Sent: Friday, April 27, 2012 00:17 Subject: Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping) Nope, it won't always be that because I get what's expected. I built the C++ files using mingw g++. Although I actually expected the reader to honor the null byte, it did not. Perhaps you are using a different version of Windows than I am or a different runtime. As I stated, I did two demo programs using Visual C++ 2008 and .NET Framework 3.5. I did not test mingw. This is on Windows 7 64-bit, fully up-to-date. Perhaps your mingw runtime doesn't have this bug, in which case, good for them! But that doesn't change the fact that not many people use mingw in comparison to Visual C++ or .NET Framework 3.5. And I'm not sure how you can complain that you can't reproduce it if you don't use similar tools to what was used to originally reproduce the issue. (Did I really need to preface my phrase of will always be with conditions that you use Visual C++ / .NET Framework? I thought it would be obvious, especially since I was referring to bugs in their runtimes...) If you don't want to install a full Visual Studio environment, you can use the C# compiler on any computer with the .NET Framework installed. For example, framework 3.5's compiler is at C:\Windows\Microsoft.NET\Framework\v3.5\csc.exe. The free Windows SDK contains a Visual C++ compiler in it. And there are free express versions of Visual Studio available if you want an IDE. You can test these sorts of things without forking over big bucks for paid Visual Studio licenses. What you are seeing may be because Cygwin was changed to use message- type pipes a couple of revisions ago. This is not going to change. The change was adopted to fix a problem with Cygwin programs and those are obviously our #1 priority. In which case, it would be a shame for Cygwin to break compatibility with the vast majority of Windows programs out there, which are probably using the two most common development platforms on Windows: Visual C++ and the .NET Framework. That sounds insane; I can't believe the developers would make that call. It seems crazy that they would not even regularly test with software compiled using non-free compilers. (May as well rip out the cygpath tool and friends; why make pretenses about supporting interoperability with Windows programs?) I liked Cygwin since I thought it offered compatibility with software designed for both Linux and Windows. Apparently the fine print is that it's now only supporting compatibility with stuff linked to cygwin1.dll, and standard Windows software need not apply - since compatibility with Windows software is not a real goal? Unfortunately, now I can't think of any good solutions that support both Linux and Windows software, whose developers place a priority on compatibility with both. If Cygwin doesn't support Windows software compiled using very common platforms like Visual C++/.NET, then nobody really does that I can think of. (Sure, there is Windows Services for UNIX, but we all know the mess that brings - we're using Cygwin for a reason... And MSYS isn't really a replacement for Cygwin either; they don't pretend to be UNIX on Windows.) More to the point, I didn't have these problems with Cygwin 1.7.9. What problem with Cygwin programs are you referring to? Because I never had any - 1.7.9 worked great. All I know, is I upgraded to a more recent version and boom - everything stopped working. Why were message pipes used? And what source code files exactly are you talking about? Cygwin has a lot of source code, but if I knew exactly where to look or better understood what it is doing and why, maybe I could offer some more constructive ideas - having done some low-level programming with pipes on Windows myself. (At first glance, I don't really see how message pipes vs byte pipes would make a difference.) And from a practical standpoint - again, while ideally it would be nice for Microsoft to go back and fix every single version of Visual C++ and .NET Framework that have this bug (which probably dates back to when these products were first created), and for Windows app developers to then redeploy their apps with these fixed runtimes - I think the chances of that happening in the next few years are zero. I'm a practical/pragmatic person, which is why I suggested working around their bugs. (As a regular reader of Raymond Chen's blog, apparently Windows works around app bugs, too...) There's no way that Cygwin could know to skip a call to WriteFile(). Cygwin doesn't interpose itself in the middle of a pipe. That would be truly disastrous. If it somehow looked at every pipe read/write rather than just allowing I/O to flow from one end to the other, the mailing list would be even more filled with people complaining that Cygwin is slow. If there is no other way, then maybe it should
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Apr 26 21:18, James Johnston wrote: I have run into an issue with Cygwin. This is arguably not a bug in Cygwin, but in other platform runtime libraries. Never-the-less, the symptoms occur only with Cygwin and not the Windows command prompt. So, from a practical standpoint, Cygwin is broken. This is almost certainly the same issue mentioned in the 1.7.10-1.7.13 : output from .NET programs does not get through pipeline to a visual c++ program thread started by cygwin AT alanhowells.e4ward.com last week. It's also related to the issue I raised over a month ago, titled Can't reliably redirect standard output from C# program in recent Cygwin. To summarize: when piping output from one program to another (i.e. running A | B command to pipe standard output from A to B's standard input), Cygwin passes through ALL writes performed by a program to standard output (and probably error), including null writes. What I don't grok is this: In your example, A and B are both native (== non-Cygwin) applications. As I already mentioned in another mail related to the .Net problem, while the Cygwin shell running at the time creates the pipe, it doesn't handle read and write calls on the pipe. It only creates the two native child processes and passes on the pipe handles. So in the end, there is no Cygwin process involved while the native processes are doing their reads and writes. It's all in the hands of these processes. There's no third process in the middle which handles the passing of information from A to B. So, what do you expect Cygwin to do? Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Fri, Apr 27, 2012 at 10:08:23AM +0100, cygwin-alanhowells.e4ward.com wrote: Christopher Faylor says Nope, it won't always be that because I get what's expected. I built the C++ files using mingw g++. Although I actually expected the reader to honor the null byte, it did not. Perhaps you are using a different version of Windows than I am or a different runtime. But the point by James is that it is the VC++ runtimes (and .NET runtimes) that think that a null write is EOF. Other runtimes may not (e.g. mingw). I was reporting what I found with the tools I have at hand. My experience refutes the use of the word always. The whole point of my mentioning a different runtime was to show that I understood that others' mileage varies. I wasn't saying that I don't believe what's being reported. I agree with James that the runtimes are probably buggy BUT I also agree that cygwin should be able to have a handle these scenarios. Your complete agreement with each other is not going to have much effect. Cygwin source code is not changed by voting. cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On 2012-04-27 AM 9:17, Christopher Faylor wrote: There's no way that Cygwin could know to skip a call to WriteFile(). Cygwin doesn't interpose itself in the middle of a pipe. That would be truly disastrous. If it somehow looked at every pipe read/write rather than just allowing I/O to flow from one end to the other, the mailing list would be even more filled with people complaining that Cygwin is slow. Maybe i can measure how much it slowed down after applying the workaround of unworkable runtime of some non-free software. -- Regards. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Fri, Apr 27, 2012 at 02:26:13PM -, James Johnston wrote: On Thu, Apr 26, 2012 at 08:17:18PM -0400, Christopher Faylor wrote: Nope, it won't always be that because I get what's expected. I built the C++ files using mingw g++. Although I actually expected the reader to honor the null byte, it did not. Perhaps you are using a different version of Windows than I am or a different runtime. As I stated, I did two demo programs using Visual C++ 2008 and .NET Framework 3.5. I did not test mingw. I did not say that you tested MinGW. I was telling you what I tested. However, as it turns out, I created the executable on Linux and my environment still ended up creating a Cygwin binary. Correctly rebuilding with MinGW shows the same behavior you reported. Sorry for getting that wrong. Btw, MinGW does not have its own runtime. This is on Windows 7 64-bit, fully up-to-date. Perhaps your mingw runtime doesn't have this bug, in which case, good for them! But that doesn't change the fact that not many people use mingw in comparison to Visual C++ or .NET Framework 3.5. And I'm not sure how you can complain that you can't reproduce it if you don't use similar tools to what was used to originally reproduce the issue. (Did I really need to preface my phrase of will always be with conditions that you use Visual C++ / .NET Framework? I thought it would be obvious, especially since I was referring to bugs in their runtimes...) I wasn't complaining that I could not reproduce the problem. I was stating a fact. And, again, to be clear, I was also not doubting that you could demonstrate a behavior. I can see it now too and I'm actually kind of relieved to see that it matches my expectations. Regardless of how many people use Visual C++ or .NET they are really not our target audience. It's nice when things work for people who don't want to use UNIX tools but they are not our primary focus. Fixing problems for people who want to use non-Cygwin stuff is not something that I find a high priority. If you want to a fix for this problem then you'd be best off if you supplied one yourself: http://cygwin.com/acronyms/#PTC Maybe I'm in the minority but I find long-winded impassioned pleas rather boring and off-putting. So, continuing in this vein is not going to be productive for you, at least as far as I'm concerned. cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Sat, Apr 28, 2012 at 12:11:35AM +0900, jojelino wrote: On 2012-04-27 AM 9:17, Christopher Faylor wrote: There's no way that Cygwin could know to skip a call to WriteFile(). Cygwin doesn't interpose itself in the middle of a pipe. That would be truly disastrous. If it somehow looked at every pipe read/write rather than just allowing I/O to flow from one end to the other, the mailing list would be even more filled with people complaining that Cygwin is slow. Maybe i can measure how much it slowed down after applying the workaround of unworkable runtime of some non-free software. Thanks for the offer but that presupposes that I'd actually be willing to go to the effort of coding something up. I'm really not. cgf -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
-Original Message- Sent: Friday, April 27, 2012 14:38 Subject: Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping) What I don't grok is this: In your example, A and B are both native (== non-Cygwin) applications. As I already mentioned in another mail related to the .Net problem, while the Cygwin shell running at the time creates the pipe, it doesn't handle read and write calls on the pipe. It only creates the two native child processes and passes on the pipe handles. So in the end, there is no Cygwin process involved while the native processes are doing their reads and writes. It's all in the hands of these processes. There's no third process in the middle which handles the passing of information from A to B. So, what do you expect Cygwin to do? Thanks for responding. I apologize for not fully understanding the problem earlier. In .NET, if you wanted to pipe the output of one process to another, you'd have to be exactly this third process as far as I can tell - the framework creates a pipe/stream for each process, so you'd need a loop to copy information from one pipe to another. To help understand how this is all working, I used API Monitor to spy and see how the Windows command prompt sets up its pipe. It's very simple: it can be simplified down to a call to CreatePipe to make an anonymous pipe. The two child processes are then spawned, with the pipe reader/writer handles being passed to CreateProcess. Unused pipe handles are closed in both the parent process and the child processes, so that the processes terminate correctly. I wrote a straight Win32 C program to spawn my child processes and direct output from one to another's input, using CreatePipe. The results were correct. What you are seeing may be because Cygwin was changed to use message-type pipes a couple of revisions ago. This is not going to change. The change was adopted to fix a problem with Cygwin programs and those are obviously our #1 priority. I will certainly not disagree with you that Cygwin programs need to work 100%! But surely there has got to be some solution that can achieve this while not widely breaking non-Cygwin programs..!? I'm actually surprised more people wouldn't be using Windows programs from Cygwin - to me, that is part of the point of using a system like Cygwin rather than just installing the Linux distro of choice - so I can mix/match both Windows and UNIX software. For me, both Windows and Cygwin software need to work - not just one or the other. While you have near-zero interest in running Windows programs from Cygwin - I would bet that does not describe a significant number of users. So to confirm your hypothesis, I modified my previously-described C program. It uses CreateNamedPipe followed by CreateFile to get reader/writer handles instead of making an anonymous pipe using CreatePipe. If I use byte mode, the results are still correct - apparently byte mode pipes won't pass through null writes. HOWEVER, if I switch to message mode - the results are incorrect - just like the bad results from Cygwin. Message mode pipes pass through null writes. So I think for sure, Cygwin's use of message pipes is breaking a lot of Windows software, because of the null writes. And ALSO additionally perhaps because of this: while reading MSDN today, I came across an interesting snippet that probably indicates a second source of problems I have not explicitly tested for, but would probably find: If the specified number of bytes to read is less than the size of the next message, the function reads as much of the message as possible before returning zero (the GetLastError function returns ERROR_MORE_DATA). The remainder of the message can be read using another read operation. While I did not test for this, I think this would also break a lot of Windows software. It's conceivable that the sender could write out a lot of data, while the receiver isn't reading anything. Then, if the reader tries to read but with a small buffer, according to the docs it will get ERROR_MORE_DATA. Most runtimes don’t expect ReadFile to fail with ERROR_MORE_DATA. For example, the decompiled .NET Framework code I read would throw an exception, as it only gracefully handles ERROR_BROKEN_PIPE. Probably others like the Visual C++ runtime will treat it as EOF or set the failbit. If there's interest I expect I could write a test case. (e.g. run cat to pipe a gigantic text file to a Windows program that takes a long time to process each line). So I have some questions... 1. Why on earth did Cygwin start using message pipes? adopted to fix a problem doesn't help... Googling site:Cygwin.com message pipe - didn't yield anything either. I do know older Cygwin versions worked fine, and I had no problem with piping data to/from both Cygwin and non-Cygwin programs. I am sure it was changed
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On 4/27/2012 3:38 PM, James Johnston wrote: So I think for sure, Cygwin's use of message pipes is breaking a lot of Windows software, because of the null writes. And ALSO additionally perhaps because of this: while reading MSDN today, I came across an interesting snippet that probably indicates a second source of problems I have not explicitly tested for, but would probably find: If the specified number of bytes to read is less than the size of the next message, the function reads as much of the message as possible before returning zero (the GetLastError function returns ERROR_MORE_DATA). The remainder of the message can be read using another read operation. Confusingly, a PIPE_TYPE_MESSAGE pipe's read end can be in either PIPE_READMODE_MESSAGE or PIPE_READMODE_BYTE mode. You're describing the behavior of PIPE_READMODE_MESSAGE; Cygwin's pipes are in PIPE_READMODE_BYTE, so they don't have this particular problem --- although the null-write problem remains. 1. Why on earth did Cygwin start using message pipes? adopted to fix a problem doesn't help... Googling site:Cygwin.com message pipe - didn't yield anything either. I do know older Cygwin versions worked fine, and I had no problem with piping data to/from both Cygwin and non-Cygwin programs. I am sure it was changed for a good reason, but would someone be willing to offer an explanation for why message pipes are now used? Do they have problems reading from a byte pipe? Writing to a byte pipe? Both? Under what conditions? IIRC, it had something to do with preserving message boundaries at the TTY later. Again IIRC (and I only faintly recall this part) the PTY layer has to deliver one line at a time to readers. If a client calls read(2) with a giant buffer, he's still supposed to get back only a single line at a time. A message-mode pipe seems like a good way to provide these semantics because it preserves write boundaries --- so if you write only a line at a time, the reader reads only a line at a time. It's a shame that this approach causes problems. Why can't only pseudoterminal pipes be created as PIPE_TYPE_MESSAGE? Cygwin already has special logic to tell whether a handle it's opening is a pty and to treat it specially in that case. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Fri, Apr 27, 2012 at 05:25:31PM -0700, Daniel Colascione wrote: IIRC, it had something to do with preserving message boundaries at the TTY later. Again IIRC (and I only faintly recall this part) the PTY layer has to deliver one line at a time to readers. If a client calls read(2) with a giant buffer, he's still supposed to get back only a single line at a time. A message-mode pipe seems like a good way to provide these semantics because it preserves write boundaries --- so if you write only a line at a time, the reader reads only a line at a time. It's a shame that this approach causes problems. Why can't only pseudoterminal pipes be created as PIPE_TYPE_MESSAGE? Cygwin already has special logic to tell whether a handle it's opening is a pty and to treat it specially in that case. Use the source code, Jake. From pipe.cc: Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE. This *seems* to more closely mimic Linux pipe behavior and is definitely required for pty handling since fhandler_pty_master writes to the pipe in chunks, terminated by newline when CANON mode is specified. The above comment shows an and relationship here. Message type pipes more closely mimic Linux (UNIX) pipe behavior AND are definitely required for ptys. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
I have run into an issue with Cygwin. This is arguably not a bug in
Cygwin, but in other platform runtime libraries. Never-the-less, the
symptoms occur only with Cygwin and not the Windows command prompt. So,
from a practical standpoint, Cygwin is broken. This is almost certainly
the same issue mentioned in the 1.7.10-1.7.13 : output from .NET programs
does not get through pipeline to a visual c++ program thread started by
cygwin AT alanhowells.e4ward.com last week. It's also related to the issue
I raised over a month ago, titled Can't reliably redirect standard output
from C# program in recent Cygwin.
To summarize: when piping output from one program to another (i.e. running
A | B command to pipe standard output from A to B's standard input),
Cygwin passes through ALL writes performed by a program to standard output
(and probably error), including null writes. Cygwin passing through these
null writes is a big problem, because multiple runtimes - I tested .NET
Framework 3.5 and the Visual C++ 2008 runtime - cannot properly handle null
writes that they receive on standard input. (Recall that a null write would
be when a program calls WriteFile Win32 API to write 0 bytes to standard
output or standard error.) While a null write appears nonsensical, every
single .NET program that uses the Console class to write to standard
output/error will do a null write, as .NET does this to verify the stream is
OK. Other software could easily decide to write zero bytes to standard
output as well (e.g. if outputting an empty string).
I think these are bugs in the runtimes that handle standard input, because
the documentation for ReadFile clearly states that if the handle is a pipe,
the call succeeds, and zero bytes are returned - then it is a null write on
the pipe and does NOT signify end-of-file. Instead, end-of-file is
signified by ERROR_BROKEN_PIPE. Yet, these runtimes erroneously handle the
null write as an end-of-file on standard input anyway, causing the software
using the runtime to malfunction. For example, I used Reflector to
decompile the Read method in the Stream class that handles standard
input/output in the .NET Framework. It's very obvious that it calls
ReadFile, and a successful ReadFile that returns zero bytes is treated as
end-of-file.
An alternative explanation might be that all this behavior is by design, and
Cygwin is buggy. But I haven't found anything in MSDN that would justify
that. (For example, documentation stating that a redirected standard input
may never have a null write would mean that runtimes could safely assume
that standard input won't have null writes, and Cygwin in error for doing a
null write. However, as an example, the STARTUPINFO structure documentation
imposes no such requirements when describing the hStdInput handle.)
These runtimes (.NET Framework and Visual C++ runtime) are in wide use. I
don't think it is realistic to expect any fixes for them any time soon:
Microsoft would need to fix both of these runtimes, and then application
vendors would need to use them. That will take years. Therefore,
realistically I hope Cygwin can work around them.
I think a workable fix would be for Cygwin not to pass through null writes
it receives on an output/error pipe. For example, somewhere in Cygwin I
assume there is a loop that calls ReadFile to read the redirected standard
output from the first program, and then calls WriteFile to send this output
to the second program's standard input. If the call to WriteFile was
skipped if it would write zero bytes (i.e. so Cygwin doesn't do null writes
itself), I think it would fix the problem and work around all these buggy
runtimes.
I am providing several sample programs that can be compiled in Visual C#
2008 / .NET Framework 3.5, or Visual C++ 2008. They should demonstrate the
problem 100% of the time on any system, because appropriate Sleep() calls
are made. This was reproduced on a Cygwin 1.7.14-2 system that I updated
this morning. Sample programs are divided as follows:
* A Sender program performs a null write on standard output, and then
writes a normal line of text.
* A Receiver program attempts to write all received lines of text.
Source code is below; directions for testing and test results at end of
e-mail following code:
== SenderCS.cs: Sender program in Visual C# 2008 ==
class SenderCS {
static void Main(string[] args) {
/* Notice how this is such a basic program */
System.Threading.Thread.Sleep(1000); /* wait for pipes to
set up and for receiving app to block on first ReadFile call */
/* This will do a null write; it could be left out and the
problem would still occur because any WriteLine() call will do it. */
/* I include it so that the null write can be placed between
two Sleep() calls. */
System.Console.Write();
System.Threading.Thread.Sleep(1000); /* wait for receiving
end to
Re: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)
On Thu, Apr 26, 2012 at 09:18:27PM -, James Johnston wrote:
== SenderC.c: Sender program in Visual C++ 2008 ==
#include windows.h
int main() {
char * test = Hello world!\n;
DWORD written;
HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE); /* Get standard output
file handle */
Sleep(1000); /* wait for pipes to set up and for receiving app to
block on first ReadFile call */
WriteFile(h, test, 0, written, NULL); /* do null write */
Sleep(1000); /* wait for receiving app to get the null write */
WriteFile(h, test, lstrlenA(test), written, NULL); /* print hello
message */
return 0;
}
== ReceiverCPP.cpp: Receiver program in Visual C++ 2008 that
demonstrates bug in VC++ 2008 runtime / STL ==
#include iostream
#include string
using namespace std;
int main() {
/* you have to use a retry loop, for exact same reasons given for C#
receiver program:
there is no way to tell difference between end-of-file and null
write. */
for (int i = 0; i 10; i++) {
string str;
/* BUG: cin will indicate end-of-file on a null write. */
getline(cin, str);
if (cin.eof()) {
cout Got end-of-file endl;
} else {
cout Got line str endl;
}
cin.clear(); /* future getline calls will always immediately
fail without attempting another read unless we clear EOF/fail flags */
}
return 0;
}
== Test results ==
The test programs are designed so that they can be run in any combination
from the command prompt. The output from a sender is piped to the input of
a receiver. Each combination delivers identical output to the other
combinations:
* ./SenderCS | ./ReceiverCS
* ./SenderCS | ./ReceiverCPP
* ./SenderC | ./ReceiverCS
* ./SenderC | ./ReceiverCPP
Output from Cygwin will always be:
Got end-of-file
Got line Hello world!
Got end-of-file
Got end-of-file
snip
Nope, it won't always be that because I get what's expected. I built
the C++ files using mingw g++. Although I actually expected the reader
to honor the null byte, it did not. Perhaps you are using a different
version of Windows than I am or a different runtime.
What you are seeing may be because Cygwin was changed to use
message-type pipes a couple of revisions ago. This is not going to
change. The change was adopted to fix a problem with Cygwin programs
and those are obviously our #1 priority.
This is wrong, because the program received end-of-file before it was
actually at the end of the input stream, due to the bug in its runtime's
handling of return values from ReadFile API. I did not do any tests using
standard error, but I assume Cygwin redirects standard error in the same way
it redirects standard output, in which case it would have the same problem.
[moved]
I think a workable fix would be for Cygwin not to pass through null writes
it receives on an output/error pipe. For example, somewhere in Cygwin I
assume there is a loop that calls ReadFile to read the redirected standard
output from the first program, and then calls WriteFile to send this output
to the second program's standard input. If the call to WriteFile was
skipped if it would write zero bytes (i.e. so Cygwin doesn't do null writes
itself), I think it would fix the problem and work around all these buggy
runtimes.
There's no way that Cygwin could know to skip a call to WriteFile().
Cygwin doesn't interpose itself in the middle of a pipe. That would be
truly disastrous. If it somehow looked at every pipe read/write rather
than just allowing I/O to flow from one end to the other, the mailing
list would be even more filled with people complaining that Cygwin is
slow.
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
