Re: GNU coreutils does not work on Cygwin because of freopen() ?
dazjorz wrote: Hey all, I'm not very great with the inner workings of GNU or Cygwin libc, so please bear with me if I make any mistakes or misunderstandings. I was trying to get GNU coreutils to work on Cygwin. Maybe this seems weird to you guys, because Cygwin has coreutils, but there are a few reasons: - Cygwin didn't have md5sum, GNU coreutils did (and I need md5sum) http://cygwin.com/cgi-bin2/package-grep.cgi?grep=bin%2Fmd5sum Did you miss this? - It should work, and Because I Can is always a valid reason in UNIX world - I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is my first try; getting coreutils from Cygwin may be done later. However, I noticed there are some very weird bugs happening. See for example, this bug in cat (don't look at the title of the page, I blamed bash first) http://paster.dazjorz.com/?p=3845 Your example works fine for me with the latest coreutils. Perhaps you're working with a buggy locally built version? -- Larry Hall http://www.rfk.com RFK Partners, Inc. (508) 893-9779 - RFK Office 216 Dalton Rd. (508) 893-9889 - FAX Holliston, MA 01746 _ A: Yes. Q: Are you sure? A: Because it reverses the logical flow of conversation. Q: Why is top posting annoying in email? -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
Larry Hall (Cygwin) wrote: dazjorz wrote: Hey all, I'm not very great with the inner workings of GNU or Cygwin libc, so please bear with me if I make any mistakes or misunderstandings. I was trying to get GNU coreutils to work on Cygwin. Maybe this seems weird to you guys, because Cygwin has coreutils, but there are a few reasons: - Cygwin didn't have md5sum, GNU coreutils did (and I need md5sum) http://cygwin.com/cgi-bin2/package-grep.cgi?grep=bin%2Fmd5sum Did you miss this? Hmm. I guess. Oh well, I didn't have md5sum before, I think :/ - It should work, and Because I Can is always a valid reason in UNIX world - I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is my first try; getting coreutils from Cygwin may be done later. However, I noticed there are some very weird bugs happening. See for example, this bug in cat (don't look at the title of the page, I blamed bash first) http://paster.dazjorz.com/?p=3845 Your example works fine for me with the latest coreutils. Perhaps you're working with a buggy locally built version? Did you try with Cygwin coreutils or GNU coreutils? :) By the way, list, I've been looking for some other things. This problem is caused by: - cat calling freopen to make the handle binary (shouldn't it be binary in the first place? shouldn't it check if it's already opened in binary mode?) - freopen() not checking if O_APPEND is set, etc. (is it possible to check this?) I'll write a test case for freopen(), that shows O_APPEND is being missed, causing this bug. Sjors -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
dazjorz wrote: - Cygwin didn't have md5sum, Cygwin does have md5sum in coreutils. You can see this at the cygwin package list: http://cygwin.com/packages/ by entering 'md5sum.exe' in the search box. - It should work, and Because I Can is always a valid reason in UNIX world However, you may have to recreate all the cygwin-specific patches that the cygwin bash maintainer hasn't managed to send upstream yet... - I'm trying to bootstrap Debian on Cygwin, and coreutils from Debian is my first try; getting coreutils from Cygwin may be done later. Dunno if your project makes sense. You'll have to end up porting everything where the differences in the underlying OS show through... which is basically what Cygwin package maintainers have already done for most of this stuff. Do you mean you're trying to cross-compile Debian on Cygwin, or do you actually want to build the whole Debian distro under Cygwin to run on Windows rather than natively under a linux kernel? However, I noticed there are some very weird bugs happening. See for example, this bug in cat (don't look at the title of the page, I blamed bash first) http://paster.dazjorz.com/?p=3845 A friend of mine and me have been trying to figure out what causes this. First, we tested if it wasn't a bug in open() + fork(). This seemed not to be the case. So we started looking further, and we're down to GNU `cat` calling freopen(). This is not POSIX-compliant and can cause problems like this. ?? Are you using an old version?? It should be fixed by now. http://www.mail-archive.com/bug-coreut...@gnu.org/msg10188.html `cat` and other coreutils tools only call freopen() if O_BINARY is defined on this platform. It's defined on Cygwin, because Cygwin makes a difference between O_TEXT and O_BINARY. 1. Why does Cygwin distinguish between O_TEXT and O_BINARY? Shouldn't this difference be removed from the Linux API side of Cygwin? Nope. Cygwin runs on windows, which uses CR-LF for line endings, but Cygwin is POSIX-compliant and uses LF internally. This is exactly what O_TEXT was invented for: it translates LF-CRLF on write and CRLF-LF on read. On Linux platforms, the native format is LF line-ends, so O_TEXT and O_BINARY are the same, but not here. 2. I don't know why coreutils is calling freopen() at all. Should I patch it out completely, or add something like an #ifndef __CYGWIN__ around the pieces of code in question? Well, if you really want to build coreutils under cygwin, why not use setup.exe to install the corresponding source package, which comes with all these problems pre-solved for you? cheers, DaveK -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Sjors Gielen wrote: Did you try with Cygwin coreutils or GNU coreutils? :) Cygwin's coreutils *is* GNU coreutils, with some patches which you will find necessary. Yaakov -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREIAAYFAklzorEACgkQpiWmPGlmQSOl0ACg5+4mOgMwaTkMMkQTirY2Xlwz xJ4An3OO1gl/7fl+GBzay8urXpDaJJDS =1fhI -END PGP SIGNATURE- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
Yaakov (Cygwin/X) wrote: Sjors Gielen wrote: Did you try with Cygwin coreutils or GNU coreutils? :) Cygwin's coreutils *is* GNU coreutils, with some patches which you will find necessary. [snip] Dave Korn wrote: However, you may have to recreate all the cygwin-specific patches that the cygwin bash maintainer hasn't managed to send upstream yet... [snip] ?? Are you using an old version?? It should be fixed by now. http://www.mail-archive.com/bug-coreut...@gnu.org/msg10188.html Oooh. Can I get a list of patches the Cygwin project has already applied to packages? :) So this bug was fixed in Cygwin's version of GNU coreutils, but not in Debian's version of GNU coreutils. Has a patch for that been sent upstream? I took GNU coreutils from Debian sid, which should be almost as up to date as it gets... Dunno if your project makes sense. You'll have to end up porting everything where the differences in the underlying OS show through... which is basically what Cygwin package maintainers have already done for most of this stuff. Do you mean you're trying to cross-compile Debian on Cygwin, or do you actually want to build the whole Debian distro under Cygwin to run on Windows rather than natively under a linux kernel? I'm trying to create a full Cygwin port for Debian. This means the base system packages, et cetera. Let's just keep it at I love apt and dpkg for package managers. Nope. Cygwin runs on windows, which uses CR-LF for line endings, but Cygwin is POSIX-compliant and uses LF internally. This is exactly what O_TEXT was invented for: it translates LF-CRLF on write and CRLF-LF on read. On Linux platforms, the native format is LF line-ends, so O_TEXT and O_BINARY are the same, but not here. But if Cygwin uses LF internally, why does it translate to CRLF on write? I remember this installation option, asking if it should use CRLF or LF for writing files. I think I chose LF only, because that's the Unix way of doing it. Do I misunderstand here? Currently, when I write a file in vim and save it, it has LF line endings anyway. If it is saved with LF line endings, there's no difference between O_TEXT and O_BINARY, right? So why not skip this at all and make *every* fd O_BINARY by default? 2. I don't know why coreutils is calling freopen() at all. Should I patch it out completely, or add something like an #ifndef __CYGWIN__ around the pieces of code in question? Well, if you really want to build coreutils under cygwin, why not use setup.exe to install the corresponding source package, which comes with all these problems pre-solved for you? That would, of course, defeat the idea of bootstrapping Debian on Cygwin. ;) Sjors -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
On Sun, Jan 18, 2009 at 5:11 PM, Sjors Gielen wrote: I'm trying to create a full Cygwin port for Debian. This means the base system packages, et cetera. Let's just keep it at I love apt and dpkg for package managers. Keep in mind that apt and dpkg will never be able to work completely on Windows, since they were written with some fundamental assumptions of how the system will behave that don't hold on Windows. dpkg expects to be able to update itself, or the shell it's running in, by just writing a new file with the same name to the same directory, and this won't work on windows; you can't overwrite an executable that is currently being executed. ~Matt -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
Hello all, * Sjors Gielen mailingl...@dazjorz.com wrote: Yaakov (Cygwin/X) wrote: Sjors Gielen wrote: Did you try with Cygwin coreutils or GNU coreutils? :) Cygwin's coreutils *is* GNU coreutils, with some patches which you will find necessary. [snip] Dave Korn wrote: However, you may have to recreate all the cygwin-specific patches that the cygwin bash maintainer hasn't managed to send upstream yet... [snip] ?? Are you using an old version?? It should be fixed by now. http://www.mail-archive.com/bug-coreut...@gnu.org/msg10188.html Oooh. Can I get a list of patches the Cygwin project has already applied to packages? :) So this bug was fixed in Cygwin's version of GNU coreutils, but not in Debian's version of GNU coreutils. Has a patch for that been sent upstream? I took GNU coreutils from Debian sid, which should be almost as up to date as it gets... But that's not a proper fix anyway. cat can't just decide to reopen the file descriptor of the standard output. That even breaks the first sentence of POSIX: The cat utility shall read files in sequence and shall write their contents to the standard output in the same sequence. If it says the standard output doesn't mean you can reopen the output descriptor. This means you end up with two different file descriptors. They can point to the same file, but they are not the same file descriptor. They fixed it by using SET_BINARY(), but not on Cygwin, because I assume SET_BINARY() is a Windows-thing, not Cygwin. I can't really understand why Cygwin just doesn't force O_BINARY to be set. File descriptors on UNIX are supposed to display raw file contents. There shouldn't be any (newline) processing. It makes no sense to cripple Cygwin only because Notepad doesn't support newlines without the carriage return. -- Ed Schouten e...@80386.nl WWW: http://80386.nl/ pgp5XghmR38PI.pgp Description: PGP signature
Re: GNU coreutils does not work on Cygwin because of freopen() ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Sjors Gielen on 1/18/2009 3:11 PM: cygwin bash maintainer hasn't managed to send upstream yet... [snip] ?? Are you using an old version?? It should be fixed by now. http://www.mail-archive.com/bug-coreut...@gnu.org/msg10188.html Oooh. Can I get a list of patches the Cygwin project has already applied to packages? :) Yes. Use setup.exe, and select the src checkbox next to coreutils (or any other package where you want to know what went into the cygwin version). For coreutils, that will get you all of my cygwin-specific patches to coreutils. So this bug was fixed in Cygwin's version of GNU coreutils, but not in Debian's version of GNU coreutils. Has a patch for that been sent upstream? I took GNU coreutils from Debian sid, which should be almost as up to date as it gets... Debian coreutils does not care about cygwin-specific patches, and the freopen in question, while it is indeed buggy upstream, only affects cygwin and other platforms that define O_BINARY. Since that excludes debian, then debian's fork of coreutils has no reason to patch it. - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklzq6cACgkQ84KuGfSFAYC0ogCfSvjhCtwLoUbTlLV7S4GR2HFV wpkAoLdjblp/mz+7OhrIcIk3jZTDv01E =xvRw -END PGP SIGNATURE- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Ed Schouten on 1/18/2009 3:19 PM: But that's not a proper fix anyway. cat can't just decide to reopen the file descriptor of the standard output. That even breaks the first sentence of POSIX: The cat utility shall read files in sequence and shall write their contents to the standard output in the same sequence. If it says the standard output doesn't mean you can reopen the output descriptor. This means you end up with two different file descriptors. They can point to the same file, but they are not the same file descriptor. That's where you are wrong. freopen can be passed a NULL pointer instead of a filename in order to reuse the same file descriptor, but with an implementation-defined different mode. In cygwin's case, this means you can use freopen to swap a file between text and binary mode. They fixed it by using SET_BINARY(), but not on Cygwin, because I assume SET_BINARY() is a Windows-thing, not Cygwin. SET_BINARY is an upstream coreutils thing designed for cygwin and other platforms that have O_BINARY in the first place (Debian doesn't). It is a macro that expands to either the non-standard setmode(), or the standardized (but implementation-dependent) freopen(). I can't really understand why Cygwin just doesn't force O_BINARY to be set. File descriptors on UNIX are supposed to display raw file contents. That's exactly what that freopen is doing - it is forcing O_BINARY to be set, regardless of how stdin was inherited. - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklzrKgACgkQ84KuGfSFAYCZzACeJcwTKhDejFWZLP8LkhKRkv/4 bj0AniqGtXHep5FcUIavGtl2YOWeSRwQ =aiau -END PGP SIGNATURE- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: GNU coreutils does not work on Cygwin because of freopen() ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Dave Korn on 1/18/2009 2:42 PM: However, you may have to recreate all the cygwin-specific patches that the cygwin bash maintainer hasn't managed to send upstream yet... Not only that, but some of those patches will never be sent upstream, because they are too platform-specific. As for this particular issue of freopen, I'm dragging my feet - the cygwin patch works, and no other mainstream platform is affected, so even though I have _already_ pointed out upstream that the current freopen use is buggy, I haven't bothered patching things upstream yet (I only have so much free time). http://lists.gnu.org/archive/html/bug-coreutils/2008-10/msg00199.html However, I noticed there are some very weird bugs happening. See for example, this bug in cat (don't look at the title of the page, I blamed bash first) http://paster.dazjorz.com/?p=3845 Yes, this is a known upstream bug in coreutils, but which only affects cygwin, and which is already fixed on cygwin, so there is very little need or desire to spend time fixing it upstream. - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklzrW4ACgkQ84KuGfSFAYBiMwCcCmf54S2kJPf8e/cO79NVVkFi Xz0AoKBlI79e4GNO/wLOHK7fv9PhOYec =PekT -END PGP SIGNATURE- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
