Re: 3.5.x regression: misquoting command line arguments from native processes

[David Allsopp via Cygwin]

Hi Corinna,

> On Apr  9 22:38, Corinna Vinschen via Cygwin wrote:
> > On Apr  3 16:53, David Allsopp via Cygwin wrote:
> > > I have what appears to be a regression in Cygwin 3.5.0 which, owing to
> > > a CI system lagging behind, we've only just discovered.
> > > [...]
> > > $ ./t.exe 'C:\Devel\реализация-mingw64\flexdll\flexdll_mingw64.o'
> > > stat: cannot stat
> > > '"C:\Devel\'$'\360\237\220\253''реализация-mingw64\flexdll\flexdll_mingw64.o':
> > > No such file or directory
> >
> > Thanks a lot for the STC!
> >
> > I think I fixed that for 3.5.4.  I pushed a patch and the test release
> > cygwin-3.6.0-0.115.g579064bf4d40 is just building and should be ready
> > for testing in an hour or two.
> >
> > Please give it a try.
>
> Sorry for nagging, but do you have some feedback, be it bad or good?

Sorry for having needed nagging! It does indeed fix it, thank you -
our smoke-test Bactrian camels can be restored 

All best,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

3.5.x regression: misquoting command line arguments from native processes

[David Allsopp via Cygwin]

I have what appears to be a regression in Cygwin 3.5.0 which, owing to
a CI system lagging behind, we've only just discovered.

In order to torture our Unicode support, OCaml's Windows CI compiles
its sources in C:\projects\реализация-mingw64 (that's a directory
under C:\projects with the camel emoji U+1F42A followed, I am told, by
the word "implementation" in Russian then with the name of the "port"
of OCaml appended in ASCII).

AppVeyor updated to Cygwin 3.5.1 at the weekend and now, during the
build, we end up with the following kind of error seen in, say,
https://ci.appveyor.com/project/avsm/ocaml/builds/49538801#L711. What
is happening here is that Cygwin's make has passed
C:\projects\реализация-mingw64\byte\bin\flexdll_mingw64.o to a
non-Cygwin program (flexlink.exe) which has called back to Cygwin's
x86_64-w64-mingw32-gcc ultimately via CreateProcess and which has then
failed.

I have reduced this chain to the following program, which takes
argv[1] and calls Cygwin's stat program using _wsystem with argv[1]
"-quoted:

#include 
#include 

int wmain(int argc, wchar_t **argv) {
  int len = 2 + 6 + wcslen(argv[1]);
  wchar_t *cmd = (wchar_t *)malloc(len * sizeof(wchar_t));
  wcscpy(cmd, L"stat ");
  cmd[5] = L'"';
  cmd[6] = 0;
  wcscat(cmd, argv[1]);
  cmd[len - 1] = L'"';
  cmd[len] = 0;
  _wsystem(cmd);
}

Compile with x86_64-w64-mingw32-gcc -municode -DUNICODE -D_UNICODE and
then see it works on Cygwin 3.4.10:

$ ./t.exe 'C:\Devel\реализация-mingw64\flexdll\flexdll_mingw64.o'
  File: C:\Devel\реализация-mingw64\flexdll\flexdll_mingw64.o
  Size: 11441   Blocks: 12 IO Block: 65536  regular file
Device: 35525,39577 Inode: 3659174698707368  Links: 1
Access: (0644/-rw-r--r--)  Uid: (1049719/ DRA)   Gid: (1049089/Domain Users)
Access: 2024-04-03 16:43:36.782037800 +0100
Modify: 2024-04-03 14:09:08.012955600 +0100
Change: 2024-04-03 14:09:08.012955600 +0100
 Birth: 2024-04-03 14:09:08.006955500 +0100

but fails on 3.5.1+ (including the latest 3.6.0-0.109 snapshot):

$ ./t.exe 'C:\Devel\реализация-mingw64\flexdll\flexdll_mingw64.o'
stat: cannot stat
'"C:\Devel\'$'\360\237\220\253''реализация-mingw64\flexdll\flexdll_mingw64.o':
No such file or directory

Note the stray double-quote at the start of the error. This appears to
be down to the presence of the emoji, rather than the Cyrillic
characters, this is fine in both:

$ ./t.exe 'C:\Devel\реализация\flexdll\flexdll_mingw64.o'
  File: C:\Devel\реализация\flexdll\flexdll_mingw64.o
  Size: 11441   Blocks: 12 IO Block: 65536  regular file
Device: 35525,39577 Inode: 16607023627376791  Links: 1
Access: (0644/-rw-r--r--)  Uid: (1049719/ DRA)   Gid: (1049089/Domain Users)
Access: 2024-04-03 16:48:43.003742000 +0100
Modify: 2024-04-03 14:09:08.012955600 +0100
Change: 2024-04-03 16:32:35.648039600 +0100
 Birth: 2024-04-03 16:32:35.647539400 +0100

All best,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Regression: Cygwin 3.5.1 freezes when launching several mingw processes in parallel

[David Allsopp via Cygwin]

On Fri, 1 Mar 2024 at 08:03, Takashi Yano via Cygwin  wrote:
> Please try: cygwin 3.6.0-0.66.gc77a5689f7bd (TEST)

I can confirm this fixes the issue for me, thank you!


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Restore SEM_FAILCRITICALERRORS [was: Aren't Windows System Error popups meant to be disabled in Cygwin?]

[David Allsopp via Cygwin]

On Sat, 3 Feb 2024 at 19:25, Corinna Vinschen wrote:
> > Sorry to say that, but SetThreadErrorMode/CreateProcess don't do what we
> > want them to do.  I just tested this myself with a modified Cygwin DLL
> > (code below) and it turns out that the child process error mode is
> > the same as the parent's process error mode.  Changing the thread
> > error mode from the Cygwin default 3 (aka SEM_FAILCRITICALERRORS |
> > SEM_NOGPFAULTERRORBOX) to 0 doesn't have any effect.
> > [...etc.]

Oh dear, what a mess!

> MSYS2 has introduced the environment variable option CYGWIN=winjitdebug.
> I backported this patch now.  So default is back to propagating Cygwin's
> error mode and if you want to reset the error mode of a non-Cygwin child
> process back to OS default, just set the option, for instance, like
> this:
>
>   $ CYGWIN=winjitdebug env 
>
> This patch will be in Cygwin 3.5.1.  For the time being, it will be
> available in the next test release cygwin-3.6.0-0.28.g918c3eda4176 as
> well.

This completely fixes it for us, thank you very much

Thanks,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Restore SEM_FAILCRITICALERRORS [was: Aren't Windows System Error popups meant to be disabled in Cygwin?]

[David Allsopp via Cygwin]

On Fri, 2 Feb 2024 at 14:18, Corinna Vinschen via Cygwin wrote:
>
> On Feb  2 13:35, David Allsopp via Cygwin wrote:
> > Jon Turney via Cygwin wrote:
> >
> > > > I'm sympathetic, and personally I would prefer to revert the patch and
> > > > stick to SEM_FAILCRITICALERRORS by default.
> > > >
> > > > The question is this: Why does, apparently, everybody expect Cygwin to
> > > > do the "right thing", with different definitions of "right", when in
> > > > fact the executable in question can easily call SetErrorMode by itself?
> > >
> > > Yeah, if cygwin wasn't involved in the process ancestry, how would you
> > > get the behaviour you want?
> >
> > Ah, but that's how I hit this (and started digging further) -
> > precisely because the non-Cygwin program I'm using _has_ called
> > SetErrorMode and its direct calls to the faulty program _are_ doing
> > what's wanted (no popup dialog) but the calls which happen via Cygwin
> > are then torching that preference.
> >
> > Not really suggesting it be done this way (it feels more complicated
> > than just reverting the change), but in some ways perhaps Cygwin
> > should be using GetErrorMode on startup and instead of not inheriting
> > it, ensuring that it sets whatever it received? i.e. just before the
> > call to CreateProcess for a non-Cygwin binary, Cygwin restores the
> > error mode (for that thread only) to the value read at startup, calls
> > CreateProcess and then sets the error mode back.
>
> This sounds like a good ide, but...
>
> Is it actually a safe bet that the error mode set by SetThreadErrorMode
> is then propagated as process error mode to the child process?
>
> I have to ask that because Microsoft conveniently forgot to document
> this scenario in the MSDN docs.

:o) Never knowingly clear, are they! It would seem to be the intent of
SetThreadErrorMode that it would behave that way but who knows.

Happy to set up a quick experiment to check that it does work (i.e.
the invoked process has GetErrorMode set as expected) and that there's
no possible race between two threads in the calling process with
differing values (i.e. that having SEM_FAILCRITICALERRORS in one
thread and not in another behaves as expected. If it does appear to
work consistently, would you be willing to go down this route? Happy
to do the patch, although it'd be very helpful to have a couple of
pointers: I'm guessing the value would want to be captured just before
the one place where SetErrorMode is already called, but in which
structure should it then be stashed away to be reused in spawn?


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Restore SEM_FAILCRITICALERRORS [was: Aren't Windows System Error popups meant to be disabled in Cygwin?]

[David Allsopp via Cygwin]

Jon Turney via Cygwin wrote:

> > I'm sympathetic, and personally I would prefer to revert the patch and
> > stick to SEM_FAILCRITICALERRORS by default.
> >
> > The question is this: Why does, apparently, everybody expect Cygwin to
> > do the "right thing", with different definitions of "right", when in
> > fact the executable in question can easily call SetErrorMode by itself?
>
> Yeah, if cygwin wasn't involved in the process ancestry, how would you
> get the behaviour you want?

Ah, but that's how I hit this (and started digging further) -
precisely because the non-Cygwin program I'm using _has_ called
SetErrorMode and its direct calls to the faulty program _are_ doing
what's wanted (no popup dialog) but the calls which happen via Cygwin
are then torching that preference.

Not really suggesting it be done this way (it feels more complicated
than just reverting the change), but in some ways perhaps Cygwin
should be using GetErrorMode on startup and instead of not inheriting
it, ensuring that it sets whatever it received? i.e. just before the
call to CreateProcess for a non-Cygwin binary, Cygwin restores the
error mode (for that thread only) to the value read at startup, calls
CreateProcess and then sets the error mode back.

To explain in the context of the actual call chain:
- I have opam.exe (compiled with mingw-w64), which is calling
SetErrorMode in main
- It's calling ocamlc.exe (in PATH) which requires the zstd DLL, but
that has not been put in PATH
- A direct call - not via Cygwin - to ocamlc -vnum therefore returns
an exit code
- Another call, already there from the Unix side, instead does sh -c
"ocamlc -config | sed ..." but Cygwin has then _removed_ the
calling applications preference

Thanks,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Restore SEM_FAILCRITICALERRORS [was: Aren't Windows System Error popups meant to be disabled in Cygwin?]

[David Allsopp via Cygwin]

On Fri, 2 Feb 2024 at 12:55, Corinna Vinschen via Cygwin wrote:
> On Feb  2 09:43, David Allsopp via Cygwin wrote:
> > On Thu, 1 Feb 2024 at 10:02, Corinna Vinschen via Cygwin
> >  wrote:
> > >
> > > The behaviour changed in 2020
> > >
> > > https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=21ec498d7f912
> > >
> > > not without a discussion
> > >
> > > https://cygwin.com/pipermail/cygwin-patches/2020q4/010870.html
> >
> > Aha, thank you! (congrats on the 3.5 release, in passing, btw).
> > Definitely not a regression, then (subject edited).
> >
> > However, this patch came from MSYS2, and subsequently they seem to
> > have found it problematic for the same reason as me
> > (https://github.com/msys2/msys2-runtime/pull/18#issuecomment-810897624)
> > and have just recently reintroduced the flag
> > (https://github.com/msys2/msys2-runtime/commit/7616b8a2e0ffcf068b47e1a66bbb1dbd7d9b5c50)
> > to control it.
> >
> > The reasoning in
> > https://cygwin.com/pipermail/cygwin/2006-August/150081.html seems as
> > valid now as it did in 2006.
> >
> > Is it possible to revisit having the flag, or even just reverting the 
> > behaviour?
> >
> > FWIW, it's been "hurting" us over in OCaml-land since zstd support was
> > added roughly a year ago - configure can tell us that mingw-w64's zstd
> > is available, but woe betide us if we run the test program to see if
> > it actually works, but the user forgot to add the sys-root into PATH,
> > because at that point the CI system is down...
>
> I'm sympathetic, and personally I would prefer to revert the patch and
> stick to SEM_FAILCRITICALERRORS by default.
>
> The question is this: Why does, apparently, everybody expect Cygwin to
> do the "right thing", with different definitions of "right", when in
> fact the executable in question can easily call SetErrorMode by itself?

The executable can't, though (at least, I'm not aware that it can)?
The DLL not found case is being triggered by the loader, before the
entrypoint code is running?

I did have a look to see if there are manifest flags or some such
which can be set to indicate this, but it does seem to be the
responsibility of the caller, coupled with a "best practice"
recommendation on MSDN to call SetErrorMode (as Cygwin is of course
doing).

The whole system with it feels like unfortunate Windows legacy, but I
guess the behaviour vastly predates Event Viewer, etc., and slightly
better (and non-blocking) ways of reporting loader errors. Perils of a
nearly 40 year old API, if not OS 

Thanks,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Restore SEM_FAILCRITICALERRORS [was: Aren't Windows System Error popups meant to be disabled in Cygwin?]

[David Allsopp via Cygwin]

On Thu, 1 Feb 2024 at 10:02, Corinna Vinschen via Cygwin
 wrote:
>
> The behaviour changed in 2020
>
> https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=21ec498d7f912
>
> not without a discussion
>
> https://cygwin.com/pipermail/cygwin-patches/2020q4/010870.html

Aha, thank you! (congrats on the 3.5 release, in passing, btw).
Definitely not a regression, then (subject edited).

However, this patch came from MSYS2, and subsequently they seem to
have found it problematic for the same reason as me
(https://github.com/msys2/msys2-runtime/pull/18#issuecomment-810897624)
and have just recently reintroduced the flag
(https://github.com/msys2/msys2-runtime/commit/7616b8a2e0ffcf068b47e1a66bbb1dbd7d9b5c50)
to control it.

The reasoning in
https://cygwin.com/pipermail/cygwin/2006-August/150081.html seems as
valid now as it did in 2006.

Is it possible to revisit having the flag, or even just reverting the behaviour?

FWIW, it's been "hurting" us over in OCaml-land since zstd support was
added roughly a year ago - configure can tell us that mingw-w64's zstd
is available, but woe betide us if we run the test program to see if
it actually works, but the user forgot to add the sys-root into PATH,
because at that point the CI system is down...

Thanks,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Aren't Windows System Error popups meant to be disabled in Cygwin?

[David Allsopp via Cygwin]

> x86_64-w64-mingw32-gcc produces a Windows program, why Cygwin should
> be involved in the execution ?

I perhaps should have made that crystal clear - in running "./test",
I'm invoking that excecutable _from_ a Cygwin program (in this case
mintty / bash / sh), so Cygwin is very much involved - it's the
caller.

In the actual problem which hit this, I have a non-Cygwin executable
which has called SetErrorMode(SEM_FAILCRITICALERRORS). If that program
calls test, there is no popup displayed. However, it also calls
Cygwin's sh and _that_ executes that program too, so something like
"C:\cygwin64\bin\sh -c "./test.exe | sed ..." but then the popup error
message appears. So somewhere along the line, Cygwin appears to be
resetting the system error mode, and that appears contrary to previous
(old) messages on the subject.

Thanks,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Aren't Windows System Error popups meant to be disabled in Cygwin?

[David Allsopp via Cygwin]

On Wed, 31 Jan 2024 at 22:27, Brian Inglis via Cygwin  wrote:
>
> On 2024-01-31 06:40, David Allsopp via Cygwin wrote:
> > Starting with this very trivial C program:
> >
> > #include 
> > #include 
> >
> > int main(void) {
> >printf("Zstandard v%d\n", ZSTD_versionNumber());
> > }
> >
> > and compiling with
> >
> > x86_64-w64-mingw32-gcc -o test.exe test.c -lzstd
> >
> > when I then run ./test.exe, I get the Windows critical-error-handler
> > dialog stating "The code execution cannot proceed because
> > libzstd-1.dll was not found. Reinstalling the program may fix this
> > problem."
> >
> > My question is not how to fix the problem (I'm well aware of that),
> > but rather why that message is being displayed at all, and is it a bug
> > in Cygwin somewhere? All I could find Googling was previous
> > suggestions that Cygwin routinely calls SetErrorMode with, amongst
> > other things, SEM_FAILCRITICALERRORS with the intention of suppressing
> > this dialog.
> >
> > Is that correct, and if so is this just me? :o)
> >
> > Windows 10 22H2, Cygwin 3.4.10, running all the commands from mintty.
> > I also get the same popup if I run C:\cygwin64\bin\sh -c
> > "/cygdrive/c/path/to/test" either from a Command Prompt or even from
> > "Start -> Run". Running this via "sh" called from a non-Cygwin process
> > (itself invoked from a Command Prompt) which has also called
> > SetErrorMode is how I hit this.
>
> Better to let you know that there is a problem, and what the problem is, so 
> you
> can fix it!

Thank you, but no - as I made clear by:

> My question is not how to fix the problem (I'm well aware of that)

I'm fully aware what has caused the issue to arise, and how to fix it
- that's not the issue. The problem is that according to previous
messages, and the Cygwin code, my impression was that mintty / bash /
sh (*Cygwin* programs) calling this executable should be returning an
exit code here, not freezing on a message dialog. The problem appears
to be a bug in the Cygwin DLL, and from previous messages on the list,
my question is whether it's a regression, and can be fixed.

The reason it's a problem is, say, a script _in Cygwin_ which is
handed a command to run, runs it, and is then completely blocked by
that popup dialog. It's the responsibility of the _caller_ (a Cygwin
program) to indicate the mode in which a program is executed - the
message box may be owned by the program called, but it's caused by it
being loaded, before it has a chance to run any code.

My understanding, based on this line:

https://github.com/cygwin/cygwin/blob/main/winsup/cygwin/dcrt0.cc#L721
in dll_crt0_0

is that Cygwin executables (in this case mintty / bash / sh, etc.)
should be running with SEM_FAILCRITICALERRORS enabled, following the
best practice recommendations in
https://learn.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-seterrormode
and that that setting should be correctly inherited by processes they
call (including non-Cygwin ones).

Some ancient history, reporting my same issue in 2004:
https://cygwin.com/pipermail/cygwin/2004-March/115553.html and this
thread from 2006
https://cygwin.com/pipermail/cygwin/2006-August/150038.html strongly
indicating that that line dcrt0.cc is there precisely to stop this
popup.

Thanks,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Aren't Windows System Error popups meant to be disabled in Cygwin?

[David Allsopp via Cygwin]

Starting with this very trivial C program:

#include 
#include 

int main(void) {
  printf("Zstandard v%d\n", ZSTD_versionNumber());
}

and compiling with

x86_64-w64-mingw32-gcc -o test.exe test.c -lzstd

when I then run ./test.exe, I get the Windows critical-error-handler
dialog stating "The code execution cannot proceed because
libzstd-1.dll was not found. Reinstalling the program may fix this
problem."

My question is not how to fix the problem (I'm well aware of that),
but rather why that message is being displayed at all, and is it a bug
in Cygwin somewhere? All I could find Googling was previous
suggestions that Cygwin routinely calls SetErrorMode with, amongst
other things, SEM_FAILCRITICALERRORS with the intention of suppressing
this dialog.

Is that correct, and if so is this just me? :o)

Windows 10 22H2, Cygwin 3.4.10, running all the commands from mintty.
I also get the same popup if I run C:\cygwin64\bin\sh -c
"/cygdrive/c/path/to/test" either from a Command Prompt or even from
"Start -> Run". Running this via "sh" called from a non-Cygwin process
(itself invoked from a Command Prompt) which has also called
SetErrorMode is how I hit this.

Thanks!


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin test 3.5.0 tar symlinks error messages and failure status

[David Allsopp via Cygwin]

Achim Gratz wrote:
> Brian Inglis writes:
> > Problem writing tar (with Cygwin default sys) symlinks before target
> > created under Cygwin 3.5.0 - error messages are issued and tar exits
> > with failure status!
> […]
> > The only likely culprit between 3.4.6 and that commit seems to be
> > commit 2023-04-18 fa84aa4dd2fb43eaf7fcdfb040aef854f2f19d01 Cygwin: fix
> > errno values set by readlinkat.
> >
> > Still seems to work as expected despite the error messages and failure 
> > status.
> >
> > Runs without any messages or failure under Cygwin stable 3.4.6.
>
> The interface mentioned above is known to be wonky on various systems.
> You might need to re-build tar in oder for it to detect any changed
> level of wonkiness and adapt accordingly.

On Cygwin 3.4.7, recompiling tar from the source package fixes this
problem; the resulting binary then seems to be fine on Cygwin 3.4.6 as
well.

Please could tar 1.34 be re-packaged?

All best,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Debugging malloc crash in gdb

[David Allsopp]

On Tue, 18 Oct 2022 at 20:09, Jon Turney wrote:
>
> On 18/10/2022 11:35, David Allsopp wrote:
> > I'm wondering if I may be able to have some pointers for debugging what
> > seems to be an unexpected interaction between mmap/mprotect/munmap and
> > malloc with the OCaml runtime.
> >
> > At the moment, I know that we crash in malloc, so my main question is how to
> > go further in gdb. I installed the cygwin-debuginfo package, but all I'm
> > getting is:
>
> Firstly, if the crash is inside the cygwin DLL, you must follow the
> advice in [1], and use 'set cygwin-exceptions on' to tell gdb to stop on
> an exception inside cygwin itself.
>
> [1] https://cygwin.com/faq.html#faq.programming.debugging-cygwin
>
>
> > /cygdrive/d/a/scallywag/gdb/gdb-11.2-1.x86_64/src/gdb-11.2/gdb/infrun.c:2550
> > : internal-error: void resume_1(gdb_signal): Assertion
> > `pc_in_thread_step_range (pc, tp)' failed.

I'm not sure now which combination of stepping directly into the
malloc call, adding set cygwin-exceptions on or switching to gdb 12.1,
but either way I was able to get to an invalid memory access in
mmap_alloc in malloc.cc. At this point, p was a pointer to the start
of the 256M block which had been passed to munmap.

What I then noticed from that is a bug in our code - the mmap'd region
was actually 256M+64K but the size passed to munmap was 256M... so the
munmap call was not releasing the entire block. Fixing that on the
OCaml side fixes the error completely - I don't know whether what we
were seeing before counts as a bug in Cygwin's allocator?

Many thanks!


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Debugging malloc crash in gdb

[David Allsopp]

I'm wondering if I may be able to have some pointers for debugging what
seems to be an unexpected interaction between mmap/mprotect/munmap and
malloc with the OCaml runtime.

At the moment, I know that we crash in malloc, so my main question is how to
go further in gdb. I installed the cygwin-debuginfo package, but all I'm
getting is:

 
/cygdrive/d/a/scallywag/gdb/gdb-11.2-1.x86_64/src/gdb-11.2/gdb/infrun.c:2550
: internal-error: void resume_1(gdb_signal): Assertion
`pc_in_thread_step_range (pc, tp)' failed.

The reproduction case is below (it's the OCaml runtime, so it's not exactly
minimal, but it seems to be very repeatable to get gdb to the position of
the crash).

In terms of memory, what OCaml is doing:

- At startup, 256M of address space is reserved (with mmap) for garbage
collected minor heaps ("minor arena")
- The first 2M of this is "committed" with mprotect for use by the program's
main thread
- The program then instructs the runtime to double the size of the minor
arena
- The 2M portion is "decommitted" with mprotect
- The 256M mmap'd region is munmap'd
- A new 512M region of address space is reserved
- The first 4M of this is "committed" with mprotect for use by the program's
main thread
- The program performs some assertion checks
- Book-keeping at the end of this causes malloc to be called, which
segfaults.

The crashing call to malloc is the first call to malloc since the 256M ->
512M munmap/map dance.

If the call to caml_mem_unmap at the end of unreserve_minor_heaps in
runtime/domain.c is omitted, then this program succeeds - i.e. malloc does
not appear to crash if the 256M region is left mapped. Obviously, I realise
this may well be unrelated to what's going wrong.

Any assistance to debug this further hugely appreciated!

Thanks,


David


---
Full repro instructions

Cygwin packages required: gcc-core, make, flexdll

Build:
  git clone https://github.com/dra27/ocaml -b restore-cygwin-break --depth 1
  cd ocaml
  ./configure --disable-native-compiler --disable-debugger
--disable-ocamldoc && make -j
  runtime/ocamlrun.exe ./ocamlc.exe -nostdlib -I stdlib
testsuite/tests/regression/pr9326/gc_set.ml -o gc_set.byte.exe

Crash:
  runtime/ocamlrun.exe ./gc_set.byte.exe

Debug:
  OCAMLRUNPARAM=v=0x1FFF gdb runtime/ocamlrun.exe
  break caml_gc_get
  run ./gc_set.byte.exe
  continue
  break alloc_generic_table
  continue
  break caml_stat_alloc_noexc
  continue
  step
  step
  step
  *boom*


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: [ITA] ocaml 4.14.0

[David Allsopp]

Corinna Vinschen wrote:
> On Aug 23 20:00, David Allsopp wrote:
> > Jon Turney wrote:
> > > I'm confused here: /usr/lib/ocaml/camlheaderd[di] look like
> > > executables (according to file etc.)
> > >
> > > If they genuinely aren't, then perhaps they shouldn't have execute
> > > permission (which is I think what is causing them to be caught by
> > > that piece of code in cygport...)
> >
> > ocamlc won't care about the executable bit, but these files are _data_
> > and so it will care if they end up with a .exe extension, because they
> > are open'd not exec'd.
> >
> > The files themselves form the start of executables produced by ocamlc.
> > For both Cygwin and native Windows, it's a tiny C program responsible
> > for locating ocamlrun (on Unix, it's just a shebang line) and the
> > bytecode itself gets appended.
> 
> Why is ocaml for Cygwin not produing shebang files as well?  After all,
> we're trying to be as Unix-compatible as possible.

I'm not 100% that this is the correct history, but I think it's down to being 
able to start the programs from outside a Cygwin bash prompt - i.e. the 
programs emitted want to have a .exe by default and it's strange to have ".exe" 
files with a shebang header (at least from outside Cygwin).

This part of OCaml is actually old enough (late nineties) that this executable 
was also used on "real" Unix systems where shebang didn't work properly :)

It is possible to configure Cygwin OCaml to use shebang headers (although at 
the moment only by changing Makefile.config after running configure)

All best,


David

RE: [ITA] ocaml 4.14.0

[David Allsopp]

Jon Turney wrote:
> On 13/07/2022 16:41, David Allsopp wrote:
> >
> >> 3) Interesting - on my machine, the camlheader[di] files had the .exe
> >> extensions. I did some digging around and found the files are *built*
> >> without  the .exe suffix, and even *initially installed* without the
> >> .exe suffix, but  ultimately come out with the .exe suffix. I ran
> >> cyport in debug mode and apparently the files are being renamed with
> >> the suffix
> >> post-install:
> >>
> >> + case "${exe##*/}" in
> >> + mv usr/lib/ocaml/camlheaderd usr/lib/ocaml/camlheaderd.exe
> >> + exe+=.exe
> >>
> >> and did a little more digging and I think these lines in cygport are
> >> the
> >> cause:
> >> https://github.com/cygwin/cygport/blob/096f27644bd3b28f29d7522e816beb
> >> d327c
> >> f24cb/lib/src_postinst.cygpart#L1010
> 
> Yeah, that seems plausible, but I'm not clear why we are seeing different
> behaviour...
> 
> >> On the topic of "testing more thoroughly", I attempted to use ocamlc
> >> to compile a simple program and it fails with "Cannot find file
> >> camlheader" but works when I remove the ".exe", so it seems that the
> >> presence of the .exe suffixes breaks the compiler. Is there a way to
> >> prevent cygport from adding it?
> >
> > The camlheader files are data files and definitely mustn't be
> > installed with a .exe extension (nor do they need to be executable).
> 
> I'm confused here: /usr/lib/ocaml/camlheaderd[di] look like executables
> (according to file etc.)
> 
> If they genuinely aren't, then perhaps they shouldn't have execute
> permission (which is I think what is causing them to be caught by that
> piece of code in cygport...)

ocamlc won't care about the executable bit, but these files are _data_ and so 
it will care if they end up with a .exe extension, because they are open'd not 
exec'd.

The files themselves form the start of executables produced by ocamlc. For both 
Cygwin and native Windows, it's a tiny C program responsible for locating 
ocamlrun (on Unix, it's just a shebang line) and the bytecode itself gets 
appended.

All best,


David

RE: [ITA] ocaml 4.14.0

[David Allsopp]

William Hu wrote:
> Hi David,
> 
> > What were the missing symbols? With the OCaml 4.10 package, I hit
> problems with this:
> >
> > echo 'print_endline "hello, world"' > hello.ml
> >
> > ocamlc -custom -runtime-variant _shared -o hello.exe hello.ml
> >
> > I think that may be an issue upstream (libasmrun_shared.so IIRC is
> broken on all platforms - on Cygwin, you can just about to persuade it to
> get to the same symbol errors because of the extra .dll.a file which gets
> generated).
> 
> It's a very long list of missing symbols (~400) so it's included as an
> attachment in case the mailing list truncates extremely lengthy emails.
> Also, just to note, the .dll.a file doesn't seem to be automatically
> generated by upstream OCaml.

I think this down to neglect - the PIC/shared versions of the runtime were 
contributed for a specific purpose and aren't properly maintained/tested AFAICT.

I'm not sure that libcamlrun_shared can ever have worked on Cygwin, at least 
certainly not since OCaml 3.11 (which is a lng time ago). The patch builds 
cygcamlrun_shared.dll as a plugin DLL - so the symbols will have been stripped 
and moved into symtbl (hence the huge number of missing symbols when linking 
against it when it gets passed to a normal linker).

What normally happens with flexlink is that either an executable or a "main 
program DLL" (compiled with flexlink -maindll) would receive that symtbl from a 
DLL it flexdll_dlopen's and the DLL receives static_symtbl from the executable. 
That then means that the DLL can relocate any symbols it expected from the main 
executable and any other flexdll_dlopen'd DLLs. However, that's all meant to 
happen at runtime - I don't think flexlink has ever supported compiling an 
executable _itself_ which must flexdll_dlopen DLLs _before_ its main function.

I think it's quite feasible to add that to flexlink, but it's not a small piece 
of work - in the meantime I'd suggest instead deleting the 
BYTECODE_SHARED_LIBRARIES += and NATIVE_SHARED_LIBRARIES += lines in 
runtime/Makefile instead of patching them.

> > What's the full configuration command and what gets inferred for the
> build, host and target triplets? fma should work without emulation in
> Cygwin32 and it should be detecting as failing on Cygwin64 but the
> emulation should be enabled by default unless you explicitly passed --
> disable-imprecise-c99-float-ops.
> 
> Configuration command:
> ./configure --prefix=/usr --enable-ocamltest
> 
> If it's relevant, I'm compiling on a 64-bit machine with Cygwin32
> installed to compile the 32-bit version. Inferred build/host/target:
> configure:2779: Configuring OCaml version 4.14.0
> configure:3007: checking build system type
> configure:3021: result: i686-pc-cygwin
> configure:3041: checking host system type
> configure:3054: result: i686-pc-cygwin
> configure:3074: checking target system type
> configure:3087: result: i686-pc-cygwin
> 
> On Cygwin64, the test does fail and emulate by default:
> 
> configure:14872: result: no
> configure:14896: WARNING: fma does not work; emulation enabled
> 
> However, on Cygwin32, I get
> 
> configure:14872: result: no
> configure:14894: error: fma does not work, enable emulation with --enable-
> imprecise-c99-float-ops
> 
> config.log conveniently includes the program that fails, so I copy-pasted
> some very redundant code (attached) and ran it on the different
> architectures:
> 
> Cygwin64:
> t264| ans: 0x0p+0   accept: 0x1.0989687cp-1044 0x1.09df287cp-1044 0x1.0ap-
> 1044
> t264| pass?: 0
> t265| ans: 0x0p+0   accept: 0x1.0988p-1060 0x1.09ep-1060 0x1.0ap-1060
> t265| pass?: 0
> t266| ans: 0x1p-1074   accept: 0x1p-1073
> t266| pass?: 0
> 
> Cygwin32:
> t264| ans: 0x1.0ap-1044   accept: 0x1.0989687cp-1044 0x1.09df287cp-1044
> 0x1.0ap-1044
> t264| pass?: 1
> t265| ans: 0x1.0ap-1060   accept: 0x1.0988p-1060 0x1.09ep-1060 0x1.0ap-
> 1060
> t265| pass?: 1
> t266| ans: 0x1p-1073   accept: 0x1p-1073
> t266| pass?: 1
> 
> The numbers look like they agree but configure doesn't think so for some
> reason.

Would you be able to send the entire config.log file (off-list is fine!). I'm 
not seeing this at all in Cygwin32 here! 

All best,


David

RE: [ITA] ocaml 4.14.0

[David Allsopp]

> 1) libcamlrun: Oops, that's another oversight, forgot to look at the old
> patches. The other 3 patches seem unnecessary, but I had trouble linking
> either libcamlrun_shared.so or libcamlrun_shared.dll.a into a program
> (unresolved symbol errors). Added but it possibly needs further patching.
> I'll keep on digging.

What were the missing symbols? With the OCaml 4.10 package, I hit problems with 
this:

  echo 'print_endline "hello, world"' > hello.ml
  ocamlc -custom -runtime-variant _shared -o hello.exe hello.ml

I think that may be an issue upstream (libasmrun_shared.so IIRC is broken on 
all platforms - on Cygwin, you can just about to persuade it to get to the same 
symbol errors because of the extra .dll.a file which gets generated). 

> 1.5) I checked for other differences between the cygports to make sure I
> didn't miss anything else. I made some edits to src_install: I removed
> `dodoc  Updating` since there doesn't seem to be a file named "Updating".
> I also removed symlinking header files to /usr/include.  From what I
> understood, the  OCaml documentation says the headers should "reside in
> the caml/ subdirectory  of the OCaml standard library directory, which is
> returned by the command ocamlc -where (usually /usr/local/lib/ocaml or
> /usr/lib/ocaml)". Obviously the  symlink doesn't move anything, but since
> their location is well documented I  didn't see a reason to have the extra
> symlinks. Please let me know if this is  too large of a change or there's
> a Cygwin (or non-Cygwin) convention that precludes this - I'm still
> getting the hang of this.

> 2) fma on x86: I'm actually getting the same error, but the tests should
> ostensibly pass on Cygwin32. I'll also look into this.

What's the full configuration command and what gets inferred for the build, 
host and target triplets? fma should work without emulation in Cygwin32 and it 
should be detecting as failing on Cygwin64 but the emulation should be enabled 
by default unless you explicitly passed --disable-imprecise-c99-float-ops.

> 3) Interesting - on my machine, the camlheader[di] files had the .exe
> extensions. I did some digging around and found the files are *built*
> without  the .exe suffix, and even *initially installed* without the .exe
> suffix, but  ultimately come out with the .exe suffix. I ran cyport in
> debug mode and apparently the files are being renamed with the suffix
> post-install:
> 
> + case "${exe##*/}" in
> + mv usr/lib/ocaml/camlheaderd usr/lib/ocaml/camlheaderd.exe
> + exe+=.exe
> 
> and did a little more digging and I think these lines in cygport are the
> cause:
> https://github.com/cygwin/cygport/blob/096f27644bd3b28f29d7522e816bebd327c
> f24cb/lib/src_postinst.cygpart#L1010
> 
> On the topic of "testing more thoroughly", I attempted to use ocamlc to
> compile a simple program and it fails with "Cannot find file camlheader"
> but works when I remove the ".exe", so it seems that the presence of the
> .exe suffixes breaks the compiler. Is there a way to prevent cygport from
> adding it?

The camlheader files are data files and definitely mustn't be installed with a 
.exe extension (nor do they need to be executable).

Incidentally, OCaml 4.12+ is also likely to run into problems if flexlink is 
older than 0.39 - I just removed the test mark from the flexdll 0.39 package 
(which I thought I'd done quite some time ago...)

HTH,


David

mmap failing with MAP_FIXED

[David Allsopp]

This program fails at the second mmap call with EINVAL:

  #include 
  #include 
  #include 

  int main (void) {
void * mem;
/* Reserve 256MB address space for the minor heaps */
mem = mmap(0, 268439552, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (mem == MAP_FAILED)
  error(1, 0, "Reservation failed");
/* Commit the first 2MB heap */
if (mmap(mem, 2097152, PROT_READ | PROT_WRITE, MAP_PRIVATE |
MAP_ANONYMOUS | MAP_FIXED, -1, 0) == MAP_FAILED)
  error(1, 0, "Commit failed");
  }

Is this something that's expected to fail for Cygwin, or a bug? The example
is extracted from OCaml 5.0's runtime, which reserves an area of address
space and then commits chunks of it as required. The above snippet comes
from the Linux side, on Windows we're using VirtualAlloc with PAGE_NOACCESS
to reserve the address space and then VirtualAlloc with MEM_COMMIT and
PAGE_READWRITE to commit smaller portions of it.

Is there a way to do that with Cygwin's mmap?

Thanks,


David


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin setup writing incorrect symlinks for native

[David Allsopp]

Jon Turney wrote:
> On 09/01/2022 09:35, David Allsopp wrote:
> > Jon Turney wrote:
> >> On 06/01/2022 16:45, David Allsopp wrote:
> >>> Jon Turney wrote:
> >>>> On 06/01/2022 10:46, David Allsopp wrote:
> >>>>> Running Cygwin setup 2.912 with --symlink-type native (or
> >>>>> CYGWIN=winsymlinks:native) is not correctly translating all
> symlinks.
> >>>>> A default install has these faulty ones:
> >>>>>
> >>>>> /etc/pki/tls/cert.pem ->
> >>>>> \??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
> >>>>> /etc/pki/tls/certs/ca-bundle.crt ->
> >>>>> \??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
> >>>>> /etc/pki/tls/certs/ca-bundle.trust.crt ->
> >>>>> \??\/etc\pki\ca-trust\extracted\openssl\ca-bundle.trust.crt
> >>>>> /etc/ssl/certs -> \??\/etc\pki\tls\certs /lib/security/cacerts ->
> >>>>> \??\/etc\pki\ca-trust\extracted\java\cacerts
> >>>>> /usr/share/doc/groff-1.22.4/pdf/mom-pdf.pdf ->
> >>>>> \??\/usr\share\doc\groff-1.22.4\examples\mom\mom-pdf.pdf
> >>>>
> >>>>
> [...]
> >>
> >> Thanks for testing.  It seems I still didn't have this quite right
> >> and these symlinks just happened to work for cygwin, but not native
> tools.
> >>
> >> Please try
> >>
> >> https://cygwin.com/setup/setup-2.914.x86_64.exe
> >> https://cygwin.com/setup/setup-2.914.x86.exe
> >
> > I'm afraid that's just changed the \??\ to \\?\ on those symlinks.
> 
> Well, that has the advantage of being correct :)
> 
> (since it's the 'Win32 File Namespace' prefix, which
> CreateSymbolicLinkW() is documented to accept for the target filename, and
> is required when that exceeds MAX_PATH (260) characters)
> 
> In my (brief) testing, e.g. the CMD builtin 'type' is able to open
> symlinks of this form.  However, it seems there are some parts of Windows
> (e.g. DIR, File Explorer), which don't handle symlinks like that well.
> 
> So, I've copied the behaviour of the cygwin DLL, which avoids using that
> prefix if the target is less than MAX_PATH characters.  This seem to work
> better in those cases with typical paths (but I'd claim we're just working
> around a bug in Windows here, as things will still be broken if the cygwin
> root is a path long enough that we can't do that).
> 
> Please try:
> 
>https://cygwin.com/setup/setup-2.915.x86_64.exe
>https://cygwin.com/setup/setup-2.915.x86.exe

That's working (for cmd as well), thanks!


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin setup writing incorrect symlinks for native

[David Allsopp]

Jon Turney wrote:
> On 06/01/2022 16:45, David Allsopp wrote:
> > Jon Turney wrote:
> >> On 06/01/2022 10:46, David Allsopp wrote:
> >>> Running Cygwin setup 2.912 with --symlink-type native (or
> >>> CYGWIN=winsymlinks:native) is not correctly translating all symlinks.
> >>> A default install has these faulty ones:
> >>>
> >>> /etc/pki/tls/cert.pem ->
> >>> \??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
> >>> /etc/pki/tls/certs/ca-bundle.crt ->
> >>> \??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
> >>> /etc/pki/tls/certs/ca-bundle.trust.crt ->
> >>> \??\/etc\pki\ca-trust\extracted\openssl\ca-bundle.trust.crt
> >>> /etc/ssl/certs -> \??\/etc\pki\tls\certs /lib/security/cacerts ->
> >>> \??\/etc\pki\ca-trust\extracted\java\cacerts
> >>> /usr/share/doc/groff-1.22.4/pdf/mom-pdf.pdf ->
> >>> \??\/usr\share\doc\groff-1.22.4\examples\mom\mom-pdf.pdf
> >>
> >>
> >> Thanks for reporting this.
> >>
> >> Yes, it seems this is messed up where the target of the symlink is an
> >> absolute path.
> >>
> >> I've uploaded setup 2.913 with an attempted fix.
> >>
> >> https://cygwin.com/setup/setup-2.913.x86_64.exe
> >> https://cygwin.com/setup/setup-2.913.x86.exe
> >>
> >> Please try that (you'll have to reinstall the effected packages), and
> >> see if it improves things for you?
> >
> > Thank you! This fixes it for the Cygwin side, although the symlinks are
> > still technically not right from the Windows side. They're still showing
> > up prefixed \??\.
> >
> > It's an age since I mucked around with those functions - IIRC the API
> > function for writing them adds the \??\ itself and doesn't expect it
> > to have been supplied? However, when you read the reparse point you
> > have to remember to remove it yourself because who needs consistency
> > :o)
> 
> Thanks for testing.  It seems I still didn't have this quite right and
> these symlinks just happened to work for cygwin, but not native tools.
> 
> Please try
> 
>https://cygwin.com/setup/setup-2.914.x86_64.exe
>https://cygwin.com/setup/setup-2.914.x86.exe

I'm afraid that's just changed the \??\ to \\?\ on those symlinks.

All best,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin setup writing incorrect symlinks for native

[David Allsopp]

Jon Turney wrote:
> On 06/01/2022 10:46, David Allsopp wrote:
> > Running Cygwin setup 2.912 with --symlink-type native (or
> > CYGWIN=winsymlinks:native) is not correctly translating all symlinks.
> > A default install has these faulty ones:
> >
> > /etc/pki/tls/cert.pem ->
> > \??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
> > /etc/pki/tls/certs/ca-bundle.crt ->
> > \??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
> > /etc/pki/tls/certs/ca-bundle.trust.crt ->
> > \??\/etc\pki\ca-trust\extracted\openssl\ca-bundle.trust.crt
> > /etc/ssl/certs -> \??\/etc\pki\tls\certs /lib/security/cacerts ->
> > \??\/etc\pki\ca-trust\extracted\java\cacerts
> > /usr/share/doc/groff-1.22.4/pdf/mom-pdf.pdf ->
> > \??\/usr\share\doc\groff-1.22.4\examples\mom\mom-pdf.pdf
> 
> 
> Thanks for reporting this.
> 
> Yes, it seems this is messed up where the target of the symlink is an
> absolute path.
> 
> I've uploaded setup 2.913 with an attempted fix.
> 
>https://cygwin.com/setup/setup-2.913.x86_64.exe
>https://cygwin.com/setup/setup-2.913.x86.exe
> 
> Please try that (you'll have to reinstall the effected packages), and see
> if it improves things for you?

Thank you! This fixes it for the Cygwin side, although the symlinks are still 
technically not right from the Windows side. They're still showing up prefixed 
\??\.

It's an age since I mucked around with those functions - IIRC the API function 
for writing them adds the \??\ itself and doesn't expect it to have been 
supplied? However, when you read the reparse point you have to remember to 
remove it yourself because who needs consistency :o)

C:\cygwin64>dir /s | findstr ??
01/06/2022  04:30 PM  cert.pem 
[\??\C:\cygwin64\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem]
01/06/2022  04:30 PM  ca-bundle.crt 
[\??\C:\cygwin64\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem]
01/06/2022  04:30 PM  ca-bundle.trust.crt 
[\??\C:\cygwin64\etc\pki\ca-trust\extracted\openssl\ca-bundle.trust.crt]
01/06/2022  04:30 PM certs [\??\C:\cygwin64\etc\pki\tls\certs]
01/06/2022  04:30 PM  cacerts 
[\??\C:\cygwin64\etc\pki\ca-trust\extracted\java\cacerts]
01/06/2022  04:30 PM  mom-pdf.pdf 
[\??\C:\cygwin64\usr\share\doc\groff-1.22.4\examples\mom\mom-pdf.pdf]

All best,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Cygwin setup writing incorrect symlinks for native

[David Allsopp]

Running Cygwin setup 2.912 with --symlink-type native (or
CYGWIN=winsymlinks:native) is not correctly translating all symlinks. A
default install has these faulty ones:

/etc/pki/tls/cert.pem ->
\??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
/etc/pki/tls/certs/ca-bundle.crt ->
\??\/etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
/etc/pki/tls/certs/ca-bundle.trust.crt ->
\??\/etc\pki\ca-trust\extracted\openssl\ca-bundle.trust.crt
/etc/ssl/certs -> \??\/etc\pki\tls\certs
/lib/security/cacerts -> \??\/etc\pki\ca-trust\extracted\java\cacerts
/usr/share/doc/groff-1.22.4/pdf/mom-pdf.pdf ->
\??\/usr\share\doc\groff-1.22.4\examples\mom\mom-pdf.pdf

In particular, we noticed that /etc/ssl/certs being invalid breaks git.


David


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Fix nanosleep returning negative rem

[David Allsopp]

> On Jul 20 16:16, David Allsopp wrote:
> > I've pushed a repro case for this to
> > https://github.com/dra27/cygwin-nanosleep-bug.git
> >
> > Originally noticed as the main CI system for OCaml has been failing
> > sporadically for the signal.ml test mentioned in that repo. This
> > morning I tried hammering that test on my dev machine and discovered
> > that it fails very frequently. No idea if that's drivers, Windows 10
> > updates, number of cores or what, but it was definitely happening, and
> > easily.
> >
> > Drilling further, it appears that NtQueryTimer is able to return a
> > negative value in the TimeRemaining field even when SignalState is
> > false. The values I've seen have always been < 15ms - i.e. less than
> > the timer resolution, so I wonder if there is a point at which the
> > timer has elapsed but has not been signalled, but WaitForMultipleObjects
> returns because of the EINTR signal.
> > Mildly surprising that it seems to be so reproducible.
> >
> > Anyway, a patch is attached which simply guards a negative return
> > value. The test on tbi.SignalState is in theory unnecessary.
> 
> Thanks for the patch, I think your patch is fine.  However, I'd like to
> dig a bit into this to see what exactly happens.  Do you have a very
> simple testcase in plain C, by any chance?

https://github.com/dra27/cygwin-nanosleep-bug/blob/main/signal.c was as simple 
as I'd gone at this stage (eliminating OCaml from the equation!). It might be 
possible to get it to happen without all the pthreads stuff: having confirmed 
it definitely wasn't OCaml and been able to put the appropriate system_printf's 
into cygwait to see that NtQueryTimer really was returning this small negative 
value, I stopped simplifying.

Does that repro case trigger on your system too?

Best,


D

Fix nanosleep returning negative rem

[David Allsopp]

I've pushed a repro case for this to
https://github.com/dra27/cygwin-nanosleep-bug.git

Originally noticed as the main CI system for OCaml has been failing
sporadically for the signal.ml test mentioned in that repo. This morning I
tried hammering that test on my dev machine and discovered that it fails
very frequently. No idea if that's drivers, Windows 10 updates, number of
cores or what, but it was definitely happening, and easily.

Drilling further, it appears that NtQueryTimer is able to return a negative
value in the TimeRemaining field even when SignalState is false. The values
I've seen have always been < 15ms - i.e. less than the timer resolution, so
I wonder if there is a point at which the timer has elapsed but has not been
signalled, but WaitForMultipleObjects returns because of the EINTR signal.
Mildly surprising that it seems to be so reproducible.

Anyway, a patch is attached which simply guards a negative return value. The
test on tbi.SignalState is in theory unnecessary.

All best,


David


0001-Ensure-nanosleep-2-never-returns-negative-rem.patch
Description: Binary data

Regenerate Cygwin FAQ

[David Allsopp via Cygwin]

The instructions for building Cygwin changed in April and the FAQ was
updated (c66797ee), but the website doesn't appear to have been regenerated
(https://cygwin.com/faq.html#faq.programming.building-cygwin).

Thanks!


David


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Building Coq in Cygwin

[David Allsopp via Cygwin]

Marco Atzeri wrote:
> On 06.05.2021 02:56, Eliot Moss wrote:
> > Folks - Before I try to Coq mailing lists, I am wondering if anyone
> > here has had success building Coq under Cygwin.  I've tried the dune
> > and the make approaches, and both fail, in different ways, but
> > seemingly because some components can't deal with the uniquenesses of
> > Cygwin - though they seem to try to provide for it.
> >
> > Regards - Eliot
> >
> 
> another of those software that thinks Automake/cmake are non needed ...
> usually they are a mess to port to un-forecasted platforms.

This has nothing to do with the build systems of these packages and
everything to do with the fact the ocaml on Cygwin64 has been broken for a 
while...

> $ ./configure
>0 [main] ocamlrun 740 child_info_fork::abort: address space needed
> by 'dllunix.so' (0x40) is already occupied ...
> by 'dllunix.so' (0x40) is already occupied
>0 [main] ocamlrun 744 child_info_fork::abort: address space needed
> by 'dllunix.so' (0x40) is already occupied I can not automatically
> find the name of your architecture.
> Give me a name, please [win32 for Win95, Win98 or WinNT]:
>^^ frontline technology I see

:) Amusingly, the git history shows it has been preserved through updates 13 
and 8 years ago,
but the line was originally written in 1999. Mature and (formally) proven is 
possibly the
description the Coq team might prefer!

> How we solve the reloc issue on 64 bit ? I am a bit ocalm rust

I fixed the underlying problem in OCaml 4.12, but I haven't had time to propose
adopting the Cygwin packages yet - I'm hoping to over the next few months.

The short-term workaround is either to use Cygwin32 or to install the opam, 
libgmp-devel and
flexdll 0.39 packages and run:

  opam init
  opam switch -y create coq ocaml-base-compiler.4.12.0
  # This takes a looong time, especially the "make install" step
  opam install -y coq
  eval $(opam env)
  coqtop
  Quit.

(NB flexdll 0.39 must be selected manually or using the new flexdll=0.39-1 
syntax on the command line,
as I've left it marked test until ocaml 4.12 is packaged)

HTH,


David

-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Cygwin base install no longer has iconv binary

[David Allsopp via Cygwin]

[Continuing second point in
https://cygwin.com/pipermail/cygwin-apps/2021-April/041212.html]

The man-db package prior to 2.9.3-1 on 3 Jan depended on the libiconv
package which, unusually, includes the iconv binary itself.

I noticed as OCaml has assumed since 2017 that iconv would be available in
base, and thus on "any" Cygwin installation. I was already working on
removing OCaml's dependency on the iconv binary, so I don't have a strong
opinion either way, but should libiconv be put in the Base category in order
to keep the installed commands consistent?


David


-- 
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: [ITA] man-db

[David Allsopp via Cygwin-apps]

On 03 January 2021 11:17, Achim Gratz wrote:
> Update to version 2.9.3 and cleaning up patches.
> 
> Change packaging so that index creation can be triggered by installing the
> man-db-create-index package.  Postinstall index update is done in the
> background by default so it won't block setup from progressing anymore,
> but can be made synchronous again if needed, via installation of the man-
> db-index-synchronously package.
> 
> https://cygwin.com/git-cygwin-packages/?p=git/cygwin-packages/man-
> db.git;a=shortlog;h=refs/heads/playground
> https://ci.appveyor.com/project/cygwin/scallywag/builds/37084308

(Apologies if this should go the main Cygwin list)

This version of the package removes the dependency on libiconv, which means 
that the base Cygwin install no longer includes the iconv command.

Was that an intentional consequence?


David

RE: [PATCH setup] Add --allow-test-versions

[David Allsopp via Cygwin-apps]

Jon Turney wrote:
> On 20/04/2021 15:37, David Allsopp via Cygwin-apps wrote:
> > Attached adds -t/--allow-test-packages to Setup which controls the
> > initial state of the "Test" checkbox.
> >
> > Motivation is to allow one CI cron job to be installing test versions
> > of packages, then we can help identify things like [1] before they're
> released.
> 
> Nice.  Thanks.
> 
> > +static BoolOption AllowTestOption (false, 't', "allow-test-packages",
> > +"Permit installation of package versions marked test");
> 
> The only quibble I have is the help text here.  The user has always been
> permitted to choose test packages, this checkbox allows setup to decide to
> use them itself.
> 
> So maybe something like "Consider package versions marked test".

No problem - see attached.

All best,


David


0001-Add-allow-test-packages.patch
Description: 0001-Add-allow-test-packages.patch

[PATCH setup] Add --allow-test-versions

[David Allsopp via Cygwin-apps]

Attached adds -t/--allow-test-packages to Setup which controls the initial
state of the "Test" checkbox.

Motivation is to allow one CI cron job to be installing test versions of
packages, then we can help identify things like [1] before they're released.


David


[1] https://cygwin.com/pipermail/cygwin/2021-April/248300.html


0001-Add-allow-test-packages.patch
Description: Binary data

RE: [PATCH setup] Handle '--packages package=version'

[David Allsopp via Cygwin-apps]

Jon Turney wrote:
> Handle '--packages package=version' to allow specifing the version of a
> package to install on the command line.
> 
> isManuallyWanted() now returns the target packageversion (if specified),
> or an empty packageversion (which is translated into an instruction to
> the solver to choose the version).
> 
> In the 'upgrade' case, this changes from using the complex logic of
> packagemeta::trustp() to determine the target version, when unspecified
> on the command line, to allowing the solver to make that decision (which
> should be broadly the same i.e. not downgrading test versions, etc.).
> 
> This also subtly changes the behaviour when the package version is not
> specified.  Instead of forcing the current version, we allow the solver
> to choose the version, so it may not pick that version, if other
> constraints exist, which prevent a solution containing that version.
> 
> There's probably some future work which can be done to simplify the
> remaining uses packagemeta::trustp(), which are only related to the UI.



This is really useful (tested locally for a couple of use-cases: one wanting
a downgrade from latest and the other to manually specify a test version).

Does libsolv give the ability to put more complex constraints? In opam, for
example, I can issue `opam install 'utop<2.7.0' and get the last version
before 2.7.0.

`--packages mingw64-x86_64-binutils=2.35.2-1` is already very useful but
`--packages mingw64-x86_64-binutils<2.36` is even more so.



David

RE: [PATCH 0/2] Fix race issues.

[David Allsopp]

Corinna Vinschen wrote:
> On Apr 19 19:30, Takashi Yano wrote:
> > Takashi Yano (2):
> >   Cygwin: console: Fix race issue regarding cons_master_thread().
> >   Cygwin: pty: Fix race issue in inheritance of pseudo console.
> >
> >  winsup/cygwin/fhandler_console.cc |  10 ++-
> >  winsup/cygwin/fhandler_tty.cc | 108 ++
> >  winsup/cygwin/tty.cc  |  15 ++---
> >  winsup/cygwin/tty.h   |   2 +-
> >  4 files changed, 77 insertions(+), 58 deletions(-)
> >
> > --
> > 2.31.1
> 
> Pushed.

Armed with this morning's Cygwin snapshot, OCaml builds again as well. Many 
thanks!


David

RE: Regression in Cygwin 3.2.0

[David Allsopp via Cygwin]

Takashi Yano wrote:
> On Fri, 16 Apr 2021 11:17:50 +0100
> David Allsopp wrote:
> > I'm unable to build OCaml using the mingw-w64 compilers with Cygwin
> 3.2.0.
> > Windows 10.0.19042.928 (and tried on three different machines so far)
> >
> > Repro:
> >
> >  - Fresh Cygwin64 installation with make, libiconv,
> > mingw64-x86_64-gcc-core and git added; fire up mintty
> >  - git clone --depth 1 --recursive https://github.com/ocaml/ocaml
> >  - cd ocaml && ./configure --host=x86_64-w64-mingw32 && make -j8
> > flexdll && make -j8 world.opt
> >
> > This is consistently hanging with Cygwin 3.2.0 and seems possibly to
> > have something to do with linking (the creation of
> > runtime/ocamlrun.exe and stdlib/camlheader, which is also an
> > executable, seem to be part of the problem).
> >
> > It can be immediately fixed by downgrading to Cygwin 3.1.7, cleaning
> > the tree and repeating the build.
> >
> > I'm of course very happy to try to reduce this to a smaller repro
> > case, but is there an obvious culprit in Cygwin 3.2.0 to check before I
> do?
> 
> Thanks for the report. I looked into this problem and found that the
> caused is the combination of some race issues for non-cygwin processes in
> pty and console code.
> 
> I will submit the patches fixing this issue.

Fantastic, thanks - I'll keep an eye on cygwin-patches and test.


David
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Regression in Cygwin 3.2.0

[David Allsopp via Cygwin]

Thomas Wolff wrote:
> Am 16.04.2021 um 16:07 schrieb David Allsopp via Cygwin:
> > Thomas Wolff wrote:
> >> Am 16.04.2021 um 12:17 schrieb David Allsopp via Cygwin:
> >>> I'm unable to build OCaml using the mingw-w64 compilers with Cygwin
> >>> 3.2.0. Windows 10.0.19042.928 (and tried on three different machines
> >>> so far)
> >>>
> >>> Repro:
> >>>
> >>>- Fresh Cygwin64 installation with make, libiconv,
> >>> mingw64-x86_64-gcc-core and git added; fire up mintty
> >>>- git clone --depth 1 --recursive https://github.com/ocaml/ocaml
> >>>- cd ocaml && ./configure --host=x86_64-w64-mingw32 && make -j8
> >>> flexdll && make -j8 world.opt
> >>>
> >>> This is consistently hanging with Cygwin 3.2.0 and seems possibly to
> >>> have something to do with linking (the creation of
> >>> runtime/ocamlrun.exe and stdlib/camlheader, which is also an
> >>> executable, seem to be part of the problem).
> >>>
> >>> It can be immediately fixed by downgrading to Cygwin 3.1.7, cleaning
> >>> the tree and repeating the build.
> >>>
> >>> I'm of course very happy to try to reduce this to a smaller repro
> >>> case, but is there an obvious culprit in Cygwin 3.2.0 to check
> >>> before I do?
> >> The build fails for me with error messages but it does not hang.
> > Thanks for confirming!
> >
> >> You could have tried at least building without -j option, and running
> >> the build steps separately for better diagnostics.
> > Indeed - the repro case above was intended to be small and fast. I was
> > still at the "is this just me/us", stage. Suggesting a non-parallel
> > build of OCaml would have been mean, as it takes about an hour if it
> > doesn't crash...!
> >
> > That said, the non-parallel build I think reveals the Cygwin problem:
> > there are console windows constantly flashing up (it renders the
> > workstation unusable for the duration of the build). The first stage
> > of the build compiles ocamlrun.exe which is non-Cygwin in this case
> > and then it uses that throughout the build, so I guess it's the conpty
> > changes which are biting.
> >
> > However, the build did succeed when run without -j, however this
> > (which will take about 20 mins or so):
> >
> > ./configure --build=x86_64-w64-mingw32 --disable-native-compiler make
> > flexdll runtop
> >
> > causes a new console Window to pop up with an OCaml prompt (which can
> > be quit by taking #quit;;) rather than the same prompt being connected
> > to mintty.
> >
> > With make -j, it doesn't appear to flash up all the console Windows,
> > but it's also reliably failing. I tried with the cygwin1.dll snapshot
> > from this morning which seems to be doing the same.
> Maybe a non-cygwin tool is injected in your build process?
> What if you strip your PATH from all non-cygwin directories?

There always has been a non-Cygwin tool - it's calling an ocamlrun.exe it us 
built at the start of the process with x86_64-w64-mingw32. Is that no longer 
permitted?!

However, I'm not sure how the environment has anything to do with this: 
everything goes back to working just by downgrading cygwin1.dll to 3.1.7.


David
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Regression in Cygwin 3.2.0

[David Allsopp via Cygwin]

Thomas Wolff wrote:
> Am 16.04.2021 um 12:17 schrieb David Allsopp via Cygwin:
> > I'm unable to build OCaml using the mingw-w64 compilers with Cygwin
> > 3.2.0. Windows 10.0.19042.928 (and tried on three different machines
> > so far)
> >
> > Repro:
> >
> >   - Fresh Cygwin64 installation with make, libiconv,
> > mingw64-x86_64-gcc-core and git added; fire up mintty
> >   - git clone --depth 1 --recursive https://github.com/ocaml/ocaml
> >   - cd ocaml && ./configure --host=x86_64-w64-mingw32 && make -j8
> > flexdll && make -j8 world.opt
> >
> > This is consistently hanging with Cygwin 3.2.0 and seems possibly to
> > have something to do with linking (the creation of
> > runtime/ocamlrun.exe and stdlib/camlheader, which is also an
> > executable, seem to be part of the problem).
> >
> > It can be immediately fixed by downgrading to Cygwin 3.1.7, cleaning
> > the tree and repeating the build.
> >
> > I'm of course very happy to try to reduce this to a smaller repro
> > case, but is there an obvious culprit in Cygwin 3.2.0 to check before I
> > do?
> The build fails for me with error messages but it does not hang.

Thanks for confirming!

> You could have tried at least building without -j option, and running the
> build steps separately for better diagnostics.

Indeed - the repro case above was intended to be small and fast. I was still
at the "is this just me/us", stage. Suggesting a non-parallel build of OCaml
would have been mean, as it takes about an hour if it doesn't crash...!

That said, the non-parallel build I think reveals the Cygwin problem: there
are console windows constantly flashing up (it renders the workstation
unusable for the duration of the build). The first stage of the build
compiles ocamlrun.exe which is non-Cygwin in this case and then it uses
that throughout the build, so I guess it's the conpty changes which are biting.

However, the build did succeed when run without -j, however this (which will
take about 20 mins or so):

./configure --build=x86_64-w64-mingw32 --disable-native-compiler
make flexdll runtop

causes a new console Window to pop up with an OCaml prompt (which can be quit
by taking #quit;;) rather than the same prompt being connected to mintty.

With make -j, it doesn't appear to flash up all the console Windows, but it's
also reliably failing. I tried with the cygwin1.dll snapshot from this morning
which seems to be doing the same.


David
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Regression in Cygwin 3.2.0

[David Allsopp via Cygwin]

I'm unable to build OCaml using the mingw-w64 compilers with Cygwin 3.2.0.
Windows 10.0.19042.928 (and tried on three different machines so far)

Repro:

 - Fresh Cygwin64 installation with make, libiconv, mingw64-x86_64-gcc-core
and git added; fire up mintty
 - git clone --depth 1 --recursive https://github.com/ocaml/ocaml
 - cd ocaml && ./configure --host=x86_64-w64-mingw32 && make -j8 flexdll &&
make -j8 world.opt

This is consistently hanging with Cygwin 3.2.0 and seems possibly to have
something to do with linking (the creation of runtime/ocamlrun.exe and
stdlib/camlheader, which is also an executable, seem to be part of the
problem).

It can be immediately fixed by downgrading to Cygwin 3.1.7, cleaning the
tree and repeating the build.

I'm of course very happy to try to reduce this to a smaller repro case, but
is there an obvious culprit in Cygwin 3.2.0 to check before I do?

Many thanks,


David

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

[ANNOUNCEMENT] Updated: flexdll 0.39-1 (TEST)

[David Allsopp via Cygwin-announce]

The following packages have been updated in the Cygwin distribution as test:

* flexdll-0.39-1.tar.xz

This is an update to the latest upstream release in advance of OCaml 4.12.0,
which requires it in order to work correctly on x86_64 Cygwin.

  *** CYGWIN-ANNOUNCE UNSUBSCRIBE INFO ***

If you want to unsubscribe from the cygwin-announce mailing list, look at
the "List-Unsubscribe: " tag in the email header of this message. Send email
to the address specified there. It will be in the format:

cygwin-announce-unsubscribe-you=yourdomain.com  cygwin.com

If you need more information on unsubscribing, start reading here:

https://sourceware.org/lists.html#unsubscribe-simple

Please read *all* of the information on unsubscribing that is available
starting at this URL.
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Updated: flexdll 0.39-1 (TEST)

[David Allsopp via Cygwin-announce]

The following packages have been updated in the Cygwin distribution as test:

* flexdll-0.39-1.tar.xz

This is an update to the latest upstream release in advance of OCaml 4.12.0,
which requires it in order to work correctly on x86_64 Cygwin.

  *** CYGWIN-ANNOUNCE UNSUBSCRIBE INFO ***

If you want to unsubscribe from the cygwin-announce mailing list, look at
the "List-Unsubscribe: " tag in the email header of this message. Send email
to the address specified there. It will be in the format:

cygwin-announce-unsubscribe-you=yourdomain.com  cygwin.com

If you need more information on unsubscribing, start reading here:

https://sourceware.org/lists.html#unsubscribe-simple

Please read *all* of the information on unsubscribing that is available
starting at this URL.

RE: SSH key for David Allsopp

[David Allsopp via Cygwin-apps]

Jon Turney wrote:
> Sent: 01 December 2020 15:21
> To: cygwin-apps@cygwin.com; David Allsopp 
> Subject: Re: SSH key for David Allsopp
> 
> On 01/12/2020 09:27, David Allsopp via Cygwin-apps wrote:
> >
> > Thanks. I can connect via sftp (or run alive) but I'm getting a
> > publickey error on cyg...@cygwin.com:/git/cygwin-packages/flexdll
> > trying to push a playground branch. Should I be able to?
> 
> Yes.
> 
> Can you be more explicit about what error you are getting?
> 
> I don't know what could cause 'ssh cyg...@cygwin.com alive' to work, but
> 'git push ssh://cyg...@cygwin.com/git/cygwin-packages/flexdll.git' to
> fail.

Ugh - please forgive the noise. PEBKAC and misconfigured Git...

RE: SSH key for David Allsopp

[David Allsopp via Cygwin-apps]

Jon Turney wrote:
> On 30/11/2020 17:30, David Allsopp via Cygwin-apps wrote:
> > Name: David Allsopp
> >  BEGIN SSH2 PUBLIC KEY 
> > Comment: "Cygwin Packaging Key"
> > C3NzaC1lZDI1NTE5IAExhdEPx3iOcR+dSXb/dByk/3+2+SRSU1UYLqGI11u9
> >  END SSH2 PUBLIC KEY 
> 
> Done. Thanks.

Thanks. I can connect via sftp (or run alive) but I'm getting a publickey error 
on cyg...@cygwin.com:/git/cygwin-packages/flexdll trying to push a playground 
branch. Should I be able to?

SSH key for David Allsopp

[David Allsopp via Cygwin-apps]

Name: David Allsopp
 BEGIN SSH2 PUBLIC KEY 
Comment: "Cygwin Packaging Key"
C3NzaC1lZDI1NTE5IAExhdEPx3iOcR+dSXb/dByk/3+2+SRSU1UYLqGI11u9
 END SSH2 PUBLIC KEY 

[ITA] flexdll

[David Allsopp via Cygwin-apps]

OCaml has been partially broken on Cygwin64 since 2018. I did some recent
work upstream to fix it and we're expecting OCaml 4.12.0 by the end of the
year to restore full support.

This requires a new release of flexdll (I also maintain upstream for that).
We could do with an updated FlexDLL package before OCaml 4.12.0 is released
to simplify testing on CI servers.

I'd like to offer to adopt the Cygwin flexdll package, therefore.

The changes for 0.39 are trivial:
https://github.com/dra27/cygport-flexdll/commit/2cb2d6787c56eb1367d7aa4588c5
24db92e45492 and test packages are in
https://github.com/dra27/cygport-flexdll/commit/eb544170af8f2a37b7ec8c7eac95
6a47ae9fef4d.

Thanks,


David

Strange behaviour with winsymlinks:native

[David Allsopp via Cygwin]

I've been doing some working around the problems with Cygwin 3.1.5+ WSL
junction points in Docker and found three unexpected pieces of behaviour
with CYGWIN=winsymlinks:native

In all cases, these work as expected with the default symlink behaviour
(i.e. CYGWIN unset or without a winsymlinks option).

1. Relative paths get unnecessarily resolved

  mkdir -p foo/bar
  cd foo/bar
  touch baz
  CYGWIN='winsymlinks:native' ln -s ../bar/baz link
  readlink link

Result is baz, not ../bar/baz

2a. Links in the magic mount directories (/usr/bin and /usr/lib) get extra
bits added (source is a default mount):

  cd /usr/lib
  CYGWIN='winsymlinks:native' ln -s ../share/terminfo terminfo2
  readlink terminfo2

Result is ../usr/share/terminfo, which makes it valid from /lib (the "real"
directory) but not in the virtual one

2b. Same but where target is a default mount

  cd /usr/libexec/p11-kit
  CYGWIN='winsymlinks:native' ln -s ../../bin/update-ca-trust
trust-extract-compat2
  readlink trust-extract-compat2

Result is ../../../bin/update-ca-trust, again it seems to be computing an
extra level back to the "real" lib directory

Are these behaviours expected? I can create those symbolic links manually
with mklink with no problem, so I'm wondering if it's an outright bug or an
unexpected consequence of something else.

Thanks!


David

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: Debugging Crash of setup-x86.exe?

[David Allsopp via Cygwin]

Jason Gross wrote:
> I'll try disabling AV and report back, thanks for the advice.  The reason
> I'm using 32-bit is because, last I checked (this might have been a year
> or two ago, so my info might be out of date), there were some OCaml or
> opam packages that only worked with 32-bit cygwin.

This is correct - OCaml has been broken on 64-bit Cygwin for a while,
however the forthcoming FlexDLL 0.39 and OCaml 4.12.0 releases (later this
year) restore 64-bit support.


David

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: [ANNOUNCEMENT] unison2.48-2.48.4-2 (Warning: possible breakage)

[David Allsopp via Cygwin]

Andrew Schulman wrote:
> > > There is unfortunately another layer of incompatibility in Unison:
> > > Two Unison executables are only compatible if they were built with
> > > the same version of OCaml.
> >
> > What a mess!
> 
> Glad you understand :)
> 
> > Would you consider embedding the OCaml version in the package name as
> > suggested by the Debian maintainer
> > (http://lists.seas.upenn.edu/pipermail/unison-hackers/2020-
> August/001975.html)?
> > Thus unison2.48-2.48.4-1 would be replaced by something like
> > unison2.48+4.04.1... and unison2.48-2.48.4-2 would be replace by
> > something like unison2.48+4.08.1  That way people who are happy
> > with unison2.48-2.48.4-1 can keep using it without being pestered by
> setup to update.
> 
> Yes, I could do that. I was hoping to avoid it, but it may be the only
> realistic solution.
> 
> It may be mainly Unison 2.48 that's affected, and maybe also 2.51 and the
> forthcoming 2.52. I guess that very few people are still using the older
> versions, and if they are they don't want any changes now.
> 
> > FWIW, here's my situation, which is probably not typical.  When you
> > released
> > unison2.48-2.48.4-1 a few years ago, it was incompatible with the
> > Linux server that I sync with.  So I simply built OCaml and unison
> > myself on that server, using the same versions that you used, and
> > installed unison in my own ~/bin directory.
> >
> > I could do that again, now using OCaml 4.08.1.  But I'd much rather
> > just keep my current arrangement, without having setup try to update
> > unison2.48 every time I run it.

Just for reference, the format of OCaml marshalled values changed in OCaml 4.08 
(https://github.com/ocaml/ocaml/pull/1683). It's changed again in OCaml 4.11 as 
well (https://github.com/ocaml/ocaml/pull/8791) but in that case the new things 
which OCaml 4.11 can marshal would have caused errors in earlier versions, so I 
don't think that will affect Unison.

Quoting https://caml.inria.fr/pub/docs/manual-ocaml/libref/Marshal.html 
(emphasis mine): "The format for the byte sequences is compatible across all 
machines **for a given version of OCaml**" so it's an interesting design choice 
of Unison's.

That said, we (OCaml devs) change the marshal format as infrequently as 
possible, and go to considerable lengths to ensure that the older format can be 
read by newer runtimes: IIRC that particular change was the first break since 
OCaml 4.01 (Sep 2013).


David
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: [PATCH] Fix incorrect sign-extension of pointer in 32-bit acl __to_entry

[David Allsopp via Cygwin-patches]

Corinna Vinschen wrote:
> On Jul 10 15:22, David Allsopp via Cygwin-patches wrote:
> > Corinna Vinschen wrote:
> > > On Jul  9 20:30, David Allsopp via Cygwin-patches wrote:
> > > > I have some code where the acl_t returned by get_file_acl is
> > > > allocated at 0x80038248. As a result the acl_entry_t generated by
> > > > acl_get_entry has an "index" of -1, since the pointer was sign-
> extended to 64-bits.
> > > >
> > > > My fix is trivial and simply casts the pointer to uintptr_t first.
> > >
> > > Pushed.  I still don't quite understand what the compiler is
> > > thinking there, sign-extending a pointer when casted to an unsigend
> > > int type, but your patch works, so all is well, I guess.
> >
> > Thank you - it is indeed hard to imagine when you'd ever want that
> behaviour!
> 
> I wonder if this is a bug in x86 gcc... Jon?

I put it to our C gurus in the OCaml team - one (who has also written a 
formally verified C compiler which on purpose 0-extends in this case) observed 
that GCC does the same for ARM32 and another (who has occasionally delighted in 
abusing language memory models to concoct highly bizarre, but legal, abuses of 
undefined behaviour) found 
https://gcc.gnu.org/onlinedocs/gcc/Arrays-and-pointers-implementation.html... 
although the Git history for GCC makes it fairly clear that this behaviour is 
retrospectively documented[1][2]!


David


[1] https://github.com/gcc-mirror/gcc/commit/cbf4c36fa373
[2] https://github.com/gcc-mirror/gcc/commit/58f4de4f271c

RE: [PATCH] Fix incorrect sign-extension of pointer in 32-bit acl __to_entry

[David Allsopp via Cygwin-patches]

Corinna Vinschen wrote:
> On Jul  9 20:30, David Allsopp via Cygwin-patches wrote:
> > I have some code where the acl_t returned by get_file_acl is allocated
> > at 0x80038248. As a result the acl_entry_t generated by acl_get_entry
> > has an "index" of -1, since the pointer was sign-extended to 64-bits.
> >
> > My fix is trivial and simply casts the pointer to uintptr_t first.
> 
> Pushed.  I still don't quite understand what the compiler is thinking
> there, sign-extending a pointer when casted to an unsigend int type, but
> your patch works, so all is well, I guess.

Thank you - it is indeed hard to imagine when you'd ever want that behaviour!

Would it be possible to have a snapshot with it, just for continuous 
integration servers which need the fix, please?


David

[PATCH] Fix incorrect sign-extension of pointer in 32-bit acl __to_entry

[David Allsopp via Cygwin-patches]

I have some code where the acl_t returned by get_file_acl is allocated at
0x80038248. As a result the acl_entry_t generated by acl_get_entry has an
"index" of -1, since the pointer was sign-extended to 64-bits.

My fix is trivial and simply casts the pointer to uintptr_t first.

All best,


David


0001-Fix-invalid-acl_entry_t-on-32-bit-Cygwin.patch
Description: Binary data

RE: opam package should depend on ocaml-compiler-libs

[David Allsopp via Cygwin]

Achim Gratz wrote:
> Brian Inglis writes:
> >>> Cygwin packages are granular and dependencies are functional: which
> >>> of the Cygwin packages opam and opam-installer uses Cygwin package
> >>> ocaml- compiler-libs, or does opam use another ocaml package to build?
> 
> >> I don't quite understand your question - opam is OCaml's package
> manager.
> 
> The question was if adding ocaml-compiler-libs to the opam requirements is
> sufficient or if there are other packages that need to be added as
> requirements for opam.  A second related question is if any of the
> existing ocaml packages are missing dependencies, especially to ocaml-
> compiler-libs.  Finally, since opam is split into two packages on Cygwin,
> which of these needs the requirement.

Indeed, adding ocaml-compiler-libs as a dependency of opam is sufficient 
(opam-installer does not in any way depend on it). The rest of opam's non-build 
dependencies were dealt with in 2018 (see 
https://github.com/andyli/opam-cygwin/blame/master/opam.cygport#L22-L23 and the 
linked https://sourceware.org/legacy-ml/cygwin/2018-01/msg00079.html).

> >> It can either compile OCaml itself from sources or use the OCaml
> >> installed by the system's package manager. In the second case, it
> >> quite reasonably assumes that the OCaml which has been installed is
> >> the one the OCaml developers intended which includes its
> >> compiler-libs package. Given that opam depends on ocaml, it should
> >> also depend on ocaml-compiler-libs, therefore.
> >
> > What you or the ocaml developers assume does not matter: what matters
> > is how the package works. If you do not understand the question and
> > can not provide the answer, I can see why the maintainer made his
> > choice: he probably never used the packages, just ran any tests
> included.
> 
> That sounds unnecessarily harsh to me.  It would however be helpful if the
> opam maintainer could have a look, since he is undoubtedly much more aware
> of the issues on both the OCaml and Cygwin sides than any one of us and
> could probably explain why the packages are built the way they are.

The dependencies were modified before so that `opam init` succeeds without it 
being:

- install Cygwin's opam package
- Run opam init
- , re-run Cygwin setup and install a load more Cygwin packages
- Run opam init again

The additional dependency ensures that the next command many users would then 
run `opam install ocamlfind` also succeeds.

> > It appears that neither ocaml nor opam use ocaml-compiler-libs,
> > otherwise the cygwin build and packaging software would include it, so
> > which package in the ocaml suite actually binds the
> > ocaml-compiler-libs to do its job, or does that package just provide
> > dynamic libraries which are expected to be present when anyone runs
> ocaml programs?
> > In which case users would be expected to manually install the package
> > ocaml-compiler-libs.
> 
> It's not uncommon that these captive "package managers" assume a certain
> environment without depending directly on it, so you'll indeed have to
> manually add the pre-requisites as manual requirements for the Cygwin
> package.  That doesn't solve the problem that usually language
> environments that come with their own package manage don't quite play nice
> with the system package manager, especially if the system is not
> GNU/Linux.

We do our best - the opam developers use a mix of FreeBSD/OpenBSD/macOS/Windows 
& Debian, so we try to remain accommodating by design :)

Thanks,


David 
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

RE: opam package should depend on ocaml-compiler-libs

[David Allsopp via Cygwin]

Brian Inglis wrote:
> On 2020-05-28 03:28, David Allsopp via Cygwin wrote:
> > opam assumes that OCaml installed by the "OS" package manager is
> "complete"
> > (i.e. is the same as "make install" from the OCaml sources), which is
> > a problem when "OS" package managers split upstream ocaml and don't
> > install the ocaml-compiler-libs package by default.
> >
> > Please could either the opam or ocaml package be updated to depend on
> > ocaml-compiler-libs. Fixes
> > https://github.com/ocaml/opam-repository/issues/16457.
> >
> > Debian/Ubuntu and Fedora both also have ocaml-compiler-libs packages,
> > but it's installed by their ocaml package as well.
> >
> > Arch chooses not to do this, having packages like ocaml-findlib
> > expressly depending on ocaml-compilerlibs and so also has its opam
> > package depend on ocaml-compilerlibs.
> >
> > As both an upstream OCaml and opam maintainer, I don't mind which
> > package has the dependency, but if opam is installed, please can any
> > system-installed ocaml definitely be "complete"!
> 
> [current maintainer has resigned so packages are up for adoption]
> 
> Cygwin packages are granular and dependencies are functional: which of the
> Cygwin packages opam and opam-installer uses Cygwin package ocaml-
> compiler-libs, or does opam use another ocaml package to build?
> 
> Current packages are:
> emacs-ocaml 4.10.0-1 x86_64
> ocaml 4.10.0-1 x86_64
> ocaml-base 4.10.0-1 x86_64
> ocaml-cairo2 0.6.1-1 x86_64
> ocaml-cairo2-gtk 0.6.1-1 x86_64
> ocaml-cairo2-pango 0.6.1-1 x86_64
> ocaml-calendar 2.04-2 x86_64
> ocaml-camlp4 4.10-0.1 x86_64
> ocaml-camlp5 7.11-1 x86_64
> ocaml-camomile 1.0.2-1 x86_64
> ocaml-camomile-data 1.0.2-1 x86_64
> ocaml-compiler-libs 4.10.0-1 x86_64
> ocaml-cppo 1.6.6-1 x86_64
> ocaml-csv 1.7-1 x86_64
> ocaml-ctypes 0.17.1-1 x86_64
> ocaml-curses 1.0.4-1 x86_64
> ocaml-curses-debuginfo 1.0.4-1 x86_64
> ocaml-debuginfo 4.04.2-1 x86_64
> ocaml-doc 4.10.0-1 x86_64
> ocaml-dune 2.5.1-1 x86_64
> ocaml-extlib 1.7.7-1 x86_64
> ocaml-facile 1.1.3-3 x86_64
> ocaml-fileutils 0.6.2-1 x86_64
> ocaml-findlib 1.8.1-1 x86_64
> ocaml-gettext 0.4.1-1 x86_64
> ocaml-graphics 5.1.0-1 x86_64
> ocaml-gv 2.40.1-5 x86_64
> ocaml-integers 0.3.0-1 x86_64
> ocaml-lablgl 1.06-1 x86_64
> ocaml-lablglade2 2.18.10-1 x86_64
> ocaml-lablgnomecanvas 2.18.10-1 x86_64
> ocaml-lablgnomeui 2.18.10-1 x86_64
> ocaml-lablgtk2 2.18.10-1 x86_64
> ocaml-lablgtk3 3.1.0-1 x86_64
> ocaml-lablgtk3-gtkspell3 3.1.0-1 x86_64
> ocaml-lablgtk3-sourceview3 3.1.0-1 x86_64
> ocaml-lablgtkgl2 2.18.10-1 x86_64
> ocaml-lablgtksourceview2 2.18.10-1 x86_64 ocaml-lablgtkspell 2.18.10-1
> x86_64 ocaml-lablrsvg 2.18.10-1 x86_64 ocaml-labltk 8.06.8-1 x86_64 ocaml-
> libvirt 0.6.1.5-1 x86_64 ocaml-libvirt-debuginfo 0.6.1.5-1 x86_64 ocaml-
> llvm 8.0.1-1 x86_64 ocaml-llvm-doc 8.0.1-1 x86_64 ocaml-num 1.3-1 x86_64
> ocaml-ocamlbuild 0.14.0-1 x86_64 ocaml-ocamldoc 4.10.0-1 x86_64 ocaml-
> ounit 2.2.2-1 x86_64 ocaml-result 1.5-1 x86_64 ocaml-runtime 4.10.0-1
> x86_64 ocaml-source 4.10.0-1 x86_64 ocaml-topkg 1.0.1-1 x86_64 ocaml-xml-
> light 2.3-0.2.svn234 x86_64

I don't quite understand your question - opam is OCaml's package manager.

It can either compile OCaml itself from sources or use the OCaml installed by 
the system's package manager. In the second case, it quite reasonably assumes 
that the OCaml which has been installed is the one the OCaml developers 
intended which includes its compiler-libs package. Given that opam depends on 
ocaml, it should also depend on ocaml-compiler-libs, therefore.


David 
--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

opam package should depend on ocaml-compiler-libs

[David Allsopp via Cygwin]

opam assumes that OCaml installed by the "OS" package manager is "complete"
(i.e. is the same as "make install" from the OCaml sources), which is a
problem when "OS" package managers split upstream ocaml and don't install
the ocaml-compiler-libs package by default.

Please could either the opam or ocaml package be updated to depend on
ocaml-compiler-libs. Fixes
https://github.com/ocaml/opam-repository/issues/16457.

Debian/Ubuntu and Fedora both also have ocaml-compiler-libs packages, but
it's installed by their ocaml package as well.

Arch chooses not to do this, having packages like ocaml-findlib expressly
depending on ocaml-compilerlibs and so also has its opam package depend on
ocaml-compilerlibs.

As both an upstream OCaml and opam maintainer, I don't mind which package
has the dependency, but if opam is installed, please can any
system-installed ocaml definitely be "complete"!

Many thanks,


David

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Packaging request: mingw-w64 versions of libargon2

[David Allsopp]

Hi (Yaakov!),

Please could libargon2 be packaged for the two mingw-w64 builds?
I've recently opened https://github.com/P-H-C/phc-winner-argon2/pull/285
which upstreams (and I think corrects) the patches in MSYS2. I also opened
https://github.com/P-H-C/phc-winner-argon2/pull/286 which includes your
build patches for Cygwin and suggests a build route for mingw-w64 (naming
the DLL cygargon2-1.dll for Cygwin and libargon2-1.dll for mingw-w64).

Many thanks (both hopefully in advance for the packaging but regardless for
the Cygwin patches in the first place, which eased my private building of
it),


David

--
Problem reports:  https://cygwin.com/problems.html
FAQ:  https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple

Re: Any chance of updating the ocaml package?

[David Allsopp]

On 28 Nov 2019, at 15:09, Andy Li  wrote:
> 
> Hi,
> 
> The ocaml package is a bit dated.
> The currently packaged version is 4.04.2, released Jun 23, 2017.
> The latest ocaml version is 4.09.0, released Sep 18, 2019.
> It would be great to have it updated.
> 

There’s a problem which needs to be fixed in flexdll which prevents OCaml 
working properly on Cygwin64 at the moment. When we’ve fixed that, it’ll be 
possible to update the OCaml package. In the meantime, on Cygwin32, opam works 
to provide the more recent versions.


David

> Best,
> Andy
> 
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
> 

RE: Bug Report: Regression in Cygwin 2.11.0-1

[David Allsopp]

Corinna Vinschen wrote:
> On Sep  1 16:41, David Allsopp wrote:
> > Corinna Vinschen wrote:
> > > Some of the path handling is seriously crippled as soon as you start
> > > using backslashes, and that's a delipberate decision and won't
> > > change, even after fixing the aforementioned bug.
> >
> > I don't quite follow this - does that mean that that path definitely
> > work even with a new Cygwin? i.e. our existing build system is
> > permanently broken for Cygwin 2.11+?
> 
> No.  The problem we're talking about here is a regression and needs fixing, no
> doubt about it.
> 
> I was just saying that win32 paths *in general* get a less complete treatment
> than POSIX paths for hopefully obvious reasons.  Just be aware that \foo\bar 
> is
> not the same as /foo/bar and may even point to another file.
>
> As for the bug in question:  I pushed a patch which should fix this issue.  I
> created new developer snapshots and uploaded them to
> https://cygwin.com/snapshots/.  Please give them a try.

Thanks! That snapshot does indeed fix our build.

How feasible would it be to generate an alternate setup.ini 
(setup-snapshots.ini or something) and include the snapshots in the actual 
mirror with a switch to setup to select the alternate file? When we finally get 
to it with OCaml's CI, that is probably how I would configure our CI to operate 
(i.e. synchronise with a mirror, download the latest snapshot and then patch 
setup.ini), but I imagine there might be other projects which would set jobs 
running if it were very simple from upstream to do. 


David

RE: Bug Report: Regression in Cygwin 2.11.0-1

[David Allsopp]

cyg Simple wrote:
> On 9/1/2018 5:52 PM, Andrey Repin wrote:
> > Greetings, David Allsopp!
> >
> >>> In terms of this OCAML build system problem:
> >>>
> >>> Please fix your build system.  Do not mix POSIX and Win32 paths, use
> >>> POSIX paths only.  Backslashes are *no* drop-in replacement for slashes.
> >
> >> The "problem" for us is more subtle than this. The program which is
> >> generating that path is not a Cygwin executable, so it is correctly
> >> combining a path it has been supplied (the ../stdlib which has been
> >> supplied via Cygwin's make) with a filename, so it uses a backslash.
> >> It's been on my TODO for years to enhance to understand that the
> >> program it's supplying the path back to is a Cygwin executable and so
> >> sort it out properly but, well, we're a small number of maintainers! That
> said, WSL is forcing us to address it properly...
> >
> > You CAN just send back forward slashes. Windows don't care.

That's not strictly true, as mkdir foo/bar will demonstrate in a command prompt 
(it is almost true when dealing with the Windows API). However, IMO, one looks 
like a Unix programmer who doesn't really understand Windows when your programs 
visibly use forward slashes everywhere. 

> > If any program would choke, it would be that program's problem after all.
> 
> Certainly not "that program's problem". The problem becomes mine and yours
> as we've not followed the design of the program.  While Windows at the
> moment doesn't care there is always the possibility that some fix could break
> that since the documentation states to use \ in paths and not /.  So while we
> "CAN just send back forward slashes" we need to be prepared for the outcome
> if something changes in an update to the OS and breaks the assumptions based
> on observed behavior rather than the documented behavior.

Just as an aside, the use of slashes is documented: see Kuhne's compulsory 
reading on the subject at 
https://blogs.msdn.microsoft.com/jeremykuhne/2016/04/21/path-normalization/ and 
also the notes in MSDN 
(https://docs.microsoft.com/en-gb/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation)
 which do mention that normalisation of / to \ in the API is not an accident.


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Bug Report: Regression in Cygwin 2.11.0-1

[David Allsopp]

Corinna Vinschen wrote:
> On Sep  1 09:56, Andreas Hauptmann wrote:
> > On Sat, 1 Sep 2018 01:24:49 +
> > Bryan Phelps wrote:
> >
> > > I'll continue to look around for a more minimal repro,
> >
> > The normalization of paths with backslashes has changed.
> >
> > The following doesn't work any longer:
> >
> > cd /tmp
> > stat "..\bin\file.exe" # or
> > stat "..\\bin\\file.exe"
> >
> > This however still works:
> >
> > stat "C:\cygwin\bin\file.exe"
> 
> I know where the problem is.  A new piece of code fixing a crash due to a 
> buffer
> underflow is too aggressively guarding the path buffer against this underflow
> when normalizing ".." in Win32 paths.
> 
> I'll fix this and release a 2.11.1 soon, but I still have a question:
> 
> Why do I push out test releases if nobody cares?

Like you, we're a (very) small number of maintainers - and, as far as I'm 
aware, the command line setup doesn't exactly make continuous integration 
testing of Cygwin test versions "trivial". I'd love to have had the time within 
the last 22 days to service my care...

> In terms of this OCAML build system problem:
> 
> Please fix your build system.  Do not mix POSIX and Win32 paths, use POSIX
> paths only.  Backslashes are *no* drop-in replacement for slashes.

The "problem" for us is more subtle than this. The program which is generating 
that path is not a Cygwin executable, so it is correctly combining a path it 
has been supplied (the ../stdlib which has been supplied via Cygwin's make) 
with a filename, so it uses a backslash. It's been on my TODO for years to 
enhance to understand that the program it's supplying the path back to is a 
Cygwin executable and so sort it out properly but, well, we're a small number 
of maintainers! That said, WSL is forcing us to address it properly... 

> Some of the path handling is seriously crippled as soon as you start using
> backslashes, and that's a delipberate decision and won't change, even after
> fixing the aforementioned bug.

I don't quite follow this - does that mean that that path definitely work even 
with a new Cygwin? i.e. our existing build system is permanently broken for 
Cygwin 2.11+?


David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Bug Report: Regression in Cygwin 2.11.0-1

[David Allsopp]

Marco Atzeri wrote:
> Am 01.09.2018 um 03:24 schrieb Bryan Phelps:
> > Hello,
> >
> >
> > Thank you for all the work on Cygwin! I've been using it to spin up an
> environment to build the OCaml compiler / toolchain, and it was working great.
> >
> >
> > However, today, all our CI builds mysteriously started failing - at first, I
> suspected it was a problem with AppVeyor, but I also failures with VSTS. We
> use an NPM package (`esy-bash`) to spin up a Cygwin environment, and then
> use that to build the OCaml toolchain.
> >
> >
> > The error message we started receiving today is:
> >
> > OCAML_FLEXLINK="../boot/ocamlrun ../flexdll/flexlink.exe"
> ../byterun/ocamlrun ../ocamlc -g -nostdlib -I ../utils -I ../parsing -I 
> ../stdlib -I
> ../compilerlibs -strict-sequence -safe-string -strict-formats -w 
> +a-4-9-41-42-44-
> 45-48 -warn-error A -custom ocamlcommon.cma -o ocamltest.exe run_win32.o
> run_stubs.o ocamltest_config.cmo testlib.cmo run_command.cmo filetype.cmo
> filecompare.cmo backends.cmo variables.cmo environments.cmo
> builtin_variables.cmo builtin_modifiers.cmo actions.cmo builtin_actions.cmo
> tests.cmo builtin_tests.cmo tsl_ast.cmo tsl_parser.cmo tsl_lexer.cmo
> tsl_semantics.cmo options.cmo main.cmo
> > x86_64-w64-mingw32-gcc: error: ../stdlib\libcamlrun.a: No such file or
> directory
> >
> 
> Question:
> is libcamlrun.a built correctly and in the same directory than before ?

I haven't double-checked, but the build cannot have got this far if the file 
were really missing.

> This mixed "../stdlib\libcamlrun.a" slash looks strange

Yeah, our build system is not terribly hygienic when it comes to mixing forward 
and backslashes in the Windows builds (the Windows shell of course is mostly 
forgiving of the mixture).

It's not immediately clear to me from the announcement whether this change in 
behaviour is a regression or an intentional change? Only asking in terms of how 
quickly we need to make OCaml's build system cleaner.

It's worth noting in this instance that x86_64-w64-mingw32-gcc would be being 
invoked by a non-Cygwin executable here (i.e. which IIRC is slightly different 
path normalisation than when it's invoked from bash).


David

RE: Cygwin x86 on Windows 10 ARM64

[David Allsopp]

David Allsopp wrote:
> Corinna Vinschen wrote:



> > This looks like a bug in the emulator.  You may want to contact
> > Microsoft.
> 
> Indeed - I can't install the fast ring insider build on this machine
> (driver problem ) but I'm now trying the slow ring instead.

And, FWIW, the same behaviour 10.0.17692.1004, so I guess treating 0 as meaning 
"x86 on arm64" is probably the way forward!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin x86 on Windows 10 ARM64

[David Allsopp]

Corinna Vinschen wrote:
> On Jul 12 07:46, David Allsopp wrote:
> > Corinna Vinschen wrote:
> > > On Jul 10 10:51, David Allsopp wrote:
> > > > I've been trying out the x86 emulation in Microsoft's ARM64
> > > > version of Windows 10 1803.
> > > >
> > > > I had two issues with Cygwin x86. The first, which is simple, is
> > > > that Windows doesn't by default create
> > > > C:\Windows\SysWOW64\drivers\etc which causes
> > > > /etc/postinstall/base-files-mketc.sh to exit with an error all the
> > > > time. I wonder if there's a possible workaround to make
> > > that less intrusive?
> > >
> > > Try if C:\Windows\Sysnative\drivers\etc works.  That should be the
> > > easiest way to fix the issue in the script.
> >
> > It does indeed. Certainly seems like a good fallback (if not possible
> > default, although I'm sure someone out there takes advantage of a
> > different hosts file between 32-bit and 64-bit!!). I'm happy to tweak
> > the script if you can remind me where its repo is?
> 
> https://sourceware.org/cygwin-apps/ has a list of Cygwin-specific projects
> hosted on cygwin.com.  The base-files project is maintained by Achim
> Gratz.  Please send patches to the cygwin-apps mailing list.

Thanks - will do!

> > > > The error message implies that it may have computed the wrong
> > > > directory, which it hasn't - it's just that the directory doesn't
> > > > exist.
> > > >
> > > > The other is that all Cygwin binaries are emitting the "Could not
> > > > compute FAST_CWD pointer" warning.
> > >
> > > Nothing we can do about, unless somebody dives into assembler code
> > > on such a system.  If the code switches to ARM64 early, this could
> > > be tricky.
> >
> > The machine I'm using is only for testing on this platform - I can
> > grant access to it if it'd be worth looking into?
> >
> > > As a workaround I pushed a patch to check for running in WOW64 under
> > > ARM64.  The warning is skipped then.  The already existing fallback
> > > code should work most of the time.  Just give the latest developer
> > > snapshot from https://cygwin.com/snapshots/ a try.
> >
> > OK, so this is very weird - both GetNativeSystemInfo and GetSystemInfo
> > are returning 0 in both wProcessorArchitecture and wReserved (and FWIW
> > 586 in dwProcessorType). This is with GCC 6.4.0 (i686-w64-mingw32-gcc)
> > and with Microsoft's own **x86** Cl (19.15.26629.1 in VS 2017.8
> > Preview 4). My test program is simply:
> 
> This looks like a bug in the emulator.  You may want to contact Microsoft.

Indeed - I can't install the fast ring insider build on this machine (driver 
problem ) but I'm now trying the slow ring instead.

> Nevertheless, we can use the current buggy reply to our advantage:
> We know we're running in an emulator.  The value of wProcessorArchitecture
> returned by GetNativeSystemInfo should never be 0.  6, 9, 12 are ok, but
> 0???

Seems reasonable!

> So, if the GetNativeSystemInfo returns 0 we can still skip the warning.
> For completeness, I'd like to see the output of `uname -a'
> in Cygwin, though.

CYGWIN_NT-10.0-WOW Envy 2.11.0(0.327/5/3)  i686 Cygwin


David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Cygwin x86 on Windows 10 ARM64

[David Allsopp]

Corinna Vinschen wrote:
> On Jul 10 10:51, David Allsopp wrote:
> > I've been trying out the x86 emulation in Microsoft's ARM64 version of
> > Windows 10 1803.
> >
> > I had two issues with Cygwin x86. The first, which is simple, is that
> > Windows doesn't by default create C:\Windows\SysWOW64\drivers\etc
> > which causes /etc/postinstall/base-files-mketc.sh to exit with an
> > error all the time. I wonder if there's a possible workaround to make
> that less intrusive?
> 
> Try if C:\Windows\Sysnative\drivers\etc works.  That should be the easiest
> way to fix the issue in the script.

It does indeed. Certainly seems like a good fallback (if not possible default, 
although I'm sure someone out there takes advantage of a different hosts file 
between 32-bit and 64-bit!!). I'm happy to tweak the script if you can remind 
me where its repo is?

> > The error message implies that it may have computed the wrong
> > directory, which it hasn't - it's just that the directory doesn't exist.
> >
> > The other is that all Cygwin binaries are emitting the "Could not
> > compute FAST_CWD pointer" warning.
> 
> Nothing we can do about, unless somebody dives into assembler code on such
> a system.  If the code switches to ARM64 early, this could be tricky.

The machine I'm using is only for testing on this platform - I can grant access 
to it if it'd be worth looking into?

> As a workaround I pushed a patch to check for running in WOW64 under
> ARM64.  The warning is skipped then.  The already existing fallback code
> should work most of the time.  Just give the latest developer snapshot
> from https://cygwin.com/snapshots/ a try.

OK, so this is very weird - both GetNativeSystemInfo and GetSystemInfo are 
returning 0 in both wProcessorArchitecture and wReserved (and FWIW 586 in 
dwProcessorType). This is with GCC 6.4.0 (i686-w64-mingw32-gcc) and with 
Microsoft's own **x86** Cl (19.15.26629.1 in VS 2017.8 Preview 4). My test 
program is simply:

#define WINVER 0x0A00
#define _WIN32_WINNT 0x0A00
#include 
#include 

int main(void) {
  SYSTEM_INFO si;
  GetNativeSystemInfo();
  printf("si.wProcessorArchitecture = %d\n", si.wProcessorArchitecture);
  printf("si.wReserved = %d\n", si.wReserved);
  printf("si.dwProcessorType = %d\n", si.dwProcessorType);
  GetSystemInfo();
  printf("si.wProcessorArchitecture = %d\n", si.wProcessorArchitecture);
  printf("si.wReserved = %d\n", si.wReserved);
  printf("si.dwProcessorType = %d\n", si.dwProcessorType);
}

However, when built with Microsoft's ARM64 CL, I get wProcessorArchitecture == 
12, as expected. I'm currently on Windows 10.0.17134.137 (1803). I've just 
switched the machine to the Fast Insider Ring to see if it's the same.

The PROCESSOR_ARCHITEW6432 variable is at least ARM64! 


David

RE: Cygwin x86 on Windows 10 ARM64

[David Allsopp]

Corinna Vinschen wrote:
> On Jul 11 00:36, Andrey Repin wrote:
> > Greetings, David Allsopp!
> >
> > > Brian Inglis wrote:
> > >> On 2018-07-10 03:51, David Allsopp wrote:
> > >> > I've been trying out the x86 emulation in Microsoft's ARM64
> > >> > version of Windows 10 1803.
> > >> >
> > >> > I had two issues with Cygwin x86. The first, which is simple, is
> > >> > that Windows doesn't by default create
> > >> > C:\Windows\SysWOW64\drivers\etc which causes
> > >> > /etc/postinstall/base-files-mketc.sh to exit with an error all
> > >> > the time. I wonder if there's a possible workaround to make
> > >> that less intrusive?
> > >> > The error message implies that it may have computed the wrong
> > >> > directory, which it hasn't - it's just that the directory doesn't
> exist.
> > >>
> > >> Do C:\Windows\{System32,SysNative,Sys*}\drivers\etc exist under the
> > >> emulator?
> > >> What does "cygpath -SU" show?
> >
> > > cygpath -SU gives /proc/cygdrive/c/Windows/SysWOW64
> >
> > > Checking within C:\Windows\SysWOW64\cmd.exe (i.e. x86 cmd):
> >
> > > C:\Windows\System32\drivers\etc exists and contains expected files
> > > C:\Windows\Sysnative\drivers\etc exists and contains expected files
> > > (looks identical to System32, as you'd hope)
> > > C:\Windows\SysArm32\drivers does not contain an etc directory (but
> > > wasn't referenced) C:\Windows\SysWOW64\drivers didn't contain an etc
> > > directory (I created one to silence the error)
> >
> > > So perhaps on WOW64 it would be worth falling back to Sysnative if
> > > etc isn't found? Or even just using Sysnative by default in WOW64?
> >
> > I'd suggest using Sysnative unconditionally, and fall back to System32
> > if Sysnative not found.
> 
> 
> Did anybody notice my reply from 8 hours earlier?
> https://cygwin.com/ml/cygwin/2018-07/msg00091.html

Yes! I answered Brian's first because it didn't involve downloading anything, 
then I tried the snapshot you'd kindly provided - but it doesn't work, for 
weird reasons, and I'm still looking into exactly why in order to give a 
helpful reply as to why!

Sorry for the lag...


David

RE: Cygwin x86 on Windows 10 ARM64

[David Allsopp]

Brian Inglis wrote:
> On 2018-07-10 03:51, David Allsopp wrote:
> > I've been trying out the x86 emulation in Microsoft's ARM64 version of
> > Windows 10 1803.
> >
> > I had two issues with Cygwin x86. The first, which is simple, is that
> > Windows doesn't by default create C:\Windows\SysWOW64\drivers\etc
> > which causes /etc/postinstall/base-files-mketc.sh to exit with an
> > error all the time. I wonder if there's a possible workaround to make
> that less intrusive?
> > The error message implies that it may have computed the wrong
> > directory, which it hasn't - it's just that the directory doesn't exist.
> 
> Do C:\Windows\{System32,SysNative,Sys*}\drivers\etc exist under the
> emulator?
> What does "cygpath -SU" show?

cygpath -SU gives /proc/cygdrive/c/Windows/SysWOW64

Checking within C:\Windows\SysWOW64\cmd.exe (i.e. x86 cmd):

C:\Windows\System32\drivers\etc exists and contains expected files
C:\Windows\Sysnative\drivers\etc exists and contains expected files (looks 
identical to System32, as you'd hope)
C:\Windows\SysArm32\drivers does not contain an etc directory (but wasn't 
referenced)
C:\Windows\SysWOW64\drivers didn't contain an etc directory (I created one to 
silence the error)

So perhaps on WOW64 it would be worth falling back to Sysnative if etc isn't 
found? Or even just using Sysnative by default in WOW64?

> > The other is that all Cygwin binaries are emitting the "Could not
> > compute FAST_CWD pointer" warning.
> 
> It is just a warning and everything should run normally: what would
> normally be an ntdll entry point on x86 is probably a thunk or trampoline
> to
> arm64(ISA)/aarch64(mode)/armv8(design) code.

Does it potentially make anything slower? (being slightly lazy - I could look 
at the code and see precisely what it's for).


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Cygwin x86 on Windows 10 ARM64

[David Allsopp]

I've been trying out the x86 emulation in Microsoft's ARM64 version of
Windows 10 1803.

I had two issues with Cygwin x86. The first, which is simple, is that
Windows doesn't by default create C:\Windows\SysWOW64\drivers\etc which
causes /etc/postinstall/base-files-mketc.sh to exit with an error all the
time. I wonder if there's a possible workaround to make that less intrusive?
The error message implies that it may have computed the wrong directory,
which it hasn't - it's just that the directory doesn't exist.

The other is that all Cygwin binaries are emitting the "Could not compute
FAST_CWD pointer" warning.

Everything's up-to-date and I also tried it with the 2018-06-29 cygwin1.dll
snapshot.

Very happy to test things and poke around, I'd just need some pointers, if
you'll excuse the pun.


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: umask not working?

[David Allsopp]

Ken Brown wrote:
> On 3/21/2018 6:36 AM, David Allsopp wrote:
> > Ken Brown
> >> On 3/19/2018 8:48 AM, David Allsopp wrote:
> >>> Is this expected behaviour:
> >>>
> >>> OPAM+DRA@OPAM ~
> >>> $ uname -a ; umask ; touch /tmp/foo ; ls -l /tmp/foo ; mkdir
> >>> /tmp/bar ; touch /tmp/bar/foo ; ls -l /tmp/bar/foo CYGWIN_NT-6.1-WOW
> >>> OPAM
> >>> 2.10.0(0.325/5/3) 2018-02-02 15:21 i686 Cygwin
> >>> 0022
> >>> -rw-r--r-- 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/foo
> >>> -rw-rw-r--+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar/foo
> >>>
> >>> Why does the file /tmp/bar/foo get g+w when /tmp/foo doesn't - I'm
> >>> not sure what to look at on my system to diagnose what I may have
> >>> inadvertently tweaked. The directory itself is:
> >>>
> >>> drwxr-xr-x+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar
> >>
> >> See if this helps:
> >>
> >> https://cygwin.com/faq/faq.html#faq.using.same-with-permissions
> >
> > Thanks for the pointer. I wonder from it if this could be to do with
> the Cygwin installation being old (but upgraded). I tried on the same
> machine creating another installation to C:\cygwin2 (which behaves as
> Roger Wells noted) and then ran getfacl /tmp on each:
> >
> > Old installation:
> >
> > # file: /tmp
> > # owner: OPAM+DRA-Admin
> > # group: OPAM+None
> > user::rwx
> > user:OPAM+DRA:rwx
> > group::r-x
> > mask:rwx
> > other:r-x
> > default:user::rwx
> > default:user:OPAM+DRA:rwx
> > default:group::r-x
> > default:mask:rwx
> > default:other:r-x
> >
> > Fresh installation:
> >
> > # file: /tmp
> > # owner: OPAM+DRA-Admin
> > # group: OPAM+None
> > # flags: --t
> > user::rwx
> > group::rwx
> > other:rwx
> > default:user::rwx
> > default:group::r-x
> > default:other:r-x
> >
> > I expect that the extra OPAM+DRA:rwx on the old installation was
> manually added by me, years ago. What are the "mask" entries all about?
> >
> > The default:mask entry seems to be the crucial one, as if I do setfacl
> default:mask:rwx /tmp on the fresh installation, then I get the same
> behaviour as on the old installation.
> >
> > However, I'm struggling to find references for either what these mask
> entries are, or how they ever appeared?
> 
> If you search the web for "Posix acl mask" you'll find lots of
> information.  Here's one that seems pretty good:
> 
>https://cs.unc.edu/help-article/posix-acls-in-linux/

Indeed, I got a lot further once I stopped looking for Cygwin-specific info. 
The most useful part was eventually finding this in the umask(2) man page 
(http://man7.org/linux/man-pages/man2/umask.2.html): "Alternatively, if the 
parent directory has a default ACL (see acl(5)), the umask is ignored". Which 
explains why I was seeing that behaviour, and it was owing to having added my 
user account (the OPAM+DRA) to the ACL as that of course added w to the mask.

Thanks for the help!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: umask not working?

[David Allsopp]

Ken Brown
> On 3/19/2018 8:48 AM, David Allsopp wrote:
> > Is this expected behaviour:
> >
> > OPAM+DRA@OPAM ~
> > $ uname -a ; umask ; touch /tmp/foo ; ls -l /tmp/foo ; mkdir /tmp/bar
> > ; touch /tmp/bar/foo ; ls -l /tmp/bar/foo CYGWIN_NT-6.1-WOW OPAM
> > 2.10.0(0.325/5/3) 2018-02-02 15:21 i686 Cygwin
> > 0022
> > -rw-r--r-- 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/foo
> > -rw-rw-r--+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar/foo
> >
> > Why does the file /tmp/bar/foo get g+w when /tmp/foo doesn't - I'm not
> > sure what to look at on my system to diagnose what I may have
> > inadvertently tweaked. The directory itself is:
> >
> > drwxr-xr-x+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar
> 
> See if this helps:
> 
>https://cygwin.com/faq/faq.html#faq.using.same-with-permissions

Thanks for the pointer. I wonder from it if this could be to do with the Cygwin 
installation being old (but upgraded). I tried on the same machine creating 
another installation to C:\cygwin2 (which behaves as Roger Wells noted) and 
then ran getfacl /tmp on each:

Old installation:

# file: /tmp
# owner: OPAM+DRA-Admin
# group: OPAM+None
user::rwx
user:OPAM+DRA:rwx
group::r-x
mask:rwx
other:r-x
default:user::rwx
default:user:OPAM+DRA:rwx
default:group::r-x
default:mask:rwx
default:other:r-x

Fresh installation:

# file: /tmp
# owner: OPAM+DRA-Admin
# group: OPAM+None
# flags: --t
user::rwx
group::rwx
other:rwx
default:user::rwx
default:group::r-x
default:other:r-x

I expect that the extra OPAM+DRA:rwx on the old installation was manually added 
by me, years ago. What are the "mask" entries all about?

The default:mask entry seems to be the crucial one, as if I do setfacl 
default:mask:rwx /tmp on the fresh installation, then I get the same behaviour 
as on the old installation.

However, I'm struggling to find references for either what these mask entries 
are, or how they ever appeared?

Thanks!


David 

RE: umask not working?

[David Allsopp]

Andrey Repin wrote:
> Greetings, David Allsopp!
> 
> > Is this expected behaviour:
> 
> > OPAM+DRA@OPAM ~
> > $ uname -a ; umask ; touch /tmp/foo ; ls -l /tmp/foo ; mkdir /tmp/bar
> > ; touch /tmp/bar/foo ; ls -l /tmp/bar/foo CYGWIN_NT-6.1-WOW OPAM
> > 2.10.0(0.325/5/3) 2018-02-02 15:21 i686 Cygwin
> > 0022
> > -rw-r--r-- 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/foo
> > -rw-rw-r--+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar/foo
> 
> > Why does the file /tmp/bar/foo get g+w when /tmp/foo doesn't - I'm not
> > sure what to look at on my system to diagnose what I may have
> > inadvertently tweaked. The directory itself is:
> 
> > drwxr-xr-x+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar
> 
> Let me guess, /tmp usertemp ?

No - it's default mounts, so /tmp = C:\cygwin\tmp, to which my 
(non-administrative) user has write access. 

> You have extended ACL on the object. And overall, umask is not a good
> idea in Windows.

"umask is not a good idea in Windows" - where's that come from? (In the actual 
scenario where I'm being bitten by this, it's because a git checkout is 
altering files which were 644 to be 664, so whether it's precisely umask or 
not, the *change* of permissions is the problem).


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: setup -g ???

[David Allsopp]

Lee wrote:
> On 3/18/18, David Allsopp   wrote:
> > Lee wrote:
> >> On 3/14/18, David Allsopp wrote:
> >> > [reformatted for top-posting]
> >> >
> >> > Lee wrote:
> >> >> > -- Forwarded message --
> >> >> > From: Jon Turney
> >> >> > Date: Fri, 3 Nov 2017 15:26:27 +
> >> >> > Subject: Re: Problem running the latest python2-2.7.14-1 under
> >> >> > AppVeyor
> >> >> > To: The Cygwin Mailing List <cygwin@cygwin.com>
> >> >> >
> >> >> > On 03/11/2017 14:45, Vadim Zeitlin wrote:
> >> >> > > Our build has started on AppVeyor, a continuous integration
> >> >> > > provider, started failing since a couple of days as a makefile
> >> >> > > command running a Python script started failing with exit code
> >> >> > > 127 without any more information. This is a strange situation
> >> >> > > as I can't reproduce the problem locally, but something
> >> >> > > definitely seems to be wrong with this package on the AppVeyor
> >> >> > > machine as Python just doesn't seem to be executable, e.g. the
> >> >> > > output of these commands in our batch file
> >> >> > driving the build:
> >> >> >
> >> >> > Perhaps you need to provide the -g (--upgrade-also) flag to
> >> >> > cygwin's setup.
> >> >> >
> >> >> > Due to setup terribleness, without this flag, it will install
> >> >> > the requested packages, and any missing dependencies of them,
> >> >> > but not upgrade any of the dependencies which are already
> >> >> > installed to the current (and perhaps needed) version...
> >> >> >
> >> >> > See also [1].
> >> >> >
> >> >> > [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
> >> >>
> >> >> Should we still be using the -g (--upgrade-also) flag on setup?
> >> >
> >> > I believe so (or at least hope so).
> >>
> >> So if I run setup and it says there are no pending updates it might
> >> be lying to me?
> >>
> >> I need to run setup -g and only then if there are no pending updates
> >> do I know that my cygwin installation is up to date?
> >
> > No - the original question here was about running setup with a command
> > line and the --packages argument. If you run setup interactively (with
> > no arguments), you should be told if any packages need upgrading, just
> > as before, without needing -g.
> 
> When I saw the msg about
> > Perhaps you need to provide the -g (--upgrade-also) flag
> I tried running setup & got a short list of updates.  Applied the
> updates and ran setup again.. and had no pending updates listed as
> expected.  Then I ran setup -g and a few libXXX things were listed as
> pending :(
> 
> Maybe I got lucky & the lib showed up at the mirror I was
> using just after running setup the second time??  dunno, but
> unless/until I can duplicate setup not showing any pending updates &
> setup -g saying there are pending updates, maybe we should just write it
> off as a one time occurrence.

It's possible that it was simply a mirror update - there are two ways to tell, 
if you see it again (or if you can remember what the actual packages were and 
the date you did it). One is to look in the history of this mailing list and 
see if there was an announce of a package update on or around the time you did 
the update. The other is to browse a mirror directly (e.g. 
https://mirrorservice.org/sites/sourceware.org/pub/cygwin/) and look at the 
timestamps of the files for the new package which appeared.

The third way, if it really worries you, is to use a private mirror (either via 
rsync or using Cygwin setup's "Download" option - though over the years I've 
found the latter remarkably unreliable) which allows completely repeatable 
installations.

HTH,


David 

umask not working?

[David Allsopp]

Is this expected behaviour:

OPAM+DRA@OPAM ~
$ uname -a ; umask ; touch /tmp/foo ; ls -l /tmp/foo ; mkdir /tmp/bar ;
touch /tmp/bar/foo ; ls -l /tmp/bar/foo
CYGWIN_NT-6.1-WOW OPAM 2.10.0(0.325/5/3) 2018-02-02 15:21 i686 Cygwin
0022
-rw-r--r-- 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/foo
-rw-rw-r--+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar/foo

Why does the file /tmp/bar/foo get g+w when /tmp/foo doesn't - I'm not sure
what to look at on my system to diagnose what I may have inadvertently
tweaked. The directory itself is:

drwxr-xr-x+ 1 OPAM+DRA OPAM+None 0 Mar 19 13:44 /tmp/bar

Thanks!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: setup -g ???

[David Allsopp]

Lee wrote:
> On 3/14/18, David Allsopp wrote:
> > [reformatted for top-posting]
> >
> > Lee wrote:
> >> > -- Forwarded message --
> >> > From: Jon Turney
> >> > Date: Fri, 3 Nov 2017 15:26:27 +
> >> > Subject: Re: Problem running the latest python2-2.7.14-1 under
> >> > AppVeyor
> >> > To: The Cygwin Mailing List <cygwin@cygwin.com>
> >> >
> >> > On 03/11/2017 14:45, Vadim Zeitlin wrote:
> >> > > Our build has started on AppVeyor, a continuous integration
> >> > > provider, started failing since a couple of days as a makefile
> >> > > command running a Python script started failing with exit code
> >> > > 127 without any more information. This is a strange situation as
> >> > > I can't reproduce the problem locally, but something definitely
> >> > > seems to be wrong with this package on the AppVeyor machine as
> >> > > Python just doesn't seem to be executable, e.g. the output of
> >> > > these commands in our batch file
> >> > driving the build:
> >> >
> >> > Perhaps you need to provide the -g (--upgrade-also) flag to
> >> > cygwin's setup.
> >> >
> >> > Due to setup terribleness, without this flag, it will install the
> >> > requested packages, and any missing dependencies of them, but not
> >> > upgrade any of the dependencies which are already installed to the
> >> > current (and perhaps needed) version...
> >> >
> >> > See also [1].
> >> >
> >> > [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
> >>
> >> Should we still be using the -g (--upgrade-also) flag on setup?
> >
> > I believe so (or at least hope so).
> 
> So if I run setup and it says there are no pending updates it might be
> lying to me?
> 
> I need to run setup -g and only then if there are no pending updates do
> I know that my cygwin installation is up to date?

No - the original question here was about running setup with a command line and 
the --packages argument. If you run setup interactively (with no arguments), 
you should be told if any packages need upgrading, just as before, without 
needing -g.

My assumption is that the new dependency solver means that installing a package 
with --packages should install that and upgrade *only* required dependencies, 
not the entire system. The OP's issue here with AppVeyor is one I see too, 
because the AppVeyor team are relatively slow to update the Cygwin installation 
in their images (that's their call, of course).

However, I workaround that using a script which installs required packages and 
then tests a known command from the package with --version - if I get a 
non-zero exit code, then I know that the package probably depends on a newer 
Cygwin DLL and the whole system needs upgrading. For example, quoting 
https://ci.appveyor.com/project/dra27/opam/build/1.0.210/job/21p7e382hp3q9lh7#L11:

 > Cygwin package patch will be installed
 > Cygwin package unzip will be installed
 > Cygwin package vim will be installed
 > Cygwin package flexdll will be installed
<< at this point, the script will have run setup with 
--packages=patch,unzip,vim,flexdll >>
 > Cygwin Package Information
 > Package  Version
 > curl 7.56.1-1
 > cygwin   2.9.0-3
 > diffutils3.5-2
 > flexdll  0.35-2
 > gcc-g++  6.4.0-4
 > make 4.2.1-2
 > patch2.7.4-1
 > tar  1.29-1
 > unzip6.0-17
 > vim  8.0.1567-1
 > Cygwin package upgrade required - please go and drink coffee
<< at this point, the script runs setup a second time with --upgrade-also >>
 > Cygwin Package Information
 > Package  Version
 > curl 7.56.1-1
 > cygwin   2.10.0-1
 > diffutils3.5-2
 > flexdll  0.35-2
 > gcc-g++  6.4.0-5
 > make 4.2.1-2
 > patch2.7.4-1
 > tar  1.29-1
 > unzip6.0-17
 > vim  8.0.1567-1

This build was on March 11th, and the problem package was Vim - the patch, 
unzip and flexdll packages hadn't been recompiled against Cygwin 2.10 and so 
could be installed without a full package upgrade. You can see from my message 
about coffee why I'd care if -g had become the default, because it takes ages 
to run (obviously), and I don't want my CI slowed down unnecessarily...


David 

RE: setup -g ???

[David Allsopp]

[reformatted for top-posting]

Lee wrote:
> > -- Forwarded message --
> > From: Jon Turney
> > Date: Fri, 3 Nov 2017 15:26:27 +
> > Subject: Re: Problem running the latest python2-2.7.14-1 under AppVeyor
> > To: The Cygwin Mailing List 
> > 
> > On 03/11/2017 14:45, Vadim Zeitlin wrote:
> > > Our build has started on AppVeyor, a continuous integration provider,
> > > started failing since a couple of days as a makefile command running a
> > > Python script started failing with exit code 127 without any more
> > > information. This is a strange situation as I can't reproduce the
> > > problem locally, but something definitely seems to be wrong with this
> > > package on the AppVeyor machine as Python just doesn't seem to be
> > > executable, e.g. the output of these commands in our batch file
> > driving the build:
> > 
> > Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
> > setup.
> > 
> > Due to setup terribleness, without this flag, it will install the
> > requested packages, and any missing dependencies of them, but not
> > upgrade any of the dependencies which are already installed to the
> > current (and perhaps needed) version...
> > 
> > See also [1].
> > 
> > [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>
> Should we still be using the -g (--upgrade-also) flag on setup?

I believe so (or at least hope so). I think it's the case that setup should now 
know to upgrade a dependency if you install a new package which requires a 
newer version of it, but I hope that's not become the same as setup effectively 
acting with --upgrade-also every time you run it (that would be a real 
nuisance, unless the entire Cygwin package universe is going to be recompiled 
on every new Cygwin release).


David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Regression for OCaml introduced by rebase 4.4.4

[David Allsopp]

Corinna Vinschen wrote:
> On Feb  9 13:12, David Allsopp wrote:
> > Corinna Vinschen wrote:
> > > On Feb  9 11:29, David Allsopp wrote:
> > > > [...]
> > > > When this was originally encountered for 64-bit MSVC (this was all



> > > I'm curious why this isn't done yet.
> >
> > I'm hoping that doing it is going to reveal that it simply wasn't
> > considered in 2008, rather than that it was and there was an issue
> > with it (I think it will just be that it wasn't thought of - like
> > Cygwin at that time in 2008, our x86_64 on Windows support was
> > extremely limited and not receiving much engineering focus).
> 
> We had similar problems back then.  The idea to move the executable and
> DLLs beyond the lower 32 bit area was nice as such... only GCC didn't
> support it at the time at all.  We had to add the x86-64 medium and
> large cmodel implementation to GCC to make this work first.
> Cygwin executables are compiled with --mcmodel=medium, IIRC.

This all seems to be working nicely, though the mcmodel stuff may also be part 
of why this wasn't fixed properly in 2008, which is because of data symbols. 
Everything does seem to be working correctly (there are various int symbols in 
the ocaml runtime which are always requiring absolute relocations, not relative 
ones, from the DLLs), but I'm trying to get my head around being certain that 
that should always be the case.

gcc -Q --help=target seems to show that the default for mcmodel is "32", but 
I'm struggling to find a description of precisely what that means? If I compile 
all the units with -mcmodel=small then, as expected, gcc starts generating 
RELOC_REL32 for data symbols as well. flexdll then starts creating thunks for 
data symbols, though the Cygwin runtime unsurprisingly blows up before there's 
time for flexdll's sins to become apparent!

Is it the case with -mcmodel=medium that an external data symbol could never be 
referred to via a RELOC_REL32? My reading of it was that that would only be the 
case if  the symbol itself refers to data which is large 
(-mlarge-data-threshold), but it seems to be happening for ints, which are 
clearly "small". Or have I still not properly understood x64 code models? The 
remaining question is what the difference between -mcmodel=32 and 
-mcmodel=medium really is?


David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Regression for OCaml introduced by rebase 4.4.4

[David Allsopp]

Corinna Vinschen wrote:
> On Feb  9 12:40, Corinna Vinschen wrote:
> > On Feb  9 11:29, David Allsopp wrote:
> > > > Apart from that, not only Cygwin DLLs but also the Windows system
> > > > DLLs are all loaded and relocated to the area beyond 0x1:8000,
> > > > so relocation beyond the 32 bit address space is no generic
> > > > problem in Windows.  Why isn't that possible in FlexDLL?  I don't
> understand this.
> > > > To me this looks like a bug in FlexDLL, not a requirement to let
> > > > certain DLLs slip through the cracks.
> > >
> > > There's a more full explanation of what and why for flexdll here:
> > > https://github.com/alainfrisch/flexdll/blob/master/README.md. I
> > > believe it's not unrelated to some of the black magic going on in
> > > Cygwin's autoload.cc, but without (at least at the moment), quite as
> > > much self-modifying code.
> > > [...]
> > > I guess one can argue over whether that's a bug or a limitation, but
> > > the problem we face is that we can engineer it so that our DLLs and
> > > executables are within a 2GB range (having looked again at this in
> > > even more detail, we could just as readily do this with addresses >
> > > 0x2), but we still run the risk of rebase messing up the
> DLLs.
> > >
> > > However, we'll scratch our heads some more on possible alternative
> > > solutions, since having a flag for DLLs which says "keep us within a
> > > 2GB range somewhere" sounds even more less likely to get merged than
> > > my previous suggestion.
> >
> > Two points:
> >
> > - You are aware that the main executable of 64 bit Cygwin processes
> are
> >   loaded to 0x1:0040, right?  The 2 GB offset problem is already
> >   imminent.
> 
> ...and you must not use the 0x0:8000 - 0x1: area because
> that's reserved for thread stacks.
> 
> To clarify, the full layout requirements:
> 
> - 0x0: - 0x0:8000 Windows
> - 0x0:8000 - 0x1: Cygwin pthreads (including main thread)
> - 0x1: - 0x1:8000 Executable
> - 0x1:8000 - 0x2: Cygwin DLL
> - 0x2: - 0x4: Rebased DLLs
> - 0x4: - 0x6: Non-rebased DLLs (hashed default
> addresses
>   generated by binutils ld with
>   -auto-image-based (default on Cygwin))
> - 0x6:Start Address Heap, growing upwards
> - 0x8: - 0x700:   Mmaps, allocated downwards
> - 0x700: and beyond   Windows

Thanks for this (and your time on this question generally!). I reckon that the 
jump tables will sort this and we'll be able to stop doing horrible things with 
--image-base completely, which should mean that flexlink (and therefore OCaml 
too) will start properly respecting the Cygwin address space layout! It's a 
shame that the layout means that the trampolines would always be needed, but I 
very much doubt their overhead will be significant in any program.


David

RE: Regression for OCaml introduced by rebase 4.4.4

[David Allsopp]

Corinna Vinschen wrote:
> On Feb  9 11:29, David Allsopp wrote:
> > Corinna Vinschen wrote:
> > > On Feb  8 11:47, David Allsopp wrote:
> > > > TL;DR flexlink-compiled DLLs (i.e. ocaml libraries) are broken by
> > > > the
> > > > 0x2 base address requirement added in rebase 4.4.4.
> > > > Possible fixes for this at the bottom.
> > > > [...]
> > > >   $ ocaml
> > > >   OCaml version 4.04.2
> > > >
> > > >   # #load "unix.cma";;
> > > >   Cannot load required shared library dllunix.
> > > >   Reason: /usr/lib/ocaml/stublibs/dllunix.so: flexdll error:
> > > > cannot relocate RELOC_REL32, target is too far: 0xfffc013d8b5f
> > > 0x13d8b5f.
> > > >
> > > > This is a known problem and fundamental limitation of flexdll
> > > > (there is no
> > > > RELOC_REL64 in COFF).
> > >
> > > Apart from that, not only Cygwin DLLs but also the Windows system
> > > DLLs are all loaded and relocated to the area beyond 0x1:8000,
> > > so relocation beyond the 32 bit address space is no generic problem
> > > in Windows.  Why isn't that possible in FlexDLL?  I don't understand
> this.
> > > To me this looks like a bug in FlexDLL, not a requirement to let
> > > certain DLLs slip through the cracks.
> >
> > There's a more full explanation of what and why for flexdll here:
> > https://github.com/alainfrisch/flexdll/blob/master/README.md. I
> > believe it's not unrelated to some of the black magic going on in
> > Cygwin's autoload.cc, but without (at least at the moment), quite as
> > much self-modifying code.
> >
> > FlexDLL is "solving" the problem of allowing a dynamically loaded
> > library to refer to symbols in the main application (or in previously
> > dynamically loaded libraries, without loading them a second time, as
> > the Windows loader I believe does). FlexDLL does this by deferring
> > COFF relocations to runtime and it achieves that by sitting in front
> > of both the linker when the DLL is constructed and also an
> > application's main (or dllmain). For normal linking, since PE limits
> > code size to 2GB, there is no need for a RELOC_REL64 relocation type.
> > However, because we're actually resolving the symbols dynamically, on
> > 64-bit the DLL may have been loaded too far from the executable (or
> > other DLL) image it's resolving to (for actual Windows resolution to
> > DLL symbols, you'd be using the stub code generated either by the
> > linker or by __declspec(dllimport), which would similarly be
> > guaranteed to be within the range of RELOC_REL32 because the stub
> > itself is static).
> >
> > When this was originally encountered for 64-bit MSVC (this was all
> > added before Cygwin64 existed), the solution at the time was to keep
> > the preferred base addresses low, but in reality what's really
> > required is that everything is within a 2GB window somewhere in the
> > address space.
> >
> > I guess one can argue over whether that's a bug or a limitation, but
> > the problem we face is that we can engineer it so that our DLLs and
> > executables are within a 2GB range (having looked again at this in
> > even more detail, we could just as readily do this with addresses >
> > 0x2), but we still run the risk of rebase messing up the DLLs.
> >
> > However, we'll scratch our heads some more on possible alternative
> > solutions, since having a flag for DLLs which says "keep us within a
> > 2GB range somewhere" sounds even more less likely to get merged than
> > my previous suggestion.
> 
> Two points:
> 
> - You are aware that the main executable of 64 bit Cygwin processes are
>   loaded to 0x1:0040, right?  The 2 GB offset problem is already
>   imminent.

Our executables are also compiled via flexdll's flexlink which sets 
--image-base in its call to the linker. I don't think the Cygwin DLL does 
anything which alters that, right? Another "fix" I tried while investigating 
was to change the --image-base we specified to be within 2GB of where rebase 
has put the DLLs, which also worked.

> - What about adding an addition jump table?  The relocation would only
>   have to point to the jump table in the vicinity of the DLL in
>   question, the jump table points to the actual 64 bit address.

That was what our head-scratching has arrived at too, which I'm in the process 
of doing.

> I'm curious why this isn't done yet.

I'm hoping that doing it is going to reveal that it simply wasn't considered in 
2008, rather than that it was and there was an issue with it (I think it will 
just be that it wasn't thought of - like Cygwin at that time in 2008, our 
x86_64 on Windows support was extremely limited and not receiving much 
engineering focus).


David

RE: Regression for OCaml introduced by rebase 4.4.4

[David Allsopp]

Corinna Vinschen wrote:
> On Feb  8 11:47, David Allsopp wrote:
> > TL;DR flexlink-compiled DLLs (i.e. ocaml libraries) are broken by the
> > 0x2 base address requirement added in rebase 4.4.4. Possible
> > fixes for this at the bottom.
> > [...]
> >   $ ocaml
> >   OCaml version 4.04.2
> >
> >   # #load "unix.cma";;
> >   Cannot load required shared library dllunix.
> >   Reason: /usr/lib/ocaml/stublibs/dllunix.so: flexdll error: cannot
> > relocate RELOC_REL32, target is too far: 0xfffc013d8b5f
> 0x13d8b5f.
> >
> > This is a known problem and fundamental limitation of flexdll (there
> > is no
> > RELOC_REL64 in COFF).
> 
> Apart from that, not only Cygwin DLLs but also the Windows system DLLs
> are all loaded and relocated to the area beyond 0x1:8000, so
> relocation beyond the 32 bit address space is no generic problem in
> Windows.  Why isn't that possible in FlexDLL?  I don't understand this.
> To me this looks like a bug in FlexDLL, not a requirement to let certain
> DLLs slip through the cracks.

There's a more full explanation of what and why for flexdll here: 
https://github.com/alainfrisch/flexdll/blob/master/README.md. I believe it's 
not unrelated to some of the black magic going on in Cygwin's autoload.cc, but 
without (at least at the moment), quite as much self-modifying code.

FlexDLL is "solving" the problem of allowing a dynamically loaded library to 
refer to symbols in the main application (or in previously dynamically loaded 
libraries, without loading them a second time, as the Windows loader I believe 
does). FlexDLL does this by deferring COFF relocations to runtime and it 
achieves that by sitting in front of both the linker when the DLL is 
constructed and also an application's main (or dllmain). For normal linking, 
since PE limits code size to 2GB, there is no need for a RELOC_REL64 relocation 
type. However, because we're actually resolving the symbols dynamically, on 
64-bit the DLL may have been loaded too far from the executable (or other DLL) 
image it's resolving to (for actual Windows resolution to DLL symbols, you'd be 
using the stub code generated either by the linker or by __declspec(dllimport), 
which would similarly be guaranteed to be within the range of RELOC_REL32 
because the stub itself is static).

When this was originally encountered for 64-bit MSVC (this was all added before 
Cygwin64 existed), the solution at the time was to keep the preferred base 
addresses low, but in reality what's really required is that everything is 
within a 2GB window somewhere in the address space.

I guess one can argue over whether that's a bug or a limitation, but the 
problem we face is that we can engineer it so that our DLLs and executables are 
within a 2GB range (having looked again at this in even more detail, we could 
just as readily do this with addresses > 0x2), but we still run the 
risk of rebase messing up the DLLs.

However, we'll scratch our heads some more on possible alternative solutions, 
since having a flag for DLLs which says "keep us within a 2GB range somewhere" 
sounds even more less likely to get merged than my previous suggestion.


David

RE: Regression for OCaml introduced by rebase 4.4.4

[David Allsopp]

Corinna Vinschen wrote:
> On Feb  8 11:47, David Allsopp wrote:
> > TL;DR flexlink-compiled DLLs (i.e. ocaml libraries) are broken by the
> > 0x2 base address requirement added in rebase 4.4.4. Possible
> > fixes for this at the bottom.
> >
> > Commit bfd383 in the rebase sources introduces a new minimum base
> > address requirement of 0x2 for Cygwin64. This is a problem for
> > the correct operation of the flexdll package and affects ocaml. On a
> > fresh up-to-date
> > Cygwin64 installation, install the ocaml package:
> >
> >   $ rebase -i /usr/lib/ocaml/stublibs/*
> >   /usr/lib/ocaml/stublibs/dllbigarray.sobase 0x0001 size
> > 0x00015000 *
> >   /usr/lib/ocaml/stublibs/dllcamlstr.so base 0x0001 size
> > 0x00014000 *
> >   /usr/lib/ocaml/stublibs/dllgraphics.sobase 0x0001 size
> > 0x00038000 *
> > [...]
> > Here you can see a problem we already know about with flexlink - all
> > libraries have a base address of 0x1
> > (https://github.com/alainfrisch/flexdll/issues/50).
> >
> > However, this allows you to load libraries dynamically:
> >
> >   $ ocaml
> >   OCaml version 4.04.2
> >
> >   # #load "unix.cma";;
> >   # #directory "+threads";;
> >   # #load "threads.cma";;
> >
> > but not fork (we know about this problem):
> >
> >   # Unix.fork ();;
> > 0 [main] ocamlrun 5688 child_info_fork::abort: address space
> > needed by 'dllunix.so' (0x40) is already occupied
> >   Exception: Unix.Unix_error (Unix.EAGAIN, "fork", "").
> >
> > Now do a rebaseall.
> >
> >   $ rebase -i /usr/lib/ocaml/stublibs/*
> >   /usr/lib/ocaml/stublibs/dllvmthreads.so   base 0x0003fec2 size
> > 0x0001f000
> >   /usr/lib/ocaml/stublibs/dllunix.sobase 0x0003fec4 size
> > 0x0004c000
> > [...]
> >
> > So forking should now be fine. However:
> >
> >   $ ocaml
> >   OCaml version 4.04.2
> >
> >   # #load "unix.cma";;
> >   Cannot load required shared library dllunix.
> >   Reason: /usr/lib/ocaml/stublibs/dllunix.so: flexdll error: cannot
> > relocate RELOC_REL32, target is too far: 0xfffc013d8b5f
> 0x13d8b5f.
> 
> The problem is this:  Given that the lib is in a safe space anyway, why
> do you still try to relocate it?  That's exactly what you don't have to
> do anymore and you shouldn't do this.  The DLL is loaded where it
> belongs, end of story.  What should another relocation gain?  So, just
> wwitch it off for 64 bit Cygwin, no?

See other message, but we're not relocating the DLL, we're performing COFF 
relocations on specific symbols which were deferred from link time.


David

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Regression for OCaml introduced by rebase 4.4.4

[David Allsopp]

TL;DR flexlink-compiled DLLs (i.e. ocaml libraries) are broken by the
0x2 base address requirement added in rebase 4.4.4. Possible fixes
for this at the bottom.

Commit bfd383 in the rebase sources introduces a new minimum base address
requirement of 0x2 for Cygwin64. This is a problem for the correct
operation of the flexdll package and affects ocaml. On a fresh up-to-date
Cygwin64 installation, install the ocaml package:

  $ rebase -i /usr/lib/ocaml/stublibs/*
  /usr/lib/ocaml/stublibs/dllbigarray.sobase 0x0001 size
0x00015000 *
  /usr/lib/ocaml/stublibs/dllcamlstr.so base 0x0001 size
0x00014000 *
  /usr/lib/ocaml/stublibs/dllgraphics.sobase 0x0001 size
0x00038000 *
  /usr/lib/ocaml/stublibs/dllnums.sobase 0x0001 size
0x00011000 *
  /usr/lib/ocaml/stublibs/dllthreads.so base 0x0001 size
0x00025000 *
  /usr/lib/ocaml/stublibs/dllunix.sobase 0x0001 size
0x0004c000 *
  /usr/lib/ocaml/stublibs/dllvmthreads.so   base 0x0001 size
0x0001f000 *

Here you can see a problem we already know about with flexlink - all
libraries have a base address of 0x1
(https://github.com/alainfrisch/flexdll/issues/50).

However, this allows you to load libraries dynamically:

  $ ocaml
  OCaml version 4.04.2

  # #load "unix.cma";;
  # #directory "+threads";;
  # #load "threads.cma";;

but not fork (we know about this problem):

  # Unix.fork ();;
0 [main] ocamlrun 5688 child_info_fork::abort: address space needed
by 'dllunix.so' (0x40) is already occupied
  Exception: Unix.Unix_error (Unix.EAGAIN, "fork", "").

Now do a rebaseall.

  $ rebase -i /usr/lib/ocaml/stublibs/*
  /usr/lib/ocaml/stublibs/dllvmthreads.so   base 0x0003fec2 size
0x0001f000
  /usr/lib/ocaml/stublibs/dllunix.sobase 0x0003fec4 size
0x0004c000
  /usr/lib/ocaml/stublibs/dllthreads.so base 0x0003fec9 size
0x00025000
  /usr/lib/ocaml/stublibs/dllnums.sobase 0x0003fecc size
0x00011000
  /usr/lib/ocaml/stublibs/dllgraphics.sobase 0x0003fece size
0x00038000
  /usr/lib/ocaml/stublibs/dllcamlstr.so base 0x0003fed2 size
0x00014000
  /usr/lib/ocaml/stublibs/dllbigarray.sobase 0x0003fed4 size
0x00015000

So forking should now be fine. However:

  $ ocaml
  OCaml version 4.04.2

  # #load "unix.cma";;
  Cannot load required shared library dllunix.
  Reason: /usr/lib/ocaml/stublibs/dllunix.so: flexdll error: cannot relocate
RELOC_REL32, target is too far: 0xfffc013d8b5f  0x13d8b5f. 

This is a known problem and fundamental limitation of flexdll (there is no
RELOC_REL64 in COFF). On our CI, we have been using a workaround for the
fork problem at
https://github.com/ocaml/ocaml/blob/trunk/tools/ci-build#L230-L231 but that
no longer works with rebase 4.4.4 because of the new minimum base address.

It was already the case that rebaseall was breaking OCaml DLLs, but now with
4.4.4 they cannot even be fixed by hand, so it's clearly a good moment to
put some effort into this (read as: I'm offering both coding and testing
time!).

For this to work at all, there needs to be some address space below
0x8000 which DLLs may be permitted to opt-in to using and which rebase
needs to respect. Assuming that's OK, I think something along the following
lines is needed:

 1. We (ab)use either a DLL characteristics flag or a section header flag to
indicate that the DLL needs to be loaded below 0x800
 2. The rebase utility warning for base addresses should take that flag into
account (to the point of requiring < 0x8000 if this new bit is set in
the image)
[2a. While we're changing validation for the image base, it'd be sensible to
add a check that the supplied address is 64K aligned :$]
 3. The flexlink utility should stop using 0x1 all the time. Probably
the best way to achieve this is if the rebase utility has a flag which
*sets* the new bit so that flexlink calls rebase after compilation to assign
an improved base address to the DLL. On x86, we don't force a given base
address at all - I assume that Cygwin's binutils stuff is already
rebase-aware and produces sensible base addresses for newly-compiled DLLs,
as I don't recall having ever seen the fork conflict problem on x86 builds
of OCaml?

Comments on the proposed need for some DLLs to occupy memory below
0x8000 and on the fixes much appreciated, thanks!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Searching full, portable Cygwin package for windows and NOT just the installer

[David Allsopp]

Ben via cygwin wrote:
> When I go to web page
> 
> http://cygwin.com/
> 
> then I can download CygWin from there (currently the file "setup-
> x86_64.exe").
> 
> Unfortunately this file is just an installer which retrieves in turn
> several other files from Internet and remote servers.
> 
> Since I have no overview what is downloaded from which server I distrust
> such installers in general.

I assume that each month you review the Microsoft Security Advisories, visit 
and digest the page for each individual update and then download and apply the 
.msu patches by hand, rather than trusting the highly fishy "Check for updates" 
feature of Windows Update?

You do have an overview: you get to pick the mirror, and the URL it will 
download files from is declared there. For example, my closest mirror is 
http://mirrorservice.org/sites/sourceware.org/pub/cygwin/ which you can browse 
online.

> I prefer full packages which contains everything needed and can be
> inspected in advance (e.g. by virus scanners) before actual installation.

The Cygwin installer downloads and saves the files - even a dumb 
filesystem-only virus scanner gets a chance to inspect the files. 

> 99,9% of all software is offered in such a way.

That number is plucked out of your mind, and simply isn't true, even on Windows 
(there's the Microsoft Store and even third party packagers such as 
https://chocolatey.org/). Microsoft's own products on Windows are now 
distributed this way too (e.g. the Visual Studio Installer and the Office 365 
installer).

> Why use Cygwin such a fishy distribution way?

Well, it's not fishy.

Cygwin is not an application, as another answer to your previous question 
pointed out, Cygwin is a library providing a Posix layer for applications on 
Windows. The setup program is an application for installing that library and 
then adding packages of programs which use it from a curated collection, but 
you can point the installer to your own collection if you wish.

> Is there really no full package to download?

The setup program includes a "Download" option which allows you to save the 
entire package repository locally, which you can use in future. Some of the 
mirrors (including the one I referenced above) also support rsync, which allows 
you to download the package repository and the installer from the website 
without running any Cygwin software initially.

Packages are updated quite frequently, and the entire repository is 10s of 
gigabytes - it would be impractical to offer a single file for download being 
updated so frequently.


David 


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: How to start Cygwin from outside Cygwin and pass a command to execute?

[David Allsopp]

Ben via cygwin wrote:
> Assume my CgyWin (on a windows 7) is currently NOT started.
> 
> Assume I want to call from Windows my CgyWin and pass a command to 
> execute.
> 
> Afterwards CygWin should automatically be closed again.
> 
> How can I achieve this?

C:\cygwin\bin\bash.exe -c "command"

You will find that successfully navigating the Command Prompt, Cygwin's and 
"bash -c"'s escaping rules to be entertaining for advanced commands.

You can also achieve similar with mintty.exe -e (which will launch the terminal 
emulator, instead of using an existing console window, or opening a new one). 
Similar fun with escaping unusual commands.

See, for example, 
https://github.com/ocaml/opam/blob/43e4c778/appveyor_build.cmd#L93


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Ejecting a USB drive using Cygwin (sync)?

[David Allsopp]

Brian Inglis wrote:
> On 2018-01-23 16:36, Steven Penny wrote:
> > On Tue, 23 Jan 2018 14:13:24, Brian Inglis wrote:
> >> I found the following utility works well without elevation - Windows
> >> code from
> >> http://www.leapsecond.com/tools/eject.{c,exe}:
> >>
> >> // eject -- Allow safe removal of USB thumb drive.
> >> // - Command line tool to flush/dismount/eject USB drive.
> >> // - Simpler than mouse clicking through taskbar "Safely Remove
> >> Hardware" icon.
> >> // - Usable in batch file scripts.
> >> // 02-Nov-2012 Tom Van Baak (tvb) www.LeapSecond.com/tools ...
> >> // Per MSDN, follow procedure for safe removal of USB drive.
> >> int drive_eject (char *drive)
> >> {
> >>     HANDLE hDev;
> >>     // Convert vintage DOS drive letter to weird Windows object
> >> pathname.
> >>     char path[10];
> >>     sprintf(path, ".\\%s", drive);
> >>     // Open (with write, but no lock) to flush pending writes.
> >>     DEV_OPEN(path, GENERIC_READ | GENERIC_WRITE);
> >>     FlushFileBuffers(hDev);
> >>     CloseHandle(hDev);
> >>     // Open (with read, and lock) to dismount and eject.
> >>     DEV_OPEN(path, GENERIC_READ);
> >>     DEV_IOCTL(FSCTL_LOCK_VOLUME);
> >>     DEV_IOCTL(FSCTL_DISMOUNT_VOLUME);
> >>     DEV_IOCTL(IOCTL_STORAGE_EJECT_MEDIA);
> >>     DEV_IOCTL(FSCTL_UNLOCK_VOLUME);
> >>     CloseHandle(hDev);
> >>     return 0;
> >> }
> >
> > No, this is not the code from that site. The code is similar, but I
> > see you have at least removed some empty lines. If you are going to do
> > so, you need to clearly present said changes to the community -
> > something like "adapted from http://...; or "modified from http://...;,
> > etc.
> >
> > Software copyright - even open source license is serious, and you
> > should give it due respect.
> 
> I was providing information, that's why I said /from/, gave the link to
> the original, and included the copyright, which does not include a link
> to the MSDN article it mentions:

I expect it's referring to this article:
https://support.microsoft.com/en-us/help/165721.
There's another quite good article on it at
https://www.codeproject.com/Articles/259577/How-to-flush-a-storage-volumes-t
he-file-cache-lock

> checking any licences are left as an
> exercise for any reader who wants to make use of the program or code
> other than for their own private use.

What licence is the ellipsis you added to the copyright line released under?
;o)


--dra


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Missing dependencies for opam package

[David Allsopp]

The dependencies for the opam package in the Cygwin repository are
incomplete. Since setup.ini doesn't support "optional" or "advised"
dependencies (at least I don't think it does), there are choices for how
many dependencies are added, which I leave to the maintainer.

As a minimum, please could libX11-devel and flexdll be added as a dependency
for the opam package, as not having those packages installed creates subtle
issues for new users (and it turns out there are a few!).

The first command any opam user must run is `opam init` and this will fail
if any of diff, tar, patch, unzip and *either* wget *or* curl cannot be
found. I'd suggest therefore that diffutils, tar, patch, unzip and curl be
added as dependencies.

`opam init` will also complain, but not fail, if any of cc, make and m4 are
missing, so gcc-core, make and m4 would be good additional dependencies.

opam will either use a preinstalled ocaml or will compile a new one from
scratch. The ocaml package could be added as a dependency (this is certainly
what Ubuntu, and I believe several other Linux distributions do), although
it is not compulsory and indeed with our forthcoming opam 2.0 release,
unnecessary. If there is not a preinstalled ocaml, then `opam init` will
seek to compile one, which requires the gcc-core and make packages. OCaml's
configure script disables features if libX11-devel and flexdll are not
installed and builds a subtly different ocaml, which has caused surprise for
new users as the error message from subsequent build systems is slightly
obscure. Unlike other dependencies which cause errors if not present, these
do not and if subsequently installed then require the user to recompile
ocaml, which is time-consuming. Therefore, although they're not strictly
dependencies, it would very helpful if libX11-devel and flexdll were both
automatically installed with opam.

Finally, normal use of opam also interacts with source code repositories. It
would be advisable, though not compulsory, to depend on rsync and also the
git and mercurial packages.

So, in summary, the following dependencies would be good to have added to
the opam package, in descending order of importance:

  flexdll
  libX11-devel
  diffutils
  tar
  patch
  unzip
  curl (or wget, if preferred)
  gcc-core
  make
  m4
  rsync
  git
  mercurial
  ocaml

Thanks!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Missing dependencies for ocaml package

[David Allsopp]

The dependencies for the ocaml package in the Cygwin repository are
incomplete. ocamlopt depends on the gcc-core and flexdll packages.

More precisely, ocamlopt itself depends on binutils and flexlink. flexlink
requires a C compiler, however it's also possible to use the flexdll package
without specifically needing gcc-core, so I'd suggest that it is the ocaml
package, rather than flexdll package, which requires the dependency.

Please could the maintainer add those two dependencies to setup.ini? The
alternative would be to split the ocaml package into two - for example,
ocaml and ocaml-native with only the latter depending on gcc-core and
flexdll and including the ocamlopt binary and native code libraries.

Thanks!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: [PATCH] Preserve order of dlopen'd modules in dll_list::topsort

[David Allsopp]

Hi Corinna,

Thanks for the feedback!

Corinna Vinschen wrote:
> Hi David,
> 
> thanks for the new patch.
> 
> On Feb 27 17:13, David Allsopp wrote:
> > Corinna Vinschen wrote:
> > > On Feb 25 16:27, David Allsopp wrote:
> > > > This patch (below - I hope I have managed to format this email
> > > > correctly) alters the behaviour of dll_list::topsort to preserve
> > > > the order of dlopen'd units.
> > > > [...]
> > > > This patch is licensed under 2-clause BSD as per
> > > > winsup/CONTRIBUTORS, Copyright (c) 2017, MetaStack Solutions Ltd.
> 
> Do you really want to make it (c) MetaStack?

Oh, I was assuming that there would just be an implied mapping! Simple is best 
(MetaStack is just me, anyway!), so the previous patch and this fixup may be 
merged as my personal copyright, yes.



> > > - Last but not least.  You add code to topsort so the loaded DLLs
> > >   are handled first.  The subsequent code is untouched.  However,
> > >   shouldn't the next loop then restrict calling populate_deps to the
> > >   linked DLLs only, at least for performance?
> >
> > Oops :$ That's an artefact of the "story" of the patch's development.
> > As it happens, the first dlopen'd DLL would have been initialised in
> > the second loop, not the first, but the presence of two loops like
> > that was indeed mostly inefficient. I've kept the original one as a
> > "fast path" for the case of no dlopen'd DLLs, though I don't know if
> > that's a worthwhile optimisation.
> 
> Well, interesting point.  Basically your new code is a drop-in
> replacement, except for the fact that it always calls an extra
> cmalloc/cfree.  However, this is only required if loaded_dlls > 0 so I
> think we may get away with removing the old loop with a simple tweak to
> your new one:
> 
>   dll** dlopen_deps = NULL;
>   if (loaded_dlls > 0)
> dlopen_deps = (dll**) cmalloc (HEAP_2_DLL, maxdeps*sizeof (dll*));
>   while ((d = d->next))
> {
>   [...]
> }
>   if (dlopen_deps)
> cfree (dlopen_deps);
> 
> Do you want to tweak your patch accordingly?

That's much neater - attached is a fixup (which obviously looks a lot clearer 
with git diff --ignore-all-space)

All best,


David


0001-Fixup-8607cf.patch
Description: Binary data

RE: [PATCH] Preserve order of dlopen'd modules in dll_list::topsort

[David Allsopp]

Hi Corinna,

Corinna Vinschen wrote:
> Hi David,
> 
> On Feb 25 16:27, David Allsopp wrote:
> > This patch (below - I hope I have managed to format this email
> > correctly) alters the behaviour of dll_list::topsort to preserve the
> > order of dlopen'd units.
> >
> > The load order of unrelated DLLs is reversed every time fork is
> > called, since dll_list::topsort finds the tail of the list and then
> > unwinds to reinsert items. My change takes advantage of what should be
> > undefined behaviour in dll_list::populate_deps (ndeps non-zero and
> > ndeps and deps not
> > initialised) to allow the deps field to be initialised prior to the
> > call and appended to, rather than overwritten.
> >
> > All DLLs which have been dlopen'd have their deps list initialised
> > with the list of all previously dlopen'd units. These extra
> > dependencies mean that the unwind preserves the order of dlopen'd units.
> >
> > The motivation for this is the FlexDLL linker used in OCaml. The
> > FlexDLL linker allows a dlopen'd unit to refer to symbols in
> > previously dlopen'd units and it resolves these symbols in DllMain
> > before anything else has initialised (including the Cygwin DLL). This
> > means that dependencies may exist between dlopen'd units (which the
> > OCaml runtime system understands) but which Windows is unaware of.
> > During fork, the process-level table which FlexDLL uses to get the
> > symbol table of each DLL is copied over but because the load order of
> > dlopen'd DLLs is reversed, it is possible for FlexDLL to attempt to
> > access memory in the DLL before it has been loaded and hence it fails
> > with an access violation. Because the list is reversed on each call to
> > fork, it means that a subsequent call to fork puts the DLLs back into
> > the correct order, hence "even" invocations of fork work!
> >
> > An interesting side-effect is that this only occurs if the DLLs load
> > at their preferred base address - if they have to be rebased, then
> > FlexDLL works because at the time that the dependent unit is loaded
> > out of order, there is still in memory the "dummy"
> > DONT_RESOLVE_DLL_REFERENCES version of the dependency which, as it
> > happens, will contain the correct symbol table in the data section.
> > For my tests, this initially appeared to be an x86-only problem, but
> > that was only because the two DLLs on x64 should have been rebased.
> >
> > I'm very happy to include the complete detail for this and, for the
> > extremely keen, the relevant Git branch in OCaml which demonstrates
> > this problem. Given the way in which FlexDLL operates, I would contend
> > that this is a sensible change of behaviour for the Cygwin DLL, though
> > not a bug fix.
> > I'd be extremely happy to see this patch integrated, as the workaround
> > necessary in FlexDLL to support Cygwin's fork is horrible (and
> > non-transparent to the library user).
> >
> > This patch is licensed under 2-clause BSD as per winsup/CONTRIBUTORS,
> > Copyright (c) 2017, MetaStack Solutions Ltd.
> 
> First of all, I think this makes perfect sense.  I just have a few
> questions in terms of the patch itself.
> 
> - Your browser inserts undesired line breaks, so the patch is broken.
>   Can you please resend the `git format-patch' output as attachment?

Darn - sorry about that (it's the first time I'd tried to send a format-patch, 
rather than as a PR). Patch attached.

> - While you're at it, please reformat your patch so the line length
>   is not longer than 80 chars.

Done - sorry, I'd inferred a longer length from a few other longer lines!

> - Last but not least.  You add code to topsort so the loaded DLLs
>   are handled first.  The subsequent code is untouched.  However,
>   shouldn't the next loop then restrict calling populate_deps to the
>   linked DLLs only, at least for performance?

Oops :$ That's an artefact of the "story" of the patch's development. As it 
happens, the first dlopen'd DLL would have been initialised in the second loop, 
not the first, but the presence of two loops like that was indeed mostly 
inefficient. I've kept the original one as a "fast path" for the case of no 
dlopen'd DLLs, though I don't know if that's a worthwhile optimisation. 

All best,


David


0001-Preserve-order-of-dlopen-d-modules-in-dll_list-topso.patch
Description: Binary data

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

Aaron Digulla wrote:
> David Allsopp wrote:
> > Aaron Digulla wrote:
> > >
> > > Am Samstag, 07. Mai 2016 09:45 CEST, "David Allsopp"
> > > schrieb:
> > >
> > >
> > > > > Then all you need is a rudimentary quoting.
> > > >
> > > > Yes, but the question still remains what that rudimentary quoting
> > > > is -
> > > i.e.
> > > > I can see how to quote spaces which appear in elements of argv,
> > > > but I cannot see how to quote double quotes!
> > >
> > > This should help:
> > > https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/0
> > > 4/23/e veryone-quotes-command-line-arguments-the-wrong-way/
> >
> > This provides documentation for how Microsoft implementations do it, not
> how Cygwin does it. The Cygwin DLL is responsible for determining how a
> Cygwin process gets argc and argv from GetCommandLineW.
> 
> That's correct but I read your question as "how do I start executables
> linked against Cygwin from another Windows process"

That's the correct reading of my question!

> To do that, you need to convert the argument list/array into the stupid
> Windows format because that's what the Cygwin process will expect.

This is not correct - both by reading the code and by testing. If you put this 
into the program caller.c I posted previously in thread which is intended to 
start with argv[1] being the literal string a\\b (i.e. 4 characters)

commandLine = "callee \"ab\"";

and then if callee.c is compiled with i686-w64-mingw32-gcc (Microsoft-world - 
escaping rules according to MSDN):

$ ./caller
argc = 2
argv[0] = callee
argv[1] = a\\b

with argv[1] correctly given in Microsoft's escaping. But if you compile 
callee.c with gcc (Cygwin-world), you get:

$ ./caller
argc = 2
argv[0] = callee
argv[1] = a\b

With the Cygwin DLL applying its interpretation of the escaping (treating \\ 
within a quoted parameter as an escaped single backslash), and quite clearly 
not MSDN's (treating \\ as two backslash characters because it's no followed by 
a double-quote). As an aside, if you have CYGWIN=noglob, you will actually get 
the same output as the native Windows case with two backslashes (more evidence, 
if you still need it, about how my question is everything to do with the Cygwin 
DLL, and nothing to do with MSDN and Microsoft's escaping rules).

There's also the small matter of Cygwin's @file trick for reading command line 
arguments from files (i.e. an extra escaping rule not indicated in MSDN because 
it's not part of Windows) - this time have commandLine = "@test", run echo 
foo>test and this time with a Microsoft-compiled callee, you'll get argv[1] = 
@test and with a Cygwin-compiled one, you'll get argv[1] = foo

> > > My line of thought is that Cygwin can't get anything which Windows
> > > can't send it. So the first step to solve this mess is to make sure
> > > the arguments which you send to CreateProcess() are correct.
> > >
> > > The next step would be to write a small C utility which dumps it's
> 
> > > arguments, so you can properly debug all kinds of characters.
> >
> > See later email, but IMHO the conversion is something Cygwin should have
> precisely documented, not determined by brittle experimentation.
> 
> Ah... no. You're mixing two or three things. Let me enumerate:
> 
> 1. You have to give your OS (Windows or Unix) the information which
> process you want to start and which arguments to pass. Unix has two ways
> (string array and single string), Windows has only single string.

Which system call in Unix allows you to start a process by giving a single 
string instead of an array of arguments (or a series of parameters)?

> 2. The OS will put the information into a structure of some kind and pass
> that to the new process.
> 3. If you have a shell (CMD.exe, bash, etc), they will take the structure
> and parse it according to some rules. They will then convert the result
> again into an OS call.
> 4. The C runtime of your executable will know where to get the OS
> structure and how to turn the structure into char ** argv.
> Where is Cygwin in all this? It's part of step #3. Cygwin emulates exec()
> and similar Unix OS functions which an emulated shell like BASH will use.
> Which means Cygwin code in the DLL is irrelevant if you don't have a Unix
> shell somewhere in your process chain.

No, Cygwin is loosely part of step #3 and definitely the whole of #4. If you 
invoke a Cygwin process from a native process, the Cygwin DLL does the work of 
#4, not the C runtime. Step #3 is entirely irrelevant to my problem because 
there is no shell involved.

> If you just want to execute a Cygwin process (=

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

Marco Atzeri wrote:
> On 09/05/2016 17:49, David Allsopp wrote:
> > Marco Atzeri wrote:
> >>
> >> Ultimate overview of MS escape howto :
> >>
> >> https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04
> >> /23/e veryone-quotes-command-line-arguments-the-wrong-way/
> >
> > This is a great article (which I'd not come across before), but this
> > relates to Microsoft's mechanisms for quoting which aren't applicable
> > here - it's definitely the Cygwin DLL which does it!
> >
> >
> > David
> >
> 
> Hi David,

Hi!

> I am puzzled, I had the impression you asked:
> "I am trying to work out the precise details for character escaping when
> starting a Cygwin process from a native (i.e. non-Cygwin) Windows
> process."
> 
> So the exec should be on "Windows process" side, why inside the Cygwin DLL
> ?

In Windows, there is no "exec" - the equivalent functions in Microsoft's C
runtime are themselves emulations boiling down to a CreateProcess call
(unlike on Unix, where the exec family are actual system calls). That
CreateProcess call takes exactly a single string for the entire command line
which the Cygwin DLL then has to convert to argc/argv in order to call the
main function in the Cygwin program started (Microsoft's C runtime has to do
the same thing which is part of what that blog explains). Any sane native
Windows program will use (or be compliant with) CommandLineToArgvW, but no
program has to do it that on Windows (indeed, Cygwin takes "advantage" of
this to do its own thing).

I have total control of my native call to CreateProcess, so it's definitely
about working out exactly what Cygwin does with the command line. AFAICT,
when a Cygwin program execs another Cygwin program, they actually
communicate argc and argv via memory, rather than through the process
invocation (although I'm really not sure that I've interpreted that
correctly) which is why this doesn't come up in pure Cygwin-land.

I think I'm there, it's just not as clean (or documented) as might be nice!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

Marco Atzeri wrote:
> 
> Ultimate overview of MS escape howto :
> 
> https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/e
> veryone-quotes-command-line-arguments-the-wrong-way/

This is a great article (which I'd not come across before), but this relates
to Microsoft's mechanisms for quoting which aren't applicable here - it's
definitely the Cygwin DLL which does it!


David 



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

Hi!

Peter Rosin wrote:
> I think cygwin emulates posix shell style command line parsing when
> invoked from a Win32 process (like you do). So, try single quotes:
> 
> commandLine = "callee.exe \"@\"te\"\n\"st fo@o bar\" \"baz baz '*'
> '\"\\'\"'";
> 
> I get this (w/o noglob):
> 
> argc = 7
> argv[0] = callee
> argv[1] = @te
> st
> argv[2] = fo@o
> argv[3] = bar baz
> argv[4] = baz
> argv[5] = *
> argv[6] = "'"

Yes, that seems to be approximately the way I arrived at too - my concern
with all the extra quoting is then hitting the Windows limit on command line
length (I'd like to avoid needing @response files as much as possible). So
with various experimenting, the slightly odd scheme I've come up with is
that if none of the arguments contain double-quote characters, then set
noglob and use the quoting mechanism previously described (so whitespace
within an argument or an @ at the beginning of an argument needing
double-quoting) and if a double-quote character appears in any of the
arguments, then don't set noglob and escape every argument with
double-quotes (to avoid globbing) - any double-quote characters within can
then be escaped with "'"'" (i.e. terminate the current quote string,
single-quote a double-quote and then resume a quote string!). Messy, but
that seems to be about the only way...


David 


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

Aaron Digulla wrote:
> 
> Am Samstag, 07. Mai 2016 09:45 CEST, "David Allsopp" <dr...@cantab.net>
> schrieb:
> 
> 
> > > Then all you need is a rudimentary quoting.
> >
> > Yes, but the question still remains what that rudimentary quoting is -
> i.e.
> > I can see how to quote spaces which appear in elements of argv, but I
> > cannot see how to quote double quotes!
> 
> This should help:
> https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/e
> veryone-quotes-command-line-arguments-the-wrong-way/

This provides documentation for how Microsoft implementations do it, not how 
Cygwin does it. The Cygwin DLL is responsible for determining how a Cygwin 
process gets argc and argv from GetCommandLineW.

> My line of thought is that Cygwin can't get anything which Windows can't
> send it. So the first step to solve this mess is to make sure the
> arguments which you send to CreateProcess() are correct.
> 
> The next step would be to write a small C utility which dumps it's
> arguments, so you can properly debug all kinds of characters.

See later email, but IMHO the conversion is something Cygwin should have 
precisely documented, not determined by brittle experimentation.

> PS: I always point people to string list/array type methods to create
> processes which fix all the problems with spaces and odd characters
> (quotes, umlauts, etc). It seems that Windows doesn't have such a method
> to create processes. Which kind of makes sense; Windows is very, very
> mouse centered.

I fail to see the connection with mice! What Windows (NT) does have is a legacy 
where the decision on how to convert a command line to a list/array of 
arguments is determined per-process (and so not the responsibility of command 
line shells) vs Unix which puts the burden of converting a single command line 
to the array on the shell instead. Nominally, the Windows way is more flexible, 
though I don't think that flexibility is actually useful (especially if you 
look at the comments in the command line -> argv conversion in Microsoft's C 
Runtime library!). 


David

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

Andrey Repin wrote:
> Greetings, David Allsopp!

And greetings to you, too!



> > I'm not using cmd, or any shell for that matter (that's actually the
> > point) - I am in a native Win32 process invoking a Cygwin process 
> > directly using the Windows API's CreateProcess call. As it happens, 
> > the program I have already has the arguments for the Cygwin process 
> > in an array, but Windows internally requires a single command line 
> > string (which is not in any related to Cmd).
> 
> Then all you need is a rudimentary quoting.

Yes, but the question still remains what that rudimentary quoting is - i.e.
I can see how to quote spaces which appear in elements of argv, but I cannot
see how to quote double quotes! 

> The rest will be handled by getopt when the command line is parsed.

That's outside my required level - I'm interested in Cygwin's emulation
handling the difference between an operating system which actually passes
argc and argv when creating processes (Posix exec/spawn) and Windows (which
only passes a single string command line). The Microsoft C Runtime and
Windows have a "clear" (at least by MS standards) specification of how that
single string gets converted to argv, I'm trying to determine Cygwin's -
getopt definitely isn't part of that.

> >> However, I've found Windows's interpretation to be inconsistent, so 
> >> often have to play with it to find what the "right combination" is 
> >> for a particular instance.
> >>
> >> I find echoing the parameters to a temporary text file and then 
> >> using the file as input to be more reliable and easier to 
> >> troubleshoot, and it breaks apart whether it is Windows cli 
> >> inconsistencies or receiving program issues very nicely with the 
> >> text file content as an intermediary
> 
> > This is an OK tack, but I don't wish to do this by experimentation 
> > and get caught out later by a case I didn't think of, so what I'm 
> > trying to determine is *exactly* how the Cygwin DLL processes the 
> > command line via its source code so that I can present it with my 
> > argv array converted to a single command line and be certain that 
> > the Cygwin will
> recover the same argv DLL.
> 
> > My reading of the relevant sources suggests that with globbing 
> > disabled, backslash escape sequences are *never* interpreted (since 
> > the quote function returns early - dcrt0.cc, line 171). If there is 
> > no way of encoding the double quote character, then perhaps I have 
> > to run with globbing enabled but ensure that the globify function 
> > will never actually expand anything - but as that's a lot of work, I 
> > was wondering
> if I was missing something with the simpler "noglob" case.
> 
> The point being, when you pass the shell and enter direct process 
> execution, you don't need much of shell magic at all.
> Shell conventions designed to ease interaction between system and 
> operator.
> But you have a system talking to the system, you can be very literal.

Indeed, which is why I'm trying to avoid the shell! But I can't be entirely
literal, because Posix and Windows are not compatible, so I need to
determine precisely how Cygwin's emulation works... and so far, it doesn't
seem to be a terribly clearly defined animal!

So, resorting to C files to try to demonstrate it further. spawn.cc seems to
suggest that there should be some kind of escaping available, but I'm
struggling to follow the code. Consider these two:

callee.c
  #include 
  int main (int argc, char* argv[])
  {
int i;

printf("argc = %d\n", argc);
for (i = 0; i < argc; i++) {
  printf("argv[%d] = %s\n", i, *argv++);
}
return 0;
  }

caller.c
  #include 
  #include 

  int main (void)
  {
LPTSTR commandLine;
STARTUPINFO startupInfo = {sizeof(STARTUPINFO), NULL, NULL, NULL, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, NULL, NULL, NULL, NULL};
PROCESS_INFORMATION process = {NULL, NULL, 0, 0};

commandLine = "callee.exe \"@\"te\"\n\"st fo@o bar\" \"baz baz *";
if (!CreateProcess("callee.exe", commandLine, NULL, NULL, FALSE, 0,
NULL, NULL, , )) {
  printf("Error spawning process!\n");
  return 1;
} else {
  WaitForSingleObject(process.hProcess, INFINITE);
  CloseHandle(process.hThread);
  CloseHandle(process.hProcess);
  return 0;
}
  }

If you compile as follows:

  $ gcc -o callee callee.c
  $ i686-w64-mingw32-gcc -o caller caller.c
  $ export CYGWIN=noglob  # Or the * will be expanded
  $ ./caller

and the output is as required:
  argc = 6
  argv[0] = callee
  argv[1] = @te
  st
  argv[2] = fo@o
  argv[3] = bar baz
  argv[4] = fliggle
  argv[5] = *

But if I want to e

RE: Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

[With apologies if threading is broken; I erroneously thought as the list was 
not subscriber-only that replies would use reply-all and so wasn't subscribed]

On Thu, May 5, 2016 at 06:47 PM, Erik Soderquist wrote:
> On Thu, May 5, 2016 at 11:24 AM, David Allsopp wrote:
> >
> > I am trying to work out the precise details for character escaping
> > when starting a Cygwin process from a native (i.e. non-Cygwin) Windows
> process.
> 
> > For example:
> >
> >   argv[0] = "foo"
> >   argv[1] = "bar baz"
> >
> > then the resulting command line string should be:
> >
> >   lpCommandLine = "foo bar\" \"baz"
> 
> If I recall correctly, Windows cmd.exe uses the carrot (^) as the general
> escape from shell character, so
> 
> C:\cygwin64\bin>.\echo.exe -e ^"hello\nworld^"
> hello
> world
> 
> works.

Indeed - but I'm not using cmd, or any shell for that matter (that's actually 
the point) - I am in a native Win32 process invoking a Cygwin process directly 
using the Windows API's CreateProcess call. As it happens, the program I have 
already has the arguments for the Cygwin process in an array, but Windows 
internally requires a single command line string (which is not in any related 
to Cmd).

> However, I've found Windows's interpretation to be inconsistent, so often
> have to play with it to find what the "right combination" is for a
> particular instance.
> 
> I find echoing the parameters to a temporary text file and then using the
> file as input to be more reliable and easier to troubleshoot, and it
> breaks apart whether it is Windows cli inconsistencies or receiving
> program issues very nicely with the text file content as an intermediary

This is an OK tack, but I don't wish to do this by experimentation and get 
caught out later by a case I didn't think of, so what I'm trying to determine 
is *exactly* how the Cygwin DLL processes the command line via its source code 
so that I can present it with my argv array converted to a single command line 
and be certain that the Cygwin will recover the same argv DLL.

My reading of the relevant sources suggests that with globbing disabled, 
backslash escape sequences are *never* interpreted (since the quote function 
returns early - dcrt0.cc, line 171). If there is no way of encoding the double 
quote character, then perhaps I have to run with globbing enabled but ensure 
that the globify function will never actually expand anything - but as that's a 
lot of work, I was wondering if I was missing something with the simpler 
"noglob" case.


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Formatting command line arguments when starting a Cygwin process from a native process

[David Allsopp]

I am trying to work out the precise details for character escaping when
starting a Cygwin process from a native (i.e. non-Cygwin) Windows process.

I have an array of command line arguments which I want passed verbatim to
the process, as though it were invoked using execv, with no globbing to take
place. I therefore disable globbing by including the noglob option in the
CYGWIN environment variable. My reading of winsup/cygwin/dcrt0.cc suggests
that I should convert argv to a single string to pass to the Windows
CreateProcess API call by protecting any whitespace characters (\t, \r, \n
and space itself) with double quotes. Then the escaped individual argv items
can be concatenated together with a space between each one.

For example:

  argv[0] = "foo"
  argv[1] = "bar baz"

then the resulting command line string should be:

  lpCommandLine = "foo bar\" \"baz"

and if I've interpreted build_argv and quoted correctly in dcrt0.cc, then as
long as allow_glob is 0 (which it is, via the noglob option in the CYGWIN
environment variable) then the Cygwin DLL will correctly reconstruct argv
based on that string returned by the Windows GetCommandLineW call made in
dll_crt0_1.

However, it appears that the single quote character may only be used to
quote strings if globbing is enabled (dcrt0.cc line 321) so how should one
encode the following argv?

  argv[0] = "foo"
  argv[1] = "bar \"baz\""

There doesn't seem to be anything along the lines of the trickery in the
Windows API's CommandLineToArgvW function if globbing is turned off?

Thanks for any pointers to the correct solution!


David


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple