Re: [bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Liviu Ionescu]

> On 29 Nov 2021, at 16:04, David Boyce  wrote:
> 
> ... why can't GNU make dig up these special "=" env vars on its own and add 
> them to its custom "envp" environment?

Sure, any suggestions are welcome, if someone knows how to do it, can use my 
test project (https://github.com/xpack-dev-tools/ms-spawn-issue), replace the 
included make.exe with anything else and experiment with various workarounds.

Regards,

Liviu

Re: [bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[David Boyce]

> The only thing that is guaranteed to work is to always create the
children processes with a NULL environment ...

This has been stated twice as an absolute fact but it leaves an obvious
question: why can't GNU make dig up these special "=" env vars on its own
and add them to its custom "envp" environment?

On Mon, Nov 29, 2021 at 5:38 AM Liviu Ionescu 
wrote:

> Follow-up Comment #8, bug #61409 (project make):
>
> > Your suggestion to modify the Make's own environment also won't work
> well,
> because Make invokes programs asynchronously, i.e. it doesn't wait for the
> child to exit, and so modifying Make's environment will affect Make itself
>
> I'm not sure this is accurate, my understanding is that the environment is
> copied to the child, so once the child is created, the parent can further
> change its own environment unhindered.
>
> > report this bug to Microsoft
>
> I'm currently investigating on how to report this to Microsoft, but, as I
> said, even if Microsoft fixes the bug, millions of older installs will
> fail.
>
> > and to MinGW64 folks, and hope that they fix it in a future release.
>
> The folks at Mingw-w64 helped with the detailed diagnose. There is not much
> they can do. The only thing that is guaranteed to work is to always create
> the
> children processes with a NULL environment, and this is up to the
> application.
>
> Remember, the BusyBox sh.exe, which was the first UCRT program to fail when
> invoked by make.exe, so the same sh.exe binary works just fine when invoked
> from cmd.exe. I'd bet that cmd.exe creates the process with a NULL
> environment.
>
> So, when I report this to Microsoft, they might very well reply that
> cmd.exe
> works, and if make.exe doesn't, I should report this to GNU make folks. ;-)
>
>
>
>
> ___
>
> Reply to this item at:
>
>   
>
> ___
>   Message sent via Savannah
>   https://savannah.gnu.org/
>
>
>

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Liviu Ionescu]

Follow-up Comment #8, bug #61409 (project make):

> Your suggestion to modify the Make's own environment also won't work well,
because Make invokes programs asynchronously, i.e. it doesn't wait for the
child to exit, and so modifying Make's environment will affect Make itself

I'm not sure this is accurate, my understanding is that the environment is
copied to the child, so once the child is created, the parent can further
change its own environment unhindered.

> report this bug to Microsoft 

I'm currently investigating on how to report this to Microsoft, but, as I
said, even if Microsoft fixes the bug, millions of older installs will fail.

> and to MinGW64 folks, and hope that they fix it in a future release.

The folks at Mingw-w64 helped with the detailed diagnose. There is not much
they can do. The only thing that is guaranteed to work is to always create the
children processes with a NULL environment, and this is up to the
application.

Remember, the BusyBox sh.exe, which was the first UCRT program to fail when
invoked by make.exe, so the same sh.exe binary works just fine when invoked
from cmd.exe. I'd bet that cmd.exe creates the process with a NULL
environment.

So, when I report this to Microsoft, they might very well reply that cmd.exe
works, and if make.exe doesn't, I should report this to GNU make folks. ;-)




___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Eli Zaretskii]

Follow-up Comment #7, bug #61409 (project make):

It is unrealistic to ask Make not to modify the environment of the programs it
invokes, because some of the GNU Make features absolutely require to put stuff
into the child environment.  Your suggestion to modify the Make's own
environment also won't work well, because Make invokes programs
asynchronously, i.e. it doesn't wait for the child to exit, and so modifying
Make's environment will affect Make itself.

So my recommendation is to report this bug to Microsoft and to MinGW64 folks,
and hope that they fix it in a future release.  Or maybe a less painful
workaround could be found.

Of course, the final decision is that of Paul, not mine.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Liviu Ionescu]

Follow-up Comment #6, bug #61409 (project make):

Now I have a slightly better understanding of the problem, although I had no
time to experiment anything yet.

The real problem is in Microsoft UCRT, which has a bug in the code managing
the environment.

Unfortunately, even if Microsoft will fix the bug tomorrow, we still have to
deal with millions of Windows instances which exhibit the bug. And in this
respect, all programs that spawn children processes should consider
workarounds, including GNU make.

If my understanding is correct (based on the explanations I received from
someone much more knowledgeable than myself), the problem is related to the
way the environment is understood by Microsoft. 

---
Internally, the "environment" in a Windows process consists of a little more
than what you see in the standard C envp array. If you run
GetEnvironmentStringsA/W and iterate over it, you'll see the normal
environment variables, but also a few other ones, like these:

=::=::\
=C:=C:\code\ms-spawn-issue\sources\spawn
=ExitCode=

These are special vars that start with an '=' - apparently the include
information about things like which directory is the "current directory" on
each drive letter.

When UCRT's spawn() tries to create a process, it wants to include these extra
variables from the current environment (from GetEnvironmentStringsA/W) on top
of the variables you provide in the 'envp' parameter.

When make starts its subprocesses, it does that by calling CreateProcess() and
passing in a string with the desired environment, but it doesn't know/care
about this kind of extra variables. So processes launched by make are missing
these extra variables in GetEnvironmentStringsA/W.

When launching a process with CreateProcess() and passing NULL to the
environment parameter, or passing NULL to the envp parameter to spawn, there's
no issue (the current exact environment is passed on). But if UCRT's spawn is
called in an environment that is missing these extra variables, it crashes.
---

So, if make creates sub-processes with a custom environment (i.e. does not
pass NULL to inherit the parent), those processes cannot use UCRT to create
sub-processes.

The same applies to sh.exe which spawns, for example, the compiler, to the
compiler who spawns the internal steps, and so on.

The workaround is relatively simple. The only known way that is guaranteed to
work is to pass a NULL environment, which means to inherit the environment
from the parent.

This means that instead of creating a custom environment, the parent must
temporarily adjust its own environment to match the desired one, spawn the
process, and restore the initial environment.

I did not check the code used by make, but if it passes a custom environment
and not a NULL, its children are affected by this Windows bug.

Regarding the question related to UCRT vs MSVCRT, the use of MSVCRT was
deprecated about 10 years ago when UCRT was introduced. mingw-w64 already
implements UCRT for some time, and it is very likely to become the default in
future mingw-w64 releases, since the MSVCRT code has many limitations and is
no longer maintained. All recent binary toolchain distributions (both clang
and gcc) produce binaries the use UCRT.

Thus the suggestion to stick to MSVCRT is not realistic.

Sure, you can close the issue, but this will not solve the problem.



___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Paul D. Smith]

Follow-up Comment #5, bug #61409 (project make):

Reading the busybox issue thread here
https://github.com/rmyorston/busybox-w32/issues/234 it _appears_ that the
problem really is not related to GNU make, although I can't understand why the
failure is only seen if the program is invoked from GNU make and not from
other tools or directly from the shell.

In any event, I'm tempted to close this issue.  @ilg please comment.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Eli Zaretskii]

Follow-up Comment #4, bug #61409 (project make):

If Make invokes other programs, and those programs have problems because the
use UCRT, how is that a problem of Make?

What I meant by my comment about MinGW was that as long as Make doesn't use
the 'spawnve' implementation from UCRT, Make itself is not directly affected
by that problem.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Paul D. Smith]

Follow-up Comment #3, bug #61409 (project make):

I agree that GNU make shouldn't / can't fix a problem in the MS runtime.  The
question is, whether there's a way GNU make could work around or avoid the
problem (and how annoying that would be).

I'm not sure I understand the comment that as long as MinGW doesn't use UCRT,
it's a theoretical problem; surely GNU make on Windows is used to invoke other
kinds of programs, that aren't built with MinGW?  Or did I misunderstand what
you meant?

Liviu: you didn't provide any information on how you compiled GNU make; can
you specify?  Did you use MSVC with the default build_w32.bat file?  Did you
use GCC with build_w32.bat?  Something else?  Maybe that doesn't matter WRT
this issue.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Eli Zaretskii]

Follow-up Comment #2, bug #61409 (project make):

AFAIU, this problem only happens when using the UCRT runtime, which is not
what MinGW uses.  So, as long as MinGW doesn't use UCRT, this is largely a
theoretical problem for GNU Make on Windows.  If and when MinGW64 will start
using UCRT as its runtime, the problem should be reported to them, so that
they provide some workaround.

IOW, I don't think it should be the GNU Make's business to fix a problem in
the MS runtime.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Paul D. Smith]

Follow-up Comment #1, bug #61409 (project make):

Unfortunately I have no experience in Windows.  Someone who has such
experience will need to track this down and figure out what's going wrong.  I
don't know if Eli has time or energy for this or not.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61409] The code used to create sub-processes on Windows triggers the spawnve() issue

[Liviu Ionescu]

URL:
  

 Summary: The code used to create sub-processes on Windows
triggers the spawnve() issue
 Project: make
Submitted by: ilg
Submitted on: Mon 01 Nov 2021 11:13:09 AM EET
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 4.3
Operating System: MS Windows
   Fixed Release: None
   Triage Status: None

___

Details:

Although I cannot tag this report as a direct bug in make, it is a major
contributing factor.

It seems that the implementation of 'spawnve()' in the modern Microsoft UCRT
API has a problem, and in certain conditions, calling the 'spawn*e()'
functions with a non-NULL environment, crashes with Error -1073741819 which is
0xC005, or ERROR_ACCESS_DENIED.

Unfortunately these conditions are met by GNU make, and some programs invoked
by 'make.exe' that try to create sub-processes with 'spawn*e()', fail.

Since 'spawnve()' is also called by BusyBox in its 'sh.exe' implementation,
this is a major issue, because it affects most common build tools.

I documented this issue in a separate project:

- https://github.com/xpack-dev-tools/ms-spawn-issue


Since ideally make should be able to start any sub-processes, including those
who use the unreliable 'spawnve()', it would be useful to review the Windows
specific code and try to avoid this problem.

A good source of inspiration might be Ninja Build code, which does not trigger
the condition that affects 'spawnve()' calls.

My knowledge of Windows is not good enough to provide a patch myself, but I
did my best to diagnose this issue, and I offer my help for further testing.






___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/