Re: [Mingw-w64-public] issues with application to run commands in parallel on Windows
On Tue, May 7, 2019 at 12:17 AM David Mathog wrote:
>
> On 04-May-2019 20:31, Liu Hao wrote:
> > 在 2019/5/4 上午6:16, David Mathog 写道:
>
> Thanks for the feedback, but I'm still nowhere near getting this to
> work.
>
> >>
> >> Issues:
> >>
> >>
> >> 1. Compiler warnings:
> >>
> >> c_spawn_n.c: In function 'main':
> >> c_spawn_n.c:47:13: warning: passing argument 3 of 'CreateThread' from
> >> incompatible pointer type [-Wincompatible-pointer-types]
> >> RunCommand, // thread function name
> >> ^~
> >
> > `LPTHREAD_START_ROUTINE` is the type of the parameter of
> > `CreateThread()`, which a pointer-to-function. The proper return type
> > for a thread procedure is `DWORD` rather than this thing.
>
> Changed the called routine to (see below):
>
> DWORD RunCommand( void *lpParam ){
try :
DWORD WINAPI Runcommand(LPVOID lpParam) { ...}
see https://docs.microsoft.com/en-us/windows/desktop/procthread/creating-threads
Vincent Torri
___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public
Re: [Mingw-w64-public] issues with application to run commands in parallel on Windows
在 2019/5/7 上午6:17, David Mathog 写道:
> Changed the called routine to (see below):
>
> DWORD RunCommand( void *lpParam ){
>
Insert an `__stdcall` before `RunCommand`. The calling convention is
part of the function type.
>>
>>> 2. when run procs is shown as -1, not sure how that happens.
>>
>> Use `WaitForSingleObject()` to await **every** thread in a loop. Do not
>> use `WaitForMultipleObjects()` as it has a upper limit of number of
>> handles to wait for, and the return value is hard to handle properly (as
>> you already did that wrong).
>
> You lost me. WaitForSingleObject() will only wait on a single handle at
> a time. There might
> be 20 subprocesses running, each with its own handle. A function is
> needed which will return
> when the first of those exits. Would that not be
> WaitForMultipleObjects()? Even if it will
> only work for lets say the first 16 processes, hopefully once those exit
> the invisible ones
> would slide into view? Or maybe it doesn't work that way?
>
If a thread terminates then its state becomes signaled; if you wait for
it a second time then the wait function returns immediately stating that
it is signaled.
To mimic the POSIX `wait()` or `pthread_join()` behavior, for each
signaled thread, you have to close its handle and remove it from the
wait list for the next `WaitForMultipleObjects()` call.
If I were you I would just write a loop that awaits ALL threads for
100ms. When the wait function returns, data can be copied from pipes to
the standard output. The loop terminates only when all threads exit,
whose handles are closed after the loop.
--
Best regards,
LH_Mouse
signature.asc
Description: OpenPGP digital signature
___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public
Re: [Mingw-w64-public] issues with application to run commands in parallel on Windows
On 04-May-2019 20:31, Liu Hao wrote:
在 2019/5/4 上午6:16, David Mathog 写道:
Thanks for the feedback, but I'm still nowhere near getting this to
work.
Issues:
1. Compiler warnings:
c_spawn_n.c: In function 'main':
c_spawn_n.c:47:13: warning: passing argument 3 of 'CreateThread' from
incompatible pointer type [-Wincompatible-pointer-types]
RunCommand, // thread function name
^~
`LPTHREAD_START_ROUTINE` is the type of the parameter of
`CreateThread()`, which a pointer-to-function. The proper return type
for a thread procedure is `DWORD` rather than this thing.
Changed the called routine to (see below):
DWORD RunCommand( void *lpParam ){
but ...
gcc -g -O0 -o c_spawn_n c_spawn_n.c
c_spawn_n.c: In function 'main':
c_spawn_n.c:47:13: warning: passing argument 3 of 'CreateThread' from
incompatible pointer type [-Wincompatible-pointer-types]
RunCommand, // thread function name
^~
In file included from
C:/progs/msys32/mingw32/i686-w64-mingw32/include/winbase.h:29:0,
from
C:/progs/msys32/mingw32/i686-w64-mingw32/include/windows.h:70,
from c_spawn_n.c:7:
C:/progs/msys32/mingw32/i686-w64-mingw32/include/processthreadsapi.h:163:28:
note: expected 'LPTHREAD_START_ROUTINE' but argument is of type 'DWORD
(*)(void *) {aka long unsigned int (*)(void *)}'
WINBASEAPI HANDLE WINAPI CreateThread (LPSECURITY_ATTRIBUTES
lpThreadAttributes, SIZE_T dwStackSize, LPTHREAD_START_ROUTINE
lpStartAddress, LPVOID lpParameter, DWORD dwCreationFlags, LPDWORD
lpThreadId);
^~~~
The processthreadsapi.h header line does indeed contain the type which
the compiler
is warning about:
grep CreateThread
C:/progs/msys32/mingw32/i686-w64-mingw32/include/processthreadsapi.h
WINBASEAPI HANDLE WINAPI CreateThread (LPSECURITY_ATTRIBUTES
lpThreadAttributes, SIZE_T dwStackSize, LPTHREAD_START_ROUTINE
lpStartAddress, LPVOID lpParameter, DWORD dwCreationFlags, LPDWORD
lpThreadId);
I don't see how to reconcile this.
2. when run procs is shown as -1, not sure how that happens.
Use `WaitForSingleObject()` to await **every** thread in a loop. Do not
use `WaitForMultipleObjects()` as it has a upper limit of number of
handles to wait for, and the return value is hard to handle properly
(as
you already did that wrong).
You lost me. WaitForSingleObject() will only wait on a single handle at
a time. There might
be 20 subprocesses running, each with its own handle. A function is
needed which will return
when the first of those exits. Would that not be
WaitForMultipleObjects()? Even if it will
only work for lets say the first 16 processes, hopefully once those exit
the invisible ones
would slide into view? Or maybe it doesn't work that way?
Don't forget to `CloseHandle()` when a thread handle is out of
interest.
The POSIX function `wait()` waits for a child process and frees its
resource, but on Windows you close its handle explicitly; wait
functions
do not free resources of it.
OK, added that. For good measure set the handle pointer to NULL after
closing it.
3. cannot tell very well if this is actually starting multiple
processes or if it
is just running sequentially.
4. there are now about about 100 cmd.exe processes showing up in
Windows Task Manager.
The `system` function passes its argument to the program designated by
the environment variable `COMSPEC` (which seems to have to be an
absolute path), which by default is `CMD.EXE` so you get a number of
instances.
If this is unacceptable you have to spawn child processes using
`CreateProcess()` and redirect standard streams yourself. There is a
nice document about this [1].
[1]
https://docs.microsoft.com/en-us/windows/desktop/ProcThread/creating-a-child-process-with-redirected-input-and-output
So something is hanging somewhere. It was run both in "MSYS2 Mingw
32"
and a W7
cmd shell. These processes are not visible in "ps -ef" in the
former.
`ps` only shows programs that run on the abstraction layer for POSIX of
MSYS2; native programs are out of its management.
5. Pretty sure I'm not checking status right on the thread creation
and
not
passing the system() exit status back correctly (or at all).
6. During the test it says this a variable number of times:
The process cannot access the file because it is being used by
another
process.
Rewrote the example. The input file was simplified to commands3.txt and
is now:
dir #c_spawn_n.c# > #c_spawn_n.c#.killme
dir accudate.1 > accudate.1.killme
dir accudate.c > accudate.c.killme
(which should run in either shell, those were just 3 files which were
available
in the current directory.)
Rewrote the code to NOT check for running processes until all (3) were
started.
Ran it like:
./c_spawn_n DEBUG firstempty 0 command >dir #c_spawn_n.c# > #c_spawn_n.c#.killme<
len 40 procs 0
DEBUG firstempty 1 comman
Re: [Mingw-w64-public] issues with application to run commands in parallel on Windows
在 2019/5/4 上午6:16, David Mathog 写道: > > Issues: > > > 1. Compiler warnings: > > c_spawn_n.c: In function 'main': > c_spawn_n.c:47:13: warning: passing argument 3 of 'CreateThread' from > incompatible pointer type [-Wincompatible-pointer-types] > RunCommand, // thread function name > ^~ `LPTHREAD_START_ROUTINE` is the type of the parameter of `CreateThread()`, which a pointer-to-function. The proper return type for a thread procedure is `DWORD` rather than this thing. > 2. when run procs is shown as -1, not sure how that happens. Use `WaitForSingleObject()` to await **every** thread in a loop. Do not use `WaitForMultipleObjects()` as it has a upper limit of number of handles to wait for, and the return value is hard to handle properly (as you already did that wrong). Don't forget to `CloseHandle()` when a thread handle is out of interest. The POSIX function `wait()` waits for a child process and frees its resource, but on Windows you close its handle explicitly; wait functions do not free resources of it. > 3. cannot tell very well if this is actually starting multiple > processes or if it > is just running sequentially. > 4. there are now about about 100 cmd.exe processes showing up in > Windows Task Manager. The `system` function passes its argument to the program designated by the environment variable `COMSPEC` (which seems to have to be an absolute path), which by default is `CMD.EXE` so you get a number of instances. If this is unacceptable you have to spawn child processes using `CreateProcess()` and redirect standard streams yourself. There is a nice document about this [1]. [1] https://docs.microsoft.com/en-us/windows/desktop/ProcThread/creating-a-child-process-with-redirected-input-and-output > So something is hanging somewhere. It was run both in "MSYS2 Mingw 32" > and a W7 > cmd shell. These processes are not visible in "ps -ef" in the former. `ps` only shows programs that run on the abstraction layer for POSIX of MSYS2; native programs are out of its management. > 5. Pretty sure I'm not checking status right on the thread creation and > not > passing the system() exit status back correctly (or at all). > 6. During the test it says this a variable number of times: > > The process cannot access the file because it is being used by another > process. It is mostly due to the fact that the same file is opened and closed repeatedly and individually by each child CMD.EXE. So you probably have to take the `CreateProcess()` approach and do your own redirection, buffering, etc. > > I think that means that it has problems with >>test_out.log running on > multiple processes > at once and that there is some implicit file locking. > -- Best regards, LH_Mouse -- Best regards, ltpmouse signature.asc Description: OpenPGP digital signature ___ Mingw-w64-public mailing list Mingw-w64-public@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mingw-w64-public
[Mingw-w64-public] issues with application to run commands in parallel on Windows
Hi all,
I maintain a little package with some text and binary utilities here:
https://sourceforge.net/projects/drmtools/
In the last release the "execinput" program was modified so that instead
of running
just one command it could run N at a time. Like so (silly example to
calculate
md5sum values for every file in a directory and store each in an ".md5"
file, using
20 parallel processes):
ls -1 | extract -fmt 'md5sum [1,] > [1,].md5 ' | execinput -t 20
This uses fork() and wait() and works fine on linux.
Problem is, on windows there is not a strict equivalent for those
functions.
Below is a first pass at getting something similar to work. It just
reads from stdin
and tries to create a thread in which it does a system() call. At the
moment it has several
issues, none of which I understand very well. Compiled in 32 bit
Mingw64 on a Windows 7 32 bit system (if that is the right way to say
that) with:
gcc -g -O0 -o c_spawn_n c_spawn_n.c
Not sure what release of Mingw64 this is but it was installed circa
8/29/2018.
For the tests described below it was given the task of "DIR onefile can
concatanate result to a single file", like so:
ls -1 | extract -fmt 'dir [1,] >> test_out.log' >many_dir.txt
head -3 many_dir.txt
dir #c_spawn_n.c# >> test_out.log
dir accudate.1 >> test_out.log
dir accudate.c >> test_out.log
./c_spawn_n oops3.txt
#preceding was msys, for cmd.exe without the leading path
Issues:
1. Compiler warnings:
c_spawn_n.c: In function 'main':
c_spawn_n.c:47:13: warning: passing argument 3 of 'CreateThread' from
incompatible pointer type [-Wincompatible-pointer-types]
RunCommand, // thread function name
^~
In file included from
C:/progs/msys32/mingw32/i686-w64-mingw32/include/winbase.h:29:0,
from
C:/progs/msys32/mingw32/i686-w64-mingw32/include/windows.h:70,
from c_spawn_n.c:7:
C:/progs/msys32/mingw32/i686-w64-mingw32/include/processthreadsapi.h:163:28:
note: expected 'LPTHREAD_START_ROUTINE' but argument is of type 'DWORD
(__attribute__((__stdcall__)) * (*)(void *))(void *)'
WINBASEAPI HANDLE WINAPI CreateThread (LPSECURITY_ATTRIBUTES
lpThreadAttributes, SIZE_T dwStackSize, LPTHREAD_START_ROUTINE
lpStartAddress, LPVOID lpParameter, DWORD dwCreationFlags, LPDWORD
lpThreadId);
^~~~
c_spawn_n.c: In function 'RunCommand':
c_spawn_n.c:70:12: warning: return makes pointer from integer without a
cast [-Wint-conversion]
return (retval);
^
2. when run procs is shown as -1, not sure how that happens.
3. cannot tell very well if this is actually starting multiple
processes or if it
is just running sequentially.
4. there are now about about 100 cmd.exe processes showing up in
Windows Task Manager.
So something is hanging somewhere. It was run both in "MSYS2 Mingw 32"
and a W7
cmd shell. These processes are not visible in "ps -ef" in the former.
5. Pretty sure I'm not checking status right on the thread creation and
not
passing the system() exit status back correctly (or at all).
6. During the test it says this a variable number of times:
The process cannot access the file because it is being used by another
process.
I think that means that it has problems with >>test_out.log running on
multiple processes
at once and that there is some implicit file locking.
Here is the small test program's code:
// c_spawn_n.c
// TEST program to run up to 20 commands in parallel as subprocesses.
// Read commands from stdin.
//
//
#include
#include
#include
#include
#include
#include
// prototypes
LPTHREAD_START_ROUTINE RunCommand( LPVOID lpParam ); // runs thread
int max_threads=20;
int procs=0;// Number of processes running
int main(int argc, char *argv[]){
#define MAXSTRING 256
char *command = malloc(MAXSTRING);
char **ThrCommands = calloc(sizeof(char *), max_threads);
HANDLE *ThrHandles = (HANDLE *) calloc(sizeof(HANDLE),max_threads);
DWORD *ThreadIds = (DWORD *) calloc(sizeof(DWORD),max_threads);
int firstempty=0;
int procs=0;
DWORD retVal;
while(1){
if(NULL == fgets(command,MAXSTRING,stdin))break;
if(procs){
firstempty = WaitForMultipleObjects(
max_threads, ThrHandles, FALSE, INFINITE);
procs--;
GetExitCodeThread(ThrHandles[firstempty], &retVal);
fprintf(stderr,"DEBUG thread %d exited with status %ul\n",
firstempty,retVal);
}
fprintf(stderr,"DEBUG firstempty %d command >%s< len %d\n",
firstempty,command,strlen(command));
ThrCommands[firstempty] = strdup(command); // save it
ThrHandles[firstempty] = CreateThread(
NULL, // default security attributes
0, // use default stack size
RunCommand, // thread function name
ThrCommands[firstempty],// argument to th
