Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 8 Dec 2021, at 17:11, Jen Kris  wrote:
> 
> I started this post on November 29, and there have been helpful comments 
> since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris 
> Angelico.  Thanks to all of you.  
> 
> I've found a solution that works for my purpose, and I said earlier that I 
> would post the solution I found. If anyone has a better solution I would 
> appreciate any feedback. 
> 
> To recap, I'm using a pair of named pipes for IPC between C and Python.  
> Python runs as a child process after fork-execv.  The Python program 
> continues to run concurrently in a while True loop, and responds to requests 
> from C at intervals, and continues to run until it receives a signal from C 
> to exit.  C sends signals to Python, then waits to receive data back from 
> Python.  My problem was that C was blocked when Python started. 
> 
> The solution was twofold:  (1) for Python to run concurrently it must be a 
> multiprocessing loop (from the multiprocessing module), and (2) Python must 
> terminate its write strings with \n, or read will block in C waiting for 
> something that never comes.  The multiprocessing module sidesteps the GIL; 
> without multiprocessing the GIL will block all other threads once Python 
> starts. 
> 
> Originally I used epoll() on the pipes.  Cameron Smith and Barry Scott 
> advised against epoll, and for this case they are right.  Blocking pipes work 
> here, and epoll is too much overhead for watching on a single file 
> descriptor. 
> 
> This is the Python code now:
> 
> #!/usr/bin/python3
> from multiprocessing import Process

You already have feedback that multiprocessing is not required.

> import os
> 
> print("Python is running")
> 
> child_pid = os.getpid()
> print('child process id:', child_pid)
> 
> def f(a, b):
> 
> print("Python now in function f")
> 
> pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
> print("File Descriptor1 Opened " + str(pr))
> pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
> print("File Descriptor2 Opened " + str(pw))
> 
> while True:
> 
> v = os.read(pr,64)
> print("Python read from pipe pr")
> print(v)
> 
> if v == b'99':
> os.close(pr)
> os.close(pw)
> print("Python is terminating")
> os._exit(os.EX_OK)
> 
> if v != "Send child PID":
> os.write(pw, b"OK message received\n")

The \n should not be required as UDS (unix domain sockets) are message based 
not a stream of bytes.

> print("Python wrote back")
> 
> if __name__ == '__main__':
> a = 0
> b = 0
> p = Process(target=f, args=(a, b,))
> p.start()
> p.join()
> 
> The variables a and b are not currently used in the body, but they will be 
> later. 
> 
> This is the part of the C code that communicates with Python:
> 
> fifo_fd1 = open(fifo_path1, O_WRONLY);
> fifo_fd2 = open(fifo_path2, O_RDONLY);
> 
> status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
> if (status_write < 0) perror("write");
You continue on after the error, exit would be better.
> 
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
> if (status_read < 0) perror("read");
> printf("C received message 1 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

> 
> status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));

How much data did you put into py_msg_01 buffer?
Is it a C string? If so you want to write sizeof(py_msg_02)-1 to avoid sending 
the trailing NUL (0)
of the C string.

> if (status_write < 0) perror("write");
If it failed exit until you have figured out the error handling.
> 
> status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
> if (status_read < 0) perror("read");
> printf("C received message 2 from Python\n");
> printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

At no point do I see in this C code the need for a \n in the data sent between 
python and C.

> 
> // Terminate Python multiprocessing
> printf("C is sending exit message to Python\n");
> status_write = write(fifo_fd1, py_msg_03, 2);

Would be better to not be using "2" for the length to write here.
If py_msg_03 only has the exit msg in it the use sizeof(py_msg_03).
If its a C string the subtract 1 for the trailing NUL (0).

> 
> printf("C is closing\n");
> close(fifo_fd1);
> close(fifo_fd2);
> 
> Screen output:
> 
> Python is running
> child process id: 5353
> Python now in function f
> File Descriptor1 Opened 6
> Thread created 0
> File Descriptor2 Opened 7
> Process ID: 5351
> Parent Process ID: 5351
> I am the parent
> Core joined 0
> I am the child
> Python read from pipe pr
> b'Hello to Python from C\x00\x00'
The \x00 is because the length you used is wrong.

> Python wrote back
> C received message 1 from Python
> OK message received

Re: Python child process in while True loop blocks parent

[Peter J. Holzer]

On 2021-12-08 18:11:48 +0100, Jen Kris via Python-list wrote:
> To recap, I'm using a pair of named pipes for IPC between C and
> Python.  Python runs as a child process after fork-execv.  The Python
> program continues to run concurrently in a while True loop, and
> responds to requests from C at intervals, and continues to run until
> it receives a signal from C to exit.  C sends signals to Python, then
> waits to receive data back from Python.  My problem was that C was
> blocked when Python started. 
> 
> The solution was twofold:  (1) for Python to run concurrently it must
> be a multiprocessing loop (from the multiprocessing module),

I don't see how this could achieve anything. It starts another (third)
process, but then it just does all the work in that process and just
waits for it. Doing the same work in the original Python process should
have exactly the same effect.

> and (2) Python must terminate its write strings with \n, or read will
> block in C waiting for something that never comes.

That's also strange. You are using os.write in Python and read in C,
both of which shoudn't care about newlines.

> The multiprocessing module sidesteps the GIL; without multiprocessing
> the GIL will block all other threads once Python starts. 

Your Python interpreter runs in a different process than your C code.
There is absolutely no way the GIL could block threads in your C
program. And your Python code doesn't need to use more than one thread
or process.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Cameron Simpson]

On 08Dec2021 18:11, Jen Kris  wrote:
>Python must terminate its write strings with \n, or read will block in 
>C waiting for something that never comes.

There are two aspects to this:
- if upstream is rding "lines of text" then you need a newline to 
  terminate the lines
- you (probably) should flush the output pipe (Python to C) after the 
  newline

I see you're using file descriptors and os.write() to sent data. This is 
unbuffered, so there is nothing to flush, so you have not encountered 
the second point above.

But if you shift to using a Python file object for your output (eg 
f=os.fdopen(pw)), which would let you use print() or any number of other 
things which do things with Python files) your file object would have a 
buffer and normally that would not be sent to the pipe unless it was 
full.

So your deadlock issue has 2 components:
- you need to terminate your records for upstream (C) to see complete 
  records. Your records are lines, so you need a newline character.
- you need to ensure the whole record has been sent upstream (been 
  written to the pipe). If you use a buffered Python file object for 
  your output, you need to flush it at your synchronisation points or 
  upstream will not receive the buffer contents. That synchronisation 
  point for you is the end of the record.

Hopefully this makes the flow considerations more clear.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

I started this post on November 29, and there have been helpful comments since 
then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico.  
Thanks to all of you.  

I've found a solution that works for my purpose, and I said earlier that I 
would post the solution I found. If anyone has a better solution I would 
appreciate any feedback. 

To recap, I'm using a pair of named pipes for IPC between C and Python.  Python 
runs as a child process after fork-execv.  The Python program continues to run 
concurrently in a while True loop, and responds to requests from C at 
intervals, and continues to run until it receives a signal from C to exit.  C 
sends signals to Python, then waits to receive data back from Python.  My 
problem was that C was blocked when Python started. 

The solution was twofold:  (1) for Python to run concurrently it must be a 
multiprocessing loop (from the multiprocessing module), and (2) Python must 
terminate its write strings with \n, or read will block in C waiting for 
something that never comes.  The multiprocessing module sidesteps the GIL; 
without multiprocessing the GIL will block all other threads once Python 
starts. 

Originally I used epoll() on the pipes.  Cameron Smith and Barry Scott advised 
against epoll, and for this case they are right.  Blocking pipes work here, and 
epoll is too much overhead for watching on a single file descriptor. 

This is the Python code now:

#!/usr/bin/python3
from multiprocessing import Process
import os

print("Python is running")

child_pid = os.getpid()
print('child process id:', child_pid)

def f(a, b):

    print("Python now in function f")

    pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
    print("File Descriptor1 Opened " + str(pr))
    pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
    print("File Descriptor2 Opened " + str(pw))

    while True:

    v = os.read(pr,64)
    print("Python read from pipe pr")
    print(v)

    if v == b'99':
    os.close(pr)
    os.close(pw)
    print("Python is terminating")
    os._exit(os.EX_OK)

    if v != "Send child PID":
    os.write(pw, b"OK message received\n")
    print("Python wrote back")

if __name__ == '__main__':
    a = 0
    b = 0
    p = Process(target=f, args=(a, b,))
    p.start()
    p.join()

The variables a and b are not currently used in the body, but they will be 
later. 

This is the part of the C code that communicates with Python:

    fifo_fd1 = open(fifo_path1, O_WRONLY);
    fifo_fd2 = open(fifo_path2, O_RDONLY);

    status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
    if (status_write < 0) perror("write");

    status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
    if (status_read < 0) perror("read");
    printf("C received message 1 from Python\n");
    printf("%.*s",(int)buf_len, fifo_readbuf);

    status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
    if (status_write < 0) perror("write");

    status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
    if (status_read < 0) perror("read");
    printf("C received message 2 from Python\n");
    printf("%.*s",(int)buf_len, fifo_readbuf);

    // Terminate Python multiprocessing
    printf("C is sending exit message to Python\n");
    status_write = write(fifo_fd1, py_msg_03, 2);

    printf("C is closing\n");
    close(fifo_fd1);
    close(fifo_fd2);

Screen output:

Python is running
child process id: 5353
Python now in function f
File Descriptor1 Opened 6
Thread created 0
File Descriptor2 Opened 7
Process ID: 5351
Parent Process ID: 5351
I am the parent
Core joined 0
I am the child
Python read from pipe pr
b'Hello to Python from C\x00\x00'
Python wrote back
C received message 1 from Python
OK message received
Python read from pipe pr
b'Message to Python 2\x00\x00'
Python wrote back
C received message 2 from Python
OK message received
C is sending exit message to Python
C is closing
Python read from pipe pr
b'99'
Python is terminating

Python runs on a separate thread (created with pthreads) because I want the 
flexibility of using this same basic code as a stand-alone .exe, or for a C 
extension from Python called with ctypes.  If I use it as a C extension then I 
want the Python code on a separate thread because I can't have two instances of 
the Python interpreter running on one thread, and one instance will already be 
running on the main thread, albeit "suspended" by the call from ctypes. 

So that's my solution:  (1) Python multiprocessing module; (2) Python strings 
written to the pipe must be terminated with \n. 

Thanks again to all who commented. 



Dec 6, 2021, 13:33 by ba...@barrys-emacs.org:

>
>
>
>> On 6 Dec 2021, at 21:05, Jen Kris <>> jenk...@tutanota.com>> > wrote:
>>
>> Here is what I don't understand from what you said.  "The child process is 
>> created with a single thread—the one that called fork()."  To me that 
>> implies that the thread that called fork() is the same 

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 6 Dec 2021, at 21:05, Jen Kris  wrote:
> 
> Here is what I don't understand from what you said.  "The child process is 
> created with a single thread—the one that called fork()."  To me that implies 
> that the thread that called fork() is the same thread as the child process.  
> I guess you're talking about the distinction between logical threads and 
> physical threads.  

The thread that called fork is cloned into a new thread in the new process.
It has a clone of the processor registers of the thread, stack memory segment 
of that thread,
which means that it has all the stack variables and stack frames from the 
parent process's thread.


> But the main issue is your suggestion that I should call fork-execv from the 
> thread that runs the main C program, not from a separate physical pthread.  
> That would certainly eliminate the overhead of creating a new pthread. 

Forget about physical threads, they only matter when you are performance tuning 
your code.

> I am working now to finish this, and I will try your suggestion of calling 
> fork-execv from the "main" thread.  When I reply back next I can give you a 
> complete picture of what I'm doing. 
> 
> Your comments, and those of Peter Holzer and Chris Angelico, are most 
> appreciated. 

Barry


> 
> 
> Dec 6, 2021, 10:37 by ba...@barrys-emacs.org:
> 
> On 6 Dec 2021, at 17:09, Jen Kris via Python-list  
> wrote:
> 
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages 
> https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
> created with a single thread—the one that called fork()."
> 
> You just quoted the evidence!
> 
> All new processes on unix (may all OS) only ever have one thread when they 
> start.
> The thread-id of the first thread is the same as the process-id and referred 
> to as the main thread.
> 
> I have a one-core one-thread instance at Digital Ocean available running 
> Ubuntu 18.04. I can fork and create a new process on it, but it doesn't 
> create a new thread because it doesn't have one available.
> 
> 
> By that logic it can only run one process...
> 
> It has one hardware core that support one hardware thread.
> Linux can create as many software threads as it likes.
> You may also want to see "Forking vs Threading" 
> (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), 
> "Fork vs Thread" 
> (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux 
> process and thread" (https://zliu.org/post/linux-process-and-thread) ("This 
> means that to create a normal process fork() is used that further calls 
> clone() with appropriate arguments while to create a thread or LWP, a 
> function from pthread library calls clone() with relvant flags. So, the main 
> difference is generated by using different flags that can be passed to 
> clone() funciton(to be exact, it is a system call"). 
> 
> You may be confused by the fact that threads are called light-weight 
> processes.
> 
> No Peter and I are not confused.
> 
> Or maybe I'm confused :)
> 
> Yes you are confused.
> 
> If you have other information, please let me know. Thanks.
> 
> Please get the book I recommended, or another that covers systems programming 
> on unix, and have a read.
> 
> Barry
> 
> Jen
> 
> 
> Dec 5, 2021, 18:08 by hjp-pyt...@hjp.at:
> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process. On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program. That
> way Python runs on a separate thread. 
> 
> I think you have the relationship between processes and threads
> backwards. A process consists of one or more threads. Fork creates a new
> process and therefore also a new thread.
> 
> hp
> 
> -- 
> _ | Peter J. Holzer | Story must make more sense than reality.
> |_|_) | |
> | | | h...@hjp.at | -- Charles Stross, "Creative writing
> __/ | http://www.hjp.at/ | challenge!"
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

Here is what I don't understand from what you said.  "The child process is 
created with a single thread—the one that called fork()."  To me that implies 
that the thread that called fork() is the same thread as the child process.  I 
guess you're talking about the distinction between logical threads and physical 
threads.  

But the main issue is your suggestion that I should call fork-execv from the 
thread that runs the main C program, not from a separate physical pthread.  
That would certainly eliminate the overhead of creating a new pthread. 

I am working now to finish this, and I will try your suggestion of calling 
fork-execv from the "main" thread.  When I reply back next I can give you a 
complete picture of what I'm doing. 

Your comments, and those of Peter Holzer and Chris Angelico, are most 
appreciated. 




Dec 6, 2021, 10:37 by ba...@barrys-emacs.org:

>
>
>> On 6 Dec 2021, at 17:09, Jen Kris via Python-list  
>> wrote:
>>
>> I can't find any support for your comment that "Fork creates a new
>> process and therefore also a new thread."  From the Linux man pages 
>> https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
>> created with a single thread—the one that called fork()."
>>
>
> You just quoted the evidence!
>
> All new processes on unix (may all OS) only ever have one thread when they 
> start.
> The thread-id of the first thread is the same as the process-id and referred 
> to as the main thread.
>
>>
>> I have a one-core one-thread instance at Digital Ocean available running 
>> Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't 
>> create a new thread because it doesn't have one available.
>>
>
>
> By that logic it can only run one process...
>
> It has one hardware core that support one hardware thread.
> Linux can create as many software threads as it likes.
>
>> You may also want to see "Forking vs Threading" 
>> (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), 
>> "Fork vs Thread" 
>> (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux 
>> process and thread" (https://zliu.org/post/linux-process-and-thread) ("This 
>> means that to create a normal process fork() is used that further calls 
>> clone() with appropriate arguments while to create a thread or LWP, a 
>> function from pthread library calls clone() with relvant flags. So, the main 
>> difference is generated by using different flags that can be passed to 
>> clone() funciton(to be exact, it is a system call"). 
>>
>> You may be confused by the fact that threads are called light-weight 
>> processes.
>>
>
> No Peter and I are not confused.
>
>>
>> Or maybe I'm confused :)
>>
>
> Yes you are confused.
>
>>
>> If you have other information, please let me know.  Thanks.
>>
>
> Please get the book I recommended, or another that covers systems programming 
> on unix, and have a read.
>
> Barry
>
>>
>> Jen
>>
>>
>> Dec 5, 2021, 18:08 by hjp-pyt...@hjp.at:
>>
>>> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>>>
 The C program creates two threads (using pthreads), one for itself and
 one for the child process.  On creation, the second pthread is pointed
 to a C program that calls fork-execv to run the Python program.  That
 way Python runs on a separate thread. 

>>>
>>> I think you have the relationship between processes and threads
>>> backwards. A process consists of one or more threads. Fork creates a new
>>> process and therefore also a new thread.
>>>
>>> hp
>>>
>>> -- 
>>> _  | Peter J. Holzer| Story must make more sense than reality.
>>> |_|_) ||
>>> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
>>> __/   | http://www.hjp.at/ |   challenge!"
>>>
>>
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>>

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 6 Dec 2021, at 17:09, Jen Kris via Python-list  
> wrote:
> 
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread."  From the Linux man pages 
> https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
> created with a single thread—the one that called fork()." 

You just quoted the evidence!

All new processes on unix (may all OS) only ever have one thread when they 
start.
The thread-id of the first thread is the same as the process-id and referred to 
as the main thread.

> 
> I have a one-core one-thread instance at Digital Ocean available running 
> Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't 
> create a new thread because it doesn't have one available. 


By that logic it can only run one process...

It has one hardware core that support one hardware thread.
Linux can create as many software threads as it likes.

> You may also want to see "Forking vs Threading" 
> (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), 
> "Fork vs Thread" 
> (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux 
> process and thread" (https://zliu.org/post/linux-process-and-thread) ("This 
> means that to create a normal process fork() is used that further calls 
> clone() with appropriate arguments while to create a thread or LWP, a 
> function from pthread library calls clone() with relvant flags. So, the main 
> difference is generated by using different flags that can be passed to 
> clone() funciton(to be exact, it is a system call"). 
> 
> You may be confused by the fact that threads are called light-weight 
> processes. 

No Peter and I are not confused.

> 
> Or maybe I'm confused :)

Yes you are confused.

> 
> If you have other information, please let me know.  Thanks. 

Please get the book I recommended, or another that covers systems programming 
on unix, and have a read.

Barry

> 
> Jen
> 
> 
> Dec 5, 2021, 18:08 by hjp-pyt...@hjp.at:
> 
>> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>> 
>>> The C program creates two threads (using pthreads), one for itself and
>>> one for the child process.  On creation, the second pthread is pointed
>>> to a C program that calls fork-execv to run the Python program.  That
>>> way Python runs on a separate thread. 
>>> 
>> 
>> I think you have the relationship between processes and threads
>> backwards. A process consists of one or more threads. Fork creates a new
>> process and therefore also a new thread.
>> 
>> hp
>> 
>> -- 
>> _  | Peter J. Holzer| Story must make more sense than reality.
>> |_|_) ||
>> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
>> __/   | http://www.hjp.at/ |   challenge!"
>> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Chris Angelico]

On Tue, Dec 7, 2021 at 4:10 AM Jen Kris via Python-list
 wrote:
>
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread."  From the Linux man pages 
> https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
> created with a single thread—the one that called fork()."
>
> I have a one-core one-thread instance at Digital Ocean available running 
> Ubuntu 18.04.  I can fork and create a new process on it, but it doesn't 
> create a new thread because it doesn't have one available.
>

A CPU core is capable of running one or more threads *concurrently*,
and having multiple cores obviously multiplies that out. But for what
you're doing here, there's no material difference between threads
running concurrently and threads switching out between themselves
(other than performance, of course). From the operating system and
application perspectives, a "thread" is one runnable thread.

Every Unix process must contain a minimum of one thread. As a rough
rule of thumb, a process owns resources, a thread runs code. Without a
thread, you can't run any code.

Forking a process creates a child process and leaves the parent
running. The child process must by definition have a minimum of one
thread, and the man page is stating that one thread is all it gets.

Hope that helps explain things a bit. I don't know exactly what's
going on in your code, but maybe that'll clarify the word "thread".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

I can't find any support for your comment that "Fork creates a new
process and therefore also a new thread."  From the Linux man pages 
https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is 
created with a single thread—the one that called fork()." 

I have a one-core one-thread instance at Digital Ocean available running Ubuntu 
18.04.  I can fork and create a new process on it, but it doesn't create a new 
thread because it doesn't have one available. 

You may also want to see "Forking vs Threading" 
(https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork 
vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and 
"Linux process and thread" (https://zliu.org/post/linux-process-and-thread) 
("This means that to create a normal process fork() is used that further calls 
clone() with appropriate arguments while to create a thread or LWP, a function 
from pthread library calls clone() with relvant flags. So, the main difference 
is generated by using different flags that can be passed to clone() funciton(to 
be exact, it is a system call"). 

You may be confused by the fact that threads are called light-weight processes. 

Or maybe I'm confused :)

If you have other information, please let me know.  Thanks. 

Jen


Dec 5, 2021, 18:08 by hjp-pyt...@hjp.at:

> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
>
>> The C program creates two threads (using pthreads), one for itself and
>> one for the child process.  On creation, the second pthread is pointed
>> to a C program that calls fork-execv to run the Python program.  That
>> way Python runs on a separate thread. 
>>
>
> I think you have the relationship between processes and threads
> backwards. A process consists of one or more threads. Fork creates a new
> process and therefore also a new thread.
>
>  hp
>
> -- 
>  _  | Peter J. Holzer| Story must make more sense than reality.
> |_|_) ||
> | |   | h...@hjp.at |-- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |   challenge!"
>

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry]

> On 5 Dec 2021, at 23:51, Jen Kris  wrote:
> 
> 
> By embedding, I think you may be referring to embedding Python in a C program 
> with the Python C API.   That's not what I'm doing here -- I'm not using the 
> Python C API.  The C program creates two threads (using pthreads), one for 
> itself and one for the child process.  On creation, the second pthread is 
> pointed to a C program that calls fork-execv to run the Python program.  That 
> way Python runs on a separate thread.  The multiprocessing library 
> "effectively side-step[s] the Global Interpreter Lock by using subprocesses 
> instead of threads."  https://docs.python.org/3/library/multiprocessing.html. 
>  This way I can get the Python functionality I want on call from the C 
> program through pipes and shared memory.  
> 
> I don't want to use the C API because I will be making certain library calls 
> from the C program, and the syntax is much easier with native Python code 
> than with C API code. 
> 
> I hope that clarifies what I'm doing. 

You are confusing the threading needs of a python program and extension written 
in C
and what you need to do to start a child process.

I see no benifit from calling fork and exec from a new thread. Indeed you may
encounter more issues.

What ever the bug is that you and trying to fix threading has nothing to do 
with it.

What if you change you code to run bash script instead of python?
Does you C program block them?

What did strace tell you was the reason that your C parent program blocked?

Barry



> 
> Jen
> 
> 
> 
> Dec 5, 2021, 15:19 by ba...@barrys-emacs.org:
> 
> 
> 
>>> On 5 Dec 2021, at 17:54, Jen Kris  wrote:
>>  
>> Thanks for your comments.  
>> 
>> I put the Python program on its own pthread, and call a small C program to 
>> fork-execv to call the Python program as a child process. 
> 
> What do you mean by putting python in it’s own pthread?
> Are you embedding python in an other program?
> 
> Barry
> 
> 
>> I revised the Python program to be a multiprocessing loop using the Python 
>> multiprocessing module.  That bypasses the GIL and allows Python to run 
>> concurrently with C.  So far so good.  
>> 
>> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC 
>> between Python and C.  Multiprocessing pipes are (as far as I can tell) only 
>> for commo between two Python processes.  I will have the parent thread send 
>> a signal through the pipe to the child process to exit when the parent 
>> thread is ready to exit, then call wait() to finalize the child process.  
>> 
>> I will reply back when it's finished and post the code so you can see what I 
>> have done.  
>> 
>> Thanks again.  
>> 
>> Jen
>> 
>> 
>> Dec 4, 2021, 09:22 by ba...@barrys-emacs.org:
>> 
>>> On 1 Dec 2021, at 16:01, Jen Kris  wrote:
>>> 
>>> Thanks for your comment re blocking.  
>>> 
>>> I removed pipes from the Python and C programs to see if it blocks without 
>>> them, and it does.
>>> It looks now like the problem is not pipes.
>> 
>> Ok.
>> 
>>> I use fork() and execv() in C to run Python in a child process, but the 
>>> Python process blocks
>> 
>> Use strace on the parent process to see what is happening.
>> You will need to use the option to follow subprocesses so that you can see 
>> what goes on in the python process.
>> 
>> See man strace and the --follow-forks and --output-separately options.
>> That will allow you to find the blocking system call that your code is 
>> making.
>> 
>>> because fork() does not create a new thread, so the Python global 
>>> interpreter lock (GIL) prevents the C program from running once Python 
>>> starts.
>> 
>> Not sure why you think this.
>> 
>>>   So the solution appears to be run Python in a separate thread, which I 
>>> can do with pthread create.
>>>   See "Thread State and the Global Interpreter Lock" 
>>> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
>>>  and the sections below that "Non-Python created threads" and "Cautions 
>>> about fork()." 
>> 
>> I take it you mean that in the parent you think that using pthreads will 
>> affect python after the exec() call?
>> I does not. After exec() the process has one main thread create by the 
>> kernel and a new address space as defined by the /usr/bin/python.
>> The only state that in inherited from the parent are open file descriptors, 
>> the current working directory and security state like UID, GID.
>> 
>>> I'm working on that today and I hope all goes well :) 
>> 
>> You seem to be missing background information on how processes work.
>> Maybe "Advanced Programming in the UNIX Environment" would be helpful?
>> 
>> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk
>>  
>> 
>> It's a great book and covers a wide range of Unix systems programming topics.
>> 
>> Have you created a small C program that just does the fork and exec of a 
>> python program to test out 

Re: Python child process in while True loop blocks parent

[Peter J. Holzer]

On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process.  On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program.  That
> way Python runs on a separate thread. 

I think you have the relationship between processes and threads
backwards. A process consists of one or more threads. Fork creates a new
process and therefore also a new thread.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

By embedding, I think you may be referring to embedding Python in a C program 
with the Python C API.   That's not what I'm doing here -- I'm not using the 
Python C API.  The C program creates two threads (using pthreads), one for 
itself and one for the child process.  On creation, the second pthread is 
pointed to a C program that calls fork-execv to run the Python program.  That 
way Python runs on a separate thread.  The multiprocessing library "effectively 
side-step[s] the Global Interpreter Lock by using subprocesses instead of 
threads."  https://docs.python.org/3/library/multiprocessing.html.  This way I 
can get the Python functionality I want on call from the C program through 
pipes and shared memory.  

I don't want to use the C API because I will be making certain library calls 
from the C program, and the syntax is much easier with native Python code than 
with C API code. 

I hope that clarifies what I'm doing. 

Jen



Dec 5, 2021, 15:19 by ba...@barrys-emacs.org:

>
>
>
>
>> On 5 Dec 2021, at 17:54, Jen Kris  wrote:
>>
>>   
>> Thanks for your comments.  
>>
>> I put the Python program on its own pthread, and call a small C program to 
>> fork-execv to call the Python program as a child process. 
>>
>
> What do you mean by putting python in it’s own pthread?
> Are you embedding python in an other program?
>
> Barry
>
>
>
>> I revised the Python program to be a multiprocessing loop using the Python 
>> multiprocessing module.  That bypasses the GIL and allows Python to run 
>> concurrently with C.  So far so good.  
>>
>> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC 
>> between Python and C.  Multiprocessing pipes are (as far as I can tell) only 
>> for commo between two Python processes.  I will have the parent thread send 
>> a signal through the pipe to the child process to exit when the parent 
>> thread is ready to exit, then call wait() to finalize the child process.  
>>
>> I will reply back when it's finished and post the code so you can see what I 
>> have done.  
>>
>> Thanks again.  
>>
>> Jen
>>
>>
>> Dec 4, 2021, 09:22 by ba...@barrys-emacs.org:
>>
>>>
>>>
 On 1 Dec 2021, at 16:01, Jen Kris < jenk...@tutanota.com > wrote:

 Thanks for your comment re blocking.  

 I removed pipes from the Python and C programs to see if it blocks without 
 them, and it does.

 It looks now like the problem is not pipes.

>>>
>>> Ok.
>>>
>>>
 I use fork() and execv() in C to run Python in a child process, but the 
 Python process blocks

>>>
>>> Use strace on the parent process to see what is happening.
>>> You will need to use the option to follow subprocesses so that you can see 
>>> what goes on in the python process.
>>>
>>> See man strace and the --follow-forks and --output-separately options.
>>> That will allow you to find the blocking system call that your code is 
>>> making.
>>>
>>>
 because fork() does not create a new thread, so the Python global 
 interpreter lock (GIL) prevents the C program from running once Python 
 starts.

>>>
>>> Not sure why you think this.
>>>
>>>
   So the solution appears to be run Python in a separate thread, which I 
 can do with pthread create.

   See "Thread State and the Global Interpreter Lock"  
 https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
   and the sections below that "Non-Python created threads" and "Cautions 
 about fork()." 

>>>
>>> I take it you mean that in the parent you think that using pthreads will 
>>> affect python after the exec() call?
>>> I does not. After exec() the process has one main thread create by the 
>>> kernel and a new address space as defined by the /usr/bin/python.
>>> The only state that in inherited from the parent are open file descriptors, 
>>> the current working directory and security state like UID, GID.
>>>
>>>
 I'm working on that today and I hope all goes well :) 

>>>
>>> You seem to be missing background information on how processes work.
>>> Maybe "Advanced Programming in the UNIX Environment" >>> would be helpful?
>>>
>>> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>>>
>>>   
>>>
>>> It's a great book and covers a wide range of Unix systems programming 
>>> topics.
>>>
>>> Have you created a small C program that just does the fork and exec of a 
>>> python program to test out your assumptions?
>>> If not I recommend that you do.
>>>
>>> Barry
>>>
>>>
>>>



 Nov 30, 2021, 11:42 by  ba...@barrys-emacs.org :

>
>
>
>> On 29 Nov 2021, at 22:31, Jen Kris <>> jenk...@tutanota.com>> > 
>> wrote:
>>
>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
>> set, but no event loop to handle it yet.  I'm putting that in now with a 
>> pipe write 

Re: Python child process in while True loop blocks parent

[Barry]

> On 5 Dec 2021, at 17:54, Jen Kris  wrote:
> 
> 
> Thanks for your comments.  
> 
> I put the Python program on its own pthread, and call a small C program to 
> fork-execv to call the Python program as a child process. 

What do you mean by putting python in it’s own pthread?
Are you embedding python in an other program?

Barry


> I revised the Python program to be a multiprocessing loop using the Python 
> multiprocessing module.  That bypasses the GIL and allows Python to run 
> concurrently with C.  So far so good.  
> 
> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC 
> between Python and C.  Multiprocessing pipes are (as far as I can tell) only 
> for commo between two Python processes.  I will have the parent thread send a 
> signal through the pipe to the child process to exit when the parent thread 
> is ready to exit, then call wait() to finalize the child process.  
> 
> I will reply back when it's finished and post the code so you can see what I 
> have done. 
> 
> Thanks again.  
> 
> Jen
> 
> 
> Dec 4, 2021, 09:22 by ba...@barrys-emacs.org:
> 
>>> On 1 Dec 2021, at 16:01, Jen Kris  wrote:
>>> 
>>> Thanks for your comment re blocking.  
>>> 
>>> I removed pipes from the Python and C programs to see if it blocks without 
>>> them, and it does.
>>> It looks now like the problem is not pipes.
>> 
>> Ok.
>> 
>> I use fork() and execv() in C to run Python in a child process, but the 
>> Python process blocks
> 
> Use strace on the parent process to see what is happening.
> You will need to use the option to follow subprocesses so that you can see 
> what goes on in the python process.
> 
> See man strace and the --follow-forks and --output-separately options.
> That will allow you to find the blocking system call that your code is making.
> 
>> because fork() does not create a new thread, so the Python global 
>> interpreter lock (GIL) prevents the C program from running once Python 
>> starts.
> 
> Not sure why you think this.
> 
>>   So the solution appears to be run Python in a separate thread, which I can 
>> do with pthread create.
>>   See "Thread State and the Global Interpreter Lock" 
>> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
>>  and the sections below that "Non-Python created threads" and "Cautions 
>> about fork()." 
> 
> I take it you mean that in the parent you think that using pthreads will 
> affect python after the exec() call?
> I does not. After exec() the process has one main thread create by the kernel 
> and a new address space as defined by the /usr/bin/python.
> The only state that in inherited from the parent are open file descriptors, 
> the current working directory and security state like UID, GID.
> 
>> I'm working on that today and I hope all goes well :) 
> 
> You seem to be missing background information on how processes work.
> Maybe "Advanced Programming in the UNIX Environment" would be helpful?
> 
> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk
>  
> 
> It's a great book and covers a wide range of Unix systems programming topics.
> 
> Have you created a small C program that just does the fork and exec of a 
> python program to test out your assumptions?
> If not I recommend that you do.
> 
> Barry
> 
> 
>> 
>> 
>> 
>> Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
>> 
>> 
>>> On 29 Nov 2021, at 22:31, Jen Kris  wrote:
>>> 
>>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
>>> set, but no event loop to handle it yet.  I'm putting that in now with a 
>>> pipe write in Python-- as Cameron pointed out that is the likely source of 
>>> blocking on C.  The pipes are opened as rdwr in Python because that's 
>>> nonblocking by default.  The child will become more complex, but not in a 
>>> way that affects polling.  And thanks for the tip about the c-string 
>>> termination. 
>>> 
>> 
>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>> You should not use O_RDWR when you only need O_RDONLY access or only 
>> O_WRONLY access.
>> 
>> You may find
>> 
>> man 2 open
>> 
>> useful to understand in detail what is behind os.open().
>> 
>> Barry
>> 
>> 
>> 
>>> 
>>> 
>>> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org:
>>> 
>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list  
>>> wrote:
>>> 
>>> I have a C program that forks to create a child process and uses execv to 
>>> call a Python program. The Python program communicates with the parent 
>>> process (in C) through a FIFO pipe monitored with epoll(). 
>>> 
>>> The Python child process is in a while True loop, which is intended to keep 
>>> it running while the parent process proceeds, and perform functions for the 
>>> C program only at intervals when the parent sends data to the child -- 
>>> similar to a daemon process. 
>>> 
>>> The C process writes to its end of the pipe and the child process reads it, 

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

Thanks for your comments.  

I put the Python program on its own pthread, and call a small C program to 
fork-execv to call the Python program as a child process.  I revised the Python 
program to be a multiprocessing loop using the Python multiprocessing module.  
That bypasses the GIL and allows Python to run concurrently with C.  So far so 
good.  

Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between 
Python and C.  Multiprocessing pipes are (as far as I can tell) only for commo 
between two Python processes.  I will have the parent thread send a signal 
through the pipe to the child process to exit when the parent thread is ready 
to exit, then call wait() to finalize the child process.  

I will reply back when it's finished and post the code so you can see what I 
have done.  

Thanks again.  

Jen


Dec 4, 2021, 09:22 by ba...@barrys-emacs.org:

>
>
>> On 1 Dec 2021, at 16:01, Jen Kris <>> jenk...@tutanota.com>> > wrote:
>>
>> Thanks for your comment re blocking.  
>>
>> I removed pipes from the Python and C programs to see if it blocks without 
>> them, and it does.
>>
>> It looks now like the problem is not pipes.
>>
>
> Ok.
>
>
>> I use fork() and execv() in C to run Python in a child process, but the 
>> Python process blocks
>>
>
> Use strace on the parent process to see what is happening.
> You will need to use the option to follow subprocesses so that you can see 
> what goes on in the python process.
>
> See man strace and the --follow-forks and --output-separately options.
> That will allow you to find the blocking system call that your code is making.
>
>
>> because fork() does not create a new thread, so the Python global 
>> interpreter lock (GIL) prevents the C program from running once Python 
>> starts.
>>
>
> Not sure why you think this.
>
>
>>   So the solution appears to be run Python in a separate thread, which I can 
>> do with pthread create.
>>
>>   See "Thread State and the Global Interpreter Lock" >> 
>> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>>
>>   and the sections below that "Non-Python created threads" and "Cautions 
>> about fork()." 
>>
>
> I take it you mean that in the parent you think that using pthreads will 
> affect python after the exec() call?
> I does not. After exec() the process has one main thread create by the kernel 
> and a new address space as defined by the /usr/bin/python.
> The only state that in inherited from the parent are open file descriptors, 
> the current working directory and security state like UID, GID.
>
>
>> I'm working on that today and I hope all goes well :) 
>>
>
> You seem to be missing background information on how processes work.
> Maybe "Advanced Programming in the UNIX Environment" > would be helpful?
>
> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>
>   
>
> It's a great book and covers a wide range of Unix systems programming topics.
>
> Have you created a small C program that just does the fork and exec of a 
> python program to test out your assumptions?
> If not I recommend that you do.
>
> Barry
>
>
>
>>
>>
>>
>> Nov 30, 2021, 11:42 by >> ba...@barrys-emacs.org>> :
>>
>>>
>>>
>>>
 On 29 Nov 2021, at 22:31, Jen Kris < jenk...@tutanota.com > wrote:

 Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
 set, but no event loop to handle it yet.  I'm putting that in now with a 
 pipe write in Python-- as Cameron pointed out that is the likely source of 
 blocking on C.  The pipes are opened as rdwr in Python because that's 
 nonblocking by default.  The child will become more complex, but not in a 
 way that affects polling.  And thanks for the tip about the c-string 
 termination. 


>>>
>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
>>> You should not use O_RDWR when you only need O_RDONLY access or only 
>>> O_WRONLY access.
>>>
>>> You may find
>>>
>>> man 2 open
>>>
>>> useful to understand in detail what is behind os.open().
>>>
>>> Barry
>>>
>>>
>>>
>>>


 Nov 29, 2021, 14:12 by  ba...@barrys-emacs.org :

>
>
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>> 
>> python-list@python.org>> > wrote:
>>
>> I have a C program that forks to create a child process and uses execv 
>> to call a Python program.  The Python program communicates with the 
>> parent process (in C) through a FIFO pipe monitored with epoll(). 
>>
>> The Python child process is in a while True loop, which is intended to 
>> keep it running while the parent process proceeds, and perform functions 
>> for the C program only at intervals when the parent sends data to the 
>> child -- similar to a daemon process. 
>>
>> The C process writes to its end of the pipe and the child process reads 
>> 

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 1 Dec 2021, at 16:01, Jen Kris  wrote:
> 
> Thanks for your comment re blocking.  
> 
> I removed pipes from the Python and C programs to see if it blocks without 
> them, and it does.
> It looks now like the problem is not pipes.

Ok.

> I use fork() and execv() in C to run Python in a child process, but the 
> Python process blocks

Use strace on the parent process to see what is happening.
You will need to use the option to follow subprocesses so that you can see what 
goes on in the python process.

See man strace and the --follow-forks and --output-separately options.
That will allow you to find the blocking system call that your code is making.

> because fork() does not create a new thread, so the Python global interpreter 
> lock (GIL) prevents the C program from running once Python starts.

Not sure why you think this.

>   So the solution appears to be run Python in a separate thread, which I can 
> do with pthread create.
>   See "Thread State and the Global Interpreter Lock" 
> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
>  
> 
>  and the sections below that "Non-Python created threads" and "Cautions about 
> fork()." 

I take it you mean that in the parent you think that using pthreads will affect 
python after the exec() call?
I does not. After exec() the process has one main thread create by the kernel 
and a new address space as defined by the /usr/bin/python.
The only state that in inherited from the parent are open file descriptors, the 
current working directory and security state like UID, GID.

> I'm working on that today and I hope all goes well :) 

You seem to be missing background information on how processes work.
Maybe "Advanced Programming in the UNIX Environment" would be helpful?

https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk
 

 

It's a great book and covers a wide range of Unix systems programming topics.

Have you created a small C program that just does the fork and exec of a python 
program to test out your assumptions?
If not I recommend that you do.

Barry


> 
> 
> 
> Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
> 
> 
>> On 29 Nov 2021, at 22:31, Jen Kris > > wrote:
>> 
>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
>> set, but no event loop to handle it yet.  I'm putting that in now with a 
>> pipe write in Python-- as Cameron pointed out that is the likely source of 
>> blocking on C.  The pipes are opened as rdwr in Python because that's 
>> nonblocking by default.  The child will become more complex, but not in a 
>> way that affects polling.  And thanks for the tip about the c-string 
>> termination. 
>> 
> 
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY 
> access.
> 
> You may find
> 
> man 2 open
> 
> useful to understand in detail what is behind os.open().
> 
> Barry
> 
> 
> 
>> 
>> 
>> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org 
>> :
>> 
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list > > wrote:
>> 
>> I have a C program that forks to create a child process and uses execv to 
>> call a Python program. The Python program communicates with the parent 
>> process (in C) through a FIFO pipe monitored with epoll(). 
>> 
>> The Python child process is in a while True loop, which is intended to keep 
>> it running while the parent process proceeds, and perform functions for the 
>> C program only at intervals when the parent sends data to the child -- 
>> similar to a daemon process. 
>> 
>> The C process writes to its end of the pipe and the child process reads it, 
>> but then the child process continues to loop, thereby blocking the parent. 
>> 
>> This is the Python code:
>> 
>> #!/usr/bin/python3
>> import os
>> import select
>> 
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>> Why open rdwr if you are only going to read the pipe?
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>> Only need to open for write.
>> 
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>> 
>> Is the only thing that the child does this:
>> 1. Read message from pr
>> 2. Process message
>> 3. Write result to pw.
>> 4. Loop from 1
>> 
>> If so as Cameron said you do not need to worry about the poll.
>> Do you plan for the child to become more complex?
>> 
>> while True:
>> 
>> events = ep.poll(timeout=2.5, maxevents=-1)
>> #events = ep.poll(timeout=None, maxevents=-1)
>> 
>> print("child is looping")
>> 
>> for fileno, event in events:
>> print("Python fileno")
>> 

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

Thanks for your comment re blocking.  

I removed pipes from the Python and C programs to see if it blocks without 
them, and it does.  It looks now like the problem is not pipes.  I use fork() 
and execv() in C to run Python in a child process, but the Python process 
blocks because fork() does not create a new thread, so the Python global 
interpreter lock (GIL) prevents the C program from running once Python starts.  
So the solution appears to be run Python in a separate thread, which I can do 
with pthread create.  See "Thread State and the Global Interpreter Lock" 
https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
 and the sections below that "Non-Python created threads" and "Cautions about 
fork()." 

I'm working on that today and I hope all goes well :) 



Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:

>
>
>
>> On 29 Nov 2021, at 22:31, Jen Kris <>> jenk...@tutanota.com>> > wrote:
>>
>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
>> set, but no event loop to handle it yet.  I'm putting that in now with a 
>> pipe write in Python-- as Cameron pointed out that is the likely source of 
>> blocking on C.  The pipes are opened as rdwr in Python because that's 
>> nonblocking by default.  The child will become more complex, but not in a 
>> way that affects polling.  And thanks for the tip about the c-string 
>> termination. 
>>
>>
>
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY 
> access.
>
> You may find
>
> man 2 open
>
> useful to understand in detail what is behind os.open().
>
> Barry
>
>
>
>
>>
>>
>> Nov 29, 2021, 14:12 by >> ba...@barrys-emacs.org>> :
>>
>>>
>>>
 On 29 Nov 2021, at 20:36, Jen Kris via Python-list < 
 python-list@python.org > wrote:

 I have a C program that forks to create a child process and uses execv to 
 call a Python program.  The Python program communicates with the parent 
 process (in C) through a FIFO pipe monitored with epoll(). 

 The Python child process is in a while True loop, which is intended to 
 keep it running while the parent process proceeds, and perform functions 
 for the C program only at intervals when the parent sends data to the 
 child -- similar to a daemon process. 

 The C process writes to its end of the pipe and the child process reads 
 it, but then the child process continues to loop, thereby blocking the 
 parent. 

 This is the Python code:

 #!/usr/bin/python3
 import os
 import select

 #Open the named pipes
 pr = os.open('/tmp/Pipe_01', os.O_RDWR)

>>> Why open rdwr if you are only going to read the pipe?
>>>
 pw = os.open('/tmp/Pipe_02', os.O_RDWR)

>>> Only need to open for write.
>>>

 ep = select.epoll(-1)
 ep.register(pr, select.EPOLLIN)

>>>
>>> Is the only thing that the child does this:
>>> 1. Read message from pr
>>> 2. Process message
>>> 3. Write result to pw.
>>> 4. Loop from 1
>>>
>>> If so as Cameron said you do not need to worry about the poll.
>>> Do you plan for the child to become more complex?
>>>

 while True:

 events = ep.poll(timeout=2.5, maxevents=-1)
 #events = ep.poll(timeout=None, maxevents=-1)

 print("child is looping")

 for fileno, event in events:
 print("Python fileno")
 print(fileno)
 print("Python event")
 print(event)
 v = os.read(pr,64)
 print("Pipe value")
 print(v)

 The child process correctly receives the signal from ep.poll and correctly 
 reads the data in the pipe, but then it continues looping.  For example, 
 when I put in a timeout:

 child is looping
 Python fileno
 4
 Python event
 1
 Pipe value
 b'10\x00'

>>> The C code does not need to write a 0 bytes at the end.
>>> I assume the 0 is from the end of a C string.
>>> UDS messages have a length.
>>> In the C just write 2 byes in the case.
>>>
>>> Barry
>>>
 child is looping
 child is looping

 That suggests that a while True loop is not the right thing to do in this 
 case.  My question is, what type of process loop is best for this 
 situation?  The multiprocessing, asyncio and subprocess libraries are very 
 extensive, and it would help if someone could suggest the best alternative 
 for what I am doing here. 

 Thanks very much for any ideas. 


 -- 
 https://mail.python.org/mailman/listinfo/python-list

>>
>>

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 29 Nov 2021, at 22:31, Jen Kris  wrote:
> 
> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, 
> but no event loop to handle it yet.  I'm putting that in now with a pipe 
> write in Python-- as Cameron pointed out that is the likely source of 
> blocking on C.  The pipes are opened as rdwr in Python because that's 
> nonblocking by default.  The child will become more complex, but not in a way 
> that affects polling.  And thanks for the tip about the c-string termination. 
> 

flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY 
access.

You may find

man 2 open

useful to understand in detail what is behind os.open().

Barry



> 
> 
> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org:
> 
> On 29 Nov 2021, at 20:36, Jen Kris via Python-list  
> wrote:
> 
> I have a C program that forks to create a child process and uses execv to 
> call a Python program. The Python program communicates with the parent 
> process (in C) through a FIFO pipe monitored with epoll(). 
> 
> The Python child process is in a while True loop, which is intended to keep 
> it running while the parent process proceeds, and perform functions for the C 
> program only at intervals when the parent sends data to the child -- similar 
> to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, 
> but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> Why open rdwr if you are only going to read the pipe?
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
> Only need to open for write.
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
> 
> Is the only thing that the child does this:
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
> 
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
> 
> while True:
> 
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
> 
> print("child is looping")
> 
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly 
> reads the data in the pipe, but then it continues looping. For example, when 
> I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
> The C code does not need to write a 0 bytes at the end.
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
> 
> Barry
> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this 
> case. My question is, what type of process loop is best for this situation? 
> The multiprocessing, asyncio and subprocess libraries are very extensive, and 
> it would help if someone could suggest the best alternative for what I am 
> doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Jen Kris via Python-list]

Thanks to you and Cameron for your replies.  The C side has an epoll_ctl set, 
but no event loop to handle it yet.  I'm putting that in now with a pipe write 
in Python-- as Cameron pointed out that is the likely source of blocking on C.  
The pipes are opened as rdwr in Python because that's nonblocking by default.  
The child will become more complex, but not in a way that affects polling.  And 
thanks for the tip about the c-string termination. 



Nov 29, 2021, 14:12 by ba...@barrys-emacs.org:

>
>
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list  
>> wrote:
>>
>> I have a C program that forks to create a child process and uses execv to 
>> call a Python program.  The Python program communicates with the parent 
>> process (in C) through a FIFO pipe monitored with epoll(). 
>>
>> The Python child process is in a while True loop, which is intended to keep 
>> it running while the parent process proceeds, and perform functions for the 
>> C program only at intervals when the parent sends data to the child -- 
>> similar to a daemon process. 
>>
>> The C process writes to its end of the pipe and the child process reads it, 
>> but then the child process continues to loop, thereby blocking the parent. 
>>
>> This is the Python code:
>>
>> #!/usr/bin/python3
>> import os
>> import select
>>
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>>
> Why open rdwr if you are only going to read the pipe?
>
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>>
> Only need to open for write.
>
>>
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>>
>
> Is the only thing that the child does this:
> 1. Read message from pr
> 2. Process message
> 3. Write result to pw.
> 4. Loop from 1
>
> If so as Cameron said you do not need to worry about the poll.
> Do you plan for the child to become more complex?
>
>>
>> while True:
>>
>>  events = ep.poll(timeout=2.5, maxevents=-1)
>>  #events = ep.poll(timeout=None, maxevents=-1)
>>
>>  print("child is looping")
>>
>>  for fileno, event in events:
>>  print("Python fileno")
>>  print(fileno)
>>  print("Python event")
>>  print(event)
>>  v = os.read(pr,64)
>>  print("Pipe value")
>>  print(v)
>>
>> The child process correctly receives the signal from ep.poll and correctly 
>> reads the data in the pipe, but then it continues looping.  For example, 
>> when I put in a timeout:
>>
>> child is looping
>> Python fileno
>> 4
>> Python event
>> 1
>> Pipe value
>> b'10\x00'
>>
> The C code does not need to write a 0 bytes at the end.
> I assume the 0 is from the end of a C string.
> UDS messages have a length.
> In the C just write 2 byes in the case.
>
> Barry
>
>> child is looping
>> child is looping
>>
>> That suggests that a while True loop is not the right thing to do in this 
>> case.  My question is, what type of process loop is best for this situation? 
>>  The multiprocessing, asyncio and subprocess libraries are very extensive, 
>> and it would help if someone could suggest the best alternative for what I 
>> am doing here. 
>>
>> Thanks very much for any ideas. 
>>
>>
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>>

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry]

> On 29 Nov 2021, at 20:36, Jen Kris via Python-list  
> wrote:
> 
> I have a C program that forks to create a child process and uses execv to 
> call a Python program.  The Python program communicates with the parent 
> process (in C) through a FIFO pipe monitored with epoll().  
> 
> The Python child process is in a while True loop, which is intended to keep 
> it running while the parent process proceeds, and perform functions for the C 
> program only at intervals when the parent sends data to the child -- similar 
> to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, 
> but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)

You will need to set the fd’s to non blocking on parent and child.
Otherwise the parent will block on its write until the child reads the message.

Barry
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)
> 
> while True:
> 
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
> 
> print("child is looping")
> 
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly 
> reads the data in the pipe, but then it continues looping.  For example, when 
> I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this 
> case.  My question is, what type of process loop is best for this situation?  
> The multiprocessing, asyncio and subprocess libraries are very extensive, and 
> it would help if someone could suggest the best alternative for what I am 
> doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Barry]

> On 29 Nov 2021, at 20:36, Jen Kris via Python-list  
> wrote:
> 
> I have a C program that forks to create a child process and uses execv to 
> call a Python program.  The Python program communicates with the parent 
> process (in C) through a FIFO pipe monitored with epoll().  
> 
> The Python child process is in a while True loop, which is intended to keep 
> it running while the parent process proceeds, and perform functions for the C 
> program only at intervals when the parent sends data to the child -- similar 
> to a daemon process. 
> 
> The C process writes to its end of the pipe and the child process reads it, 
> but then the child process continues to loop, thereby blocking the parent. 
> 
> This is the Python code:
> 
> #!/usr/bin/python3
> import os
> import select
> 
> #Open the named pipes
> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
Why open rdwr if you are only going to read the pipe?
> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
Only need to open for write.
> 
> ep = select.epoll(-1)
> ep.register(pr, select.EPOLLIN)

Is the only thing that the child does this:
1. Read message from pr
2. Process message
3. Write result to pw.
4. Loop from 1

If so as Cameron said you do not need to worry about the poll.
Do you plan for the child to become more complex?

> 
> while True:
> 
> events = ep.poll(timeout=2.5, maxevents=-1)
> #events = ep.poll(timeout=None, maxevents=-1)
> 
> print("child is looping")
> 
> for fileno, event in events:
> print("Python fileno")
> print(fileno)
> print("Python event")
> print(event)
> v = os.read(pr,64)
> print("Pipe value")
> print(v)
> 
> The child process correctly receives the signal from ep.poll and correctly 
> reads the data in the pipe, but then it continues looping.  For example, when 
> I put in a timeout:
> 
> child is looping
> Python fileno
> 4
> Python event
> 1
> Pipe value
> b'10\x00'
The C code does not need to write a 0 bytes at the end.
I assume the 0 is from the end of a C string.
UDS messages have a length.
In the C just write 2 byes in the case.

Barry

> child is looping
> child is looping
> 
> That suggests that a while True loop is not the right thing to do in this 
> case.  My question is, what type of process loop is best for this situation?  
> The multiprocessing, asyncio and subprocess libraries are very extensive, and 
> it would help if someone could suggest the best alternative for what I am 
> doing here. 
> 
> Thanks very much for any ideas. 
> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python child process in while True loop blocks parent

[Cameron Simpson]

On 29Nov2021 21:34, Jen Kris  wrote:
>I have a C program that forks to create a child process and uses execv to call 
>a Python program.  The Python program communicates with the parent process (in 
>C) through a FIFO pipe monitored with epoll(). 
>
>The Python child process is in a while True loop, which is intended to keep it 
>running while the parent process proceeds, and perform functions for the C 
>program only at intervals when the parent sends data to the child -- similar 
>to a daemon process. 
>
>The C process writes to its end of the pipe and the child process reads it, 
>but then the child process continues to loop, thereby blocking the parent. 

It seems to me that the child Python process never writes anything back 
to the parent. If the parent is waiting for some response, of course it 
will be blocked.

Personally I wouldn't be using epoll at all. I'd just read data from pr, 
act on it, and write back to pw. That way the child blocks waiting for 
the parent istead of polling. You only want epoll if you're either 
polling something _while_ you do other work, or monitoring more than one 
thing. A plain read-from-one-pipe, work, write-back-to-the-other does 
not need it.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list