Re: Python child process in while True loop blocks parent

[Barry Scott]

> On 1 Dec 2021, at 16:01, Jen Kris  wrote:
> 
> Thanks for your comment re blocking.  
> 
> I removed pipes from the Python and C programs to see if it blocks without 
> them, and it does.
> It looks now like the problem is not pipes.

Ok.

> I use fork() and execv() in C to run Python in a child process, but the 
> Python process blocks

Use strace on the parent process to see what is happening.
You will need to use the option to follow subprocesses so that you can see what 
goes on in the python process.

See man strace and the --follow-forks and --output-separately options.
That will allow you to find the blocking system call that your code is making.

> because fork() does not create a new thread, so the Python global interpreter 
> lock (GIL) prevents the C program from running once Python starts.

Not sure why you think this.

>   So the solution appears to be run Python in a separate thread, which I can 
> do with pthread create.
>   See "Thread State and the Global Interpreter Lock" 
> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
>  
> 
>  and the sections below that "Non-Python created threads" and "Cautions about 
> fork()." 

I take it you mean that in the parent you think that using pthreads will affect 
python after the exec() call?
I does not. After exec() the process has one main thread create by the kernel 
and a new address space as defined by the /usr/bin/python.
The only state that in inherited from the parent are open file descriptors, the 
current working directory and security state like UID, GID.

> I'm working on that today and I hope all goes well :) 

You seem to be missing background information on how processes work.
Maybe "Advanced Programming in the UNIX Environment" would be helpful?

https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk
 

 

It's a great book and covers a wide range of Unix systems programming topics.

Have you created a small C program that just does the fork and exec of a python 
program to test out your assumptions?
If not I recommend that you do.

Barry


> 
> 
> 
> Nov 30, 2021, 11:42 by ba...@barrys-emacs.org:
> 
> 
>> On 29 Nov 2021, at 22:31, Jen Kris > > wrote:
>> 
>> Thanks to you and Cameron for your replies.  The C side has an epoll_ctl 
>> set, but no event loop to handle it yet.  I'm putting that in now with a 
>> pipe write in Python-- as Cameron pointed out that is the likely source of 
>> blocking on C.  The pipes are opened as rdwr in Python because that's 
>> nonblocking by default.  The child will become more complex, but not in a 
>> way that affects polling.  And thanks for the tip about the c-string 
>> termination. 
>> 
> 
> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK.
> You should not use O_RDWR when you only need O_RDONLY access or only O_WRONLY 
> access.
> 
> You may find
> 
> man 2 open
> 
> useful to understand in detail what is behind os.open().
> 
> Barry
> 
> 
> 
>> 
>> 
>> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org 
>> :
>> 
>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list > > wrote:
>> 
>> I have a C program that forks to create a child process and uses execv to 
>> call a Python program. The Python program communicates with the parent 
>> process (in C) through a FIFO pipe monitored with epoll(). 
>> 
>> The Python child process is in a while True loop, which is intended to keep 
>> it running while the parent process proceeds, and perform functions for the 
>> C program only at intervals when the parent sends data to the child -- 
>> similar to a daemon process. 
>> 
>> The C process writes to its end of the pipe and the child process reads it, 
>> but then the child process continues to loop, thereby blocking the parent. 
>> 
>> This is the Python code:
>> 
>> #!/usr/bin/python3
>> import os
>> import select
>> 
>> #Open the named pipes
>> pr = os.open('/tmp/Pipe_01', os.O_RDWR)
>> Why open rdwr if you are only going to read the pipe?
>> pw = os.open('/tmp/Pipe_02', os.O_RDWR)
>> Only need to open for write.
>> 
>> ep = select.epoll(-1)
>> ep.register(pr, select.EPOLLIN)
>> 
>> Is the only thing that the child does this:
>> 1. Read message from pr
>> 2. Process message
>> 3. Write result to pw.
>> 4. Loop from 1
>> 
>> If so as Cameron said you do not need to worry about the poll.
>> Do you plan for the child to become more complex?
>> 
>> while True:
>> 
>> events = ep.poll(timeout=2.5, maxevents=-1)
>> #events = ep.poll(timeout=None, maxevents=-1)
>> 
>> print("child is looping")
>> 
>> for fileno, event in events:
>> print("Python fileno")
>> print(fil

Re: TypeError: 'bytes' object is not callable error while trying to converting to bytes.

[Wasia Maya]

You have assigned a bytes value to the name bytes:

>>> bytes([10, 20, 30, 40])
b'\n\x14\x1e('
>>> bytes = bytes([10, 20, 30, 40])
>>> bytes([10, 20, 30, 40])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'bytes' object is not callable
bytes is now bound to the value b'\n\x14\x1e(', which is not callable. This 
global is shadowing the built-in. Delete it:

del bytes
-- 
https://mail.python.org/mailman/listinfo/python-list