Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Wednesday, 22 February 2017 18:54:34 GMT Ralph Corderoy wrote: > There's at least three other experienced Python programmers on the list > so they can pipe up if they disagree, but it's idiomatic to test for > `truthness' rather than absolute values. > > Python defines truth very consistently. (Unlike PHP.) > https://docs.python.org/2/library/stdtypes.html#truth-value-testing > > So it's written `if foo' rather than `if foo == True', and `if not foo' > for the opposite. After all, your `foo == True' is itself an expression > that is True or False so needs testing again: `foo == True == True'. > And so's that... That's what I intended to do, once I had defined the var as boolean. > It follows that the choice of identifier makes it read well with the > right meaning. `if ringing_changes' perhaps, I don't know the subject. The var is intended to mean 'change rings control'. I have added a comment in the code to that effect. > It doesn't have to be a Boolean to do this. If `child' is an int that's > either 0 when there's no child yet, or its PID when there is, then the > test is `if child' rather than `if child != 0'. It reads more simply > too. Now that I wasn't sure of, so I was sticking to making use of the boolean form. I think that I still will in the interests of clarity for future 'makers' who might take over the maintenance of this system. I feel that the meaning is much more obvious if the var is declared as a boolean than as a simple int, unless you have a background in CS. Bearing in mind the situation here, my view is that if I'm not sure of the meaning of something in the code, then there will be others who will have similar problems; particularly since they won't necessarily know what I'm trying to do. Of course, I appreciate that this code is not Python 101 anymore; as mentioned earlier, my 'Learning Python' book (1540pp) doesn't even cover multi-processing and multi-threading, so I guess that this point will be pretty moot if they have a similar background to me :-) I'm still working my way through the rest of your message and the earlier one, but I have other stuff to do elsewhere today, so I may not get round to trying out your suggestions until tomorrow. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Will,
> "Small integers", I believe it's the range -5 through 256 inclusive,
> also have this "Highlander property" ("there can be only one").
Yes, [-5, 256].
https://github.com/python/cpython/blob/master/Objects/longobject.c#L18
> This is as an optimisation. Python reference counts everything and
> the churn from constantly allocating and freeing small integers poses
> quite an overhead.
They refer to it as `interning' IIRC. As soon as a garbage collector is
involved the focus moves to stop producing garbage to lessen the work.
:-) Go is steadily improving here at the moment with its escape
analysis to know when something can be stack allocated or whether it
must be on the heap for the GC to track.
> Hmmm I wonder what else works in this way? So far we've identified --
> None, int(-5) through to int(256), and I think True and False work
> this way too.
Yes, they are just simple variables.
https://github.com/python/cpython/blob/master/Objects/boolobject.c#L174
Anything that's a frequently used and immutable is a contender, e.g. (),
the empty tuple.
https://github.com/python/cpython/blob/master/Objects/tupleobject.c#L85
Strings, and bytes now with Python 3, are shared too, e.g. 'A' will be
found in characters[65] in Objects/bytesobject.c, but that's not
pre-built, probably to avoid start-up time suffering even more.
What's pre-allocated has changed over time as they've tried to speed up
the byte-code interpreter. I don't know of a current list.
Cheers, Ralph.
--
Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00
Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/
New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING
Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
>x refers to one int(4200), and the constant I typed refers to another,
>distinct int(4200) object.
>
>>>> id(x), id(4200)
>(93918839372072, 93918839372024)
> ...
>
>There's only one None.
"Small integers", I believe it's the range -5 through 256 inclusive, also
have this "Highlander property" ("there can be only one").
This is as an optimisation. Python reference counts everything and the churn
from constantly allocating and freeing small integers poses quite an
overhead.
Hmmm I wonder what else works in this way? So far we've identified -- None,
int(-5) through to int(256), and I think True and False work this way too.
Will
_
William R Sowerbutts w...@sowerbutts.com
"Carpe post meridiem" http://sowerbutts.com
main(){char*s=">#=0> ^#X@#@^7=",c=0,m;for(;c<15;c++)for
(m=-1;m<7;putchar(m++/6%3/2?10:s[c]-31&1<
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > > if c_player.returncode == None or chr_control == 0: > > I just noticed a bug in that bit of code. The var chr_control was > originally an int, but it made sense to change it to a boolean, so the > test in the final if should be checking for False, not 0. There's at least three other experienced Python programmers on the list so they can pipe up if they disagree, but it's idiomatic to test for `truthness' rather than absolute values. Python defines truth very consistently. (Unlike PHP.) https://docs.python.org/2/library/stdtypes.html#truth-value-testing So it's written `if foo' rather than `if foo == True', and `if not foo' for the opposite. After all, your `foo == True' is itself an expression that is True or False so needs testing again: `foo == True == True'. And so's that... It follows that the choice of identifier makes it read well with the right meaning. `if ringing_changes' perhaps, I don't know the subject. It doesn't have to be a Boolean to do this. If `child' is an int that's either 0 when there's no child yet, or its PID when there is, then the test is `if child' rather than `if child != 0'. It reads more simply too. An empty string is false, so's an empty list. It's well designed. So you might run through, and destroy, a list with while workq: w = workq.pop() toil(w) Not `while len(workq) != 0:'. Related, there's only one None. It's a built-in constant. https://docs.python.org/2/library/constants.html?highlight=none#None To test if two objects are identical the `is' operator is used rather than `==', and `is not' rather than `!='. >>> x = 42 >>> y = 6 * 7 >>> x is y True x and y above are two identifiers that refer to the same object; an int(42). That's deceptive though... >>> x *= 100 >>> x is 4200 False >>> x 4200 >>> x == 4200 True x refers to one int(4200), and the constant I typed refers to another, distinct int(4200) object. >>> id(x), id(4200) (93918839372072, 93918839372024) id() returns an opaque number for the object referred to; same number means same object. Thus `is' just needs to compare the two numbers. The number is not meant to have meaning inferred, but it's actually the memory address of the object's data; that's an easy way to ensure they're distinct for different objects and the same for the one object. There's only one None. >>> x = None >>> x is None True >>> id(x), id(None) (140441554349504, 140441554349504) Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry,
> They seem to get variously called Procedures and Functions in my
> 'Learning Python' Book (which doesn't cover multiprocessing). I mean
> a piece of code defined within a 'def' block.
The book's wrong. :-) In Python, they're functions, lower F. Even a
function without an explicit return statement still returns None whether
the caller wants it or not.
>>> import dis
>>> def f():
... pass
...
>>> dis.dis(f)
2 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
>>>
Just as the subprocess module defines a Popen class of object, so some
Pi threading library might decide to create a Procedure one, that's why
I asked.
> Well the documentation for RPi.GPIO seems to consist only of the
> Tutorials on Alex Eames' website
I was looking at
https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/
Don't know if that's the same thing. And then
https://sourceforge.net/p/raspberry-gpio-python/code/ci/default/tree/source/event_gpio.c#l36
> I'm not sure that I need 'closure' in the called function, just the
> ability to run a few lines of code and then exit.
It just allows the possibility of re-using the callback for similar
functionality and having it use the passed-in closure to tailor its
work. Less code, a bit more data; tends to be the best direction.
> def change_rings(): # Play a file of change rings
> global chr_subdir
> global chr_control
>
> c_file = 'Wimborne_Minster_StedmanCinques_16-10-2016.mp3' # Change rings.
> c_path = os.path.join(chr_subdir, c_file)
> blackhole = open(os.devnull, 'w')
I'd open this once as a global variable. Python will garbage collect
and close the file descriptors over time as all the references die out,
but that's assuming they do die out. If they don't, the process will
use up ever more file descriptors and there's a couple of limits it
could reach that would make open() fail.
> print 'Playing Changes - ', c_file, 'to the Tower'
> c_player = subprocess.Popen(['mpg321', '-q', '-m', '-a', 'hw:0,0',
> c_path],
> stdin=subprocess.PIPE, stdout=blackhole, stderr=blackhole)
>
> while chr_control:
> c_player.poll() # check to see if terminated
> if c_player.returncode != None:
> break
> time.sleep(0.1)
Does this busy loop really need to exist? Instead, change_rings() could
just kick off the subprocess, stash the `c_player' somewhere, and then
return. Callbacks often do a small slice of work and then return,
otherwise the `main loop' can't continue, as you've found.
The callback for the switch to nobble the bells can then terminate(),
since it's OK to do that whether mpg321 has finished or not as long as
the zombie child hasn't been reaped with poll().
Here's me killing a still-playing mpg321.
>>> import os, time, subprocess
>>> p = subprocess.Popen(('sleep', '42h')) # Pretend it's mpg321.
>>> os.system('ps lf')
F UID PID PPID PRI NIVSZ RSS WCHAN STAT TTYTIME COMMAND
0 1000 603 570 20 0 34900 4144 wait Ss pts/2 0:03 -bash
0 1000 20845 603 20 0 45932 7868 wait S+ pts/2 0:00 \_
python2
0 1000 20852 20845 20 0 6112 680 hrtime S+ pts/2 0:00
\_ sleep 42h
0 1000 20853 20845 20 0 32804 3244 - R+ pts/2 0:00
\_ ps lf
0
>>> p.terminate(),
(None,)
>>> os.system('ps lf')
F UID PID PPID PRI NIVSZ RSS WCHAN STAT TTYTIME COMMAND
0 1000 603 570 20 0 34900 4144 wait Ss pts/2 0:03 -bash
0 1000 20845 603 20 0 45932 7868 wait S+ pts/2 0:00 \_
python2
0 1000 20852 20845 20 0 0 0 - Z+ pts/2 0:00
\_ [sleep]
0 1000 20854 20845 20 0 32804 3212 - R+ pts/2 0:00
\_ ps lf
0
PID 20852 is now a zombie process; `Z' in stat. It still has an entry
in the process table, that's what ps(1) is showing, but most of the
resources it was using, memory, file descriptors, etc., have been freed
by the kernel for re-use. What remains are things the parent, PPID,
might be interested in like its exit(2) value indicating success, or
not.
I can keep signal(2)ing a zombie process.
>>> p.terminate(),
(None,)
>>> p.terminate(),
(None,)
Now I "reap" the zombie child process with poll().
>>> p.poll()
-15
That -15 is negative indicates death by signal. 15 is SIGTERM's number;
see signal(7). You might want to check poll()'s return value in case
mpg321 exits due to an error as you're already discarding stdout and
stderr.
The zombie is no longer in the kernel's process table.
>>> os.system('ps lf')
F UID PID PPID PRI NIVSZ RSS WCHAN STAT TTYTIME COMMAND
0 1000 603 570 20 0 34900 4144 wait Ss pts/2 0:03 -bash
0 1000 20845 603 20 0 45932 7868 wait S+ pts/2
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Wednesday, 22 February 2017 15:10:21 GMT Terry Coles wrote: > while chr_control: > c_player.poll() # check to see if terminated > if c_player.returncode != None: > break > time.sleep(0.1) > > if c_player.returncode == None or chr_control == 0: # kill the > player if it's still running > c_player.terminate() > c_player.wait() I just noticed a bug in that bit of code. The var chr_control was originally an int, but it made sense to change it to a boolean, so the test in the final if should be checking for False, not 0. Of course, I haven't been able to test that bit of code yet, (I needed the switches to be working), so there could well be other bugs :-) -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, Seeking clarification... Python 2 or 3? > GPIO.add_event_detect(20, GPIO.RISING, callback=extended_hours_triggered, > bouncetime=500) > GPIO.add_event_detect(21, GPIO.RISING, callback=extended_hours_disabled, > bouncetime=500) ... > The idea is that when the user presses a switch down, the associated > callback enables the function by setting a global variable and calling > the appropriate Procedure. Is `Procedure' here just a general term for a Python function, or a particular thing from some threading, etc., package that you're using? I see that GPIO maintains a separate thread for calling the registered callbacks sequentially. And there is no `closure' argument possible for a callback, though it does get passed the pin by the looks of it, e.g. 20. > Earlier discussions have allowed me to ensure that these functions are > in separate subprocesses and code within the Procedure that started > the subprocess monitors the appropriate global so that the process can > be terminated when necessary. `subprocesses' using the Python subprocess library? How is it started? What's the `monitoring' code in the Procedure? Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
[Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi, My bells software for the Wimborne Model Town is getting close to fruition. With lots of help from you guys, I have been able to demonstrate the chimes ringing whilst at the same time playing music in another area of the Model. Pretty much the last thing left to do is the User Interface. This consists of a row of four SPDT (centre biased off) switches on the front of the control box which contains the Pi and other hardware; such as the audio amplifiers and MOSFETs to drive the Quarter Jack solenoids. Each of these four switches is connected to a couple of GPIO pins configured as inputs. Using the information provided here: http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio[1] I have been able to set up eight event detectors of the form: GPIO.add_event_detect(20, GPIO.RISING, callback=extended_hours_triggered, bouncetime=500) GPIO.add_event_detect(21, GPIO.RISING, callback=extended_hours_disabled, bouncetime=500) and five more GPIO.add_event_detect(27, GPIO.RISING, callback=wedding_sequence_disabled, bouncetime=500) The idea is that when the user presses a switch down, the associated callback enables the function by setting a global variable and calling the appropriate Procedure. Pressing the switch up unsets the var. These functions do such things as provide an MP3 player and fire off a wedding sequence of music and bells. Earlier discussions have allowed me to ensure that these functions are in separate subprocesses and code within the Procedure that started the subprocess monitors the appropriate global so that the process can be terminated when necessary. The above events work perfectly until the associated function has started, then switch toggling has no effect until the procedure has terminated. Clearly the Procedures need to be themselves running in separate threads. Having looked at various sources, it would seem that multiprocessing rather than multithreading might be more appropriate, but I'm finding the documentation hard going and none of the examples that I've found seem to lend themselves to what I want to do. Can anyone help? I think that all that's needed is to get the Procedure defined in the callback to fire even when the MP3 player (say) is playing. I'm fairly sure that the events are happening, it's what happens afterwards that seems to be the problem. All contributions gratefully received, as always. -- Terry Coles [1] http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
