Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Saturday, 25 February 2017 15:20:58 GMT Ralph Corderoy wrote:
> > I'm still struggling a bit (well a lot actually ;-) ). The problem is
> > still how to get all the things done that I need to get done.
>
> If you keep cycling back over the old emails as you learn more things,
> they may make more sense. :-)
Oh they do. It's just that my 66 year-old brain doesn't multi-task very well
(it never did much) and my memory is not getting any better. Add to that a
very good long-term memory, a very poor short-term memory and a not too
brilliant ability to sort out logic, you'll perhaps understand why I
specialised in systems engineering and not programming :-)
> > it would probably still not help because the new code needs to run in
> > a separate thread, not just the call to it.
>
> Don't know what `new code' is here.
I was meaning my functions to control the players. ('New' was probably not
the right description.)
> No, GPIO provides handle_events(); it was under the `Library code, used
> by everyone' comment. You write a bunch of callbacks, register them
> all, and forget about it in that main thread, getting on with your timed
> chime playing.
>
> You'll end up with this.
>
> python-process
> │
> ├── python-main-thread
> │ │
> │ └─/─ quarters-process
> │
> └── python-gpio-thread
> │
> ├─/─ bell-player-mpg321-process
> ├─/─ playlist-mpg321-process
> └─/─ wedding-sequence-mpg321-process
Light is beginning to dawn. What you are saying is that everything except the
code to time and ring the quarters (and the hours) will be registered as a
callback; one callback for each thing that I want to do (eg play the change
rings, stop them, setup the MP3 Playlist and play it, stop the player etc.
> A lot of your callbacks will use the subprocess module to run mpg321.
> These will again be new processes, just as quarters-process was. Some
> will .call() it, waiting for it to finish, and others will .Popen()
> where you Python callback code continues to run without waiting for,
> say, bell-player-mpg321-process, to finish. You callback will finish,
> having kicked off mpg321, and you'll return to the C code in
> python-gpio-thread that called you. That will release Python's GIL and
> your Python code in python-main-thread is free to continue if it needs.
So the key thing that I need to keep in mind is that I cannot fire off a new
block of code until the old one is finished and a new event from GPIO triggers
a new callback. This is going to take a bit of thinking about.
Unfortunately, I'll have to sleep on it, because my daughter came to stay for
the weekend this morning, so I'll not be able to dedicate as much time to it
as I'd have liked.
I'll report back tomorrow with an update on what's working, (or not).
--
Terry Coles
--
Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00
Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/
New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING
Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > I'm still struggling a bit (well a lot actually ;-) ). The problem is > still how to get all the things done that I need to get done. If you keep cycling back over the old emails as you learn more things, they may make more sense. :-) > There are a number of references in the code to 'callbacks' (as > opposed to 'callback'), so I started off thinking that I could add > multiple callbacks to the GPIO.add_event_detect() statement by listing > them as you indicated for your example code in your later message that > I reference below. IIRC you can't do that. The C code refers to callbacks for the same reason my example Python callback example had callbacks = collections.defaultdict(list) Because it has to track all the callbacks for all the different events. It doesn't mean the interface for registering callbacks necessarily allows multiple ones to be given at once. > it would probably still not help because the new code needs to run in > a separate thread, not just the call to it. Don't know what `new code' is here. > So, I'm think that what you were leading up in your example to is some > scheme whereby I set up and create an event handler similar to the one > you created in this message. This code is run before the > GPIO.add_event_detect() statements and the callback in each of those > statements references the appropriate function. These functions are > where I put my code to change the state of the global control flags, > kill the mpg321 player and play the message to the user. No, GPIO provides handle_events(); it was under the `Library code, used by everyone' comment. You write a bunch of callbacks, register them all, and forget about it in that main thread, getting on with your timed chime playing. You'll end up with this. python-process │ ├── python-main-thread │ │ │ └─/─ quarters-process │ └── python-gpio-thread │ ├─/─ bell-player-mpg321-process ├─/─ playlist-mpg321-process └─/─ wedding-sequence-mpg321-process You run your Python script. That's `python-process' with its own PID, and it has its python-main-thread running your code. As it stands now, you're already using a library that tells you when it's on the quarter and you use the subprocess module to run mpg321. That runs in quarters-process which is a separate process with its own PID. I've marked the relationship line with `/' to indicate the process boundary. On registering callbacks with GPIO, it will create python-gpio-thread, still in the same python-process. When a GPIO event is seen, it will block Python in python-main-thread from running and call the matching Python callback function that you've provided. Your function will be running in python-gpio-thread, so still in python-process. A lot of your callbacks will use the subprocess module to run mpg321. These will again be new processes, just as quarters-process was. Some will .call() it, waiting for it to finish, and others will .Popen() where you Python callback code continues to run without waiting for, say, bell-player-mpg321-process, to finish. You callback will finish, having kicked off mpg321, and you'll return to the C code in python-gpio-thread that called you. That will release Python's GIL and your Python code in python-main-thread is free to continue if it needs. Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Saturday, 25 February 2017 14:56:53 GMT Ralph Corderoy wrote: > Although your code now has two threads, your first initial main one, and > the one GPIO creates once you start wanting it to watch for interrupt > events, there's still Python's GIL, Global Interpreter Lock, and the > second thread's C code has to aquire it before it can run the Python > code of your callback function, and then release it. Once released, > Python code in your main thread, that was held up waiting for the lock, > will continue. > > Does that explain the `isn't busy' behaviour? Yes. And I kind of understood that. I have read about the GIL without necessarily understanding it other than that it limits how many threads Python can manage at once. It's one reason why I thought multiprocessing might be needed, because I understand that this gets over the problem with the GIL. > Your callbacks will need a `gpio' or `channel' parameter else I'd expect > an error when GPIO tries to call them. I never tried it without one; if I copy code then I don't change anything that doesn't look as if it needs changing :-) -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > > but I'd be surprised if that works because foo's expecting no > > arguments, but one, the `gpio', is going to be passed in. > > Oh OK. The example on Alex Eames website uses (channel). I wasn't > really sure what that was for either. It works OK as long as the > program isn't busy elsewhere. Although your code now has two threads, your first initial main one, and the one GPIO creates once you start wanting it to watch for interrupt events, there's still Python's GIL, Global Interpreter Lock, and the second thread's C code has to aquire it before it can run the Python code of your callback function, and then release it. Once released, Python code in your main thread, that was held up waiting for the lock, will continue. Does that explain the `isn't busy' behaviour? Your callbacks will need a `gpio' or `channel' parameter else I'd expect an error when GPIO tries to call them. >>> def foo(): ... pass ... >>> callback = foo >>> callback(42) Traceback (most recent call last): File "", line 1, in TypeError: foo() takes no arguments (1 given) Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Cross-over :-) On Saturday, 25 February 2017 14:37:53 GMT Ralph Corderoy wrote: > > How do these functions tie in with the GPIO.add.event.detect > > statement. Also, what calls the function? > > They are the callbacks registered using GPIO.add_event_detect(). They > are called by the GPIO module when it learns an event has happened. The > module sets up a second thread, separate from the first where your main > Python code is running, and it's that thread that waits for events and > calls your callbacks. Yes. I understand that part. This is the bit that works. > > What is the 'gpio' in this and the other functions? > > Like my other email with the callback example, the GPIO module passed in > "event data" to the callback function when it calls it. You've > > GPIO.add_event_detect(20, GPIO.RISING, callback=foo, bouncetime=500) > > def foo(): > > from your earlier emails, but I'd be surprised if that works because > foo's expecting no arguments, but one, the `gpio', is going to be passed > in. Oh OK. The example on Alex Eames website uses (channel). I wasn't really sure what that was for either. It works OK as long as the program isn't busy elsewhere. > Ah, OK, so it's basically `sleeping' as far as the kernel is concerned. > Is there any problem with the quarters playing in the middle of one of > your other mpg321 playbacks so you need to step in and ensure it doesn't > happen? No I've proved that. The chimes are fired off using subprocess.Popen() and so is the mp3 player, change rings etc. It works well. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > > > 2. Change Rings Control - The simplest function. It either plays the > > > > bell_player = None > > > > def start_bell_player(gpio): > > global bell_player > > How do these functions tie in with the GPIO.add.event.detect > statement. Also, what calls the function? They are the callbacks registered using GPIO.add_event_detect(). They are called by the GPIO module when it learns an event has happened. The module sets up a second thread, separate from the first where your main Python code is running, and it's that thread that waits for events and calls your callbacks. > What is the 'gpio' in this and the other functions? Like my other email with the callback example, the GPIO module passed in "event data" to the callback function when it calls it. You've GPIO.add_event_detect(20, GPIO.RISING, callback=foo, bouncetime=500) def foo(): from your earlier emails, but I'd be surprised if that works because foo's expecting no arguments, but one, the `gpio', is going to be passed in. As for its value, it's an int describing the GPIO pin, or `channel', where the event has been detected. Quite what that is, I don't know. It seems there's two methods of numbering the GPIOs, and the module supports both, but if you print it out you should see the correspondence, e.g. `20' from your add_event_detect() call above. > The current version of the software (which works, but without any User > Interface) simply sits around waiting for the apscheduler events to > trigger the hours and quarters Ah, OK, so it's basically `sleeping' as far as the kernel is concerned. Is there any problem with the quarters playing in the middle of one of your other mpg321 playbacks so you need to step in and ensure it doesn't happen? (For others that are interested, having looked at some of the GPIO module, it's not really handling interrupts, but just asking the kernel to let it know when there's some data to read from a bunch of `gpio' files the kernel provides and makes data available on when a change occurs. It uses epoll(2), a more modern form of select(2). And provides another layer of abstraction/obsfucation. :-) Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Ralph, I'm still struggling a bit (well a lot actually ;-) ). The problem is still how to get all the things done that I need to get done. In this message, I'll try to answer some of your questions that I've failed to address and raise a few more for you. On Friday, 24 February 2017 15:43:04 GMT Ralph Corderoy wrote: > > The are four functions that are controlled by the toggle switches on > > the front of the Control Box: > And the switches can be held up or down and spring back to the centre > "off" position. Yes. > def start_bell_player(gpio): > global bell_player I'm still unsure what the 'gpio' in the function call is for in your example. > > There are generally at least one other thing that needs doing; hence > > my query about threaded or multi-processed function calls. > > Still don't think you need it. The Pi GPIO library takes care of your > second thread to call your registered event callbacks. The subprocess > module handles starting child processes running external programs. You > don't want any more parallelism that this else you have to worry about > things happening at the same time and communicating between parts to > stop it going wrong, e.g. a quick switch up then down could have two > callbacks running at once as a race. This is the key point I think. Reading through the C code in the RPi.GPIO Library that you provided the link to yesterday, I can see the code to handle the callbacks, but my understanding of C is insufficient to inform me what is going on. There are a number of references in the code to 'callbacks' (as opposed to 'callback'), so I started off thinking that I could add multiple callbacks to the GPIO.add_event_detect() statement by listing them as you indicated for your example code in your later message that I reference below. However, if this is the case, I can't work out how to do it and even if I could, it would probably still not help because the new code needs to run in a separate thread, not just the call to it. On Saturday, 25 February 2017 00:10:59 GMT Ralph Corderoy wrote: > Once everything is set up, your code has to pass control over to the > library by calling its handle_events(). It then toils and calls back to > your code if it so deigns. This is the `inversion of control' compared > to normal imperative programming. So, I'm think that what you were leading up in your example to is some scheme whereby I set up and create an event handler similar to the one you created in this message. This code is run before the GPIO.add_event_detect() statements and the callback in each of those statements references the appropriate function. These functions are where I put my code to change the state of the global control flags, kill the mpg321 player and play the message to the user. Is that what you had in mind? -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
