Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > > But that's not "using your existing... functional-programming > > thingy" mentioned above. > > > > mp3_files = sorted(os.path.join(mp3_subdir, x) for x in > > filter(lambda f: f.lower().endswith(".mp3"), os.listdir(mp3_subdir))) > > No it's not, but that bit of code produces a list of files, including > paths, separated by commas (all the os methods that I tried (such as > os.list dir do). No, it produces a `list', a Python type that's an array-like data structure that can change length. You've been writing lists literally with brackets in your code, e.g. Popen() is being passed one parameter here: subprocess.Popen(['mpg321', '-q', '-m', '-a', 'hw:0,0', 'foo.mp3']) Python prints lists with commas between items, but a list isn't a string and the commas are just formatting. You need to join your mp3_files list onto the end of your hard-coded ['mpg321'...] one so Popen() sees just one long list. https://docs.python.org/2/tutorial/introduction.html#lists > I suspected as much. Presumably in the shell the * is expanded before > it is given to mpg321 (without commas) so it works. Yes, unlike DOS, glob expansion is normally done by the shell and programs just see literal file names to process. This aids consistency as each program doesn't develop their own dialect, and allows centralised improvements, e.g. bash's `**' globstar. Historically, it was a separate program to the shell due to memory constraints, called glob(1), but these days shells have it built in, and larger languages provide similar logic in libraries, e.g. glob.glob(). Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 17:46:43 GMT Ralph Corderoy wrote: > Hi Terry, > > > > Change it back to that and take them to be directory names instead, > > > and `daisy/*.mp3' is what you pass to mpg321 using your existing > > > `sorted(...listdir)' functional-programming thingy. Well actually, that is one of the things that I tried before I got down and dirty and really basic. > > even if I explicitly change directory to the one containing the files > > Don't do that. No I don't want to, but I was trying things out to see if it would work with the simplest approach. > But that's not "using your existing... functional-programming thingy" > mentioned above. > > mp3_files = sorted(os.path.join(mp3_subdir, x) for x in > filter(lambda f: f.lower().endswith(".mp3"), os.listdir(mp3_subdir))) No it's not, but that bit of code produces a list of files, including paths, separated by commas (all the os methods that I tried (such as os.list dir do). Comma separated filenames don't work in the shell either. > There's also glob.glob() that would probably make the above simpler. > https://docs.python.org/2/library/glob.html > > mp3_files = sorted(glob.glob(mp3_subdir + '/*.[Mm][Pp]3')) That also puts the commas in. I've been trying to work out how to strip the commas out, but then I tried making the list manually and I couldn't get that to work either. > mpg321 sees the literal string `*.mp3' and tries to open it. If it > exists, it would play it. > A raw `*.mp3' in the source don't mean those filenames to Python as it > doesn't glob strings like a shell. That's why you're doing listdir(), > etc. Yes. I suspected as much. Presumably in the shell the * is expanded before it is given to mpg321 (without commas) so it works. > I think part of the problem here is you're trying to write a program to > meet your needs in Python, not trying to learn Python so you can then > write a program to meet your needs. :-) Learning all of Python before I set out to write the code would be a major task in itself. Then I'd probably learn the wrong bits. I have followed some tutorials online, but none of them got down to this stuff. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 15:00:00 GMT Ralph Corderoy wrote: > mpg321 has a -l option to loop N times. If N is 0 then forever. So > your callback starts mpg321 with -l and gives it the sorted list of all > the MP3s in the current playlist-directory entry, e.g. > `valkyries/*.mp3'. Then it forgets all about it until it's time to > SIGTERM it. Except it doesn't (for me anyway). If I issue the command: mpg321 -l ./Playlist1/*.mp3 in a shell it works. In my Python program it doesn't, even if I explicitly change directory to the one containing the files and format my statement as: mp3_player = subprocess.Popen(['mpg321', '-q', '-m', '-a', 'hw:0,0', '*.mp3']) If I use: mp3_player = subprocess.Popen(['mpg321', '-q', '-m', '-a', 'hw:0,0', '01 Marche Episcopale.mp3']) it works. If I use: mp3_player = subprocess.Popen(['mpg321', '-q', '-m', '-a', 'hw:0,0', *.mp3]) (eg no quotes around the *.mp3), I get a syntax error on the asterisk. Can this be escaped somehow? Or is there something else that I've missed? -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > > They're the files to be played. > > Ah. We will have up to six independent playlists, so we would have to > manage that too. Oh, that explains why I called the tuple `playlists' originally. :-) Change it back to that and take them to be directory names instead, and `daisy/*.mp3' is what you pass to mpg321 using your existing `sorted(...listdir)' functional-programming thingy. `announce-daisy.mp3', alongside the `daisy' directory, not in it, is still the announcement. > 3. MP3 Control - The most complicated. Depressing the switch > increments a variable, plays a user message to indicate which Playlist > is selected and then fires off the player which will then loop through > the Playlist until stopped by an upwards click. mpg321 has a -l option to loop N times. If N is 0 then forever. So your callback starts mpg321 with -l and gives it the sorted list of all the MP3s in the current playlist-directory entry, e.g. `valkyries/*.mp3'. Then it forgets all about it until it's time to SIGTERM it. IOW, pretty much the same as my last walkthrough other than mpg321's arguments being different; you're starting a long-running program with the subprocess module. BTW, if mpg321 didn't have a -l option, you'd just have a little looping shell script to call mpg321 forever and call that from your Python programming. The key point is, let the Unix process model handle doing things in parallel rather than trying to do it all in Python using threads. Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 14:03:27 GMT Terry Coles wrote: > On Friday, 3 March 2017 14:00:43 GMT Ralph Corderoy wrote: > > Where do you get your mpg321 program from? A package in the > > distribution? If so, which package, which distribution? Or somewhere > > else? > > On the Pi its the Raspbian repository and on this machine it's the Kubuntu > one. > > Both appear to behave the same. Sorry. The package name is simply mpg321 and the version is 0.3.2-1.1 on both. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 14:00:43 GMT Ralph Corderoy wrote: > Where do you get your mpg321 program from? A package in the > distribution? If so, which package, which distribution? Or somewhere > else? On the Pi its the Raspbian repository and on this machine it's the Kubuntu one. Both appear to behave the same. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, Where do you get your mpg321 program from? A package in the distribution? If so, which package, which distribution? Or somewhere else? Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 13:40:07 GMT Terry Coles wrote: > On the principle that I don't need the inner while loop (around > mp3_player.poll() ) as you mentioned in an earlier response, I think that > this would work, but someone would have to go over and start the player > again every 30 mins or so. Also, as specified, it would then step on to > the next playlist and the user would have to keep clicking the switch until > he got back to the one that was playing if that's what was required. The > town often has different themes on different days, so it would be hymns one > day and organ music another, and so on. Actually, I've just realised that I was wrong here in that there is the for loop which presents each file to the mpg321 player after the previous one has been completed. However, I believe that if the filenames were presented to mpg321 in a list, with a space between each one, that would work. It doesn't get round the outer loop problem so that the Playlist is replayed for ever. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 12:44:30 GMT Ralph Corderoy wrote: > They're the files to be played. Ah. We will have up to six independent playlists, so we would have to manage that too. > Right, so your fchimes_path is always the same file, but it needs to > alter on each cycle for a playlist. Not quite. At least, not quite how you describe it below. > > I'm assuming that the playlists[playlist_index] element is intended to > > cycle through the files in the playlist directory (see my first > > question). > > playlists[playlist_index] is the current element, playlist_index cycles > through playlists's indexes. > > > Will it do this forever (until terminated) or only once? > > Each time cycle_playlist() is called it does > > playlist_index += 1 > playlist_index %= len(playlists) > > The `%' operator is Python's "modulo". > https://docs.python.org/2/reference/expressions.html#binary-arithmetic-opera > tions Yes. I understood that bit. > daisy.mp3 is the tune, announce-daisy.mp3 is your short announcement of > what's coming up next. OK. I missed that. > Bump the index on by one, ready for the next call to cycle(). It will > go 01234012340... If I've understood you correctly, the problem with that is that only one file will be played each time the user flicks the switch. To re-iterate what I said originally: 3. MP3 Control - The most complicated. Depressing the switch increments a variable, plays a user message to indicate which Playlist is selected and then fires off the player which will then loop through the Playlist until stopped by an upwards click. This is a precis of the Requirements that we agreed with the WMT Staff. Up until this year, they had an ancient 'boom-box' sitting on the floor of the chancel with a memory stick plugged into it. The stick contained several directories, each containing one to many MP3 files. The boom-box was set to play all of the files in a given directory, in a loop, until someone came along to stop it or change the directory. This function has to replicate that. The code that I was using could do that, except that it played through the files in the directory only once. One of my concerns with multi-processing / threading was that I needed to continuously loop round the playlist, which was a bit difficult if only one thread was available. Here is the code that worked without the outer loop: def mp3_player(): # Setup and run the MP3 Player global mp3_subdir mp3_files = sorted(os.path.join(mp3_subdir, x) for x in filter(lambda f: f.lower().endswith(".mp3"), os.listdir(mp3_subdir))) print 'MP3 Player - Playing ', mp3_files, 'to the Chancel' blackhole = open(os.devnull, 'w') for file in mp3_files: mp3_player = subprocess.Popen(['mpg321', '-q', '-m', '-a', 'hw:1,0', file], stdin=subprocess.PIPE, stdout=blackhole, stderr=blackhole)# launch the player while True: mp3_player.poll() # check to see if terminated if mp3_player.returncode != None: break time.sleep(0.1) if mp3_player.returncode == None or mp3_control == 0: # kill the player if it's still running mp3_player.terminate() mp3_player.wait() On the principle that I don't need the inner while loop (around mp3_player.poll() ) as you mentioned in an earlier response, I think that this would work, but someone would have to go over and start the player again every 30 mins or so. Also, as specified, it would then step on to the next playlist and the user would have to keep clicking the switch until he got back to the one that was playing if that's what was required. The town often has different themes on different days, so it would be hymns one day and organ music another, and so on. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > > playlists = ('daisy', 'peal', 'vikings', 'march', 'fugue') > > Are these intended to be just names or should they link in some way to > the files to be played. They're the files to be played. > >r = subprocess.call(...mpg321..., 'announce-' + > > playlists[playlist_index]) > > I originally assumed that your ...mpg321..., was intended to represent > the code need to get the mpg player to play to the right place etc, so > I substituted my own code there (see below). Correct. > here is an example of a subprocess.call() from my program that > actually works: > > subprocess.call(['mpg321', '-q', '-m', '-m', '-a', 'hw:0,0', > fchimes_path]) > > Where fchimes_path is assembled from the filename and the directory > path using os.path.join(). Right, so your fchimes_path is always the same file, but it needs to alter on each cycle for a playlist. > I'm assuming that the playlists[playlist_index] element is intended to > cycle through the files in the playlist directory (see my first > question). playlists[playlist_index] is the current element, playlist_index cycles through playlists's indexes. > Will it do this forever (until terminated) or only once? Each time cycle_playlist() is called it does playlist_index += 1 playlist_index %= len(playlists) The `%' operator is Python's "modulo". https://docs.python.org/2/reference/expressions.html#binary-arithmetic-operations Here's the code again, I'll describe it. playlist = ('daisy', 'peal', 'valkyries', 'march', 'fugue') A list of the files in the playlist. I didn't bother with the suffixes because they'll all be the same? playlist_index = 0 Start by playing https://en.wikipedia.org/wiki/Daisy,_Daisy#In_technology_and_culture def cycle_playlist(gpio): global playlist_index, playlist_player stop_playlist(None) Don't rely on the user to "stop" before cycling onto the next tune. mp3 = '%s/announce-%s.%s' % ('path/to/tunes', playlist[playlist_index], '.mp3') daisy.mp3 is the tune, announce-daisy.mp3 is your short announcement of what's coming up next. r = subprocess.call(['mpg321', '-q', '-m', '-m', '-a', 'hw:0,0', mp3]) if r: print 'cycle_playlist: mpg321 failed: %d %#x\n' % (playlist_index, r) mp3 = '%s/%s.%s' % ('path/to/tunes', playlist[playlist_index], '.mp3') Now for daisy.mp3. playlist_player = subprocess.Pipe(['mpg321', '-q', '-m', '-m', '-a', 'hw:0,0', mp3] playlist_index += 1 playlist_index %= len(playlist) Bump the index on by one, ready for the next call to cycle(). It will go 01234012340... Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 24 February 2017 15:43:04 GMT Ralph Corderoy wrote: Ralph, Thanks for all your help on the subprocess stuff. I'm sure I have a way forward now. In the meantime, I've been trying to get the MP3 Player to work. From your original suggestions: > playlists = ('daisy', 'peal', 'vikings', 'march', 'fugue') Are these intended to be just names or should they link in some way to the files to be played. I've assumed the latter, but when I add the path it still doesn't work (that may be because of an earlier error, see below). > r = subprocess.call(...mpg321..., 'announce-' + > playlists[playlist_index]) I originally assumed that your ...mpg321..., was intended to represent the code need to get the mpg player to play to the right place etc, so I substituted my own code there (see below). After I got a runtime error. I tried it as written and got a syntax error and when I put mpg321 into quotes, I got the following runtime error: TypeError: bufsize must be an integer I've tried running it in the IDLE Debugger, but that doesn't really help me (it might someone else who actually understood the Debugger output). However, here is an example of a subprocess.call() from my program that actually works: subprocess.call(['mpg321', '-q', '-m', '-m', '-a', 'hw:0,0', fchimes_path]) Where fchimes_path is assembled from the filename and the directory path using os.path.join(). The thing is that each of the strings in quotation marks after the element containing mpg321 are options to it. However, 'announce' isn't a valid option, so is that what is causing the error? If so, what is it trying to do? Is that another representation of something? It was at this point that I was intending to play a short mp3 file to announce the identity of the current playlist. Also, and related to my first question; in your invocation, I'm assuming that the playlists[playlist_index] element is intended to cycle through the files in the playlist directory (see my first question). Will it do this forever (until terminated) or only once? -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > So what happens if the .terminate() doesn't work? Well, it's likely to work in that it will successfully send SIGTERM to mpg321. That's all it's trying to do; the Unix function is called signal(3). (The misleading `terminate' is typical of later Python's cross-platform abstractions obscuring the truth!) The default action is for the kernel to kill the process; see signal(7). But it is possible for that to be altered so mpg321 catches the signal, i.e. has explicit code to handle it and know it's occurred, and decides to ignore it, or just do something else first and that takes an unexpectedly long time. > Presumably we have to wait for the playback to end, which could be a > long time if we are playing the Playlists in the MP3 Player. Yes, the wait() will only return when mpg321 finally exits so you can examine its returncode. I'd stick with that. If you want, you could note the time.time() before and after the wait() and if it's outside the expected [0, max] interval then print it to aid investigation if it should ever be a problem. But I wouldn't try and fix what shouldn't be happening as that could just disguise the root cause of several problems, making debugging harder. Fail early. Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Friday, 3 March 2017 11:31:44 GMT Ralph Corderoy wrote: > Correct. So `if r not in (0, -15):' looks apt. It does to me too. Thanks. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
Hi Terry, > So what was your intention when you wrote those lines: > > r = bell_player.wait(): > if r: > print 'bell_player: mpg321 failed: %#x\n' % r > bell_player = None I was intending that you'd see the results familiar from https://www.mail-archive.com/dorset@mailman.lug.org.uk/msg07338.html when I talked about zombie processes, etc. > It seems to me that if the mpg321 process exits with -15, that is no > more a failure than if it exits with 0, because terminating the > process was the intended behaviour. Correct. So `if r not in (0, -15):' looks apt. Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Thursday, 2 March 2017 20:17:53 GMT Ralph Corderoy wrote: > Because this wait() doesn't take any parameters, unlike the threading > module's one. :-) So what happens if the .terminate() doesn't work? Presumably we have to wait for the playback to end, which could be a long time if we are playing the Playlists in the MP3 Player. I suppose, having a timeout wouldn't gain us much in this instance, because we would still have to wait. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Python Multiprocessing (or Maybe Multithreading)
On Thursday, 2 March 2017 19:36:29 GMT Ralph Corderoy wrote: > GPIO creates *one* thread, not one per add_event_detect(). That one > thread calls the registered callbacks sequentially. You end up with two > threads. > I did keep trying to get across you didn't need more than this. :-) Yes I know. I was hung up on the idea that my functions would take a while to return. > I think you saw -0xf. Yes, I did. > > http://www.chiark.greenend.org.uk/doc/python2.7-dev/html/library/threading > > .html#event-objects > That's a copy of the documentation for the threading module. ;-( I know that now. > > I think that I should be testing for not r in the if statement > > (because it works) > > Not a great reason! :-) Deleting the if statement completely also > works. True. Now you know why I was a systems engineer and not a software engineer. > Did you try letting mpg321 finish its playing so it exited normally, and > then flick the switch to request it to stop? You should see a > non-negative number that's its exit(3) value, hopefully 0 to indicate > it's happy. Yes. it was. So what was your intention when you wrote those lines: r = bell_player.wait(): if r: print 'bell_player: mpg321 failed: %#x\n' % r bell_player = None It seems to me that if the mpg321 process exits with -15, that is no more a failure than if it exits with 0, because terminating the process was the intended behaviour. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-03-07 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR