Re: Python as a scripting language. Alternative to bash script?
In message slrni3478o.1qf.user...@rutherford.ilthio.net, Tim Harig wrote: On 2010-07-05, Lawrence D'Oliveiro l...@geek-central.gen.new_zealand wrote: I’ve never come across any system where you could string together multiple GUI apps, or even multiple GUI operations in the same app, in any sensible or effective way at all. GUIs just aren???t designed to work that way. You can if they are designed to be used externally. COM is an execellent example of this ... But COM is an IPC architecture, which has nothing to do with GUI apps. Conflating the two leads to precisely the sort of inflexibility that is characteristic of GUIs. For example, things happening with the GUI on-screen while a script is running that are at best distracting to the user, or at worst vulnerable to breaking if the user accidentally presses a key or clicks in the wrong place. I have worked with complex business process automation centered around Excel and I have automated some really ugly AJAX style web based applications that would only work inside of IE6 by accessing IE through its COM interface. If those apps really were using AJAX, then why did you need to script a Web browser at all? You could just make direct HTTP requests to retrieve the data. Automating GUI applications requires interal access to the program through some kind of interface ... Some kind of interface that bypasses the GUI. This is why so many Open Source apps are structured as a GUI front end to a completely separate command-line tool (e.g. MPlayer). That way a script can use the latter without having to go through the former. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
In message mailman.301.1278423632.1673.python-l...@python.org, Michael Torrie wrote: While it's possible to set up pipes and spawn programs in parallel to operate on the pipes, in practice it's simpler to tell subprocess.Popen to use a shell and then just rely on Bash's very nice syntax for setting up the pipeline. Just be careful about properly escaping any special characters in the file names. See my other thread about escaping data, where some respondents have expressed a strong allergy to any form of embedding one language inside another. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 2010-07-24, Lawrence D'Oliveiro l...@geek-central.gen.new_zealand wrote: In message slrni3478o.1qf.user...@rutherford.ilthio.net, Tim Harig wrote: On 2010-07-05, Lawrence D'Oliveiro l...@geek-central.gen.new_zealand wrote: I???ve never come across any system where you could string together multiple GUI apps, or even multiple GUI operations in the same app, in any sensible or effective way at all. GUIs just aren???t designed to work that way. You can if they are designed to be used externally. COM is an execellent example of this ... But COM is an IPC architecture, which has nothing to do with GUI apps. Not exactly. IPC allows you to send and receive information from another process; but, it doesn't give you internal access to the process. COM is much closer to an OO style of RPC. Conflating the two leads to precisely the sort of inflexibility that is characteristic of GUIs. For example, things happening with the GUI on-screen while a script is running that are at best distracting to the user, or at worst vulnerable to breaking if the user accidentally presses a key or clicks in the wrong place. COM object have a visibility property that determines whether or not they are displayed to the user. I have worked with complex business process automation centered around Excel and I have automated some really ugly AJAX style web based applications that would only work inside of IE6 by accessing IE through its COM interface. If those apps really were using AJAX, then why did you need to script a Web browser at all? You could just make direct HTTP requests to retrieve the data. The first and primary reason was that the application server required a client side SSL certificate. We didn't have access to a certificate that we could get to work Python's SSL or HTTPS modules. The second reason was that we had considerable difficulty figuring out how the server side interfaces worked from the slew of horrible Javascript. You really have to have seen it to know just what a mess this really was. Every call was literally wrapped in dozens of almost identical wrapper functions that you had to follow to attempt to figure out what was going on. Parts of the rendered form which looked like text field entries really weren't until a series of events took place to change them. They were used as flags indicating the state of various information. I really can't overstate how ugly this application really was. I was working on trying to figure out the serverside interface while my partner was trying to get an open SSL connection. By the time he gave up trying to convert our key to something we could use, I decided it was simpler just to use COM and internet explorer. It worked pretty well. Automating GUI applications requires interal access to the program through some kind of interface ... Some kind of interface that bypasses the GUI. This is why so many Open Source apps are structured as a GUI front end to a completely separate command-line tool (e.g. MPlayer). That way a script can use the latter without having to go through the former. Which is great where the a separate backend exists. That isn't the case for a *huge* number of GUI applications; but, You don't always get to pick what you are going to need to automate. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
John Nagle, 28.06.2010 19:57: Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. Except that you just broke the simplicity of the pipe mechanism. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 07/06/2010 09:34 PM, Chris Rebert wrote: On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie torr...@gmail.com wrote: While it's possible to set up pipes and spawn programs in parallel to operate on the pipes, in practice it's simpler to tell subprocess.Popen to use a shell and then just rely on Bash's very nice syntax for setting up the pipeline. Until there's a Python variable involved that is, unless you want to overlook all the edge cases or do the escaping all by yourself (and then pray you did it right). Very good point. This is a problem that the pipes module suffers from as well. Although we learned in the other thread on escaping SQL statements that escaping is faster, easier and just as safe as other parameterization mechanisms. Uh huh. Back on target, a library similar to pipes that was safe (pipes is not) and had a pythonic syntax would be cool. pipes module works alright, syntax wise, but it's not a clean syntax. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Wed, Jul 7, 2010 at 8:31 AM, Michael Torrie torr...@gmail.com wrote: On 07/06/2010 09:34 PM, Chris Rebert wrote: On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie torr...@gmail.com wrote: While it's possible to set up pipes and spawn programs in parallel to operate on the pipes, in practice it's simpler to tell subprocess.Popen to use a shell and then just rely on Bash's very nice syntax for setting up the pipeline. Until there's a Python variable involved that is, unless you want to overlook all the edge cases or do the escaping all by yourself (and then pray you did it right). Very good point. This is a problem that the pipes module suffers from as well. Although we learned in the other thread on escaping SQL statements that escaping is faster, easier and just as safe as other parameterization mechanisms. Uh huh. Back on target, a library similar to pipes that was safe (pipes is not) and had a pythonic syntax would be cool. pipes module works alright, syntax wise, but it's not a clean syntax. Actually, your original post inspired me to take a crack at writing something like that yesterday: http://rebertia.com/code/subproc_pipelines.py Thoughts anyone? (Written on a whim, so no tests or docs at present.) Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 28 Jun, 19:39, Michael Torrie torr...@gmail.com wrote:
In python I could simply take the output of ps ax and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Or you just pass the stdout of one command as stdin to another. That
is equivalent of piping with bash.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 07/06/2010 04:12 AM, sturlamolden wrote:
On 28 Jun, 19:39, Michael Torrie torr...@gmail.com wrote:
In python I could simply take the output of ps ax and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Or you just pass the stdout of one command as stdin to another. That
is equivalent of piping with bash.
Consider this contrived example:
tail -f /var/log/messages | grep openvpn
While it's possible to set up pipes and spawn programs in parallel to
operate on the pipes, in practice it's simpler to tell subprocess.Popen
to use a shell and then just rely on Bash's very nice syntax for setting
up the pipeline. Then just read the final output in python. If you set
the stdout descriptor to non-blocking, you could read output as it came.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie torr...@gmail.com wrote:
On 07/06/2010 04:12 AM, sturlamolden wrote:
On 28 Jun, 19:39, Michael Torrie torr...@gmail.com wrote:
In python I could simply take the output of ps ax and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Or you just pass the stdout of one command as stdin to another. That
is equivalent of piping with bash.
Consider this contrived example:
tail -f /var/log/messages | grep openvpn
While it's possible to set up pipes and spawn programs in parallel to
operate on the pipes, in practice it's simpler to tell subprocess.Popen
to use a shell and then just rely on Bash's very nice syntax for setting
up the pipeline. Then just read the final output in python. If you set
the stdout descriptor to non-blocking, you could read output as it came.
--
http://mail.python.org/mailman/listinfo/python-list
Is this a discussion about the pipes module in the std library?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Tue, Jul 6, 2010 at 1:35 PM, member thudfoo thud...@opensuse.us wrote:
On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie torr...@gmail.com wrote:
On 07/06/2010 04:12 AM, sturlamolden wrote:
On 28 Jun, 19:39, Michael Torrie torr...@gmail.com wrote:
In python I could simply take the output of ps ax and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Or you just pass the stdout of one command as stdin to another. That
is equivalent of piping with bash.
Consider this contrived example:
tail -f /var/log/messages | grep openvpn
While it's possible to set up pipes and spawn programs in parallel to
operate on the pipes, in practice it's simpler to tell subprocess.Popen
to use a shell and then just rely on Bash's very nice syntax for setting
up the pipeline. Then just read the final output in python. If you set
the stdout descriptor to non-blocking, you could read output as it came.
Is this a discussion about the pipes module in the std library?
No, though that module is not irrelevant to Mr. Torrie's argument.
Cheers,
Chris
--
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie torr...@gmail.com wrote:
On 07/06/2010 04:12 AM, sturlamolden wrote:
On 28 Jun, 19:39, Michael Torrie torr...@gmail.com wrote:
In python I could simply take the output of ps ax and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Or you just pass the stdout of one command as stdin to another. That
is equivalent of piping with bash.
Consider this contrived example:
tail -f /var/log/messages | grep openvpn
While it's possible to set up pipes and spawn programs in parallel to
operate on the pipes, in practice it's simpler to tell subprocess.Popen
to use a shell and then just rely on Bash's very nice syntax for setting
up the pipeline.
Until there's a Python variable involved that is, unless you want to
overlook all the edge cases or do the escaping all by yourself (and
then pray you did it right).
Cheers,
Chris
--
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
In message 7xpqzbj8st@ruckus.brouhaha.com, Paul Rubin wrote: ... and argc/argv were passed to the child process on its stack. I’ve always felt that to be a misfeature. It means you can’t implement a self-contained argument-parsing library, it still needs the mainline to explicitly pass/put its arguments somewhere. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
In message xns9da77f36b9f6emithrandirisawes...@80.93.112.4, Mithrandir wrote:
I think that Python could be a alternative to bash and have some
advantages, but it's a long way off from being fully implemented.
Would you prefer to do the following sort of thing in Python or Bash?
AudioParms = -f s16le -ar 48000 -ac 2 # because I can't seem to pipe
compressed audio
ImportCmd = \
(
ffmpeg -v 0 -i (%(src_video)s)
%(audio_parms)s -i (%(src_audio)s)
-vcodec copy -acodec %(audio_codec)s -y %(dstfile)s
%
{
src_video :
; .join
(
[
ffmpeg -v 0 -i %(srcfile)s -f image2pipe
-vcodec copy
-y /dev/stdout
%
{srcfile : ShellEscape(SrcFile)}
for
SrcFile in SrcFiles
]
), # pipe through compressed video without
recompression
src_audio :
; .join
(
[
ffmpeg -v 0 -i %(srcfile)s %(audio_parms)s
-y /dev/stdout
%
{
srcfile : ShellEscape(SrcFile),
audio_parms : AudioParms,
}
for
SrcFile in SrcFiles
]
),
dstfile : ShellEscape(DstFile),
audio_parms : AudioParms,
audio_codec : mp2, # assumption!
}
)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
In message op.ve06nlvia8n...@gnudebst, Rhodri James wrote: Classic Unix programming is a matter of stringing a bunch of tools together with pipes to get the output you want. This isn't a great paradigm for GUIs (not without tweaking that hasn't really been done), but then again it was never meant to be. I’ve never come across any system where you could string together multiple GUI apps, or even multiple GUI operations in the same app, in any sensible or effective way at all. GUIs just aren’t designed to work that way. The command line (or scripting, the difference isn’t that important) remains the only workable way to string together complex combinations of simpler operations. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 2010-07-05, Lawrence D'Oliveiro l...@geek-central.gen.new_zealand wrote: In message op.ve06nlvia8n...@gnudebst, Rhodri James wrote: Classic Unix programming is a matter of stringing a bunch of tools together with pipes to get the output you want. This isn't a great paradigm for GUIs (not without tweaking that hasn't really been done), but then again it was never meant to be. I???ve never come across any system where you could string together multiple GUI apps, or even multiple GUI operations in the same app, in any sensible or effective way at all. GUIs just aren???t designed to work that way. You can if they are designed to be used externally. COM is an execellent example of this as is script-fu and any number of other technologies that allow external access to the subroutines or automation capability of a GUI application. The problem is not that it cannot be done; but, that it must be explicilty designed to be used this way. I have worked with complex business process automation centered around Excel and I have automated some really ugly AJAX style web based applications that would only work inside of IE6 by accessing IE through its COM interface. The command line (or scripting, the difference isn???t that important) remains the only workable way to string together complex combinations of simpler operations. The difference is that it is almost always possible to automate using text based applications, even when the author of the software never designed the software to be scripted. Text based IO streams are easy to capture and manipulate. Automating GUI applications requires interal access to the program through some kind of interface and, ideally, decent documention of the interface, something that is missing from many, if not most, GUIs. Anything else relies on ugly and, generally fragile, mechanisms to intercept the programs output and send event triggers to the application. The recent thread to automate Minesweeper by processing its screenshot is an example of this. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 7/5/2010 11:02 AM Tim Harig said... Automating GUI applications requires interal access to the program through some kind of interface and, ideally, decent documention of the interface, something that is missing from many, if not most, GUIs. Anything else relies on ugly and, generally fragile, mechanisms to intercept the programs output and send event triggers to the application. I've been doing the 'anything else...' stuff for years now without issue, so ugly, yes, but fragile, not really. Unless you let users on the same machine... Emile -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
I'm the OP btw. On 1 July 2010 18:10, Dennis Lee Bieber wlfr...@ix.netcom.com wrote: I think that Python could be a alternative to bash and have some advantages, but it's a long way off from being fully implemented. While a somewhat klutzier language in aspects (the , is both an parameter separator, AND the continued line indicator, leading to lines that end in , ,) I'd consider a decent* REXX implementation first... Any program line that is not recognized as starting a REXX statement is automatically passed to the currently defined command processor. And one could ensure this by just quoting the first word on the line G Useful. However, I know Python, never looked at Rexx, so not much use to me. Tks -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ. http://www.dpawson.co.uk -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
* Dave Pawson dave.paw...@gmail.com [2010-07-02 08:22]: I'm the OP btw. On 1 July 2010 18:10, Dennis Lee Bieber wlfr...@ix.netcom.com wrote: I think that Python could be a alternative to bash and have some advantages, but it's a long way off from being fully implemented. Take a look at Python for Unix and Linux System Administration: http://www.amazon.com/Python-Unix-Linux-System-Administration/dp/0596515820 I quite like the book, I learned about Ipython from this book and I'm quite glad I did! Jerry -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
Michael Torrie torr...@gmail.com wrote in news:mailman.2313.1277759925.32709.python-l...@python.org: On 06/28/2010 02:06 PM, Mithrandir wrote: I can't see Python as an alt. to bash. (As I recall) Python is much more object-oriented than bash, but also there are many commands (such as apt- get, etc.) that would need Python equivs. However, I can see Python being used as a scripting alt. to C. OO is a plus for most things, in my book. As for commands they have *nothing* to do with Bash. apt-get is not a Bash command. By your logic tcsh or zsh would not be an alternate to bash, but in fact they are. I use python for shell scripting quite often now. Anytime one of my own Bash scripts exceeds 100 lines, I know it's time to switch it to python. Please read that link I posted a while back on how you can use generators in python to replace many of the things that piping to external commands did in Bash. There certainly are a few tasks that Bash is best at (chaining commands together through pipes), but often Python already has support for many of the things I'd use external commands and pipes in Bash for. Bash is designed for working down at the level of files, directories, and processes, but Python works pretty well too, if you make some abstraction modules like my runcmd module that I use extensively. You both are correct. :) (I wrote that before my first cup of coffee, so my wording was way off. That, and I'm new to Python.) :) I think that Python could be a alternative to bash and have some advantages, but it's a long way off from being fully implemented. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Mon, 2010-06-28, John Nagle wrote: On 6/28/2010 7:58 AM, Benjamin Kaplan wrote: How does a program return anything other than an exit code? Ah, yes, the second biggest design mistake in UNIX. Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. Like others said, you have standard output. sys.stdout for data, sys.stderr for human-readable errors and warnings, and the exit code for machine-readable errors. C was like that once. In the 1970s, all you could return was an int or a float. But that got fixed. Huh? The C we have today cannot return a float, and not even a full int. 0 and 1 work, small integers up to 255 are likely to work, but beyond that common systems (Unix) will chop off the high bits. /Jorgen -- // Jorgen Grahn grahn@ Oo o. . . \X/ snipabacken.se O o . -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Mon, 2010-06-28, Dave Pawson wrote: I've a fairly long bash script and I'm wondering how easy it would be to port to Python. Main queries are: Ease of calling out to bash to use something like imageMagick or Java? Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Can Python access the exit status of a program? I'd prefer the advantages of using Python, just wondering if I got so far with the port then found it wouldn't do something? As other remarked, bash has advantages, too. Personally, if my main job is chaining other non-trivial programs into pipelines and sequences, I don't hesitate to use Bourne shell or bash. Perl is for heavier text processing, and Python for problems with more elaborate data types. Note the distinction Bourne shell/bash. If you can get away with it, use bash for medium/large-sized scripts. Many people try to avoid bash-specific syntax, but they miss out on lots of things that make programs maintainable, like local variables. /Jorgen -- // Jorgen Grahn grahn@ Oo o. . . \X/ snipabacken.se O o . -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Jun 28, 5:48 am, Dave Pawson dave.paw...@gmail.com wrote: I've a fairly long bash script and I'm wondering how easy it would be to port to Python. Main queries are: Ease of calling out to bash to use something like imageMagick or Java? Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Can Python access the exit status of a program? I'd prefer the advantages of using Python, just wondering if I got so far with the port then found it wouldn't do something? Has anyone made this comparison please? TIA -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ.http://www.dpawson.co.uk You do have a couple of couple of python scriting programs that are like bash. I don't know if they are worth using though (you would have to look them up I don't recall the names right now) -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 6/29/10 4:06 AM, Jorgen Grahn wrote: On Mon, 2010-06-28, John Nagle wrote: On 6/28/2010 7:58 AM, Benjamin Kaplan wrote: How does a program return anything other than an exit code? Ah, yes, the second biggest design mistake in UNIX. Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. Like others said, you have standard output. sys.stdout for data, sys.stderr for human-readable errors and warnings, and the exit code for machine-readable errors. C was like that once. In the 1970s, all you could return was an int or a float. But that got fixed. Huh? The C we have today cannot return a float, and not even a full int. 0 and 1 work, small integers up to 255 are likely to work, but beyond that common systems (Unix) will chop off the high bits. I think he's talking about C functions now, not programs. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Python as a scripting language. Alternative to bash script?
I've a fairly long bash script and I'm wondering how easy it would be to port to Python. Main queries are: Ease of calling out to bash to use something like imageMagick or Java? Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Can Python access the exit status of a program? I'd prefer the advantages of using Python, just wondering if I got so far with the port then found it wouldn't do something? Has anyone made this comparison please? TIA -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ. http://www.dpawson.co.uk -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Mon, Jun 28, 2010 at 4:48 AM, Dave Pawson dave.paw...@gmail.com wrote: I've a fairly long bash script and I'm wondering how easy it would be to port to Python. Main queries are: Ease of calling out to bash to use something like imageMagick or Java? Easiest way is os.system, most flexible way is subprocess.Popen. Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? How does a program return anything other than an exit code? Subprocess allows you to read the program's stdout if that's what you're looking for. In the case of ImageMagick, you can use a Python wrapper to the library instead of calling the program from the command line, and then you can get all the return values you want. Can Python access the exit status of a program? proc = subprocess.Popen(args) retcode = proc.wait() There's a shortcut of retcode = subprocess.call(args), but that doesn't give you access to stdout, just the return code. I'd prefer the advantages of using Python, just wondering if I got so far with the port then found it wouldn't do something? If there's anything it can't do that bash can, you can always just call the shell command. Has anyone made this comparison please? If you already have a working shell script, it's probably not worth your time. But if you've having trouble getting bash to cooperate, it's not that difficult to rewrite a shell script in Python. TIA -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ. http://www.dpawson.co.uk -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Mon, 28 Jun 2010 12:48:51 +0100 Dave Pawson dave.paw...@gmail.com wrote: I've a fairly long bash script and I'm wondering how easy it would be to port to Python. That's too big a question without seeing more of what your script does. I will try to suggest some direction though. First, if you have a complicated bash script that works, the best choice may be to just leave it alone. Think about Python for your next project instead. Main queries are: Ease of calling out to bash to use something like imageMagick or Java? You don't need to call bash to call an external program. Check out the subprocess module. If you do need a shell to simplify calling a program (environment and wild card expansione.g.) don't call bash. Just use a basic sh. You won't be using the bash control structures so keep to whatever is supplied by your OS. If that turns out to be bash anyway then no harm. Another option is to write small Python scripts that you can call from your bash script. You can even create them in your bash script. Here is a silly example. uc=import sys s = sys.argv[1] print s.upper() ... echo -n Upper case of $SOMESTRING is ; python -c $uc $SOMESTRING Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Just to set the terminology straight, a parameter is what you call the function with. The return value is what it returns. The program output is what it emits (prints.) Programs return an integer value. This is also called the exxit status. On success this is 0 but can be otherwise on failure. You can use this, for example, with diff to determine if two files differ when you don't care how they differ. What you want is the output of the program. For this you need to capture the output and parse it. Look at the subprocess module. -- D'Arcy J.M. Cain da...@druid.net | Democracy is three wolves http://www.druid.net/darcy/| and a sheep voting on +1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
Thanks for the replies (and Benjamin). Not met with the subprocess idea. On 28 June 2010 16:29, D'Arcy J.M. Cain da...@druid.net wrote: Main queries are: Ease of calling out to bash to use something like imageMagick or Java? You don't need to call bash to call an external program. Check out the subprocess module. Will do. If you do need a shell to simplify calling a program (environment and wild card expansione.g.) don't call bash. I can get what I want from Python. No envars needed. Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Just to set the terminology straight, a parameter is what you call the function with. The return value is what it returns. The program output is what it emits (prints.) My bad. I mean return values, though I do want program out from (for example) identify Programs return an integer value. This is also called the exit status. Sheer greed, for identify I may get either a return value or an exit status (bad input etc) :-) Looks like subprocess can hack it though. What you want is the output of the program. For this you need to capture the output and parse it. Look at the subprocess module. Will do. tks D'Arcy (and Benjamin) -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ. http://www.dpawson.co.uk -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 06/28/2010 06:08 PM, Dave Pawson wrote: Thanks for the replies (and Benjamin). Not met with the subprocess idea. On 28 June 2010 16:29, D'Arcy J.M. Cain da...@druid.net wrote: Main queries are: Ease of calling out to bash to use something like imageMagick or Java? You don't need to call bash to call an external program. Check out the subprocess module. Will do. If you do need a shell to simplify calling a program (environment and wild card expansione.g.) don't call bash. I can get what I want from Python. No envars needed. Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Just to set the terminology straight, a parameter is what you call the function with. The return value is what it returns. The program output is what it emits (prints.) My bad. I mean return values, though I do want program out from (for example) identify If you're working with images, have a look at the PIL (Python Imaging Library). Programs return an integer value. This is also called the exit status. Sheer greed, for identify I may get either a return value or an exit status (bad input etc) :-) Looks like subprocess can hack it though. What you want is the output of the program. For this you need to capture the output and parse it. Look at the subprocess module. Will do. tks D'Arcy (and Benjamin) -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 06/28/2010 05:48 AM, Dave Pawson wrote:
Main queries are: Ease of calling out to bash to use something like
imageMagick or Java? Ease of grabbing return parameters? E.g. convert
can return both height and width of an image. Can this be returned to
the Python program? Can Python access the exit status of a program?
Sure. I've created a module called runcmd that does 90% of what I want
(easy access to stdout, stderr, error code). I've attached it to this
e-mail. Feel free to use it; this post puts my code into the public domain.
I'd prefer the advantages of using Python, just wondering if I got so
far with the port then found it wouldn't do something?
Python really isn't a shell scripting language. So there are things
that Bash does much better, such as spawning processes and piping them
together. I've tried over the years to create a pythonic library that
would let me do that, but haven't found a good syntax that I like.
It turns out, though, that much of what I use piping for in Bash is to
run external processes to do things that I could use python modules for.
For example, I typically pipe stuff to cut a lot to get certain fields.
For example:
ps ax | cut -c1-5
In python I could simply take the output of ps ax and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Sure it's a couple more lines in this case, but in other cases, python's
abilities make it simpler than bash. A great document on how you can
exploit python's abilities (particularly generators) to replace bash
pipelines is here: http://www.dabeaz.com/generators/
#!/usr/bin/python
import subprocess
import sys
import fcntl
import os
import select
import StringIO
This module contains a single function run() that spawns a
process with arguments, and returns it's errcode, stdout,
and stderr.
It currently has a number of deficiencies, including the lack
of a way to kill the child.
def _make_non_blocking(fd):
Takes a file descriptor and sets it to non-blocking IO mode,
allowing us to use the posix select() on it. Used by this
module to allow us to read from a process as it writes to
stderr and stdout
@type fd: number
@param fd: file descriptor number to set to nonblocking
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
def run(cmd, **kwargs):
Runs a command using the subprocess.Popen class, and
returns the err, stdout, and stderr of the process.
Optionally it can print to stdout the process's stderr
and stdout, and can also call callbacks when stdout and
stderr is available from the process. The following
kwargs are useful:
- print_stdout - if set to True will print process stdout
to stdout as data arrives
- print_stderr - if set to True will print process stderr
to stdout as data arrives
- stdin - Text to feed to process's stdin
- stdout_callback - function to call with stdout data
- stderr_callback - function to call with stderr data
@type cmd: text or list
@param cmd: the command to run, use a list if args are passed
@type kwargs: keyword args
@param kwargs: optional keyword arguments (see above)
@rtype: tuple
@return: Tuple of error code, stdout text, stderr text
# Set flags depending on kwargs
if print_stdout in kwargs:
print_stdout=True
else:
print_stdout=False
if print_stderr in kwargs:
print_stderr=True
else:
print_stderr=False
if stdout_callback in kwargs:
stdout_callback=kwargs[stdout_callback]
print setting stdout callback
else:
stdout_callback=None
if stdout_callback in kwargs:
stderr_callback=kwargs[stderr_callback]
print setting stderr callback
else:
stderr_callback=None
if shell in kwargs:
shell=kwargs[shell]
else:
shell=False
# create process object, set up pipes
child = subprocess.Popen(cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=0,
shell=shell
)
# give the process any stdin that we might have, then
# close stdin to prevent deadlocks
if stdin in kwargs:
child.stdin.write(kwargs[stdin])
child.stdin.close()
# get output pipes, make them non-blocking
fo=child.stdout
fe=child.stderr
_make_non_blocking(fo.fileno())
eof=False
stdout=StringIO.StringIO()
stderr=StringIO.StringIO()
# Loop, checking for data on process's stdout and stderr
# until
Re: Python as a scripting language. Alternative to bash script?
On 6/28/2010 7:58 AM, Benjamin Kaplan wrote: How does a program return anything other than an exit code? Ah, yes, the second biggest design mistake in UNIX. Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. C was like that once. In the 1970s, all you could return was an int or a float. But that got fixed. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 28 June 2010 18:39, Michael Torrie torr...@gmail.com wrote: On 06/28/2010 05:48 AM, Dave Pawson wrote: Main queries are: Ease of calling out to bash to use something like imageMagick or Java? Ease of grabbing return parameters? E.g. convert can return both height and width of an image. Can this be returned to the Python program? Can Python access the exit status of a program? Sure. I've created a module called runcmd that does 90% of what I want (easy access to stdout, stderr, error code). I've attached it to this e-mail. Feel free to use it; this post puts my code into the public domain. Thanks Michael. I'd prefer the advantages of using Python, just wondering if I got so far with the port then found it wouldn't do something? Python really isn't a shell scripting language. So there are things that Bash does much better, such as spawning processes and piping them together. I've tried over the years to create a pythonic library that would let me do that, but haven't found a good syntax that I like. I'm using xml quite a bit and tried xmlsh which does the job but has what I think of as irregular syntax? Bash is fine for smaller scripts, but I had to be really careful by the time I got to around 1200 loc. Hence my preference for Python. It turns out, though, that much of what I use piping for in Bash is to run external processes to do things that I could use python modules for. I kept thinking that.. Then I had to use Java/Python for some plain old calculations, so it kept nagging me that a cleaner approach would be a single language. Sure it's a couple more lines in this case, but in other cases, python's abilities make it simpler than bash. A great document on how you can exploit python's abilities (particularly generators) to replace bash pipelines is here: http://www.dabeaz.com/generators/ Thanks for the reference. I'm nearly there with the mix of return codes and output values, using subprocess. p1 = subprocess.Popen(['identify', '-format' , '%[fx:w]' , f ], stdout=subprocess.PIPE) resp1= p1.communicate() #print dir(p1) p2 = subprocess.Popen(['identify', '-format' , '%[fx:h]' , f ], stdout=subprocess.PIPE) resp2= p2.communicate() if (p1.returncode or p2.returncode): print Error ,p1.returncode sys.exit(2) else: width=int(resp1[0]) height=int(resp2[0]) print Height %s, width %s % (height, width) Not happy with the error handling yet, but this was the bit that I thought I'd struggle with. regards -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ. http://www.dpawson.co.uk -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 06/28/2010 12:05 PM, Dave Pawson wrote: On 28 June 2010 18:39, Michael Torrie torr...@gmail.com wrote: Sure. I've created a module called runcmd that does 90% of what I want (easy access to stdout, stderr, error code). I've attached it to this e-mail. Feel free to use it; this post puts my code into the public domain. Request please Michael. Examples of usage in the docstring? Well the simplest form is as I showed in my e-mail: (err,stdout,stderr) = runcmd.run (['command','arg1','arg2', ... ]) An example where you need to pass stdin: input=some string\n (err,stdout,stderr) = runcmd.run (['cmd','arg1',], stdin=input) err is the return error code (int) and stdout and stderr are the strings the result from the process's output, both strings. There are some other features (or bugs probably) that you can get from the source code itself. I think at one time I wanted to be able to have callbacks that would do something with stdout and stderr (in a real-time, unbuffered way), but I have never used them and don't think they work. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
John Nagle na...@animats.com writes: Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. That's interesting but I'm having a hard time seeing how it would work. I think environment variables didn't exist in early versions of Unix, and argc/argv were passed to the child process on its stack. I guess the reverse side could involve the wait system call taking a callback parameter with a buffer to receive the returned data. But that still only happens when the child actually exits, and presumably intercommunicating netween programs should be bidirectional. But Unix has always had pipes for that. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
Paul Rubin no.em...@nospam.invalid wrote in news:7xpqzbj8st@ruckus.brouhaha.com: Re: Python as a scripting language. Alternative to bash script? That's interesting but I'm having a hard time seeing how it would work. I think environment variables didn't exist in early versions of Unix, and argc/argv were passed to the child process on its stack. I guess the reverse side could involve the wait system call taking a callback parameter with a buffer to receive the returned data. But that still only happens when the child actually exits, and presumably intercommunicating netween programs should be bidirectional. But Unix has always had pipes for that. I can't see Python as an alt. to bash. (As I recall) Python is much more object-oriented than bash, but also there are many commands (such as apt- get, etc.) that would need Python equivs. However, I can see Python being used as a scripting alt. to C. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 6/28/10 1:06 PM, Mithrandir wrote: Paul Rubinno.em...@nospam.invalid wrote in news:7xpqzbj8st@ruckus.brouhaha.com: Re: Python as a scripting language. Alternative to bash script? That's interesting but I'm having a hard time seeing how it would work. I think environment variables didn't exist in early versions of Unix, and argc/argv were passed to the child process on its stack. I guess the reverse side could involve the wait system call taking a callback parameter with a buffer to receive the returned data. But that still only happens when the child actually exits, and presumably intercommunicating netween programs should be bidirectional. But Unix has always had pipes for that. I can't see Python as an alt. to bash. (As I recall) Python is much more object-oriented than bash, but also there are many commands (such as apt- get, etc.) that would need Python equivs. However, I can see Python being used as a scripting alt. to C. Wait, what? o.O First, you don't really have to use any of the object-orientedness of Python. It doesn't push anything on you. Second, why would apt-get need a Python equiv? Just use apt-get. And, ... a scripting alternative to C? How's C a scripting anything, that you need an alternate for? :) I mean, scripting is what you use to push around C programs :) -- ... Stephen Hansen ... Also: Ixokai ... Mail: me+list/python (AT) ixokai (DOT) io ... Blog: http://meh.ixokai.io/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On 06/28/2010 02:06 PM, Mithrandir wrote: I can't see Python as an alt. to bash. (As I recall) Python is much more object-oriented than bash, but also there are many commands (such as apt- get, etc.) that would need Python equivs. However, I can see Python being used as a scripting alt. to C. OO is a plus for most things, in my book. As for commands they have *nothing* to do with Bash. apt-get is not a Bash command. By your logic tcsh or zsh would not be an alternate to bash, but in fact they are. I use python for shell scripting quite often now. Anytime one of my own Bash scripts exceeds 100 lines, I know it's time to switch it to python. Please read that link I posted a while back on how you can use generators in python to replace many of the things that piping to external commands did in Bash. There certainly are a few tasks that Bash is best at (chaining commands together through pipes), but often Python already has support for many of the things I'd use external commands and pipes in Bash for. Bash is designed for working down at the level of files, directories, and processes, but Python works pretty well too, if you make some abstraction modules like my runcmd module that I use extensively. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Mon, 28 Jun 2010 18:57:45 +0100, John Nagle na...@animats.com wrote: On 6/28/2010 7:58 AM, Benjamin Kaplan wrote: How does a program return anything other than an exit code? Ah, yes, the second biggest design mistake in UNIX. Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. That's not just a little harsh, given that I've never known any operating system that returned more than an integer exit code, it's blithely ignoring the Unix paradigm. It's stream-based processing; argv and the environment are better thought of as switches, controlling how a program treats stdin to produce stdout. Classic Unix programming is a matter of stringing a bunch of tools together with pipes to get the output you want. This isn't a great paradigm for GUIs (not without tweaking that hasn't really been done), but then again it was never meant to be. C was like that once. In the 1970s, all you could return was an int or a float. But that got fixed. Strangely, these facts are not unconnected. -- Rhodri James *-* Wildebeeste Herder to the Masses -- http://mail.python.org/mailman/listinfo/python-list
Re: Python as a scripting language. Alternative to bash script?
On Mon, 28 Jun 2010 23:12:47 +0100, Rhodri James wrote: On Mon, 28 Jun 2010 18:57:45 +0100, John Nagle na...@animats.com wrote: On 6/28/2010 7:58 AM, Benjamin Kaplan wrote: How does a program return anything other than an exit code? Ah, yes, the second biggest design mistake in UNIX. Programs have argv and argc, plus environment variables, going in. So, going in, there are essentially subroutine parameters. But all that comes back is an exit code. They should have had something similar coming back, with arguments to exit() returning the results. Then the many small intercommunicating programs concept would have worked much better. That's not just a little harsh, given that I've never known any operating system that returned more than an integer exit code, it's blithely ignoring the Unix paradigm. It's stream-based processing; argv and the environment are better thought of as switches, controlling how a program treats stdin to produce stdout. Classic Unix programming is a matter of stringing a bunch of tools together with pipes to get the output you want. This isn't a great paradigm for GUIs (not without tweaking that hasn't really been done), but then again it was never meant to be. 'zactly. Arguements aren't data. They're essentially passing data by reference. Some unix utilities allow you to specify output where to put output (that is, an output file); some only to one output stream. -- I didn't need to sabotage anything. Not being around to say No that won't work or you can't do it that way is more than enough damage. (Ego problem? It's not a problem.) -- Graham Reed, on job endings -- http://mail.python.org/mailman/listinfo/python-list
