piping with subprocess
I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Thanks RD -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
Rick Dooling wrote: I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. It is always a good idea to post your best effort failed attempt, if only to give us an idea of your level of expertise. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Try to convert the example from the above page output=`dmesg | grep hda` # becomes p1 = Popen([dmesg], stdout=PIPE) p2 = Popen([grep, hda], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] to your usecase. Namely, replace [dmesg] -- [textutil, -convert, html, infile, -stdout] [grep, hda] -- [pandoc, -f, html, -t, marktown, -o, outfile] Don't forget to set infile = ... outfile = ... to filenames (with absolute paths, to avoid one source of error). If that doesn't work post the code you wrote along with the error messages. -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
Try this: from subprocess import check_output import sys check_output(textutil -convert html %s -stdout | pandoc -f html -t markdown -o %s % sys.argv[1:3], shell=True) On Sat, Feb 1, 2014 at 7:19 AM, Rick Dooling rpdool...@gmail.com wrote: I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Thanks RD -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote: Rick Dooling wrote: I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. It is always a good idea to post your best effort failed attempt, if only to give us an idea of your level of expertise. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Try to convert the example from the above page output=`dmesg | grep hda` # becomes p1 = Popen([dmesg], stdout=PIPE) p2 = Popen([grep, hda], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] to your usecase. Namely, replace [dmesg] -- [textutil, -convert, html, infile, -stdout] [grep, hda] -- [pandoc, -f, html, -t, marktown, -o, outfile] Don't forget to set infile = ... outfile = ... to filenames (with absolute paths, to avoid one source of error). If that doesn't work post the code you wrote along with the error messages. p1 = subprocess.Popen([textutil, -convert, html, file], stdout=subprocess.PIPE) p2 = subprocess.check_call([pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Errors Traceback (most recent call last): File /Users/me/Python/any2pandoc.py, line 70, in module convert_word_file(file, markdown_file) File /Users/me/Python/any2pandoc.py, line 59, in convert_word_file output = p2.communicate()[0] AttributeError: 'int' object has no attribute 'communicate' I get a markdown_file created but it's empty. Thanks, RD ps - Daniel's works fine but I still don't learn to pipe :) -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
On Saturday, February 1, 2014 7:54:34 AM UTC-6, Rick Dooling wrote: On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote: Rick Dooling wrote: I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. It is always a good idea to post your best effort failed attempt, if only to give us an idea of your level of expertise. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Try to convert the example from the above page output=`dmesg | grep hda` # becomes p1 = Popen([dmesg], stdout=PIPE) p2 = Popen([grep, hda], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] to your usecase. Namely, replace [dmesg] -- [textutil, -convert, html, infile, -stdout] [grep, hda] -- [pandoc, -f, html, -t, marktown, -o, outfile] Don't forget to set infile = ... outfile = ... to filenames (with absolute paths, to avoid one source of error). If that doesn't work post the code you wrote along with the error messages. p1 = subprocess.Popen([textutil, -convert, html, file], stdout=subprocess.PIPE) p2 = subprocess.check_call([pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Errors Traceback (most recent call last): File /Users/me/Python/any2pandoc.py, line 70, in module convert_word_file(file, markdown_file) File /Users/me/Python/any2pandoc.py, line 59, in convert_word_file output = p2.communicate()[0] AttributeError: 'int' object has no attribute 'communicate' I get a markdown_file created but it's empty. Thanks, RD ps - Daniel's works fine but I still don't learn to pipe :) Okay, sorry. I fixed that obvious goof p1 = subprocess.Popen([textutil, -convert, html, file], stdout=subprocess.PIPE) p2 = subprocess.Popen([pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Now I get no errors, but I still get a blank markdown file. -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
On 01/02/2014 13:54, Rick Dooling wrote: On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote: Rick Dooling wrote: I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. It is always a good idea to post your best effort failed attempt, if only to give us an idea of your level of expertise. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Try to convert the example from the above page output=`dmesg | grep hda` # becomes p1 = Popen([dmesg], stdout=PIPE) p2 = Popen([grep, hda], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] to your usecase. Namely, replace [dmesg] -- [textutil, -convert, html, infile, -stdout] [grep, hda] -- [pandoc, -f, html, -t, marktown, -o, outfile] Don't forget to set infile = ... outfile = ... to filenames (with absolute paths, to avoid one source of error). If that doesn't work post the code you wrote along with the error messages. Would you please read and action this https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing the double line spacing above, thanks. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
On Saturday, February 1, 2014 8:00:59 AM UTC-6, Rick Dooling wrote: On Saturday, February 1, 2014 7:54:34 AM UTC-6, Rick Dooling wrote: On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote: Rick Dooling wrote: I spent half a day trying to convert this bash script (on Mac) textutil -convert html $1 -stdout | pandoc -f html -t markdown -o $2 into Python using subprocess pipes. It works if I save the above into a shell script called convert.sh and then do subprocess.check_call([convert.sh, file, markdown_file]) where file and markdown_file are variables. But otherwise my piping attempts fail. It is always a good idea to post your best effort failed attempt, if only to give us an idea of your level of expertise. Could someone show me how to pipe in subprocess. Yes, I've read the doc, especially http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline But I'm a feeble hobbyist, not a computer scientist. Try to convert the example from the above page output=`dmesg | grep hda` # becomes p1 = Popen([dmesg], stdout=PIPE) p2 = Popen([grep, hda], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] to your usecase. Namely, replace [dmesg] -- [textutil, -convert, html, infile, -stdout] [grep, hda] -- [pandoc, -f, html, -t, marktown, -o outfile] Don't forget to set infile = ... outfile = ... to filenames (with absolute paths, to avoid one source of error). If that doesn't work post the code you wrote along with the error messages. p1 = subprocess.Popen([textutil, -convert, html, file], stdout=subprocess.PIPE) p2 = subprocess.check_call([pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Errors Traceback (most recent call last): File /Users/me/Python/any2pandoc.py, line 70, in module convert_word_file(file, markdown_file) File /Users/me/Python/any2pandoc.py, line 59, in convert_word_file output = p2.communicate()[0] AttributeError: 'int' object has no attribute 'communicate' I get a markdown_file created but it's empty. Thanks, RD ps - Daniel's works fine but I still don't learn to pipe :) Okay, sorry. I fixed that obvious goof p1 = subprocess.Popen([textutil, -convert, html, file], stdout=subprocess.PIPE) p2 = subprocess.Popen([pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Now I get no errors, but I still get a blank markdown file. Okay, blank lines removed. Apologies. I didn't know Google inserted them. RD -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
On 01/02/2014 15:35, Rick Dooling wrote: Okay, blank lines removed. Apologies. I didn't know Google inserted them. RD No problem, the whole snag is people don't know about this flaw in this tool until they're told about it. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: piping with subprocess
Rick Dooling wrote: On Saturday, February 1, 2014 6:54:09 AM UTC-6, Peter Otten wrote: Try to convert the example from the above page output=`dmesg | grep hda` # becomes p1 = Popen([dmesg], stdout=PIPE) p2 = Popen([grep, hda], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] to your usecase. Namely, replace [dmesg] -- [textutil, -convert, html, infile, -stdout] [grep, hda] -- [pandoc, -f, html, -t, marktown, -o, outfile] Don't forget to set infile = ... outfile = ... to filenames (with absolute paths, to avoid one source of error). If that doesn't work post the code you wrote along with the error messages. p1 = subprocess.Popen([textutil, -convert, html, file], stdout=subprocess.PIPE) p2 = subprocess.check_call([pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] Errors Traceback (most recent call last): File /Users/me/Python/any2pandoc.py, line 70, in module convert_word_file(file, markdown_file) File /Users/me/Python/any2pandoc.py, line 59, in convert_word_file output = p2.communicate()[0] AttributeError: 'int' object has no attribute 'communicate' I get a markdown_file created but it's empty. Well, you replaced the Popen() from the example with a check_call() which uses a Popen instance internally, but does not expose it. I recommend that you stick as closely to the example as possible until you have a working baseline version. I'd try textutil = subprocess.Popen( [textutil, -convert, html, file], stdout=subprocess.PIPE) pandoc = subprocess.Popen( [pandoc, -f, html, -t, markdown, -o, markdown_file], stdin=textutil.stdout) textutil.stdout.close() pandoc.communicate() -- https://mail.python.org/mailman/listinfo/python-list
