Re: Call a shell command from Python

[Bob Martin]

in 767198 20161104 142132 Thomas 'PointedEars' Lahn  wrote:
>Ben Finney wrote:
>
>> Note that ‘sudo’ is specifically designed to be invoked interactively,
>
>Nonsense.
>
>> seeking to verify that the current user has credentials to run the
>> command.
>
>NOPASSWD is not the default in sudoers(5),

It is on the Raspberry Pi

 but conclude from that what you
>concluded is a bit far-fetched, to say the least.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Wildman via Python-list]

On Tue, 01 Nov 2016 11:23:09 -0400, D'Arcy Cain wrote:

> On 2016-11-01 01:23 AM, Ben Finney wrote:
>> Wildman via Python-list  writes:
>> So the way your script was invoked has no bearing on whether Bash will
>> get involved in what your script does. Your script is *directly*
>> invoking programs, and if you don't ask for a shell to be involved you
>> won't get it.
> 
> In other words, the OP's script *is* bash in this scenario or a 
> reasonable facsimile thereof.
> 
> The subject probably should have been "Emulating a shell in Python."  Of 
> course, if the OP knew to use that subject he probably would have 
> already had the answer.  :-)

You are absolutely correct.  And thank you for your understanding.

-- 
 GNU/Linux user #557453
Keyboard not detected! Press any key to continue...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[D'Arcy Cain]

On 2016-11-01 01:23 AM, Ben Finney wrote:

Wildman via Python-list  writes:
So the way your script was invoked has no bearing on whether Bash will
get involved in what your script does. Your script is *directly*
invoking programs, and if you don't ask for a shell to be involved you
won't get it.


In other words, the OP's script *is* bash in this scenario or a 
reasonable facsimile thereof.


The subject probably should have been "Emulating a shell in Python."  Of 
course, if the OP knew to use that subject he probably would have 
already had the answer.  :-)


--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:da...@vex.net
VoIP: sip:da...@vex.net
--
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Wildman via Python-list]

On Tue, 01 Nov 2016 13:42:03 +, Grant Edwards wrote:

> On 2016-11-01, Steve D'Aprano  wrote:
>> On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:
>>
>>> You are correct about that but, in this case grep never "sees" the '$'
>>> sign.  Bash expands $USER to the actual user name beforehand.  If you
>>> are on a Linux system, enter this into a terminal to illustrate:
>>> 
>>> sudo grep ^$USER\: /etc/shadow
>>
>> Bash is not involved here. Python is calling grep directly.
> 
> He's already been told this 6 times, and is willfully ignoring it.
> 
> Just give up.

That was not my intention.  I was trying to understand that
which I did not understand.  Ever been there?

-- 
 GNU/Linux user #557453
The cow died so I don't need your bull!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Wildman via Python-list]

On Tue, 01 Nov 2016 16:52:18 +1100, Steve D'Aprano wrote:

> On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:
> 
>> You are correct about that but, in this case grep never "sees" the '$'
>> sign.  Bash expands $USER to the actual user name beforehand.  If you
>> are on a Linux system, enter this into a terminal to illustrate:
>> 
>> sudo grep ^$USER\: /etc/shadow
> 
> Bash is not involved here. Python is calling grep directly.
> 
> 
> You don't have to believe us, you can test this yourself. Create a simple
> text file with a single line containing your username, and a simple Python
> script that calls grep as you have been:
> 
> 
> [steve@ando ~]$ echo $USER
> steve
> [steve@ando ~]$ cat foo.txt
> blah blah steve blah blah
> [steve@ando ~]$ cat greptest.py
> import subprocess
> cmdlist = ['grep', '$USER', 'foo.txt']
> p = subprocess.Popen(cmdlist, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
> line, err = p.communicate()
> print err, line
> 
> [steve@ando ~]$ python2.7 greptest.py
> 
> [steve@ando ~]$
> 
> 
> 
> So there you have it: categorical proof that bash does not expand the
> string '$USER'. It cannot: bash is not involved in the subprocess call.
> Python calls grep directly.
> 
> If you want to expand the '$USER' string from Python, do it yourself:

I understand now.  That explains more clearly why my original
code did not work.  Thank you.

-- 
 GNU/Linux user #557453
The cow died so I don't need your bull!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Wildman via Python-list]

On Tue, 01 Nov 2016 16:23:08 +1100, Ben Finney wrote:

> Wildman via Python-list  writes:
> 
>> […] in this case grep never "sees" the '$' sign. Bash expands $USER to
>> the actual user name beforehand.
> 
> I understand how Bash substitutes variables on the command line.
> 
> What I need to repeat, though: In this case, no, Bash doesn't do that
> because Bash isn't getting involved. Grep does in fact see the “$” in
> the command argument, because nothing ever replaces it.

I understand now.  Thank you.

-- 
 GNU/Linux user #557453
The cow died so I don't need your bull!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Grant Edwards]

On 2016-11-01, Steve D'Aprano  wrote:
> On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:
>
>> You are correct about that but, in this case grep never "sees" the '$'
>> sign.  Bash expands $USER to the actual user name beforehand.  If you
>> are on a Linux system, enter this into a terminal to illustrate:
>> 
>> sudo grep ^$USER\: /etc/shadow
>
> Bash is not involved here. Python is calling grep directly.

He's already been told this 6 times, and is willfully ignoring it.

Just give up.

-- 
Grant Edwards   grant.b.edwardsYow! With YOU, I can be
  at   MYSELF ...  We don't NEED
  gmail.comDan Rather ...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Steve D'Aprano]

On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:

> You are correct about that but, in this case grep never "sees" the '$'
> sign.  Bash expands $USER to the actual user name beforehand.  If you
> are on a Linux system, enter this into a terminal to illustrate:
> 
> sudo grep ^$USER\: /etc/shadow

Bash is not involved here. Python is calling grep directly.


You don't have to believe us, you can test this yourself. Create a simple
text file with a single line containing your username, and a simple Python
script that calls grep as you have been:


[steve@ando ~]$ echo $USER
steve
[steve@ando ~]$ cat foo.txt
blah blah steve blah blah
[steve@ando ~]$ cat greptest.py
import subprocess
cmdlist = ['grep', '$USER', 'foo.txt']
p = subprocess.Popen(cmdlist, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
line, err = p.communicate()
print err, line

[steve@ando ~]$ python2.7 greptest.py

[steve@ando ~]$



So there you have it: categorical proof that bash does not expand the
string '$USER'. It cannot: bash is not involved in the subprocess call.
Python calls grep directly.

If you want to expand the '$USER' string from Python, do it yourself:


py> import os
py> os.environ['USER']
'steve'


> If the user name is ben then grep would see this:
> 
> grep ben\: /etc/shadow

If would, if you called grep from bash. But you didn't.


>>> > Maybe you are expecting Bash to be involved somehow (and so “$USER”
>>> > will be substituted by Bash with some other value). That's not what
>>> > happens.
>>>
>>> No, the shell is already running.
>> 
>> I don't know what you mean by this. If you mean that some *other*
>> instances of the shell ar running: that isn't relevant to how your
>> Python program invokes a subprocess.
> 
> I simply meant that the script is run from a terminal.

That's irrelevant. Just because the script is running from a terminal
doesn't mean that the shell can peer deep inside each and every process and
magically apply the shell's string expansion rules.





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Ben Finney]

Wildman via Python-list  writes:

> […] in this case grep never "sees" the '$' sign. Bash expands $USER to
> the actual user name beforehand.

I understand how Bash substitutes variables on the command line.

What I need to repeat, though: In this case, no, Bash doesn't do that
because Bash isn't getting involved. Grep does in fact see the “$” in
the command argument, because nothing ever replaces it.

> >> No, the shell is already running.
> > 
> > I don't know what you mean by this.
>
> I simply meant that the script is run from a terminal.

Which means that shell isn't involved in that command you're invoking
from inside the script.

The Bash instance which invoked your script is now *doing nothing*, and
will never do anything again until control returns to it (by your script
exiting, or suspending, or some other control interruption).

Your script invoking further programs will *not* get the earlier Bash
involved in that process (except in the special cases where those
programs themselves manipulate running processes; ‘sudo’ and ‘grep’ are
not special in this way).

So the way your script was invoked has no bearing on whether Bash will
get involved in what your script does. Your script is *directly*
invoking programs, and if you don't ask for a shell to be involved you
won't get it.

-- 
 \  “… a Microsoft Certified System Engineer is to information |
  `\ technology as a McDonalds Certified Food Specialist is to the |
_o__)   culinary arts.” —Michael Bacarella |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Wildman via Python-list]

On Tue, 01 Nov 2016 12:08:52 +1100, Ben Finney wrote:

> Wildman via Python-list  writes:
> 
>> On Mon, 31 Oct 2016 15:44:13 +1100, Ben Finney wrote:
>>
>> > One immediate difference I see is that you specify different
>> > arguments to ‘grep’. You have a different pattern for each command.
>> > 
>> > * The ‘^user\:’ pattern matches “user\:” at the start of a line.
>> > 
>> > * The ‘^$USER\:’ pattern I think won't match anything, since “$” matches
>> >   end-of-line and then you expect further characters *past* the end of
>> >   the line. I think that will always fail to match any line.
>>
>> Yes, the '^' indicates the start of the line and the ':' indicates
>> the character where to stop.  The colon has a special meaning so it
>> has to be escaped, '\:'.  The dollar sign precedes a variable.  In
>> this case it is an environment variable.
> 
> The ‘grep’ program you're invoking knows nothing of such variables, and
> the ‘$’ sign means to ‘grep’ what I said above.

You are correct about that but, in this case grep never "sees" the '$'
sign.  Bash expands $USER to the actual user name beforehand.  If you
are on a Linux system, enter this into a terminal to illustrate:

sudo grep ^$USER\: /etc/shadow

If the user name is ben then grep would see this:

grep ben\: /etc/shadow

>> > Maybe you are expecting Bash to be involved somehow (and so “$USER”
>> > will be substituted by Bash with some other value). That's not what
>> > happens.
>>
>> No, the shell is already running.
> 
> I don't know what you mean by this. If you mean that some *other*
> instances of the shell ar running: that isn't relevant to how your
> Python program invokes a subprocess.

I simply meant that the script is run from a terminal.

> The shell is not involved in the command as you invoke it directly as a
> subprocess, without asking for a shell.
> 
>> And $USER will be substituted by the name of the user that invoked the
>> shell.
> 
> It will not, because there is no shell involved: your Python program
> invokes ‘sudo’, which invokes ‘grep’. The shell is never involved in
> that chain, so its substitutions rules are irrelevant.

I think my terminology is causing confusion.  I apologize for that. 

-- 
 GNU/Linux user #557453
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python

[Ben Finney]

Wildman via Python-list  writes:

> On Mon, 31 Oct 2016 15:44:13 +1100, Ben Finney wrote:
>
> > One immediate difference I see is that you specify different
> > arguments to ‘grep’. You have a different pattern for each command.
> > 
> > * The ‘^user\:’ pattern matches “user\:” at the start of a line.
> > 
> > * The ‘^$USER\:’ pattern I think won't match anything, since “$” matches
> >   end-of-line and then you expect further characters *past* the end of
> >   the line. I think that will always fail to match any line.
>
> Yes, the '^' indicates the start of the line and the ':' indicates
> the character where to stop.  The colon has a special meaning so it
> has to be escaped, '\:'.  The dollar sign precedes a variable.  In
> this case it is an environment variable.

The ‘grep’ program you're invoking knows nothing of such variables, and
the ‘$’ sign means to ‘grep’ what I said above.

> > Maybe you are expecting Bash to be involved somehow (and so “$USER”
> > will be substituted by Bash with some other value). That's not what
> > happens.
>
> No, the shell is already running.

I don't know what you mean by this. If you mean that some *other*
instances of the shell ar running: that isn't relevant to how your
Python program invokes a subprocess.

The shell is not involved in the command as you invoke it directly as a
subprocess, without asking for a shell.

> And $USER will be substituted by the name of the user that invoked the
> shell.

It will not, because there is no shell involved: your Python program
invokes ‘sudo’, which invokes ‘grep’. The shell is never involved in
that chain, so its substitutions rules are irrelevant.

-- 
 \ “Try adding “as long as you don't breach the terms of service – |
  `\  according to our sole judgement” to the end of any cloud |
_o__)  computing pitch.” —Simon Phipps, 2010-12-11 |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python (was: Calling Bash Command From Python)

[Grant Edwards]

On 2016-10-31, Wildman via Python-list  wrote:
> On Mon, 31 Oct 2016 15:44:13 +1100, Ben Finney wrote:
>> Wildman via Python-list  writes:
>> 
>>> Python 2.7.9 on Linux
>>>
>>> Here is a bash command that I want to run from a python
>>> program:  sudo grep "^user\:" /etc/shadow
>> 
>> Some points to note:
>> 
>> * Those commands are not special to Bash, or any particular shell. They
>>   invoke commands, without AFAIK any specific Bash features. So this is
>>   asking rather to invoke a shell command.

Actually it's not a shell command either.

> Yes, I know.  I perhaps should have used the word "shell" instead
> of bash.  However, bash is a shell.

To most people, "bash command" or "shell command" refers to

 1) a commands built in to bash (or other shell)

 or

 2) a command line that must be interpreted by bash (or other shell) because it
uses I/O redirection, variable substituion, globbing, etc.

Your code does not use bash (or any other shell), nor does it need to.

It's just running the sudo executable.  There are no shells involved,
so using the word shell isn't any better than using the word bash.

-- 
Grant Edwards   grant.b.edwardsYow! I have seen these EGG
  at   EXTENDERS in my Supermarket
  gmail.com... I have read the
   INSTRUCTIONS ...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python (was: Calling Bash Command From Python)

[Wildman via Python-list]

On Mon, 31 Oct 2016 15:44:13 +1100, Ben Finney wrote:

> Wildman via Python-list  writes:
> 
>> Python 2.7.9 on Linux
>>
>> Here is a bash command that I want to run from a python
>> program:  sudo grep "^user\:" /etc/shadow
> 
> Some points to note:
> 
> * Those commands are not special to Bash, or any particular shell. They
>   invoke commands, without AFAIK any specific Bash features. So this is
>   asking rather to invoke a shell command.

Yes, I know.  I perhaps should have used the word "shell" instead
of bash.  However, bash is a shell.

>   Nothing wrong with that; but on that basis, I've changed the subject
>   field.
> 
> * You're asking to invoke the ‘sudo’ command, which itself is designed
>   to switch to a separate user identity and run another program.

Yes, I know.

> * The above command is (I assume) typed into a shell, but your Python
>   program never invokes Bash or any other shell.

The program is run in a shell so invocation is not needed.

>> If I enter the command directly into a terminal it works perfectly.
> 
> Note that ‘sudo’ is specifically designed to be invoked interactively,
> seeking to verify that the current user has credentials to run the
> command.
> 
> Note further that ‘sudo’ will record when the *current user session*
> last invoked ‘sudo’ and seek re-verification if that is too long in the
> past.
> 
> Both of these are security measures, and are designed to avoid
> non-interactive use of ‘sudo’. Rather, it's meant to be used
> interactively by a real, present human with credentials to run the
> command.

Yes, I know all that.

>> If I run it from a python program it returns an empty string.
> 
> You can also check the exit status of a command; ‘grep’ will give
> different exit status for a match versus no match.
> 
>> Below is the code I am using.  Suggestions
>> appreciated.
>>
>> cmdlist = ["sudo", "grep", '"^$USER\:"', "/etc/shadow"]
> 
> One immediate difference I see is that you specify different arguments
> to ‘grep’. You have a different pattern for each command.
> 
> * The ‘^user\:’ pattern matches “user\:” at the start of a line.
> 
> * The ‘^$USER\:’ pattern I think won't match anything, since “$” matches
>   end-of-line and then you expect further characters *past* the end of
>   the line. I think that will always fail to match any line.

Yes, the '^' indicates the start of the line and the ':' indicates
the character where to stop.  The colon has a special meaning so it
has to be escaped, '\:'.  The dollar sign precedes a variable.  In
this case it is an environment variable.

The Linux shadow file contains information about the users of the
system.  It has the user name, encrypted password, salt, encryption
type and other information.  The format is such that the user name
is at the start of the line ending with a colon.  Here is an example:

wildman:$6$hODbsJJp$/NWGXZ3fMIVB4U.v/oLtAv.CnL0l0I39.IwsDx1ZAlKW3wUSjTfwJdnQvOMpYNbqNqqFfZ52vgYWBmnjsaX9R.:16177:0:9:7:::


>> p = subprocess.Popen(cmdlist,
>>  stdout=subprocess.PIPE,
>>  stderr=subprocess.PIPE)
>> shadow, err = p.communicate()
> 
> Maybe you are expecting Bash to be involved somehow (and so “$USER” will
> be substituted by Bash with some other value). That's not what happens.

No, the shell is already running.  And $USER will be substituted by the
name of the user that invoked the shell.

> Instead, the ‘subprocess.Popen.communicate’ method will invoke the
> program directly, without involving a shell. See the documentation for
> ‘subprocess.Popen’.

I will look into that.  Thanks for the reply.

-- 
 GNU/Linux user #557453
The cow died so I don't need your bull!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call a shell command from Python (was: Calling Bash Command From Python)

[Chris Angelico]

On Mon, Oct 31, 2016 at 3:44 PM, Ben Finney  wrote:
> Note that ‘sudo’ is specifically designed to be invoked interactively,
> seeking to verify that the current user has credentials to run the
> command.
>
> Note further that ‘sudo’ will record when the *current user session*
> last invoked ‘sudo’ and seek re-verification if that is too long in the
> past.
>
> Both of these are security measures, and are designed to avoid
> non-interactive use of ‘sudo’. Rather, it's meant to be used
> interactively by a real, present human with credentials to run the
> command.

I don't know that non-interactive sudo is so bad a thing. In fact,
sudo has a --non-interactive option that appears specifically designed
for this kind of thing - it causes the command to fail rather than
prompt. You can configure a sudoers file to allow passwordless
execution of specific commands, and then permit scripts to elevate
privileges in very limited ways, safely.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Call a shell command from Python (was: Calling Bash Command From Python)

[Ben Finney]

Wildman via Python-list  writes:

> Python 2.7.9 on Linux
>
> Here is a bash command that I want to run from a python
> program:  sudo grep "^user\:" /etc/shadow

Some points to note:

* Those commands are not special to Bash, or any particular shell. They
  invoke commands, without AFAIK any specific Bash features. So this is
  asking rather to invoke a shell command.

  Nothing wrong with that; but on that basis, I've changed the subject
  field.

* You're asking to invoke the ‘sudo’ command, which itself is designed
  to switch to a separate user identity and run another program.

* The above command is (I assume) typed into a shell, but your Python
  program never invokes Bash or any other shell.

> If I enter the command directly into a terminal it works perfectly.

Note that ‘sudo’ is specifically designed to be invoked interactively,
seeking to verify that the current user has credentials to run the
command.

Note further that ‘sudo’ will record when the *current user session*
last invoked ‘sudo’ and seek re-verification if that is too long in the
past.

Both of these are security measures, and are designed to avoid
non-interactive use of ‘sudo’. Rather, it's meant to be used
interactively by a real, present human with credentials to run the
command.

> If I run it from a python program it returns an empty string.

You can also check the exit status of a command; ‘grep’ will give
different exit status for a match versus no match.

> Below is the code I am using.  Suggestions
> appreciated.
>
> cmdlist = ["sudo", "grep", '"^$USER\:"', "/etc/shadow"]

One immediate difference I see is that you specify different arguments
to ‘grep’. You have a different pattern for each command.

* The ‘^user\:’ pattern matches “user\:” at the start of a line.

* The ‘^$USER\:’ pattern I think won't match anything, since “$” matches
  end-of-line and then you expect further characters *past* the end of
  the line. I think that will always fail to match any line.

> p = subprocess.Popen(cmdlist,
>  stdout=subprocess.PIPE,
>  stderr=subprocess.PIPE)
> shadow, err = p.communicate()

Maybe you are expecting Bash to be involved somehow (and so “$USER” will
be substituted by Bash with some other value). That's not what happens.

Instead, the ‘subprocess.Popen.communicate’ method will invoke the
program directly, without involving a shell. See the documentation for
‘subprocess.Popen’.

-- 
 \“If you continue running Windows, your system may become |
  `\unstable.” —Microsoft, Windows 95 bluescreen error message |
_o__)  |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list