Re: line to argv transformation
On 16-06-14 13:09, Chris Angelico wrote: On Mon, Jun 16, 2014 at 8:51 PM, Tim Chase python.l...@tim.thechases.com wrote: On 2014-06-16 20:41, Chris Angelico wrote: Oops! I made the cardinal error of trying in one and assuming it'd work in both. Just needs a b prefix on the split string: def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(b\0) You'll get back a list of byte strings, in any case. Feel free to pass them through a decode operation, or to incorporate a .decode() into the above stream, as you wish. Tested on Win32? The behavior of shell expansion on Windows cmd.exe differs from most *nix shells (i.e., it *doesn't* expand, so you have to do it yourself), so Antoon would need to describe the desired behavior on Win32. Well, his example commands began ls, which is a common Unix command, but isn't present on most Windows systems. If he'd started out with something that looked more Windowsish, I'd totally understand - you start with a single line, and you need to do what the cmd.exe shell hasn't done for you. (Although splitting is done, so it still wouldn't be quite as clear.) But he said treated as a command line. So that's exactly what I did. :) He didn't ask about globbing, he asked about doing what the shell does... maybe he wants variable expansion too? That would be interresting too. The problem with your solution would be that it would only substitute environment variables, and not shell variables within the interactive session. Of course I could use os.putenv to put all those variables in the environment but that might have troublesome effects on other subprocesses. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
line to argv transformation
I am looking for an interface that takes a string as argument. The string is to be treated as if it is a command line and transformed into an argv list. ls file - ['ls', 'file'] ls *.py - ['ls', 'file1.py', 'file2.py', ...] ls '*.py' - ['ls', '*.py'] Does something like this already exist? I looked around but seem to find only things only partially do things like this, like shlex.split. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 7:56 PM, Antoon Pardon antoon.par...@rece.vub.ac.be wrote: I am looking for an interface that takes a string as argument. The string is to be treated as if it is a command line and transformed into an argv list. ls file - ['ls', 'file'] ls *.py - ['ls', 'file1.py', 'file2.py', ...] ls '*.py' - ['ls', '*.py'] Does something like this already exist? I looked around but seem to find only things only partially do things like this, like shlex.split. def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(\0) :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 16-06-14 12:06, Chris Angelico wrote: def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(\0) Nice idea, unfortunatly it doesn't work in python3.3 shell_split(ls *.py) Traceback (most recent call last): File stdin, line 1, in module File stdin, line 3, in shell_split TypeError: Type str doesn't support the buffer API -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 8:24 PM, Antoon Pardon antoon.par...@rece.vub.ac.be wrote: On 16-06-14 12:06, Chris Angelico wrote: def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(\0) Nice idea, unfortunatly it doesn't work in python3.3 shell_split(ls *.py) Traceback (most recent call last): File stdin, line 1, in module File stdin, line 3, in shell_split TypeError: Type str doesn't support the buffer API Oops! I made the cardinal error of trying in one and assuming it'd work in both. Just needs a b prefix on the split string: def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(b\0) You'll get back a list of byte strings, in any case. Feel free to pass them through a decode operation, or to incorporate a .decode() into the above stream, as you wish. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
Chris Angelico ros...@gmail.com:
def shell_split(cmd):
return subprocess.check_output(python -c 'import sys;
print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(b\0)
You'll get back a list of byte strings, in any case. Feel free to pass
them through a decode operation, or to incorporate a .decode() into
the above stream, as you wish.
Now you are subject to the quirks of /bin/sh. For example, '*.xyz'
expands itself ('*.xyz') if there is no match in the current working
directory.
Moreover, you need to guard against arguments like
$(cat ~/.ssh/id_dsa)
which would spill the beans.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
Antoon Pardon wrote: I am looking for an interface that takes a string as argument. The string is to be treated as if it is a command line and transformed into an argv list. ls file - ['ls', 'file'] ls *.py - ['ls', 'file1.py', 'file2.py', ...] ls '*.py' - ['ls', '*.py'] Does something like this already exist? I looked around but seem to find only things only partially do things like this, like shlex.split. You might combine shlex and glob: def parse_and_expand(s): parts = shlex.split(s) expanded = [] for part in parts: matches = glob.glob(part) if matches: expanded.extend(sorted(matches)) else: expanded.append(part) return expanded -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 8:51 PM, Tim Chase python.l...@tim.thechases.com wrote: On 2014-06-16 20:41, Chris Angelico wrote: Oops! I made the cardinal error of trying in one and assuming it'd work in both. Just needs a b prefix on the split string: def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(b\0) You'll get back a list of byte strings, in any case. Feel free to pass them through a decode operation, or to incorporate a .decode() into the above stream, as you wish. Tested on Win32? The behavior of shell expansion on Windows cmd.exe differs from most *nix shells (i.e., it *doesn't* expand, so you have to do it yourself), so Antoon would need to describe the desired behavior on Win32. Well, his example commands began ls, which is a common Unix command, but isn't present on most Windows systems. If he'd started out with something that looked more Windowsish, I'd totally understand - you start with a single line, and you need to do what the cmd.exe shell hasn't done for you. (Although splitting is done, so it still wouldn't be quite as clear.) But he said treated as a command line. So that's exactly what I did. :) He didn't ask about globbing, he asked about doing what the shell does... maybe he wants variable expansion too? shell_split(echo $DISPLAY) [b'echo', b'localhost:10.0'] (This is running over SSH, so my $DISPLAY is a perhaps unusual value.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 2014-06-16 20:41, Chris Angelico wrote: Oops! I made the cardinal error of trying in one and assuming it'd work in both. Just needs a b prefix on the split string: def shell_split(cmd): return subprocess.check_output(python -c 'import sys; print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(b\0) You'll get back a list of byte strings, in any case. Feel free to pass them through a decode operation, or to incorporate a .decode() into the above stream, as you wish. Tested on Win32? The behavior of shell expansion on Windows cmd.exe differs from most *nix shells (i.e., it *doesn't* expand, so you have to do it yourself), so Antoon would need to describe the desired behavior on Win32. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 8:59 PM, Marko Rauhamaa ma...@pacujo.net wrote:
Chris Angelico ros...@gmail.com:
def shell_split(cmd):
return subprocess.check_output(python -c 'import sys;
print(\\0.join(sys.argv[1:]))' +cmd,shell=True)[:-1].split(b\0)
You'll get back a list of byte strings, in any case. Feel free to pass
them through a decode operation, or to incorporate a .decode() into
the above stream, as you wish.
Now you are subject to the quirks of /bin/sh. For example, '*.xyz'
expands itself ('*.xyz') if there is no match in the current working
directory.
Moreover, you need to guard against arguments like
$(cat ~/.ssh/id_dsa)
which would spill the beans.
As I said in response to Tim, the somewhat underspecified question
does leave open followups of whether both of those would be features,
rather than bugs. For instance, if *.py should expand to a list of
all files matching that glob, should [123].py expand to any files
matching that pattern? I'm not sure that your typical glob function
handles that. And should spam{eggs,spam} become
spameggs,spamspam? (Though that one's bash-specific, I believe.)
Where do you draw the line?
My reading is that it should be one of two options:
1) Split, according to shell rules, and then glob. Nothing more. No
square brackets (probably), no braces, nothing.
2) Do exactly what $CHELL would do, for some value of $CHELL.
But neither is quite clear, and I can see exactly why there isn't
anything in the stdlib. And what shell do you want to imitate, for
option 2? Using /bin/sh makes a lot of sense... but so does /bin/bash.
Or maybe you should use the user's own login shell, if you're wrapping
a command prompt in some way. Perhaps you want to use GLaDOS; but
remember that although fun and learning are the primary goals of this
activity, serious injuries may occur, especially when using backticks
or $( ) in the command.
OP needs to specify better. Otherwise Black Mesa will get the contract.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 16-06-14 13:01, Peter Otten wrote: Antoon Pardon wrote: I am looking for an interface that takes a string as argument. The string is to be treated as if it is a command line and transformed into an argv list. ls file - ['ls', 'file'] ls *.py - ['ls', 'file1.py', 'file2.py', ...] ls '*.py' - ['ls', '*.py'] Does something like this already exist? I looked around but seem to find only things only partially do things like this, like shlex.split. You might combine shlex and glob: def parse_and_expand(s): parts = shlex.split(s) expanded = [] for part in parts: matches = glob.glob(part) if matches: expanded.extend(sorted(matches)) else: expanded.append(part) return expanded No that doesn't work because of this: shlex.split(ls *.py) ['ls', '*.py'] shlex.split(ls '*.py') ['ls', '*.py'] After the split you have no idea wether the glob pattern you see was quoted or not. -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 2014-06-16 13:51, Antoon Pardon wrote: shlex.split(ls *.py) ['ls', '*.py'] shlex.split(ls '*.py') ['ls', '*.py'] To accommodate this, I'd probably just clone the shlib.py to my local project under a new name and then tweak the source to emit whether a token was quoted or not, something like the diff below. You can then iterate over your string/token-stream and know whether it was quoted or not, allowing you to do any post-processing/globbing on that file. -tkc --- /usr/lib/python2.7/shlex.py 2014-03-13 05:54:53.0 -0500 +++ /home/tim/tmp/myshlex.py2014-06-16 07:39:34.130645270 -0500 @@ -93,29 +93,30 @@ print shlex: popping token + repr(tok) return tok # No pushback. Get a token. -raw = self.read_token() +was_quoted, raw = self.read_token() # Handle inclusions if self.source is not None: while raw == self.source: -spec = self.sourcehook(self.read_token()) +was_quoted, token = self.read_token() +spec = self.sourcehook(roken) if spec: (newfile, newstream) = spec self.push_source(newstream, newfile) -raw = self.get_token() +was_quoted, raw = self.get_token() # Maybe we got EOF instead? while raw == self.eof: if not self.filestack: return self.eof else: self.pop_source() -raw = self.get_token() +was_quoted, raw = self.get_token() # Neither inclusion nor EOF if self.debug = 1: if raw != self.eof: print shlex: token= + repr(raw) else: print shlex: token=EOF -return raw +return was_quoted, raw def read_token(self): quoted = False @@ -243,7 +244,7 @@ print shlex: raw token= + repr(result) else: print shlex: raw token=EOF -return result +return quoted, result def sourcehook(self, newfile): Hook called on a filename to be sourced. @@ -266,10 +267,10 @@ return self def next(self): -token = self.get_token() +was_quoted, token = self.get_token() if token == self.eof: raise StopIteration -return token +return was_quoted, token def split(s, comments=False, posix=True): lex = shlex(s, posix=posix) @@ -285,7 +286,7 @@ file = sys.argv[1] lexer = shlex(open(file), file) while 1: -tt = lexer.get_token() +was_quoted, tt = lexer.get_token() if tt: print Token: + repr(tt) else: -- https://mail.python.org/mailman/listinfo/python-list
