Executing python script stored as a string
Hello,
please, how to execute a python script stored as a string? But let me
impose several limitations, so simple exec wont work:
- if I understood it correctly defining a function in the string and
exec-ing it created the function in current scope. This is something I
really don't want
- simple exec also blocks the rest of the program
- I also would like the string to be able to use and return some parts
of the caller
So to give an example what I try to achieve:
result = []
def up(s):
result.append(s.upper())
code = '''
up(abc)
print 'hello'
i = i + 3
def x(s):
up(s)
x('def')
print i
'''
somehow_execute(code)
Couple of points:
- the script in string should behave just like any other ordinary
python script executed in separate process, except it should also know
about a function caller up. Nothing else. (I read that something
similar is possible while embedding python into your C project - that
you could invoke the VM and provide some default imports)
- if the other script runs in separate process how should it call the
remote function? And how to pass its arguments? I really hope I don't
have to serialize every communication, maybe I should use threading
instead of process? All I want is that running it wont block the
caller and that it cannot modify callers code/variables/scope (apart
from calling the predefined callers' functions). Or maybe even better,
let it block the caller but provide a way to stop its execution?
- how to know that the script finished? I was thinking about atexit()
- could it work here?
Think of it as a text editor with a special ability to execute its
content, while providing access of some of its functionality to the
script.
The reason I *think* I cannot just simple import the editor module
into the script is that theeditor is GUI application and script
should have access to just this instance of editor.
Anyway, I hope I was not too confusing. Thanks for any help!
--
http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Mon, 31 Aug 2009 16:29:45 -0700, Ecir Hana wrote:
Hello,
please, how to execute a python script stored as a string? But let me
impose several limitations, so simple exec wont work:
- if I understood it correctly defining a function in the string and
exec-ing it created the function in current scope. This is something I
really don't want
You can pass in a global and local namespaces to exec as arguments:
x = 4
ns = {'x': 4}
exec x += 1 in ns
x
4
ns['x']
5
See the docs for details.
- simple exec also blocks the rest of the program
Run it in a thread.
- I also would like the string to be able to use and return some parts
of the caller
You can copy the parts of the current scope into the namespace you pass
to exec, then later copy the revised values out again.
But are you sure you really want to take this approach? exec is up to ten
times slower than just executing the code directly. And if the string is
coming from an untrusted source, it is a *huge* security risk.
Couple of points:
- the script in string should behave just like any other ordinary python
script executed in separate process, except it should also know about a
function caller up. Nothing else. (I read that something similar is
possible while embedding python into your C project - that you could
invoke the VM and provide some default imports)
If you want it to execute in a separate *process*, that's a whole
different question. If you do that, you get separation of code for free,
as well as separate namespaces. My approach would be to have a special
module common which subprocesses can import, to get access to the
shared functions. You will probably need to create some sort of message
passing infrastructure to get results out of the subprocess into the
parent process.
- if the other script runs in separate process how should it call the
remote function? And how to pass its arguments? I really hope I don't
have to serialize every communication, maybe I should use threading
instead of process?
If you want separate processes, they're *separate*. Threads are not.
All I want is that running it wont block the caller
and that it cannot modify callers code/variables/scope (apart from
calling the predefined callers' functions). Or maybe even better, let it
block the caller but provide a way to stop its execution?
As far as I know, you can't kill threads, you can only ask them to kill
themselves.
- how to know that the script finished? I was thinking about atexit() -
could it work here?
I doubt it. You would need to poll each thread to see if it has completed.
Think of it as a text editor with a special ability to execute its
content, while providing access of some of its functionality to the
script.
Something like this?
In the text editor, you have contents:
text goes here
and more text
# Python script starts here
x = 'a'
up(x)
print foo
# Python script stops here
more text again
and the user selects lines 4 and 5 and chooses the command Execute. The
script executes, and its output (foo) is appended to the end of the file:
text goes here
and more text
# Python script starts here
x = 'a'
up(x)
print foo
# Python script stops here
more text again
foo
Is this what you mean?
If so, I think you are making this much too complicated for such a simple
use-case. Just publish an API which the script can use, and have the main
text editor application specify a script namespace containing only that
API. That could be a module:
import math # pretend this is your API shared module
exec myvalue = 42 in math.__dict__
math.myvalue
42
Then execute the text using exec, but don't bother about putting it into
a thread or subprocess. That just makes it harder to implement, and you
have to worry about concurrency issues.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Tue, Sep 1, 2009 at 9:29 AM, Ecir Hana ecir.h...@gmail.com wrote: - if I understood it correctly defining a function in the string and exec-ing it created the function in current scope. This is something I really don't want Oops, missed that point. May I ask what is there you don't want, and what about putting the function inside a different module and importing? What's so bad about being in scope? Sincerely, -Xav -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Sep 1, 5:31 am, Steven D'Aprano
ste...@remove.this.cybersource.com.au wrote:
You can pass in a global and local namespaces to exec as arguments:
x = 4
ns = {'x': 4}
exec x += 1 in ns
x
4
ns['x']
5
See the docs for details.
Thanks! This is very useful!
You can copy the parts of the current scope into the namespace you pass
to exec, then later copy the revised values out again.
But are you sure you really want to take this approach? exec is up to ten
times slower than just executing the code directly. And if the string is
coming from an untrusted source, it is a *huge* security risk.
I don't know if I should use exec. I don't really mind that it's slow
(btw., why is it so?). But I don't quite understand why is it security
risk. How is it different to run:
exec 'format(your_hdd)'
than:
/bin/python format.py
?
As far as I know, you can't kill threads, you can only ask them to kill
themselves.
Also, I'm not sure if I follow. What does this mean? If a thread runs:
while True:
pass
it is not possible to kill it from another thread? (Bacause it doesn't
check whether some other thread asks to stop it..?)
Something like this?
Well, something more like:
data = [1, 2, 3]
map(lambda x: x * 2, data)
display_data_in_editor_viewport(data) #this renders into part of main
editor window (may take some time)
If so, I think you are making this much too complicated for such a simple
use-case. Just publish an API which the script can use, and have the main
text editor application specify a script namespace containing only that
API. That could be a module:
import math # pretend this is your API shared module
exec myvalue = 42 in math.__dict__
math.myvalue
42
Then execute the text using exec, but don't bother about putting it into
a thread or subprocess. That just makes it harder to implement, and you
have to worry about concurrency issues.
Ok, I could try exec, thanks for the explanation. But what about those
security concerns you mentioned above?
Thanks a lot, very informative!
--
http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Tue, 01 Sep 2009 01:34:33 -0700, Ecir Hana wrote:
You can copy the parts of the current scope into the namespace you pass
to exec, then later copy the revised values out again.
But are you sure you really want to take this approach? exec is up to
ten times slower than just executing the code directly. And if the
string is coming from an untrusted source, it is a *huge* security
risk.
I don't know if I should use exec. I don't really mind that it's slow
(btw., why is it so?).
Because it has to parse and compile the string into a code object before
it can run it.
But I don't quite understand why is it security
risk. How is it different to run:
exec 'format(your_hdd)'
than:
/bin/python format.py
?
It's not different. But read what I said -- if the string is coming from
an UNTRUSTED source -- presumably you trust yourself. If you run 'exec
format(your_hdd)' it is because *you* want to format your hard disk.
Now imagine you have a web-app which gets a string from the user and
calls exec on it. Then you might have this:
exec search('%d') % user_input
and the user, who is halfway across the world, enters the following
search string:
places to eat'); import os; os.system('#rm -rf /
Your web app will go right ahead and erase itself. That's why you need to
keep untrusted strings away from exec, execfile, and eval.
As far as I know, you can't kill threads, you can only ask them to kill
themselves.
Also, I'm not sure if I follow. What does this mean? If a thread runs:
while True:
pass
it is not possible to kill it from another thread? (Bacause it doesn't
check whether some other thread asks to stop it..?)
No, I believe that the only way to halt that is to halt the entire
process.
Possibly there is a way to have a thread halt itself after a certain
amount of time? I'm not an expert on threads, I've hardly ever used them.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Sep 1, 11:32 am, Steven D'Aprano
ste...@remove.this.cybersource.com.au wrote:
But I don't quite understand why is it security
risk. How is it different to run:
exec 'format(your_hdd)'
than:
/bin/python format.py
?
It's not different. But read what I said -- if the string is coming from
an UNTRUSTED source -- presumably you trust yourself. If you run 'exec
format(your_hdd)' it is because *you* want to format your hard disk.
Now imagine you have a web-app which gets a string from the user and
calls exec on it. Then you might have this:
exec search('%d') % user_input
and the user, who is halfway across the world, enters the following
search string:
places to eat'); import os; os.system('#rm -rf /
Your web app will go right ahead and erase itself. That's why you need to
keep untrusted strings away from exec, execfile, and eval.
Ah, I see! Ok.
No, I believe that the only way to halt that is to halt the entire
process.
Possibly there is a way to have a thread halt itself after a certain
amount of time? I'm not an expert on threads, I've hardly ever used them.
Thank you once again!
--
http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Tuesday 01 September 2009 11:32:29 Steven D'Aprano wrote: Possibly there is a way to have a thread halt itself after a certain amount of time? I'm not an expert on threads, I've hardly ever used them. Not automagically, as far as I can see. You are on your own if you want to somehow kill a thread. What I do is to wrap what I want to do in a while clause: while runbool: do_what_you_want() where runbool is a global that I set to false elsewhere when I want to stop. There is of course nothing to stop you writing a thread with something like: import time start_time = time.time() while time.time() - start_time some_parameter: do_what_you_want() Which will have the effect of running for a while and then stop. I cannot see much use use for that as it is, as it will be bloody-minded about the time, and might chop off before the whole job is done, but one could make it more intelligent, such that it keeps track of idle time, and aborts after say a second (or a minute) of not doing anything useful. Unlike you, I am a thread and queue fanatic - use them all the time. - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
