Executing python script stored as a string
Hello, please, how to execute a python script stored as a string? But let me impose several limitations, so simple exec wont work: - if I understood it correctly defining a function in the string and exec-ing it created the function in current scope. This is something I really don't want - simple exec also blocks the rest of the program - I also would like the string to be able to use and return some parts of the caller So to give an example what I try to achieve: result = [] def up(s): result.append(s.upper()) code = ''' up(abc) print 'hello' i = i + 3 def x(s): up(s) x('def') print i ''' somehow_execute(code) Couple of points: - the script in string should behave just like any other ordinary python script executed in separate process, except it should also know about a function caller up. Nothing else. (I read that something similar is possible while embedding python into your C project - that you could invoke the VM and provide some default imports) - if the other script runs in separate process how should it call the remote function? And how to pass its arguments? I really hope I don't have to serialize every communication, maybe I should use threading instead of process? All I want is that running it wont block the caller and that it cannot modify callers code/variables/scope (apart from calling the predefined callers' functions). Or maybe even better, let it block the caller but provide a way to stop its execution? - how to know that the script finished? I was thinking about atexit() - could it work here? Think of it as a text editor with a special ability to execute its content, while providing access of some of its functionality to the script. The reason I *think* I cannot just simple import the editor module into the script is that theeditor is GUI application and script should have access to just this instance of editor. Anyway, I hope I was not too confusing. Thanks for any help! -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Mon, 31 Aug 2009 16:29:45 -0700, Ecir Hana wrote: Hello, please, how to execute a python script stored as a string? But let me impose several limitations, so simple exec wont work: - if I understood it correctly defining a function in the string and exec-ing it created the function in current scope. This is something I really don't want You can pass in a global and local namespaces to exec as arguments: x = 4 ns = {'x': 4} exec x += 1 in ns x 4 ns['x'] 5 See the docs for details. - simple exec also blocks the rest of the program Run it in a thread. - I also would like the string to be able to use and return some parts of the caller You can copy the parts of the current scope into the namespace you pass to exec, then later copy the revised values out again. But are you sure you really want to take this approach? exec is up to ten times slower than just executing the code directly. And if the string is coming from an untrusted source, it is a *huge* security risk. Couple of points: - the script in string should behave just like any other ordinary python script executed in separate process, except it should also know about a function caller up. Nothing else. (I read that something similar is possible while embedding python into your C project - that you could invoke the VM and provide some default imports) If you want it to execute in a separate *process*, that's a whole different question. If you do that, you get separation of code for free, as well as separate namespaces. My approach would be to have a special module common which subprocesses can import, to get access to the shared functions. You will probably need to create some sort of message passing infrastructure to get results out of the subprocess into the parent process. - if the other script runs in separate process how should it call the remote function? And how to pass its arguments? I really hope I don't have to serialize every communication, maybe I should use threading instead of process? If you want separate processes, they're *separate*. Threads are not. All I want is that running it wont block the caller and that it cannot modify callers code/variables/scope (apart from calling the predefined callers' functions). Or maybe even better, let it block the caller but provide a way to stop its execution? As far as I know, you can't kill threads, you can only ask them to kill themselves. - how to know that the script finished? I was thinking about atexit() - could it work here? I doubt it. You would need to poll each thread to see if it has completed. Think of it as a text editor with a special ability to execute its content, while providing access of some of its functionality to the script. Something like this? In the text editor, you have contents: text goes here and more text # Python script starts here x = 'a' up(x) print foo # Python script stops here more text again and the user selects lines 4 and 5 and chooses the command Execute. The script executes, and its output (foo) is appended to the end of the file: text goes here and more text # Python script starts here x = 'a' up(x) print foo # Python script stops here more text again foo Is this what you mean? If so, I think you are making this much too complicated for such a simple use-case. Just publish an API which the script can use, and have the main text editor application specify a script namespace containing only that API. That could be a module: import math # pretend this is your API shared module exec myvalue = 42 in math.__dict__ math.myvalue 42 Then execute the text using exec, but don't bother about putting it into a thread or subprocess. That just makes it harder to implement, and you have to worry about concurrency issues. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Tue, Sep 1, 2009 at 9:29 AM, Ecir Hana ecir.h...@gmail.com wrote: - if I understood it correctly defining a function in the string and exec-ing it created the function in current scope. This is something I really don't want Oops, missed that point. May I ask what is there you don't want, and what about putting the function inside a different module and importing? What's so bad about being in scope? Sincerely, -Xav -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Sep 1, 5:31 am, Steven D'Aprano ste...@remove.this.cybersource.com.au wrote: You can pass in a global and local namespaces to exec as arguments: x = 4 ns = {'x': 4} exec x += 1 in ns x 4 ns['x'] 5 See the docs for details. Thanks! This is very useful! You can copy the parts of the current scope into the namespace you pass to exec, then later copy the revised values out again. But are you sure you really want to take this approach? exec is up to ten times slower than just executing the code directly. And if the string is coming from an untrusted source, it is a *huge* security risk. I don't know if I should use exec. I don't really mind that it's slow (btw., why is it so?). But I don't quite understand why is it security risk. How is it different to run: exec 'format(your_hdd)' than: /bin/python format.py ? As far as I know, you can't kill threads, you can only ask them to kill themselves. Also, I'm not sure if I follow. What does this mean? If a thread runs: while True: pass it is not possible to kill it from another thread? (Bacause it doesn't check whether some other thread asks to stop it..?) Something like this? Well, something more like: data = [1, 2, 3] map(lambda x: x * 2, data) display_data_in_editor_viewport(data) #this renders into part of main editor window (may take some time) If so, I think you are making this much too complicated for such a simple use-case. Just publish an API which the script can use, and have the main text editor application specify a script namespace containing only that API. That could be a module: import math # pretend this is your API shared module exec myvalue = 42 in math.__dict__ math.myvalue 42 Then execute the text using exec, but don't bother about putting it into a thread or subprocess. That just makes it harder to implement, and you have to worry about concurrency issues. Ok, I could try exec, thanks for the explanation. But what about those security concerns you mentioned above? Thanks a lot, very informative! -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Tue, 01 Sep 2009 01:34:33 -0700, Ecir Hana wrote: You can copy the parts of the current scope into the namespace you pass to exec, then later copy the revised values out again. But are you sure you really want to take this approach? exec is up to ten times slower than just executing the code directly. And if the string is coming from an untrusted source, it is a *huge* security risk. I don't know if I should use exec. I don't really mind that it's slow (btw., why is it so?). Because it has to parse and compile the string into a code object before it can run it. But I don't quite understand why is it security risk. How is it different to run: exec 'format(your_hdd)' than: /bin/python format.py ? It's not different. But read what I said -- if the string is coming from an UNTRUSTED source -- presumably you trust yourself. If you run 'exec format(your_hdd)' it is because *you* want to format your hard disk. Now imagine you have a web-app which gets a string from the user and calls exec on it. Then you might have this: exec search('%d') % user_input and the user, who is halfway across the world, enters the following search string: places to eat'); import os; os.system('#rm -rf / Your web app will go right ahead and erase itself. That's why you need to keep untrusted strings away from exec, execfile, and eval. As far as I know, you can't kill threads, you can only ask them to kill themselves. Also, I'm not sure if I follow. What does this mean? If a thread runs: while True: pass it is not possible to kill it from another thread? (Bacause it doesn't check whether some other thread asks to stop it..?) No, I believe that the only way to halt that is to halt the entire process. Possibly there is a way to have a thread halt itself after a certain amount of time? I'm not an expert on threads, I've hardly ever used them. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Sep 1, 11:32 am, Steven D'Aprano ste...@remove.this.cybersource.com.au wrote: But I don't quite understand why is it security risk. How is it different to run: exec 'format(your_hdd)' than: /bin/python format.py ? It's not different. But read what I said -- if the string is coming from an UNTRUSTED source -- presumably you trust yourself. If you run 'exec format(your_hdd)' it is because *you* want to format your hard disk. Now imagine you have a web-app which gets a string from the user and calls exec on it. Then you might have this: exec search('%d') % user_input and the user, who is halfway across the world, enters the following search string: places to eat'); import os; os.system('#rm -rf / Your web app will go right ahead and erase itself. That's why you need to keep untrusted strings away from exec, execfile, and eval. Ah, I see! Ok. No, I believe that the only way to halt that is to halt the entire process. Possibly there is a way to have a thread halt itself after a certain amount of time? I'm not an expert on threads, I've hardly ever used them. Thank you once again! -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing python script stored as a string
On Tuesday 01 September 2009 11:32:29 Steven D'Aprano wrote: Possibly there is a way to have a thread halt itself after a certain amount of time? I'm not an expert on threads, I've hardly ever used them. Not automagically, as far as I can see. You are on your own if you want to somehow kill a thread. What I do is to wrap what I want to do in a while clause: while runbool: do_what_you_want() where runbool is a global that I set to false elsewhere when I want to stop. There is of course nothing to stop you writing a thread with something like: import time start_time = time.time() while time.time() - start_time some_parameter: do_what_you_want() Which will have the effect of running for a while and then stop. I cannot see much use use for that as it is, as it will be bloody-minded about the time, and might chop off before the whole job is done, but one could make it more intelligent, such that it keeps track of idle time, and aborts after say a second (or a minute) of not doing anything useful. Unlike you, I am a thread and queue fanatic - use them all the time. - Hendrik -- http://mail.python.org/mailman/listinfo/python-list