Re: adding a simulation mode
2012/7/13 Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: Well of course it does. If copytree fails, the try block ends and execution skips straight to the except block, which runs, and then the program halts because there's nothing else to be done. That at least is my guess, based on the described symptoms. Well I think that's what I was stupidly missing, I always had only one possibly failing thing in a try/except block, and I always gave for granted that it doesn't jump to the except block on first error, but of course it makes more sense if it does... Thanks a lot -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On 13/07/12 04:16:53, Steven D'Aprano wrote: On Thu, 12 Jul 2012 16:37:42 +0100, andrea crotti wrote: 2012/7/12 John Gordon gor...@panix.com: In mailman.2043.1342102625.4697.python-l...@python.org andrea crotti andrea.crott...@gmail.com writes: Well that's what I thought, but I can't find any explicit exit anywhere in shutil, so what's going on there? Try catching SystemExit specifically (it doesn't inherit from Exception, so except Exception won't catch it.) Ah yes that actually works, but I think is quite dodgy, why was it done like this? It may be that the function you're calling found a problem that the author thinks is so grave that they shouldn't give you an opportunity to deal with it. If that's the case, I would be inclined to think that they are wrong. Built-in exceptions SystemExit, KeyboardInterrupt and GeneratorExit deliberately do not inherit from Exception since they are not meant to be caught by catch-all try...except Exception clauses. You can see the exception hierarchy here: http://docs.python.org/library/exceptions.html#exception-hierarchy Please do NOT catch BaseException, since that is the wrong thing to do. I would agree if you had said in production code. If you are investigating why a third-party function is stopping your interpreter, then catching BaseException may tell you that the code is raising the wrong kind of Exception. Once you know what kind the function is raising, you should catch only that particular excpetion subclass. If you must catch SystemExit, KeyboardInterrupt, etc. they you should do so as separate catch clauses: try: main() except SystemExit as e: print(e) # see if we can find out who is raising this If you want to find out who is raising the exception, you could try this: except SystemExit: import traceback traceback.print_exc() That will print a complete stack trace. If you only need to know what kind of exception you have, you can do: print(repr(e)) A simple print(e) will print str(e), which in the case of SystemExit, is an empty string. That's not very informative. except KeyboardInterrupt: print(Mwahahaha my pretty, you cannot cancel this!!!) print(...er, now what do I do?) except Exception: print(why am I catching exceptions I can't recover from?) Hope this helps, -- HansM -- http://mail.python.org/mailman/listinfo/python-list
RE: adding a simulation mode
Please do NOT catch BaseException, since that is the wrong thing to do. I would agree if you had said in production code. If you are investigating why a third-party function is stopping your interpreter, then catching BaseException may tell you that the code is raising the wrong kind of Exception. Once you know what kind the function is raising, you should catch only that particular excpetion subclass. I would say the opposite. In production code usually I want it to recover, log as much information as I need (including sending any notifications), and NOT just die. In development, not catching the exception will give me a full trace back automatically. Why bother with trying to catch and print something when the interpreter will do it for me? Not to mention that removes any hassle of trying to catch the right exception or figuring out the best way to print it. I suppose if there are arguments on the exception that were not printed then I might want to catch it, but has been rare in my experience. Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On Sat, Jul 14, 2012 at 3:08 AM, Prasad, Ramit ramit.pra...@jpmorgan.com wrote: I would say the opposite. In production code usually I want it to recover, log as much information as I need (including sending any notifications), and NOT just die. In development, not catching the exception will give me a full trace back automatically. Here's another take on the matter. In development, your script is your execution unit, so you let the interpreter print out your tracebacks. In production, there will usually be one, maybe two subunits (for instance, a TCP-based server might have the socket connection as an execution unit, and possibly a command parser inside that), and at the top of that subunit, you have a broad exception handler that resets that one unit (goes back and accepts another client, or waits for another command). Otherwise, wide-scope exception handling is usually a bad thing. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
One thing that I don't quite understand is why some calls even if I
catch the exception still makes the whole program quit.
For example this
try:
copytree('sjkdf', 'dsflkj')
Popen(['notfouhd'], shell=True)
except Exception as e:
print(here)
behaves differently from:
try:
Popen(['notfouhd'], shell=True)
copytree('sjkdf', 'dsflkj')
except Exception as e:
print(here)
because if copytree fails it quits anyway.
I also looked at the code but can't quite get why.. any idea?
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
One way instead that might actually work is this
def default_mock_action(func_name):
def _default_mock_action(*args, **kwargs):
print(running {} with args {} and {}.format(func_name, args, kwargs))
return _default_mock_action
def mock_fs_actions(to_run):
Take a function to run, and run it in an environment which
mocks all the possibly dangerous operations
side_effect = [
'copytree',
'copy',
]
acts = dict((s, default_mock_action(s)) for s in side_effect)
with patch('pytest.runner.commands.ShellCommand.run',
default_mock_action('run')):
with patch.multiple('shutil', **acts):
to_run()
So I can just pass the main function inside the mock like
mock_fs_actions(main)
and it seems to do the job, but I have to list manually all the things
to mock and I'm not sure is the best idea anyway..
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
In mailman.2039.1342099220.4697.python-l...@python.org andrea crotti
andrea.crott...@gmail.com writes:
try:
copytree('sjkdf', 'dsflkj')
Popen(['notfouhd'], shell=True)
except Exception as e:
print(here)
behaves differently from:
try:
Popen(['notfouhd'], shell=True)
copytree('sjkdf', 'dsflkj')
except Exception as e:
print(here)
because if copytree fails it quits anyway.
I also looked at the code but can't quite get why.. any idea?
copytree() could contain a call to sys.exit(), although that seems like
a rude thing to do.
--
John Gordon A is for Amy, who fell down the stairs
gor...@panix.com B is for Basil, assaulted by bears
-- Edward Gorey, The Gashlycrumb Tinies
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
Well that's what I thought, but I can't find any explicit exit anywhere in shutil, so what's going on there? -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
In mailman.2043.1342102625.4697.python-l...@python.org andrea crotti andrea.crott...@gmail.com writes: Well that's what I thought, but I can't find any explicit exit anywhere in shutil, so what's going on there? Try catching SystemExit specifically (it doesn't inherit from Exception, so except Exception won't catch it.) -- John Gordon A is for Amy, who fell down the stairs gor...@panix.com B is for Basil, assaulted by bears -- Edward Gorey, The Gashlycrumb Tinies -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
2012/7/12 John Gordon gor...@panix.com: In mailman.2043.1342102625.4697.python-l...@python.org andrea crotti andrea.crott...@gmail.com writes: Well that's what I thought, but I can't find any explicit exit anywhere in shutil, so what's going on there? Try catching SystemExit specifically (it doesn't inherit from Exception, so except Exception won't catch it.) -- Ah yes that actually works, but I think is quite dodgy, why was it done like this? In shutil there is still no mention of SystemExit, and trying to raise the single exceptions by and doens't still make it exit, so I would still like to know how it is possible just for curiosity.. -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
andrea crotti wrote at 2012-7-12 14:20 +0100:
One thing that I don't quite understand is why some calls even if I
catch the exception still makes the whole program quit.
For example this
try:
copytree('sjkdf', 'dsflkj')
Popen(['notfouhd'], shell=True)
except Exception as e:
print(here)
behaves differently from:
try:
Popen(['notfouhd'], shell=True)
copytree('sjkdf', 'dsflkj')
except Exception as e:
print(here)
because if copytree fails it quits anyway.
I also looked at the code but can't quite get why.. any idea?
There are ways to quit a program immediately without giving
exception handlers a chance to intervene -- though Python does
not make this easy.
Your code above should not do this. If it does, there is likely a bug.
You told us, that the two alternatives above behaved differently --
I expect 'behaved differently with respect to the printing of here'.
If you tell us, which alternative printed here and which did not,
we would be able to deduce which of the Popen or copytree
caused the immediate exit.
Popen might contain a call to os._exit (one of the ways to immediately
quit) -- though it should only call it in the forked child not in the
calling process. coyptree might under exceptional circumstances (extremely
deeply nested structures -- surely not for non-existent source and target)
cause a stack overflow (which, too, can lead to immediate death).
In addition, Popen and maybe even copytree may call platform
dependent functions. Thus, platform information could be relevant.
Under *nix, you should be able to get some information from
the exit code of a suddenly quiting process. It tells whether
the process died from a fatal signal (a stack overflow would
result in the fatal SIGSEGV) or whether it existed willingly with
an exit code.
--
Dieter
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On Thu, 12 Jul 2012 15:17:03 +0100, andrea crotti wrote: Well that's what I thought, but I can't find any explicit exit anywhere in shutil, so what's going on there? Hard to say, since you don't give any context to your question. When replying to posts, please leave enough quoted to establish context. Neither email nor usenet are guaranteed delivery services, and they certainly don't guarantee to deliver messages in order. Assume that your readers may not have seen the message you are replying to, and you will probably get more and better responses. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On Thu, 12 Jul 2012 16:37:42 +0100, andrea crotti wrote: 2012/7/12 John Gordon gor...@panix.com: In mailman.2043.1342102625.4697.python-l...@python.org andrea crotti andrea.crott...@gmail.com writes: Well that's what I thought, but I can't find any explicit exit anywhere in shutil, so what's going on there? Try catching SystemExit specifically (it doesn't inherit from Exception, so except Exception won't catch it.) Ah yes that actually works, but I think is quite dodgy, why was it done like this? Built-in exceptions SystemExit, KeyboardInterrupt and GeneratorExit deliberately do not inherit from Exception since they are not meant to be caught by catch-all try...except Exception clauses. You can see the exception hierarchy here: http://docs.python.org/library/exceptions.html#exception-hierarchy Please do NOT catch BaseException, since that is the wrong thing to do. If you must catch SystemExit, KeyboardInterrupt, etc. they you should do so as separate catch clauses: try: main() except SystemExit as e: print(e) # see if we can find out who is raising this except KeyboardInterrupt: print(Mwahahaha my pretty, you cannot cancel this!!!) print(...er, now what do I do?) except Exception: print(why am I catching exceptions I can't recover from?) -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On Thu, 12 Jul 2012 14:20:18 +0100, andrea crotti wrote:
One thing that I don't quite understand is why some calls even if I
catch the exception still makes the whole program quit.
Without seeing your whole program, we can't possibly answer this. But by
consulting my crystal ball, I bet you have something like this:
try:
do_stuff() # run your program
except Exception as e:
# pointlessly catch exceptions I can't handle, which has the
# bonus of making debugging MUCH MUCH harder
print(here)
# end of file, nothing further to do
When do_stuff() fails, here gets printed, and then the program exits
because there's nothing else to do.
Catching exceptions doesn't magically cause the code to continue from the
point of the error. It doesn't work like that. Execution skips from where
the error occurred to the except clause. Once the except clause has run,
anything following the except clause runs, and then the program ends as
normal.
If you haven't already done so, I recommend you go through the tutorial:
http://docs.python.org/py3k/tutorial/index.html
in particular the part about exception handling:
http://docs.python.org/py3k/tutorial/errors.html
For example this
try:
copytree('sjkdf', 'dsflkj')
Popen(['notfouhd'], shell=True)
except Exception as e:
print(here)
What is Popen and where is it from?
My first guess was os.popen, but that doesn't take a shell argument:
py os.popen(['ls', '-l'], shell=True)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: popen() got an unexpected keyword argument 'shell'
behaves differently from:
try:
Popen(['notfouhd'], shell=True)
copytree('sjkdf', 'dsflkj')
except Exception as e:
print(here)
because if copytree fails it quits anyway.
Well of course it does. If copytree fails, the try block ends and
execution skips straight to the except block, which runs, and then the
program halts because there's nothing else to be done.
That at least is my guess, based on the described symptoms.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
andrea crotti andrea.crott...@gmail.com writes: I'm writing a program which has to interact with many external resources, at least: - mysql database - perforce - shared mounts - files on disk And the logic is quite complex, because there are many possible paths to follow depending on some other parameters. This program even needs to run on many virtual machines at the same time so the interaction is another thing I need to check... Now I successfully managed to mock the database with sqlalchemy and only the fields I actually need, but I now would like to simulate also everything else. There is a paradigm called inversion of control which can be used to handle those requirements. With inversion of control, the components interact on the bases of interfaces. The components themselves do not know each other, they know only the interfaces they want to interact with. For the interaction to really take place, a component asks a registry give me a component satisfying this interface, gets it and uses the interface. If you follow this paradigm, it is easy to switch components: just register different alternatives for the interface at hand. zope.interface and zope.component are python packages that support this paradigm. Despite the zope in their name, they can be used outside of Zope. zope.interface models interfaces, while zope.component provides so called utilities (e.g. database utility, filesystem utility, ...) and adapters and the corresponding registries. Of course, they contain only the infrastructure for the inversion of control paradigm. Up to you to provide the implementation for the various mocks. -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
2012/7/5 Dieter Maurer die...@handshake.de: There is a paradigm called inversion of control which can be used to handle those requirements. With inversion of control, the components interact on the bases of interfaces. The components themselves do not know each other, they know only the interfaces they want to interact with. For the interaction to really take place, a component asks a registry give me a component satisfying this interface, gets it and uses the interface. If you follow this paradigm, it is easy to switch components: just register different alternatives for the interface at hand. zope.interface and zope.component are python packages that support this paradigm. Despite the zope in their name, they can be used outside of Zope. zope.interface models interfaces, while zope.component provides so called utilities (e.g. database utility, filesystem utility, ...) and adapters and the corresponding registries. Of course, they contain only the infrastructure for the inversion of control paradigm. Up to you to provide the implementation for the various mocks. Thanks that's a good point, in short I could do something like: class FSActions: @classmethod def copy_directories(cls, src, dest) raise NotImplementedError @classmethod And then have different implementations of this interface. This would work, but I don't really like the idea of constructing interfaces that provide only the few things I need. Instead of being good APIs they might become just random functionalities put together to make my life easier, and at that point maybe just some clear mocking might be even better.. -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On Wed, 04 Jul 2012 10:42:56 +0100, andrea crotti wrote: I'm writing a program which has to interact with many external resources, at least: - mysql database - perforce - shared mounts - files on disk And the logic is quite complex, because there are many possible paths to follow depending on some other parameters. This program even needs to run on many virtual machines at the same time so the interaction is another thing I need to check... Now I successfully managed to mock the database with sqlalchemy and only the fields I actually need, but I now would like to simulate also everything else. I would like for example that if I simulate I can pass a fake database, a fake configuration and get the log of what exactly would happen. But I'm not sure how to implement it now.. One possibility would be to have a global variable (PRETEND_ONLY = False) that if set should be checked before every potentially system-dependent command. I think a better way would be to use a mock database, etc. For each thing which you want to simulate, create a class that has the same interface but a simulated implementation. Then, have your code accept the thing as an argument. E.g. instead of having a hard-coded database connection, allow the database connection to be set (perhaps as an argument, perhaps as a config option, etc.). There are libraries to help with mocks, e.g.: http://garybernhardt.github.com/python-mock-comparison/ For example copytree(src, dest) becomes: if not PRETEND_ONLY: copytree(src, dest) Ewww :( Mocking the file system is probably the hardest part, because you generally don't have a FileSystem object available to be replaced. In effect, your program has one giant global variable, the file system. Worse, it's not even a named variable, it's hard-coded everywhere you use it. I don't know of any good solution for that. I've often thought about it, but don't have an answer. I suppose you could monkey-patch a bunch of stuff: if ONLY_PRETEND: open = my_mock_open copytree = my_mock_copytree # etc. main() # run your application but that would also be painful. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
2012/7/4 Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:
Then, have your code accept the thing as an argument.
E.g. instead of having a hard-coded database connection, allow the
database connection to be set (perhaps as an argument, perhaps as a
config option, etc.).
There are libraries to help with mocks, e.g.:
http://garybernhardt.github.com/python-mock-comparison/
Ah yes this part is already done, I pass an object to the entry point
of the program which represents the database connection, which looks
like this:
class MockMysqlEngine(MySqlEngine):
# TODO: make the engine more generic would avoid this dirty hack
def __init__(self, *args, **kwargs):
# self.engine =
create_engine('sqlite:home/andrea/testdb.sqlite', echo=True)
self.engine = create_engine('sqlite://', echo=True)
self.meta = MetaData(bind=self.engine)
self.session_maker = sessionmaker(bind=self.engine)
Now I populate statically the schema and populate with some test data
too, but I'm also implementing a weay to just pass some CSV files so
that other people can easily write some test cases with some other
possible database configurations.
(And I use mock for a few other things)
For example
copytree(src, dest) becomes:
if not PRETEND_ONLY:
copytree(src, dest)
Ewww :(
Mocking the file system is probably the hardest part, because you
generally don't have a FileSystem object available to be replaced. In
effect, your program has one giant global variable, the file system.
Worse, it's not even a named variable, it's hard-coded everywhere you use
it.
I don't know of any good solution for that. I've often thought about it,
but don't have an answer.
I suppose you could monkey-patch a bunch of stuff:
if ONLY_PRETEND:
open = my_mock_open
copytree = my_mock_copytree
# etc.
main() # run your application
but that would also be painful.
Yes there is no easy solution apparently.. But I'm also playing
around with vagrant and virtual machine generations, suppose I'm able
to really control what will be on the machine at time X, creating it
on demand with what I need, it might be a good way to solve my
problems (a bit overkill and slow maybe though).
I'll try the sys.excepthook trick first, any error should give me an
exception, so if I catch them all I think it might work already..
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
Yes there is no easy solution apparently.. But I'm also playing
around with vagrant and virtual machine generations, suppose I'm able
to really control what will be on the machine at time X, creating it
on demand with what I need, it might be a good way to solve my
problems (a bit overkill and slow maybe though).
I'll try the sys.excepthook trick first, any error should give me an
exception, so if I catch them all I think it might work already..
I actually thought that the sys.excepthook would be easy but it's not
so trivial apparently:
This simple sample never reaches the print(here), because even if
the exception is catched it still quits with return code=1.
I also tried to catch the signal but same result, how do I make it
continue and just don't complain?
The other option if of course to do a big try/except, but I would
prefer the excepthook solution..
import sys
from shutil import copy
def my_except_hook(etype, value, tb):
print(got an exception of type, etype)
if __name__ == '__main__':
sys.excepthook = my_except_hook
copy('sdflsdk')
print(here)
--
http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
On 12-07-04 05:42 AM, andrea crotti wrote: ... copytree(src, dest) becomes: if not PRETEND_ONLY: copytree(src, dest) import globalsub, unittest class MyTest( unittest.TestCase ): def setUp( self ): globalsub.subs( shutil.copytree ) def tearDown( self ): globalsub.restore( shutil.copytree ) You can also pass a function to subs like so: def setUp( self ): self.copied_trees = [] def fake_copytree( src, dest ): assert os.path.exists( src ) self.copied_trees.append( (src, dest )) return dest # or whatever the thing should return globalsub.subs( shutil.copytree, fake_copytree ) $ pip install globalsub HTH, Mike -- Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
For what it's worth, this is the reason that Allen Short wrote Exocet. This way, you can test your code without having to resort to sys.modules hackery, and you can better factor your applications by separating configuration and environment concerns from the rest of your code. See: - http://washort.twistedmatrix.com/2011/01/introducing-exocet.html - http://washort.twistedmatrix.com/2011/03/exocet-second-look.html -- Devin -- http://mail.python.org/mailman/listinfo/python-list
Re: adding a simulation mode
andrea crotti andrea.crott...@gmail.com writes: copytree(src, dest) becomes: if not PRETEND_ONLY: copytree(src, dest) But I don't like it too much because I would have to add a lot of garbage around.. I've had good results writing the module under test in the style of a java applet, i.e. one of its args is a class instance representing the outside world, and ALL interaction that you might want to simulate is done through this object: def your_prog(parent): conn = parent.db.make_connection(...) blah = parent.copytree(...) Then you make real and mock versions of the external interface, and pass in an appropriate instance. -- http://mail.python.org/mailman/listinfo/python-list
