Re: Bounds checking
Martin De Kauwe wrote: Sorry, are you trying to say that it is not practical to write correct code that isn't buggy? Well, you're honest, at least, still I can't help but feel that you're admitting defeat before even starting. No. What I am saying is the code is written has been well tested and *appears* to be working well. However the code is complicated and there is potential for bugs. I think I am just been practical here, evidently I can't think of everything, but there are some clear and obvious errors that would be worth checking for. I can only explain this in the terms of the code (sorry)...but for example the model estimates plant photosynthesis and then allocates the carbon. So one clear example is that the model cuts back carbon production if there is water stress for the plant. This involves removing carbon from the state. Clearly if you ended up in a situation where there is negative carbon in a leaf, i.e. the leaf doesn't exist well this is not physically possible and would be a code issue. Whilst this is unlikely I think it would be nice to have a catch for it. Another example would be the amount of soil water available to the plant, again there can be zero but not negative soil water. It wouldn't be meaningful. I hope that makes sense? thanks Not that much. You'll spot bugs where negative numbers will be set to some attribute but what if 42 is put instead of 43 in one of your attribute ? Same consequence, it will mess up with your model but none of your check will spot that error. Try to identify those complicated functions you mentioned, and write unitary tests for them. Use a set of input parameters with the expected function return value. That way you'll be able to spot errors, whether the attribute is negative or not. JM PS : writing unitary tests takes time, a lot of time. -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On Mar 21, 9:43 pm, Jean-Michel Pichavant jeanmic...@sequans.com wrote: Martin De Kauwe wrote: Sorry, are you trying to say that it is not practical to write correct code that isn't buggy? Well, you're honest, at least, still I can't help but feel that you're admitting defeat before even starting. No. What I am saying is the code is written has been well tested and *appears* to be working well. However the code is complicated and there is potential for bugs. I think I am just been practical here, evidently I can't think of everything, but there are some clear and obvious errors that would be worth checking for. I can only explain this in the terms of the code (sorry)...but for example the model estimates plant photosynthesis and then allocates the carbon. So one clear example is that the model cuts back carbon production if there is water stress for the plant. This involves removing carbon from the state. Clearly if you ended up in a situation where there is negative carbon in a leaf, i.e. the leaf doesn't exist well this is not physically possible and would be a code issue. Whilst this is unlikely I think it would be nice to have a catch for it. Another example would be the amount of soil water available to the plant, again there can be zero but not negative soil water. It wouldn't be meaningful. I hope that makes sense? thanks Not that much. You'll spot bugs where negative numbers will be set to some attribute but what if 42 is put instead of 43 in one of your attribute ? Same consequence, it will mess up with your model but none of your check will spot that error. Try to identify those complicated functions you mentioned, and write unitary tests for them. Use a set of input parameters with the expected function return value. That way you'll be able to spot errors, whether the attribute is negative or not. JM PS : writing unitary tests takes time, a lot of time. Hi, Yes you make a good point. I think what I was trying to get across is there are certain scenarios which can't physically happen and they are straight forward to check for. I just thought it might be nice to add a check for these. However you make a good case and I definitely take your point. I guess I should look into unitary tests then (any suggestions?) ta. -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Martin De Kauwe wrote: On Mar 21, 9:43 pm, Jean-Michel Pichavant jeanmic...@sequans.com wrote: Martin De Kauwe wrote: Sorry, are you trying to say that it is not practical to write correct code that isn't buggy? Well, you're honest, at least, still I can't help but feel that you're admitting defeat before even starting. No. What I am saying is the code is written has been well tested and *appears* to be working well. However the code is complicated and there is potential for bugs. I think I am just been practical here, evidently I can't think of everything, but there are some clear and obvious errors that would be worth checking for. I can only explain this in the terms of the code (sorry)...but for example the model estimates plant photosynthesis and then allocates the carbon. So one clear example is that the model cuts back carbon production if there is water stress for the plant. This involves removing carbon from the state. Clearly if you ended up in a situation where there is negative carbon in a leaf, i.e. the leaf doesn't exist well this is not physically possible and would be a code issue. Whilst this is unlikely I think it would be nice to have a catch for it. Another example would be the amount of soil water available to the plant, again there can be zero but not negative soil water. It wouldn't be meaningful. I hope that makes sense? thanks Not that much. You'll spot bugs where negative numbers will be set to some attribute but what if 42 is put instead of 43 in one of your attribute ? Same consequence, it will mess up with your model but none of your check will spot that error. Try to identify those complicated functions you mentioned, and write unitary tests for them. Use a set of input parameters with the expected function return value. That way you'll be able to spot errors, whether the attribute is negative or not. JM PS : writing unitary tests takes time, a lot of time. Hi, Yes you make a good point. I think what I was trying to get across is there are certain scenarios which can't physically happen and they are straight forward to check for. I just thought it might be nice to add a check for these. However you make a good case and I definitely take your point. I guess I should look into unitary tests then (any suggestions?) ta. if your python version 2.7: http://pypi.python.org/pypi/unittest2 http://docs.python.org/library/unittest.html This is the standard way of writing unitary tests in python (AKA pyunit) JM -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On Mar 19, 8:40 pm, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: On Sat, 19 Mar 2011 01:38:10 -0700, Martin De Kauwe wrote: Why don't you do the range check *before* storing it in state? That way you can identify the calculation that was wrong, instead of merely noticing that at some point some unknown calculation went wrong. I guess no reason really. I suppose in my mind I was thinking it was an unlikely safeguard but I liked the idea of adding so would just do it at the end of a time step. In reality I think there is practically no difference and this way it is done once, in a single location vs. potential 10 separate checks? I don't see the advantage? You should always aim to fail as close as possible to the source of the error as is practical. That decreases the amount of debugging required when something fails: instead of searching your entire program, you only have to search a very small amount of code. -- Steven OK I take your point and can see the superior logic! I shall amend what I was planning -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On Fri, 18 Mar 2011 07:24:33 -0700, Martin De Kauwe wrote:
Hi,
if one has a set of values which should never step outside certain
bounds (for example if the values were negative then they wouldn't be
physically meaningful) is there a nice way to bounds check? I
potentially have 10 or so values I would like to check at the end of
each iteration.
assert all(x = 0 for x in (a, b, c, d, e, f, g, h, i, j))
However as the loop is over many years I figured I
probably want to be as optimal as possible with my check. Any thoughts?
e.g. this is my solution
# module contain data
# e.g. print state.something might produce 4.0 import state as state
def main():
for i in xrange(num_days):
# do stuff
# bounds check at end of iteration
bounds_check(state)
Why don't you do the range check *before* storing it in state? That way
you can identify the calculation that was wrong, instead of merely
noticing that at some point some unknown calculation went wrong.
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
You're looking at every single attribute, including those of super
classes, when you only want to check 10 or so attributes. That's
probably not wise. At the very least, dir() will be a fairly expensive
call.
If you insist on checking state *after* the value is stored, instead of
preventing it from being stored in the first place, it is better to make
the state object responsible for doing it's own bounds checking. That way
only the state object needs to be updated when you change those ten
attributes, instead of some arbitrary number of places scattered all
throughout your code.
print Error state values 0: %s % (attr)
sys.exit()
Python already has a mechanism for printing an error message and exiting:
raise.
if condition:
raise ValueError(attribute %s is negative % name)
This will still halt the application, but it does so without stomping all
over normal conventions for command line applications (error messages
should go to stderr, not stdout; the return result should be non-zero) as
well as normal conventions for Python code (the caller should be able to
easily catch the exception -- catching sys.exit can be done, but it is a
pretty unusual thing to do).
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On Fri, 18 Mar 2011 15:35:40 -0700, Martin De Kauwe wrote:
Don't check for bounds, fix any bug in the code that would set your
values out of bounds and use asserts while debugging.
whilst that is a nice idea in practice this just is not a practical
solution.
Sorry, are you trying to say that it is not practical to write correct
code that isn't buggy? Well, you're honest, at least, still I can't help
but feel that you're admitting defeat before even starting.
Otherwise if you really need dynamic checks, it will cost you cpu, for
sure.
Yes I agree and I hadn't decided whether to add it or not as there
aren't any current issues. However I can see that the check would
overall be safer. I was just wondering if there was some super smartie
pants solution :P
Make each of the attributes a computed attribute with a setter that
raises an exception if you try to store a negative value. That way each
attribute is responsible for ensuring that itself is never negative.
Here's a toy example:
class Demo(object):
... def __init__(self, x):
... self.x = x
... def _getx(self):
... return self._x
... def _setx(self, value):
... if value 0:
... raise ValueError('attempt to set x to negative value')
... self._x = value
... x = property(_getx, _setx)
...
d = Demo(-42)
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 3, in __init__
File stdin, line 8, in _setx
ValueError: attempt to set x to negative value
d = Demo(42)
d.x
42
d.x = -23
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 8, in _setx
ValueError: attempt to set x to negative value
d.x = 23
d.x
23
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
dir() has to do a bit a computation. I would be tempted to give 'state' a set of attributes to check. Call it 'nonnegatives'. for attr in nonnegatives: if ... This allows for attributes not subject to that check. -- Terry Jan Reedy Agreed. I was trying to just write a dummy example quickly to go with my question and that was just the way that came to mind so that I could loop over them to test. It wasn't a great example sorry! In reality I would just have like you said a subset a list I guess of the ones I would check. -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
assert all(x = 0 for x in (a, b, c, d, e, f, g, h, i, j)) yep neat! Why don't you do the range check *before* storing it in state? That way you can identify the calculation that was wrong, instead of merely noticing that at some point some unknown calculation went wrong. I guess no reason really. I suppose in my mind I was thinking it was an unlikely safeguard but I liked the idea of adding so would just do it at the end of a time step. In reality I think there is practically no difference and this way it is done once, in a single location vs. potential 10 separate checks? I don't see the advantage? You're looking at every single attribute, including those of super classes, when you only want to check 10 or so attributes. That's probably not wise. At the very least, dir() will be a fairly expensive call. yes your right, sorry I was just trying to post a quick example to go with my question. I would use a list of the attributes to check. If you insist on checking state *after* the value is stored, instead of preventing it from being stored in the first place, it is better to make the state object responsible for doing it's own bounds checking. That way only the state object needs to be updated when you change those ten attributes, instead of some arbitrary number of places scattered all throughout your code. yes I think this is what I was getting at, I will look at your suggestion thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Sorry, are you trying to say that it is not practical to write correct code that isn't buggy? Well, you're honest, at least, still I can't help but feel that you're admitting defeat before even starting. No. What I am saying is the code is written has been well tested and *appears* to be working well. However the code is complicated and there is potential for bugs. I think I am just been practical here, evidently I can't think of everything, but there are some clear and obvious errors that would be worth checking for. I can only explain this in the terms of the code (sorry)...but for example the model estimates plant photosynthesis and then allocates the carbon. So one clear example is that the model cuts back carbon production if there is water stress for the plant. This involves removing carbon from the state. Clearly if you ended up in a situation where there is negative carbon in a leaf, i.e. the leaf doesn't exist well this is not physically possible and would be a code issue. Whilst this is unlikely I think it would be nice to have a catch for it. Another example would be the amount of soil water available to the plant, again there can be zero but not negative soil water. It wouldn't be meaningful. I hope that makes sense? thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On Sat, 19 Mar 2011 01:38:10 -0700, Martin De Kauwe wrote: Why don't you do the range check *before* storing it in state? That way you can identify the calculation that was wrong, instead of merely noticing that at some point some unknown calculation went wrong. I guess no reason really. I suppose in my mind I was thinking it was an unlikely safeguard but I liked the idea of adding so would just do it at the end of a time step. In reality I think there is practically no difference and this way it is done once, in a single location vs. potential 10 separate checks? I don't see the advantage? You should always aim to fail as close as possible to the source of the error as is practical. That decreases the amount of debugging required when something fails: instead of searching your entire program, you only have to search a very small amount of code. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Bounds checking
Hi,
if one has a set of values which should never step outside certain
bounds (for example if the values were negative then they wouldn't be
physically meaningful) is there a nice way to bounds check? I
potentially have 10 or so values I would like to check at the end of
each iteration. However as the loop is over many years I figured I
probably want to be as optimal as possible with my check. Any
thoughts?
e.g. this is my solution
# module contain data
# e.g. print state.something might produce 4.0
import state as state
def main():
for i in xrange(num_days):
# do stuff
# bounds check at end of iteration
bounds_check(state)
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
print Error state values 0: %s % (attr)
sys.exit()
if __name__ == __main__:
sys.exit(main())
thanks
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
What sort of checks are you making ? - in general greater than/less than
tend to be fairly optimal, although you might be able to do a faster is
negative test
Katie
On Fri, Mar 18, 2011 at 2:24 PM, Martin De Kauwe mdeka...@gmail.com wrote:
Hi,
if one has a set of values which should never step outside certain
bounds (for example if the values were negative then they wouldn't be
physically meaningful) is there a nice way to bounds check? I
potentially have 10 or so values I would like to check at the end of
each iteration. However as the loop is over many years I figured I
probably want to be as optimal as possible with my check. Any
thoughts?
e.g. this is my solution
# module contain data
# e.g. print state.something might produce 4.0
import state as state
def main():
for i in xrange(num_days):
# do stuff
# bounds check at end of iteration
bounds_check(state)
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
print Error state values 0: %s % (attr)
sys.exit()
if __name__ == __main__:
sys.exit(main())
thanks
Martin
--
http://mail.python.org/mailman/listinfo/python-list
--
CoderStack
http://www.coderstack.co.uk/python-jobs
The Software Developer Job Board
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Martin De Kauwe wrote:
Hi,
if one has a set of values which should never step outside certain
bounds (for example if the values were negative then they wouldn't be
physically meaningful) is there a nice way to bounds check? I
potentially have 10 or so values I would like to check at the end of
each iteration. However as the loop is over many years I figured I
probably want to be as optimal as possible with my check. Any
thoughts?
e.g. this is my solution
# module contain data
# e.g. print state.something might produce 4.0
import state as state
def main():
for i in xrange(num_days):
# do stuff
# bounds check at end of iteration
bounds_check(state)
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
print Error state values 0: %s % (attr)
sys.exit()
if __name__ == __main__:
sys.exit(main())
thanks
Martin
Don't check for bounds, fix any bug in the code that would set your
values out of bounds and use asserts while debugging.
Otherwise if you really need dynamic checks, it will cost you cpu, for
sure. Howeverver you could for instance override the __setatttr__ of
state object, and call the attribute's associated function.
class State(object):
funcTable = {
'foo': lambda x: x = 0.0
}
def __init__(self):
self.foo = 0
def __setattr__(self, attribute, value):
if not self.funcTable.get(attribute, lambda x: True)(value):
sys.exit('error out of bound')
return object.__setattr(self, attribute, value)
Untested, however it's just an idea. I'm not even sure that would be
less cpu consuming :D
That way only attributes in functable execute a (cpu consuming ?) test
function, all other attributes will execute 'lambda x: True'.
The check occurs everytime you set an attribute however.
JM
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Jean-Michel Pichavant wrote:
Martin De Kauwe wrote:
Don't check for bounds, fix any bug in the code that would set your
values out of bounds and use asserts while debugging.
[ ... ]
def __setattr__(self, attribute, value):
if not self.funcTable.get(attribute, lambda x: True)(value):
sys.exit('error out of bound')
return object.__setattr(self, attribute, value)
Offhand, my only quibble is that sys.exit is not helpful for debugging.
Much better to raise an error:
if not self.funcTable.get(attribute, lambda x: True)(value):
raise ValueError ('error out of bound')
or define a subclass of ValueError just for this purpose. On error, the
program will stop just as dead, but you'll get a trace.
Mel.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
print Error state values 0: %s % (attr)
sys.exit()
Not that related to the question. But it's usually better to raise an exception.
Exiting in the middle of a function usually make debugging later more
interesting.
You might find Traits interesting for validation, see
http://code.enthought.com/projects/traits/
HTH
--
Miki Tebeka miki.teb...@gmail.com
http://pythonwise.blogspot.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On Fri, Mar 18, 2011 at 3:01 PM, Jean-Michel Pichavant jeanmic...@sequans.com wrote: Don't check for bounds, fix any bug in the code that would set your values out of bounds and use asserts while debugging. Otherwise if you really need dynamic checks, it will cost you cpu, for sure. Howeverver you could for instance override the __setatttr__ of state object, and call the attribute's associated function. If the codes something critical (i.e. it's used for financial calculations, hardware control, etc.) it's probably safer to test it dynamically, unless you only have a finite number of inputs/outputs it's often hard to ensure you've fixed all the bugs. Katie -- CoderStack h http://www.coderstack.co.ukttp://www.coderstack.co.uk/perl-jobs The Software Developer Job Board -- http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Don't check for bounds, fix any bug in the code that would set your
values out of bounds and use asserts while debugging.
whilst that is a nice idea in practice this just is not a practical
solution.
Otherwise if you really need dynamic checks, it will cost you cpu, for
sure.
Yes I agree and I hadn't decided whether to add it or not as there
aren't any current issues. However I can see that the check would
overall be safer. I was just wondering if there was some super smartie
pants solution :P
Howeverver you could for instance override the __setatttr__ of
state object, and call the attribute's associated function.
class State(object):
funcTable = {
'foo': lambda x: x = 0.0
}
def __init__(self):
self.foo = 0
def __setattr__(self, attribute, value):
if not self.funcTable.get(attribute, lambda x: True)(value):
sys.exit('error out of bound')
return object.__setattr(self, attribute, value)
Untested, however it's just an idea. I'm not even sure that would be
less cpu consuming :D
thanks I will look at what you suggested.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Offhand, my only quibble is that sys.exit is not helpful for debugging.
Much better to raise an error:
if not self.funcTable.get(attribute, lambda x: True)(value):
raise ValueError ('error out of bound')
or define a subclass of ValueError just for this purpose. On error, the
program will stop just as dead, but you'll get a trace.
Mel.
I think generally I prefer my code to die and as long as I know where
(from the statement) the error occurred I know generally what point I
have to debug up until. Can you explain how your solution would be
easier with a trace as I don't tend to use the raise/assert
functionality so I am interested.
thanks
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
Actually, I'd probably create a class with 3 arguments - an initial value, a
lower bound, and an upper bound, give it a _check method, and call _check
from the various operator methods. The class would otherwise impersonate an
int.
In code that isn't performance-critical, it's better to check for errors on
a level that seems excessive to many; this is how I like to program - it
tends to stop bugs in their tracks, rather than allowing them to manifest
somewhere far away, elsewhere in the code, yielding a potentially
time-consuming bug chase. So I'll have if's and asserts checking
assumptions everywhere - at the beginning of a callable, at the end, in the
middle - inspired by Eiffel's heavy assumption checking.
Python code doesn't need to be troublesome. I'm told this is how a famous
chess engine (was it Crafty?) got pretty much all of its bugs out.
Or if things don't need to be in the correct range every step of the way (or
rather, if they must be able to temporarily step outside their bounds), then
you could make self._check() be self.check(), and call it externally on an
as-needed basis instead.
Performance-oriented programming is fun, but not all programming is
performance-oriented, and shouldn't be treated as though it is.
On Fri, Mar 18, 2011 at 7:24 AM, Martin De Kauwe mdeka...@gmail.com wrote:
Hi,
if one has a set of values which should never step outside certain
bounds (for example if the values were negative then they wouldn't be
physically meaningful) is there a nice way to bounds check? I
potentially have 10 or so values I would like to check at the end of
each iteration. However as the loop is over many years I figured I
probably want to be as optimal as possible with my check. Any
thoughts?
e.g. this is my solution
# module contain data
# e.g. print state.something might produce 4.0
import state as state
def main():
for i in xrange(num_days):
# do stuff
# bounds check at end of iteration
bounds_check(state)
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
print Error state values 0: %s % (attr)
sys.exit()
if __name__ == __main__:
sys.exit(main())
thanks
Martin
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bounds checking
On 3/18/2011 10:24 AM, Martin De Kauwe wrote:
def bounds_check(state):
check state values are 0
for attr in dir(state):
if not attr.startswith('__') and getattr(state, attr) 0.0:
print Error state values 0: %s % (attr)
dir() has to do a bit a computation. I would be tempted to give 'state'
a set of attributes to check. Call it 'nonnegatives'.
for attr in nonnegatives:
if ...
This allows for attributes not subject to that check.
--
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
