Re: advice needed for lazy evaluation mechanism
On Thu, 12 Nov 2009 08:53:57 +0100, Dieter Maurer wrote: Steven D'Aprano st...@remove-this-cybersource.com.au writes on 10 Nov 2009 19:11:07 GMT: ... So I am trying to restructure it using lazy evaluation. Oh great, avoiding confusion with something even more confusing. Lazy evaluation may be confusing if it is misused. But, it may be very clear and powerful if used appropriately. I was referring to the specific example given, not the general concept of lazy evaluation. I went on to give another example of simple, straightforward lazy evaluation: using properties for computed attributes. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: advice needed for lazy evaluation mechanism
Steven D'Aprano st...@remove-this-cybersource.com.au writes on 10 Nov 2009 19:11:07 GMT: ... So I am trying to restructure it using lazy evaluation. Oh great, avoiding confusion with something even more confusing. Lazy evaluation may be confusing if it is misused. But, it may be very clear and powerful if used appropriately. Lazy evaluation essentially means: you describe beforehand how a computation should be performed but do this computation only when its result is immediately required. Of course, this makes it more difficult to predict when the computation actually happens -- something potentially very confusing when the computation has side effects. If the computation is side effect free, potentially great gains can be achieved without loss of clarity. Python supports lazy evaluation e.g. by its genenerators (and generator expressions). Its itertools module provides examples to efficiently use iterators (and by inclusion generators) without sacrificing clarity. -- Dieter -- http://mail.python.org/mailman/listinfo/python-list
Re: advice needed for lazy evaluation mechanism
On Sun, 08 Nov 2009 14:41:27 -0800, markolopa wrote: Hi, Could you please give me some advice on the piece of code I am writing? My system has several possible outputs, some of them are not always needed. I started to get confused with the code flow conditions needed to avoid doing unnecessary work. How many dozens of man-hours (yours, and the people who have to maintain the software after you have moved on) of confusion are you going to spend to avoid how many microseconds of execution time? So what if your system takes 35ms instead of 18ms to calculate the result? As Tony Hoare famously said: We should forget about the small efficiencies, say about 97% of the time: Premature optimization is the root of all evil. Of course, all of this assumes that the routines you are trying to avoid calling don't require hours of running time each time you call them... So I am trying to restructure it using lazy evaluation. Oh great, avoiding confusion with something even more confusing. - Is there a more standard (pythonic) way to do what I am trying to do? Yes. Avoid it. Do the simplest thing that works until you *know* -- because you have profiled it -- that it is too slow. Until then, all that complication you've built, all that over-engineered jungle of classes and abstract classes, is unnecessary. I find it beyond all credibility that your data is so complicated that you need a plug-in system just to manage the methods you need to calculate your data. Just create some properties, like this one: class Example(object): def __init__(self, birth_year): self.birth_year = birth_year @property def age(self): return 2009 - self.birth_year # FIXME -- it won't be 2009 forever And there you have a lazily-evaluated age property. Not complicated enough? The 3ms it takes to calculate the age is too long? Cache it! class Example(object): def __init__(self, birth_year): self.birth_year = birth_year self._age = None @property def age(self): a = self._age if a is None: a = 2009 - self.birth_year self._age = a return a Now all you need is to invalidate the cache if the birth_year changes. So you make birth_year a property too: class Example(object): def __init__(self, birth_year): self.birth_year = birth_year @property def birth_year(self): return self._birth_year @property.setter # Requires Python 2.6. Also untested. def birth_year(self, year): self._birth_year = year self._age = None @property def age(self): a = self._age if a is None: a = 2009 - self.birth_year self._age = a return a Are there libraries, design patterns, functional programming structures to use to achieve what I am looking for (i.e. am I trying to reinvent the wheel)? The most important buzzwords you want are YAGNI and Premature Generalisation, and perhaps a touch of Architecture Astronaut: http://www.joelonsoftware.com/items/2008/05/01.html - Is the coding style good? - Can I avoid the eval command in Repository.add_routine? What I want there is to be able to have a generic code for the repository which does not depend on the files containing the routines I want it to hold. You mean the exec? cmd = from %s import %s\nroutine = %s() % (file_name, class_name, class_name) exec(cmd) # XXX: ugly Yes. Untested: module = __import__(filename) class_object = getattr(module, class_name) routine = class_object() Of course you can turn that into a one-liner: routine = getattr(__import__(filename), class_name)() -- Steven -- http://mail.python.org/mailman/listinfo/python-list
advice needed for lazy evaluation mechanism
Hi,
Could you please give me some advice on the piece of code I am
writing?
My system has several possible outputs, some of them are not always
needed. I started to get confused with the code flow conditions needed
to avoid doing unnecessary work. So I am trying to restructure it
using lazy evaluation.
In the new mechanism I am coding I have a repository with two types of
objects: infos and routines. In the begining I have a list of
routines. Each routine tells which infos it can compute. The execution
is triggered when the value of an info is requested. In the example
below I have 3 routines
Routine ReadData computes info gender and info birth_year
Routine YearToAge computes info age (using info birth_year)
Routine ComputeMHF computes info max_heart_frequency (using info
gender and info age)
/-- gender \
ReadData --| | -- ComputeMHF --
max_heart_frequency
\-- birth_year -- YearToAge -- age --/
So for instance if all I need is info age, only the routines
ReadData and YearToAge are computed.
The code below implements the example. There are 3 files:
- test.py: the test case for the example
- routines.py: the routines (classes) of the example
- repository.py: the lazy evaluation mechanism (independent of the
example)
My questions are:
- Is there a more standard (pythonic) way to do what I am trying to
do? Are there libraries, design patterns, functional programming
structures to use to achieve what I am looking for (i.e. am I trying
to reinvent the wheel)?
- Is the coding style good?
- Can I avoid the eval command in Repository.add_routine? What I want
there is to be able to have a generic code for the repository which
does not depend on the files containing the routines I want it to
hold.
Note: The routines do not need to declare the info they depend on.
They request the info in the computation phase.
test.py
===
import unittest
from repository import Repository
ROUTINE_LIST =
ReadData routines
YearToAge routines
ComputeMHF routines
class Test(unittest.TestCase):
def test_age(self):
repo = Repository(ROUTINE_LIST)
self.assertEqual(repo['age'], 30)
def test_max_heart_frequency(self):
repo = Repository(ROUTINE_LIST)
self.assertEqual(repo['max_heart_frequency'], 181)
===
routines.py
===
from repository import AbstractRoutine
class ReadData(AbstractRoutine):
def __init__(self):
super(ReadData, self).__init__(self.__class__.__name__,
['birth_year', 'gender'])
def compute(self, repo):
repo['birth_year'] = 1979
repo['gender'] = 'F'
class YearToAge(AbstractRoutine):
def __init__(self):
super(YearToAge, self).__init__(self.__class__.__name__,
['age'])
def compute(self, repo):
repo['age'] = 2009 - repo['birth_year']
class ComputeMHF(AbstractRoutine):
def __init__(self):
super(ComputeMHF, self).__init__(self.__class__.__name__,
['max_heart_frequency'])
def compute(self, repo):
gender = repo['gender']
age = repo['age']
mhf = 211 - age if gender == 'F' else 205 - age
repo['max_heart_frequency'] = mhf
===
repostory.py
===
from StringIO import StringIO
class AbstractRoutine(object):
def __init__(self, name, infos_provided):
self.name = name
self.infos_provided = infos_provided
self.computed = False
def compute(self):
raise NotImplementedError
class Info(object):
def __init__(self, name, routine):
self.name = name
self.routine = routine
self.computed = False
self.value = None
class Repository(object):
def __init__(self, routine_definition_lines):
self._infos = {}
self.add_routines(routine_definition_lines)
def add_routines(self, definition_lines):
for line in StringIO(definition_lines):
line = line.strip()
if line == '':
continue
name, file_name = line.split()
self.add_routine(name, file_name)
def add_routine(self, class_name, file_name):
routine = None # only to cheat pylint
cmd = from %s import %s\nroutine = %s() % (file_name,
class_name,
class_name)
exec(cmd) # XXX: ugly
if not isinstance(routine, AbstractRoutine):
raise ValueError('Class %s is not AbstractRoutine'
% class_name)
for info_name in routine.infos_provided:
info = Info(info_name, routine)
self._infos[info_name] = info
def __setitem__(self, key, value):
if key not in self._infos:
raise ValueError('info %s not defined in repository' %
key)
info = self._infos[key]
if info.computed:
raise
Re: advice needed for lazy evaluation mechanism
Hi again, I put a copy of the message and the tarball of the code here (because of the problem of line breaks): http://python-advocacy.wikidot.com/comp-lang-python-question Thanks! Marko -- http://mail.python.org/mailman/listinfo/python-list
Re: advice needed for lazy evaluation mechanism
markolopa wrote:
Hi again,
I put a copy of the message and the tarball of the code here (because
of the problem of line breaks):
http://python-advocacy.wikidot.com/comp-lang-python-question
Here's a slightly different approach:
repository.py
=
class Repository(object):
def __init__(self):
self._providers = {}
self._values = {}
def add_provider(self, func, keys):
for k in keys:
self._providers[k] = func
def __getitem__(self, key):
if key not in self._values:
self._providers[key](self)
return self._values[key]
def __setitem__(self, key, value):
self._values[key] = value
def register(*modules):
Register the provider modules and return a repository.
repo = Repository()
for mod in modules:
# Scan the module for providers.
# A provider is a function which lists what it provides in its __doc__
string.
for name, value in mod.__dict__.items():
if callable(value) and value.__doc__:
repo.add_provider(value, value.__doc__.split())
return repo
routines.py
===
# The providers are functions which list what they provide in their __doc__
strings.
def ReadData(repo):
'birth_year gender'
repo['birth_year'] = 1979
repo['gender'] = 'F'
def YearToAge(repo):
'age'
repo['age'] = 2009 - repo['birth_year']
def ComputeMHF(repo):
'max_heart_frequency'
gender = repo['gender']
age = repo['age']
mhf = 211 - age if gender == 'F' else 205 - age
repo['max_heart_frequency'] = mhf
test.py
===
import unittest
import repository
import routines
class Test(unittest.TestCase):
def test_age(self):
repo = repository.register(routines)
self.assertEqual(repo['age'], 30)
def test_max_heart_frequency(self):
repo = repository.register(routines)
self.assertEqual(repo['max_heart_frequency'], 181)
unittest.main()
--
http://mail.python.org/mailman/listinfo/python-list
