Re: [Zope-dev] an Interface expressing a list of generator function
On Tue, Nov 9, 2010 at 2:58 AM, Jean-Daniel jeandaniel.bro...@gmail.com wrote: Ok, If I attach an item_type to the the list, I do not think I will able to use the native Python list, and should use a subclass instead. It is a bit heavyweight, especially when the intent of the use of the interface was documentation. Tres could have avoided the generalization and just said something like: class IGeneratorList(Interface): A sequence of generators. def __getitem__(index): Return the 'index'th item. 'index' must be an integer between 0 and __len__() - 1, else raise IndexError. blah blah... Note that zope interfaces are only *semi*-formal. List and generators are very common in Python and are expressed in very few and clear characters, Huh? What does this mean? I thought there would be a shorter way to document their interfaces. What exactly are you hoping to accomplish? Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
Am 08.11.2010, 15:35 Uhr, schrieb Brian Sutherland
br...@vanguardistas.net:
If no-one replies, I'll assume that 3% is just not enough to be
interesting and do nothing;)
Hi Brian,
thanks for sharing this but it looks to me like a micro-optimisation that
isn't really worth going against best practice for - by using explicit
name-mangling.
-return self.__tagged_values.get(tag, default)
+return getattr(self, '_Element__tagged_values', {}).get(tag,
default)
Any improvements higher up the stack (using slots, different Python
version or compilers like LLVM) are likely to have a more significant
effect. Might be worth timing a slots-based implementation.
Charlie
--
Charlie Clark
Managing Director
Clark Consulting Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Tue, Nov 09, 2010 at 01:21:00AM +0200, Marius Gedminas wrote:
On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote:
I've committed 2 patches to a jinty-mem branch of zope.interface.
Together these patches reduce the startup memory use of my ZTK based
application by 3%. Is that enough to justify their risk/complexity?
https://mail.zope.org/pipermail/checkins/2010-November/052516.html
https://mail.zope.org/pipermail/checkins/2010-November/052517.html
If no-one replies, I'll assume that 3% is just not enough to be
interesting and do nothing;)
It's so interesting I'm going to ask for even more numbers: how much is
that 3% in kilobytes, or objects?
Ok, you asked for it ;)
The 3% was in kilobytes as measured using /bin/ps on OSX leopard with
python2.6 from MacPorts. The app I measured is a ZTK 1.0 based web
application.
My application without the changes:
Objects:245614
RSS memory (kb):59116
My application with the changes:
Objects:227147
RSS memory (kb):57180
I also did a micro-benchmark (more below) which indicate that when
creating an interface with 5 attributes and 5 methods you save 41% in
kilobytes and 34% in objects.
Is there any measurable speed impact
for application startup or test runs (faster/slower)?
My app doesn't startup measurably faster or slower. The ZTK tests seem
to run a little faster, but the error is high on my laptop:
Without patches:
./bin/test-ztk 71.93s
./bin/test-ztk 70.49s
./bin/test-ztk 70.31s
With patches:
./bin/test-ztk 70.28s
./bin/test-ztk 70.00s
./bin/test-ztk 69.98s
I also did some micro-benchmarking (script is attached). There were
repeatable results which indicate that startup will be faster and
runtime only probably faster:
Without patches:
Performance bench
-
one interface, 5 methods, 5 functions: 181.00 usec/pass
five inheriting interfaces: 355.76 usec/pass
one interface: 59.79 usec/pass
query uninitialized tagged value: 1.26 usec/pass
query initialized tagged value: 1.27 usec/pass
Memory bench
Memory before (kb):
RSS VSZ
33684 109780
objects created: 150001
Memory after (kb):
RSS VSZ
47856 124116
With patches:
Performance bench
-
one interface, 5 methods, 5 functions: 165.97 usec/pass
five inheriting interfaces: 341.22 usec/pass
one interface: 53.03 usec/pass
query uninitialized tagged value: 0.75 usec/pass
query initialized tagged value: 1.37 usec/pass
Memory bench
Memory before (kb):
RSS VSZ
33328 109524
objects created: 99001
Memory after (kb):
RSS VSZ
41728 117972
Otherwise I'm +1, assuming this doesn't somehow make the code 10%
slower.
Marius Gedminas
--
http://pov.lt/ -- Zope 3/BlueBream consulting and development
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
--
Brian Sutherland
#
# Perdormance Benchmark
#
import os
import timeit
import subprocess
from zope.interface import Interface, Attribute
print Performance bench
print -
s1 = \
class I1(Interface):
a1 = Attribute('one')
a2 = Attribute('two')
a3 = Attribute('thee')
a4 = Attribute('four')
a5 = Attribute('five')
def f1(arg):
pass
def f2(arg):
pass
def f3(arg):
pass
def f4(arg):
pass
def f5(arg):
pass
t = timeit.Timer(stmt=s1, setup=from zope.interface import Interface,
Attribute)
print one interface, 5 methods, 5 functions: %.2f usec/pass % (100 *
t.timeit(number=1)/1)
s1 = \
class I1(Interface):
pass
class I2(I1):
pass
class I3(I2):
pass
class I4(I3):
pass
class I5(I4):
pass
t = timeit.Timer(stmt=s1, setup=from zope.interface import Interface,
Attribute)
print five inheriting interfaces: %.2f usec/pass % (100 *
t.timeit(number=1)/1)
s1 = \
class I1(Interface):
pass
t = timeit.Timer(stmt=s1, setup=from zope.interface import Interface,
Attribute)
print one interface: %.2f usec/pass % (100 * t.timeit(number=1)/1)
setup = \
from zope.interface import Interface, Attribute
class I1(Interface):
pass
s1 = \
I1.queryTaggedValue('bob')
t = timeit.Timer(stmt=s1, setup=setup)
print query uninitialized tagged value: %.2f usec/pass % (100 *
t.timeit(number=5)/5)
setup = \
from zope.interface import Interface, Attribute
class I1(Interface):
pass
I1.setTaggedValue('bob', 'bobby')
s1 = \
I1.queryTaggedValue('bob')
t = timeit.Timer(stmt=s1, setup=setup)
print
Re: [Zope-dev] an Interface expressing a list of generator function
What exactly are you hoping to accomplish? In brief, I wish there were a concise and clear way to express that an object implements a list of generator functions, in the context of CS students learning the concept of interface. I am writing an article on a sudoku solver for an audience of CS students, They do not know much of Python, have recently heard about recursivity and the yield keyword was not obvious for them. The goal is that they meet a nice collaboration between three objects, in Python, to manipulate the concepts of code reuse and interface. There is 1. a Sudoku class modelling a sudoku board and rules, 2. a backtracking algorithm, 3. and a function which returns the data structure (a list of generator functions) used by the backtracking algorithm, and which uses an instance of the Sudoku class, (The solver code can be read at jdb.github.com/modules/sudoku.htmlhttp://jdb.github.com/modules/sudoku.html#Sudoku, a draft of the article is at : jdb.github.com/sudoku.html, the nice backtracking algorithm comes from the conjoin algorithm described in test_generators.py in the Python sources) As the naive implementation the students would write is one big fat function, I have to convince them that going through a more sophisticated design is a better path. The idea is to exemplify code reuse by swapping the Sudoku class with an optimized equivalent class, for instance. Also the same backtracking algorithm can cook a solution to the eight queens problems or the knight's tour problem, without modification but given the adapted list of generator functions. The interfaces help designing formalizing the collaboration between objects and the article wishes to convey this idea. The Sudoku interface is nice and clean but explaining the interface for a list of generator functions is likely to raise more questions from the student than it brings clarity. The interface will require the students to know that list and its bracket notation implies the __getitem__ function, and that generator implies __iter__ and next(). Roughly a dozen elegant lines are needed to implement the interface with the Python keywords and syntax but the expected __getitem__, __iter__ and next() will be absent from the implementation which uses [] and yield. I understand that the interfaces presented by Tres Seaver will work well for the purpose of code reuse in a real application, and it is the way to go if I want to use a global registry and/or a validation of the interface provided by a class. At this point, I think it is easier to explain the interface in plain english in the documentation with the help of the sphinx documentation directives extracting the public methods of the class. Here is (maybe) what I wish I could write: from zope.interface import IList, IGenerator, Interface, provides, validates def make_generator_functions(): [...] make_generator_functions = , make_generator_functions) def stack_assumption(gen_funcs): validates(IList(IGenerator), gen_funcs) [ ... ] Thanks for your attention, Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] an Interface expressing a list of generator function
The end of the last message was not finished... Ideally, here is (maybe) what I wish I could write: from zope.interface import IList, IGenerator, implements, requires def make_generator_functions(): implements(IList(IGenerator)) [...] def stack_assumption(gen_funcs): requires(IList(IGenerator), gen_funcs) [ ... ] Regards, ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Tue, Nov 09, 2010 at 03:49:38PM +0100, Charlie Clark wrote:
Am 08.11.2010, 15:35 Uhr, schrieb Brian Sutherland
br...@vanguardistas.net:
If no-one replies, I'll assume that 3% is just not enough to be
interesting and do nothing;)
Hi Brian,
thanks for sharing this but it looks to me like a micro-optimisation that
isn't really worth going against best practice for - by using explicit
name-mangling.
-return self.__tagged_values.get(tag, default)
+return getattr(self, '_Element__tagged_values', {}).get(tag,
default)
In response to Marius' comments I changed the patch a bit earlier to
this:
def queryTaggedValue(self, tag, default=None):
Returns the value associated with 'tag'.
tv = self.__tagged_values
if tv is None:
return default
return tv.get(tag, default)
Which is is faster as well as not doing the explicit name mangling you
object to.
Any improvements higher up the stack (using slots, different Python
version or compilers like LLVM) are likely to have a more significant
effect.
That would be great, when it arrives.
Actually, in my case, memory is the issue. So the only optimization
that'll really help in a big way is if multiple python processes running
similar code start sharing data.
Some people have poked at that problem, but I havn't seen anything
significant yet.
Might be worth timing a slots-based implementation.
I just tried to add slots to Element, Attribute and Method classes, but
it there's too many tests that break to benchmark it running the ZTK
tests.
On the micro benchmarks, the only difference I see is a memory
improvement which I think will not survive in practice because most
Charlie
--
Charlie Clark
Managing Director
Clark Consulting Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
--
Brian Sutherland
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SVN: zope.interface/branches/jinty-mem/src/zope/interface/interface.py Improve CPU performance of previous memory optimization
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/09/2010 08:26 AM, Wichert Akkerman wrote:
On 11/9/10 14:22 , Brian Sutherland wrote:
Log message for revision 118295:
Improve CPU performance of previous memory optimization
Changed:
U zope.interface/branches/jinty-mem/src/zope/interface/interface.py
-=-
Modified: zope.interface/branches/jinty-mem/src/zope/interface/interface.py
===
--- zope.interface/branches/jinty-mem/src/zope/interface/interface.py
2010-11-09 08:31:37 UTC (rev 118294)
+++ zope.interface/branches/jinty-mem/src/zope/interface/interface.py
2010-11-09 13:22:27 UTC (rev 118295)
@@ -51,6 +51,7 @@
# infrastructure in place.
#
#implements(IElement)
+__tagged_values = None
def __init__(self, __name__, __doc__=''):
Create an 'attribute' description
@@ -72,22 +73,27 @@
def getTaggedValue(self, tag):
Returns the value associated with 'tag'.
-return getattr(self, '_Element__tagged_values', {})[tag]
+if self.__tagged_values is None:
+return default
+return self.__tagged_values[tag]
You can even optimise this further:
tv = self.__tagged_values
if tv is None:
return default
return tv[tv]
that avoids a second attribute lookup. You may also want to benchmark
that versus using a __tagged_values={} on the class and doing a simple
return self.__tagged_values.get(tag, default_
- -1: mutable class defaults are a bug magnet.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzZlFUACgkQ+gerLs4ltQ5ZYQCfRyDUGofCMiER447yJjBeEduu
E5IAniZu6SbOmYZC0XJt/4WeXOY2u5oD
=cNXP
-END PGP SIGNATURE-
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SVN: zope.interface/branches/jinty-mem/src/zope/interface/interface.py Improve CPU performance of previous memory optimization
On 9 November 2010 18:35, Tres Seaver tsea...@palladion.com wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/09/2010 08:26 AM, Wichert Akkerman wrote:
On 11/9/10 14:22 , Brian Sutherland wrote:
Log message for revision 118295:
Improve CPU performance of previous memory optimization
Changed:
U zope.interface/branches/jinty-mem/src/zope/interface/interface.py
-=-
Modified: zope.interface/branches/jinty-mem/src/zope/interface/interface.py
===
--- zope.interface/branches/jinty-mem/src/zope/interface/interface.py
2010-11-09 08:31:37 UTC (rev 118294)
+++ zope.interface/branches/jinty-mem/src/zope/interface/interface.py
2010-11-09 13:22:27 UTC (rev 118295)
@@ -51,6 +51,7 @@
# infrastructure in place.
#
#implements(IElement)
+ __tagged_values = None
def __init__(self, __name__, __doc__=''):
Create an 'attribute' description
@@ -72,22 +73,27 @@
def getTaggedValue(self, tag):
Returns the value associated with 'tag'.
- return getattr(self, '_Element__tagged_values', {})[tag]
+ if self.__tagged_values is None:
+ return default
+ return self.__tagged_values[tag]
You can even optimise this further:
tv = self.__tagged_values
if tv is None:
return default
return tv[tv]
that avoids a second attribute lookup. You may also want to benchmark
that versus using a __tagged_values={} on the class and doing a simple
return self.__tagged_values.get(tag, default_
- -1: mutable class defaults are a bug magnet.
None is immutable so I don't think that is a problem in this case.
I think the is a possible threading issue with Element.setTaggedValue
and Specification.subscribe - if two threads called the method
concurrently, then one of the values might be lost. I think the
correct way to do it would be:
tv = self.__tagged_values
if tv is None:
tv = self.__dict__.setdefault('_Element__tagged_values', {})
tv[tag] = value
This does bring the name mangling back though.
Laurence
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Tue, Nov 09, 2010 at 05:04:11PM +0100, Brian Sutherland wrote: Might be worth timing a slots-based implementation. I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests. On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most ... ? Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Wed, Nov 10, 2010 at 12:14:54AM +0200, Marius Gedminas wrote: On Tue, Nov 09, 2010 at 05:04:11PM +0100, Brian Sutherland wrote: Might be worth timing a slots-based implementation. I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests. On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most ... ? Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] logrotation/logreopening in ZTK-based applications
Hi, Zope 2 provides a command to ask it to re-open its log files thus supporting outside logrotation. We have a ZTK-based app and found that the server provides neither signal handling to rotate logs nor a zopectl command. Does anybody remember wether this was ever built? I see an old discussion on the Python list (http://mail.python.org/pipermail/python-dev/2006-January/059566.html) but no conclusion. If this isn't in yet, does anybody know what the right place would be? (I guess the packages providing the main loops and configuring the process would be the right ones.) Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
