[Zope3-dev] Re: 64-bit BTrees
Fred Drake wrote: > I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), > and I'm not the first. I've created a feature development branch for > this, and checked in my initial implementation. > > I've modified the existing code to use PY_LONG_LONG instead of int for > the key and/or value type; there's no longer a 32-bit version in the > modified code. Any Python int or long that can fit in 64 bits is > accepted; ValueError is raised for values that require 65 bits (or > more). Keys and values that can be reported as Python ints are, and > longs are only returned when the value cannot be converted to a Python > int. > > This can have a substantial effect on memory consumption, since keys > and/or values now take twice the space. There may be performance > issues as well, but those have not been tested. > > There are new unit tests, but more are likely needed. > > If you're interested in getting the code from Subversion, it's available at: > > svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ > > Ideally, this or some variation on this could be folded back into the > main development for ZODB. If this is objectionable, making 64-bit > btrees available would require introducing new versions of the btrees > (possibly named LLBTree, LOBTree, and OLBTree). -1 to the L*BTree flavours. The 'long' type is disappearing from Python anyways, why should the implementation detail worry us regarding BTrees? +1 to making the I*BTree flavours 64bit-aware. If Jim is right, existing pickles will be read back just fine and as long as the integers stay under 32 bit, they won't even be bigger. The only real implication therefore is the pushing around twice as many bits in memory. Dieter estimates 20% to 35% slowdown for the C algorithms (whatever that means), Tim seems to think it won't have such a big effect. I guess we'll only know after some benchmarks. Philipp ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: [ZODB-Dev] Re: [Zope-dev] Re: 64-bit BTrees
On 4/17/06, Jim Fulton <[EMAIL PROTECTED]> wrote: > The fact that IIBTrees is so widely used is exatly the reason > I want to use 64-bits for the existing types rather than having to > introduce a new type. Oops, I was checking in the separated version of 64-bit BTrees while this was landing in my inbox. ;-/ Once we determine which approach we're going with, I should make an alpha release of ZODB 3.7 and knit that into the Zope 3 trunk so we can get more testing in context. -Fred -- Fred L. Drake, Jr. "Don't let schooling interfere with your education." -- Mark Twain ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: [Zope-dev] Re: 64-bit BTrees
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). I think coming up with new types is the only reasonable thing to do, given the prevalence of persistent BTrees out in the wild. Changing the runtime behavior (footprint, performance) of those objects is probably not something which most users are going to want, at least not without carefully considering the implications. It really depends on what the impact is. It would be nice to get a feel for whether this really impacts memory or performance for real applications. This adds 4-bytes per key or value. That isn't much, especially in a typical Zope application. Similarly, it's hard to say what the difference in C integer operations will be. I can easily imagine it being negligible (or being significant :). OTOH, adding a new type could be a huge PITA. We'd like to use these with existing catalog and index code, all of which uses IIBTrees. If the performance impacts are modest, I'd much rather declare IIBTrees to use 64-bit rather than 32-bit integers. I suppose an alternative would be to add a mechanism to configure IIBTrees to use either 32-bit or 64-bit integers at run-time. Who uses IOBTree / OIBTree / IIBTree? - Catalogs map RIDs to UIDs as IOBTrees (one record per indexed object) - Most indexes (those derived from Unindex) map RID to indexed value as an IOBTree (one record per object with a value meaningful to that index) and map values to RIDs as OOBTrees (where the second O is usually an IITreeSet). - ZCTextIndex uses IIBTrees to map word IDs to RIDs, in various ways, and make use of IOBTrees as wel.. - Relationship "indexes" (typically not stored within catalogs) usually have an IIBTree which is the mapping of the edges as pairs of internal node IDs (one per explicit relationship), with OIBTrees to map the user-supplied node value to a node ID. I would guess that if you could do a census of all the OIDs in all the Datas.fs in the world, a significant majority of them would be instances of classes declared in IOBTree / IIBTree (certainly the bulk of *transaction* records are going to be tied up with them). OK. I think we are misscommunicating. Using 64 bits for IIBTrees types would not in any way invalidate existing pickles. 64-bit IIBTrees types can be unpickled from existing data. Of course, someone who created 64-bit BTrees type instances that had values outside the 32-bit range would have trouble reading these values with 32-bit IIBTrees, The fact that IIBTrees is so widely used is exatly the reason I want to use 64-bits for the existing types rather than having to introduce a new type. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: what is ZCML?
Shane Hathaway wrote: Sidnei da Silva wrote: On Mon, Mar 13, 2006 at 02:16:17PM -0700, Jeff Shell wrote: | And I think it's | very important for the Python code to say what it does, so when I come | back to a module five months later I'm not staring at MyFactory going | "yeah, but what is it?" One thing that must not pass by unnoticed is that one of the points of 'Why ZCML' is that it allows you to 'do stuff' (configuration?) with plain python code that wasn't written for, nor depends on, Zope 3 directly. That is, to me, a very important feature. To be able to write some python module that does not depend on Zope 3 at import time, but is 'hooked into' Zope 3 externally, with ZCML, at 'configuration time'. As I understand, no other framework out there gives you this. I would suggest that is a component architecture feature, not a ZCML feature. true, but zcml + configuration system is how you currently access this feature. The runtime execution of configuration + having a unified configuration format has been fairly important to creating systems that have to deal with backward compatibility in a transparent, organized way(I'm thinking of Five here). in zope3 such monkeypatch madness isn't a concern. but zcml isn't used just in zope3 anymore. -w ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: zope.app.rdb bug?
On Mon, Apr 17, 2006 at 09:27:07AM -0500, Jeff Rush wrote:
> > Specifically, we're using the connection in a
> > non-persistent shopping cart object. Upon initialization the cart object
> > retrieves the connection from a persistent parent in the context in which
> > the cart is initialized. In this way, different shopping cart items can be
> > stored in different databases depending upon the container. The parent is
> > essentially a copy of ISQLScript.
>
> I copied SQLScript as well. Be sure that on each SQL operation you re-fetch
> the connection object using the connection name. Do not hold on to the
> connection object itself, as it may change during system re-configuration.
>
> Here is my calling method:
>
> def invoke_SQL(self, query):
>
> cache = getCacheForObject(self)
> location = getLocationForCache(self)
> if cache and location:
> _marker = object()
> result = cache.query(location, {'query': query}, default=_marker)
> if result is not _marker:
> return result
>
> try:
> connection = zapi.getUtility(IZopeDatabaseAdapter,
> self.connection_name)()
> except KeyError:
> raise AttributeError, (
> "The database connection '%s' cannot be found." % (
> self.connection_name))
>
> result = queryForResults(connection, query)
> if cache and location:
> cache.set(result, location, {'query': query})
> return result
>
> Notice the zapi.getUtility() call to re-fetch each time.
At one point in sqlos development we were re-fetching the connection
every time to get local utilities working. That turned out to have a
noticeable performance penalty, i.e. people noticed and started to
complain;)
So these days we have a per thread cache that is cleaned out before
traversing a site and at the end of every request.
--
Brian Sutherland
Metropolis - "it's the first movie with a robot. And she's a woman.
And she's EVIL!!"
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] RE: zope.app.rdb bug?
Good. I think I'm clear. That procedure is exactly what I am following,
except I don't cache queries for this application. My database connection
is a persistent utility. I'll give the global utility a try.
> -Original Message-
> From: Jeff Rush [mailto:[EMAIL PROTECTED]
> Sent: Monday, April 17, 2006 9:27 AM
> To: David Johnson
> Cc: zope3-dev@zope.org
> Subject: Re: zope.app.rdb bug?
>
> David Johnson wrote:
> >>
> > In regards to persistence, are we saying this problem occurs when the
> > connection is an attribute of a persistent object? In our case the
> > connection resides in a persistent object but is called and used in a
> > non-persistent one.
>
> It doesn't matter if the 'caller' of the connection is persistent or not,
> but
> that the connector is itself a persistent, local utility stored in the
> site
> manager folder.
>
> > Specifically, we're using the connection in a
> > non-persistent shopping cart object. Upon initialization the cart
> object
> > retrieves the connection from a persistent parent in the context in
> which
> > the cart is initialized. In this way, different shopping cart items can
> be
> > stored in different databases depending upon the container. The parent
> is
> > essentially a copy of ISQLScript.
>
> I copied SQLScript as well. Be sure that on each SQL operation you re-
> fetch
> the connection object using the connection name. Do not hold on to the
> connection object itself, as it may change during system re-configuration.
>
> Here is my calling method:
>
> def invoke_SQL(self, query):
>
> cache = getCacheForObject(self)
> location = getLocationForCache(self)
> if cache and location:
> _marker = object()
> result = cache.query(location, {'query': query},
> default=_marker)
> if result is not _marker:
> return result
>
> try:
> connection = zapi.getUtility(IZopeDatabaseAdapter,
> self.connection_name)()
> except KeyError:
> raise AttributeError, (
> "The database connection '%s' cannot be found." % (
> self.connection_name))
>
> result = queryForResults(connection, query)
> if cache and location:
> cache.set(result, location, {'query': query})
> return result
>
> Notice the zapi.getUtility() call to re-fetch each time.
>
> -Jeff
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: zope.app.rdb bug?
David Johnson wrote:
In regards to persistence, are we saying this problem occurs when the
connection is an attribute of a persistent object? In our case the
connection resides in a persistent object but is called and used in a
non-persistent one.
It doesn't matter if the 'caller' of the connection is persistent or not, but
that the connector is itself a persistent, local utility stored in the site
manager folder.
> Specifically, we're using the connection in a
non-persistent shopping cart object. Upon initialization the cart object
retrieves the connection from a persistent parent in the context in which
the cart is initialized. In this way, different shopping cart items can be
stored in different databases depending upon the container. The parent is
essentially a copy of ISQLScript.
I copied SQLScript as well. Be sure that on each SQL operation you re-fetch
the connection object using the connection name. Do not hold on to the
connection object itself, as it may change during system re-configuration.
Here is my calling method:
def invoke_SQL(self, query):
cache = getCacheForObject(self)
location = getLocationForCache(self)
if cache and location:
_marker = object()
result = cache.query(location, {'query': query}, default=_marker)
if result is not _marker:
return result
try:
connection = zapi.getUtility(IZopeDatabaseAdapter,
self.connection_name)()
except KeyError:
raise AttributeError, (
"The database connection '%s' cannot be found." % (
self.connection_name))
result = queryForResults(connection, query)
if cache and location:
cache.set(result, location, {'query': query})
return result
Notice the zapi.getUtility() call to re-fetch each time.
-Jeff
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] RE: zope.app.rdb bug?
> If it's related to persistence, then perhaps a workaround for David in the > meantime would be to supply the connector as a global utility not stored > in > the ZODB. I haven't seen the problem occur here and perhaps that's > because I > declared my adapter in site.zcml so it never gets ghosted: > >name="dbfinance" >component="psycopgda.adapter.PsycopgAdapter" >dsn="dbi://webaccountant:[EMAIL PROTECTED]/finance" >/> > In regards to persistence, are we saying this problem occurs when the connection is an attribute of a persistent object? In our case the connection resides in a persistent object but is called and used in a non-persistent one. Specifically, we're using the connection in a non-persistent shopping cart object. Upon initialization the cart object retrieves the connection from a persistent parent in the context in which the cart is initialized. In this way, different shopping cart items can be stored in different databases depending upon the container. The parent is essentially a copy of ISQLScript. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: [Zope-dev] 64-bit BTrees
Fred Drake wrote: (possibly named LLBTree, LOBTree, and OLBTree). For my half-penny's worth, this is the way I'd like to see it go. Explicit is better than implicit and all that. If you need more than 32-bits, you can explicitly use them. The implicit change to make them all 64-bits which results in some unknown slowdown for all "I" BTree users seems a bit too scary to bite off... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: zope.app.rdb bug?
Jeff Rush wrote: Dmitry Vasiliev wrote: David Johnson wrote: I am using the zope database adapter, and on occasion (1/20) the following code intermittently throws an exception (1/20 times), even though the connection is connected. It's a known bug, see http://www.zope.org/Collectors/Zope3-dev/582. The problem is that the adapter sometimes can't be loaded from the ghost state. However, I still can't understand why the connection is closed (or reported as closed) while the request should be still in process... If it's related to persistence, then perhaps a workaround for David in the meantime would be to supply the connector as a global utility not stored in the ZODB. I haven't seen the problem occur here and perhaps that's because I declared my adapter in site.zcml so it never gets ghosted: I guess the adapter should work just fine as global utility, but we need to fix the bug anyway. -- Dmitry Vasiliev (dima at hlabs.spb.ru) http://hlabs.spb.ru ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
