Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?
On Jan 17, 2013, at 10:31 AM, Claudiu Saftoiu wrote:
> What I don't understand is that this doesn't seem to work in the long run.
> Just before writing this email I ran a view that required a simple query
> after not having restarted the server in a while, and it took a minute or two
> to complete. Running the view again, it took only a few seconds. So it seems
> something had been moved out of the cache, which makes no sense to me as the
> server has plenty of RAM and the cache size is plenty large.
You can use the 'cacheDetail' method of the ZODB to inspect your object
('connection cache') and see how many objects are in there. You are using a
connection cache size of 50, which means 50 ZODB objects per
connection/thread. 'cacheDetail' will help you see how many objects are being
used towards that count of 500,000.
I did some recent investigations where I was looking at what happened as the
result of a catalog query used on part of the home page on a customer site that
is exhibiting similar behaviors. The query in question is for '10 most recent
published weblog articles'. Here's looking at the cachedetail. You can get the
'db' object a number of ways depending on your framework. From any persistent
object, you can get it via '._p_jar.db()'.
from pprint import pprint as pp
from operator import itemgetter
pp(sorted(db.cacheDetail(), key=itemgetter(1), reverse=True)[:20])
[('BTrees.IFBTree.IFSet', 79122),
('BTrees.IOBTree.IOBucket', 21516),
('BTrees.IFBTree.IFTreeSet', 3441),
('BTrees.OIBTree.OIBTree', 415),
Between IFSet and IOBucket, there's 100,000 objects alone that are going into
our object/connection cache count (although another method, 'cacheSize()', says
there are 83,758 items in the cache; I believe this is the non-ghost count).
This is for just one query.
So look at methods like cacheSize(), cacheDetailSize(), cacheDetail(), and if
you're feeling adventurous: cacheExtremeDetail(). They will let you know how
the object/connection cache is actually being used.
I think it's possible that with multiple, rather large BTree based catalog
indexes that some of those IFSets and IOBuckets that make up their internals
can still get flushed out if not exercised by a frequently used query. We've
seen the same behavior on a couple of our biggest customers.
It's also quite possible that those big old catalog indexes have individual
IFSets and Buckets that are getting invalidated since they change state as
object data gets re-indexed. The invalidation causes the ZEO client cache to
need to request a new copy, and I presume this invalidates data in the
connection/object cache as well. Once that happens, IO is required to transfer
the data over the network and/or disk into memory.
> Further, after having preloaded the indices once, shouldn't it preload quite
> rapidly upon further server restarts, if it's all in the cache and the cache
> is persisted?
Again, there are two caches here and they are not really related. The
"persistent cache" is for ZEO to keep local copies instead of having to
constantly hit the network. The object or 'connection' cache is what is in
memory being used by the application. It still requires IO operations to find
all of the bytes from the persistent ZEO cache and move them into memory as
objects. The connection/object cache does not get preserved between restarts.
The client/persistent cache is not a memory dump. If you run the ZODB with just
a local FileStorage file, there is no 'persistent cache' aside from the
database file itself.
Thanks,
Jeff Shell
j...@bottlerocket.net
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repoze.catalog.query very slow
Dear gods, I hope you get an answer to this question, as I've noticed the same
thing with very large indexes (using zc.catalog). I believe that at the root
layers repoze.catalog is built around the same concepts and structures.
When I tried to trace down the problems with a profiler, they all revolved
around loading the relevant portions of the indexes into memory. It had nothing
to do with the final results of the query; had nothing to do with waking up the
'result' objects; all of the slowness seemed to be in loading the indexes
themselves into memory. In one case, we were only using one index (a SetIndex),
with about 15 document ids.
This is from my own profiling. All I know is that this is very slow, and then
very fast, and the object cache and the relevant indexes ability to keep all of
their little BTrees or Buckets or Sets or whatever in that object cache seem to
have a tremendous impact on query and response time - far more than is taken up
by then waking up the content objects in your result set.
When the indexes aren't in memory, in my case I found the slowness to be in
BTrees's 'multiunion' function; the real slowness was in calling ZODB's
setstate (which is loading into memory). This is just BTree (catalog index)
data being loaded at this point:
Profiling a fresh site (no object cache / memory population yet)
winterhome-firstload.profile% callees multiunion
Function called...
ncalls tottime cumtime
{BTrees._IFBTree.multiunion} -> 659800.132 57.891
Connection.py:848(setstate)
winterhome-firstload.profile% callers multiunion
Function was called by...
ncalls tottime cumtime
{BTrees._IFBTree.multiunion} <- 270.348 58.239 index.py:203(apply)
(yep, 58 seconds; very slow ZEO network load in a demostorage setup where ZEO
cannot update its client cache, which makes these setstate problems very
exaggerated). 'multiunion' is called 27 times, but one of those times takes 58
seconds).
Profiling the same page again with everything all loaded
winterhome-secondload.profile% callees multiunion
Function called...
ncalls tottime cumtime
{BTrees._IFBTree.multiunion} ->
winterhome-secondload.profile% callers multiunion
Function was called by...
ncalls tottime cumtime
{BTrees._IFBTree.multiunion} <- 270.1930.193 index.py:203(apply)
(this time, multiunion doesn't require any ZODB loads, and its 27 calls
internal time and cumulative time are relatively speedy)
If there's a good strategy for getting and keeping these things in memory, I'd
love to know it; but when the catalog indexes are competing with all of the
content objects that make up a site, it's hard to know what to do or even how
to configure the object cache counts well without running into serious memory
problems.
On Jan 3, 2013, at 2:50 PM, Claudiu Saftoiu wrote:
> Hello all,
>
> Am I doing something wrong with my queries, or is repoze.catalog.query very
> slow?
>
> I have a `Catalog` with ~320,000 objects and 17 `CatalogFieldIndex`es. All
> the objects are indexed and up to date. This is the query I ran (field names
> renamed):
>
> And(InRange('float_field', 0.01, 0.04),
> InRange('datetime_field', seven_days_ago, today),
> Eq('str1', str1),
> Eq('str2', str2),
> Eq('str3', str3),
> Eq('str4', str4))
>
> It returned 15 results so it's not a large result set by any means. The
> strings are like labels - there are <20 things any one of the string fields
> can be.
>
> This query took a few minutes to run the first time. Re-running it again in
> the same session took <1 second each time. When I restarted the session it
> took only 30 seconds, and again 1 second each subsequent time.
>
> What makes it run so slow? Is it that the catalog isn't fully in memory? If
> so, is there any way I can guarantee the catalog will be in memory given that
> my entire database doesn't fit in memory all at once?
>
> Thanks,
> - Claudiu
> ___
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list - ZODB-Dev@zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
Thanks,
Jeff Shell
j...@bottlerocket.net
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Big catalog indexes/btrees, reads, and the object cache
We've recently noticed some fairly significant speed issues cropping up on a
customer site (zope 3 / bluebream-ish stack). We use zc.catalog and its Value
and Set indexes and it's serves us rather well, up until recently.
In a *cold start* in one environment, profiling the home page reported 27 calls
of the 'index.apply' method, which turned into 27 calls of
'BTrees._IFBtree.multiunion' function, and those 27 multiunion calls had a
Cumulative Time of 58 seconds. Then I looked at the multiunion callees and got
this:
> winterhome-firstload.profile% callees multiunion
>Random listing order was used
>List reduced from 1445 to 1 due to restriction <'multiunion'>
>
> Function called...
> ncalls tottime cumtime
> {BTrees._IFBTree.multiunion} -> 659800.132 57.891
> Connection.py:848(setstate)
I believe this is just doing the raw BTree index loading - not doing any waking
of the objects referenced by the int (document) ids. If I profile this page
immediately afterwards, those 27 calls to multiunion have a cumulative time of
0.193 seconds with no ZODB activity (this is running from a script).
These numbers vary depending on the machine and storages in use, but this seems
to be a constant issue of performance. We seldom see the 57 second times in
production, but we're still seeing pages that are significantly slow, or that
will seem speedy unless they're not used for an hour or more (ie, looking at
some customer-specific reports), and then they're very sluggish again. It feels
like at least some the BTree nodes/buckets/whatever are falling out of the
object cache, because it seems to be doing the raw query that seems to be
taking the most time.
The document counts of these indexes are around 17 items each, and tend to
cover every major-ish content object. I'm wondering how this is impacting the
object caches and how we should be sizing them.
Are we hitting a limit on BTree and/or catalog index size?
Do we need to keep upping the zodb-object-cache size? (What is a good metric
for setting that cache size?)
Thanks,
Jeff Shell
j...@bottlerocket.net
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Empty client cache with beforestorage or demostorage?
We use the combination of 'beforestorage' (zc.beforestorage) and 'demostorage' for development and some testing. For most development, we've been using a local set of ZEO servers built off of backups of production databases, with desktop development changes going into a 'Changes.fs' local filestorage. This has generally worked out fine, and we've started using this practice with a new 'beta' server for some customers that connects to the production ZEO server, but puts changes (again) into a local Changes.fs file. For desktop development this has worked quite well, but I've noticed that when starting up our Zope 3-ish servers I'd see "No verification necessary -- empty cache" messages when connecting to ZEO. But performance was always fine since most of the ZEO loads were small and local. In our new beta setup, however, there seems to be quite a performance impact. Even when configured with a large ZEO client cache for our base/'before' storage setup, some page loads were taking achingly long times and then would appear to be fine. I decided to look at the ZEO client cache file in a hex dump and noticed that it was nothing but zeros (null bytes). Is there a way to configure the demostorage/beforestorage setup so that the client cache can still get populated? This is what our setups look like: cache-size 25000 before now cache-size 800MB client example read-only true server zeo.example.com: storage example var /home/blah/projects/example/parts/zeoclient path /home/blah/projects/example/skiutah/var/db/Changes.fs Or is the non-touching of the client cache just an artifact of one or both of these extra storages (beforestorage or demostorage)? Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Transaction notes
I'd like to add something to our application framework code so that when page messages are displayed to the user (simple messages like 'saved changes' or 'deleted ') that they are also added to the transaction description (using the 'note' method on transactions). Our fsdumps and fstails have very little useful information so I want to re-use an internal API that we use to help provide some information when debugging the amount of writing our application does. My question is: Will using transaction 'note' in a non-writing transaction cause a record to be written out when using FileStorage and/or ZEO? Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Low-level reference finding, tracking, deleting?
On Apr 17, 2010, at 7:43 AM, Laurence Rowe wrote: > On 17 April 2010 05:27, Jeff Shell wrote: >> We encountered a problem during an export/import in a Zope 3 based >> application that resulted in something not being importable. This is from >> our very first Zope 3 based application, and I stumbled across some very old >> adapter/utility registrations that I thought I had cleared out. There are >> references to `zope.interface.adapter.Null` which haven't been around for >> years. This is in an old `LocalAdapterRegistry` which, again, I thought I >> had removed along time ago. These objects and what they reference are not >> part of our normal object graph, and I was surprised to see them. >> >> Given an oid, how can I trace what references that object/oid? There is >> something in our normal object hierarchy retaining a reference, but I don't >> know how to find it, and imagine that trying to investigate/load the objects >> from the ZODB level will help me find the culprit. > > I describe how to do this in an article here: > http://plone.org/documentation/kb/debug-zodb-bloat Oh yeah! I remember looking at that article a few months back. Thanks for the reminder. Looking at our fsdump results, this database seems to have quite a bit of old items that should have been packed / garbage collected a long time ago. This could help me find out why they're not going away. > Since then, Jim has written zc.zodbgc in which the > "multi-zodb-check-refs script will optionally produce a database > of reverse references". > http://www.mail-archive.com/zodb-dev@zope.org/msg04389.html ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Low-level reference finding, tracking, deleting?
On Apr 17, 2010, at 2:07 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Jeff Shell wrote: >> >> Given an oid, how can I trace what references that object/oid? There >> is something in our normal object hierarchy retaining a reference, >> but I don't know how to find it, and imagine that trying to >> investigate/load the objects from the ZODB level will help me find >> the culprit. >> >> Are there low level deletion tools in the ZODB to delete individual >> objects? > > Hmm, maybe you could hack a dummy version of > 'zope.inteface.adapter.Null' back into place, and set a breakpoint > inside its __setstate__? Just a thought. We did put a dummy Null in place to try and get through the import, but didn't think about the breakpoint in __setstate__. Good thought. JPS ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Low-level reference finding, tracking, deleting?
We encountered a problem during an export/import in a Zope 3 based application that resulted in something not being importable. This is from our very first Zope 3 based application, and I stumbled across some very old adapter/utility registrations that I thought I had cleared out. There are references to `zope.interface.adapter.Null` which haven't been around for years. This is in an old `LocalAdapterRegistry` which, again, I thought I had removed along time ago. These objects and what they reference are not part of our normal object graph, and I was surprised to see them. Given an oid, how can I trace what references that object/oid? There is something in our normal object hierarchy retaining a reference, but I don't know how to find it, and imagine that trying to investigate/load the objects from the ZODB level will help me find the culprit. Are there low level deletion tools in the ZODB to delete individual objects? Thanks, JPS ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Understanding the ZODB cache-size option
Are there any metrics about how to set the ZODB 'cache-size' (or cache-size-bytes) option? We've been using '5000' (arbitrarily chosen) for our Zope 3.4-ish app servers. We have access to zc.z3monitor which can output the "number of objects in the object caches (combined)" and "number of non-ghost objects in the object caches (combined)". But I don't know understand how to interpret those numbers and use them to make better settings. Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
On Dec 23, 2009, at 2:26 PM, Jim Fulton wrote: > On Wed, Dec 23, 2009 at 4:20 PM, Jeff Shell wrote: >> >> Are you asking at the ZODB level, or at a higher level, such as Zope? > > I'm asking if applications are using undo these days. > >> We use it in our Zope 3 based CMS. We usually only expose a single-level >> undo and have not exposed any interface to undo more. > > What do you mean by single undo? Undoing just one transaction? Undoing just one transaction. When doing like an action like 'delete' inside of a folder, we add a link to the end of the message that is displayed on the next page to undo: "Deleted 3 items (undo)". If 'undo' is clicked, then the next page has a message to redo (undo the undo). It's always meant to undo a single, recent (but maybe not the most recent) transaction, there as a quick "oops!" recovery mechanism. Once they move on to the next page, the link to 'undo' is gone. Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
On Dec 23, 2009, at 1:26 PM, Jim Fulton wrote: > I'm looking at undo today because it is complicating ZEO transaction > management. > > Undo is broken in a number of ways. Does anyone care? Does anyone use undo? Are you asking at the ZODB level, or at a higher level, such as Zope? We use it in our Zope 3 based CMS. We usually only expose a single-level undo and have not exposed any interface to undo more. I have noticed that trying to access the Rotterdam skin (I guess) "undo more / undo all" views blow up when using ZEO. But again, we don't expose those to our customers and never really use them much ourselves. All we use is some of the APIs from `zope.app.undo` (v3.4.0). The closest we get to interacting with ZODB level code directly is setting extended info on a transaction, basically to make a smarter kind of 'undo last' action. Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Dec 2, 2009, at 5:39 AM, Jim Fulton wrote: > On Wed, Dec 2, 2009 at 7:19 AM, Martin Aspeli > wrote: >> Jim Fulton wrote: >> >>> In any case, nothing is deleted in subversion. If someone creates even >>> minimal tests that exercise the code you changed, then I'll >>> be happy to reapply the change. At some point, if no one takes >>> responsibility for repozo, I'm inclined to remove it from the >>> distribution, as there is an implied promise of support, >>> which seems to be lacking. >> >> I think all hope of anyone taking the ZODB seriously outside those >> already wedded to it would be lost if there wasn't some kind of >> supported backup solution. > > Have I ever said a supported backup solution wasn't important? > > BTW, offline back ups if file-storage even with Blobs is > straightforward without repozo. No, it's not. What do we use to check for integrity of the other backups? fsrefs? fstest? fsoids? Many of the ZODB Scripts are barely useful and certainly not 'straightforward'. Tangent: 'straightforward' is the new 'simply'. Just because one is close to the tools and has been using them for many years does not mean that everyone else has them. The ZODB tool chain is not discoverable. And I think it's funny that you give so much concern to Blobs when I have abandoned them in sheer frustration. And I recognize your concerns about angering / frustrating / losing users because a tool does not seem to work, because that's how I feel about Blobs. >> The correct course of action here, in my opinonion, is not to make >> threats, but to try to encourage people to step up. I find that a >> positive message works much better for encouragement than a negative one. > > I haven't made any threats. It is dishonest to include a backup > tool in ZODB that isn't supported. Users would have expectations > that aren't met. I don't think that any of the ZODB scripts are in good shape. There's nothing I can confidently give to a system administrator. There's an old Wiki page of documentation for a handful of the tools, and that's about it. At least 'repozo' has good and useful information to spit out when you use the '-h' / '—help' switch. I'm not sure whose expectations you're satisfying as the items in ZODB.scripts are a wild and inconsistent mess. It's been trial and error just to find ones that seem to work, outside of repozo. — Jeff Shell ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
On Nov 20, 2009, at 11:52 AM, Jim Fulton wrote: >> If configured, sure. > > I would not trust repozo + blob backups. I don't trust Blobs. A year ago, I reverted all of the code I had for our CMS that supported Blobs as it was impossible to copy their data in a basic Zope 3 copy/past/move. I think that may have been fixed, but since there's practically zero documentation on how to use blobs, use them wisely, use them well, and use them in fairly plain Zope 3-ish applications, I lost confidence. > Fair enough. Given that this provokes the symptom that was fixed. > I can live with this. > > I also would fear to rely on repozo in it's current state, but I don't have > to. We live in fear of most of the tools in their current state. So they pass the tests - whoopee! But the documentation is so bare and minimal that we (Bottlerocket) still don't properly configure anything. I guessed at some default values and those get copied around, hoping for the best. We've encountered problems with ZEO in the long-ago past that I'm sure were due to configuration misunderstandings that still hold over us to this day. We use ZEO, but I don't think we use it well. Last time I looked for documentation, many of the documents seemed either from the early Zope 2 era or came from the Plone community. While thankful for some of the documents from the Plone community, we're not a Plone shop, nor a Zope 2 shop any more (except for a couple of customers). Some of the tools they reference are unavailable to us. I still don't know what the data from zc.monitor means or how we can use that to adjust ZEO / ZODB configuration settings to improve performance of a site. So while the tests can make us all confident that the ZODB/ZEO system works, how about some documentation coverage that could help us be confident that we're using it correctly? For what it's worth, I rely on but also somewhat fear repozo - not because of its tests or lack thereof (it seems to work for me!), but because documentation and help for the scripts are lacking (fortunately, 'repozo' is the one exception. 'fsrefs' and 'fsdump' are mysterious. There's a lot of other tools in ZODB.scripts that seem rather antiquated, or intimately tied to Zope 2. But chances are that no one even knows those tools are there). Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
On Nov 20, 2009, at 7:32 AM, Chris Withers wrote: > Jim Fulton wrote: >> I'd prefer that there be a file-storage backup solution out of the box. >> repozo is the logical choice. It sounds like it needs some love though. >> This isn't something I'd likely get to soon. > > I'm not sure how much love repozo needs. It works, and it won't need > changing until FileStorage's format changes, which I don't see happening > any time soon. For what it's worth, I've dealt with broken / unloved / mildly-out-of-date ZODB scripts by making a new package that my little company uses internally. It allowed me to get some useful tools available again, and was independent from any ZODB release cycle. Now I have additional scripts that get generated by my ZODB-related buildouts with names like 'fsdumpx'. You could always explore this option, and even do it better than me by sharing with the community. The ZODB 'scripts' are a wild handful of relics, some useful, some not, some just woefully out of date. It would be helpful to have an independent project that provided new or fixed tools that were maintained, were better command-line citizens, had consistent command line options, etc. There's no reason for such a project to be tied to the longer tail of the ZODB release cycle. So if 'repozo' needs love, even if it's simple love, I would recommend that someone make a 'z3c.zodbtools' or 'z3c.repozo' or 'z3c.fsbackup' project. Hell, I'll look at the package I've made and see if I can do something along those lines as a starting/example point. (I believe my zodbtools package has some dependencies on a couple of internal packages that can not be shared, but I could scrub those uses out). — Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB.__version__
You can try the `pkg_resources` APIs and get the version from the
setuptools data.
>>> import pkg_resources
>>> pkg_resources.require('ZODB3')[0].version
'3.8.1'
Which only works if ZODB3 was installed via buildout/setuptools, of
course.
On Jun 8, 2009, at 2:30 AM, Pedro Ferreira wrote:
> Hello,
>
> I've noticed that the __version__ variable hasn't been updated lately:
>
> i.e. on 3.8.1:
>
>>>> import ZODB
>>>> ZODB.__version__
> '3.7.0b3'
>
> And on 3.9x it doesn't even exist.
> Was it officially deprecated? If so, is there any other way to check
> the
> library version?
>
> Thanks in advance,
Thanks,
Jeff Shell
j...@bottlerocket.net
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Analyzing the committed data during one transaction
On Nov 17, 2008, at 10:20 AM, Alan Runyan wrote: > On Mon, Nov 17, 2008 at 9:01 AM, Jim Fulton <[EMAIL PROTECTED]> wrote: >> >> On Nov 17, 2008, at 8:21 AM, Andreas Jung wrote: >> ... >>> Rephrasing my question in a different way: how can I get hold of the >>> "parent" object based a persistent object with a _p_jar >>> connection? E.g. when the ZODB commits a subobject (e.g. a BTree or >>> a bucket) of portal_catalog/Indexes/some_index then I would like to >>> get hold of the the related parent "first-class" Zope 2 object (e.g. >>> derived from SimpleItem/Item or something like that). >> >> There isn't a practical way to do that. (If you're willing to work >> very very very hard, you can iterate over a database and build up a >> reverese reference structure.) > > FWIW: I've wanted to do this same thing before but found no toolchain > support and gave up. It would probably be best to start with the `fsdump` tool. Here's a transaction record from a dump I had lying around. Trans #00720 tid=03744947f76dbf66 time=2008-03-07 20:23:57.991102 offset=179864483 status='p' user='/ example.user.1' description="Deleted 'thirty foot flat bar' [galleryphoto5]" data #0 oid=45f6 size=278 class=BTrees._IOBTree.IOBucket data #1 oid=0002b67d size=305 class=BTrees._IFBTree.IFSet data #2 oid=0002b648 size=320 class=BTrees._IFBTree.IFSet data #3 oid=0002b67f size=320 class=BTrees._IFBTree.IFSet data #4 oid=6db8 size=295 class=BTrees._IFBTree.IFSet data #5 oid=4f15 size=823 class=BTrees._IFBTree.IFBucket data #6 oid=4f0e size=325 class=BTrees._IFBTree.IFSet data #7 oid=3a9d size=1604 class=BTrees._IOBTree.IOBucket data #8 oid=0002b61b size=830 class=BTrees._OIBTree.OIBucket data #9 oid=00037ad9 size=300 class=BTrees._IFBTree.IFSet data #00010 oid=0002c6f5 size=568 class=BTrees._OOBTree.OOBucket data #00011 oid=0002b661 size=448 class=BTrees._OOBTree.OOBucket Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Oct 24, 2008, at 8:36 AM, Christian Theune wrote: > Again: blobs export as expected. > > However, Jeff pickles/unpickles the subtree to get rid of the > '__parent__' attribute. This and blobs' inability to get copied around > in this specific manner make him end up with empty blobs. This is the real issue. What can be done to address it? This sounds like something that should be handled at/near the ZODB/Pickle/Copy (deep copy?) level. While doing some google searches on this issue, I saw others having same or similar concerns when using Blobs with other tools that did deep copies of objects, although their bugs seemed related more to the issue of having 'None' values in the readers and writers attributes of the Blob file (a bug which has been fixed). The initial traceback in this bug report is tied to that issue: https://bugs.launchpad.net/zope3/+bug/240381 But then things swing around to actual content copying. Christophe Combelles last comment (from June) shows manually patching 'zope.locationlocationCopy' to address his particular situation. 'loc' is the original 'located' object being copied. copied = unpickler.load() if hasattr(copied, '_data') and type(copied._data) is Blob: copied._data._create_uncommitted_file() targetfile = open(copied._data._current_filename(), 'w') sourcefile = loc.openDetached() chunk = sourcefile.read(100) while chunk: targetfile.write(chunk) chunk = sourcefile.read(100) targetfile.close() return copied That's obviously not a general solution, or the most efficient, but I'm guessing it's pretty close to what needs to happen... I just verified that with our CMS (built on Zope 3) and custom 'BlobFile' content type object, copying the BlobFile content type object copies all of the other persistent data of the object, but the Blob contents are indeed empty. This is using a copy/paste system built on top of zope.copypastemove. Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Oct 24, 2008, at 7:51 AM, Jim Fulton wrote: > A few high-level observations. (I don't have time to get into the > weeds.) > > 1. The parent pointers used in Zope 3 are, obviously, a problem for > ZODB exports. You've already noticed this. > A possible way to mitigate this would be to extend the export > API to be able to omit objects from the export. >Then, when copying a tree, you could tell it to omit the parent > of the root. There would have to be some way >to tell it what to use instead. Since Export appears to work by following OID references, I guess a stupid simple way of providing alternate values/lookups for certain OIDs might work. I'm not sure how zc.copy / zope.location.pickling.locationCopy actually do their work (the guts of pickling are a mystery to me), but I imagine that there could be a way of providing something similar for exporting - basically some way of finding out if a referenced object is outside the containment zone and providing an alternative value/ reference. > 2. I doubt that blobs have been factored into ZODB exports. This is, > obviously, an oversight. They're actually factored in quite well. The copy stunt which I use to kill the __parent__ link of the root exported object is the problem. This is the scenario I now worry about - how Blobs will work in applications like Zope which provide 'copy/paste' functionality. > 3. I think that zope.fssync/zope.app.fssync might provide a better > export/import technology, although they *may* require more work. On > the up site, they are more pluggable. We are, *finally*, about to > start using zope.fssync in a production application. I'll take a look at those. I think they're what we (Bottlerocket) need (at least at a conceptual level). Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Oct 24, 2008, at 5:06 AM, Christophe Combelles wrote: >> What can I do here? I'm in the midst of a botched deployment of some >> content updates for a customer. I thought that copying something >> 'containing' a Blob would cause its contents to be copied. If this is >> an exercise for application layer code, how can I get plugged in and >> know when a Blob is being copied inside of a deep copy? > > > I suppose it is related to this? > https://bugs.launchpad.net/zope3/+bug/240381 It is exactly related to that. As Christian mentions elsewhere in this thread, it's the copy that kills the blob content due to something happening at Pickle time. Does the patch in that bug report (which patches locationCopy) work on all objects inside of a deep copy? Or does it only work on the root object being copied? Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
For the past few years we've been using an ugly, but working, trick to export data out of a Zope 3 instance without exporting every bit of content. Given an object to export, we use 'locationCopy' (zope.location.pickling) to get a deep copy, set the __parent__ attribute of the copied object to None, put it in the ZODB root (root[EXPORT_KEY] = detached_copy), and then commit the transaction. This is so that we have the 'oid' and connection needed for ZODB's 'exportFile' to work. If there's a better way of setting `__parent__` to None (or just not following the __parent__ ref during Export, without affecting the saved state in the database), I'd love to know. But that's an aside. I exported some content tonight that had Blobs in it. Upon import, I noticed that the blob files in the destination were empty (zero- length). Looking at the ZEXP file, I could see the BLOBSTART markers, but could see no binary data. Tracing down into the export, and then just playing around with 'locationCopy' to make other detached copies, I noticed that the Blob contents were never copied into the new file. What can I do here? I'm in the midst of a botched deployment of some content updates for a customer. I thought that copying something 'containing' a Blob would cause its contents to be copied. If this is an exercise for application layer code, how can I get plugged in and know when a Blob is being copied inside of a deep copy? Thanks, -- Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] conflict errors in persistent ZEO client cache
On Sep 25, 2008, at 6:31 AM, Jim Fulton wrote: > On Sep 25, 2008, at 1:12 AM, Jeff Shell wrote: >> In ZODB 3.8.0, while prepping a customer's site to go from staging to >> production and doing a fair amount of big operations, we started >> seeing conflict errors. It was a 200MB cache size, with a persistent >> client cache. Keeping the same cache size (against a base storage of >> about 700MB) but turning off the persistent client cache seemed to >> make the problem go away. >> >> The persistent cache file at the time of the conflict error was close >> to 200MB in size, but doesn't it always go to the size configured, >> regardless of how much data is inside? > > Since ZODB 3.8.0, I have fixed *many* ZEO cache bugs. What you > describe is one symptom. I suggest upgrading to 3.8.1b8. Just to follow up and hopefully help get 3.8.1 out the door - trying these sites and configurations on 3.8.1b8 worked much better than on 3.8.0. Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] conflict errors in persistent ZEO client cache
In ZODB 3.8.0, while prepping a customer's site to go from staging to production and doing a fair amount of big operations, we started seeing conflict errors. It was a 200MB cache size, with a persistent client cache. Keeping the same cache size (against a base storage of about 700MB) but turning off the persistent client cache seemed to make the problem go away. The persistent cache file at the time of the conflict error was close to 200MB in size, but doesn't it always go to the size configured, regardless of how much data is inside? Anyways, I saw this in the ZODB 3.8.1 beta release notes: > - When using ZEO Client Storages, Errors occured when trying to > store objects too big to fit in the ZEO cache file. Out of curiosity, what Errors were occurring? I don't think we were trying to store individual objects too big to fit in the ZODB, but the transactions were likely touching a lot of objects. I'm trying to figure out if I had a configuration problem, software problem, or both. Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
