Re: [ZODB-Dev] Re: ZODB Benchmarks
On Feb 4, 2008, at 1:54 PM, Roché Compaan wrote: I don't follow? There are 2 insertions and there are 1338046 calls to persistent_id. Doesn't this suggest that there are 66 objects persisted per insertion? This seems way to high? It seems like there is some confusion about the correspondence between persisting an object and calls to persistent_id(). The pickler makes lots of calls to persistent_id() as it is making pickles. In my mind, persisting an object means saving the new state of an instance of Persistent. When you insert a new persistent instance in a BTree, you are persisting the new instance, the bucket/node that holds the reference to the new instance, and in some cases, some small number of other bucket/nodes that are changed as part of the insertion. That's it. If you insert a bunch of things in one commit(), the number of persistent instances committed is even smaller because some buckets will get multiple changes in one write. There are usually many calls to persistent_id() when one btree bucket is pickled, but I would count that as 1 persistent object. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Why does this useage of __setstate__ fail?
You need to saved the user object with the color attribute.
The __setstate__ is just giving you the unsaved one every time.
Add 'newuser._p_changed = 1' after the first
assert statement below and it will be saved along with
your change to the Color instance.
On Jan 17, 2008, at 11:21 AM, Mika, David P (GE, Research) wrote:
def test_2_setstate(self):
print 'in test2'
newuser = self.userdb[self.id]
# setstate is called subsequently
assert hasattr(newuser, 'color')
assert newuser.getColor() == 'blue'
newuser.setColor('red')
assert newuser.getColor() == 'red'
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 2, 2007, at 8:39 AM, Lennart Regebro wrote: On 11/2/07, Matt Hamilton [EMAIL PROTECTED] wrote: That may just end up causing delays periodically in transactions... ie delays that the user sees, as opposed to doing it via another thread or something. But then as only one thread would be doing this at a time it might not be too bad. But wouldn't then all other threads get a conflict? If they are trying to do insertions at the same time as the consolidation, yes. This data structure won't stop insertion conflicts, the intent is to make them less frequent. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 2, 2007, at 6:20 AM, Lennart Regebro wrote: Lots of people don't do nightly packs, I'm pretty sure such a process needs to be completely automatic. The question is weather doing it in a separate process in the background, or ever X transactions, or every X seconds, or something. Okay, perhaps the trigger should be the depth of the small-bucket tree. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 2, 2007, at 5:48 AM, Lennart Regebro wrote: On 11/1/07, Matt Hamilton [EMAIL PROTECTED] wrote: An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/ branch factor. Would it be possible to not occasionally consolidate, but actually do it ongoing, but just one process, thereby always inserting just one transaction into the normal BTree at a time? Or does that cause troubles? I think that option would work. I think it would suffice to do a Big.update(Small); Small.clear() operation before a nightly pack. It might invalidate every bucket in every cache, but BTrees are designed to perform reasonably without a cache. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote: David Binger dbinger at mems-exchange.org writes: On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote: Ie we perhaps look at a catalog data structure in which writes are initially done to some kind of queue then moved to the BTrees at a later point. A suggestion: use a pair of BTrees, one with a high branching factor (bucket size) and one with a very low branching factor. Force all writes into the tree with little buckets. Make every search look in both trees. Consolidate occasionally. An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/ branch factor. You may be right about that. Conflict resolution makes it harder for me to predict which way is better. If you don't have conflict resolution for insertions, then I think the smaller buckets are definitely better for avoiding conflicts. In either case, smaller buckets reduce the size and serialization time of the insertion transactions, and that alone *might* be a reason to favor them. I think I'd still bet on smaller buckets, but tests would expose the trade-offs. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Import/Export code
On Aug 15, 2007, at 9:34 AM, Christian Theune wrote: pfile = StringIO(data) unpickler = Unpickler(pfile) unpickler.persistent_load = persistent_load newp = StringIO() pickler = Pickler(newp, 1) pickler.persistent_id = persistent_id pickler.dump(unpickler.load()) pickler.dump(unpickler.load()) data = newp.getvalue() Why do we call pickler.dump(unpickler.load()) twice? Just a guess, but I think it is because the original data record is actually the concatenation of two pickles, the first for the class (or something like that) and the second is the actual pickled object state. It looks like this code extracts the pickled state from the record. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problem Storing Infinity
On Sep 12, 2006, at 2:11 PM, Tim Peters wrote: [David Binger] That's interesting. It appears that pickle protocol 2 chokes on inf. As above. BTW, why protocol 2 specifically? Protocols 1 and 2 treat floats the same way. I was thinking that the default protocol is 1, but I see now that the default is 0. 2.4.2 on OS X can pickle infinity using protocol 0, but attempts to pickle infinity using protocols 1 or 2 raise a SystemError. Pickling infinity works fine, just as you say, in 2.5, using any of the protocols. Thanks for the explanation. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Adopting ZODB
On Jul 13, 2006, at 1:35 PM, Chris S wrote:
On 7/13/06, David Binger [EMAIL PROTECTED] wrote:
On Jul 13, 2006, at 12:55 PM, Chris S wrote:
I don't think this is the case. Consider my simple example
below. None
of my classes inherit Persistent, and even though I set
_p_changed =
1, nothing's persisted.
In your example, it seems like there is a missing
root['bob'] = User('bob')
transaction.commit()
to get things started.
That would overwrite my User with a new instance each run, defeating
the purpose of persistence.
The line:
parent = root.get(parentName, User(parentName))
initializes things if the user was not yet created.
Chris
Okay, but the thing that is missing is any assignment to the root
instance
or any instance of Persistent.
I think that is necessary.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Adopting ZODB
On Jul 13, 2006, at 3:55 PM, Chris S wrote: Uh, calling root.get(name, obj) assigns obj to the root if the name is not yet defined. And of course I'm not assigning to any other Persistent instances, since the point of the code was to show that Zope can't persist objects that don't inherit the Persistent class. Chris Okay, sorry. I thought the get method of the root would work like it does on a regular dict. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Getting an up to date instance of a persistent object
On Jun 2, 2006, at 5:05 AM, Pascal Peregrina wrote: Below is a partial illustration of what I was saying earlier: in a given connection, just after a conn.sync(), if I access the _p_serial attribute of a persistent object and the object state is outdated, then the up to date state won't be loaded. But if I then access the _p_mtime attribute, the up to date state is loaded. The _p_mtime attribute is really the unusual one among the _p_* attributes in that accessing it will unghostify the instance (and this seems reasonable). I guess that _p_serial attribute should be understood to mean the serial number of the last loaded state. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Undo differences between Z2 and Z3
On May 17, 2006, at 6:54 AM, Jim Fulton wrote: It was claimed earlier in this thread, that plone only lets you undo the most recent transactions. If this is true, then that should not be a problem. A facility that let you only undo last transactions would be safe. It would also be useful, at least for single-user systems. (Note that definition of last gets a little tricky in the presense of transactional undo.) I would say that the undo the last transaction would be useful for all systems, but safe only for systems where all writing comes from one user. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Query Regrading ZODB FileStorage(.fs file)
On Jan 3, 2006, at 5:31 PM, Tim Peters wrote: [Monica chopra] ... I am still hanging at my point how to hide that data in .fs file as i can see completely all the data if i open that file in notepad or any other editor. ZODB doesn't support encryption directly, and there are no current plans to add such a feature. The alternatives given by others and Tim all seem more attractive for reasons they describe, but I'll just throw out another approach that might be of interest. I think your application could define and use a custom subclass of Persistent, with __getstate__ and __setstate__ methods overridden to provide and use encrypted pickles of the dicts that would otherwise be the state. This requires the encryption key to be available to your application code, so the problem of hiding something is moved (and concentrated). As others have noted, a change like this doesn't really provide additional security in most situations. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:39 AM, Chris Withers wrote: Sorry, my question was that if DateTime's were persistent, would the following code result in a complete pickle for 'a' being written on the second transaction commit? a.someTime = DateTime() get_transaction().commit() wait/do stuff... a.someTime = DateTime() have we just committed a pickle containing all of 'a'?... Not quite. But if you call commit() again, you do indeed save a new pickle of a. Does mixing persistence into DateTime make a difference here? Not with respect to whether or not a second commit contains a pickle of a. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On Nov 10, 2005, at 5:09 AM, Syver Enstad wrote: Thank you Stefan. I am right if I suppose that the reason this fixes all references is that it saves all persistent objects in the database? Will this also fix references to persistent objects in non persistent objects stored in the database? Yes. All such references appear somewhere in the pickle of a persistent object. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On Nov 9, 2005, at 10:37 AM, Jeremy Hylton wrote: How do you find all the references? It seems like you would need to execute a transaction that iterated over every object in the storage and searched the pickle for references to the class. You do need to do this to find the references (using referencesf()?), but it doesn't necessarily need to happen in one giant transaction. The script just needs to (load and) re-pickle every instance with a reference while the sys.modules hack is in place. Commit each referring instance in a new transaction if you like, and conflict errors should not be a problem. I can imagine the update transaction taking a really long time to run, which increases the possibility that it will get a conflict error. Maybe it's a resolvable conflict, but not using the current mechanism. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
