Re: [ZODB-Dev] Persisting Python objects in MongoDB
as you might have seen from my checkins on Friday, I released mongopersist. In short, it implements a persistence backend in MongoDB (that, with a little bit of effort, could be used as a ZODB replacement in Zope): http://pypi.python.org/pypi/mongopersist Sweet! Thanks a lot for this great development! Pedro -- José Pedro Ferreira Software Developer, Indico Project http://indico-software.org +---+ + '``'--- `+ CERN - European Organization for Nuclear Research + |CERN| / + 1211 Geneve 23, Switzerland + ..__. \. + IT-CIS-AVC + \\___.\ + Office: 513-1-005 + /+ Tel. +41227677159 +---+ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persisting Python objects in MongoDB
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Very cool and excellent work. Andreas Stephan Richter wrote: Hi everyone, as you might have seen from my checkins on Friday, I released mongopersist. In short, it implements a persistence backend in MongoDB (that, with a little bit of effort, could be used as a ZODB replacement in Zope): http://pypi.python.org/pypi/mongopersist Some background and motivation: Many years ago, Jim told me that the persistence code could be used to implement any storage backend. But until now I never had the need to confirm that statement. The ZODB and its various storages served me just fine. But then I decided to use MongoDB as storage for my latest project. (I just recently used storm with PostGreSQL for a project and was once again annoyed about the impedance mismatch between the stiff relational and flexible object worlds.) It was out of question that I just wanted to use pymongo straight up, since it makes admin interfaces just very painful to write. I first tried p01.mongo, but unfortunately it tries to re-implement persistence with some of the most annoying ORM-style features (such as explicit attribute declarations). So, I came to the conclusion I wanted to build an abstraction layer myself and see how well it would work. An ORM-style solution was out of question. I consider attribute-type declarations ORM-style, since it prohibits me to extend objects in arbitrary ways (think annotations, sub-classing, etc). I then looked into implementing a ZODB storage. Unfortunately, the ZODB really wants to deal with pickles and I wanted to store objects in a way that the data could be used without Python. Thus, implementing a storage mechanism on top persistence seemed like the best choice. It was not too hard, since ZODB's storage implementations provided good examples. Here are some of the main features: * Full persistent data manager implementation making storing objects seamless. * Data manager can be used to load and dump objects directly. * Automatic and explicit specification of the mongo collection to use. * Store multiple object types in the same collection. This is particularly useful, if you have sub-classes of an object type. * Storage of non-persistent objects. Pretty much everything that can be pickled can be stored. There is some work left to do to support all pickle hooks and object constructors. * Plugins to provide custom serializers and de-serializers for objects. This is useful for objects whose reduced state is not suitable for Mongo storage, such as datetime.date. * Control over document granularity. In the ZODB, any persistent object causes a separate pickle/document. In mongopersist you can choose whether a persistent object is stored as part of its parent object or not. (The Mongo guys make a big deal about choosing document granularity.) * Cross database support, i.e. full DBRef object references are used everywhere. * Optional write-conflict detection. * Simple Mapping implementation that represents a collection. * Optional Zope Support: Multiple container implementations for various use cases. The main container supports one collection for multiple instances by keeping a parent reference. The container itself can be connected to a ZODB or mongopersist. The container also provides full access to the MongoDB query API, filtering by parent automatically and supporting raw or object results. Before you jump on this right away, here are some features that are missing: * BTreeContainer instances will probably store, but they would not provide sensible output in Mongo. Custom hooks need to be written to handle BTreeContainers sensibly. * Since the pickle module does not cleanly separate reducing an object and creating a pickle string, mongopersist must effectively re-implement the pickle API. I have implemented the most common use cases, but some pickle hooks and handling of classes with __new__ constructors are still missing. * Cyclic references of non-persistent objects is not supported. That said, we are about to head into production with mongopersist. If you need any help or want those above two problems addressed, feel free to contact me. So was Jim right? Absolutely! Regards, Stephan - -- ZOPYX Limited | zopyx group Charlottenstr. 37/1 | The full-service network for Zope Plone D-72070 Tübingen| Produce Publish www.zopyx.com | www.produce-and-publish.com - E-Publishing, Python, Zope Plone development, Consulting -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJOuUnuAAoJEADcfz7u4AZj7Z4LwKPmjsATpiOvlf1s/KsH49l3 x0ky1sLmeCeyQkcafTdo+jxXvPrmDvy1N8bVFBoL2ciXgEatC4VWAo6qJfJ7ozHq
Re: [ZODB-Dev] Persisting Python objects in MongoDB
On Tuesday, November 08, 2011 07:25:34 AM Andreas Jung wrote: Very cool and excellent work. Thanks very much! I appreciate it, especially after your initial reaction on the #mongo IRC channel. :-) Regards, Stephan -- Entrepreneur and Software Geek Google me. Zope Stephan Richter ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persisting Python objects in MongoDB
Hi Stephan, Adam A very nice Idee but it ignores the most important aspects of mongodb. And this is exactly what I was preventing to do with m01.mongo. I still think it's just wrong to implement multi-object transactions with a mongodb backend. And this is what mongopersist is based on. m01.mongo was developed with the focus to offer everything you need for atomic object transactions and offers all you need for implement rollback concepts at an application level. In my point of view an application using the mongodb must be implemented in parts or called services. Each of them must handle it's own part in atomic transactions and this could make an application stable. A persistent layer using multi-object transactions can't do that. Here are some short examples what we do with m01.mongo. We allways store or change one item in one transaction. Everything else is done in background tasks. e.g. - indexing - remove references (based on items data) - remove items (we just mark them as removed in a transaction) - update references (based on items data) - archive items - create invoice items (based on items data) This means if one item changes and other things needs to be done, do it explicit, rollback or do what your apps need to do in another transaction. Rolling back a master transaction isn't possible if the database doesn't support this. That's the responsibility of your app. Everything else could and in my experience with mongodb it will fail. But the good thing is, a concept like described above could also get implemented with mongopersist. But then, you need to catch much more use case based on the DBRef in mongopersist then with m01.mongo. Anyway I highly recommend to anybody only using mongopersist and of corse also m01.mongo if you know how to implement rollback and catching other sideeffects in our app by your own. Or you will quickly see what this could mean not having a multi-object transaction based database by our own. Regards Roger Ineichen -Ursprüngliche Nachricht- Von: zodb-dev-bounces+dev=projekt01...@zope.org [mailto:zodb-dev- bounces+dev=projekt01...@zope.org] Im Auftrag von Stephan Richter Gesendet: Dienstag, 8. November 2011 19:50 An: li...@zopyx.com Cc: zodb-dev@zope.org Betreff: Re: [ZODB-Dev] Persisting Python objects in MongoDB On Tuesday, November 08, 2011 07:25:34 AM Andreas Jung wrote: Very cool and excellent work. Thanks very much! I appreciate it, especially after your initial reaction on the #mongo IRC channel. :-) Regards, Stephan -- Entrepreneur and Software Geek Google me. Zope Stephan Richter ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persisting Python objects in MongoDB
On Wednesday, November 09, 2011 05:41:33 AM Roger wrote: I still think it's just wrong to implement multi-object transactions with a mongodb backend. And this is what mongopersist is based on. I have not made any claim on how to use the code. You can call transaction.commit() after each object modification, if you like. Also, I assume that anyone interested in mongopersist is aware that MongoDB does not provide transaction safety. That said, I think that a large class of applications (or large parts of an app) does not need transaction safety. In our application, for example, most object modifications are compartmentalized, so that modifying small sets of objects is pretty safe. The largest risk is posed by competing modifications, which I tried to address using optional write-conflict detection. Regards, Stephan -- Entrepreneur and Software Geek Google me. Zope Stephan Richter ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persisting Python objects in MongoDB
Hi Stephan
Betreff: Re: AW: [ZODB-Dev] Persisting Python objects in MongoDB
On Wednesday, November 09, 2011 05:41:33 AM Roger wrote:
I still think it's just wrong to implement multi-object transactions
with a mongodb backend. And this is what mongopersist is based on.
I have not made any claim on how to use the code. You can call
transaction.commit() after each object modification, if you like. Also, I
assume that anyone interested in mongopersist is aware that MongoDB does
not
provide transaction safety.
That said, I think that a large class of applications (or large parts of
an
app) does not need transaction safety. In our application, for example,
most
object modifications are compartmentalized, so that modifying small sets
of
objects is pretty safe. The largest risk is posed by competing
modifications,
which I tried to address using optional write-conflict detection.
This means we now have a persistent layer for mongodb like we have
with ZODB but we should not use it the same way as we whould with
the ZODB in our applications.
There are many use cases if you develope with mongodb where you
have to use an own application logic and m01.mongo and mongopersist
does not fit at any time. e.g. update the same properties for 2 or more
items at the same time. This could be done with an atomic mongodb
transaction and whould be done with 2 or more independent collection
calls with m01.mongo or mongopersist.
like:
collection.update({color: 'dark'}, {color: 'black'})
There are many other usecases where anything else then
using an explicit pymongo connection is just a wrong.
All I can say is, that it's wrong to use persistence for hide the
mongodb from the developers. Then mongodb is very different
then a ZODB. The big question for me is, how simple is it to optimize
the implementation if you need to do so based on the used framework.
Regards
Roger Ineichen
Regards,
Stephan
--
Entrepreneur and Software Geek
Google me. Zope Stephan Richter
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
