Re: [ZODB-Dev] merging databases containing multidatabase references
On Tue, Apr 28, 2009 at 14:27, Jim Fulton wrote: > > On Apr 28, 2009, at 1:06 AM, Andreas Jung wrote: > > On 27.04.2009 15:13 Uhr, Jim Fulton wrote: >> >>> Note that ZODB 3.9 is pretty stable. We're using it in production. It >>> is alpha mainly because Shane and I are still sorting out integration >>> of RelStorage. >>> >> >> Speaking of ZODB 3.9 - when do you expect going beta (I know it's done >> when it is done) since ZODB is a dependency of Zope 2.12 and I want to >> beta with Zope 2.12 at the time ZODB 3.0 enters beta. >> > > > I merged Shane's branch to support RelStorage yesterday. I have one other > small feature to check in and then I'll be ready for beta. I may make the > first beta release today. If not, then almost certainly this week. > Great news! Thanks, Andreas ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On Apr 28, 2009, at 1:06 AM, Andreas Jung wrote: > On 27.04.2009 15:13 Uhr, Jim Fulton wrote: >> Note that ZODB 3.9 is pretty stable. We're using it in production. It >> is alpha mainly because Shane and I are still sorting out integration >> of RelStorage. > > Speaking of ZODB 3.9 - when do you expect going beta (I know it's done > when it is done) since ZODB is a dependency of Zope 2.12 and I want to > beta with Zope 2.12 at the time ZODB 3.0 enters beta. I merged Shane's branch to support RelStorage yesterday. I have one other small feature to check in and then I'll be ready for beta. I may make the first beta release today. If not, then almost certainly this week. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On 27.04.2009 15:13 Uhr, Jim Fulton wrote: > Note that ZODB 3.9 is pretty stable. We're using it in production. It > is alpha mainly because Shane and I are still sorting out integration > of RelStorage. Speaking of ZODB 3.9 - when do you expect going beta (I know it's done when it is done) since ZODB is a dependency of Zope 2.12 and I want to beta with Zope 2.12 at the time ZODB 3.0 enters beta. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
David Glick wrote: > Taking the existing ExportImport.py code as a starting point, it looks > like I would need to: > 1. modify the persistent_load method to handle rewriting extended > references of types 'n' and 'm' > 2. provide some way to process several export files at once, and input > a mapping of filename to database name (to handle when I am, say, > importing 2 databases with references between each other) On second thought... if it were me, I think I'd write a special export function, rather than import function. The export function would cross database boundaries as needed. It would rewrite every OID and OID reference as "db_name:oid". Then you wouldn't need to write an import function. I hope the import/export code doesn't have some buried assumption that OIDs are 8 bytes long, though. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On Apr 27, 2009, at 8:57 AM, Alan Runyan wrote: >> In ZODB 3.9, you can pack without doing garbage collection. This >> should at least buy you time, if not solve your problem (depending on >> how much garbage your application creates. > > What about zc.FileStorage? Does this support packing w/o GC? Yes. > And does that work on versions of ZODB < 3.9? It works with 3.8. Note that if you're using ZEO, you can use ZODB 3.9 on the server and use an older version on the client, so you use Zope versions as far back as 2.7. Note that ZODB 3.9 is pretty stable. We're using it in production. It is alpha mainly because Shane and I are still sorting out integration of RelStorage. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
> In ZODB 3.9, you can pack without doing garbage collection. This > should at least buy you time, if not solve your problem (depending on > how much garbage your application creates. What about zc.FileStorage? Does this support packing w/o GC? And does that work on versions of ZODB < 3.9? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On Apr 24, 2009, at 2:37 AM, Shane Hathaway wrote: > Christian Theune wrote: >> On Thu, 2009-04-23 at 15:02 -0700, David Glick wrote: >>> Having now realized the folly of trying to split things into >>> multiple databases, I would really like to merge these 3 >>> databases back into 1. Any ideas for how I could go about this? >>> Based on >>> http://docs.zope.org/zope3/Code/ZODB/cross-database-references.txt/index.html >>> >>>and some experimentation, multi-database references are not >>> supported by the import/export code. Do I have other options? >> I think it should be possible to write a tool that merges one >> database >> into another. This needs some processing on the database objects as >> their OIDs (and thus the internal references) need to be reorganized. > > The import machinery in ZODB/ExportImport.py provides a good example > of how to rewrite OIDs. However, I am concerned that understanding > that code requires a lot of experience with ZODB. > > David, if you want to tackle writing a tool that merges multi- > databases by rewriting OIDs, it probably won't be easy, but I think > you can count on help from this list. I also think the tool would > be useful for others. Taking the existing ExportImport.py code as a starting point, it looks like I would need to: 1. modify the persistent_load method to handle rewriting extended references of types 'n' and 'm' 2. provide some way to process several export files at once, and input a mapping of filename to database name (to handle when I am, say, importing 2 databases with references between each other) Does this sound right? I'm not in any rush but I may tackle this if I find time. It looks like OFS has a custom importer/exporter for XML that I may be able to use as a modelhmm, though on closer inspection this only rewrites the XML format into the standard format and then uses the same _importDuringCommit implementation on the Connection; it looks like I probably can't provide an alternate _importDuringCommit without monkeypatching. thanks all for the advice so far... David Glick Web Developer ONE/Northwest New tools and strategies for engaging people in protecting the environment http://www.onenw.org davidgl...@onenw.org work: (206) 286-1235 x32 mobile: (206) 679-3833 Subscribe to ONEList, our email newsletter! Practical advice for effective online engagement http://www.onenw.org/full_signup ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On Apr 23, 2009, at 6:02 PM, David Glick wrote: > I have a multidatabase of 3 databases containing a handful of > multidatabase references. (This was created out of a combination of > naivety along with a desire to be able to work with smaller files if I > need to pull down a local copy of the database from the server.) Of > course, now I am running into trouble because I cannot pack one of the > databases without occasionally having an object get garbage collected > that is still referenced from another database, which leads to > POSKeyErrors when I try to load the object containing that reference. In ZODB 3.9, you can pack without doing garbage collection. This should at least buy you time, if not solve your problem (depending on how much garbage your application creates. In the (near?) future, we'll be providing a multi-database garbage collector that works independently of packing. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
Christian Theune wrote: > On Thu, 2009-04-23 at 15:02 -0700, David Glick wrote: >> Having now realized the folly of trying to split things into multiple >> databases, I would really like to merge these 3 databases back into >> 1. Any ideas for how I could go about this? Based on >> http://docs.zope.org/zope3/Code/ZODB/cross-database-references.txt/index.html >> >> and some experimentation, multi-database references are not >> supported by the import/export code. Do I have other options? > > I think it should be possible to write a tool that merges one database > into another. This needs some processing on the database objects as > their OIDs (and thus the internal references) need to be reorganized. The import machinery in ZODB/ExportImport.py provides a good example of how to rewrite OIDs. However, I am concerned that understanding that code requires a lot of experience with ZODB. David, if you want to tackle writing a tool that merges multi-databases by rewriting OIDs, it probably won't be easy, but I think you can count on help from this list. I also think the tool would be useful for others. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On Thu, 2009-04-23 at 15:02 -0700, David Glick wrote: > I have a multidatabase of 3 databases containing a handful of > multidatabase references. (This was created out of a combination of > naivety along with a desire to be able to work with smaller files if I > need to pull down a local copy of the database from the server.) Of > course, now I am running into trouble because I cannot pack one of the > databases without occasionally having an object get garbage collected > that is still referenced from another database, which leads to > POSKeyErrors when I try to load the object containing that reference. > > Having now realized the folly of trying to split things into multiple > databases, I would really like to merge these 3 databases back into > 1. Any ideas for how I could go about this? Based on > http://docs.zope.org/zope3/Code/ZODB/cross-database-references.txt/index.html > and some experimentation, multi-database references are not > supported by the import/export code. Do I have other options? I think it should be possible to write a tool that merges one database into another. This needs some processing on the database objects as their OIDs (and thus the internal references) need to be reorganized. Christian -- Christian Theune · c...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] merging databases containing multidatabase references
I have a multidatabase of 3 databases containing a handful of multidatabase references. (This was created out of a combination of naivety along with a desire to be able to work with smaller files if I need to pull down a local copy of the database from the server.) Of course, now I am running into trouble because I cannot pack one of the databases without occasionally having an object get garbage collected that is still referenced from another database, which leads to POSKeyErrors when I try to load the object containing that reference. Having now realized the folly of trying to split things into multiple databases, I would really like to merge these 3 databases back into 1. Any ideas for how I could go about this? Based on http://docs.zope.org/zope3/Code/ZODB/cross-database-references.txt/index.html and some experimentation, multi-database references are not supported by the import/export code. Do I have other options? thanks, David Glick Web Developer ONE/Northwest New tools and strategies for engaging people in protecting the environment http://www.onenw.org davidgl...@onenw.org work: (206) 286-1235 x32 mobile: (206) 679-3833 Subscribe to ONEList, our email newsletter! Practical advice for effective online engagement http://www.onenw.org/full_signup ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
