Re: [ZODB-Dev] PyTables Storage
Hi guys. repoze has package is useful for solving the transaction management issue. http://svn.repoze.org/repoze.tm2 Regards, David On Nov 1, 2008, at 4:39 AM, Andreas Jung wrote: > On 31.10.2008 22:17 Uhr, Christophe Combelles wrote: >> Tim Cook a écrit : >>> Does anyone know of any work being done on a PyTables >>> http://www.pytables.org storage? >> >> No, but I might be interested in such a thing, since I've created >> last year a >> medical data analysis software on which I encountered such memory >> issues, while >> looping over large structures in zodb. >> I have looked at pytables a few times, but I've never tried it >> yet. (I had also >> looked at SOOM (a component of NetEpi Analysis), which allow to >> compute stats on >> very large datasets.) >> Is it worth creating a new storage implementation for such cases? >> Don't just we >> need to implement (or use an existing) specific structure in >> filestorage? >> >> Christophe >> >>> I see a future need in my project to be able to manage real time >>> storage >>> of medical data and the author of PyTables lists the memory usage >>> of the >>> ZODB as being a drawback in these situations. > > The problem with all such implementations is: they don't support > transactions which is a major requirement. I also looked at 10gen > lately (something similar like GAE but open-source) and they also > provide some nice storage (of course "the cloud") but also lack > transaction integration. > > Andreas___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Amazon SimpleDB Adapter
Hi guys, I had thought of something along these lines a while back for s3 and discovered laurence had already started something - but it was too slow which I can understand. Can't really see things being much better with simpledb and I think cost would not be great when you start thinking of paying for all the pickles. I think a better way of interacting with databases these days is to use models in a direct way rather than bend the storage to the zodb api. Martijn and Laurence have been doing this with megrok.rdb, megrok.rdf for illustration. The work to tie transactions together with zope is really the important stuff. There is no lack of storage options out there these days. My opinion is that if you want the true characteristics of the storage, you won't make it behave like a zodb. Lovely systems came to a similar conclusion after trying relstorage and then moving to use storm in a more direct way. Regards, David On Oct 12, 2008, at 12:50 AM, Shane Hathaway wrote: > Laurence Rowe wrote: >> I'm not sure RelStorage is the best place for it - SimpleDB is very >> different to relational databases. > > RelStorage doesn't use much of a relational database either (except > during packing). > >> A couple of years ago I experimented with s3storage [1]. This >> turned out >> to be very slow due to the number of writes performed every >> transaction >> - one per object, though this could be improved if the writes were >> parallelized. It reached the point where zope2 would start up. >> This took >> about 10 or 15 mintutes at the time (I did not have access to EC2 >> at the >> time and this was over public wifi). >> >> It worked by creating it's own indexes in S3. I don't think SimpleDB >> will give any advantage unless it is shown to be faster to query than >> S3. You cannot store pickles directly in SimpleDB because it is >> limited >> to an attribute size of 1024 bytes. >> >> The challenge in building such a system is in Amazon's eventual >> consistency model means you cannot know how up to date your view >> of the >> data is. I think it could make a great backend for storing pickles >> (keyed by oid, tid) but it is probably much easier to have a separate >> index to consult during loadSerial. > > Thanks for the background on S3 and SimpleDB. Using Amazon's storage > services as a ZODB backend is sounding ever more like an interesting > challenge. > > Shane > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.8.1b2 available and persistent caches
Hi Jim. Any chance of incorporating Shane's patch for relstorage for this release? Many thanks. Regards, David Jim Fulton wrote: Lately, we've started to use persistent caches. As we've done so, we've found and fixed a number of problems. I'm planning to make a 3.8.1 release soon. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and BLOBs
Hi Christian. Curious, what were you using for postgres replication? Many thanks. David Christian Theune wrote: I'm kind of curious what a match of RelStorage and ZEORaid may give. FWIW my experiences with Postgresql replication aren't that good so you could still go that route using two RelStorages with one (or more) ZEORaids in front. Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Zeo Server as a single point of failure
Hi Shane. Not sure if you have been following the whole thread or not. In any case, I had committed to maintaining DirectoryStorage a little while ago and it will be releasing in the near future under a new license. I think it is a good project too and I am excited about bringing it up to date :-). Once I have finished with the packaging, a bit of clean up and a first commit done of the updated source to sourceforge, I am hoping you may be interested in participating in ts development and co-maintaining it with me. If so, I can provide you with access once the new code has landed in the repository. I am happy that there is genuine interest for this storage though it is not widely deployed. I have a continued interest in RelStorage (formerly PGStorage). Though I cannot commit to svn, I'd be happy to help with a buildout configuration that could apply the ZODB patch also. Had you though about putting into the z3c namespace. Many thanks. Regards, David Shane Hathaway wrote: Tres Seaver wrote: What does Shane have to do with DirectoryStorage? For the record: I am not at all associated with DirectoryStorage, other than I think it's a good project. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Zeo Server as a single point of failure
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David Pratt wrote: Hi Alan. I'm keeping the thread on the list since you raised the issue here. There are several repositories of zope related code that exist outside of svn.zope.org. I have made the decision to host DirectoryStorage at sourceforge on the basis that I have committed to maintaining and contributing to the project. I volunteered to maintain the project since I saw it falling into disrepair following two years without a single commit to the code base. I have become the maintainer on the basis of an agreement with Tobi back in January. This agreement permits me to relicense the code under my choice of a ZPL or MIT license. I cannot work with sources in zope repository since it would require a contributor agreement with Zope Corp. I am unable to enter into this agreement for the forseeable future. The zif collective (zif.sourceforge.net) is a way that I may contribute back to the zope community without the requirement of the agreement. You don't identify the problem(s) which prevent you from entering into that agreement: we might be able to clarify or help resolve them if you did so. Hi Tres. Any discussion of the ZCA will have to occur another day. I am not seeking a way to enter the agreement or to resolve anything publicly. Toby and I made these arrangements a couple months back and have committed to ongoing communication. It should be pointed out that users of the code will be getting releases from a package index in future. More important than the location of the repository is the fact the code will be eggified, maintained and there will be future releases. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Zeo Server as a single point of failure
Hi Martijn. Probably not at this point. Unfortunately, the assignment of IP and contribution to open source are not synonymous. Perhaps when the relationship between ZC and ZF is clearer it could be reconsidered in future. Regards, David Martijn Faassen wrote: Hey, David Pratt wrote: [snip] I cannot work with sources in zope repository since it would require a contributor agreement with Zope Corp. I am unable to enter into this agreement for the forseeable future. The zif collective (zif.sourceforge.net) is a way that I may contribute back to the zope community without the requirement of the agreement. Just out of interest, would a contributor agreement with the Zope Foundation be any better for you? Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo Server as a single point of failure
Hi Alan. I'm keeping the thread on the list since you raised the issue here. There are several repositories of zope related code that exist outside of svn.zope.org. I have made the decision to host DirectoryStorage at sourceforge on the basis that I have committed to maintaining and contributing to the project. I volunteered to maintain the project since I saw it falling into disrepair following two years without a single commit to the code base. I have become the maintainer on the basis of an agreement with Tobi back in January. This agreement permits me to relicense the code under my choice of a ZPL or MIT license. I cannot work with sources in zope repository since it would require a contributor agreement with Zope Corp. I am unable to enter into this agreement for the forseeable future. The zif collective (zif.sourceforge.net) is a way that I may contribute back to the zope community without the requirement of the agreement. Anyone interested in DirectoryStorage is encouraged to participate in its development since it is an open project that will be getting a more liberal license. There will be something more to build upon once the egg repackaging is released. It will be released under zif namespaced package. Shane and I have a relationship since I was also a project admin for PGStorage when it was located on sourceforge. There is nothing that would prohibit our communication or collaboration on any future development of DirectoryStorage. HTH. Regards, David Alan Runyan wrote: > i understand. I wanted to see if I could rile the zodb community > into saying "we want to maintain it". as in the 'collective' we. > > I respect your rights. But if I could get Shane to say "make it ZPL > and I will maintain it" -- it would be well worth while to see if > I could make this happen > > alan > > On Tue, Mar 11, 2008 at 8:16 AM, David Pratt <[EMAIL PROTECTED]> wrote: >> Hi Alan. I am working on cleanup and will releasing a version with new >> licensing at some point in near future. I am working at this with bits >> of time I have between other work. Unfortunately, it had been left for a >> couple of years without attention and my interest is in keeping it up to >> date with ZODB development. We discussed this about month or so ago when >> I communicated I would be maintaining it at sourceforge for the reasons >> I provided at that time. >> >> Regards, >> David >> >> >> >> >> Alan Runyan wrote: >> >> in that vein, DirectoryStorage-behind-ZEO has worked perfectly for >> >> years... but this new stuff is very interesting! >> > >> > wishing we could get DirectoryStorage into svn.zope.org >> > >> > ~alan >> >> >>> ___ >> > For more information about ZODB, see the ZODB Wiki: >> > http://www.zope.org/Wikis/ZODB/ >> > >> > ZODB-Dev mailing list - ZODB-Dev@zope.org >> > http://mail.zope.org/mailman/listinfo/zodb-dev >> > >> > > > ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo Server as a single point of failure
Hi Alan. I am working on cleanup and will releasing a version with new licensing at some point in near future. I am working at this with bits of time I have between other work. Unfortunately, it had been left for a couple of years without attention and my interest is in keeping it up to date with ZODB development. We discussed this about month or so ago when I communicated I would be maintaining it at sourceforge for the reasons I provided at that time. Regards, David Alan Runyan wrote: in that vein, DirectoryStorage-behind-ZEO has worked perfectly for years... but this new stuff is very interesting! wishing we could get DirectoryStorage into svn.zope.org ~alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Hi Benji. Have you any settings to recommend or use a default. Many thanks. Regards, David Benji York wrote: Roché Compaan wrote: On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: Maybe if you set up a ZODB cache that allows just over 10 million objects, the lookup time will drop to microseconds. You might need a lot of RAM to do that, though. Maybe, but somehow I think that disk IO will prevent this. I'll check. If you're on Linux, you can tweak swappiness (/proc/sys/vm/swappiness; http://lwn.net/Articles/83588/) to affect how much RAM is used for the page cache and how much for your process. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Hi Roche. I figured this out once and it was included in PGStorage so it should be in relstorage also. Take a look at get_db_size method in postgres adapter. relstorage is in the zope repository. Regards, David Roché Compaan wrote: - How much disk space does each database consume when there are 10M objects? ZODB: 19GB How do you check the size of a Postgres database? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL
Ok, great. Woo hoo - 64 bit 2.5. Many thanks Shane. Regards, David Shane Hathaway wrote: David Pratt wrote: Hi Shane. Congratulations. Can you advise of compatibility with ZODB 3.8. Many thanks. Yes, it's fully compatible now. A few tests didn't pass at first, but they only failed due to the lack of some stub methods. After I added those methods and created a patch that applies cleanly to ZODB 3.8.0, all the tests passed (with 64 bit Python 2.5, in fact). Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL
Hi Shane. Congratulations. Can you advise of compatibility with ZODB 3.8. Many thanks. Regards, David Stephan Richter wrote: On Wednesday 30 January 2008, Shane Hathaway wrote: Kudos to the PostgreSQL team for building such a nice database. :-) I am saying this since at least 8 years now. :-) But everyone went on the MySQL bandwagon. But the storage is great news! I will certainly keep this in mind for the current project. Regards, Stephan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Hi Andreas. No need to rely on rumors. The current source is available here: http://pgstorage.cvs.sourceforge.net/pgstorage/ Regards, David Andreas Jung wrote: --On 24. Januar 2008 15:42:01 +0100 Andreas Jung <[EMAIL PROTECTED]> wrote: --On 23. Januar 2008 18:17:18 +0100 Andreas Jung <[EMAIL PROTECTED]> wrote: --On 22. Januar 2008 21:17:45 -0500 Stephan Richter <[EMAIL PROTECTED]> wrote: On Tuesday 22 January 2008, Dieter Maurer wrote: "OracleStorage" was abandoned because it was almost an order or magnitude slower than "FileStorage". Actually, Lovely Systems uses PGStorage because it is faster for them. It would be interesting where PGS is faster than filestorage. Ok, I just tried made a simple benchmark for testing write performance. I created a Plone site and then created a copy using copy/paste within the ZMI. (AMD Dualcore 2.6 GHz, Postgres 7.4.7 running on dedicated DB server): Copy&Paste using Filestorage: 3-4 seconds Copy&Paste using PGStorage: 30-40 seconds Alan asked me to so some further testing. I wrote a small script creating 100 Documents within one transaction (both in CMF and Plone) and I calculated the transaction time (not included the overhead for creating the instances)...the numbers are pretty much the self-speaking: CMFPlone PGStorage10-12 secs 15-18 secs Filestorage 0.3-0.4 secs 0.4-0.6 secs I must mention that all tests were done using PGStorage 0.1 - the only version available to me. Rumors say that Shane might release improved versions. So the numbers must be taken with care as they reflect the state of PGStorage as of 2006. Andreas ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Graphical test comparison (open office format) http://pgstorage.cvs.sourceforge.net/pgstorage/PGStorage/tests/comparison.ods Regards, David Alan Runyan wrote: Andreas, Could you try to mount the catalog separately? My understanding is the catalog is what makes storages a misery. CMF/portal_catalog mounted as Filestorage and CMF could be mounted as PGStorage. I presume you would see much more reasonable performance? cheers ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Cool. Jim Fulton wrote: Berkeley DB storage didnt' work out the first time around. My experience optimizing packing for FileStorage reminded me how much I want an alternative to FileStorage for large active databases. I intend to revisit Berkeley DB storage one of these days. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Yes, Shane had done some benchmarking about a year or so ago. PGStorage was actually faster with small writes but slower for larger ones. As far as packing, as a zodb implementation, packing is still required to reduce the size of data in Postgres. BTW Stephan, where is Lovely using it - a site example? I had read some time ago that they were exploring it but not that it was being used. Regards, David Stephan Richter wrote: On Tuesday 22 January 2008, Dieter Maurer wrote: "OracleStorage" was abandoned because it was almost an order or magnitude slower than "FileStorage". Actually, Lovely Systems uses PGStorage because it is faster for them. Regards, Stephan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] some interesting benchmarks
Cool! Thanks Martijn. Regards, David Martijn Faassen wrote: Hi there, Recently there were two blog entries which did some simple benchmarks of the ZODB and compared to other databases. Possibly others hadn't noticed those yet, so here are the references: ZODB vs Relational Database: a simple benchmark http://pyinsci.blogspot.com/2007/09/zodb-vs-relational-database-simple.html ZODB vs Durus http://pyinsci.blogspot.com/2007/09/zodb-vs-durus.html Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zc.recipe.zeo
Hi Jim. My apologies. I'll move this thread to the other list. Many
thanks for your reply and help.
Regards,
David
Jim Fulton wrote:
On Aug 3, 2007, at 10:00 AM, David Pratt wrote:
Hi Fred. Working with the new recipe but there does not seem to be a
way to use the filestorage recipe together with it. I would like to do
something like this
[zeoscripts]
recipe = zc.recipe.egg:script
eggs = ZODB3
[database]
recipe = zc.recipe.filestorage
[server]
recipe = zc.zodbrecipes:server
database = database
zeo.conf =
address 8105
monitor-address 8106
transaction-timeout 300
to keep the database contained within the buildout and to produce a
zeo.conf like
address 8105
monitor-address 8106
transaction-timeout 300
path /my/buildout/path/database/Data.fs
I'll take a look at the recipe since it should be possible to utilize
some of what is in zc.recipe.zeo which has this behaviour when
database is defined as an option, otherwise it would pass to do what
it is currently doing. I use other backend storages also so would like
to have it operate automatic with filestorage recipe or manually
configured. Many thanks.
Use:
path ${database:path}
Jim
--
Jim Fultonmailto:[EMAIL PROTECTED]Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporationhttp://www.zope.comhttp://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zc.recipe.zeo
Hi Fred. Working with the new recipe but there does not seem to be a way to use the filestorage recipe together with it. I would like to do something like this [zeoscripts] recipe = zc.recipe.egg:script eggs = ZODB3 [database] recipe = zc.recipe.filestorage [server] recipe = zc.zodbrecipes:server database = database zeo.conf = address 8105 monitor-address 8106 transaction-timeout 300 to keep the database contained within the buildout and to produce a zeo.conf like address 8105 monitor-address 8106 transaction-timeout 300 path /my/buildout/path/database/Data.fs I'll take a look at the recipe since it should be possible to utilize some of what is in zc.recipe.zeo which has this behaviour when database is defined as an option, otherwise it would pass to do what it is currently doing. I use other backend storages also so would like to have it operate automatic with filestorage recipe or manually configured. Many thanks. Regards, David Fred Drake wrote: On 8/3/07, David Pratt <[EMAIL PROTECTED]> wrote: Hi there. zc.recipe.zeo has not yet made it to the cheeseshop. It would be great if someone could put it there. If you want me to do it that is fine since I have other packages on cheeseshop. The port for zeo on is also hardcoded for 8100 but I am wanting to run more than a single ZEO server instance per physical server so it looks like it needs a fix. I've been using zc.zodbrecipes:server, including address 9000 in the value for the zeo.conf key (replacing 9000 with your chosen port). Question about thread setting in buildout. In zope2's zope.conf you just set the no of threads. In zc.zope3recipes, there is no mention of setting threads. I ask this since setting threads to 1 improved performance according to the recent high performance zope session by Lovely systems at europython. Many thanks. You can include the line "threads 1" as part of the value for zope.conf in the instance part. -Fred ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zc.recipe.zeo
Hi Fred. The zc.recipe.zeo is a bit old and I seemed to have missed the newer zeo server recipe in svn which handles imports also. Thank you also for the info on threads. Regards, David Fred Drake wrote: On 8/3/07, David Pratt <[EMAIL PROTECTED]> wrote: Hi there. zc.recipe.zeo has not yet made it to the cheeseshop. It would be great if someone could put it there. If you want me to do it that is fine since I have other packages on cheeseshop. The port for zeo on is also hardcoded for 8100 but I am wanting to run more than a single ZEO server instance per physical server so it looks like it needs a fix. I've been using zc.zodbrecipes:server, including address 9000 in the value for the zeo.conf key (replacing 9000 with your chosen port). Question about thread setting in buildout. In zope2's zope.conf you just set the no of threads. In zc.zope3recipes, there is no mention of setting threads. I ask this since setting threads to 1 improved performance according to the recent high performance zope session by Lovely systems at europython. Many thanks. You can include the line "threads 1" as part of the value for zope.conf in the instance part. -Fred ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zc.recipe.zeo
Hi there. zc.recipe.zeo has not yet made it to the cheeseshop. It would be great if someone could put it there. If you want me to do it that is fine since I have other packages on cheeseshop. The port for zeo on is also hardcoded for 8100 but I am wanting to run more than a single ZEO server instance per physical server so it looks like it needs a fix. Question about thread setting in buildout. In zope2's zope.conf you just set the no of threads. In zc.zope3recipes, there is no mention of setting threads. I ask this since setting threads to 1 improved performance according to the recent high performance zope session by Lovely systems at europython. Many thanks. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO client poll times
Hi Jim. I understood that you are opening connection. Looks like I misunderstood what will happen when max is reached. :-) So after reaching the max it will not disconnect - just attempt to reconnect at the max interval. I guess if this is the case, I don't see an issue with the shorter interval between retries. Worst case, it will probably just mean some extra lines in the log file before it connects (if it can). Regards, David Jim Fulton wrote: On May 11, 2007, at 4:39 PM, David Pratt wrote: 6 seconds as max time sounds a bit short to me. I think there can be latency sometimes for whatever reason over the Internet. I would not want it necessarily closing connections as a result. Maybe max of 30 - 60 sec might be better - and still not too long to wait. If folks use the configuration, it adjustable on the other hand. The timeout I'm talking about isn't for closing connections, it's for opening them. It's the length of time to wait before trying again after being disconnected or after failing a connection attempt. Jim -- Jim Fultonmailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporationhttp://www.zope.comhttp://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO client poll times
Hi Jim. Twisted has a reconnecting client with a similar feature. It starts at a short interval also but backs off at increasingly longer intervals which I think is the way to go when you are unable to reach the other side (but when you are obviously running a server). 6 seconds as max time sounds a bit short to me. I think there can be latency sometimes for whatever reason over the Internet. I would not want it necessarily closing connections as a result. Maybe max of 30 - 60 sec might be better - and still not too long to wait. If folks use the configuration, it adjustable on the other hand. Many thanks. Regards, David Jim Fulton wrote: When a ZEO Client is disconnected from a ZEO server, it will try to reconnect and keep trying until it reconnects or is closed. It waits between attempts. The waiting period starts at 5 seconds and increases to 5 minutes by default. These times are configurable using the min-disconnect-poll and max-disconnect-poll ZConfig options or with the min_disconnect_poll and max_disconnect_poll constructor arguments. Since most people don't change these defaults, most people's clients take too long to reconnect. I'd like to change these defaults to something much smaller. I'm thinking 1 and 6 seconds. Anybody have any objections to this change? Jim -- Jim Fultonmailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporationhttp://www.zope.comhttp://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
Bottom line is that storage methods need to be executed on both sides. I don't think the storages need to care what or how objects are transported nor what version of zope you are using so long as in the end, the object is one that can be used by a storage method - and that the storage methods are compatible with the older zopes (which seems to be what you are concerned about). At least this is how I am looking at it - whether it is zope 2 or 3, it should not matter if they are methods that can act on ZODB and are suited to the zodb version. In any case, thats the direction I have been going. On top of it, who wants a transport that does only one thing, that can get tired pretty fast. There are many more possibilities than exchanging zodb data using a more generic transport. That's the other key reason I tossed out my attempt to replicate ZEO directly using twisted - if you want to think a bit beyond ZEO. Regards, David Jim Fulton wrote: On Apr 28, 2007, at 9:13 AM, David Pratt wrote: ... As far as compatibility, I am not sure the past always needs to equal the future. It is probably the methods for zrpc that are important, not the transport - at least that is my thinking. I can use current ZEO servers with ZODB 3.2 clients running in very old versions of Zope (2.6 and 2.7). This is extremely useful to me at this point. I couldn't do this if I changed the wire protocol. Jim -- Jim Fultonmailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporationhttp://www.zope.comhttp://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
Jim Fulton wrote: On Apr 27, 2007, at 9:30 PM, David Pratt wrote: Hi Jim. I have been quite interested in this also so have experimented and continue on this also. The way I am doing the networking it similar to ngi I have put this together before you defined these interfaces. I'm not sure which "this" you're referring to. I assume you mean using Twisted for ZEO. I don't see that that has much to do with the new storage interfaces. Yes, Twisted for ZEO. No, I was referring to the interfaces you were setting up - the interfaces to ngi. In what way is your networking similar to ngi? The interfaces are similar. The system I am putting together is pretty symmetrical and I like the language which is more consistent with twisted's use of factories. You were using this language before in your initial proposal to ngi but many of the interfaces are now name differently named. See how nice and symmetrical this looks :-) clientconnfactoryhandler clientconnhandler clientconnfactory clientreconnfactory clientmethods clientstorage connection serverconnfactory serverconnhandler serverconnfactoryhandler servermethods serverstorage Note that I wrote ngi based on a misunderstanding of Twisted. :( I didn't realize that Twisted application code didn't touch sockets and therefore could be tested without using sockets, threads or subprocesses. IMO, this is Twisted's most important feature. I recently finished my first Twisted application and I was very pleased with my ability to create sane tests for it, although it's test support infrastructure could use improvement, which I plan to do. I have got a basic transport to use prospective broker and banana protocol. So a client and server but I have not yet put together the methods for interacting with the storage. I plan to stick with the existing ZEO protocol, both for compatibility and for performance reasons. Banana protocol is efficient and extensible. It has its own producer/consumer so no need to roll you own system as you have for ZEO. I think you will find banana an efficient serialization of objects. As far as compatibility, I am not sure the past always needs to equal the future. It is probably the methods for zrpc that are important, not the transport - at least that is my thinking. This is the obvious downside of ZEO at the moment is that is too tied to asyncore. In fact, I started by trying to replicate much of what was in ZEO - then I threw it away because something simpler I believe is possible. If a new system can transport the objects and execute the methods, isn't that all it is supposed to do? On top of it, what if something better comes around - the methods that you are executing on both sides are the important part and the transport should be transparent. I think this could produce a much cleaner and understandable system. On top of it, it doesn't upset anything with current zeo for users that currently use this. It could in fact be developed and run in parallel, not that you would want to run it that way :-). Another obvious advantage is that twisted project has tests for most of this already so that is also a big plus. Out of curiosity, does perspective broker support one-way calls, messages sent without replies? Scanning the docs, I can't tell. Of course, not waiting for replies is inherent in Twisted's defered model I suppose. Yes To set up something to work with I create both a TCP and SSL clientfactory so I bring them into the multiservice when zope is started. This way you do not need a separate loop to run the service which is the current way zeo runs. At the same time this does not interfere with the current zope setup. I have no proof one way or the other that this will not work as efficiently on a single twisted loop with the second asyncore loop. I still have to hook up database methods and I had put this on a backburner for a while since I have had other things to do. In any case, the client setup in zope requires only minor modification of the zope.app.twisted main.py since the script only deals with servers and not clients at the moment. The clients are reconnecting clients so will continue to try reconnecting if connection is lost in gradually increasing time intervals. On the server side I have the server working using zconfig and zdaemon in a similar way to way it is setup for zope itself. I have been looking at the zope3recipes since I am working on getting this into a buildout to continue since I am quite happy with the way buildouts work. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
Hi Jim. I have been quite interested in this also so have experimented and continue on this also. The way I am doing the networking it similar to ngi I have put this together before you defined these interfaces. I have got a basic transport to use prospective broker and banana protocol. So a client and server but I have not yet put together the methods for interacting with the storage. Just opening it at the moment and executing a couple of methods on the objects on the other side. Regards, David Jim Fulton wrote: BTW, an upcoming project of mine will be to change ZEO to use Twisted. I suppose that will make it easier to support encrypted ZEO connections. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Cleaning up storage interfaces
Hi. I am going to be doing an egg release for pgstorage at some point in the next couple of weeks. The current repository is: http://pgstorage.cvs.sourceforge.net/pgstorage/PGStorage/ getSerial is not used in the source. Many thanks. Regards, David Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Theune wrote: Am Dienstag, den 24.04.2007, 13:16 -0400 schrieb Jim Fulton: I'm finally trying to clean up the storage interfaces. I have a question. I'll probably have more later, but I'm going to deal with them one by one rather than trying to save them up. Does anyone know if getSerial is used for anything? I suspect that it is an old name for getTid and would like to remove it in favor of getTid. In particular, it's plumbed through the storage and client servers, but I can't see any evidence that people are using it. Hmm. Looks like it. getTid looks like it's used from ZEO in the verify() method. I didn't find any place where getSerial was used. Zope 3 uses neither (and it probably shouldn't anyway.) DirectoryStorage[1] is probably the major "out-of-tree" storage implementation to check; PGStorage[2] is likel a distant second in terms of usage: [1] http://dirstorage.sourceforge.net/ [2] http://hathawaymix.org/Software/PGStorage/ Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGLtS7+gerLs4ltQ4RAvn7AJ98Wgm7D/EbA3YS/A/gbJ5bKjbR5QCgtuyv tX4is0ClS1zf1Pa6Oc2RwkE= =I2S0 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.8 and Blobs
Many thanks to the team for this awesome effort. Regards David Christian Theune wrote: Hi, it's been about 21 months since work on blobs for ZODB started. I finally sat down today and fixed the last broken test for blobs on windows and merged the branch to the trunk. Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Experimental ZODB 3.7 release
Hi Christian. I have been anticipating the merge of blobs in the trunk for a while so also hoping this happens for this release. Many thanks. Regards, David Christian Theune wrote: Hi, from the project management perspective: Does that mean we're on a feature freeze for 3.7? I still have to deal with one bug on windows (which is likely just a problem in the test itself that fails, not in the blob code) until I can merge into the trunk and I'd be very happy to commit this soon. Christian Jim Fulton wrote: I've released a fairly experimental 3.7 release, http://www.python.org/pypi/ZODB3/3.7.0b3. Unlike older ZODB releases, this release only installs the packages that are part of ZODB. It doesn't include ZConfig zdaemon, zope.testing, zope.proxy and zope.interface. These are now dependencies that need to be installed separately. If this release is installed with easy_install or zc.buildout then the dependencies should be installed automatically. I haven't modified mkzeoinst to reflect the fact that the dependencies are no-longer included in the distribution. As a consequence, the instances built by mkzeoinst will only work if the dependencies are on the python path (e.g. installed into site-packages.) I will need to modify mkzeiinst to deal with this. Anyway, if people want to play with it, feedback is welcome. I'm currently using the release in a project in which I install the release with zc.buildout. Jim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB
Hi Christian. Yay :-) Looking forwards to it. Many thanks. Regards, David Christian Theune wrote: David Pratt wrote: Can you advise whether blobs will make it the trunk any time soon. I see an extfile package now for z3 but would rather see filesystem storage dealt with at the backend than in the app itself. Many thanks. Chris McDonough and I have this on our schedule. It's not much to do and I hope to get my todo list done during the next week. After that we might be able to merge into trunk during September. Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB
Can you advise whether blobs will make it the trunk any time soon. I see an extfile package now for z3 but would rather see filesystem storage dealt with at the backend than in the app itself. Many thanks. Regards, David Gary Poster wrote: On Apr 29, 2006, at 6:08 AM, Christian Theune wrote: Hi everybody, the ZODB team is proud to announce a technical preview of a Blob implementation for ZODB. Hey. This seems like there's no way that it will make it into the next ZODB release, right? I assume no one is using it in production, on the basis of the branch TODO. Does anyone have a hard need to get this production-ready (address the TODOs) by a certain date? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Getting clientstorage object in main.py
Hi Pascal. Yes, this is exactly what I was after. Many thanks for your help! Regards David Pascal Peregrina wrote: I think that through the DB instance, you can access the storage instance with the "_storage" attribute. Pascal -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de David Pratt Envoyé : jeudi 15 juin 2006 19:29 À : zodb-dev@zope.org Objet : Re: [ZODB-Dev] Getting clientstorage object in main.py Perhaps I should qualify this a bit. On realize ClientStorage is initialized during ZEO startup. main.py provides a ZODB.DB.DB instance as db but I am interested in setting an attribute of the ClientStorage instance that was initialized. Regards, David David Pratt wrote: Hi. I am needing to get hold of my clientstorage obj instance that was created on realize in zope.app.twisted.main.py. Can someone advise how to get a handle on it in main.py after is has it has been created (realized). Many thanks. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Getting clientstorage object in main.py
Perhaps I should qualify this a bit. On realize ClientStorage is initialized during ZEO startup. main.py provides a ZODB.DB.DB instance as db but I am interested in setting an attribute of the ClientStorage instance that was initialized. Regards, David David Pratt wrote: Hi. I am needing to get hold of my clientstorage obj instance that was created on realize in zope.app.twisted.main.py. Can someone advise how to get a handle on it in main.py after is has it has been created (realized). Many thanks. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Getting clientstorage object in main.py
Hi. I am needing to get hold of my clientstorage obj instance that was created on realize in zope.app.twisted.main.py. Can someone advise how to get a handle on it in main.py after is has it has been created (realized). Many thanks. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO Authentication
Well, you might need a few monkeys for sure :-) I guess it is safe to say, that the authentication built into ZEO is not used extensively. You never know until you ask. I guess if it were, it might depend on the use case you have for ZEO more than anything. Many thanks. Regards David You can use a VLAN, too. Or a monkey to control each ZEO packet :-) -aj ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO Authentication
Hi Jens, this is available in ZEO and looks to me that the original idea was to support more than a single mode of authentication. I was curious of those using it to see how it was generally being used (or whether others have worked out something against other authentication schemes). Many thanks. Regards, David Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26 May 2006, at 05:46, David Pratt wrote: I am curious how folks are handling authentication for ZEO in a general way (client against the server). Is is common to attempt to use user/passwords from acl in zope as a means of creating a lists for authenticating against a ZEO server? Or is it more typical to manage a separate list of user/passwords just for the purpose of authenticating the client server connection? I'd like to hear what sort of things are being done to handle this generally. Many thanks. I don't know anyone who uses authenticated ZEO connections, does it even work? IMHO most people are in a situation where the traffic between the ZEO clients and the ZEO server runs on an internal network, so it simply doesn't matter. Or small setups where the communication is on one and the same machine. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEdseGRAx5nvEhZLIRApfTAJ9MjasVk9UHp1yvlBP2BNPQl6GXKACeIpIC Yx5XIHtQzPX9+xQzbL71zh0= =G8RQ -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZEO Authentication
I am curious how folks are handling authentication for ZEO in a general way (client against the server). Is is common to attempt to use user/passwords from acl in zope as a means of creating a lists for authenticating against a ZEO server? Or is it more typical to manage a separate list of user/passwords just for the purpose of authenticating the client server connection? I'd like to hear what sort of things are being done to handle this generally. Many thanks. Regards David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB
Hi Gary. I have been experimenting with it an posed the question to Andreas a couple of days ago about when it might get into the trunk. I am using the modified ZODB and ZEO files to accomodate it in the interim but would be happy if it was in the trunk for sure. When is 3.7 final to be tagged? Will this now be for 3.8? Regards, David Gary Poster wrote: On Apr 29, 2006, at 6:08 AM, Christian Theune wrote: Hi everybody, the ZODB team is proud to announce a technical preview of a Blob implementation for ZODB. Hey. This seems like there's no way that it will make it into the next ZODB release, right? I assume no one is using it in production, on the basis of the branch TODO. Does anyone have a hard need to get this production-ready (address the TODOs) by a certain date? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Undo differences between Z2 and Z3
Hi Jim. I managed to fix this by modifying the storage but it seems to me strange to differentiate in a storage whether it is a backend to a z2 or z3 app. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Undo differences between Z2 and Z3
Jim Fulton wrote:
David Pratt wrote:
There seems to be enough of a difference between Z2 and Z3 undo to
have an effect on a backend like PGStorage from working on Z3. I
believe the objective in the storage api is that it should not matter
to the backend whether you are connected to Z2 or Z3. Can someone more
familiar with the history of undo in z2 and z3 explain the current
situation?
Undo has a fairly complicated API, and I can easily believe that Z2 and
Z3 call that API in very different ways.
BTW, I strongly discourage use of Undo except in emergencies.
Sadly, except when undoing the last (non-undo) transactions in
a database can lead to inconsistency. Undo should be used with
caution and should generally not be exposed to non-expert users.
Undo has been a part of CMF and Plone for non-expert users for quite
some time by exposing only the last non-undo transaction. This should be
fine for Z3 I believe. In any case, my interest is initially to give Z3
what it is asking for without crapping out so contents view will work
for a principal through z3 zmi.
In Z3, before bringing up the contents view, there is a check of the
undo transactions for the principal. It uses z3's undo manager to
getPrincipalTransactions which eventually results in a lookup in the
undoLog in the external storage where it dies. Just a brief look at
the undo code in z3 gives me the feeling the format
Format of what?
I am referring to is this which is the route this is taking to the undoLog :
def _getUndoInfo(self, context, principal, first, last):
specification = {}
if context is not None:
locatable = IPhysicallyLocatable(context, None)
if locatable is not None:
location = Prefix(locatable.getPath())
specification.update({'location': location})
if principal is not None:
# TODO: The 'user' in the transactions is a concatenation
# of 'path' and 'user' (id). 'path' used to be the path
# of the user folder in Zope 2. ZopePublication currently
# does not set a path, so ZODB lets the path default to
# '/'. We should change ZODB3 to set no default path at
# some point
path = '/' # default for now
specification.update({'user_name': path + ' ' + principal.id})
entries = self.__db.undoInfo(first, last, specification)
It's possible that there is a bug in PGStorage in handling
extension data. I can easily believe that Z2 doesn't use extension
data and perhaps z3 does.
Thanks. I have reported this to Shane but am also experimenting with
storages so have an interest in understanding what is happening.
Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Undo differences between Z2 and Z3
There seems to be enough of a difference between Z2 and Z3 undo to have an effect on a backend like PGStorage from working on Z3. I believe the objective in the storage api is that it should not matter to the backend whether you are connected to Z2 or Z3. Can someone more familiar with the history of undo in z2 and z3 explain the current situation? In Z3, before bringing up the contents view, there is a check of the undo transactions for the principal. It uses z3's undo manager to getPrincipalTransactions which eventually results in a lookup in the undoLog in the external storage where it dies. Just a brief look at the undo code in z3 gives me the feeling the format between z2 and z3 may be incompatible which may be the problem. The relevant portion of the traceback I receive is this: File "/usr/local/zope3/z3trunk/src/zope/tales/expressions.py", line 199, in _eval return ob() File "/usr/local/zope3/z3trunk/src/zope/app/publisher/browser/viewmeta.py", line 419, in __call__ return meth(*a, **k) File "/usr/local/zope3/z3trunk/src/zope/app/undo/browser.py", line 33, in principalLastTransactionIsUndo last=1) File "/usr/local/zope3/z3trunk/src/zope/app/undo/__init__.py", line 108, in getPrincipalTransactions return self._getUndoInfo(context, principal, first, last) File "/usr/local/zope3/z3trunk/src/zope/app/undo/__init__.py", line 129, in _getUndoInfo entries = self.__db.undoInfo(first, last, specification) File "/usr/local/zope3/z3trunk/src/ZODB/UndoLogCompatible.py", line 37, in undoInfo return self.undoLog(first, last, filter) File "/usr/local/zope3/z3trunk/src/ZODB/PGStorage/pgstorage.py", line 478, in undoLog d.update(cPickle.loads(ext)) EOFError Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Really want to refactor ZEO's networking archiecture
Hi Jim and Roderigo! This is encouraging news. I have been putting much thought into this also. There is a downside to refactoring ZEO in that there are many folks heavily dependent on it and also reasonably happy with it. Secondly, zeo has a specific place and relationship. When I initiated the twisted zeo discussion about a week ago, it was with the hope of drawing interest to some possible changes. I was thinking about security, my own positive experiences with twisted, and how things could be modified to bring tighter integration. That said, the general experience of twisted and zope has been short (since 3.2) and this is a departure that has risks that I respect. I was encouraged to challenge my own thinking on what may be needed. Before committing, I am planning to build and test a few ideas to explore this further without confining the scope. I don't know where this is leading me just yet. On the notion of multiple independent loops for twisted, I believe there are possibilities, some interested folks, and a patch I found that could be evaluated. The work would be murky as Glyph has pointed out - but he has offered to review tests for this as well. Many thanks, Regards, David Jim Fulton wrote: I think that my next big project might be to refactor ZEO's networking architecture. Wonder what the main reason is? - Provide more secure connections? Nope - Leverage Twisted? Nope - Get rid if the insane async/sync client madness? Nope - Get better storage-server performance? Nope - Allow simultaneous syncronous calls from the same client? Nope I'd like to do all of these things, but the main reason to refactor ZEO's networking architecture is to make it testable. Writing ZEO tests now requires actually starting servers and clients. This is nuts. IMO, application code shouldn't touch sockets. Application code, like ZODB and ZEO should be insulated from actual network APIs. They use simpler APIs that are easy to interccept and control. I've been thinking of a someone general networking API with these aims in mind, but I realized that the ZEO existing frameworks could probably be refactored in a more limited way to do this. We'll see. If anyone is interested in workingon this with me, let me know. Jim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Hi Rodrigo! Great post on this. You provide an insightful and compelling summary of what can be accomplished. Add on top of this, the things that twisted "just does" like SSL or SSH out of the box and we can also have the security issue beat. Jim, what do you think? Do you think you could give us a place to experiment in zope's svn? Regards, David Rodrigo Dias Arruda Senra wrote: You probably already know all of these, but just in case ... [ Jim Fulton ]: --- | | I am strongly against linking ZEO to an application's main loop. Twisted performs better if the protocols build upon it rely on its reactor (asynchronous loop). That means breaking long computations to use promises (deferreds in Twisted vocabulary), therefore increasing the interleaving of data handlers. Implementing ZEO in Twisted would probably isolate the zeo protocol from the underlying transport (tcp, udp, whatever). OTOH, the implementation would be dependent on the typical deferred patterns (only common in Twisted nowadays). Twisted also support threads, but that defeats most of the other benefits of using Twisted in the first place (like not bothering about critical sections and other synch stuff), so they are just used when blocking calls must be made. | If twisted supports using multiple independent main loops, then it | would be an option. I know it supports multiple implementations of reactors, each specialized for a particular purpose, for instance to conciliate two different loops into a single reactor, which is the case of twisted_networking_loop + GUI_event_loop. Therefore, it would be possible (my educated guess) to create a zeoreactor, would that suffice ? That was done to marry Twisted to: gtk, gtk2, glib2, wxPython, win32event loops. | Then the possible issues are: | | - Whether we want Twisted to be a dependency of ZEO If it comes to that, probably just TwistedCore would be required, and perhaps even a subset. | - Performance There is a lot of this-and-that [1] about what communication pattern performs better. Twisted is definetely better for long sessions, and not necessarily much worse for short-termed sessions (where "people" say thread-pools are much sexier). IMVHO, the strong argument behind Twisted adoption would be flexibility and not huge speedups. [ David Pratt ]: - | > Is there is any strong opposition to using the twistd daemon for a | > twisted zeo service? I volunteer to help in that effort to the best of my modest skills and limited (just like everybody else) spare time. [1] http://www.cs.wustl.edu/~schmidt/patterns-ace.html cheers, Senra - Rodrigo Senra GPr Sistemas ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Hi Jim. My experience has been in running a single selectreactor in an app together with its own main loop. I found a patch on google to allow multiple selectreactors but have not seen the patch applied to svn or attempted the change in twisted as yet. I have emailed the maintainer to ask about it. I was looking at the zeo server service first which should be happy to run on its own. There is no risk at this stage to experiment with my ideas other than learning more than I did before about twisted and ZEO. Regards, David Jim Fulton wrote: David Pratt wrote: ... Is there is any strong opposition to using the twistd daemon for a twisted zeo service? I am strongly against linking ZEO to an application's main loop. If twisted supports using multiple independent main loops, then it would be an option. Then the possible issues are: - Whether we want Twisted to be a dependency of ZEO - Performance Switching to twisted would be a big change. If I was to make such a change, which I anticipate, it would be great if twised could be an option. Jim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: ZEO and Twisted
Florent Guillaume wrote: Huh, I thought you were talking about the ZEO client, ClientStorage, not the ZEO server. For the ZEO server I don't see the point of changing it, it works well. OTOH a ClientStorage has to integrate with the other servers in Zope, and that's the one that would benefit from being moved to the twisted event loop if twisted is used. Hi Florent. Sorry I was meaning the whole not just asyncore and twisted integration for instances. I think re-implementing the zrpc protocol for ZEO server would clean things up quite a bit and make the security issue easier to deal with. Then we are talking apples and probably less code as well since twisted handles a fair amount and its already in the zope distribution. Reading posts about about securing zeo etc from as far back as six years ago makes me wonder why not? Twisted uses zope interfaces, has all the right stuff and makes it easier on top of it. I am no networking guru for sure, but at the end of the day, zrpc is a simple custom protocol that should be pretty doable in twisted without building custom add ons to secure network communication. Is there is any strong opposition to using the twistd daemon for a twisted zeo service? Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO Twisted and Z3
Good news. Sorry to reply to own post. I have found the following interfaces: twisted.internet.interfaces.IReactorSSL.listenSSL twisted.internet.interfaces.IReactorSSL.connectSSL this looks promising. I believe authentication is passed in the protocol so maybe there is not that much to worry about there. So maybe this is an issue of overriding asnychronous core with twisted. Regards, David David Pratt wrote: Is there a plan to provide security for ZEO client / server communication using Twisted now that it is in Zope3? I only have not read all source of ZEO but wish to use it with encryption and authentication and Twisted is an excellent toolbox for network communication. Anyone done this or have any recommendations or advice. Many thanks David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZEO Twisted and Z3
Is there a plan to provide security for ZEO client / server communication using Twisted now that it is in Zope3? I only have not read all source of ZEO but wish to use it with encryption and authentication and Twisted is an excellent toolbox for network communication. Anyone done this or have any recommendations or advice. Many thanks David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
