Re: [Zope] ZEO with Multiple Storages
On 9/14/06, Jonathan [EMAIL PROTECTED] wrote: If you mean having zope commit a single transaction to multiple storages, then 'vanilla' zope won't do it (Zope Corp has a commercial product called ZRS which does this type of thing). No, that's not true. ZRS is used to create hot read-only or stand-by copies of a ZEO server (and all storages within such a server are copied). One ZEO server can deal with multiple storages just fine. Transactions involving multiple storages commit just fine. First off, does this make any sense? Is it worth pursuing? I think so. I would think, but do not know for sure, that transactions only involving one storage should not delay transactions involving other storages. Secondly, I assume that where an object is stored does not limit what other objects it can access in a transaction (e.g. object A could add a property to object B if they were in different storages) because all changes would occur in a client. (this ties into the single transaction committing to multiple storages) You assume correctly. Note that a stock Zope install already runs multiple storages; one for Session data (tempstorage) and one for all other data. Lastly, any thoughts on how we could split up our existing data.fs file? One thought I had was to attempt to import/export data. Either before or after upgrading to 2.9 (probably best after to take advantage of the more robust ZODB), export and import. -- Martijn Pieters ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14 Sep 2006, at 08:54, Martijn Pieters wrote: One ZEO server can deal with multiple storages just fine. Transactions involving multiple storages commit just fine. Caveat: You buy nothing by having a single ZEO process serve several ZODBs when you're trying to decrease commit times. You should run a ZEO process per ZODB you're serving out, that way the writes can be segregated and parallelized. That also makes it easier to move processes/databases around disks/hosts to spread I/O load. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFFCQSiRAx5nvEhZLIRAhihAKC7V0lCz3deSz0vN/0RhvdWmFAj0ACgoguT I8NDYcUbz5t25xniZZ1akfo= =63My -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
- Original Message - From: Martijn Pieters [EMAIL PROTECTED] To: Jonathan [EMAIL PROTECTED] Cc: Brian Brinegar [EMAIL PROTECTED]; zope@zope.org Sent: Thursday, September 14, 2006 2:54 AM Subject: Re: [Zope] ZEO with Multiple Storages On 9/14/06, Jonathan [EMAIL PROTECTED] wrote: If you mean having zope commit a single transaction to multiple storages, then 'vanilla' zope won't do it (Zope Corp has a commercial product called ZRS which does this type of thing). No, that's not true. ZRS is used to create hot read-only or stand-by copies of a ZEO server (and all storages within such a server are copied). One ZEO server can deal with multiple storages just fine. Transactions involving multiple storages commit just fine. Yes and No! Yes: a single zeo server can deal with multiple storages (ie. a storage can contain mount points to other storages) Yes: if a single transaction creates/modifies multiple objects and each of those objects is stored in a different storage (all storages 'served up' by a single zeo server), then the single transaction will commit those objects to their respective storages with no problem. (however, there are little/no performance improvements in this scenario) No: if a single transaction creates/modifies a single object, that single object can not be stored over multiple storages (ie. you cannot achieve RAID 'disk striping' type of performance improvements) However, that being said there may be some improvement, if the application is disk-bound, by giving each storage its own disk subsystem (never tried this myself, but it may buy some small performance improvement). For the cost/effort involved I would stick to having a single zeo server 'serve up' a single storage, and have each zeo server on its own hardware platform (much more scalable, easy to implement). Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14 Sep 2006, at 13:56, Jonathan wrote: Yes: a single zeo server can deal with multiple storages (ie. a storage can contain mount points to other storages) That's misleading wording, a storage knows nothing at all about mounts. The storage server can't do anything except for reading from and writing to ZODB databases. Mounting is a client concept only. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFFCVdQRAx5nvEhZLIRAkACAJwM7Cv0+dmnjgtsGsnYhYb6Vzl8BwCgqO8Y z4WxwoWO/XOrRkW8FwLZx9k= =4Lax -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
- Original Message - From: Jens Vagelpohl [EMAIL PROTECTED] To: [Zope] List Mailing zope@zope.org Sent: Thursday, September 14, 2006 9:21 AM Subject: Re: [Zope] ZEO with Multiple Storages -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14 Sep 2006, at 13:56, Jonathan wrote: Yes: a single zeo server can deal with multiple storages (ie. a storage can contain mount points to other storages) That's misleading wording, a storage knows nothing at all about mounts. The storage server can't do anything except for reading from and writing to ZODB databases. Mounting is a client concept only. P.S. Or is the mount point totally controlled by zope.conf and nothing relating to the 'mount point' is stored in the storage? Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14 Sep 2006, at 15:34, Jonathan wrote: Yes: a single zeo server can deal with multiple storages (ie. a storage can contain mount points to other storages) That's misleading wording, a storage knows nothing at all about mounts. The storage server can't do anything except for reading from and writing to ZODB databases. Mounting is a client concept only. P.S. Or is the mount point totally controlled by zope.conf and nothing relating to the 'mount point' is stored in the storage? You got it. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFFCV48RAx5nvEhZLIRAsLWAKCIYC9D+/QpX6oq3QvHvWSBtMMCGQCfY5fS LLsShBUeXJI+O6FSZRiYqxw= =/czT -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
- Original Message - From: Jens Vagelpohl [EMAIL PROTECTED] To: [Zope] List Mailing zope@zope.org Sent: Thursday, September 14, 2006 9:21 AM Subject: Re: [Zope] ZEO with Multiple Storages -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14 Sep 2006, at 13:56, Jonathan wrote: Yes: a single zeo server can deal with multiple storages (ie. a storage can contain mount points to other storages) That's misleading wording, a storage knows nothing at all about mounts. The storage server can't do anything except for reading from and writing to ZODB databases. Mounting is a client concept only. I didn't say that the storage 'knew' anything about mount points. I simply stated that a storage can contain mount points to other storages (storages contain objects, if one of those objects happens to be a mount point to another storage the storage containing the 'mount point' object doesn't care). So the wording is not misleading (if the storage doesn't contain the mount point object, where is it stored?) Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
Brian Brinegar wrote at 2006-9-13 21:22 -0400: ... partioning large data sets onto different storages ... First off, does this make any sense? I makes sense and we do it successfully with our large editorial system. However, ensure that each storage is self contained data wise: e.g. do not put data in one storage and have them catalogued in a different one. Otherwise, selective backup of individual storages will introduce inconsistencies. Is it worth pursuing? Yes. Secondly, I assume that where an object is stored does not limit what other objects it can access in a transaction (e.g. object A could add a property to object B if they were in different storages) because all changes would occur in a client. (this ties into the single transaction committing to multiple storages) Right. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
Jens Vagelpohl wrote at 2006-9-14 15:50 +0200: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14 Sep 2006, at 15:34, Jonathan wrote: Yes: a single zeo server can deal with multiple storages (ie. a storage can contain mount points to other storages) That's misleading wording, a storage knows nothing at all about mounts. The storage server can't do anything except for reading from and writing to ZODB databases. Mounting is a client concept only. P.S. Or is the mount point totally controlled by zope.conf and nothing relating to the 'mount point' is stored in the storage? You got it. I am not sure about the most recent Zope/ZODB versions. But, up to Zope 2.8/ZODB 3.4, Jonathan was right: Beside configuration, there have been MountPoint objects in the ZODB that handled the mounting magic. While not completely impossible, I would be very surprised if the need for MountPoint objects were dropped in newer Zope versions... -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] ZEO with Multiple Storages
We currently have a ZEO environment with FileStorage (data.fs) over 100 gigs, which continues to grow. Currently we are approaching a major hardware and software upgrade. All new hardware and moving from Zope 2.6 to Zope 2.9. Our server hosts sites for several different schools and departments within our college, what I have considered doing is trying to split up the data.fs into multiple smaller storages, possibly one per school. I suspect if this is possible it could potentially have several benefits. The first being that we would not have to deal with 100 gig files which are very difficult to move around backup/restore, etc. Secondly, we often have a very large number of users updating content at a given time and will run into slowdowns where a transaction is blocked while another commits. I hope that multiple storages could allow for a transaction to commit to one storage at the same time as another, though I am not sure. Along this line can a single transaction commit to multiple storages? First off, does this make any sense? Is it worth pursuing? Secondly, I assume that where an object is stored does not limit what other objects it can access in a transaction (e.g. object A could add a property to object B if they were in different storages) because all changes would occur in a client. (this ties into the single transaction committing to multiple storages) If this does makes sense is there any documentation, recommendations, best practices for how to set something like this up? Lastly, any thoughts on how we could split up our existing data.fs file? One thought I had was to attempt to import/export data. Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network Purdue University ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO with Multiple Storages
- Original Message - From: Brian Brinegar [EMAIL PROTECTED] To: zope@zope.org Sent: Wednesday, September 13, 2006 9:22 PM Subject: [Zope] ZEO with Multiple Storages We currently have a ZEO environment with FileStorage (data.fs) over 100 gigs, which continues to grow. Currently we are approaching a major hardware and software upgrade. All new hardware and moving from Zope 2.6 to Zope 2.9. Our server hosts sites for several different schools and departments within our college, what I have considered doing is trying to split up the data.fs into multiple smaller storages, possibly one per school. I suspect if this is possible it could potentially have several benefits. The first being that we would not have to deal with 100 gig files which are very difficult to move around backup/restore, etc. Secondly, we often have a very large number of users updating content at a given time and will run into slowdowns where a transaction is blocked while another commits. I hope that multiple storages could allow for a transaction to commit to one storage at the same time as another, though I am not sure. Along this line can a single transaction commit to multiple storages? If you mean having zope commit a single transaction to multiple storages, then 'vanilla' zope won't do it (Zope Corp has a commercial product called ZRS which does this type of thing). Alternatively, if your server is disk-bound then you could look into RAID disk striping to try to improve performance. First off, does this make any sense? Is it worth pursuing? Secondly, I assume that where an object is stored does not limit what other objects it can access in a transaction (e.g. object A could add a property to object B if they were in different storages) because all changes would occur in a client. (this ties into the single transaction committing to multiple storages) If this does makes sense is there any documentation, recommendations, best practices for how to set something like this up? Lastly, any thoughts on how we could split up our existing data.fs file? One thought I had was to attempt to import/export data. If you have the time and hardware you could easily set up several zeo servers on different 'back-end' hardware servers (ie. put each zeo server on its on computer) to improve performance, increase reliability/robustness, etc. Your 'use case' in which you have different schools and departments would lend itself to this kind of distribution (put the biggest/heaviest users on their own servers, group together smaller/lighter users on shared servers). You just need to define a mount point (in zope.conf) for each different zeo server (and set up each zeo server of course). If you already have the data for the various schools depts stored in separate folders, it would be easy to export each 'school' folder and then import it into the new configuration. hth Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
