Re: [Zope] Re: Packing Data.fs
Jens Vagelpohl wrote: It's also f'ing slow and doesn't scale ;-) Translation: In our experience, large databases with a lot of objects can lead to very slow operations, such as packing and backing up the data. This might be improved by using a file system designed for many small files, but on ext3 the whole thing became a terrible dog. This slowness also appears when doing automated buildouts which create lots of objects. Even simple stuff like just instantiating a Plone portal from the ZMI was noticeably slower. I thought my version was much more understandable ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing Data.fs
Tres Seaver wrote: Mark Gibson wrote: I know zope makes a copy of Data.fs before packing. Is it possible to specify a location other than zope/var to place that copy? Actually, it makes a *new* file when packing, and then unlinks the old file and re-links the copy under its name after finishing the pack successfully. If you are looking to spread the love across multiple partitions, you can symlink the original file into a directory on another drive / partition, and then do the pack there. One the pack completes, you can move or copy the packed version back to the original directory. I recently had to move a fairly large Data.fs onto a different partition, but just copying it would have involved downtime (at least for authoring) that I wanted to avoid. The solution I came up with was to create a var folder on the new partition symlinking back to Data.fs in the original location. Then I switched ZEO to using the var on the new partition and packed the database. After packing the database was on the new partition with no downtime apart from a single ZEO restart. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing Data.fs
Duncan Booth wrote: Tres Seaver wrote: Mark Gibson wrote: I know zope makes a copy of Data.fs before packing. Is it possible to specify a location other than zope/var to place that copy? Actually, it makes a *new* file when packing, and then unlinks the old file and re-links the copy under its name after finishing the pack successfully. If you are looking to spread the love across multiple partitions, you can symlink the original file into a directory on another drive / partition, and then do the pack there. One the pack completes, you can move or copy the packed version back to the original directory. I recently had to move a fairly large Data.fs onto a different partition, but just copying it would have involved downtime (at least for authoring) that I wanted to avoid. The solution I came up with was to create a var folder on the new partition symlinking back to Data.fs in the original location. Then I switched ZEO to using the var on the new partition and packed the database. After packing the database was on the new partition with no downtime apart from a single ZEO restart. Nice. I should note that DirectoryStorage (http://dirstorage.sf.net) has excellent replication support which make things like moving live Zope instances from one partition to another one w/o lots of downtime almost trivial. Philipp ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing Data.fs
Philipp von Weitershausen wrote: I should note that DirectoryStorage (http://dirstorage.sf.net) has excellent replication support which make things like moving live Zope instances from one partition to another one w/o lots of downtime almost trivial. It's also f'ing slow and doesn't scale ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing Data.fs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 30 Aug 2006, at 15:48, Chris Withers wrote: Philipp von Weitershausen wrote: I should note that DirectoryStorage (http://dirstorage.sf.net) has excellent replication support which make things like moving live Zope instances from one partition to another one w/o lots of downtime almost trivial. It's also f'ing slow and doesn't scale ;-) Translation: In our experience, large databases with a lot of objects can lead to very slow operations, such as packing and backing up the data. This might be improved by using a file system designed for many small files, but on ext3 the whole thing became a terrible dog. This slowness also appears when doing automated buildouts which create lots of objects. Even simple stuff like just instantiating a Plone portal from the ZMI was noticeably slower. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFE9ZyQRAx5nvEhZLIRAtDYAKCl83nHzOHCuqpfCAQ7xuY5a1nG0QCgtUG0 fPg4mYLK4ILq9e/yERDRNbs= =A5c2 -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing Data.fs
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Mark Gibson wrote:
I know zope makes a copy of Data.fs before packing. Is it possible to
specify a location other than zope/var to place that copy?
Actually, it makes a *new* file when packing, and then unlinks the old
file and re-links the copy under its name after finishing the pack
successfully. If you are looking to spread the love across multiple
partitions, you can symlink the original file into a directory on
another drive / partition, and then do the pack there. One the pack
completes, you can move or copy the packed version back to the original
directory.
To do the pack in the temporary directory, something like the following
Python script should work::
#!/path/to/your/python
import time
from ZODB.FileStorage import FileStorage
to_pack = FileStorage('/mnt/space/packing/Data.fs')
when = time.time() - (7 * 86400) # one week ago
to_pack.pack(when)
Tres.
- --
===
Tres Seaver +1 202-558-7113 [EMAIL PROTECTED]
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFE9Nal+gerLs4ltQ4RAnIDAJ4uMnZFWxT2QUGIxUX8pYEU3ZAHCgCff2p5
ptA83pek4hISALzRwiX7wIU=
=gz4O
-END PGP SIGNATURE-
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing data.fs programmatically
On 25 Feb 2006, at 04:27, Derrick Hudson wrote: | Then introduce it, it's simple to set up and adds a world of | functionality! I could. I'd have to learn how to work with it :-). The deployments I've worked with are all single uniprocessor machines, thus ZEO wouldn't yield a performance improvement. I should learn ZEO one of these days. Thinking of ZEO just as a performance improvement is doing it a great disservice. Think of it simply in terms of decoupling the appserver from the database server and the flexibility that gives you by being able to access the database server separately. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing data.fs programmatically
Derrick Hudson wrote: Chris, On Tue, Jan 31, 2006 at 01:36:36PM +, Chris Withers wrote: | ZEO Stepper Is ZEO required to use Stepper? Yes, otherwise you'd have to shut down your Zope instance to run Stepper ;-) Does 'zopectl run' require exclusive access to the database? (IOW must the zope server be stopped before zopectl run can be used?) Not if you're using ZEO. I just came across this thread and have a situation where this would be useful, however ZEO is not used for that deployment. Then introduce it, it's simple to set up and adds a world of functionality! Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing data.fs programmatically
On Fri, Feb 24, 2006 at 09:18:35PM +, Chris Withers wrote: | Derrick Hudson wrote: | Chris, | | On Tue, Jan 31, 2006 at 01:36:36PM +, Chris Withers wrote: | | ZEO Stepper | | Is ZEO required to use Stepper? | | Yes, otherwise you'd have to shut down your Zope instance to run Stepper ;-) | | Does 'zopectl run' require exclusive access to the database? (IOW | must the zope server be stopped before zopectl run can be used?) | | Not if you're using ZEO. Thanks Paul and Chris. | I just came across this thread and have a situation where this would | be useful, however ZEO is not used for that deployment. | | Then introduce it, it's simple to set up and adds a world of | functionality! I could. I'd have to learn how to work with it :-). The deployments I've worked with are all single uniprocessor machines, thus ZEO wouldn't yield a performance improvement. I should learn ZEO one of these days. -D -- If you hold to [Jesus'] teaching, you are really [Jesus'] disciples. Then you will know the truth, and the truth will set you free. John 8:31-32 www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] signature.asc Description: Digital signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing data.fs programmatically
Chris, On Tue, Jan 31, 2006 at 01:36:36PM +, Chris Withers wrote: | ZEO Stepper Is ZEO required to use Stepper? | Or if you don't like Stepper, then zopectl run... Does 'zopectl run' require exclusive access to the database? (IOW must the zope server be stopped before zopectl run can be used?) I just came across this thread and have a situation where this would be useful, however ZEO is not used for that deployment. -D -- The fear of the Lord leads to life: Then one rests content, untouched by trouble. Proverbs 19:23 www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] signature.asc Description: Digital signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing data.fs programmatically
On Thu, Feb 23, 2006 at 11:27:59AM -0500, Derrick Hudson wrote: Does 'zopectl run' require exclusive access to the database? (IOW must the zope server be stopped before zopectl run can be used?) Yes, IFF you are not running zeo. That's another useful reason to use zeo :-) I just came across this thread and have a situation where this would be useful, however ZEO is not used for that deployment. Ah, that's a shame. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing data.fs programmatically
Ah-ha, that's another approach I had not thought of! Many thanks! Cheers Jonathan Reinoud van Leeuwen wrote: On Tue, Jan 31, 2006 at 11:38:34AM +, Jonathan Cleaver wrote: I am using the following module to pack my data.fs programmatically before a backup is made. This was largely taken out of the Zopelabs cookbook - thanks! I do it though the webinterface with a wget from the crontab: @daily cd /some_path/zeo_server/var \ cp Data.fs Data.fs.pre_pack_backup \ /usr/local/bin/wget \ 'http://myzopeserver/Control_Panel/Database/manage_pack?days:float=0'\ --http-user=special_user \ --http-passwd='some_password' \ --spider \ cp Data.fs Data.fs.post_pack_backup ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Packing data.fs programmatically
Chris Withers wrote: Reinoud van Leeuwen wrote: I do it though the webinterface with a wget from the crontab: @daily cd /some_path/zeo_server/var \ cp Data.fs Data.fs.pre_pack_backup \ /usr/local/bin/wget \ 'http://myzopeserver/Control_Panel/Database/manage_pack?days:float=0'\ --http-user=special_user \ --http-passwd='some_password' \ --spider \cp Data.fs Data.fs.post_pack_backup Oh, ouch ouch bad fragile pain failure suffering... URL whacking is evil and must be punished... What is the better way then? ZEO and ordinary Python scripts? -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing data.fs programmatically
ZEO Stepper Or if you don't like Stepper, then zopectl run... Chris Max M wrote: Chris Withers wrote: Reinoud van Leeuwen wrote: I do it though the webinterface with a wget from the crontab: @daily cd /some_path/zeo_server/var \ cp Data.fs Data.fs.pre_pack_backup \ /usr/local/bin/wget \ 'http://myzopeserver/Control_Panel/Database/manage_pack?days:float=0'\ --http-user=special_user \ --http-passwd='some_password' \ --spider \cp Data.fs Data.fs.post_pack_backup Oh, ouch ouch bad fragile pain failure suffering... URL whacking is evil and must be punished... What is the better way then? ZEO and ordinary Python scripts? -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing data.fs programmatically
On Tue, Jan 31, 2006 at 01:36:36PM +, Chris Withers wrote: ZEO Stepper Or if you don't like Stepper, then zopectl run... Chris Sounds interesting. Do you have a working example that you can post here? -- __ Nothing is as subjective as reality Reinoud van Leeuwen[EMAIL PROTECTED] http://www.xs4all.nl/~reinoud __ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Packing data.fs programmatically
The product comes with a whole raft of examples for doing things: http://www.simplistix.co.uk/software/zope/stepper base.py and config.py contain the examples, lemme know if you have any problems... Chris Reinoud van Leeuwen wrote: On Tue, Jan 31, 2006 at 01:36:36PM +, Chris Withers wrote: ZEO Stepper Or if you don't like Stepper, then zopectl run... Chris Sounds interesting. Do you have a working example that you can post here? -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
