Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
Dieter Maurer wrote: Why do you think so? Almost everything is run off SANs. Why should "Data.fs" make a problem? Under high load, an awful lot of data is read from this file. SANs have higher latency than local disks. In some apps this can make a noticeable performance difference. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
Patrick Gerken wrote: You run your live Data.fs off a SAN? That usually makes for interesting performance problems in the best case! Well, yes, that is/was the idea when suddenly HA requests popped up. You might be right, Painful experience has taught me that I am ;-) need to access many many objects we use an SQL index, How strange... and nobody will access too many objects. Er, yeah... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
Chris Withers wrote at 2006-9-25 18:28 +0100: >Patrick Gerken wrote: >> system with ZEO for an ERP5 deployment. In my case I don't need to >> care for data replication, all is stored on a SAN considered HA by the >> customer already. > >You run your live Data.fs off a SAN? >That usually makes for interesting performance problems in the best case! Why do you think so? Almost everything is run off SANs. Why should "Data.fs" make a problem? -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
On 9/25/06, Chris Withers <[EMAIL PROTECTED]> wrote: Patrick Gerken wrote: > system with ZEO for an ERP5 deployment. In my case I don't need to > care for data replication, all is stored on a SAN considered HA by the > customer already. You run your live Data.fs off a SAN? That usually makes for interesting performance problems in the best case! Well, yes, that is/was the idea when suddenly HA requests popped up. You might be right, but I would still give it a try, whenever we would need to access many many objects we use an SQL index, and nobody will access too many objects. Most of them will probably lay around and never touched again. Though we didn't start with profiling yet, I feel pretty safe to say that the typical number of users for one zeo client will not try to access 1000 different objects. > So my data.fs and index and all that stuff will already be available > on my backup server. How does it measure "down"? More important is the question what we will do if we think it is down. Through I do not think about details currently. > The thing which scared me to hell was the "rumour" that it can happen > that the index can get corrupted and then everything has to be indexed > again. without ZRS, don't even try to have a "live" backup. Best you can aim for is a half-hour swapover if you have a decent sized (10-20GB) database. With ZRS, it may be less, but I remain to be convinced. > Given that writing a file and renaming a file can be considered > atomic, and that no solar winds or similar things can screw up my > filesystem, how can I screw up my index file? The index checking is likely pessemistic, as you've noticed. Optimisations, I'm sure, would be welcome, but you'd need to prove they can't result in dangerous corruption... Lets see what can be done. Thanks for your remarks! best regards, Patrick ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
Patrick Gerken wrote: system with ZEO for an ERP5 deployment. In my case I don't need to care for data replication, all is stored on a SAN considered HA by the customer already. You run your live Data.fs off a SAN? That usually makes for interesting performance problems in the best case! So my data.fs and index and all that stuff will already be available on my backup server. ...but better be very very careful that both the backup and live server dont' try and use the .fs and index files at the same time... The idea is that the backup server will watch the real ZEO, and start all services if the server goes down. How does it measure "down"? The thing which scared me to hell was the "rumour" that it can happen that the index can get corrupted and then everything has to be indexed again. without ZRS, don't even try to have a "live" backup. Best you can aim for is a half-hour swapover if you have a decent sized (10-20GB) database. With ZRS, it may be less, but I remain to be convinced. Given that writing a file and renaming a file can be considered atomic, and that no solar winds or similar things can screw up my filesystem, how can I screw up my index file? The index checking is likely pessemistic, as you've noticed. Optimisations, I'm sure, would be welcome, but you'd need to prove they can't result in dangerous corruption... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
Patrick Gerken wrote at 2006-9-22 10:17 +0200: > ... >Even the first checkin of the filestorage makes a >self.read_index(...start = start...) >which updates the index from the oldest tid known to the index. >I can only imagine that the index gets invalidated if the >self._check_sanity() returns 0 thus saying the index is not valid. But >from my reading through the websvn this should not be the case in the >scenarios I can imagine (journaled filesystem, no solar winds). My experience is that "_check_sanity" returns false whenever there is a single modification after the index file was written and then the index is not used at all. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
On Sep 22, 2006, at 2:48 AM, Christian Theune wrote: ... One of the last things I remember is that indexes can not be rebuilt partially but are rebuilt completely. If the existing index is valid, then only new data are read if it isn't current. I think this is what Patrick was referring to. If the old index is determined not to be valid, then of course, the index has to be rebuild from scratch. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
On 9/22/06, Christian Theune <[EMAIL PROTECTED]> wrote: Hi, Patrick Gerken wrote: > Hello, > > its funny, like Garth in may this year I am looking into making a HA > system with ZEO for an ERP5 deployment. In my case I don't need to > care for data replication, all is stored on a SAN considered HA by the > customer already. > > So my data.fs and index and all that stuff will already be available > on my backup server. > The idea is that the backup server will watch the real ZEO, and start > all services if the server goes down. It seems quite safe for me to > take over the files from the filesystem and restart everything (While > writing I realise I should be really really sure the other zeo is > down, but well, that is out of scope for this mail). > > The thing which scared me to hell was the "rumour" that it can happen > that the index can get corrupted and then everything has to be indexed > again. With a large number of objects, this will be slow, either > because of millions of seeks or many many blocks written to memory. So > I wanted to look into the FileStorage implementation to see if this > could be optimised. But I did not find a single place where killing > the server would result in a corrupt index file (Simply assuming that > we have journaling filesystems). Also if the index file is not up to > date, restarting the zeo server would result in updating the zeo > server starting from the latest transaction written to the index. That > can take up to 5 minutes (assuming 1 objects, the distance between > writes, multiplied with 20 ms for processing and seeking, rounded up > to the next prime. > > Given that writing a file and renaming a file can be considered > atomic, and that no solar winds or similar things can screw up my > filesystem, how can I screw up my index file? One of the last things I remember is that indexes can not be rebuilt partially but are rebuilt completely. I had a request on my plate to modify the index code to take old indexes into account as well. I'm not sure if anybody else already did that (but that would be available only on trunk anyway) Even the first checkin of the filestorage makes a self.read_index(...start = start...) which updates the index from the oldest tid known to the index. I can only imagine that the index gets invalidated if the self._check_sanity() returns 0 thus saying the index is not valid. But from my reading through the websvn this should not be the case in the scenarios I can imagine (journaled filesystem, no solar winds). Is there a way to reproduce this behaviour easily? best regards, Patrick Gerken ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage
Hi, Patrick Gerken wrote: > Hello, > > its funny, like Garth in may this year I am looking into making a HA > system with ZEO for an ERP5 deployment. In my case I don't need to > care for data replication, all is stored on a SAN considered HA by the > customer already. > > So my data.fs and index and all that stuff will already be available > on my backup server. > The idea is that the backup server will watch the real ZEO, and start > all services if the server goes down. It seems quite safe for me to > take over the files from the filesystem and restart everything (While > writing I realise I should be really really sure the other zeo is > down, but well, that is out of scope for this mail). > > The thing which scared me to hell was the "rumour" that it can happen > that the index can get corrupted and then everything has to be indexed > again. With a large number of objects, this will be slow, either > because of millions of seeks or many many blocks written to memory. So > I wanted to look into the FileStorage implementation to see if this > could be optimised. But I did not find a single place where killing > the server would result in a corrupt index file (Simply assuming that > we have journaling filesystems). Also if the index file is not up to > date, restarting the zeo server would result in updating the zeo > server starting from the latest transaction written to the index. That > can take up to 5 minutes (assuming 1 objects, the distance between > writes, multiplied with 20 ms for processing and seeking, rounded up > to the next prime. > > Given that writing a file and renaming a file can be considered > atomic, and that no solar winds or similar things can screw up my > filesystem, how can I screw up my index file? One of the last things I remember is that indexes can not be rebuilt partially but are rebuilt completely. I had a request on my plate to modify the index code to take old indexes into account as well. I'm not sure if anybody else already did that (but that would be available only on trunk anyway) Christian -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: OpenPGP digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
