Re: [ZODB-Dev] Using zodb and blobs
On 04/14/2010 03:30 AM, Nitro wrote: Am 14.04.2010, 04:39 Uhr, schrieb Tim Peterstim.pet...@gmail.com: [Nitro] ... I wonder if _commit is really *that* slow Six years ago I timed factor-of-100 speed differences due to using MS _commit() on WinXP at the time: https://mail.zope.org/pipermail/zodb-dev/2004-July/007720.html or if there's another (faster) function which can be called... No MS function that does the same thing. Seems like ZODB has a long history of discussions on this matter: http://www.mail-archive.com/zodb-dev@zope.org/msg01874.html There's even a proposal for improvement in that thread, also on the wiki: http://wiki.zope.org/ZODB/FsyncBehaviourSetting What I don't really get is why you should never use None on windows. As far as I can judge from the various transaction rates in the thread Tim mentioned, fsync is just a no-op on linux anyways (depending on the specific file system of course). I'm pretty sure it's not. IIRC fsync is defined by POSIX and absolutely requires the implementor to flush data physically to disk ensuring its persistency. If that doesn't hold true then all transactions are borked. I've seen virtualised environments like VMWare ESX lie about fsyncs from a virtual hardware perspective. A similar issue with breaking fsync was the discussion around ext4 on notebooks with a only actually flush discs every 10 seconds. In the end it really depends on what you need your data for. If I'd store the information that a customer paid me 50 EUR for something and I presented a screen to him that told him he'll receive some good for that then I'd rather stick with compliant transactions. I am almost tempted to do os.fsync = lambda fd: 0 and rely on yesterday's backup. 0.49 j/k. j/k? -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
On 4/14/10 08:24 , Christian Theune wrote: I'm pretty sure it's not. IIRC fsync is defined by POSIX and absolutely requires the implementor to flush data physically to disk ensuring its persistency. If that doesn't hold true then all transactions are borked. That was the problem with fsync on Linux: it effectively flushed all pending filesystem work, not just that for your current filehandle. That was needed to satisfy ordering constraints for the filesystem. And even though the result might be a lie since disks or other bits of hardwire can lie to you. It is generally better to use fdatasync() instead of fsync(), but you could still end up waiting much longer than you would expect. Wichert. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
Am 14.04.2010, 09:24 Uhr, schrieb Christian Theune c...@gocept.com: What I don't really get is why you should never use None on windows. As far as I can judge from the various transaction rates in the thread Tim mentioned, fsync is just a no-op on linux anyways (depending on the specific file system of course). I'm pretty sure it's not. IIRC fsync is defined by POSIX and absolutely requires the implementor to flush data physically to disk ensuring its persistency. If that doesn't hold true then all transactions are borked. They are. See https://mail.zope.org/pipermail/zodb-dev/2004-July/007683.html . Important quote Linux will allow fsync() to return even if the data isn't actually on disk. Also you can do the math yourself. If you have 10 ms average seek time, then you can only do 100 fsync/s. If you get more, there's buffering of some sort. Be it in hardware or file system. Since you cannot rely on fsync to work here I see the fsync as an hopeful wish, but nothing you can rely on. Things you cannot rely on give a false sense of security even if they might actually help in some cases. That's why I trust Windows' _commit to do it's work much more I trust linux' fsync to do it's job. 50 transaction = 20ms per transaction which sounds reasonable. In the end it really depends on what you need your data for. If I'd store the information that a customer paid me 50 EUR for something and I presented a screen to him that told him he'll receive some good for that then I'd rather stick with compliant transactions. Yes, in my case it's nothing critical or related to money. If there's a hardware outage a day of work is lost at worst. In case of corruption (which can happen also without fsync as data within the file can just be garbled) you need a backup anyways. j/k? just kidding. I don't think the transfer rate is actually that interesting. For small but many transactions the seek time/spinning speed should have the limiting influence. Yes, seek time is important here. I didn't recall the seek times of my hard disk, but wanted to mention it's not a slow hard disk. -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
On Wed, Apr 14, 2010 at 11:52 AM, Nitro ni...@dr-code.org wrote: Yes, in my case it's nothing critical or related to money. If there's a hardware outage a day of work is lost at worst. In case of corruption (which can happen also without fsync as data within the file can just be garbled) you need a backup anyways. Usually you will only loose the last transaction and not a days of work. The Data.fs is an append-only file, with one transaction appended after another. If there's a garbled or incomplete write, you'll typically loose the last transaction. The ZODB is smart enough to detect broken transactions and skip them on restart. I have witnessed one ZEO installation myself, where the physical machine hosting the ZEO server restarted multiple times a day, over a period of months. Nobody noticed for a long time, as the application was accessible all the time and no data had been lost. Obviously this wasn't a very write-intense application. But it still showed me how stable the ZODB really is. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
Am 14.04.2010, 14:45 Uhr, schrieb Hanno Schlichting ha...@hannosch.eu: Usually you will only loose the last transaction and not a days of work. The Data.fs is an append-only file, with one transaction appended after another. If there's a garbled or incomplete write, you'll typically loose the last transaction. The ZODB is smart enough to detect broken transactions and skip them on restart. I have witnessed one ZEO installation myself, where the physical machine hosting the ZEO server restarted multiple times a day, over a period of months. Nobody noticed for a long time, as the application was accessible all the time and no data had been lost. Obviously this wasn't a very write-intense application. But it still showed me how stable the ZODB really is. Yes, I agree with your opinion in general. There's still a chance that broken transactions are written IIUC (see https://mail.zope.org/pipermail/zodb-dev/2004-July/007683.html ): Doesn't this mean that if the system suddenly crashes in the middle of os.fsync, the Data.fs on disk will contain an incomplete transaction, but the transaction status byte would claim that the transaction is complete. Wouldn't that be bad? If that happened, perhaps. The chance exists because fsync does not work as advertised on many systems. On the systems where it seems to work, the slowdown is massive. So I am doubting the usefulness of using fsync in current ZODB at all. As your observations seem to hint, it's probably very unlikely to encounter this problem in practice. And I doubt Tim finally got Jim to pay him for a 48 hour pull-the-plug session :-) That's why I am not going to dig further into this and am satisfied with the current reliability. -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Request for merge: tseaver-lp143158-feature branch
On Mon, Apr 12, 2010 at 6:29 AM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Implements '--kill-old-on-full' command line switch for repozo, which removes older full / incremental backups on successful completion of a full backup. http://svn.zope.org/ZODB/branches/tseaver-lp143158-feature/?rev=106808view=rev Done. Jim P.S. Do I owe you any other ZODB merges? -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
