Re: [ZODB-Dev] Blob directory structure scalability limits
On Fri, Jun 20, 2008 at 07:35:10PM +0200, Christian Theune wrote: > [...] and I'm working on an offline migration script that should be > completed by tomorrow. The offline migration script is completed. I'm currently trying to make the tests run on windows. The code probably does run, but I get lots of errors due to path names in doctests and bogus permission information. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Hi, On Fri, Jun 20, 2008 at 07:55:17PM +0200, Andreas Jung wrote: > > > --On 19. Juni 2008 14:31:47 +0200 Martijn Faassen > <[EMAIL PROTECTED]> wrote: > >> Hi there, >> >> On Thu, Jun 19, 2008 at 1:38 PM, Christian Theune <[EMAIL PROTECTED]> wrote: >> [snip] >>> We propose to keep both implementations around and allow to select which >>> one to use. We would extend the FileSystemHelper to abstract the two >>> strategies. >> >> Could the default be to choose the more scalable one, or would this >> lead to backwards compatibility issues? >> >> > > Good point especially because Zope 2.11 is now inofficially released and > people will start using blobs based on the limited directory layout...so > how would a migration look like when people do encounter the current > limitations? It's 100% backwards compatible. When the old layout is used a warning is issued with a hint to migrate but I continue to fully support the old data structures. When a new directory is started, the new layout becomes the default. Migration comes in the form of a command line script that can convert the old structure to the new one. It requires the database to be offline, but I'm working hard to make it reasonably fast. The script will be available ... later today. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
--On 19. Juni 2008 14:31:47 +0200 Martijn Faassen <[EMAIL PROTECTED]> wrote: Hi there, On Thu, Jun 19, 2008 at 1:38 PM, Christian Theune <[EMAIL PROTECTED]> wrote: [snip] We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. Could the default be to choose the more scalable one, or would this lead to backwards compatibility issues? Good point especially because Zope 2.11 is now inofficially released and people will start using blobs based on the limited directory layout...so how would a migration look like when people do encounter the current limitations? Andreas pgpSPnMGLkDpH.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Me again, On Thu, Jun 19, 2008 at 01:38:38PM +0200, Christian Theune wrote: > Hi, > > We propose to introduce a new mode for the blob storage which breaks the > directory structure into one level per byte of the oid. This would lead to > directories 0x00-0xFF nested in 8 levels. I implemented a `bushy` layout and started calling the old layout `lawn` in reference to the DirectoryStorage terminology. The branch is 'ctheune-bushy-directory`. The layouts are pluggable now, provide 100% compatibility to the current implementation and I'm working on an offline migration script that should be completed by tomorrow. I'd like to get feedback and eventually the allowance to merge this into the trunk, we need this rather urgently so I'd be happy to fix any issues quickly that hinder the merge if they get pointed out. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Hi, On Thu, Jun 19, 2008 at 09:29:13AM -0400, Jim Fulton wrote: >> We propose to introduce a new mode for the blob storage which breaks >> the >> directory structure into one level per byte of the oid. This would >> lead to >> directories 0x00-0xFF nested in 8 levels. > > +1 > > ... > >> We propose to keep both implementations around and allow to select >> which one >> to use. We would extend the FileSystemHelper to abstract the two >> strategies. > > -1 to making this selectable. > >> We would also provide a migration tool that can convert the old format >> to the >> new format. > > Maybe we can support, but deprecate the old layout and provide a > conversion scriptt. Good for me as well. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
On Jun 19, 2008, at 7:38 AM, Christian Theune wrote: Hi, one of our installations hit a scalability limits with the current blob directory structure. The current structure looks like: \blobs\ \ .blob .blob ... \ ... ...\ We hit a limit with a database that contains more than 32k blob objects because ext3 doesn't allow more than 32k entries in a directory. We propose to introduce a new mode for the blob storage which breaks the directory structure into one level per byte of the oid. This would lead to directories 0x00-0xFF nested in 8 levels. +1 ... We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. -1 to making this selectable. We would also provide a migration tool that can convert the old format to the new format. Maybe we can support, but deprecate the old layout and provide a conversion scriptt. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Hi there, On Thu, Jun 19, 2008 at 1:38 PM, Christian Theune <[EMAIL PROTECTED]> wrote: [snip] > We propose to keep both implementations around and allow to select which one > to use. We would extend the FileSystemHelper to abstract the two strategies. Could the default be to choose the more scalable one, or would this lead to backwards compatibility issues? Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
On Thu, Jun 19, 2008 at 08:00:27AM -0400, Benji York wrote: > On Thu, Jun 19, 2008 at 7:38 AM, Christian Theune <[EMAIL PROTECTED]> wrote: > > We propose to introduce a new mode for the blob storage which breaks the > > directory structure into one level per byte of the oid. This would lead to > > directories 0x00-0xFF nested in 8 levels. > > I suppose the directories would be built on-demand, right? Right. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
On Thu, Jun 19, 2008 at 7:38 AM, Christian Theune <[EMAIL PROTECTED]> wrote: > We propose to introduce a new mode for the blob storage which breaks the > directory structure into one level per byte of the oid. This would lead to > directories 0x00-0xFF nested in 8 levels. I suppose the directories would be built on-demand, right? -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Blob directory structure scalability limits
Hi, one of our installations hit a scalability limits with the current blob directory structure. The current structure looks like: \blobs\ \ .blob .blob ... \ ... ...\ We hit a limit with a database that contains more than 32k blob objects because ext3 doesn't allow more than 32k entries in a directory. We propose to introduce a new mode for the blob storage which breaks the directory structure into one level per byte of the oid. This would lead to directories 0x00-0xFF nested in 8 levels. The last directory denotes the blob itself and looks like the current directory: a list of blob files named by the tids they were committed for. We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. We would also provide a migration tool that can convert the old format to the new format. Comments? Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
