#9316: Unify folder implementations
----------------------------+-----------------------------------------------
Reporter: smcmahon | Owner: witsch
Type: PLIP | Status: assigned
Priority: n/a | Milestone: 4.0
Component: Infrastructure | Resolution:
Keywords: |
----------------------------+-----------------------------------------------
Old description:
> ,,Copied from [http://plone.org/products/plone/roadmap/191/ PLIP #191] in
> the roadmap:,,
>
> = Unify folder implementations =
>
> ''We currently have "Folder" and "Large Plone Folder" implementations.
> [[br]]
> There should be only one.''
>
> Proposed by::
> Martin Aspeli
> Seconded by::
> Andreas Zeidler
> Proposal type::
> Architecture
> Repository branch::
> [browser:plone.folder], [browser:plone.app.folder]
>
> == Motivation ==
>
> Shipping with two folder types is unnecessary for several reasons:
> - It forces the user to make an a-priori choice about the number of
> objects they plan to put into a folder
> - We ship with the "Large" type disabled by default to avoid UI
> confusion
> - We don't have a proper search-based UI for large folders anyway
>
> Also the standard Folder type stores attributes, and has a single list
> _objects tuple which keeps the list of objects and order. This is prone
> to ConflictErrors and is slower. In simple benchmarks, a BTree-based
> folder performs orders of magnitude faster than a basic folder.
>
> == Proposal ==
>
> Have a single folder implementation.
> - The internal storage is BTrees
> - It still supports ordering, by storing a separate sort order list/tree
> - It has at least two views - one search-based for "large" folders, one
> batch-based for "small" folders. This is either just a "display" menu
> choice, or a choice in the object's schema. Explicit sorting may be
> turned off for "large" folders.
>
> == Implementation ==
>
> The package `plone.folder` in the Plone SVN provides a base
> implementation of a folder base class, which is not Archetypes specific,
> based on BTreeFolder2, but adding ordering. The exact ordering
> implementation is left up to an adapter, with a default providing
> explicit ordering. This allows other implementations, such as auto-
> sorting based on some key.
>
> The diagram below shows the folderish base- and mix-in classes used in
> OFS, CMF, Archetypes and Plone. Count 'em:
>
> [[Image(http://plone.org/products/plone/roadmap/191/Folder%20mess.png)]]
>
> The new base class from `plone.folder`, i.e. `OrderedBTreeFolderBase`, is
> used by `plone.app.folder` to provide two folderish classes, one targeted
> to `Archetypes` (`BaseBTreeFolder`) as well as one for `ATContentTypes`
> (`ATFolder`). Both add relatively little extra code and setup in order
> to make them fully compatible with the original, to be replaced classes.
>
> The package also provides a `GenericSetup` profile replacing the standard
> "Folder" content type with the new one. In-place migration will convert
> the internal data-structures when upgrading to use the new folders. Such
> migration code (including thorough tests) already exists in a project-
> specific package, and just needs to be moved into `plone.folder` itself.
> The migration runs relatively fast |---| in several performance tests
> about 13,000 folders holding some 200,000 items in total were migrated in
> about 5 minutes.
>
> .. |---| unicode:: U+2014 .. em dash
>
> The Container type in `plone.app.content` does not use BTrees, nor does
> it support ordering. Rather than changing this class and providing
> migration, we could add a new type, e.g. called `OrderedContainer` or
> just `Folder` (to signify closer resemblance to the standard folder
> behaviour) that mixes in the new class instead of `PortalFolder`.
>
> In addition, `plone.folder` also provides an ordering adapter, which only
> considers certain content to be orderable (implemented via marker
> interfaces). This allows for ordering of content that requires it, for
> example navigational items, without any added performance penalties if
> those share a folder with other, non-orderable content in large
> quantities.
>
> == Deliverables ==
>
> - New folderish base classes that are BTree-based and support large
> content sets as well as ordering.
> - Improved `Products.ATContentTypes.content.ATFolder` and
> `Products.ATContentTypes.content.ATBTreeFolder` versions which use BTrees
> and supports ordering.
> - New class plone.app.content.container.Folder (?) that uses BTrees and
> supports ordering.
> - The ability to switch to a folder view/behaviour that's optimised for
> "large" content sets.
> - In-place Migration for all existing sites.
>
> == Risks ==
>
> - There may be migration issues involved for derived folderish types
> with additional data-structures regarding containment and/or ordering.
> - This vision makes the existing `BaseBTreeFolder` in Archetypes
> orderable (though in some UI configurations you may not see the ordering,
> to prevent the user invoking slow operations), which may be unexpected.
>
> == Progress ==
>
> Complete:
>
> - Create a base folder implementation - OrderedBTreeFolderBase
> - Hook this into BaseBTreeFolder in Archetypes
> - Fix CMFPlone so that it detects the order support based on a Zope 3
> interface rather than a Zope 2 one
> - Test that order support works on the new folder
> - Update FTIs and portal_type's so that the default "Folder" portal_type
> is a btree-based folder
> - Enable next/previous navigation for this folder type
> - Enable photo-album support for this folder type
> - Enable archiving support for this folder type
> - Enable ordering policy to be specified by adapter
>
> Releases:
>
> - 1.0b3 of `plone.folder` has been released in May and is in Dexterity.
> - 1.0a1 of `plone.app.folder` has been released in May in order to
> factor out all code specific to `ATContentTypes`
> - Both releases have been heavily tested in a large-scale Jarn project
> and will go into production next week.
>
> To-do - ATContentTypes:
>
> - Merge in-place migration to new folder types
> - Write migrations for FTI and portal_type changes
> - Ensure compatibility with Plone 4.0, i.e. Python 2.6 & Zope 2.12
>
> To-do - plone.app.content:
>
> - Create new Folder base class, using the new mixin, with tests
>
> == Benchmark results ==
>
> The following two graphs show some simple benchmarks using the various
> types of folders (regular = ATFolder, large = ATBTreeFolder, ordered =
> ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The
> tests conducted where creating content, retrieving a batch of items from
> the folder, getting all items from the folder as well as random accessing
> the items. The benchmark tests used can be found in the
> [browser:plone.folder/trunk/src/plone/folder/tests/ tests/] folder.
>
> [[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_50_items.png)]]
> [[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_500_items.png)]]
>
> A more realistic scenario was provided by some benchmarks based on
> [http://jakarta.apache.org/jmeter/ JMeter] (available in
> [browser:plone.folder/jmeter/ jmeter]):
>
> [[Image(http://plone.org/products/plone/roadmap/191/jmeter_performance_tests.png)]]
>
> The graphs show the time it took to create a folder with 500 news items
> both using a standard folder and the new btree-based folder with order
> support. Also, and that's probably the most interesting figure so far,
> the time for repeatedly browsing the "folder contents" view is decreased
> to slightly less than 75% by using the btree folder.
>
> Another interesting thing to note is that the "content creation" tests
> show that btree-based folders are faster than the currently used
> implementation (from ATFolder) even for very small numbers of contained
> objects. The tests still run faster when adding 500, 50, 10, 5, 2 or
> even just a single object to the folder.
>
> == Participants ==
>
> - Martin Aspeli (IRC nickname: <optilude>)
> - Andreas Zeidler (IRC nickname: <witsch>)
> - Martijn Pieters (IRC nickname: <MJ>)
> - Alec Mitchell (IRC nickname: <alecm>)
New description:
,,Copied from [http://plone.org/products/plone/roadmap/191/ PLIP #191] in
the roadmap:,,
= Unify folder implementations =
''We currently have "Folder" and "Large Plone Folder" implementations.
[[br]]
There should be only one.''
Proposed by::
Martin Aspeli
Seconded by::
Andreas Zeidler
Proposal type::
Architecture
Repository branch::
[browser:plone.folder], [browser:plone.app.folder]
== Motivation ==
Shipping with two folder types is unnecessary for several reasons:
- It forces the user to make an a-priori choice about the number of
objects they plan to put into a folder
- We ship with the "Large" type disabled by default to avoid UI confusion
- We don't have a proper search-based UI for large folders anyway
Also the standard Folder type stores attributes, and has a single list
_objects tuple which keeps the list of objects and order. This is prone to
ConflictErrors and is slower. In simple benchmarks, a BTree-based folder
performs orders of magnitude faster than a basic folder.
== Proposal ==
Have a single folder implementation.
- The internal storage is BTrees
- It still supports ordering, by storing a separate sort order list/tree
- It has at least two views - one search-based for "large" folders, one
batch-based for "small" folders. This is either just a "display" menu
choice, or a choice in the object's schema. Explicit sorting may be turned
off for "large" folders.
== Implementation ==
The package `plone.folder` in the Plone SVN provides a base implementation
of a folder base class, which is not Archetypes specific, based on
BTreeFolder2, but adding ordering. The exact ordering implementation is
left up to an adapter, with a default providing explicit ordering. This
allows other implementations, such as auto-sorting based on some key.
The diagram below shows the folderish base- and mix-in classes used in
OFS, CMF, Archetypes and Plone. Count 'em:
[[Image(http://plone.org/products/plone/roadmap/191/Folder%20mess.png)]]
The new base class from `plone.folder`, i.e. `OrderedBTreeFolderBase`, is
used by `plone.app.folder` to provide two folderish classes, one targeted
to `Archetypes` (`BaseBTreeFolder`) as well as one for `ATContentTypes`
(`ATFolder`). Both add relatively little extra code and setup in order to
make them fully compatible with the original, to be replaced classes.
The package also provides a `GenericSetup` profile replacing the standard
"Folder" content type with the new one. In-place migration will convert
the internal data-structures when upgrading to use the new folders. Such
migration code (including thorough tests) already exists in a project-
specific package, and just needs to be moved into `plone.folder` itself.
The migration runs relatively fast — in several performance tests about
13,000 folders holding some 200,000 items in total were migrated in about
5 minutes.
The Container type in `plone.app.content` does not use BTrees, nor does it
support ordering. Rather than changing this class and providing migration,
we could add a new type, e.g. called `OrderedContainer` or just `Folder`
(to signify closer resemblance to the standard folder behaviour) that
mixes in the new class instead of `PortalFolder`.
In addition, `plone.folder` also provides an ordering adapter, which only
considers certain content to be orderable (implemented via marker
interfaces). This allows for ordering of content that requires it, for
example navigational items, without any added performance penalties if
those share a folder with other, non-orderable content in large
quantities.
== Deliverables ==
- New folderish base classes that are BTree-based and support large
content sets as well as ordering.
- Improved `Products.ATContentTypes.content.ATFolder` and
`Products.ATContentTypes.content.ATBTreeFolder` versions which use BTrees
and supports ordering.
- New class plone.app.content.container.Folder (?) that uses BTrees and
supports ordering.
- The ability to switch to a folder view/behaviour that's optimised for
"large" content sets.
- In-place Migration for all existing sites.
== Risks ==
- There may be migration issues involved for derived folderish types with
additional data-structures regarding containment and/or ordering.
- This vision makes the existing `BaseBTreeFolder` in Archetypes
orderable (though in some UI configurations you may not see the ordering,
to prevent the user invoking slow operations), which may be unexpected.
== Progress ==
Complete:
- Create a base folder implementation - OrderedBTreeFolderBase
- Hook this into BaseBTreeFolder in Archetypes
- Fix CMFPlone so that it detects the order support based on a Zope 3
interface rather than a Zope 2 one
- Test that order support works on the new folder
- Update FTIs and portal_type's so that the default "Folder" portal_type
is a btree-based folder
- Enable next/previous navigation for this folder type
- Enable photo-album support for this folder type
- Enable archiving support for this folder type
- Enable ordering policy to be specified by adapter
Releases:
- 1.0b3 of `plone.folder` has been released in May and is in Dexterity.
- 1.0a1 of `plone.app.folder` has been released in May in order to factor
out all code specific to `ATContentTypes`
- Both releases have been heavily tested in a large-scale Jarn project
and will go into production next week.
To-do - ATContentTypes:
- Merge in-place migration to new folder types
- Write migrations for FTI and portal_type changes
- Ensure compatibility with Plone 4.0, i.e. Python 2.6 & Zope 2.12
To-do - plone.app.content:
- Create new Folder base class, using the new mixin, with tests
== Benchmark results ==
The following two graphs show some simple benchmarks using the various
types of folders (regular = ATFolder, large = ATBTreeFolder, ordered =
ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The
tests conducted where creating content, retrieving a batch of items from
the folder, getting all items from the folder as well as random accessing
the items. The benchmark tests used can be found in the
[browser:plone.folder/trunk/src/plone/folder/tests/ tests/] folder.
[[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_50_items.png)]]
[[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_500_items.png)]]
A more realistic scenario was provided by some benchmarks based on
[http://jakarta.apache.org/jmeter/ JMeter] (available in
[browser:plone.folder/jmeter/ jmeter]):
[[Image(http://plone.org/products/plone/roadmap/191/jmeter_performance_tests.png)]]
The graphs show the time it took to create a folder with 500 news items
both using a standard folder and the new btree-based folder with order
support. Also, and that's probably the most interesting figure so far,
the time for repeatedly browsing the "folder contents" view is decreased
to slightly less than 75% by using the btree folder.
Another interesting thing to note is that the "content creation" tests
show that btree-based folders are faster than the currently used
implementation (from ATFolder) even for very small numbers of contained
objects. The tests still run faster when adding 500, 50, 10, 5, 2 or even
just a single object to the folder.
== Participants ==
- Martin Aspeli (IRC nickname: <optilude>)
- Andreas Zeidler (IRC nickname: <witsch>)
- Martijn Pieters (IRC nickname: <MJ>)
- Alec Mitchell (IRC nickname: <alecm>)
--
Comment(by witsch):
this is not ReSt, so unicode characters can be included directly... yay!
:)
--
Ticket URL: <http://dev.plone.org/plone/ticket/9316#comment:22>
Plone <http://plone.org>
Plone Content Management System
_______________________________________________
PLIP-Advisories mailing list
plip-advisor...@lists.plone.org
http://lists.plone.org/mailman/listinfo/plip-advisories
