[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [34040]) tag [http://pypi.python.org/pypi/plone.app.blob/1.0b12 12th] beta (refs #6805, #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:91 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [33691]) the [http://pypi.python.org/pypi/plone.recipe.zope2instance/4.0a3 plone 4.0-specific] versions of `plone.recipe.zope2instance` don't work with python 2.4 since they rely on the [http://docs.python.org/library/hashlib.html hashlib] module — this should hopefully fix the [http://blobot.zitc.de/grid bot] again (refs #6805, #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:90 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [31040]) provide helper methods for easier migration of custom content types (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:81 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [30833]) try to mimic the value of `repr(...)` for the old file and image types for improved test compatibility (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:80 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [30597]) fix the missing dependency after [changeset:30566 not applying the blob gs profile] anymore (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:78 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [30615]) upgrade to ZODB 3.9.2 — the [http://pypi.python.org/pypi/ZODB3/3.9.2#change-history fix in 3.9.1] sounds like something we want (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:79 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [30561]) with r30550 in place the ZODB pin (to 3.8.x) can be lifted again (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:76 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [30566]) partly revert r30541: it doesn't make sense to apply `plone.app.blob`'s own gs profiles in plone 4 — the FTI changes can simply be integrated instead (refs #7822) please note that the profiles need to stay for 3.x, though. -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:77 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [30519]) aha, all new packages need to be added to `[test]` in order to make `$ bin/test ...` work (refs #7822, #9316) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:69 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [30541]) merge [browser:Plone/branches/plip7822-blob-supp...@30540 branch for plip7822] into [browser:Plone/branches/4...@30540 4.0] (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:70 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [30544]) merge [browser:buildouts/plone- coredev/branches/4.0/plips/plip7822-blob-support@30543 buildout changes for plip7822] (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:71 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Changes (by witsch): * status: assigned = closed * resolution: = fixed Comment: the PLIP has been merged, and even though there are still [browser:buildouts/plone-coredev/branches/4.0/plips/plip7822-blob- support@30545#l19 things left to wrap up polish] this ticket is done as far as the PLIP is concerned... -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:73 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: closed Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: fixed Keywords: focusarea | +--- Comment(by witsch): (In [30549]) triggered by the insight that the blob test setup doesn't work with plone 4.0 and zodb 3.9, because it is messing with things too much when zodb 3.9's `DemoStorage` actually already is a `BlobStorage` as well, i thought i should try if the `custom_zodb.py` is still needed at all... (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:74 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [30354]) tag [http://pypi.python.org/pypi/plone.app.blob/1.0b6 6th] beta (refs #6805, #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:68 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by esteele): This PLIP has been accepted for merging into Plone 4.0 The final vote was: Alec Mitchell +1 David Glick +1 Erik Rose +1 Laurence Rowe +1 Matthew Wilkes - Ross Patterson +1 Please merge your branches into the Plone 4.0 head by end-of-day Friday Oct 16. If you need assistance with merging, please contact me. We'll be assigning a documentation ticket to this PLIP shortly. Please assist the docs team in documenting the changes and new features that this PLIP introduces. -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:66 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by rossp): FWT vote, +1 for merge -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:65 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [29875]) use the right permissions when registering the replacement types for File and Image content, i.e. the original ones from `ATContentTypes` (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:49 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [29857]) actually, r29755 should have gone here: use [http://pypi.python.org/pypi/archetypes.schemaextender archetypes.schemaextender] trunk in order to verify if the [http://dev.plone.org/archetypes/changeset/11344 recent performance fix] brings blob support back on par with vanilla plone 4 (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:48 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): * Using `archetypes.schemaextender` and marker interfaces introduces a new level of indirection when implementing content types. It also adds performance issues due to the additional and generally more expensive schema lookups. said performance issues should be mostly fixed after [http://dev.plone.org/archetypes/changeset/11344 r11...@archetypes]. the slowdown for serving small images as reported [browser:buildouts/plone- coredev/branches/4.0/plips/plip7822-review-alecm@29529#l29 in alec's review] is now almost gone: the request rate for a 600+ byte file when up from about 80 to a little more than 100 again — pending further investigation the rest is likely due to the extra overhead of fetching two zodb objects and opening/reading the blob file... This can be solved by also shipping with `archetypes.schematuning`, which has been proposed as PLIP 9376. adding `archetypes.schematuning` to the mix now actually make the above benchmark run ever so slightly slower — 100 vs 101 req/s... -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:43 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by alecm): (In [29529]) Add reviw for PLIP #7822 (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:42 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [29374]) tag [http://pypi.python.org/pypi/plone.app.blob/1.0b5 5th] beta (refs #6805, #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:41 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Old description: == task description == * create an integration package for zodb blob support for plone * implement a new content type intended to be used to store binary data * provide a sub-type layer to allow this type to mimick the existing ATFile and ATImage types * provide migration tools to allow existing content to be converted to use zodb blobs * provide hooks for 3rd-party products to use a blob-aware AT-field and add additional sub-types and migrations * document the setup, migration as well as best practices of how to use the integration layer with custom types == deliverables == * an add-on package for plone 3.0/3.1 providing blob support for existing sites (incl. migration) * a core package intended to be shipped with plone 4.0 (or a release based on zope 2.11) * documentation for users, site integrators as well as developers, i.e. various howtos * perhaps also a wsgi layer helping to overcome the blocking problem due to the limited number of threads in the zope2 publisher == success would look like... == * lots of plone sites start using zodb blobs and migrate their existing content from standard AT types (or other blob solutions) * add-on packages adopt the sub-typing approach for their content types, e.g. the plone4artists suite * people start using alternative file-systems for their blob storages, e.g. amazon s3, linux drbd, coda, gfs etc * the confusion about the various existing other blob solutions and which of them to use vanishes and people are referring to zodb blob (and the above mentioned integration package) when they talk about blobs * the zope2 thread blocking issue has become a non-issue, since people are running repoze anyway... == current status == * the integration package is already half-way there — it is called [http://plone.org/products/plone.app.blob plone.app.blob] and lives in the plone [http://dev.plone.org/plone/browser/plone.app.blob/ svn repository] * a [http://dev.plone.org/plone/ticket/6805 trac ticket] was abused as some sort of progress log and to discuss things — it contains a lot of technical details for the interested reader (but also a lot of redundancy :)) * the package provides a new content type that can be used as a drop- in replacement for ATFile * it also provides a migration path for existing ATFile content * a lot of people have helped with testing the package and it is already used in a few production sites * thanks to a generous sponsorship of headnet there's a working branch supporting plone 2.5 * thanks to the tireless efforts of enfoldsystems it also works on windows as of recently * several buildouts (based on ploneout and plone.recipe.plone) are provided * however, support for mimicking ATImage content is still missing == timeline == * next week will see a first beta release including some recent bugfixes * after that i'll start working on image support, which should be ready by the end of march * rc and 1.0 final releases in april i would be glad to hear your concerns, suggestions and questions about this. also testing the package and providing feedback would be of great value! and last but not least, if you'd like to participate in the development please contact me directly via email. cheers, andi New description: ,,Copied in part from [http://plone.org/products/plone/roadmap/154/ PLIP #154] in the roadmap:,, = Large file handling = ''It is possible to configure Zope to work with very large files, but the out-of-the-box story is not terribly great. It should be obvious how to configure Plone so that it can handle large volumes of MS Office, PDF or media files, for example.'' Proposed by:: Martin Aspeli Seconded by:: Martijn Pieters Proposal type:: Architecture Repository branch:: [browser:plone.app.blob], [browser:plone.app.imaging] == Motivation == In many ways, Plone is well-suited to document management and the management of files in general. Tools such as `ExternalEditor` and `Enfold Desktop` make this even more true. However, due to the way the ZODB works, large files can be problematic to work with — if you're not careful, your ZODB could balloon because each change to a file revisions the whole object. There are solutions to this problem, which usually involve storing some content outside the ZODB. However, the out-of-the-box story in Plone isn't good enough. It
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): (In [28898]) add review buildout for [https://dev.plone.org/plone/ticket/7822 blob support] (refs #7822) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:39 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by esteele): Approved by FWT vote. -- Ticket URL: https://dev.plone.org/plone/ticket/7822#comment:38 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
[PLIP-Advisories] Re: [Plone] #7822: Make standard file content types use ZODB BLOB support
#7822: Make standard file content types use ZODB BLOB support +--- Reporter: limi|Owner: witsch Type: PLIP| Status: assigned Priority: major |Milestone: 4.0 Component: Infrastructure | Resolution: Keywords: focusarea | +--- Comment(by witsch): Replying to [comment:33 klm]: [...] just trying to factor in what might be pitfalls to address when the time comes to decide about adoption. i know you are, and like i said, much appreciate it... :) -- Ticket URL: http://dev.plone.org/plone/ticket/7822#comment:35 Plone http://plone.org Plone Content Management System ___ PLIP-Advisories mailing list plip-advisor...@lists.plone.org http://lists.plone.org/mailman/listinfo/plip-advisories
