Caleb, your analysis seems to match very well what I had observed. Does it mean that such changes would also affect the XML-serailization?
paul On 14 oct. 2010, at 15:24, Caleb James DeLisle wrote: > Hi, > I have some changes to the attachment system which will allow XWiki to handle > much larger > attachments without memory exhaustion. I have found that there are some > places where I cannot make > any changes because the code is not in XWiki but rather in JRCS. > > XWiki versions attachments by creating a JRCS node for the XML version of > each version of each > attachment. This means that memory consumption improvements hit a hard wall > at 2 * 2 * 1.3 * the > size of the attachment. base-64 encoding for XML increases the size by 1.3 > times, storage as a > String (array of 16 bit chars) doubles the size and the need to copy the > String doubles the size again. > > The second issue is that the database and JDBC do not handle multiple > hundreds of megabytes in a > single query well. If I try to attach a 500MB attachment with attachment > versioning disabled, my > changes allow the attachment to be streamed to the database but postgresql is > not able to save it. I > am able to attach a 256MB attachment but with 512MB of heap space, the > attachment cannot be loaded > from the database because JDBC lacks the necessary streaming functionality. > > An option which I am now considering is adding a binary table to the database > schema. The table > would contain a composite id made of the id of the data and the part number > of that entry, and a > data column slightly smaller than 1MB (default max_allowed_packet in mysql). > All interaction with > this table would go through a storage engine which would require InputStreams > and OutputStreams and > the streams would be written and read by the storage mechanism which would ID > tag them and break > them up into parts to be sent to the database individually. > > WDYT? > > Caleb > > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
