Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
shivshan wrote: > Hi > There was another discussion on this topic, where an effort has been made to > store the files directly on the filesystem. Not sure how this would reduce > memory consumption. Please read Sergiu's comments carefully though, before > you try this out. > Link > http://www.nabble.com/AttachmentStore---my-filesystem-based-implementation-td16987518.html#a17040326 > here > Of course, in the meantime, we await the WebDAV implementation. > Shiva I'm not sure the webdav implementation will solve the memory problem. If I understand correctly how it will work, then it will just be an interface over the same core, meaning that all the performance issues will still be there. > > vmassol wrote: >> >> On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote: >> >>> Hi, Vincent. >>> >>> Vincent Massol wrote: Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory). >>> Look at >>> http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query >>> :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results Worse, if this is not the first version of the attachment, then the complete attachment history is loaded in memory, so add another 24*versionsize*versions of memory needed during upload. >>> This is a feature of JRCS :) >> I see. So I guess the only solution would be drop JRCS and do our own >> stuff... (or find another library that does it fast). >> >> Thanks >> -Vincent >> -- Sergiu Dumitriu http://purl.org/net/sergiu/ ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Hi There was another discussion on this topic, where an effort has been made to store the files directly on the filesystem. Not sure how this would reduce memory consumption. Please read Sergiu's comments carefully though, before you try this out. Link http://www.nabble.com/AttachmentStore---my-filesystem-based-implementation-td16987518.html#a17040326 here Of course, in the meantime, we await the WebDAV implementation. Shiva vmassol wrote: > > > On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote: > >> Hi, Vincent. >> >> Vincent Massol wrote: >>> Why is it using some much memory? Isn't that a bug? (I don't >>> understand why we would need to load more than 1 attachment version >>> at >>> a time in memory). >> >> Look at >> http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query >> :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results >>> Worse, if this is not the first version of the attachment, then the >>> complete attachment history is loaded in memory, so add another >>> 24*versionsize*versions of memory needed during upload. >> >> This is a feature of JRCS :) > > I see. So I guess the only solution would be drop JRCS and do our own > stuff... (or find another library that does it fast). > > Thanks > -Vincent > > ___ > users mailing list > users@xwiki.org > http://lists.xwiki.org/mailman/listinfo/users > > -- View this message in context: http://www.nabble.com/XWiki-Does-Not-Handle-Attachments-Efficiently-tp17741586p17801734.html Sent from the XWiki- Users mailing list archive at Nabble.com. ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote: > Hi, Vincent. > > Vincent Massol wrote: >> Why is it using some much memory? Isn't that a bug? (I don't >> understand why we would need to load more than 1 attachment version >> at >> a time in memory). > > Look at > http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query > :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results >> Worse, if this is not the first version of the attachment, then the >> complete attachment history is loaded in memory, so add another >> 24*versionsize*versions of memory needed during upload. > > This is a feature of JRCS :) I see. So I guess the only solution would be drop JRCS and do our own stuff... (or find another library that does it fast). Thanks -Vincent ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Hi, Vincent. Vincent Massol wrote: > Why is it using some much memory? Isn't that a bug? (I don't > understand why we would need to load more than 1 attachment version at > a time in memory). Look at http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query:attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results > Worse, if this is not the first version of the attachment, then the > complete attachment history is loaded in memory, so add another > 24*versionsize*versions of memory needed during upload. This is a feature of JRCS :) -- Artem Melentyev ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Hi Artem, On Jun 12, 2008, at 2:21 PM, Artem Melentyev wrote: > Hi. > > You can set xwiki.store.attachment.versioning=0 in xwiki.cfg > This will turn off attachment versioning and *very* much reduce memory > consumption. > see http://jira.xwiki.org/jira/browse/XWIKI-2275 > and > http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Configuration#HOptionalStoreFeatures Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory). Thanks -Vincent > AW wrote: >> Is there anything I can do to make XWiki 1.4 handle large >> attachments? That >> is, attachments from a few megabytes up to a few dozen megabytes. ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Hi. You can set xwiki.store.attachment.versioning=0 in xwiki.cfg This will turn off attachment versioning and *very* much reduce memory consumption. see http://jira.xwiki.org/jira/browse/XWIKI-2275 and http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Configuration#HOptionalStoreFeatures AW wrote: > Is there anything I can do to make XWiki 1.4 handle large attachments? That > is, attachments from a few megabytes up to a few dozen megabytes. > >... -- Artem Melentyev ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Yes storing attachments directly in the file system and referencing them would be good. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of AW Sent: 09 June 2008 21:15 To: XWiki Users Subject: [xwiki-users] XWiki Does Not Handle Attachments Efficiently Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes. XWiki does not appear to efficiently handle attachments in general; large attachments appear to require extraordinary measures and resources. In testing prior to a deployment I attempted to attach a 56 MB file to an instance running on a server with 2GB of RAM and using MySQL as the database. To attach the file: 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a larger value, since by default it's only 1MB and XWiki stores attachments directly in the database, rather than saving them in the file system and referencing the attachment's path. 2. Next I needed change the maximum attachment size in XWiki, which can be done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a NumberProperty field of type long, named upload_maxsize" which I finally figured out I could accomplish by visiting a page similar to "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"; ...although this wasn't immediately obvious. 3. At this point attempting to attach the file resulted in a JDBC error, which I resolved by manually changing some columns in the database from type mediumblob to longblob. 4. Attempting to attach the file still didn't work, this time due to Java heap space limitations. I increased the size of the heap, modifying the JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes to 512MB, up from 256). This had no effect on the error, so I increased the size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the file. Next I attempted to delete the attachment along with the page to which it is attached. My naïve assumption was that deleting an attachment should be easier than adding one and I didn't anticipate problems. Unfortunately, this again resulted in out of memory errors. To work around the problem I tried deleting the attachment by clicking the delete button in the show --> attachments screen. This still resulted in an out of memory error, so I upped the heap size to 1.6 GB, or 80% of the RAM in the server. This resulted in very heavy swapping, yet still failed due to an out of memory exception. (On the positive side, at least the swapping helped keep the processor utilization under 100%.) I've ordered another 2GB of RAM and will try increasing the heap size again after it's installed. Until then, is there anything I can do to keep attachments from eating the server alive? If I recall correctly, there has been talk about saving attachments externally, rather than within the database. Has anyone built a mechanism to allow this, or is there any news to report on that front? Also, does anyone have any advice on how I can excise this page and its 56MB attachment from the system prior to more memory arriving? (I'd like to do so as, aside from the attachment problems and dysfunctional WYSIWYG editor, the system looks deployable.) As it stands, unless I'm doing something incorrectly, I have to conclude that it's basically impractical to use attachments larger than a few megabytes with XWiki, and even small attachments are frightfully ungainly. - AW ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Thales UK Ltd (Wells) DISCLAIMER: The information contained in this e-mail is confidential. It may also be legally privileged. It is intended only for the stated addressee(s) and access to it by any other person is unauthorised. If you are not an addressee, you must not disclose, copy, circulate or in any other way use or rely on the information contained in this e-mail. Such unauthorised use may be unlawful. We may monitor all e-mail communications through our networks. If you have received this e-mail in error, please inform us immediately on +44 (0) 1749 672081 and delete it and all copies from your system. We accept no responsibility for changes to any e-mail which occur after it has been sent. Attachments to this e-mail may contain software viruses which could damage your system. We therefore recommend you virus-check all attachments before opening. A business of Thales UK Ltd. Registered Office: 2 Dashwood Lang Road, The Bourne Business Park, Addlestone, Weybridge, Surrey KT15 2NX Registered in England No. 868273 ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Hi AW, you may want to look at this thread that addresses most of the issues you encountered AFAIR : http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query:attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results Guillaume On Mon, Jun 9, 2008 at 11:24 PM, [Ricardo Rodriguez] Your EPEC Network ICT Team <[EMAIL PROTECTED]> wrote: > Hi AW, > > I think there is a general agreement about the fact that an option to > store attachments, particularly big ones, outside the database. I think > it was of the main reasons to accept/promote WebDAV integration as one > of the projects to be developed during the current Google Summer of Code > (GSoC) mentored by Ludovic Dubost. > > http://code.google.com/soc/2008/xwiki/about.html > > Perhaps these threads will help you to catch up with this issues and > make some contribution!!! > > http://www.nabble.com/About-WebDav-Integration-tp16358565p16358565.html > > http://www.nabble.com/Need-help-with-WebDAV-Support.-tp16453419p16453419.html > http://www.nabble.com/-GSOC--WebDAV--Status-...-tp17588475p17588475.html > > http://www.nabble.com/-GSOC---WebDAV--Ground-work-done.-tp17674844p17674844.html > > > Hope this helps, > > Ricardo > > AW wrote: > > Is there anything I can do to make XWiki 1.4 handle large attachments? > That > > is, attachments from a few megabytes up to a few dozen megabytes. > > > > XWiki does not appear to efficiently handle attachments in general; large > > attachments appear to require extraordinary measures and resources. > > > > In testing prior to a deployment I attempted to attach a 56 MB file to an > > instance running on a server with 2GB of RAM and using MySQL as the > > database. > > > > To attach the file: > > > > 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf > to a > > larger value, since by default it's only 1MB and XWiki stores attachments > > directly in the database, rather than saving them in the file system and > > referencing the attachment's path. > > > > 2. Next I needed change the maximum attachment size in XWiki, which can > be > > done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a > > NumberProperty field of type long, named upload_maxsize" which I finally > > figured out I could accomplish by visiting a page similar to > > "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"; > > ...although this wasn't immediately obvious. > > > > 3. At this point attempting to attach the file resulted in a JDBC error, > > which I resolved by manually changing some columns in the database from > type > > mediumblob to longblob. > > > > 4. Attempting to attach the file still didn't work, this time due to Java > > heap space limitations. I increased the size of the heap, modifying the > > JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap > sizes > > to 512MB, up from 256). This had no effect on the error, so I increased > the > > size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the > > file. > > > > Next I attempted to delete the attachment along with the page to which it > is > > attached. My naïve assumption was that deleting an attachment should be > > easier than adding one and I didn't anticipate problems. Unfortunately, > this > > again resulted in out of memory errors. To work around the problem I > tried > > deleting the attachment by clicking the delete button in the show --> > > attachments screen. This still resulted in an out of memory error, so I > > upped the heap size to 1.6 GB, or 80% of the RAM in the server. This > > resulted in very heavy swapping, yet still failed due to an out of memory > > exception. (On the positive side, at least the swapping helped keep the > > processor utilization under 100%.) > > > > I've ordered another 2GB of RAM and will try increasing the heap size > again > > after it's installed. Until then, is there anything I can do to keep > > attachments from eating the server alive? If I recall correctly, there > has > > been talk about saving attachments externally, rather than within the > > database. Has anyone built a mechanism to allow this, or is there any > news > > to report on that front? > > > > Also, does anyone have any advice on how I can excise this page and its > 56MB > > attachment from the system prior to more memory arriving? (I'd like to do > so > > as, aside from the attachment problems and dysfunctional WYSIWYG editor, > the > > system looks deployable.) > > > > As it stands, unless I'm doing something incorrectly, I have to conclude > > that it's basically impractical to use attachments larger than a few > > megabytes with XWiki, and even small attachments are frightfully > ungainly. > > > > - AW > > > > > > ___ > > users mailing list > > users@xwiki.org > > http://lists.xwiki.org/mailman/listinfo/users > > > > > > > > > -- > Ricardo Rodríguez > Your EPEC Network ICT Team > >
Re: [xwiki-users] XWiki Does Not Handle Attachments Efficiently
Hi AW, I think there is a general agreement about the fact that an option to store attachments, particularly big ones, outside the database. I think it was of the main reasons to accept/promote WebDAV integration as one of the projects to be developed during the current Google Summer of Code (GSoC) mentored by Ludovic Dubost. http://code.google.com/soc/2008/xwiki/about.html Perhaps these threads will help you to catch up with this issues and make some contribution!!! http://www.nabble.com/About-WebDav-Integration-tp16358565p16358565.html http://www.nabble.com/Need-help-with-WebDAV-Support.-tp16453419p16453419.html http://www.nabble.com/-GSOC--WebDAV--Status-...-tp17588475p17588475.html http://www.nabble.com/-GSOC---WebDAV--Ground-work-done.-tp17674844p17674844.html Hope this helps, Ricardo AW wrote: > Is there anything I can do to make XWiki 1.4 handle large attachments? That > is, attachments from a few megabytes up to a few dozen megabytes. > > XWiki does not appear to efficiently handle attachments in general; large > attachments appear to require extraordinary measures and resources. > > In testing prior to a deployment I attempted to attach a 56 MB file to an > instance running on a server with 2GB of RAM and using MySQL as the > database. > > To attach the file: > > 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a > larger value, since by default it's only 1MB and XWiki stores attachments > directly in the database, rather than saving them in the file system and > referencing the attachment's path. > > 2. Next I needed change the maximum attachment size in XWiki, which can be > done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a > NumberProperty field of type long, named upload_maxsize" which I finally > figured out I could accomplish by visiting a page similar to > "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"; > ...although this wasn't immediately obvious. > > 3. At this point attempting to attach the file resulted in a JDBC error, > which I resolved by manually changing some columns in the database from type > mediumblob to longblob. > > 4. Attempting to attach the file still didn't work, this time due to Java > heap space limitations. I increased the size of the heap, modifying the > JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes > to 512MB, up from 256). This had no effect on the error, so I increased the > size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the > file. > > Next I attempted to delete the attachment along with the page to which it is > attached. My naïve assumption was that deleting an attachment should be > easier than adding one and I didn't anticipate problems. Unfortunately, this > again resulted in out of memory errors. To work around the problem I tried > deleting the attachment by clicking the delete button in the show --> > attachments screen. This still resulted in an out of memory error, so I > upped the heap size to 1.6 GB, or 80% of the RAM in the server. This > resulted in very heavy swapping, yet still failed due to an out of memory > exception. (On the positive side, at least the swapping helped keep the > processor utilization under 100%.) > > I've ordered another 2GB of RAM and will try increasing the heap size again > after it's installed. Until then, is there anything I can do to keep > attachments from eating the server alive? If I recall correctly, there has > been talk about saving attachments externally, rather than within the > database. Has anyone built a mechanism to allow this, or is there any news > to report on that front? > > Also, does anyone have any advice on how I can excise this page and its 56MB > attachment from the system prior to more memory arriving? (I'd like to do so > as, aside from the attachment problems and dysfunctional WYSIWYG editor, the > system looks deployable.) > > As it stands, unless I'm doing something incorrectly, I have to conclude > that it's basically impractical to use attachments larger than a few > megabytes with XWiki, and even small attachments are frightfully ungainly. > > - AW > > > ___ > users mailing list > users@xwiki.org > http://lists.xwiki.org/mailman/listinfo/users > > > -- Ricardo Rodríguez Your EPEC Network ICT Team ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
[xwiki-users] XWiki Does Not Handle Attachments Efficiently
Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes. XWiki does not appear to efficiently handle attachments in general; large attachments appear to require extraordinary measures and resources. In testing prior to a deployment I attempted to attach a 56 MB file to an instance running on a server with 2GB of RAM and using MySQL as the database. To attach the file: 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a larger value, since by default it's only 1MB and XWiki stores attachments directly in the database, rather than saving them in the file system and referencing the attachment's path. 2. Next I needed change the maximum attachment size in XWiki, which can be done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a NumberProperty field of type long, named upload_maxsize" which I finally figured out I could accomplish by visiting a page similar to "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object"; ...although this wasn't immediately obvious. 3. At this point attempting to attach the file resulted in a JDBC error, which I resolved by manually changing some columns in the database from type mediumblob to longblob. 4. Attempting to attach the file still didn't work, this time due to Java heap space limitations. I increased the size of the heap, modifying the JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes to 512MB, up from 256). This had no effect on the error, so I increased the size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the file. Next I attempted to delete the attachment along with the page to which it is attached. My naïve assumption was that deleting an attachment should be easier than adding one and I didn't anticipate problems. Unfortunately, this again resulted in out of memory errors. To work around the problem I tried deleting the attachment by clicking the delete button in the show --> attachments screen. This still resulted in an out of memory error, so I upped the heap size to 1.6 GB, or 80% of the RAM in the server. This resulted in very heavy swapping, yet still failed due to an out of memory exception. (On the positive side, at least the swapping helped keep the processor utilization under 100%.) I've ordered another 2GB of RAM and will try increasing the heap size again after it's installed. Until then, is there anything I can do to keep attachments from eating the server alive? If I recall correctly, there has been talk about saving attachments externally, rather than within the database. Has anyone built a mechanism to allow this, or is there any news to report on that front? Also, does anyone have any advice on how I can excise this page and its 56MB attachment from the system prior to more memory arriving? (I'd like to do so as, aside from the attachment problems and dysfunctional WYSIWYG editor, the system looks deployable.) As it stands, unless I'm doing something incorrectly, I have to conclude that it's basically impractical to use attachments larger than a few megabytes with XWiki, and even small attachments are frightfully ungainly. - AW ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
