Hi, I have a Groovy script in my XWiki 7.1.1 site that reads files sent to it. I've been using the fileUpload plugin as shown below.
{{groovy}} import java.io.InputStream; import java.io.OutputStream; import org.apache.commons.io.IOUtils; import org.apache.commons.fileupload.FileItem // Function to actually read the file data from file.getInputStream(); def processFile(String filename, FileItem file, String title) { ... } def params = request.getParameterNames().toList() def fileUpload = xwiki.fileupload def results = [:] if (!(params.contains("title"))) { results.put('error',"No title specified") } else { fileUpload.loadFileList(2147483648L, 100000, (String) (xcontext.getContext().getWiki().Param("xwiki.upload.tempdir"))) FileItem fileItem = fileUpload.getFileItems().find {it.getFieldName().equals("newfile")} if (fileItem == null) { results.put('error',"Couldn't load file - did you forget to choose one?") } else { results = processFile(fileUpload.getFileName("newfile"), fileItem, request.getParameter('title')) } } // Handle display of results based on how script was called def jsontool = new org.xwiki.velocity.tools.JSONTool() if (request.getParameter('outputSyntax') == 'plain' || request.getParameter('xpage') == 'plain') { response.setContentType('application/json') if (results.error) { response.setStatus(400) } print jsontool.serialize(results) } else { if(results.error) { println "{{error}}${results.error}{{/error}}" } else { println "{{success}}File processed{{/success}}" } println "Return to [[Original page>>${request.getHeader('referer')}]]"; } {{/groovy}} Files are sent to this page via a form: <form action="FileProcessor" method="post" enctype="multipart/form-data"> <input type="file" name="newfile" /> <input type="hidden" name="title" value="foo" /> <button type="submit">Upload</button> </form> The files I need to read can sometimes be large (~1GB), which is why the call to fileUpload.loadFileList explicitly gives a large maximum file size (10GB, although I don't expect files to reach this size for the time being). However, this limit seems to get ignored: when files exceed the limit specified in the Maximum Upload Size setting on xwiki/bin/edit/XWiki/XWikiPreferences?editor=object the script fails at the step (!(params.contains("title"))) { ... This is despite the title definitely being set. My guess is the an exception gets thrown by the File Upload plugin that causes the parameters to fail being processed - when I increase the upload limit from 32MB to 50MB the problem goes away for a 47MB file, for instance. So problem 1: is there a way to get fileUpload.loadFileList to respect the file limit I specify? I don't want to allow very large files to be included in attachments, so it would be nice if I can leave the default setting lower. Then comes problem 2: Even if I do set the maximum size to something very large, I get a similar problem for files over about 500Mb. I've tried testing just uploading such files as attachments and I end up with a page saying "waiting for server confirmation", and then "an error occurred while uploading the file". Digging around in the Tomcat logs I see exceptions containing: Caused by: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. No space left on device at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:351) ~[commons-fileupload-1.3.1.jar:1.3.1] at com.xpn.xwiki.plugin.fileupload.FileUploadPlugin.loadFileList(FileUploadPlugin.java:248) ~[xwiki-platform-legacy-oldcore-7.1.2.jar:na] ... 47 common frames omitted Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.7.0_79] at java.io.FileOutputStream.write(FileOutputStream.java:345) ~[na:1.7.0_79] at org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:129) ~[commons-io-2.4.jar:2.4] at org.apache.commons.fileupload.util.Streams.copy(Streams.java:107) ~[commons-fileupload-1.3.1.jar:1.3.1] at org.apache.commons.fileupload.util.Streams.copy(Streams.java:70) ~[commons-fileupload-1.3.1.jar:1.3.1] at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:347) ~[commons-fileupload-1.3.1.jar:1.3.1] ... 48 common frames omitted Looking at this I see that there is probably an earlier call to loadFileList with default values, which is probably why my first problem was occurring. I can't see where this is being called - probably the crucial info is in one of those 48 omitted frames. My guess is that it's in com.xpn.xwiki.web.Utils.handleMultipart Also, looking in https://github.com/xwiki/xwiki-platform/blob/7392653655d1a8524c67851393b0ac16b07974f5/xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/web/Utils.java#L611 it appears that all files in a request are processed using org.apache.commons.fileupload.FileUploadBase.parseRequest() which writes all files to file, which I'm guessing is the ultimate problem once files get too big to store in temporary storage. Is there any way that I can use the streaming API http://commons.apache.org/proper/commons-fileupload/streaming.html instead? Thanks, Bryn _______________________________________________ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users