Hi David, my solution ideally needs to support these storage backends
- file system - Swift - SoftLayer Object Storage So, once I have a way of shoe-horning a raw binary POST into Isis, which I wouldn't have to read fully into memory, I would code the rest around JClouds [3] with its BlobStore [4], which supports (and glosses over the differences between): - AWS S3 - Google Cloud Storage - Microsoft Azure - Rackspace US - Rackspace UK The rest of it could be coded as a blob-storage module with one or two tables which store storage paths, blob IDs, etc. like the security module does for users/groups. Cheers Kambiz [1] http://docs.openstack.org/developer/swift/ [2] http://www.softlayer.com/object-storage [3] https://jclouds.apache.org/ [4] https://jclouds.apache.org/start/blobstore/ On 2016-08-06 08:23 CEST, David Tildesley <[email protected]> wrote: > Good point Kambiz - with the likes of S3 object storage this is > becoming a common requirement - especially for very large files where > loading the whole file into memory is not appropriate. And much better > than storing in a rdbms blob column. > > I suggest a hidden mandatory attribute on a domain object (actually to > hold the stored object ID once it has uploaded) to prevent the form > from being submitted and javascript (delivered via a custom wicket > component) using Fine Uploader Javascript Upload Library since it is > Ajax your form is not submitted until your javascript allows it by > storing the uploaded object ID in the hidden mandatory field. Also the > user sees a fine progress bar whilst uploading and this library > supports chunking and direct to S3 and Azure or to traditional server > side (servlet). > > What do you think? > > Regards, > David. > > > image > > Fine Uploader Javascript Upload Library > > Fine Uploader. A dependency-free, open-source, native browser > > upload tool. > > > > View on fineuploader.com > > > > image > > On Saturday, 6 August 2016 7:31 AM, Martin Grigorov > <[email protected]> wrote: > > Hi Kambiz, > > I guess this code works only because you don't need another request > parameter in the same action. > Latest versions of Isis recommend to have an action per request > parameter, > so maybe this is not an issue. > But imagine submitting a form with one or more text fields and one > file > upload field. To be able to pass the values of the input fields Isis > will > need to call request.getParameter(someName) and this will consume the > request body and later your code won't see anything in the > inputstream. > > Martin Grigorov > Wicket Training and Consulting > https://twitter.com/mtgrigorov > > On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <[email protected]> > wrote: > >> Hello Martin, >> >> On 2016-08-05 13:42 CEST, Martin Grigorov <[email protected]> > wrote: >> >> > [...] >> > The problem here is that you have to make sure that your code is > the very >> > first one that reads from the ServletInputStream. Otherwise the > body will >> > be already consumed. >> > You may need to override some Wicket/RESTeasy code to be able to > do this >> > for the default viewers. >> > And this would be much harder than custom Servlet Filter in front > of the >> > ones by Wicket/RESTeasy >> >> If you look at the change in my original post: >> >> > https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73 >> f16857a628 >> >> you can see that I added this functionality to the >> restfulobjects-viewer, so that I can be sure that the full stream is >> handed to the action without any change. >> >> That code is tested against the Isis archetype with this addition to > the >> SimpleObjectMenu service: >> >> @Action(semantics = SemanticsOf.SAFE) >> public void upload(HttpServletRequest request, HttpServletResponse >> response) { >> FileOutputStream out = null; >> try { >> File outFile = new File("/tmp/request"); >> out = new FileOutputStream(outFile, false); >> IOUtils.copy(request.getInputStream(), out); >> out.close(); >> } catch (IOException e) { >> if(out != null) { >> try { >> out.close(); >> } catch (IOException e1) { >> // LOG something >> } >> } >> // throw something >> } >> } >> >> >> and the file in /tmp/request is identical to the posted binary file > which >> is sent by >> curl: >> >> curl -X POST --data-binary @big-binary-file >> --header "Authorization: Basic c3ZlbjpwYXNz" \ >> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \ >> --header "Content-Type: application/octet-stream" \ >> http://localhost:8080/restful/services/SimpleObjectMenu/ >> actions/upload/invoke >> >> >> Cheers >> >> >> Kambiz >> >> > >> >> action with the metadata of the files/request, but it would help > a lot >> >> to have such a facility inside Isis instead of writing a separate >> >> servlet and manually integrating its deployment and the calling > of >> >> domain actions etc. >> >> >> >> Cheers >> >> >> >> >> >> Kambiz >> >> >> >> >> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan < >> [email protected]> >> >> wrote: >> >> >> >> > Hi Kambiz, >> >> > >> >> > We had a requirement similar to yours. The problem with > injecting >> >> > HttpServletRequest to domain services is that it kind of > violates the >> >> DDD's >> >> > hexagonal architecture by letting viewer implementation leak > into the >> >> > domain layer. >> >> > >> >> > The specific requirement that we had was to be able to > pre-upload >> files >> >> > before submitting an action. We developed a custom blob that > stores >> an id >> >> > instead of the actual data, a service layer that manages a blob >> storage >> >> > abstracting and running on top of jclouds (we plan to be able > to >> support >> >> > other types of storage aside from file system), and a file > service >> that >> >> > manages the state of "transient files" (files that were > pre-uploaded >> or >> >> > that are returned from actions as result of export functions, > etc.) as >> >> well >> >> > as means to persist them (i.e. move them to another >> >> > container/bucket/folder). >> >> > >> >> > We then developed a custom servlet that handles download and > upload >> and >> >> > integrate them with isis. This enabled us to implement features > like >> >> forced >> >> > attachment content disposition, or on-demand image resizing. >> >> > >> >> > Should you have other ideas or questions, feel free to ask. >> >> > >> >> > >> >> > Best regards, >> >> > Willie >> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi > <[email protected]> >> >> wrote: >> >> > >> >> >> Hi, >> >> >> >> >> >> I have to implement a legacy file upload service for a > customer. >> Files >> >> >> of up to some hundred megabytes are uploaded to a service over > HTTP >> POST >> >> >> and a unique ID is returned which can be used to refer to that > file, >> add >> >> >> metadata to it etc. >> >> >> >> >> >> Using Blobs is not viable as parsing of the base64 encoded > binary >> data >> >> >> and the allocation of multiple strings during request > processing >> >> >> increases both the time and space requirements. >> >> >> >> >> >> Ideally, I would be able to create an action on one of my > domain >> >> >> services which has a HttpServletRequest argument and take care > of >> >> >> reading the binary data and writing them to the file system. >> >> >> >> >> >> I have played around with the request processing part of the >> >> >> restfulobjects viewer/server and managed to get the > request/response >> >> >> objects through. >> >> >> >> >> >> I know that this is a real edge case, but I think that it is >> important >> >> >> to have solutions for edge cases, too, if Isis is to be used > as a >> >> >> general purpose platform. >> >> >> >> >> >> Could some committers please review the changes and maybe > propose a >> >> >> cleaner way of implementing this? >> >> >> >> >> >> >> >> >> https://github.com/m-creations/isis/commit/ >> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628 >> >> >> >> >> >> Thanks >> >> >> >> >> >> >> >> >> Kambiz >> >> >> >> >> >> >> >> >>
