Coming late to this thread, apologies (been on vacation). With respect to a mechanism for uploading BLOBs via RO, in the first event I think I'd rather this is done through separate server-side resources that sit alongside the Isis runtime. Using the headless APIs [1] it ought to be relatively easy to have a custom controller get access into the Isis runtime. (I'm hoping to make the handling of sessions and transactions more consistent between RO and Wicket viewer, eg through a standard set of filters in web.xml; hopefully that will make it easier for us to support custom functionality, eg this and no-wicket).
With respect to the storage of the BLOBs themselves, I don't think (at least I couldn't find) any datatype mapping to be able to map from a stream (obtained from ServletInputStream) into a JDO entity, ie to avoid the creation of the BLOB in-memory. If it *were* possible, then I don't think Isis would care; just map the property as @Programmatic and Isis would ignore it. Perhaps Andy might be able to chip in on this? Somewhat related, within Estatio we have a requirement to generate PDFs (invoices) but want to eventually store these outside of the RDBMS, eg S3 or in an external CMS. Since our BLOBs aren't too large, we are considering defining an entity that stores the BLOBs locally, but then have a Camel or Quartz background job that would then move the BLOB data out into the external storage. When the BLOB data is moved, it would then hold a reference to the S3 BLOB. The user experience is that the PDF would be immediately available (stored within the same transaction that creates the invoice), and without all the hassle of having to handle eventual consistency issues. HTH Dan [1] http://isis.apache.org/guides/ugbtb.html#_ugbtb_headless-access [2] https://github.com/subes/invesdwin-nowicket On 6 August 2016 at 09:04, Kambiz Darabi <[email protected]> wrote: > Hi David, > > my solution ideally needs to support these storage backends > > - file system > - Swift > - SoftLayer Object Storage > > So, once I have a way of shoe-horning a raw binary POST into Isis, which > I wouldn't have to read fully into memory, I would code the rest around > JClouds [3] with its BlobStore [4], which supports (and glosses over the > differences between): > > - AWS S3 > - Google Cloud Storage > - Microsoft Azure > - Rackspace US > - Rackspace UK > > The rest of it could be coded as a blob-storage module with one or two > tables which store storage paths, blob IDs, etc. like the security > module does for users/groups. > > Cheers > > > Kambiz > > [1] http://docs.openstack.org/developer/swift/ > [2] http://www.softlayer.com/object-storage > [3] https://jclouds.apache.org/ > [4] https://jclouds.apache.org/start/blobstore/ > > On 2016-08-06 08:23 CEST, David Tildesley <[email protected]> wrote: > > > Good point Kambiz - with the likes of S3 object storage this is > > becoming a common requirement - especially for very large files where > > loading the whole file into memory is not appropriate. And much better > > than storing in a rdbms blob column. > > > > I suggest a hidden mandatory attribute on a domain object (actually to > > hold the stored object ID once it has uploaded) to prevent the form > > from being submitted and javascript (delivered via a custom wicket > > component) using Fine Uploader Javascript Upload Library since it is > > Ajax your form is not submitted until your javascript allows it by > > storing the uploaded object ID in the hidden mandatory field. Also the > > user sees a fine progress bar whilst uploading and this library > > supports chunking and direct to S3 and Azure or to traditional server > > side (servlet). > > > > What do you think? > > > > Regards, > > David. > > > > > > image > > Fine Uploader Javascript Upload Library > > Fine Uploader. A dependency-free, open-source, native browser > > upload tool. > > > > View on fineuploader.com > > > > > > image > > > > On Saturday, 6 August 2016 7:31 AM, Martin Grigorov > > <[email protected]> wrote: > > > > Hi Kambiz, > > > > I guess this code works only because you don't need another request > > parameter in the same action. > > Latest versions of Isis recommend to have an action per request > > parameter, > > so maybe this is not an issue. > > But imagine submitting a form with one or more text fields and one > > file > > upload field. To be able to pass the values of the input fields Isis > > will > > need to call request.getParameter(someName) and this will consume the > > request body and later your code won't see anything in the > > inputstream. > > > > Martin Grigorov > > Wicket Training and Consulting > > https://twitter.com/mtgrigorov > > > > On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi <[email protected]> > > wrote: > > > >> Hello Martin, > >> > >> On 2016-08-05 13:42 CEST, Martin Grigorov <[email protected]> > > wrote: > >> > >> > [...] > >> > The problem here is that you have to make sure that your code is > > the very > >> > first one that reads from the ServletInputStream. Otherwise the > > body will > >> > be already consumed. > >> > You may need to override some Wicket/RESTeasy code to be able to > > do this > >> > for the default viewers. > >> > And this would be much harder than custom Servlet Filter in front > > of the > >> > ones by Wicket/RESTeasy > >> > >> If you look at the change in my original post: > >> > >> > > https://github.com/m-creations/isis/commit/ > aa3b16a5cf463466f5abadbcc8cc73 > >> f16857a628 > >> > >> you can see that I added this functionality to the > >> restfulobjects-viewer, so that I can be sure that the full stream is > >> handed to the action without any change. > >> > >> That code is tested against the Isis archetype with this addition to > > the > >> SimpleObjectMenu service: > >> > >> @Action(semantics = SemanticsOf.SAFE) > >> public void upload(HttpServletRequest request, HttpServletResponse > >> response) { > >> FileOutputStream out = null; > >> try { > >> File outFile = new File("/tmp/request"); > >> out = new FileOutputStream(outFile, false); > >> IOUtils.copy(request.getInputStream(), out); > >> out.close(); > >> } catch (IOException e) { > >> if(out != null) { > >> try { > >> out.close(); > >> } catch (IOException e1) { > >> // LOG something > >> } > >> } > >> // throw something > >> } > >> } > >> > >> > >> and the file in /tmp/request is identical to the posted binary file > > which > >> is sent by > >> curl: > >> > >> curl -X POST --data-binary @big-binary-file > >> --header "Authorization: Basic c3ZlbjpwYXNz" \ > >> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \ > >> --header "Content-Type: application/octet-stream" \ > >> http://localhost:8080/restful/services/SimpleObjectMenu/ > >> actions/upload/invoke > >> > >> > >> Cheers > >> > >> > >> Kambiz > >> > >> > > >> >> action with the metadata of the files/request, but it would help > > a lot > >> >> to have such a facility inside Isis instead of writing a separate > >> >> servlet and manually integrating its deployment and the calling > > of > >> >> domain actions etc. > >> >> > >> >> Cheers > >> >> > >> >> > >> >> Kambiz > >> >> > >> >> > >> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan < > >> [email protected]> > >> >> wrote: > >> >> > >> >> > Hi Kambiz, > >> >> > > >> >> > We had a requirement similar to yours. The problem with > > injecting > >> >> > HttpServletRequest to domain services is that it kind of > > violates the > >> >> DDD's > >> >> > hexagonal architecture by letting viewer implementation leak > > into the > >> >> > domain layer. > >> >> > > >> >> > The specific requirement that we had was to be able to > > pre-upload > >> files > >> >> > before submitting an action. We developed a custom blob that > > stores > >> an id > >> >> > instead of the actual data, a service layer that manages a blob > >> storage > >> >> > abstracting and running on top of jclouds (we plan to be able > > to > >> support > >> >> > other types of storage aside from file system), and a file > > service > >> that > >> >> > manages the state of "transient files" (files that were > > pre-uploaded > >> or > >> >> > that are returned from actions as result of export functions, > > etc.) as > >> >> well > >> >> > as means to persist them (i.e. move them to another > >> >> > container/bucket/folder). > >> >> > > >> >> > We then developed a custom servlet that handles download and > > upload > >> and > >> >> > integrate them with isis. This enabled us to implement features > > like > >> >> forced > >> >> > attachment content disposition, or on-demand image resizing. > >> >> > > >> >> > Should you have other ideas or questions, feel free to ask. > >> >> > > >> >> > > >> >> > Best regards, > >> >> > Willie > >> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi > > <[email protected]> > >> >> wrote: > >> >> > > >> >> >> Hi, > >> >> >> > >> >> >> I have to implement a legacy file upload service for a > > customer. > >> Files > >> >> >> of up to some hundred megabytes are uploaded to a service over > > HTTP > >> POST > >> >> >> and a unique ID is returned which can be used to refer to that > > file, > >> add > >> >> >> metadata to it etc. > >> >> >> > >> >> >> Using Blobs is not viable as parsing of the base64 encoded > > binary > >> data > >> >> >> and the allocation of multiple strings during request > > processing > >> >> >> increases both the time and space requirements. > >> >> >> > >> >> >> Ideally, I would be able to create an action on one of my > > domain > >> >> >> services which has a HttpServletRequest argument and take care > > of > >> >> >> reading the binary data and writing them to the file system. > >> >> >> > >> >> >> I have played around with the request processing part of the > >> >> >> restfulobjects viewer/server and managed to get the > > request/response > >> >> >> objects through. > >> >> >> > >> >> >> I know that this is a real edge case, but I think that it is > >> important > >> >> >> to have solutions for edge cases, too, if Isis is to be used > > as a > >> >> >> general purpose platform. > >> >> >> > >> >> >> Could some committers please review the changes and maybe > > propose a > >> >> >> cleaner way of implementing this? > >> >> >> > >> >> >> > >> >> >> https://github.com/m-creations/isis/commit/ > >> >> aa3b16a5cf463466f5abadbcc8cc73f16857a628 > >> >> >> > >> >> >> Thanks > >> >> >> > >> >> >> > >> >> >> Kambiz > >> >> >> > >> >> >> > >> >> > >> >
