Hi Ruwan, Thank you for the suggestion. I couldn't mention in the previous mail, we will be supporting an approach similar to the one you suggested as well for the users who are not willing to sacrifice easiness for performance.
Regards, Samiyuru On Tue, Nov 10, 2015 at 10:38 PM, Ruwan Abeykoon <[email protected]> wrote: > Hi Samiyuru, > I like the event like architecture you are proposing for some of the > applications which has large content upload. Because it will give really > high performance. > +1 on newStream.listen(...) or nice name > > However we need to keep easy programming model too. So how about something > like [1]. We can process the incoming form data and zero copy into temp > file [2] if it is large enough. Provide the wrapped (custom) stream to the > temp file which supports nio and zero copy to the annotated arguments. > Custom finalizer to delete the temp file. > > This way form validation and file upload features can be done easily than > the event driven architecture you are proposing for most of the cases, > unless extremely high performance is needed. Yes, in this we have to parse > the whole HTTP content. But memory and CPU requirements are almost the same > as above proposed method. > > [1] > https://jersey.java.net/nonav/apidocs/1.0.3/contribs/jersey-multipart/com/sun/jersey/multipart/FormDataParam.html > > [2] > http://docs.oracle.com/javase/7/docs/api/java/io/File.html#createTempFile(java.lang.String,%20java.lang.String) > > Cheers, > Ruwan > > On Mon, Nov 9, 2015 at 10:20 PM, Samiyuru Senarathne <[email protected]> > wrote: > >> In MSS, we implemented support for streaming chunked HTTP requests by >> providing the following interface. >> >> HttpStreamHandler >> >> chunk() >> >> finished() >> >> error() >> >> When a resource method implements this interface to handle a chunked >> input: chunk() will be called each time a data chunk is received, >> finished() will be called when all chunks have been received, error() will >> be called when exceptions are thrown while processing a request. >> >> We can use this behaviour in order to handle large request payloads >> without consuming much memory in a zero copy manner. IE. We can write the >> chunks directly to another stream (file output stream etc.) without first >> aggregating the chunks in memory. >> >> But this streaming behaviour falls in short when the payload is not an >> individual entity. IE. When the payload is multipart [1] (Common when html >> forms are submitted with files). In order to parse such payloads either we >> have to have the whole request body in place or we have to parse the HTTP >> chunks while they are streaming to produce meaningful elements. Having the >> whole request body in place works for small payloads but not a solution for >> large ones since they can consume a huge amount of memory. Therefore, for >> large payloads we have to support a way of processing the chunks while >> streaming to get meaningful entities out of them. In this case, If we >> provide only the HttpStreamHandler low level interface to the >> developers, they will have to do the stream processing themselves. Then >> this will be a big burden that creates lot of trouble. >> >> Considering the above facts I suggest to discourage the use of the >> HttpStreamHandler >> interface and provide a high level alternative streaming solution. For >> that, we can introduce a new high level @Context injectable stream >> parameter for our JAX-RS like resource methods. This new stream should be >> able to behind the scenes process the streaming HTTP chunks and provide a >> meaningful object stream. IE. The objects in the stream will be, key value >> pairs for form fields etc. and inputstreams for files etc. Basically this >> stream will be a form of pipe that consumes very low amount of memory in >> terms of the implementation. Developers will be able to conveniently use >> this new stream inside the resource methods without worrying about low >> level details of the requests as shown in the following code fragment. >> >> // Note: newStream is the suggested new stream. We should give it a >> proper name later. >> // newStream should be injected to the resource method using @Context >> >> newStream.listen(elem -> { >> >> if(elem instanceof Field){ >> >> ... >> >> } else if (elem instanceof StreamedFile ){ >> >> … >> >> // for these types of content we can provide another zero copy solution to >> >> // pipe the stream to a file etc. >> >> // eg: elem.stream(outputStream1, outputStream2).whenFinished(()->{}); >> >> } >> >> }); >> >> WDYT? >> >> [1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.2 >> >> >> Regards, >> Samiyuru >> >> -- >> Samiyuru Senarathne >> *Software Engineer* >> Mobile : +94 (0) 71 134 6087 >> [email protected] >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *Ruwan Abeykoon* > *Architect,* > *WSO2, Inc. http://wso2.com <http://wso2.com/> * > *lean.enterprise.middleware.* > > email: [email protected] > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Samiyuru Senarathne *Software Engineer* Mobile : +94 (0) 71 134 6087 [email protected]
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
