> This should be absolutely doable without a change in the runtimes at all, as > they "shouldn't" (to be verified) need a change to support TCP streaming.
I think some of the runtimes are currently reading whole request in memory. For e.g. in Java [1] it constructs a json instance. So would need some logic to stream the content to filesystem (possiby via multipart support). For nodejs some sort of multi part handling or stream handling would be required. Chetan Mehrotra [1] https://github.com/apache/incubator-openwhisk-runtime-java/blob/master/core/javaAction/proxy/src/main/java/openwhisk/java/action/Proxy.java#L80 On Thu, Mar 29, 2018 at 4:40 PM, Markus Thoemmes <[email protected]> wrote: > Heya, > >>> So for #C is there a way to proxy the stream without changing the runtime >>> http api where binary is passed embedded in the json object on /init? > >> They would need to be modified. A less intrusive approach would have >> been to pass signed url and use client like wget to fetch the binary. >> But as David mentioned some setups may lock down access to external >> services. For such cases we would need to implement the support in >> client runtimes as part of init protocol > >> Chetan Mehrotra > > > I believe they don't need to be changed at all. TCP as the transport protocol > is streamed by default (stuff is broken down into smallish packets). Say for > example our payload we need to pass is: {"main": "foo", "code": "Some > veeeeery long string here"}. Let's also assume our attachement is available > as Source[ByteString]. What we can do, is generate the first bit until the > part we *need* to stream, in this case: {"main": "foo", "code": ". Now we > consume the chunks of the attachement Source and put them on the downstream. > In the end, we put an element like "} on the stream, to close the JSON object. > > Pseudocode: Source.single(ByteString("""{"main": "foo", > "code":""")).concat(attachementSource).to(httpConnectionToContainer).run() > > This should be absolutely doable without a change in the runtimes at all, as > they "shouldn't" (to be verified) need a change to support TCP streaming. > > Does that make sense? > > Cheers, > Markus >
