Since Beam is all about being able to distribute work, you'll need an HTTP request/response system where the response can be given from any machine. This will most likely require some device/service which holds connections open for you which you can connect back to serve the response. You would need to have a custom source which "streams" requests from this device and a custom sink that "streams" back responses. Beam currently is lacking support for custom unbounded sinks but as always, anything can be done with ParDo even if it shouldn't.
On Tue, Apr 5, 2016 at 8:28 AM, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Bertrand, > > it sounds like Camel route, as you don't really manipulate the data, and > you need service/reply. > > I'm not sure Beam will match > > Anyway, first, we would need an IO exposing a HTTP service, getting the > PCollections (bounded), doing some Fn, the updated PCollections have to be > sent back to the HTTP IO, which is not possible AFAIR. > > Let me know if I can help. IMHO, Camel is more appropriate. > > Regards > JB > > > On 04/05/2016 03:28 PM, Bertrand Delacretaz wrote: > >> Hi, >> >> Besides being an incubation mentor for Beam (which is not relevant to >> this message) I'm a long-time Apache Sling committer and PMC member - >> Sling is a web applications framework [1] where an important task is >> generating web pages dynamically, by assembling and transforming atoms >> of content. Sling is very modular and "malleable", uses OSGi services >> to implement this. >> >> In this context, I'm interested in experimenting with Beam to >> implement an HTTP request -> response pipeline (unless people think >> this is crazy ;-) >> >> I see how that would work for the HTTP key-value pairs: request >> method, path, parameters, headers as input and HTTP status and headers >> as output, shouldn't be a problem. >> >> What I don't see is how to handle the binary request/response streams >> - are there any Beam examples which would put me on the right track >> for accepting a binary stream and returning another one to my HTTP >> clients? >> >> -Bertrand >> >> [1] http://sling.apache.org/ >> >> > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
