The Spark Job that has the main DStream, could have another DStream that is listening for "stream subscription" requests. So when a subscription is received, you could do a filter/forEach on the main DStream and respond to that one request. So you're basically creating a stream server that is capable of accepting "filter requests".
On Mon, Aug 18, 2014 at 10:30 AM, Levi Bowman <levi.bow...@markit.com> wrote: > Based on my understanding something like this doesn’t seem to be > possible out of the box, but I thought I would write it up anyway in case > someone has any ideas. > > > > We have conceptually one high volume input stream, each streaming job is > either interested in a subset of the stream or the entire stream. We would > like to get to the point where we could be running a large number of > streaming jobs concurrently across a cluster. It does not seem that > putting a buffer like Kafka in between the source and the streaming jobs > would be a sustainable route as all jobs would be consuming the whole > stream and the majority of them would only be interested in a small subset > of the available data. As we don’t know exactly what data each job would > be interested in up front it would be difficult to separate/partition Kafka > topics up front. > > > > What I think we want is a way to have one streaming job whose output is > passed to n other streaming jobs. Has anyone though about implementing > something like this? I don’t see a way to have the Receiver for one > streaming job be the output of another streaming job. > > > > > > ------------------------------ > > This e-mail, including accompanying communications and attachments, is > strictly confidential and only for the intended recipient. Any retention, > use or disclosure not expressly authorised by Markit is prohibited. This > email is subject to all waivers and other terms at the following link: > http://www.markit.com/en/about/legal/email-disclaimer.page > > Please visit http://www.markit.com/en/about/contact/contact-us.page? for > contact information on our offices worldwide. >
