Glad that you worked it out, Johannes & Abhijit, thanks for sharing! > 7 okt 2015 kl. 06:18 skrev Abhijit Sarkar <[email protected]>: > > Thank you Johannes. However, there was a small issue that I was able to solve > myself. To keep the design clean, I wanted the source and the graph to be > generated in separate classes. As I said earlier, basically I wanted to say > "You've a source, I've a sink, let's hook 'em up". Even though your code > sketch doesn't separate them as cleanly as I'd have liked, I was able to take > your idea and convert it to the way I wanted. > For anyone else who might visit this thread through a Google search, I post > the majority of working code below: > > Graph builder: > > val firstSubscriber = Sink.actorSubscriber(RsvpSubscriber.props("first")) > > val secondSubscriber = Sink.actorSubscriber(RsvpSubscriber.props("second")) > > val rsvpFlow: Flow[ByteString, Rsvp, Unit] = Flow[ByteString].map { > import name.abhijitsarkar.scala.meetup.model.RsvpJsonSupport._ > import spray.json._ > > _.utf8String.parseJson.convertTo[Rsvp] > } > > val rsvpSink: Graph[SinkShape[ByteString], Unit] = FlowGraph.partial() { > implicit builder => > import FlowGraph.Implicits._ > > val broadcast = builder.add(Broadcast[Rsvp](2)) > val rsvp = builder.add(rsvpFlow) > > broadcast ~> firstSubscriber > broadcast ~> secondSubscriber > rsvp ~> broadcast.in > > SinkShape(rsvp.inlet) > } > > > > > Source builder and runner: > > > val httpRequest = HttpRequest(uri = baseUri, method = GET) > > val flow = { > val host = httpRequest.uri.authority.host.address() > Http().outgoingConnectionTls(host) > } > val src: Source[ByteString, Unit] = Source.single(httpRequest).via(flow).map { > _.entity.dataBytes > }.flatten(FlattenStrategy.concat) > > src.runWith(sink) > > > > On Tuesday, October 6, 2015 at 12:41:48 AM UTC-7, Johannes Rudolph wrote: > Here is a sketch: > > val dataStream: Future[Source[ByteString]] = > Http().singleRequest(...).map(_.entity.dataBytes) > val parseData: Flow[ByteString, DomainEvent] = ... > val parsedDataStream: Future[Source[DomainEvent]] = > dataStream.map(_.map(parseData)) > > val broadcastSink: SinkShape[DomainEvent] = FlowGraph.partial() { implicit > builder => > import FlowGraph.Implicits._ > val broadcast = builder.add(Broadcast[DomainEvent](2)) > > broadcast ~> sink1 > broadcast ~> sink2 > > SinkShape(broadcast.in <http://broadcast.in/>) > } > > parsedDataStream.onComplete { > case Success(parsedStream) => parsedStream.runWith(broadcastSink) > case Failure(e) => // handleFailure > } > > > On Tuesday, October 6, 2015 at 9:23:23 AM UTC+2, Abhijit Sarkar wrote: > Hi Johannes, > Thank you for your response. Can you show me what do you mean by "put the > broadcast behind the entity.dataByes'? Please pardon me if this is obvious as > I'm just learning. > > I'm using Source.single(httpRequest), not fanoutPub. The example you pointed > me to use fanout, which I intend to study but not apply in this case. > > Regards, > Abhijit > > > On Monday, October 5, 2015 at 11:29:54 PM UTC-7, Johannes Rudolph wrote: > Hi Abhijit, > > from a quick glance to your code it seems that you are broadcasting an > HttpResponse and then access its `entity.dataBytes` in several branches. This > is not supported. You need to put the broadcast behind the `entity.dataBytes` > and not behind the stream of responses. > > We prepared a similar example for scala.world but didn't get to show the > fanoutPublisher in the end. There's still in an early version of it in the > example repository: > > https://github.com/jrudolph/scala-world-2015/blob/43e6a4e50c68fdf22e70e0137d6bb54e9614e808/backend/src/main/scala/example/repoanalyzer/Webservice.scala > > <https://github.com/jrudolph/scala-world-2015/blob/43e6a4e50c68fdf22e70e0137d6bb54e9614e808/backend/src/main/scala/example/repoanalyzer/Webservice.scala> > > I'd suggest using `Http.singleRequest` for creating the single request to > meetup to simplify things. If you use fanoutPublisher you need to be aware of > its several potential traps: > > * it backpressures over all its subscribers, if one is slow or stalled, this > will stop all the others as well at some point > * this means you may want to add a `.buffer` directly behind each branch to > relieve the fanoutPublisher from backpressure and add a OverflowStrategy that > suits your needs > * OverflowStrategies may not be what you want, so you may want to use > `conflate`, instead, (which is somewhat harder to use) to implement custom > overflow strategies > * the fanoutPublisher shuts down if the last subscriber cancels its > subscription, to make it persistent you need to attach a dummy sink to it to > keep it alive > > HTH > Johannes > > On Tuesday, October 6, 2015 at 6:21:52 AM UTC+2, Abhijit Sarkar wrote: > Hi Victor, > > The reason I didn't want to totally do away with streaming and create some > local source was that the point of my question would be lost. In an effort to > simplify, I might end up with a working but completely unrelated and somewhat > trivial example, which is IMHO the issue with the examples on most blogs. > > So I converted to Twitter streaming example to a minimal Meetup streaming > example. The later doesn't require any auth and hence is a similar but > simpler representation of what I'm after. And guess what, I was able to > reproduce the issue again. That project is attached (meetup-streaming.zip). > All you need to do is download it and execute the class > name.abhijitsarkar.scala.meetup.MeetupStreamingApp. > > Just for kicks, I did also run an example without using a streaming HTTP > source and I didn't get the IllegalStateException. That's precisely the > reason I didn't want to deviate too far from my actual use case. > > Thanks for your time. If you need anything else from me, please let me know. > > BTW, in your sample code below allTweets ~> broadcast doesn't compile. > They're different types. > > Regards, > Abhijit > > On Monday, October 5, 2015 at 1:44:43 AM UTC-7, √ wrote: > Hi Abhijit, > > I think it would be much more helpful with a *minimized* reproducer. > Have you tried something similar to: > > val partial = FlowGraph.partial() { implicit builder => > import FlowGraph.Implicits._ > val allTweets = Flow[HttpResponse].map { _.entity.dataBytes > }.flatten(FlattenStrategy.concat).map { > b => parseTweet(b.utf8String) > } > > val isAfterEpoch = (t: Tweet) => t.createdAt.getYear > 1970 > val broadcast = builder.add(Broadcast[HttpResponse](2)) > > allTweets ~> broadcast > broadcast ~> Flow[Tweet].filter(isAfterEpoch) ~> > Sink.actorSubscriber(TwitterSubscriber.props("good")) > broadcast ~> Flow[Tweet].filter(!isAfterEpoch(_)) ~> > Sink.actorSubscriber(TwitterSubscriber.props("bad")) > > SinkShape(broadcast.in <http://broadcast.in/>) > } > > On Mon, Oct 5, 2015 at 10:28 AM, Abhijit Sarkar <[email protected] <>> > wrote: > I've attached the maven project for picture 1. If you run the class > name.abhijitsarkar.scala.scauth.TwitterStreamingApp, you'll get an > IllegalStateException as I've posted before. There's no code for picture 2 as > I couldn't figure out how to attach multiple subscribers to a fanout > publisher (as shown by ? in the pic). > > The only problem is that in order to connect to Twitter and get data, you > need OAuth credentials. Without those, you can't actually execute the program > (well, technically you can but Twitter will return auth error). > > On Monday, October 5, 2015 at 12:55:27 AM UTC-7, √ wrote: > Hi, > > both of those should work, so I'm looking fwd to the code. > > -- > Cheers, > √ > > On 5 Oct 2015 09:46, "Abhijit Sarkar" <[email protected] <>> wrote: > > <https://lh3.googleusercontent.com/-oPEGfrczTSM/VhIqepO9W4I/AAAAAAAACUA/3Y4OuViNZ6g/s1600/1.jpg> > > <https://lh3.googleusercontent.com/-sDLrEeNF3qo/VhIpfhXCCMI/AAAAAAAACT0/TCi96zVz09I/s1600/2.jpg> > > > I read that > <http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-graphs.html>. > In fact, I always change the version in the URL to "current" before reading > any Akka Streams doc as I don't want to waste time reading outdated ones. > > I've attached pictures showing what I'm trying to do. The gray bounding boxes > enclosing the graphs show that those are generated elsewhere and then > attached to the source later. > > > > On Monday, October 5, 2015 at 12:07:45 AM UTC-7, rkuhn wrote: > Hi Abhijit, > > 5 okt 2015 kl. 08:57 skrev Abhijit Sarkar <[email protected] <>>: > > Hi, > What Victor gave me, and I myself came up with, doesn't work. If I attach two > subscribers to the stream, it throws an IllegalStateException with a > reference to reactive streams spec. > > Which “stream” are we talking about here? There definitely is a disconnect > between what we are describing and I cannot help you unless you show what is > failing. > > I suppose I could materialize the source as a publisher and then attach > multiple subscribers to it but that's more than what I'm looking to do at > this point. > > It helps to have actual code examples for non-trivial scenarios. > Documentation is a sore spot for almost every open source but it is a big > pain point with Akka Streams. > > I do know that there are certain things to be improved about the Streams > documentation, but from this comment I wonder whether you are another victim > of Google weirdness: please make sure to look at version 1.0 of the docs (and > not 1.0-M2), there are dozens of pages of non-trivial examples—which is > precisely the problem right now (i.e. we lack the absolutely beginner > section). > > In particular the questions you ask should be discussed here: > http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-graphs.html > > <http://www.google.com/url?q=http%3A%2F%2Fdoc.akka.io%2Fdocs%2Fakka-stream-and-http-experimental%2F1.0%2Fscala%2Fstream-graphs.html&sa=D&sntz=1&usg=AFQjCNHv8TTpCFds9Q-iHtFPqCHnjAdbNg> > > Regards, > > Roland > > > Regards, > Abhijit Sarkar > > On Sunday, October 4, 2015 at 11:45:22 PM UTC-7, rkuhn wrote: > Hi Abhijit, > > fanoutPublisher is something that you’d need when the downstream subscribers > are not yet known at the point where you need to materialize the upstream > graph, which is to say that in the problem you describe you won’t need it. > What Viktor gave you is already a Sink, your program is finished there. > Wherever you have your Source you say “source.runWith(sink)” and that’s it. > > It helps to draw these stream topologies on paper or on a whiteboard, the > whole concept is very much graphical and the perfect metaphor consists of > water and pipes and flanges. A Source is a tap and a Sink is a drain. > > Regards, > > Roland > > 5 okt 2015 kl. 00:01 skrev Abhijit Sarkar <[email protected] <>>: > > > > java.lang.IllegalStateException: Substream publisher only supports one > subscriber (which is allowed, see reactive-streams specification, rule 1.12) > > > > On Sunday, October 4, 2015 at 2:31:52 PM UTC-7, Abhijit Sarkar wrote: > Without the pub (fanout), it complains that a publisher can't handle more > than one subscribers. > > On Sunday, October 4, 2015 at 2:00:47 PM UTC-7, √ wrote: > Why `pub` at all? > > a ~> c should be enough, and then you don't need close() at all, > > `src to partial` should be all you need. (Source + Sink) > > On Sun, Oct 4, 2015 at 10:29 PM, Abhijit Sarkar <[email protected] <>> > wrote: > Thank you, that's exactly what I'd come up with (feels good to know that I'm > starting to get it :) ). So now I've a Source, a publisher and a partial. I'm > trying to hook them up as follows but it doesn't compile. > > What I'd like to do is to attach the publisher with the partial and then hand > the whole thing to the source. I've not been able to find a way to do that. > Basically my idea is to say that "you've a source, I'll give you a SinkShape, > go run it." Instead, now the class with the source is having to construct the > publisher, and as I said, that even doesn't compile. > > val pub = Sink.fanoutPublisher[Tweet](1, 1) > > val src = Source.single(httpRequest).via(flow) > > FlowGraph.closed() { implicit builder => > > import FlowGraph.Implicits._ > > val a = builder.add(src) > > val b = builder.add(pub) > > val c = builder.add(partial) > > a ~> pub ~> c // compiler isn't happy here! > } > > > > On Sunday, October 4, 2015 at 1:14:13 PM UTC-7, √ wrote: > val partial = FlowGraph.partial() { implicit builder => > import FlowGraph.Implicits._ > val broadcast = builder.add(Broadcast[HttpResponse](2)) > > broadcast ~> goodTweets > broadcast ~> badTweets > > SinkShape(broadcast.in) > }.named("partial") > > On Sun, Oct 4, 2015 at 8:30 PM, Abhijit Sarkar <[email protected] <>> > wrote: > I'm afraid it doesn't help much without code example. I've only one inlet but > I'm also not sure how to hook up the fanout publisher with the 2 sinks. Based > on the above code, can you show me what you're suggesting? > > On Sunday, October 4, 2015 at 3:59:33 AM UTC-7, √ wrote: > Hi! > > Since what you want only has one inlet, that is what a Sink is, which means > you should use SinkShape iso FlowShape. > Does that help? > > On Sun, Oct 4, 2015 at 12:03 PM, Abhijit Sarkar <[email protected] <>> > wrote: > I'm cutting my teeth on Akka streams, building the quintessential Twitter > streaming client. What I'm trying to do is some processing on the tweet > stream and then based on the result, divide the stream into "good" and "bad" > streams. Both streams are directed to the same ActorPublisher class > instantiated with different names. Following is my code so far but I'm at a > loss connecting all the components. Please help. > > val s = Sink.fanoutPublisher[Tweet](1, 1) > > val goodSink = > Sink.actorSubscriber(TwitterSubscriber.props("GoodTwitterSubscriber")) > val badSink = > Sink.actorSubscriber(TwitterSubscriber.props("BadTwitterSubscriber")) > > val allTweets = Flow[HttpResponse].map { _.entity.dataBytes > }.flatten(FlattenStrategy.concat).map { > b => parseTweet(b.utf8String) > } > > val afterEpoch = (t: Tweet) => t.createdAt.getYear > 1970 > > val goodTweets: Sink[HttpResponse, Unit] = allTweets.splitWhen { afterEpoch > }.to(goodSink) > val badTweets: Sink[HttpResponse, Unit] = allTweets.splitWhen { > !afterEpoch(_)}.to(badSink) > > // I think this is wrong! > val partial = FlowGraph.partial() { implicit builder => > import FlowGraph.Implicits._ > val broadcast = builder.add(Broadcast[HttpResponse](2)) > > broadcast ~> goodTweets > broadcast ~> badTweets > > // I don't need an outlet, only an inlet, but partial needs an outlet > val outlet = builder.add(Source.empty) > FlowShape(broadcast.in, outlet) > }.named("partial") > > > In another class: > > val httpRequest = this.httpRequest { queryParams(follow, track) } > > val flow = this.flow { httpRequest } > > val src = Source.single(httpRequest).via(flow) > > src.via(partial).to(Sink.ignore) > > > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ <http://akka.io/docs/> > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > >>>>>>>>>> <https://groups.google.com/group/akka-user> > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <>. > To post to this group, send email to [email protected] <>. > Visit this group at http://groups.google.com/group/akka-user > <http://groups.google.com/group/akka-user>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > > -- > Cheers, > √ > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ <http://akka.io/docs/> > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > >>>>>>>>>> <https://groups.google.com/group/akka-user> > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <>. > To post to this group, send email to [email protected] <>. > Visit this group at http://groups.google.com/group/akka-user > <http://groups.google.com/group/akka-user>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > > -- > Cheers, > √ > > -- > >>>>>>>>>> Read the docs: > ... > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ <http://akka.io/docs/> > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> <http://doc.akka.io/docs/akka/current/additional/faq.html> > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > >>>>>>>>>> <https://groups.google.com/group/akka-user> > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at http://groups.google.com/group/akka-user > <http://groups.google.com/group/akka-user>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>.
Dr. Roland Kuhn Akka Tech Lead Typesafe <http://typesafe.com/> – Reactive apps on the JVM. twitter: @rolandkuhn <http://twitter.com/#!/rolandkuhn> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
