Hi Abhijit,
I think it would be much more helpful with a *minimized* reproducer.
Have you tried something similar to:
val partial = FlowGraph.partial() { implicit builder =>
import FlowGraph.Implicits._
val allTweets = Flow[HttpResponse].map { _.entity.dataBytes
}.flatten(FlattenStrategy.concat).map {
b => parseTweet(b.utf8String)
}
val isAfterEpoch = (t: Tweet) => t.createdAt.getYear > 1970
val broadcast = builder.add(Broadcast[HttpResponse](2))
allTweets ~> broadcast
broadcast ~> Flow[Tweet].filter(isAfterEpoch) ~>
Sink.actorSubscriber(TwitterSubscriber.props("good"))
broadcast ~> Flow[Tweet].filter(!isAfterEpoch(_)) ~>
Sink.actorSubscriber(TwitterSubscriber.props("bad"))
SinkShape(broadcast.in)
}
On Mon, Oct 5, 2015 at 10:28 AM, Abhijit Sarkar <[email protected]>
wrote:
> I've attached the maven project for picture 1. If you run the class
> name.abhijitsarkar.scala.scauth.TwitterStreamingApp, you'll get an
> IllegalStateException as I've posted before. There's no code for picture 2
> as I couldn't figure out how to attach multiple subscribers to a fanout
> publisher (as shown by ? in the pic).
>
> The only problem is that in order to connect to Twitter and get data, you
> need OAuth credentials. Without those, you can't actually execute the
> program (well, technically you can but Twitter will return auth error).
>
> On Monday, October 5, 2015 at 12:55:27 AM UTC-7, √ wrote:
>>
>> Hi,
>>
>> both of those should work, so I'm looking fwd to the code.
>>
>> --
>> Cheers,
>> √
>> On 5 Oct 2015 09:46, "Abhijit Sarkar" <[email protected]> wrote:
>>
>>>
>>> <https://lh3.googleusercontent.com/-oPEGfrczTSM/VhIqepO9W4I/AAAAAAAACUA/3Y4OuViNZ6g/s1600/1.jpg>
>>>
>>>
>>> <https://lh3.googleusercontent.com/-sDLrEeNF3qo/VhIpfhXCCMI/AAAAAAAACT0/TCi96zVz09I/s1600/2.jpg>
>>>
>>> I read that
>>> <http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-graphs.html>.
>>> In fact, I always change the version in the URL to "current" before reading
>>> any Akka Streams doc as I don't want to waste time reading outdated ones.
>>>
>>> I've attached pictures showing what I'm trying to do. The gray bounding
>>> boxes enclosing the graphs show that those are generated elsewhere and then
>>> attached to the source later.
>>>
>>>
>>>
>>> On Monday, October 5, 2015 at 12:07:45 AM UTC-7, rkuhn wrote:
>>>>
>>>> Hi Abhijit,
>>>>
>>>> 5 okt 2015 kl. 08:57 skrev Abhijit Sarkar <[email protected]>:
>>>>
>>>> Hi,
>>>> What Victor gave me, and I myself came up with, doesn't work. If I
>>>> attach two subscribers to the stream, it throws an IllegalStateException
>>>> with a reference to reactive streams spec.
>>>>
>>>>
>>>> Which “stream” are we talking about here? There definitely is a
>>>> disconnect between what we are describing and I cannot help you unless you
>>>> show what is failing.
>>>>
>>>> I suppose I could materialize the source as a publisher and then attach
>>>> multiple subscribers to it but that's more than what I'm looking to do at
>>>> this point.
>>>>
>>>> It helps to have actual code examples for non-trivial scenarios.
>>>> Documentation is a sore spot for almost every open source but it is a big
>>>> pain point with Akka Streams.
>>>>
>>>>
>>>> I do know that there are certain things to be improved about the
>>>> Streams documentation, but from this comment I wonder whether you are
>>>> another victim of Google weirdness: please make sure to look at version 1.0
>>>> of the docs (and not 1.0-M2), there are dozens of pages of non-trivial
>>>> examples—which is precisely the problem right now (i.e. we lack the
>>>> absolutely beginner section).
>>>>
>>>> In particular the questions you ask should be discussed here:
>>>> http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-graphs.html
>>>> <http://www.google.com/url?q=http%3A%2F%2Fdoc.akka.io%2Fdocs%2Fakka-stream-and-http-experimental%2F1.0%2Fscala%2Fstream-graphs.html&sa=D&sntz=1&usg=AFQjCNHv8TTpCFds9Q-iHtFPqCHnjAdbNg>
>>>>
>>>> Regards,
>>>>
>>>> Roland
>>>>
>>>>
>>>> Regards,
>>>> Abhijit Sarkar
>>>>
>>>> On Sunday, October 4, 2015 at 11:45:22 PM UTC-7, rkuhn wrote:
>>>>
>>>> Hi Abhijit,
>>>>
>>>> fanoutPublisher is something that you’d need when the downstream
>>>> subscribers are not yet known at the point where you need to materialize
>>>> the upstream graph, which is to say that in the problem you describe you
>>>> won’t need it. What Viktor gave you is already a Sink, your program is
>>>> finished there. Wherever you have your Source you say
>>>> “source.runWith(sink)” and that’s it.
>>>>
>>>> It helps to draw these stream topologies on paper or on a whiteboard,
>>>> the whole concept is very much graphical and the perfect metaphor consists
>>>> of water and pipes and flanges. A Source is a tap and a Sink is a drain.
>>>>
>>>> Regards,
>>>>
>>>> Roland
>>>>
>>>> 5 okt 2015 kl. 00:01 skrev Abhijit Sarkar <[email protected]>:
>>>>
>>>>
>>>>
>>>> java.lang.IllegalStateException: Substream publisher only supports one
>>>> subscriber (which is allowed, see reactive-streams specification, rule
>>>> 1.12)
>>>>
>>>>
>>>>
>>>> On Sunday, October 4, 2015 at 2:31:52 PM UTC-7, Abhijit Sarkar wrote:
>>>>
>>>> Without the pub (fanout), it complains that a publisher can't handle
>>>> more than one subscribers.
>>>>
>>>> On Sunday, October 4, 2015 at 2:00:47 PM UTC-7, √ wrote:
>>>>
>>>> Why `pub` at all?
>>>>
>>>> a ~> c should be enough, and then you don't need close() at all,
>>>>
>>>> `src to partial` should be all you need. (Source + Sink)
>>>>
>>>> On Sun, Oct 4, 2015 at 10:29 PM, Abhijit Sarkar <[email protected]>
>>>> wrote:
>>>>
>>>> Thank you, that's exactly what I'd come up with (feels good to know
>>>> that I'm starting to get it :) ). So now I've a Source, a publisher and a
>>>> partial. I'm trying to hook them up as follows but it doesn't compile.
>>>>
>>>> What I'd like to do is to attach the publisher with the partial and
>>>> then hand the whole thing to the source. I've not been able to find a way
>>>> to do that. Basically my idea is to say that "you've a source, I'll give
>>>> you a SinkShape, go run it." Instead, now the class with the source is
>>>> having to construct the publisher, and as I said, that even doesn't
>>>> compile.
>>>>
>>>> val pub = Sink.fanoutPublisher[Tweet](1, 1)
>>>>
>>>> val src = Source.single(httpRequest).via(flow)
>>>>
>>>> FlowGraph.closed() { implicit builder =>
>>>>
>>>> import FlowGraph.Implicits._
>>>>
>>>> val a = builder.add(src)
>>>>
>>>> val b = builder.add(pub)
>>>>
>>>> val c = builder.add(partial)
>>>>
>>>> a ~> pub ~> c // compiler isn't happy here!
>>>> }
>>>>
>>>>
>>>>
>>>> On Sunday, October 4, 2015 at 1:14:13 PM UTC-7, √ wrote:
>>>>
>>>> val partial = FlowGraph.partial() { implicit builder =>
>>>> import FlowGraph.Implicits._
>>>> val broadcast = builder.add(Broadcast[HttpResponse](2))
>>>>
>>>> broadcast ~> goodTweets
>>>> broadcast ~> badTweets
>>>>
>>>> SinkShape(broadcast.in)
>>>> }.named("partial")
>>>>
>>>> On Sun, Oct 4, 2015 at 8:30 PM, Abhijit Sarkar <[email protected]>
>>>> wrote:
>>>>
>>>> I'm afraid it doesn't help much without code example. I've only one
>>>> inlet but I'm also not sure how to hook up the fanout publisher with the 2
>>>> sinks. Based on the above code, can you show me what you're suggesting?
>>>>
>>>> On Sunday, October 4, 2015 at 3:59:33 AM UTC-7, √ wrote:
>>>>
>>>> Hi!
>>>>
>>>> Since what you want only has one inlet, that is what a Sink is, which
>>>> means you should use SinkShape iso FlowShape.
>>>> Does that help?
>>>>
>>>> On Sun, Oct 4, 2015 at 12:03 PM, Abhijit Sarkar <[email protected]>
>>>> wrote:
>>>>
>>>> I'm cutting my teeth on Akka streams, building the quintessential
>>>> Twitter streaming client. What I'm trying to do is some processing on the
>>>> tweet stream and then based on the result, divide the stream into "good"
>>>> and "bad" streams. Both streams are directed to the same ActorPublisher
>>>> class instantiated with different names. Following is my code so far but
>>>> I'm at a loss connecting all the components. Please help.
>>>>
>>>> val s = Sink.fanoutPublisher[Tweet](1, 1)
>>>>
>>>> val goodSink = Sink.actorSubscriber(TwitterSubscriber.props(
>>>> "GoodTwitterSubscriber"))
>>>> val badSink = Sink.actorSubscriber(TwitterSubscriber.props(
>>>> "BadTwitterSubscriber"))
>>>>
>>>> val allTweets = Flow[HttpResponse].map { _.entity.dataBytes }.flatten(
>>>> FlattenStrategy.concat).map {
>>>> b => parseTweet(b.utf8String)
>>>> }
>>>>
>>>> val afterEpoch = (t: Tweet) => t.createdAt.getYear > 1970
>>>>
>>>> val goodTweets: Sink[HttpResponse, Unit] = allTweets.splitWhen {
>>>> afterEpoch }.to(goodSink)
>>>> val badTweets: Sink[HttpResponse, Unit] = allTweets.splitWhen { !
>>>> afterEpoch(_)}.to(badSink)
>>>>
>>>> // I think this is wrong!
>>>> val partial = FlowGraph.partial() { implicit builder =>
>>>> import FlowGraph.Implicits._
>>>> val broadcast = builder.add(Broadcast[HttpResponse](2))
>>>>
>>>> broadcast ~> goodTweets
>>>> broadcast ~> badTweets
>>>>
>>>> // I don't need an outlet, only an inlet, but partial needs an outlet
>>>> val outlet = builder.add(Source.empty)
>>>> FlowShape(broadcast.in, outlet)
>>>> }.named("partial")
>>>>
>>>>
>>>> In another class:
>>>>
>>>> val httpRequest = this.httpRequest { queryParams(follow, track) }
>>>>
>>>> val flow = this.flow { httpRequest }
>>>>
>>>> val src = Source.single(httpRequest).via(flow)
>>>>
>>>> src.via(partial).to(Sink.ignore)
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> √
>>>>
>>>>
>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> √
>>>>
>>>>
>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> √
>>>>
>>>>
>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> ...
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
Cheers,
√
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
