One thought is to use DistributedData <http://doc.akka.io/docs/akka/2.4.2/java/distributed-data.html> for the active hashtags, and have something listening on that an spawn/stop publishers on demand. There are limitations of how much data you can have in DistributedData, but you can be creative with the keyspace to get rather far.
/Patrik On Fri, Feb 26, 2016 at 12:59 PM, Akka Team <[email protected]> wrote: > Hi Guy, > > In general, this is a hard (or even impossible, depending on the > guarantees) problem. In many times though you can get away with good-enough > solutions. > > One possibility is to use a topic where Publishers can announce that they > go away. Since PubSub is best effort, sending one such announcement is > risky. What you can still do though is to keep the publisher alive for, > let's say 5 minutes longer, and repeat the announcement every minute. While > this does not guarantee that every interested party gets the announcement, > it does not need complicated logic and works rather well. > > On the subscriber side, you can also defend from problematic situations > (publisher is gone but subscriber did not receive an announcement). For > example, if there were no arrivals from the publisher for a while, it > should try to ping the publisher to decide whether it is still alive just > has nothing to say, or dead. Again, repeating the ping for a while is > necessary. Alternatively, you can have the publisher from time to time > publish a heartbeat. If a subscriber does not get a heartbeat for 4-5 > periods, it can consider the publisher to be dead. > > The combination of the above two approaches guarantee that if the > publisher goes away, eventually all subscribers notice it (explicit > announcement, or no reply to several pings after silence, or no heartbeat > for a while). It does not guarantee however the absence of false positives, > i.e. subscribers might think that the publisher is gone because of a > network split. At that point though the failure detector of Akka Cluster > should kick in and some ops need go remove unreachable nodes and resolve > the split (or use the commercial RP build which has a built-in split brain > resolver, SBR). > > -Endre > > On Tue, Feb 23, 2016 at 2:40 PM, Guy Dawson <[email protected]> wrote: > >> Hi, >> >> I posted the following on SO, but it hasn't got much love. I'm reposting >> it here in the hopes someone will be able to answer the question: >> >> Using DistributedPubSub, say an actor can subscribe to a publisher which >> publishes tweets with a certain hashtag. Obviously publishers would have to >> be created on demand of subscribers: you can't have a publisher for every >> possible hashtag. Similarly, you want to be able to shutdown publishers who >> no longer have any subscribers: you don't want the number of publishers to >> grow indefinitely. Which brings us onto my question: *in a distributed >> system, how do you determine whether a publisher no longer has any >> subscribers (so you can shut it down)*? >> >> Cheers, >> >> Guy >> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Akka Team > Typesafe - Reactive apps on the JVM > Blog: letitcrash.com > Twitter: @akkateam > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM Twitter: @patriknw [image: Lightbend] <http://www.lightbend.com/> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
