You can still achieve it by implementing an actor in each partition but I am not sure it's a good design regarding scalability because your distributed actors would send a message for each event to your single app actor, it would be a huge load.... If you want to experiment this and because actor is thread safe you can use the following pattern which allows to reuse actors between micro batches in each partitions http://allegro.tech/2015/08/spark-kafka-integration.html
2016-11-29 2:18 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: > Hello All, > > I just want to make sure this is a right use case for Kafka --> Spark > Streaming > > Few words about my use case : > > When the user watches a video, I get the position events from the user > that indicates how much they have completed viewing and at a certain point, > I mark that Video as complete and that triggers a lot of other events. I > need a way to notify the app about the creation of the completion event. > > Appreciate any suggestions. > > Thanks > > > On Mon, Nov 28, 2016 at 2:35 PM, shyla deshpande <deshpandesh...@gmail.com > > wrote: > >> In this case, persisting to Cassandra is for future analytics and >> Visualization. >> >> I want to notify that the app of the event, so it makes the app >> interactive. >> >> Thanks >> >> On Mon, Nov 28, 2016 at 2:24 PM, vincent gromakowski < >> vincent.gromakow...@gmail.com> wrote: >> >>> Sorry I don't understand... >>> Is it a cassandra acknowledge to actors that you want ? Why do you want >>> to ack after writing to cassandra ? Your pipeline kafka=>spark=>cassandra >>> is supposed to be exactly once, so you don't need to wait for cassandra >>> ack, you can just write to kafka from actors and then notify the user ? >>> >>> 2016-11-28 23:15 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: >>> >>>> Thanks Vincent for the input. Not sure I understand your suggestion. >>>> Please clarify. >>>> >>>> Few words about my use case : >>>> When the user watches a video, I get the position events from the user >>>> that indicates how much they have completed viewing and at a certain point, >>>> I mark that Video as complete and persist that info to cassandra. >>>> >>>> How do I notify the user that it was marked complete? >>>> >>>> Are you suggesting I write the completed events to kafka(different >>>> topic) and the akka consumer could read from this? There could be many >>>> completed events from different users in this topic. So the akka consumer >>>> should pretty much do what a spark streaming does to process this without >>>> the knowledge of the kafka offset. >>>> >>>> So not sure what you mean by kafka offsets will do the job, how will >>>> the akka consumer know the kafka offset? >>>> >>>> On Mon, Nov 28, 2016 at 12:52 PM, vincent gromakowski < >>>> vincent.gromakow...@gmail.com> wrote: >>>> >>>>> You don't need actors to do kafka=>spark processing=>kafka >>>>> Why do you need to notify the akka producer ? If you need to get back >>>>> the processed message in your producer, then implement an akka consummer >>>>> in >>>>> your akka app and kafka offsets will do the job >>>>> >>>>> 2016-11-28 21:46 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: >>>>> >>>>>> Thanks Daniel for the response. >>>>>> >>>>>> I am planning to use Spark streaming to do Event Processing. I will >>>>>> have akka actors sending messages to kafka. I process them using Spark >>>>>> streaming and as a result a new events will be generated. How do I notify >>>>>> the akka actor(Message producer) that a new event has been generated? >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Nov 28, 2016 at 9:51 AM, Daniel van der Ende < >>>>>> daniel.vandere...@gmail.com> wrote: >>>>>> >>>>>>> Well, I would say it depends on what you're trying to achieve. Right >>>>>>> now I don't know why you are considering using Akka. Could you please >>>>>>> explain your use case a bit? >>>>>>> >>>>>>> In general, there is no single correct answer to your current >>>>>>> question as it's quite broad. >>>>>>> >>>>>>> Daniel >>>>>>> >>>>>>> On Mon, Nov 28, 2016 at 9:11 AM, shyla deshpande < >>>>>>> deshpandesh...@gmail.com> wrote: >>>>>>> >>>>>>>> My data pipeline is Kafka --> Spark Streaming --> Cassandra. >>>>>>>> >>>>>>>> Can someone please explain me when would I need to wrap akka around >>>>>>>> the spark streaming app. My knowledge of akka and the actor system is >>>>>>>> poor. >>>>>>>> Please help! >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Daniel >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
