Re: Do I have to wrap akka around spark streaming app?

Thanks Vincent for the feedback. I appreciate. On Tue, Nov 29, 2016 at 1:34 AM, vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > You can still achieve it by implementing an actor in each partition but I > am not sure it's a good design regarding scalability because your >

Re: Do I have to wrap akka around spark streaming app?

You can still achieve it by implementing an actor in each partition but I am not sure it's a good design regarding scalability because your distributed actors would send a message for each event to your single app actor, it would be a huge load If you want to experiment this and because actor

Re: Do I have to wrap akka around spark streaming app?

Hello All, I just want to make sure this is a right use case for Kafka --> Spark Streaming Few words about my use case : When the user watches a video, I get the position events from the user that indicates how much they have completed viewing and at a certain point, I mark that Video as

Re: Do I have to wrap akka around spark streaming app?

In this case, persisting to Cassandra is for future analytics and Visualization. I want to notify that the app of the event, so it makes the app interactive. Thanks On Mon, Nov 28, 2016 at 2:24 PM, vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Sorry I don't understand... > Is

Re: Do I have to wrap akka around spark streaming app?

Sorry I don't understand... Is it a cassandra acknowledge to actors that you want ? Why do you want to ack after writing to cassandra ? Your pipeline kafka=>spark=>cassandra is supposed to be exactly once, so you don't need to wait for cassandra ack, you can just write to kafka from actors and

Re: Do I have to wrap akka around spark streaming app?

Thanks Vincent for the input. Not sure I understand your suggestion. Please clarify. Few words about my use case : When the user watches a video, I get the position events from the user that indicates how much they have completed viewing and at a certain point, I mark that Video as complete and

Re: Do I have to wrap akka around spark streaming app?

You don't need actors to do kafka=>spark processing=>kafka Why do you need to notify the akka producer ? If you need to get back the processed message in your producer, then implement an akka consummer in your akka app and kafka offsets will do the job 2016-11-28 21:46 GMT+01:00 shyla deshpande

Re: Do I have to wrap akka around spark streaming app?

Thanks Daniel for the response. I am planning to use Spark streaming to do Event Processing. I will have akka actors sending messages to kafka. I process them using Spark streaming and as a result a new events will be generated. How do I notify the akka actor(Message producer) that a new event

Re: Do I have to wrap akka around spark streaming app?

Well, I would say it depends on what you're trying to achieve. Right now I don't know why you are considering using Akka. Could you please explain your use case a bit? In general, there is no single correct answer to your current question as it's quite broad. Daniel On Mon, Nov 28, 2016 at 9:11

Re: Do I have to wrap akka around spark streaming app?

Anyone with experience of spark streaming in production, appreciate your input. Thanks -shyla On Mon, Nov 28, 2016 at 12:11 AM, shyla deshpande wrote: > My data pipeline is Kafka --> Spark Streaming --> Cassandra. > > Can someone please explain me when would I need to

Do I have to wrap akka around spark streaming app?

My data pipeline is Kafka --> Spark Streaming --> Cassandra. Can someone please explain me when would I need to wrap akka around the spark streaming app. My knowledge of akka and the actor system is poor. Please help! Thanks