Re: inter spark application communication

[Michael Segel]

Hi, 

Putting this back in User email list… 

Ok, 

So its not inter-job communication, but taking the output of Job 1 and using it 
as input to job 2. 
(Chaining of jobs.) 

So you don’t need anything fancy… 


> On Apr 18, 2016, at 1:47 PM, Soumitra Johri  
> wrote:
> 
> Yes , I want to chain spark applications. 
> On Mon, Apr 18, 2016 at 4:46 PM Michael Segel  > wrote:
> Yes, but I’m confused. Are you chaining your spark jobs? So you run job one 
> and its output is the input to job 2? 
> 
>> On Apr 18, 2016, at 1:21 PM, Soumitra Johri > > wrote:
>> 
>> By Akka you mean the Akka Actors ?
>> 
>> I am sending basically sending counts which are further aggregated on the 
>> second  application.
>> 
>> 
>> Warm Regards
>> Soumitra
>> 
>> On Mon, Apr 18, 2016 at 3:54 PM, Michael Segel > > wrote:
>> have you thought about Akka?
>> 
>> What are you trying to send? Why do you want them to talk to one another?
>> 
>> > On Apr 18, 2016, at 12:04 PM, Soumitra Johri > > > wrote:
>> >
>> > Hi,
>> >
>> > I have two applications : App1 and App2.
>> > On a single cluster I have to spawn 5 instances os App1 and 1 instance of 
>> > App2.
>> >
>> > What would be the best way to send data from the 5 App1 instances to the 
>> > single App2 instance ?
>> >
>> > Right now I am using Kafka to send data from one spark application to the 
>> > spark application  but the setup doesn't seem right and I hope there is a 
>> > better way to do this.
>> >
>> > Warm Regards
>> > Soumitra
>> 
>> 
> 

Re: inter spark application communication

[Michael Segel]

have you thought about Akka? 

What are you trying to send? Why do you want them to talk to one another? 

> On Apr 18, 2016, at 12:04 PM, Soumitra Johri  
> wrote:
> 
> Hi,
> 
> I have two applications : App1 and App2. 
> On a single cluster I have to spawn 5 instances os App1 and 1 instance of 
> App2.
> 
> What would be the best way to send data from the 5 App1 instances to the 
> single App2 instance ?
> 
> Right now I am using Kafka to send data from one spark application to the 
> spark application  but the setup doesn't seem right and I hope there is a 
> better way to do this.
> 
> Warm Regards
> Soumitra


-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

inter spark application communication

[Soumitra Johri]

Hi,

I have two applications : App1 and App2.
On a single cluster I have to spawn 5 instances os App1 and 1 instance of
App2.

What would be the best way to send data from the 5 App1 instances to the
single App2 instance ?

Right now I am using Kafka to send data from one spark application to the
spark application  but the setup doesn't seem right and I hope there is a
better way to do this.

Warm Regards
Soumitra