Re: WorkFlow Processing - Spark

2015-06-24 Thread ayan guha
What kind of custom logic? On 25 Jun 2015 01:33, Ashish Soni asoni.le...@gmail.com wrote: Hi All , We are looking to use spark as our stream processing framework and it would be helpful if experts can weigh if we made a right choice given below requirement Given a stream of data we need to

Re: WorkFlow Processing - Spark

2015-06-24 Thread asoni . learn
Any custom script ( python or java or scala ) Thanks , Ashish On Jun 24, 2015, at 4:39 PM, ayan guha guha.a...@gmail.com wrote: What kind of custom logic? On 25 Jun 2015 01:33, Ashish Soni asoni.le...@gmail.com wrote: Hi All , We are looking to use spark as our stream processing

Re: WorkFlow Processing - Spark

2015-06-24 Thread ayan guha
As long as the logic can be run in parallel, yes. You should not however load any logic in driver. All logic should run in executors. On 25 Jun 2015 07:58, asoni.le...@gmail.com wrote: Any custom script ( python or java or scala ) Thanks , Ashish On Jun 24, 2015, at 4:39 PM, ayan guha

WorkFlow Processing - Spark

2015-06-24 Thread Ashish Soni
Hi All , We are looking to use spark as our stream processing framework and it would be helpful if experts can weigh if we made a right choice given below requirement Given a stream of data we need to take those event to multiple stage ( pipeline processing ) and in those stage customer will