Re: Creating topology in spark streaming

sm without a Central Scheduler > > > > *From:* Juan Rodríguez Hortalá [mailto:juan.rodriguez.hort...@gmail.com] > *Sent:* Wednesday, May 6, 2015 11:20 AM > *To:* Evo Eftimov > *Cc:* anshu shukla; ayan guha; user@spark.apache.org > > *Subject:* Re: Creating topology in spark streaming > > >

RE: Creating topology in spark streaming

To: Evo Eftimov Cc: anshu shukla; ayan guha; user@spark.apache.org Subject: Re: Creating topology in spark streaming Hi, I agree with Evo, Spark works at a different abstraction level than Storm, and there is not a direct translation from Storm topologies to Spark Streaming jobs. I think

Re: Creating topology in spark streaming

ectly > relates to the corresponding number of Threads > > > > *From:* anshu shukla [mailto:anshushuk...@gmail.com] > *Sent:* Wednesday, May 6, 2015 9:33 AM > *To:* ayan guha > *Cc:* user@spark.apache.org; d...@spark.apache.org > *Subject:* Re: Creating topology in spark streamin

RE: Creating topology in spark streaming

corresponding number of Threads From: anshu shukla [mailto:anshushuk...@gmail.com] Sent: Wednesday, May 6, 2015 9:33 AM To: ayan guha Cc: user@spark.apache.org; d...@spark.apache.org Subject: Re: Creating topology in spark streaming But main problem is how to increase the level of

Re: Creating topology in spark streaming

Thanks alot Juan, That was a great post, One more thing if u can .Any there any demo/blog telling how to configure or create a topology of different types .. i mean how we can decide the pipelining model in spark as done in storm for https://storm.apache.org/documentation/images/topology.p

Re: Creating topology in spark streaming

Hi, You can use the method repartition from DStream (for the Scala API) or JavaDStream (for the Java API) defrepartition(numPartitions: Int): DStream [T] Return a new DStream with an increased or dec

Re: Creating topology in spark streaming

But main problem is how to increase the level of parallelism for any particular bolt logic . suppose i want this type of topology . https://storm.apache.org/documentation/images/topology.png How we can manage it . On Wed, May 6, 2015 at 1:36 PM, ayan guha wrote: > Every transformation on a

Re: Creating topology in spark streaming

Every transformation on a dstream will create another dstream. You may want to take a look at foreachrdd? Also, kindly share your code so people can help better On 6 May 2015 17:54, "anshu shukla" wrote: > Please help guys, Even After going through all the examples given i have > not understood

Creating topology in spark streaming

Please help guys, Even After going through all the examples given i have not understood how to pass the D-streams from one bolt/logic to other (without writing it on HDFS etc.) just like emit function in storm . Suppose i have topology with 3 bolts(say) *BOLT1(parse the tweets nd emit tweet u