Re: Normal Spark Streaming vs Streaming on Beam with Spark Runner

2018-05-18 Thread chandan prakash
Thanks Lukasz. It was really helpful to understand. Regards, Chandan On Thu, May 17, 2018 at 8:26 PM, Lukasz Cwik wrote: > > > On Wed, May 16, 2018 at 10:46 PM chandan prakash < > chandanbaran...@gmail.com> wrote: > >> Thanks Ismaël. >> Your answers were quite useful for a novice user. >> I gu

Re: Normal Spark Streaming vs Streaming on Beam with Spark Runner

2018-05-17 Thread Lukasz Cwik
On Wed, May 16, 2018 at 10:46 PM chandan prakash wrote: > Thanks Ismaël. > Your answers were quite useful for a novice user. > I guess this answer will help many like me. > > *Regarding your answer to point 2 :* > > > *"Checkpointing is supported, Kafka offset management (if I understand > whatyo

Re: Normal Spark Streaming vs Streaming on Beam with Spark Runner

2018-05-16 Thread chandan prakash
Thanks Ismaël. Your answers were quite useful for a novice user. I guess this answer will help many like me. *Regarding your answer to point 2 :* *"Checkpointing is supported, Kafka offset management (if I understand whatyoumean) is managed by the KafkaIO connector + the runner"* Beam provides

Re: Normal Spark Streaming vs Streaming on Beam with Spark Runner

2018-05-16 Thread Ismaël Mejía
Hello, Answers to the questions inline: > 1. Are there any limitations in terms of implementations, functionalities or performance if we want to run streaming on Beam with Spark runner vs streaming on Spark-Streaming directly ? At this moment the Spark runner does not support some parts of the B

Re: Normal Spark Streaming vs Streaming on Beam with Spark Runner

2018-05-15 Thread chandan prakash
Also, 3. with spark 2.x structured streaming , if we want to switch across different modes like from micro-batching to continuous streaming mode, how it can be done while using Beam? These are some of the initial questions which I am not able to understand currently. Regards, Chandan On Tue, M