Re: spark streaming Directkafka with checkpointing : changed parameters not considered

Ohh that explains the reason. My use case does not need state management. So i guess i am better off without checkpointing. Thank you for clarification. Regards, Chandan On Sat, Aug 20, 2016 at 8:24 AM, Cody Koeninger wrote: > Checkpointing is required to be turned on in

Re: spark streaming Directkafka with checkpointing : changed parameters not considered

Checkpointing is required to be turned on in certain situations (e.g. updateStateByKey), but you're certainly not required to rely on it for fault tolerance. I try not to. On Fri, Aug 19, 2016 at 1:51 PM, chandan prakash wrote: > Thanks Cody for the pointer. > > I am

Re: spark streaming Directkafka with checkpointing : changed parameters not considered

Thanks Cody for the pointer. I am able to do this now. Not using checkpointing. Rather storing offsets in zookeeper for fault tolerance. Spark Config changes now getting reflected in code deployment. *Using this api :* *KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder,

Re: spark streaming Directkafka with checkpointing : changed parameters not considered

Yes, i looked into the source code implementation. sparkConf is serialized and saved during checkpointing and re-created from the checkpoint directory at time of restart. So any sparkConf parameter which you load from application.config and set in sparkConf object in code cannot be changed and

Re: spark streaming Directkafka with checkpointing : changed parameters not considered

Checkpointing is not kafka-specific. It encompasses metadata about the application. You can't re-use a checkpoint if your application has changed. http://spark.apache.org/docs/latest/streaming-programming-guide.html#upgrading-application-code On Thu, Aug 18, 2016 at 4:39 AM, chandan prakash

Re: spark streaming Directkafka with checkpointing : changed parameters not considered

Is it possible that i use checkpoint directory to restart streaming but with modified parameter value in config file (e.g. username/password for db connection) ? Thanks in advance. Regards, Chandan On Thu, Aug 18, 2016 at 1:10 PM, chandan prakash wrote: > Hi, > I

spark streaming Directkafka with checkpointing : changed parameters not considered

Hi, I am using direct kafka with checkpointing of offsets same as : https://github.com/koeninger/kafka-exactly-once/blob/master/src/main/scala/example/IdempotentExample.scala I need to change some parameters like db connection params : username/password for db connection . I stopped streaming