The warnings regarding configuration on the executor are for the executor kafka consumer, not the driver kafka consumer.
In general, the executor kafka consumers should consume only exactly the offsets the driver told them to, and not be involved in committing offsets / part of the same group as the driver. That's why those configurations are forcibly overriden. In short those warnings are normal, the main thing you should be concerned about is the checkpoint. On Mon, Jul 10, 2017 at 10:26 AM, shyla deshpande <deshpandesh...@gmail.com> wrote: > WARN Use an existing SparkContext, some configuration may not take effect. > I wanted to restart the spark streaming app, so stopped the running > and issued a new spark submit. Why and how it will use a existing > SparkContext? > => you are using checkpoint to restore the sparkcontext. > => No, I am not using checkpoint for recovery. I need the checkpoint because > I am doing stateful streaming. > > WARN Spark is not running in local mode, therefore the checkpoint directory > must not be on the local filesystem. Directory 'file:/efs/checkpoint' > appears to be on the local filesystem. > =>the CP path should be HDFSand so on. If you want to use local path, the > cluster model should be local. > => I am using the AWS EFS mount for checkpoint because I am running in > standalone mode. > > WARN overriding enable.auto.commit to false for executor > =>stop the executor to commit the offset auto > =>No, I don't want the autocommit. I do the commit later after my output > operation using the commitAsync API > > WARN overriding auto.offset.reset to none for executor > =>it set the index where the executor read msg > => I set this to none, because I want it to continue from where it left off > > WARN overriding executor group.id to spark-executor-mygroupid > => set the groupid of consumer. If you do not set, it will set a default and > give a warning. > => I am setting the groupid, mygroupid and it is adding spark-executor- as a > prefix... > > > On Mon, Jul 10, 2017 at 12:39 AM, 萝卜丝炒饭 <1427357...@qq.com> wrote: >> >> It seems you are usibg kafka 0.10. >> See my comments below. >> >> ---Original--- >> From: "shyla deshpande"<deshpandesh...@gmail.com> >> Date: 2017/7/10 08:17:10 >> To: "user"<user@spark.apache.org>; >> Subject: Spark streaming giving me a bunch of WARNINGS, please help >> meunderstand them >> >> WARN Use an existing SparkContext, some configuration may not take >> effect. >> I wanted to restart the spark streaming app, so stopped the >> running and issued a new spark submit. Why and how it will use a existing >> SparkContext? >> => you are using checkpoint to restore the sparkcontext. >> WARN Spark is not running in local mode, therefore the checkpoint >> directory must not be on the local filesystem. Directory >> 'file:/efs/checkpoint' appears to be on the local filesystem. >> =>the CP path should be HDFSand so on. >> If you want to use local path, the cluster model should be local. >> >> >> WARN overriding enable.auto.commit to false for executor >> =>stop the executor to commit the offset auto. >> >> WARN overriding auto.offset.reset to none for executor >> =>it set the index where the executor read msg >> WARN overriding executor group.id to spark-executor-mygroupid >> => set the groupid of consumer. If you do not set, it will set a default >> and give a warning. >> >> WARN overriding receive.buffer.bytes to 65536 see KAFKA-3135 >> WARN overriding enable.auto.commit to false for executor >> WARN overriding auto.offset.reset to none for executor > > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org