WARN Use an existing SparkContext, some configuration may not take effect.
I wanted to restart the spark streaming app, so stopped the
running and issued a new spark submit. Why and how it will use a existing
SparkContext?
=> you are using checkpoint to restore the sparkcontext.
=> No, I am not using checkpoint for recovery. I need the checkpoint
because I am doing stateful streaming.
WARN Spark is not running in local mode, therefore the checkpoint
directory must not be on the local filesystem. Directory
'file:/efs/checkpoint' appears to be on the local filesystem.
=>the CP path should be HDFSand so on. If you want to use local path, the
cluster model should be local.
=> I am using the AWS EFS mount for checkpoint because I am running in
standalone mode.
WARN overriding enable.auto.commit to false for executor
=>stop the executor to commit the offset auto
=>No, I don't want the autocommit. I do the commit later after my output
operation using the commitAsync API
WARN overriding auto.offset.reset to none for executor
=>it set the index where the executor read msg
=> I set this to none, because I want it to continue from where it left off
WARN overriding executor group.id to spark-executor-mygroupid
=> set the groupid of consumer. If you do not set, it will set a default
and give a warning.
=> I am setting the groupid, mygroupid and it is adding spark-executor- as
a prefix...
On Mon, Jul 10, 2017 at 12:39 AM, 萝卜丝炒饭 <[email protected]> wrote:
> It seems you are usibg kafka 0.10.
> See my comments below.
>
> ---Original---
> *From:* "shyla deshpande"<[email protected]>
> *Date:* 2017/7/10 08:17:10
> *To:* "user"<[email protected]>;
> *Subject:* Spark streaming giving me a bunch of WARNINGS, please help
> meunderstand them
>
> WARN Use an existing SparkContext, some configuration may not take effect.
> I wanted to restart the spark streaming app, so stopped the
> running and issued a new spark submit. Why and how it will use a existing
> SparkContext?
> => you are using checkpoint to restore the sparkcontext.
> WARN Spark is not running in local mode, therefore the checkpoint
> directory must not be on the local filesystem. Directory
> 'file:/efs/checkpoint' appears to be on the local filesystem.
> =>the CP path should be HDFSand so on.
> If you want to use local path, the cluster model should be local.
>
>
> WARN overriding enable.auto.commit to false for executor
> =>stop the executor to commit the offset auto.
>
> WARN overriding auto.offset.reset to none for executor
> =>it set the index where the executor read msg
> WARN overriding executor group.id to spark-executor-mygroupid
> => set the groupid of consumer. If you do not set, it will set a default
> and give a warning.
>
> WARN overriding receive.buffer.bytes to 65536 see KAFKA-3135
> WARN overriding enable.auto.commit to false for executor
> WARN overriding auto.offset.reset to none for executor
>
