Spark 2.3.1 leaves _temporary dir back on s3 even after write to s3 is done.

[sushil.chaudhary]

folks,
We recently upgraded to 2.3.1 and we started seeing that, the spark jobs
leaves _temporary directory in the s3 even though write to s3 already
finish. It do not cleanup the temporary directory. 
Hadoop version 2.8.  is there a way to control it?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Worker Re-register to Master

[sushil.chaudhary]

Can anyone please have a look and put thoughts here..



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark Worker Re-register to Master

[sushil.chaudhary]

All, 

We have a problem with Spark Worker. The worker goes down whenever we are
not able to get the spark master up and running before starting the worker. 
Of course- it does try to ReregisterWithMaster  upto 16 attemps : 

1. First 6 attempts it make in interval of appx 10 seconds 
2. Next 10 attempts it make in interval of appox 60 seconds.

But if Master is late, it fails and go down. Can anyone please suggest the
configuration to change the retry behavior.  Is there way, to have more
number of retry with a configurable wait interval so that it can wait for
master for longer period of time. I am unable to find the props which
control these behavior.

Thanks in advance.
Sushil



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark Worker Re-register to Master

[sushil.chaudhary]

All, 

We have a problem with Spark Worker. The worker goes down whenever we are
not able to get the spark master up and running before starting the worker. 
Of course- it does try to ReregisterWithMaster  upto 16 attemps : 

1. First 6 attempts it make in interval of appx 10 seconds 
2. Next 10 attempts it make in interval of appox 60 seconds.

But if Master is late, it fails and go down. Can anyone please suggest the
configuration to change the retry behavior.  Is there way, to have more
number of retry with a configurable wait interval so that it can wait for
master for longer period of time. I am unable to find the props which
control these behavior.

Thanks in advance.
Sushil



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org