ded so that default was
> increased from 4 to 16
> .set("spark.task.maxFailures","64") // Didn't really matter as I had
> no failures in this run
> .set("spark.storage.blockManagerSlaveTimeoutMs","30");
>
>
>
> From: Sven
ased
from 4 to 16
.set("spark.task.maxFailures","64") // Didn't really matter as I had no
failures in this run
.set("spark.storage.blockManagerSlaveTimeoutMs","30");
________________
From: Sven Krasser
To: Darin McBeath
Cc: User
Sent:
Hey Darin,
Are you running this over EMR or as a standalone cluster? I've had
occasional success in similar cases by digging through all executor logs
and trying to find exceptions that are not caused by the application
shutdown (but the logs remain my main pain point with Spark).
That aside, ano
I've tried various ideas, but I'm really just shooting in the dark.
I have an 8 node cluster of r3.8xlarge machines. The RDD (with 1024 partitions)
I'm trying to save off to S3 is approximately 1TB in size (with the partitions
pretty evenly distributed in size).
I just tried a test to dial back
