Can you add more details like are you using rdds/datasets/sql ..; are you
doing group by/ joins ; is your input splittable?
btw, you can pass the config the same way you are passing memryOverhead:
e.g.
--conf spark.default.parallelism=1000 or through spark-context in code
Regards,
Sushrut Ikhar
[
Hi All,
Any updates on this?
On Wednesday 28 September 2016 12:22 PM, Sushrut Ikhar wrote:
Try with increasing the parallelism by repartitioning and also you may
increase - spark.default.parallelism
You can also try with decreasing num-executor cores.
Basically, this happens when the executor
:
Thanks Sushrut for the reply.
Currently I have not defined spark.default.parallelism property.
Can you let me know how much should I set it to?
Regards,
Aditya Calangutkar
On Wednesday 28 September 2016 12:22 PM, Sushrut Ikhar wrote:
Try with increasing the parallelism by repartitioning and
Do you mind providing a bit more information ?
release of Spark
code snippet of your app
version of Java
Thanks
On Tue, Aug 18, 2015 at 8:57 AM, unk1102 wrote:
> Hi this GC overhead limit error is making me crazy. I have 20 executors
> using
> 25 GB each I dont understand at all how can it t
It could be stuck on a GC pause, Can you check a bit more in the executor
logs and see whats going on? Also from the driver UI you would get to know
at which stage it is being stuck etc.
Thanks
Best Regards
On Sun, Aug 16, 2015 at 11:45 PM, unk1102 wrote:
> Hi I have written Spark job which see
It says connection refused, just make sure the network is configured
properly (open the ports between master and the worker nodes). If the ports
are configured correctly, then i assume the process is getting killed for
some reason and hence connection refused.
Thanks
Best Regards
On Fri, Dec 5, 2
Here is a sample exception I collected from a spark worker node: (there are
many such errors across over work nodes). It looks to me that spark worker
failed to communicate to executor locally.
14/12/04 04:26:37 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkwor...@spark-prod1.xxx:7079]
bq. to get the logs from the data nodes
Minor correction: the logs are collected from machines where node managers
run.
Cheers
On Wed, Dec 3, 2014 at 3:39 PM, Ganelin, Ilya
wrote:
> You want to look further up the stack (there are almost certainly other
> errors before this happens) and thos
You want to look further up the stack (there are almost certainly other errors
before this happens) and those other errors may give your better idea of what
is going on. Also if you are running on yarn you can run "yarn logs
-applicationId " to get the logs from the data nodes.
Sent with Good