Re: Spark 3.1.2 full thread dumps

figure, but it will be in the ballpark. From: Maksim Grinman Date: Friday, February 11, 2022 at 2:21 PM To: "Lalwani, Jayesh" Cc: Mich Talebzadeh , Holden Karau , Sean Owen , "user @spark" Subject: RE: [EXTERNAL] Spark 3.1.2 full thread dumps CAUTION: This email originated

Re: Spark 3.1.2 full thread dumps

get the dreaded thread dump > if your tasks are taking few seconds > > > > *From: *Maksim Grinman > *Date: *Thursday, February 10, 2022 at 7:21 PM > *To: *"Lalwani, Jayesh" > *Cc: *Mich Talebzadeh , Holden Karau < > hol...@pigscanfly.ca>, Sean Owen , "

Re: Spark 3.1.2 full thread dumps

dump if your tasks are taking few seconds From: Maksim Grinman Date: Thursday, February 10, 2022 at 7:21 PM To: "Lalwani, Jayesh" Cc: Mich Talebzadeh , Holden Karau , Sean Owen , "user @spark" Subject: RE: [EXTERNAL] Spark 3.1.2 full thread dumps CAUTION: This email ori

Re: Spark 3.1.2 full thread dumps

, "user @spark" Subject: RE: [EXTERNAL] Spark 3.1.2 full thread dumps CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Not that this discussion is not interesting (i

Re: Spark 3.1.2 full thread dumps

Indeed. Apologies for going on a tangent. view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on

Re: Spark 3.1.2 full thread dumps

Not that this discussion is not interesting (it is), but this has strayed pretty far from my original question. Which was: How do I prevent spark from dumping huge Java Full Thread dumps when an executor appears to not be doing anything (in my case, there's a loop where it sleeps waiting for a serv

Re: Spark 3.1.2 full thread dumps

OK basically, do we have a scenario where Spark or for that matter any cluster manager can deploy a new node (after the loss of an existing node) with the view of running the failed tasks on the new executor(s) deployed on that newly spun node? view my Linkedin profile

Re: Spark 3.1.2 full thread dumps

We don’t block scaling up after node failure in classic Spark if that’s the question. On Fri, Feb 4, 2022 at 6:30 PM Mich Talebzadeh wrote: > From what I can see in auto scaling setup, you will always need a min of > two worker nodes as primary. It also states and I quote "Scaling primary > work

Re: Spark 3.1.2 full thread dumps

>From what I can see in auto scaling setup, you will always need a min of two worker nodes as primary. It also states and I quote "Scaling primary workers is not recommended due to HDFS limitations which result in instability while scaling. These limitations do not exist for secondary workers". So

Re: Spark 3.1.2 full thread dumps

I have not seen stack traces under autoscaling, so not even sure what the error in question is. There is always delay in acquiring a whole new executor in the cloud as it usually means a new VM is provisioned. Spark treats the new executor like any other, available for executing tasks. On Fri, Feb

Re: Spark 3.1.2 full thread dumps

Thanks for the info. My concern has always been on how Spark handles autoscaling (adding new executors) when the load pattern changes.I have tried to test this with setting the following parameters (Spark 3.1.2 on GCP) spark-submit --verbose \ ... --conf spark.dynami

Re: Spark 3.1.2 full thread dumps

It's actually on AWS EMR. The job bootstraps and runs fine -- the autoscaling group is to bring up a service that spark will be calling. Some code waits for the autoscaling group to come up before continuing processing in Spark, since the Spark cluster will need to make requests to the service in t

Re: Spark 3.1.2 full thread dumps

Sounds like you are running this on Google Dataproc cluster (spark 3.1.2) with auto scaling policy? Can you describe if this happens before Spark starts a new job on the cluster or somehow half way through processing an existing job? Also is the job involved doing Spark Structured Streaming? HT

Spark 3.1.2 full thread dumps

We've got a spark task that, after some processing, starts an autoscaling group and waits for it to be up before continuing processing. While waiting for the autoscaling group, spark starts throwing full thread dumps, presumably at the spark.executor.heartbeat interval. Is there a way to prevent th