Re: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

User Sent: Thursday, November 3, 2022 8:35 PM To: user@spark.apache.org Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. Now I see what you want to do. If you have access to the cluster con

Re: [EXTERNAL] Re: Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

, 2022 4:19 PM To: Shay Elbaz Cc: Artemis User ; Tom Graves ; Tom Graves ; user@spark.apache.org Subject: [EXTERNAL] Re: Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. May I ask why the ETL job and DL

Re: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

t; > I hope that was more clear. > Thank you very much for helping. > > Shay > > -- > *From:* Tom Graves > *Sent:* Friday, November 4, 2022 4:19 PM > *To:* Tom Graves ; Artemis User < > arte...@dtechspace.com>; user@spark.apache.org ; > Shay Elba

Re: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

mis User ; user@spark.apache.org ; Shay Elbaz Subject: [EXTERNAL] Re: Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. So I'm not sure I completely follow. Are you asking for a way to change the limit with

Re: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

ch more. Any failure during this long time is pretty expensive. ShayFrom: Tom Graves Sent: Thursday, November 3, 2022 7:56 PM To: Artemis User ; user@spark.apache.org ; Shay Elbaz Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs 

Re: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs

, November 3, 2022 7:56 PM To: Artemis User ; user@spark.apache.org ; Shay Elbaz Subject: [EXTERNAL] Re: Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM. Stage level scheduling does not allow you to change configs

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

level scheduling - lower the number of executors when using GPUs *ATTENTION:*This email originated from outside of GM. Shay,  You may find this video helpful (with some API code samples that you are looking for). https://www.youtube.com/watch?v=JNQu-226wUc=171s <https://www.youtube.com/watc

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

GPU per executor. So, the question is how do I limit the stage resources to 20 GPUs total?  Thanks again,Shay From: Artemis User Sent: Thursday, November 3, 2022 5:23 PM To: user@spark.apache.org Subject: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

Er, wait, this is what stage-level scheduling is right? this has existed since 3.1 https://issues.apache.org/jira/browse/SPARK-27495 On Thu, Nov 3, 2022 at 12:10 PM bo yang wrote: > Interesting discussion here, looks like Spark does not support configuring > different number of exe

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

Interesting discussion here, looks like Spark does not support configuring different number of executors in different stages. Would love to see the community come out such a feature. On Thu, Nov 3, 2022 at 9:10 AM Shay Elbaz wrote: > Thanks again Artemis, I really appreciate it. I have watc

Re: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs

? Thanks again, Shay From: Artemis User Sent: Thursday, November 3, 2022 5:23 PM To: user@spark.apache.org Subject: [EXTERNAL] Re: Re: Stage level scheduling - lower the number of executors when using GPUs ATTENTION: This email originated from outside of GM

Re: [EXTERNAL] Re: Stage level scheduling - lower the number of executors when using GPUs

Shay,  You may find this video helpful (with some API code samples that you are looking for). https://www.youtube.com/watch?v=JNQu-226wUc=171s.  The issue here isn't how to limit the number of executors but to request for the right GPU-enabled executors dynamically.  Those executors used

Re: [EXTERNAL] Re: Stage level scheduling - lower the number of executors when using GPUs

Thanks Artemis. We are not using Rapids, but rather using GPUs through the Stage Level Scheduling feature with ResourceProfile. In Kubernetes you have to turn on shuffle tracking for dynamic allocation, anyhow. The question is how we can limit the number of executors when building a new

Re: Stage level scheduling - lower the number of executors when using GPUs

for a GPU stage than for a CPU stage. We are using dynamic allocation with stage level scheduling, and Spark tries to maximize the number of executors also during the GPU stage, causing a bit of resources chaos in the cluster. This forces us to use a lower value for 'maxExecutors' in the fi

Stage level scheduling - lower the number of executors when using GPUs

Hi, Our typical applications need less executors for a GPU stage than for a CPU stage. We are using dynamic allocation with stage level scheduling, and Spark tries to maximize the number of executors also during the GPU stage, causing a bit of resources chaos in the cluster. This forces us

Spark streaming job not able to launch more number of executors

, after which we fork out the DAG into two. Each of the forks has an action (forEach) at the end. In this case, we are observing that the number of executors is not exceeding the number of input kafka partitions. Job is not spawning more than 60 executors (2*10*3). And we see that the tasks from

Re: Databricks - number of executors, shuffle.partitions etc

Thanks Ayan, I wasn't aware of such user group specifically for databricks. Thanks for the input, much appreciated! On Wed, May 15, 2019 at 10:07 PM ayan guha wrote: > Well its a databricks question so better be asked in their forum. > > You can set up cluster level params when you create new

Re: Databricks - number of executors, shuffle.partitions etc

Well its a databricks question so better be asked in their forum. You can set up cluster level params when you create new cluster or add them later. Go to cluster page, ipen one cluster, expand additional config section and add your param there as key value pair separated by space. On Thu, 16

Re: Databricks - number of executors, shuffle.partitions etc

Hi All, Any idea? Thanks, -Rishi On Tue, May 14, 2019 at 11:52 PM Rishi Shah wrote: > Hi All, > > How can we set spark conf parameter in databricks notebook? My cluster > doesn't take into account any spark.conf.set properties... it creates 8 > worker nodes (dat executors) but doesn't honor

Databricks - number of executors, shuffle.partitions etc

Hi All, How can we set spark conf parameter in databricks notebook? My cluster doesn't take into account any spark.conf.set properties... it creates 8 worker nodes (dat executors) but doesn't honor the supplied conf parameters. Any idea? -- Regards, Rishi Shah

Re: Spark Streaming - Increasing number of executors slows down processing rate

Hi Edwin, I have faced a similar issue as well and this behaviour is very abrupt. I even created a question on StackOverflow but no solution yet. https://stackoverflow.com/questions/43496205/spark-job-processing-time-increases-to-4s-without-explanation For us, we sometimes had this constant

Spark Streaming - Increasing number of executors slows down processing rate

Hi All, I am struggling with an odd issue and would like your help in addressing it. Environment AWS Cluster (40 Spark Nodes & 4 node Kafka cluster) Spark Kafka Streaming submitted in Yarn cluster mode Kafka - Single topic, 400 partitions Spark 2.1 on Cloudera Kafka 10.0 on Cloudera We have zero

Spark streaming uses lesser number of executors

get 2 executors. My batch interval is 1 second. While running what I observe from event timeline is that only 3 of the executors are being used. The other 3 are not being used. As far as I know, there is no parameter in spark standalone mode to specify the number of executors. How do I make spark

What factors decide the number of executors when doing a Spark SQL insert in Mesos?

Hi, What factors decide the number of executors when doing a Spark SQL insert? Right now when I submit my job in Mesos I see only 2 executors getting allocated all the time. Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/What-factors-decide

Re: Number of executors change during job running

will give deeper understanding of the problem. Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-change-during-job-running-tp9243p26866.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Number of executors in spark-1.6 and spark-1.5

l thread but within same executor. >> 3. In Spark-1.6, If I increase no of worker instances on each node then >> jobs >> are running in parallel as no of workers but within same executor. >> >> Can anyone suggest, why spark 1.6 can not use multiple executors across >

Re: Number of executors in spark-1.6 and spark-1.5

node at a time for parallel processing. > Your suggestion will be highly appreciated. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-in-spark-1-6-and-

Number of executors in spark-1.6 and spark-1.5

for parallel processing. Your suggestion will be highly appreciated. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-in-spark-1-6-and-spark-1-5-tp26733.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: reasonable number of executors

Hi Igor, That's a great talk and an exact answer to my question. Thank you. Cheers, Alex. On Tue, Feb 23, 2016 at 8:27 PM, Igor Berman wrote: > > http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications > > there is a section

Re: reasonable number of executors

http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications there is a section that is connected to your question On 23 February 2016 at 16:49, Alex Dzhagriev wrote: > Hello all, > > Can someone please advise me on the pros and cons on

Re: reasonable number of executors

Hi Alex, take a look here : https://blogs.aws.amazon.com/bigdata/post/Tx3RD6EISZGHQ1C/The-Impact-of-Using-Latest-Generation-Instances-for-Your-Amazon-EMR-Job

reasonable number of executors

Hello all, Can someone please advise me on the pros and cons on how to allocate the resources: many small heap machines with 1 core or few machines with big heaps and many cores? I'm sure that depends on the data flow and there is no best practise solution. E.g. with bigger heap I can perform

Re: Specify number of executors in standalone cluster mode

Max number of cores per executor can be controlled using spark.executor.cores. And maximum number of executors on a single worker can be determined by environment variable: SPARK_WORKER_INSTANCES. However, to ensure that all available cores are used, you will have to take care of how the stream

Specify number of executors in standalone cluster mode

Hi, I'm running a spark streaming application onto a spark cluster that spans 6 machines/workers. I'm using spark cluster standalone mode. Each machine has 8 cores. Is there any way to specify that I want to run my application on all 6 machines and just use 2 cores on each machine? Thanks

Number of executors in Spark - Kafka

the number executors to six. If I don't specific anything it just create one executor. Looking for information I have read: "The --num-executors command-line flag or spark.executor.instances configuration property control the number of executors requested. Starting in CDH 5.4/Spark 1.3, you will be

Re: Number of executors in Spark - Kafka

it should have six executors to process each one > one RDD. To do it, when I execute spark-submit (I use YARN) I specific the > number executors to six. > If I don't specific anything it just create one executor. Looking for > information I have read: > > &q

number of executors in sparkR.init()

Hi, I'm having trouble working out how to get the number of executors set when using sparkR.init(). If I start sparkR with sparkR --master yarn --num-executors 6 then I get 6 executors However, if start sparkR with sparkR followed by sc <- sparkR.init(master="yar

Re: number of executors in sparkR.init()

; Sent: Friday, December 25, 2015 9:23 PM Subject: number of executors in sparkR.init() To: <user@spark.apache.org> Hi, I'm having trouble working out how to get the number of executors set when using sparkR.init(). If I

Re: number of executors in sparkR.init()

> > Could you try setting that with sparkR.init()? > > > _ > From: Franc Carter <franc.car...@gmail.com> > Sent: Friday, December 25, 2015 9:23 PM > Subject: number of executors in sparkR.init() > To: <user@spark.apache.org>

RE: ideal number of executors per machine

the spark jobs unstable. Thx tri From: Veljko Skarich [mailto:veljko.skar...@gmail.com] Sent: Tuesday, December 15, 2015 3:08 PM To: user@spark.apache.org Subject: ideal number of executors per machine Hi, I'm looking for suggestions on the ideal number of executors per machine. I run my jobs on 64G

Re: ideal number of executors per machine

gt;> >> On Tue, Dec 15, 2015 at 9:07 PM, Veljko Skarich >> <veljko.skar...@gmail.com> wrote: >>> Hi, >>> >>> I'm looking for suggestions on the ideal number of executors per machine. I >>> run my jobs on 64G 32 core machines, and at the m

Re: ideal number of executors per machine

Hi Veljko, I would assume keeping the number of executors per machine to a minimum is best for performance (as long as you consider memory requirements as well). Each executor is a process that can run tasks in multiple threads. On a kernel/hardware level, thread switches are much cheaper than

ideal number of executors per machine

Hi, I'm looking for suggestions on the ideal number of executors per machine. I run my jobs on 64G 32 core machines, and at the moment I have one executor running per machine, on the spark standalone cluster. I could not find many guidelines for figuring out the ideal number of executors

Re: ideal number of executors per machine

Best Regards, Jerry > On Dec 15, 2015, at 5:18 PM, Jakob Odersky <joder...@gmail.com> wrote: > > Hi Veljko, > I would assume keeping the number of executors per machine to a minimum is > best for performance (as long as you consider memory requirements as well). > Each e

Re: ideal number of executors per machine

o.skar...@gmail.com> wrote: > Hi, > > I'm looking for suggestions on the ideal number of executors per machine. I > run my jobs on 64G 32 core machines, and at the moment I have one executor > running per machine, on the spark standalone cluster. > > I could not find many guideli

Re: How to set the number of executors and tasks in a Spark Streaming job in Mesos

can try doing a dstream.repartition to see if it increase from 11 to a higher number. Thanks Best Regards On Thu, Aug 20, 2015 at 2:28 AM, swetha swethakasire...@gmail.com wrote: Hi, How to set the number of executors and tasks in a Spark Streaming job in Mesos? I have the following settings

Re: Finding the number of executors.

As I was writing a long-ish message to explain how it doesn't work, it dawned on me that maybe driver connects to executors only after there's some work to do (while I was trying to find the number of executors BEFORE starting the actual work). So the solution was to simply execute a dummy task

Finding the number of executors.

Is there any reliable way to find out the number of executors programatically - regardless of how the job is run? A method that preferably works for spark-standalone, yarn, mesos, regardless whether the code runs from the shell or not? Things that I tried and don't work

Re: Finding the number of executors.

Hi Akhil, I'm using spark 1.4.1. Number of executors is not in the command line, not in the getExecutorMemoryStatus (I already mentioned that I tried that, works in spark-shell but not when executed via spark-submit). I tried looking at defaultParallelism too, it's 112 (7 executors * 16 cores

Re: Finding the number of executors.

Which version spark are you using? There was a discussion happened over here http://apache-spark-user-list.1001560.n3.nabble.com/Determine-number-of-running-executors-td19453.html http://mail-archives.us.apache.org/mod_mbox/spark-user/201411.mbox/%3ccacbyxk+ya1rbbnkwjheekpnbsbh10rykuzt

Re: Finding the number of executors.

= sc.getConf.get(spark.driver.host)    allExecutors.filter(! _.split(:)(0).equals(driverHost)).toList  } On Friday, August 21, 2015 1:53 PM, Virgil Palanciuc virg...@gmail.com wrote: Hi Akhil, I'm using spark 1.4.1. Number of executors is not in the command line

How to set the number of executors and tasks in a Spark Streaming job in Mesos

Hi, How to set the number of executors and tasks in a Spark Streaming job in Mesos? I have the following settings but my job still shows me 11 active tasks and 11 executors. Any idea as to why this is happening ? sparkConf.set(spark.mesos.coarse, true) sparkConf.set(spark.cores.max, 128

Re: Controlling number of executors on Mesos vs YARN

Hi Tim, An option like spark.mesos.executor.max to cap the number of executors per node/application would be very useful. However, having an option like spark.mesos.executor.num to specify desirable number of executors per node would provide even/much better control. Thanks, Ajay On Wed, Aug

Re: Controlling number of executors on Mesos vs YARN

specify desirable number of executors. If not available, Mesos (in a simple implementation) can provide/offer whatever is available. In a slightly complex implementation, we can build a simple protocol to negotiate. Regards, Ajay On Wed, Aug 12, 2015 at 5:51 PM, Tim Chen t...@mesosphere.io wrote

Re: Controlling number of executors on Mesos vs YARN

You're referring to both fine grain and coarse grain? Desirable number of executors per node could be interesting but it can't be guaranteed (or we could try to and when failed abort the job). How would you imagine this new option to actually work? Tim On Wed, Aug 12, 2015 at 11:48 AM, Ajay

Re: Controlling number of executors on Mesos vs YARN

Ayyalasomayajula aharipriy...@gmail.com wrote: Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores

Re: Controlling number of executors on Mesos vs YARN

aharipriy...@gmail.com wrote: Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores

Re: Controlling number of executors on Mesos vs YARN

Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores? Is there a default number it assumes? On Mon

Re: Controlling number of executors on Mesos vs YARN

: Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores? Is there a default number

Re: Controlling number of executors on Mesos vs YARN

aharipriy...@gmail.com wrote: Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores

Re: Determining number of executors within RDD

if you want you can determine the number of executor as well by setting 'spark.executor.instances' property in 'sparkConf' object. Thank you. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23241.html Sent from

Re: Determining number of executors within RDD

Note that this property is only available for YARN -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23256.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Determining number of executors within RDD

from Samsung Mobile Original message From: maxdml Date:2015/06/10 19:56 (GMT+00:00) To: user@spark.apache.org Subject: Re: Determining number of executors within RDD Actually this is somehow confusing for two reasons: - First, the option 'spark.executor.instances', which

Re: Determining number of executors within RDD

/executot Sent from Samsung Mobile div Original message /divdivFrom: maxdml max...@cs.duke.edu /divdivDate:2015/06/10 19:56 (GMT+00:00) /divdivTo: user@spark.apache.org /divdivSubject: Re: Determining number of executors within RDD /divdiv /divActually this is somehow confusing

Re: Determining number of executors within RDD

that it is also available for this mode - Second, a post from Andrew Or states that this properties define the number of workers in the cluster, not the number of executors on a given worker. (http://apache-spark-user-list.1001560.n3.nabble.com/clarification-for-some-spark-on-yarn-configuration-options

Re: Determining number of executors within RDD

workers per machine Sent from Samsung Mobile Original message From: Sandy Ryza Date:2015/06/10 21:31 (GMT+00:00) To: Evo Eftimov Cc: maxdml ,user@spark.apache.org Subject: Re: Determining number of executors within RDD On YARN, there is no concept of a Spark Worker

Re: Determining number of executors within RDD

.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23234.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional

Re: number of executors

Oh BTW, it's spark 1.3.1 on hadoop 2.4. AIM 3.6. Sorry for lefting out this information. Appreciate for any help! Ed 2015-05-18 12:53 GMT-04:00 edward cui edwardcu...@gmail.com: I actually have the same problem, but I am not sure whether it is a spark problem or a Yarn problem. I set up a

Re: number of executors

*All On Mon, May 18, 2015 at 9:07 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi Xiaohe, The all Spark options must go before the jar or they won't take effect. -Sandy On Sun, May 17, 2015 at 8:59 AM, xiaohe lan zombiexco...@gmail.com wrote: Sorry, them both are assigned task

Re: number of executors

Hi Xiaohe, The all Spark options must go before the jar or they won't take effect. -Sandy On Sun, May 17, 2015 at 8:59 AM, xiaohe lan zombiexco...@gmail.com wrote: Sorry, them both are assigned task actually. Aggregated Metrics by Executor Executor IDAddressTask TimeTotal TasksFailed

Re: number of executors

I actually have the same problem, but I am not sure whether it is a spark problem or a Yarn problem. I set up a five nodes cluster on aws emr, start yarn daemon on the master (The node manager will not be started on default on the master, I don't want to waste any resource since I have to pay).

Re: number of executors

Yeah, I read that page before, but it does not mention the options should come before the application jar. Actually, if I put the --class option before the application jar, I will get ClassNotFoundException. Anyway, thanks again Sandy. On Tue, May 19, 2015 at 11:06 AM, Sandy Ryza

Re: number of executors

Hi Sandy, Thanks for your information. Yes, spark-submit --master yarn --num-executors 5 --executor-cores 4 target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp is working awesomely. Is there any documentations pointing to this ? Thanks, Xiaohe On Tue, May 19, 2015 at 12:07 AM,

Re: number of executors

Awesome! It's documented here: https://spark.apache.org/docs/latest/submitting-applications.html -Sandy On Mon, May 18, 2015 at 8:03 PM, xiaohe lan zombiexco...@gmail.com wrote: Hi Sandy, Thanks for your information. Yes, spark-submit --master yarn --num-executors 5 --executor-cores 4

Re: number of executors

Did you try --executor-cores param? While you submit the job, do a ps aux | grep spark-submit and see the exact command parameters. Thanks Best Regards On Sat, May 16, 2015 at 12:31 PM, xiaohe lan zombiexco...@gmail.com wrote: Hi, I have a 5 nodes yarn cluster, I used spark-submit to submit

Re: number of executors

Sorry, them both are assigned task actually. Aggregated Metrics by Executor Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded TasksInput Size / RecordsShuffle Write Size / RecordsShuffle Spill (Memory)Shuffle Spill (Disk)1host1:61841.7 min505640.0 MB / 12318400382.3 MB / 121007701630.4

Re: number of executors

bash-4.1$ ps aux | grep SparkSubmit xilan 1704 13.2 1.2 5275520 380244 pts/0 Sl+ 08:39 0:13 /scratch/xilan/jdk1.8.0_45/bin/java -cp

Re: number of executors

What Spark release are you using ? Can you check driver log to see if there is some clue there ? Thanks On Sat, May 16, 2015 at 12:01 AM, xiaohe lan zombiexco...@gmail.com wrote: Hi, I have a 5 nodes yarn cluster, I used spark-submit to submit a simple app. spark-submit --master yarn

Re: Number of Executors per worker process

process (JMV)? 2:Should I think executor like a Java Thread Executor where the pool size is equal with the number of the given cores (set up by the SPARK_WORKER_CORES)? 3. If the worker can have many executors, how this is handled by the Spark? Or can I handle by myself to set up the number

Re: Number of Executors per worker process

of the given cores (set up by the SPARK_WORKER_CORES)? 3. If the worker can have many executors, how this is handled by the Spark? Or can I handle by myself to set up the number of executors per JVM? I look forward for your answers. Regards, Florin

Number of Executors per worker process

by the SPARK_WORKER_CORES)? 3. If the worker can have many executors, how this is handled by the Spark? Or can I handle by myself to set up the number of executors per JVM? I look forward for your answers. Regards, Florin

Setting the number of executors in standalone mode

Hi there, I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that. I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores and did that: spark.executor.memory 1g

Re: Setting the number of executors in standalone mode

Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller moham...@glassbeam.com wrote: SPARK_WORKER_MEMORY=8g Will allocate 8GB

RE: Setting the number of executors in standalone mode

SPARK_WORKER_MEMORY=8g Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors. Mohammed From: Yiannis Gkoufas [mailto:johngou...@gmail.com] Sent: Friday, February 20, 2015 4:55 AM To: user@spark.apache.org Subject: Setting the number of executors in standalone

RE: Setting the number of executors in standalone mode

: Re: Setting the number of executors in standalone mode Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller moham

Re: Setting the number of executors in standalone mode

though. Mohammed *From:* Yiannis Gkoufas [mailto:johngou...@gmail.com] *Sent:* Friday, February 20, 2015 9:48 AM *To:* Mohammed Guller *Cc:* user@spark.apache.org *Subject:* Re: Setting the number of executors in standalone mode Hi Mohammed, thanks a lot for the reply. Ok, so from

Re: Problems with GC and time to execute with different number of executors.

configurations with YARN. yarn.nodemanager.resource.memory-mb 196Gb yarn.nodemanager.resource.cpu-vcores 24 I have tried to execute the job with different number of executors a memory (1-4g) With 20 executors takes 25s each iteration (128mb) and it never has a really long time waiting

Re: Problems with GC and time to execute with different number of executors.

. yarn.nodemanager.resource.memory-mb 196Gb yarn.nodemanager.resource.cpu-vcores 24 I have tried to execute the job with different number of executors a memory (1-4g) With 20 executors takes 25s each iteration (128mb) and it never has a really long time waiting because GC. When I

Re: Problems with GC and time to execute with different number of executors.

. yarn.nodemanager.resource.memory-mb 196Gb yarn.nodemanager.resource.cpu-vcores 24 I have tried to execute the job with different number of executors a memory (1-4g) With 20 executors takes 25s each iteration (128mb) and it never has a really long time waiting because GC. When I

Re: Problems with GC and time to execute with different number of executors.

/ 7physical disks) x 5 I have been trying many different configurations with YARN. yarn.nodemanager.resource.memory-mb 196Gb yarn.nodemanager.resource.cpu-vcores 24 I have tried to execute the job with different number of executors a memory (1-4g) With 20 executors takes 25s

Re: Problems with GC and time to execute with different number of executors.

processing a file of 80Gb in HDFS. I have 5 slaves: (32cores /256Gb / 7physical disks) x 5 I have been trying many different configurations with YARN. yarn.nodemanager.resource.memory-mb 196Gb yarn.nodemanager.resource.cpu-vcores 24 I have tried to execute the job with different number

Re: Problems with GC and time to execute with different number of executors.

configurations with YARN. yarn.nodemanager.resource.memory-mb 196Gb yarn.nodemanager.resource.cpu-vcores 24 I have tried to execute the job with different number of executors a memory (1-4g) With 20 executors takes 25s each iteration (128mb) and it never has a really long time waiting because GC. When

Problems with GC and time to execute with different number of executors.

the job with different number of executors a memory (1-4g) With 20 executors takes 25s each iteration (128mb) and it never has a really long time waiting because GC. When I execute around 60 executors the process time it's about 45s and some tasks take until one minute because GC. I have no idea why

Fwd: Controlling number of executors on Mesos vs YARN

Forgot to hit reply-all. -- Forwarded message -- From: Tim Chen t...@mesosphere.io Date: Sun, Jan 4, 2015 at 10:46 PM Subject: Re: Controlling number of executors on Mesos vs YARN To: mvle m...@us.ibm.com Hi Mike, You're correct there is no such setting in for Mesos coarse

Controlling number of executors on Mesos vs YARN

I'm trying to compare the performance of Spark running on Mesos vs YARN. However, I am having problems being able to configure the Spark workload to run in a similar way on Mesos and YARN. When running Spark on YARN, you can specify the number of executors per node. So if I have a node with 4

Re: Specifying number of executors in Mesos

Not that I am aware of. Spark will try to spread the tasks evenly across executors, its not aware of the workers at all. So if the executors to worker allocation is uneven, I am not sure what can be done. Maybe others can get smoe ideas. On Tue, Dec 9, 2014 at 6:20 AM, Gerard Maas

Re: Specifying number of executors in Mesos

Gerard, Are you familiar with spark.deploy.spreadOut http://spark.apache.org/docs/latest/spark-standalone.html in Standalone mode? It sounds like you want the same thing in Mesos mode. On Thu, Dec 11, 2014 at 6:48 AM, Tathagata Das tathagata.das1...@gmail.com wrote: Not that I am aware of.

Specifying number of executors in Mesos

Hi, We've a number of Spark Streaming /Kafka jobs that would benefit of an even spread of consumers over physical hosts in order to maximize network usage. As far as I can see, the Spark Mesos scheduler accepts resource offers until all required Mem + CPU allocation has been satisfied. This

Number of executors and tasks

Hi, I am running Spark in the stand alone mode. 1) I have a file of 286MB in HDFS (block size is 64MB) and so is split into 5 blocks. When I have the file in HDFS, 5 tasks are generated and so 5 files in the output. My understanding is that there will be a separate partition for each block and

Re: Number of executors and tasks

1. On HDFS files are treated as ~64mb in block size. When you put the same file in local file system (ext3/ext4) it will be treated as different (in your case it looks like ~32mb) and that's why you are seeing 9 output files. 2. You could set *num-executors *to increase the number of executor

  1   2   >