Re: Is the executor number fixed during the lifetime of one app ?
It depends on how you use Spark, if you use Spark with Yarn and enable dynamic allocation, the number of executor is not fixed, will change dynamically according to the load. Thanks Jerry 2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com: It seems the executor number is fixed for the standalone mode, not sure other modes.
Re: Is the executor number fixed during the lifetime of one app ?
How does the dynamic allocation works ? I mean does it related with parallelism of my RDD and how does driver know how many executor it needs ? On Wed, May 27, 2015 at 2:49 PM, Saisai Shao sai.sai.s...@gmail.com wrote: It depends on how you use Spark, if you use Spark with Yarn and enable dynamic allocation, the number of executor is not fixed, will change dynamically according to the load. Thanks Jerry 2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com: It seems the executor number is fixed for the standalone mode, not sure other modes.
Re: Is the executor number fixed during the lifetime of one app ?
The drive has a heuristic mechanism to decide the number of executors in the run-time according the pending tasks. You could enable with configuration, you could refer to spark document to find the details. 2015-05-27 15:00 GMT+08:00 canan chen ccn...@gmail.com: How does the dynamic allocation works ? I mean does it related with parallelism of my RDD and how does driver know how many executor it needs ? On Wed, May 27, 2015 at 2:49 PM, Saisai Shao sai.sai.s...@gmail.com wrote: It depends on how you use Spark, if you use Spark with Yarn and enable dynamic allocation, the number of executor is not fixed, will change dynamically according to the load. Thanks Jerry 2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com: It seems the executor number is fixed for the standalone mode, not sure other modes.
Re: Is the executor number fixed during the lifetime of one app ?
If with mesos, how do we control the number of executors? In our cluster, each node only has one executor with very big JVM. Sometimes, if the executor dies, all the concurrent running tasks will be gone. We would like to have multiple executors in one node but can not figure out a way to do it in Yarn. On Wednesday, May 27, 2015, Saisai Shao sai.sai.s...@gmail.com wrote: The drive has a heuristic mechanism to decide the number of executors in the run-time according the pending tasks. You could enable with configuration, you could refer to spark document to find the details. 2015-05-27 15:00 GMT+08:00 canan chen ccn...@gmail.com javascript:_e(%7B%7D,'cvml','ccn...@gmail.com');: How does the dynamic allocation works ? I mean does it related with parallelism of my RDD and how does driver know how many executor it needs ? On Wed, May 27, 2015 at 2:49 PM, Saisai Shao sai.sai.s...@gmail.com javascript:_e(%7B%7D,'cvml','sai.sai.s...@gmail.com'); wrote: It depends on how you use Spark, if you use Spark with Yarn and enable dynamic allocation, the number of executor is not fixed, will change dynamically according to the load. Thanks Jerry 2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com javascript:_e(%7B%7D,'cvml','ccn...@gmail.com');: It seems the executor number is fixed for the standalone mode, not sure other modes. -- Sent from my iPhone
Re: Is the executor number fixed during the lifetime of one app ?
Typo. We can not figure a way to increase the number of executor in one node in mesos. On Wednesday, May 27, 2015, DB Tsai dbt...@dbtsai.com wrote: If with mesos, how do we control the number of executors? In our cluster, each node only has one executor with very big JVM. Sometimes, if the executor dies, all the concurrent running tasks will be gone. We would like to have multiple executors in one node but can not figure out a way to do it in Yarn. On Wednesday, May 27, 2015, Saisai Shao sai.sai.s...@gmail.com javascript:_e(%7B%7D,'cvml','sai.sai.s...@gmail.com'); wrote: The drive has a heuristic mechanism to decide the number of executors in the run-time according the pending tasks. You could enable with configuration, you could refer to spark document to find the details. 2015-05-27 15:00 GMT+08:00 canan chen ccn...@gmail.com: How does the dynamic allocation works ? I mean does it related with parallelism of my RDD and how does driver know how many executor it needs ? On Wed, May 27, 2015 at 2:49 PM, Saisai Shao sai.sai.s...@gmail.com wrote: It depends on how you use Spark, if you use Spark with Yarn and enable dynamic allocation, the number of executor is not fixed, will change dynamically according to the load. Thanks Jerry 2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com: It seems the executor number is fixed for the standalone mode, not sure other modes. -- Sent from my iPhone -- Sent from my iPhone
Re: Is the executor number fixed during the lifetime of one app ?
I'm not sure about Mesos, maybe someone has the Mesos experience can help answer this. Thanks Jerry 2015-05-27 15:21 GMT+08:00 DB Tsai dbt...@dbtsai.com: Typo. We can not figure a way to increase the number of executor in one node in mesos. On Wednesday, May 27, 2015, DB Tsai dbt...@dbtsai.com wrote: If with mesos, how do we control the number of executors? In our cluster, each node only has one executor with very big JVM. Sometimes, if the executor dies, all the concurrent running tasks will be gone. We would like to have multiple executors in one node but can not figure out a way to do it in Yarn. On Wednesday, May 27, 2015, Saisai Shao sai.sai.s...@gmail.com wrote: The drive has a heuristic mechanism to decide the number of executors in the run-time according the pending tasks. You could enable with configuration, you could refer to spark document to find the details. 2015-05-27 15:00 GMT+08:00 canan chen ccn...@gmail.com: How does the dynamic allocation works ? I mean does it related with parallelism of my RDD and how does driver know how many executor it needs ? On Wed, May 27, 2015 at 2:49 PM, Saisai Shao sai.sai.s...@gmail.com wrote: It depends on how you use Spark, if you use Spark with Yarn and enable dynamic allocation, the number of executor is not fixed, will change dynamically according to the load. Thanks Jerry 2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com: It seems the executor number is fixed for the standalone mode, not sure other modes. -- Sent from my iPhone -- Sent from my iPhone
