Looks like that message would be triggered if
spark.dynamicAllocation.initialExecutors was not set, or 0, if I read
this right. Yeah, that might have to be positive. This requires you
set initial executors to 1 if you want 0 min executors. Hm, maybe that
shouldn't be an error condition in the args
On Thu, Apr 16, 2015 at 12:16:13PM -0700, Marcelo Vanzin wrote:
I think Michael is referring to this:
"""
Exception in thread "main" java.lang.IllegalArgumentException: You
must specify at least 1 executor!
Usage: org.apache.spark.deploy.yarn.Client [options]
"""
Yes, sorry, there were too man
I think Michael is referring to this:
"""
Exception in thread "main" java.lang.IllegalArgumentException: You
must specify at least 1 executor!
Usage: org.apache.spark.deploy.yarn.Client [options]
"""
spark-submit --conf spark.dynamicAllocation.enabled=true --conf
spark.dynamicAllocation.minExecut
On Thu, Apr 16, 2015 at 08:10:54PM +0100, Sean Owen wrote:
Yes, look what it was before -- would also reject a minimum of 0.
That's the case you are hitting. 0 is a fine minimum.
How can 0 be a fine minimum if it's rejected? Changing the value is easy
enough, but in general it's nice for defau
Yes, look what it was before -- would also reject a minimum of 0.
That's the case you are hitting. 0 is a fine minimum.
On Thu, Apr 16, 2015 at 8:09 PM, Michael Stone wrote:
> On Thu, Apr 16, 2015 at 07:47:51PM +0100, Sean Owen wrote:
>>
>> IIRC that was fixed already in 1.3
>>
>>
>> https://gith
On Thu, Apr 16, 2015 at 07:47:51PM +0100, Sean Owen wrote:
IIRC that was fixed already in 1.3
https://github.com/apache/spark/commit/b2047b55c5fc85de6b63276d8ab9610d2496e08b
From that commit:
+ private val minNumExecutors =
conf.getInt("spark.dynamicAllocation.minExecutors", 0)
...
+ if (max
IIRC that was fixed already in 1.3
https://github.com/apache/spark/commit/b2047b55c5fc85de6b63276d8ab9610d2496e08b
On Thu, Apr 16, 2015 at 7:41 PM, Michael Stone wrote:
> The default for spark.dynamicAllocation.minExecutors is 0, but that value
> causes a runtime error and a message that the min
