Github user mengxr commented on a diff in the pull request:
    --- Diff: 
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
    @@ -577,4 +577,17 @@ package object config {
           .checkValue(v => v > 0, "The value should be a positive time value.")
    +      .doc("Time in seconds to wait between a max concurrent tasks check 
failure and the next " +
    +        "check. A max concurrent tasks check ensures the cluster can 
launch more concurrent " +
    +        "tasks than required by a barrier stage on job submitted. The 
check can fail in case " +
    +        "a cluster has just started and not enough executors have 
registered, so we wait for a " +
    +        "little while and try to perform the check again. If the check 
fails consecutively for " +
    +        "three times for a job then fail current job submission. Note this 
config only applies " +
    +        "to jobs that contain one or more barrier stages, we won't perform 
the check on " +
    +        "non-barrier jobs.")
    +      .timeConf(TimeUnit.SECONDS)
    +      .createWithDefaultString("10s")
    --- End diff --
    Would you make the default higher like `30s`? This is to cover the case 
when applications starts immediately with a barrier while master is adding new 
executors. Let me know if this won't happen.


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to