GitHub user szyszy opened a pull request:

    [SPARK-20327][CORE][YARN] Add CLI support for YARN custom resources, like 

    ## What changes were proposed in this pull request?
    This PR adds CLI support for YARN custom resources, e.g. GPUs and any other 
resources YARN defines.
    The custom resources are defined with Spark properties, no additional CLI 
arguments were introduced.
    The properties can be defined in the following form:
    **Driver resources, client mode:** 
    Format: `<resource-name>`
    The property name follows the naming convention of YARN AM cores / memory 
properties: ` and
    **Driver resources, cluster mode:** 
    Format: `spark.yarn.driver.resource.<resource-name>`
    The property name follows the naming convention of driver cores / memory 
properties: `spark.driver.memory and spark.driver.cores.`
    **Executor resources:** 
    Format: `spark.yarn.executor.resource.<resource-name>`
    The property name follows the naming convention of executor cores / memory 
properties: `spark.executor.memory / spark.executor.cores`.
    For the driver resources (cluster mode) and executor resources properties, 
we use the `yarn` prefix here as custom resource types are specific to YARN, 
    Please note that a validation logic is added to avoid having requested 
resources defined in 2 ways, for example defining the following configs:
    "--conf", "spark.driver.memory=2G",
    "--conf", "spark.yarn.driver.resource.memory=1G"
    will not start execution and prints an error message: 
    `spark.driver.memory and spark.yarn.driver.resource.memory configs are both 
present, only one of them is allowed at the same time!`
    ## How was this patch tested?
    Unit tests + manual execution.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-20327

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20761
commit 0bb46134ce61eee8658b40dc3f5598dcc83c34fa
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-14T11:36:14Z

    SPARK-20327. Introduce custom resource type configs and use new yarn 
resource api

commit 9a17a740d7b4700ecd3aebbce3cf4a42241698f7
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-14T11:45:06Z

    SPARK-20327. add yarn prefix to new resource type configs

commit e4c63b043d7c13959c65bc16229ff71bd0d714af
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-14T12:59:46Z

    SPARK-20327. add yarn prefix to constant

commit 6819f95add04ec9679827c569cd2d4d6ebf747a3
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-14T13:12:13Z

    SPARK-20327. add missing call to 

commit 825e95782c6df6ce27c2da7384c4cba6f91c9de3
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-14T17:18:54Z

    SPARK-20327. use reflection for all yarn api calls (hadoop 2.x 

commit b0667f6bc18473289a1065b2337641c9fecc8ee3
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-15T13:07:16Z

    SPARK-20327. add clarifying comment to SparkSubmitArguments

commit 99673ddc51fa112f5e13846f7a92b864bc264a49
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-15T14:22:38Z

    SPARK-20327. refactor SparkSubmitArguments: validate duplicate resource 
type configs

commit d20e94e928ca10e08fe19572cd3d2c2466dca417
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-15T14:44:48Z

    SPARK-20327. renamings, comment fixes

commit 6c9d5e086d87916ec505bed56d1b4d883fa698ac
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-20T14:11:19Z

    SPARK-20327. fix scalastyle issues

commit d44c4183ab4324eb159516171ee714f5a4c655be
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-22T13:59:17Z

    SPARK-20327. added debug log to yarn Client

commit 106bbde7db4efc07a85186e4df905e5af99cb9e5
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-22T14:31:02Z

    SPARK-20327. unit test fixes

commit f43d8d9f2ef25fb777bac5830f771e1de656684b
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-22T14:31:30Z

    SPARK-20327. improve resource type logging

commit 2117fa005185fe5d80e1366ebbb0978903b92954
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-22T16:09:13Z

    SPARK-20327. collect and print resource type error messages instead of 
printing the first

commit 0d41ff747b3b6d5d515103c9d2d4396a65b4ecc0
Author: Szilard Nemeth <snemeth@...>
Date:   2018-02-22T16:44:20Z

    SPARK-20327. documentation update

commit 48b6baa073559f65d5e1a785c6b2982d34fec54f
Author: Szilard Nemeth <snemeth@...>
Date:   2018-03-06T21:24:32Z

    SPARK-20327. test fixes



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to