[jira] [Commented] (YARN-9116) Capacity Scheduler: add the default maximum-allocation-mb and maximum-allocation-vcores for the queues

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719676#comment-16719676 ] Wangda Tan commented on YARN-9116: -- [~aihuaxu],  This sounds like a plan, but existing maximum memory,

[jira] [Commented] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719658#comment-16719658 ] Wangda Tan commented on YARN-9055: -- [~aihuaxu], I agree with Thomas, this looks like a change of

[jira] [Commented] (YARN-9015) [DevicePlugin] Add an interface for device plugin to provide customized scheduler

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719376#comment-16719376 ] Wangda Tan commented on YARN-9015: -- Committed to trunk, thanks [~tangzhankun]! > [DevicePlugin] Add an

[jira] [Updated] (YARN-8885) [DevicePlugin] Support NM APIs to query device resource allocation

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8885: - Summary: [DevicePlugin] Support NM APIs to query device resource allocation (was: Phase 1 - Support NM

[jira] [Updated] (YARN-9015) [DevicePlugin] Add an interface for device plugin to provide customized scheduler

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-9015: - Summary: [DevicePlugin] Add an interface for device plugin to provide customized scheduler (was: Phase 1

[jira] [Commented] (YARN-9112) [Submarine] Support polling applicationId when it's not ready in cluster

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719349#comment-16719349 ] Wangda Tan commented on YARN-9112: -- LGTM, +1. Thanks [~tangzhankun]. > [Submarine] Support polling

[jira] [Commented] (YARN-8885) Phase 1 - Support NM APIs to query device resource allocation

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719347#comment-16719347 ] Wangda Tan commented on YARN-8885: -- Thanks [~tangzhankun], patch LGTM, will commit by today. > Phase 1 -

[jira] [Commented] (YARN-9078) [Submarine] Clean up the code of CliUtils#parseResourcesString

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719354#comment-16719354 ] Wangda Tan commented on YARN-9078: -- Change looks good. Thanks [~tangzhankun].  > [Submarine] Clean up

[jira] [Commented] (YARN-9015) Phase 1 - Add an interface for device plugin to provide customized scheduler

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719351#comment-16719351 ] Wangda Tan commented on YARN-9015: -- Thanks [~tangzhankun], latest patch LGTM, +1. > Phase 1 - Add an

[jira] [Commented] (YARN-9075) Dynamically add or remove auxiliary services

2018-12-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719307#comment-16719307 ] Wangda Tan commented on YARN-9075: -- Thanks [~billie.rinaldi],  The overall code flow looks good to me.

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-12-11 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718234#comment-16718234 ] Wangda Tan commented on YARN-8714: -- [~tangzhankun], sounds like a plan, but let's try to solve the issue

[jira] [Commented] (YARN-9087) Better logging for initialization of Resource plugins

2018-12-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711837#comment-16711837 ] Wangda Tan commented on YARN-9087: -- [~snemeth], Device plugin framework is for the future plugins. We

[jira] [Updated] (YARN-8822) Nvidia-docker v2 support

2018-12-05 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8822: - Priority: Critical (was: Major) > Nvidia-docker v2 support > > >

[jira] [Commented] (YARN-8822) Nvidia-docker v2 support

2018-12-05 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710646#comment-16710646 ] Wangda Tan commented on YARN-8822: -- [~Charo Zhang], Thanks for the patch, apologize missed this Jira. I

[jira] [Updated] (YARN-8822) Nvidia-docker v2 support

2018-12-05 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8822: - Fix Version/s: (was: 3.1.2) > Nvidia-docker v2 support > > >

[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts

2018-12-04 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709341#comment-16709341 ] Wangda Tan commented on YARN-8870: -- As we discussed offline, reverted the patch from branches. It's

[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts

2018-12-04 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8870: - Target Version/s: (was: 3.2.0) > [Submarine] Add submarine installation scripts >

[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts

2018-12-04 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8870: - Fix Version/s: (was: 3.2.0) > [Submarine] Add submarine installation scripts >

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-12-04 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709178#comment-16709178 ] Wangda Tan commented on YARN-8714: -- Thanks [~tangzhankun], what I remember is YARN doesn't support

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-12-03 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707602#comment-16707602 ] Wangda Tan commented on YARN-8714: -- [~liuxun323], fair enough.  [~tangzhankun], I think we can add a

[jira] [Commented] (YARN-9015) Phase 1 - Add an interface for device plugin to provide customized scheduler

2018-12-03 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707587#comment-16707587 ] Wangda Tan commented on YARN-9015: -- [~tangzhankun], 1) DevicePluginScheduler: Why use Integer instead

[jira] [Commented] (YARN-8885) Phase 1 - Support NM APIs to query device resource allocation

2018-12-03 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707577#comment-16707577 ] Wangda Tan commented on YARN-8885: -- [~tangzhankun], could u provide example output of the API? Thanks,

[jira] [Commented] (YARN-9078) [Submarine] Clean up the code of CliUtils#parseResourcesString

2018-12-03 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707574#comment-16707574 ] Wangda Tan commented on YARN-9078: -- [~tangzhankun], I'm wondering if {code} 82 if

[jira] [Commented] (YARN-9050) Usability improvements for scheduler activities

2018-12-03 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707553#comment-16707553 ] Wangda Tan commented on YARN-9050: -- [~Tao Yang], make sense to me. Once you figured out details, I can

[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts

2018-12-01 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706070#comment-16706070 ] Wangda Tan commented on YARN-8870: -- [~liuxun323], I figured out how to do it manually First you need to

[jira] [Commented] (YARN-9010) Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl

2018-11-29 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703996#comment-16703996 ] Wangda Tan commented on YARN-9010: -- Committed to trunk, thanks [~tangzhankun]. > Fix the incorrect

[jira] [Updated] (YARN-9010) Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl

2018-11-29 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-9010: - Priority: Major (was: Minor) > Fix the incorrect trailing slash deletion in constructor method of >

[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts

2018-11-29 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703895#comment-16703895 ] Wangda Tan commented on YARN-8870: -- That's my bad, [~liuxun323], could u work on an addendum patch to get

[jira] [Commented] (YARN-9050) Usability improvements for scheduler activities

2018-11-29 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703553#comment-16703553 ] Wangda Tan commented on YARN-9050: -- [~Tao Yang], thanks for filing the JIRA. The all issues you

[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation in native container-executor

2018-11-29 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703546#comment-16703546 ] Wangda Tan commented on YARN-9060: -- [~tangzhankun], explanation makes sense, and the issue about GPU

[jira] [Resolved] (YARN-8975) [Submarine] Use predefined Charset object StandardCharsets.UTF_8 instead of String "UTF-8"

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-8975. -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Committed to trunk, thanks

[jira] [Commented] (YARN-8989) Move DockerCommandPlugin volume related APIs' invocation from DockerLinuxContainerRuntime#prepareContainer to #launchContainer

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702480#comment-16702480 ] Wangda Tan commented on YARN-8989: -- LGTM, thanks [~tangzhankun], committing. > Move DockerCommandPlugin

[jira] [Updated] (YARN-8882) [YARN-8851] Add a shared device mapping manager (scheduler) for device plugins

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8882: - Summary: [YARN-8851] Add a shared device mapping manager (scheduler) for device plugins (was: Phase 1 -

[jira] [Commented] (YARN-9061) Improve the GPU/FPGA module log message of container-executor

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702451#comment-16702451 ] Wangda Tan commented on YARN-9061: -- +1, thanks [~tangzhankun], committing. > Improve the GPU/FPGA module

[jira] [Commented] (YARN-7277) Container Launch expand environment needs to consider bracket matching

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702450#comment-16702450 ] Wangda Tan commented on YARN-7277: -- [~tangzhankun], typically what you can do is add a new line or empty

[jira] [Comment Edited] (YARN-7277) Container Launch expand environment needs to consider bracket matching

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702450#comment-16702450 ] Wangda Tan edited comment on YARN-7277 at 11/28/18 10:30 PM: - [~tangzhankun],

[jira] [Comment Edited] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation in native container-executor

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702438#comment-16702438 ] Wangda Tan edited comment on YARN-9060 at 11/28/18 10:26 PM: - [~tangzhankun],

[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation in native container-executor

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702438#comment-16702438 ] Wangda Tan commented on YARN-9060: -- [~tangzhankun], just want to understand some high level

[jira] [Commented] (YARN-8882) Phase 1 - Add a shared device mapping manager for device plugin to use

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702429#comment-16702429 ] Wangda Tan commented on YARN-8882: -- Thanks [~tangzhankun], existing code looks good, committing .. >

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-11-28 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702424#comment-16702424 ] Wangda Tan commented on YARN-8714: -- Thanks [~tangzhankun] for working on the patch, several comments:

[jira] [Updated] (YARN-9030) Log aggregation changes to handle filesystems which do not support setting permissions

2018-11-21 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-9030: - Summary: Log aggregation changes to handle filesystems which do not support setting permissions (was:

[jira] [Commented] (YARN-8882) Phase 1 - Add a shared device mapping manager for device plugin to use

2018-11-21 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695450#comment-16695450 ] Wangda Tan commented on YARN-8882: -- [~tangzhankun], why rename "device-scheduler" to

[jira] [Commented] (YARN-9030) Log aggregation changes to handle filesystems which do not support permissions

2018-11-20 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693852#comment-16693852 ] Wangda Tan commented on YARN-9030: -- Thanks [~suma.shivaprasad], +1, will get it committed later today.

[jira] [Commented] (YARN-8881) [YARN-8851] Add basic pluggable device plugin framework

2018-11-19 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692007#comment-16692007 ] Wangda Tan commented on YARN-8881: -- [~tangzhankun], patch committed to trunk. Thanks for reviews from

[jira] [Updated] (YARN-8881) [YARN-8851] Add basic pluggable device plugin framework

2018-11-19 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8881: - Fix Version/s: 3.3.0 > [YARN-8851] Add basic pluggable device plugin framework >

[jira] [Updated] (YARN-8881) [YARN-8851] Add basic pluggable device plugin framework

2018-11-19 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8881: - Summary: [YARN-8851] Add basic pluggable device plugin framework (was: Phase 1 - Add basic pluggable

[jira] [Commented] (YARN-8960) [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment

2018-11-19 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691972#comment-16691972 ] Wangda Tan commented on YARN-8960: -- +1, committing, thanks [~yuan_zac]. > [Submarine] Can't get

[jira] [Updated] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8299: - Target Version/s: (was: 3.1.2) > Yarn Service Upgrade: Add GET APIs that returns instances matching

[jira] [Updated] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8299: - Fix Version/s: 3.1.2 > Yarn Service Upgrade: Add GET APIs that returns instances matching query > params

[jira] [Commented] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689823#comment-16689823 ] Wangda Tan commented on YARN-8299: -- Committing to branch-3.1 now .. > Yarn Service Upgrade: Add GET APIs

[jira] [Updated] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8299: - Priority: Critical (was: Major) > Yarn Service Upgrade: Add GET APIs that returns instances matching

[jira] [Commented] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689805#comment-16689805 ] Wangda Tan commented on YARN-8299: -- Reopened to backport to 3.1.2 > Yarn Service Upgrade: Add GET APIs

[jira] [Updated] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8299: - Target Version/s: 3.1.2 > Yarn Service Upgrade: Add GET APIs that returns instances matching query >

[jira] [Reopened] (YARN-8299) Yarn Service Upgrade: Add GET APIs that returns instances matching query params

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reopened YARN-8299: -- > Yarn Service Upgrade: Add GET APIs that returns instances matching query > params >

[jira] [Updated] (YARN-8779) Fix few discrepancies between YARN Service swagger spec and code

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8779: - Target Version/s: 3.2.0, 3.1.3 (was: 3.2.0, 3.1.2) > Fix few discrepancies between YARN Service swagger

[jira] [Updated] (YARN-8161) ServiceState FLEX should be removed

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8161: - Target Version/s: 3.2.0, 3.1.3 (was: 3.2.0, 3.1.2) > ServiceState FLEX should be removed >

[jira] [Updated] (YARN-8366) Expose debug log information when user intend to enable GPU without setting nvidia-smi path

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8366: - Target Version/s: 3.2.0, 3.1.3 (was: 3.2.0, 3.1.2) > Expose debug log information when user intend to

[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8986: - Fix Version/s: (was: 3.1.2) > publish all exposed ports to random ports when using bridge network >

[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8986: - Target Version/s: 3.1.3 (was: 3.1.2) > publish all exposed ports to random ports when using bridge

[jira] [Updated] (YARN-8552) [DS] Container report fails for distributed containers

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8552: - Target Version/s: 3.1.3 (was: 3.1.2) > [DS] Container report fails for distributed containers >

[jira] [Updated] (YARN-8509) Total pending resource calculation in preemption should use user-limit factor instead of minimum-user-limit-percent

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8509: - Target Version/s: 3.2.0, 3.1.3 (was: 3.2.0, 3.1.2) > Total pending resource calculation in preemption

[jira] [Updated] (YARN-8453) Additional Unit tests to verify queue limit and max-limit with multiple resource types

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8453: - Target Version/s: 3.0.4, 3.1.3 (was: 3.0.4, 3.1.2) > Additional Unit tests to verify queue limit and

[jira] [Updated] (YARN-8052) Move overwriting of service definition during flex to service master

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8052: - Target Version/s: 3.1.3 (was: 3.1.2) > Move overwriting of service definition during flex to service

[jira] [Updated] (YARN-8417) Should skip passing HDFS_HOME, HADOOP_CONF_DIR, JAVA_HOME, etc. to Docker container.

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8417: - Target Version/s: 3.1.3 (was: 3.1.2) > Should skip passing HDFS_HOME, HADOOP_CONF_DIR, JAVA_HOME, etc.

[jira] [Updated] (YARN-8657) User limit calculation should be read-lock-protected within LeafQueue

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8657: - Target Version/s: 3.2.1, 3.1.3 (was: 3.1.2, 3.2.1) > User limit calculation should be

[jira] [Updated] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8234: - Target Version/s: 3.1.3 (was: 3.1.2) > Improve RM system metrics publisher's performance by pushing

[jira] [Updated] (YARN-8257) Native service should automatically adding escapes for environment/launch cmd before sending to YARN

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8257: - Target Version/s: 3.1.3 (was: 3.1.2) > Native service should automatically adding escapes for

[jira] [Commented] (YARN-9030) Log aggregation changes to handle filesystems which do not support permissions

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689755#comment-16689755 ] Wangda Tan commented on YARN-9030: -- [~suma.shivaprasad], it seems the logic of verifyAndCreateRemoteDir

[jira] [Commented] (YARN-8881) Phase 1 - Add basic pluggable device plugin framework

2018-11-16 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689699#comment-16689699 ] Wangda Tan commented on YARN-8881: -- +1 to the latest patch, will commit later today if no objections.

[jira] [Updated] (YARN-8917) Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource

2018-11-14 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8917: - Target Version/s: 3.2.1 > Absolute (maximum) capacity of level3+ queues is wrongly calculated for >

[jira] [Updated] (YARN-8917) Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource

2018-11-14 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8917: - Priority: Critical (was: Major) > Absolute (maximum) capacity of level3+ queues is wrongly calculated

[jira] [Resolved] (YARN-9020) set a wrong AbsoluteCapacity when call ParentQueue#setAbsoluteCapacity

2018-11-14 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-9020. -- Resolution: Duplicate Thanks [~jutia] for reporting this. It is a valid issue. This is dup of

[jira] [Commented] (YARN-8917) Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource

2018-11-14 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686903#comment-16686903 ] Wangda Tan commented on YARN-8917: -- This JIRA somehow dropped from our radar, retriggering Jenkins job

[jira] [Assigned] (YARN-6223) [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation on YARN

2018-11-14 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned YARN-6223: Assignee: Wangda Tan (was: Antal Bálint Steinbach) > [Umbrella] Natively support GPU

[jira] [Commented] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685750#comment-16685750 ] Wangda Tan commented on YARN-9001: -- Pushed to trunk, but backport to branch-3.2 failed, [~yuan_zac], if

[jira] [Updated] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-9001: - Fix Version/s: 3.3.0 > [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs >

[jira] [Comment Edited] (YARN-8960) [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685618#comment-16685618 ] Wangda Tan edited comment on YARN-8960 at 11/13/18 7:50 PM: Thanks

[jira] [Commented] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685635#comment-16685635 ] Wangda Tan commented on YARN-9001: -- Rebased to latest trunk to run Jenkins. > [Submarine] Use

[jira] [Updated] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-9001: - Attachment: YARN-9001.005.patch > [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

[jira] [Commented] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685631#comment-16685631 ] Wangda Tan commented on YARN-9001: -- Thanks [~yuan_zac], +1, committing the patch. > [Submarine] Use

[jira] [Commented] (YARN-8960) [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685618#comment-16685618 ] Wangda Tan commented on YARN-8960: -- Thanks [~yuan_zac], Some comments: 1) doLoginIfSecure, could u

[jira] [Updated] (YARN-8960) [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8960: - Description: After submitting a submarine job, we tried to get service status using the following

[jira] [Commented] (YARN-8881) Phase 1 - Add basic pluggable device plugin framework

2018-11-13 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685605#comment-16685605 ] Wangda Tan commented on YARN-8881: -- Thanks [~tangzhankun], Regarding Integer vs. int, I would suggest

[jira] [Commented] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs

2018-11-12 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684468#comment-16684468 ] Wangda Tan commented on YARN-9001: -- [~yuan_zac], checked the patch, in general patch looks good, could u

[jira] [Created] (YARN-8993) [Submarine] Add support to run deep learning workload in non-Docker containers

2018-11-08 Thread Wangda Tan (JIRA)
Wangda Tan created YARN-8993: Summary: [Submarine] Add support to run deep learning workload in non-Docker containers Key: YARN-8993 URL: https://issues.apache.org/jira/browse/YARN-8993 Project: Hadoop

[jira] [Commented] (YARN-8877) Extend service spec to allow setting resource attributes

2018-11-08 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680617#comment-16680617 ] Wangda Tan commented on YARN-8877: -- [~cheersyang], make sense to me. > Extend service spec to allow

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-11-08 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680615#comment-16680615 ] Wangda Tan commented on YARN-8714: -- [~tangzhankun] , I'm still not quite sure about: {code:java}

[jira] [Commented] (YARN-8960) [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment

2018-11-08 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680601#comment-16680601 ] Wangda Tan commented on YARN-8960: -- [~yuan_zac] , as we discussed offline, do we still need the service

[jira] [Updated] (YARN-8135) Hadoop {Submarine} Project: Simple and scalable deployment of deep learning training / serving jobs on Hadoop

2018-11-08 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8135: - Description: Description: *Goals:* - Allow infra engineer / data scientist to run *unmodified*

[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet

2018-11-08 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680570#comment-16680570 ] Wangda Tan commented on YARN-8763: -- [~sunilg] , I highly suggest reverting this from branch-3.2 if

[jira] [Updated] (YARN-8220) Running Tensorflow on YARN with GPU and Docker - Examples

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8220: - Description: -Tensorflow could be run on YARN and could leverage YARN's distributed features.- -This

[jira] [Updated] (YARN-8237) mxnet yarn spec file to add to native service examples

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8237: - Description: Mxnet -could be run on YARN. This- jira -will help to add examples,- yarnfile-, docker

[jira] [Updated] (YARN-8238) [Umbrella] YARN deep learning framework examples to run on native service

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8238: - Description: -Umbrella- jira -to track various deep learning frameworks which can run on yarn native

[jira] [Resolved] (YARN-8237) mxnet yarn spec file to add to native service examples

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-8237. -- Resolution: Duplicate > mxnet yarn spec file to add to native service examples >

[jira] [Resolved] (YARN-8238) [Umbrella] YARN deep learning framework examples to run on native service

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-8238. -- Resolution: Fixed Closing as dup of YARN-8135.  > [Umbrella] YARN deep learning framework examples to

[jira] [Commented] (YARN-8877) Extend service spec to allow setting resource attributes

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677039#comment-16677039 ] Wangda Tan commented on YARN-8877: -- [~cheersyang],  If YARN-8940 will satisfy all needs for volume,

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677029#comment-16677029 ] Wangda Tan commented on YARN-8714: -- [~tangzhankun] , could u please explain a little bit about what does

[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677030#comment-16677030 ] Wangda Tan commented on YARN-8714: -- + [~liuxun323] / [~yuan_zac] to take a look at this as well. >

[jira] [Commented] (YARN-8902) Add volume manager that manages CSI volume lifecycle

2018-11-06 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677027#comment-16677027 ] Wangda Tan commented on YARN-8902: -- {quote}I prefer not to do this rename. As the package already has 

[jira] [Commented] (YARN-8858) CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.

2018-11-05 Thread Wangda Tan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676232#comment-16676232 ] Wangda Tan commented on YARN-8858: -- Thanks [~cheersyang] / [~ajisakaa] for rebasing and committing the

  1   2   3   4   5   6   7   8   9   10   >