[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802739#comment-17802739 ] Shilun Fan commented on YARN-5683: -- Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a blocker. Retarget 3.5.0. > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: oct16-hard > Attachments: YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch, > flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories, and choose to not launch container if fallback > through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and > ENSURE_LOG_STORAGE_TYPE). > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories and fallback > strategy (MapReduce configurations: mapreduce.job.local-storage-type, > mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and > mapreduce.job.ensure-log-storage-type). > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce-2.png! > h3. Further Discussion > * Scheduling : The requirement of storage type for local/log directories may > not be satisfied for a part of nodes on heterogeneous clusters. To achieve > global optimum, scheduler should aware and manage disk resources. > ** Approach-1: Based on node attributes (YARN-3409), Scheduler can allocate > containers which have SSD requirement on nodes with attribute:ssd=true. > ** Approach-2: Based on extended resource model (YARN-3926), it's easy to > support scheduling through extending resource models like vdisk and vssd > using this feature, but hard to measure for applications and isolate for > non-CFQ based disks. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? We have implemented a fallback strategy that fail > to launch container when none of desired storage type disk
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080876#comment-17080876
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} YARN-5683 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25857/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Labels: oct16-hard
> Attachments: YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch,
> flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png
>
>
> h3. Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc)
> using local storage (checkpoint, shuffle etc) might require high IO
> performance. It's useful to allocate local directories to high performance
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications
> cannot selectively use storage media with different performance
> characteristics. Adding awareness of storage media can allow YARN to make
> better decisions about the placement of local directories.
> h3. Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
> should allow the cluster administrator to optionally specify the storage type
> for each local directories. Example:
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default
> storage type.
> ** StorageLocation separates storage type and directory path, used by
> LocalDirAllocator to aware the types of local dirs, the default storage type
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
> local directory of the specified storage type, and will fallback to not care
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch.
> All application frameworks can set the environment variables
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage
> type of local/log directories, and choose to not launch container if fallback
> through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and
> ENSURE_LOG_STORAGE_TYPE).
> * Allow specified storage type for various frameworks (Take MapReduce as an
> example)
> ** Add new configurations should allow application administrator to
> optionally specify the storage type of local/log directories and fallback
> strategy (MapReduce configurations: mapreduce.job.local-storage-type,
> mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and
> mapreduce.job.ensure-log-storage-type).
> ** Support for container work directories. Set the environment variables
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local
> directories of frameworks (such as shuffle directories for MapReduce).
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce-2.png!
> h3. Further Discussion
> * Scheduling : The requirement of storage type for local/log directories may
> not be satisfied for a part of nodes on heterogeneous clusters. To achieve
> global optimum, scheduler should aware and manage disk resources.
> ** Approach-1: Based on node attributes (YARN-3409), Scheduler can
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690339#comment-16690339
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-5683 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22588/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Labels: oct16-hard
> Attachments: YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch,
> flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png
>
>
> h3. Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc)
> using local storage (checkpoint, shuffle etc) might require high IO
> performance. It's useful to allocate local directories to high performance
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications
> cannot selectively use storage media with different performance
> characteristics. Adding awareness of storage media can allow YARN to make
> better decisions about the placement of local directories.
> h3. Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
> should allow the cluster administrator to optionally specify the storage type
> for each local directories. Example:
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default
> storage type.
> ** StorageLocation separates storage type and directory path, used by
> LocalDirAllocator to aware the types of local dirs, the default storage type
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
> local directory of the specified storage type, and will fallback to not care
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch.
> All application frameworks can set the environment variables
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage
> type of local/log directories, and choose to not launch container if fallback
> through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and
> ENSURE_LOG_STORAGE_TYPE).
> * Allow specified storage type for various frameworks (Take MapReduce as an
> example)
> ** Add new configurations should allow application administrator to
> optionally specify the storage type of local/log directories and fallback
> strategy (MapReduce configurations: mapreduce.job.local-storage-type,
> mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and
> mapreduce.job.ensure-log-storage-type).
> ** Support for container work directories. Set the environment variables
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local
> directories of frameworks (such as shuffle directories for MapReduce).
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce-2.png!
> h3. Further Discussion
> * Scheduling : The requirement of storage type for local/log directories may
> not be satisfied for a part of nodes on heterogeneous clusters. To achieve
> global optimum, scheduler should aware and manage disk resources.
> ** Approach-1: Based on node attributes (YARN-3409), Scheduler can
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594814#comment-16594814 ] Bibin A Chundatt commented on YARN-5683: [~Tao Yang] Any thoughts on updating design based on Node attributes YARN-3409?? > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: oct16-hard > Attachments: YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch, > flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories, and choose to not launch container if fallback > through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and > ENSURE_LOG_STORAGE_TYPE). > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories and fallback > strategy (MapReduce configurations: mapreduce.job.local-storage-type, > mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and > mapreduce.job.ensure-log-storage-type). > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce-2.png! > h3. Further Discussion > * Scheduling : The requirement of storage type for local/log directories may > not be satisfied for a part of nodes on heterogeneous clusters. To achieve > global optimum, scheduler should aware and manage disk resources. > ** Approach-1: Based on node attributes (YARN-3409), Scheduler can allocate > containers which have SSD requirement on nodes with attribute:ssd=true. > ** Approach-2: Based on extended resource model (YARN-3926), it's easy to > support scheduling through extending resource models like vdisk and vssd > using this feature, but hard to measure for applications and isolate for > non-CFQ based disks. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? We have implemented a fallback strategy that fail > to launch container when none of desired storage type disk are
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575900#comment-16575900
]
genericqa commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-5683 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21559/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Labels: oct16-hard
> Attachments: YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch,
> flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png
>
>
> h3. Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc)
> using local storage (checkpoint, shuffle etc) might require high IO
> performance. It's useful to allocate local directories to high performance
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications
> cannot selectively use storage media with different performance
> characteristics. Adding awareness of storage media can allow YARN to make
> better decisions about the placement of local directories.
> h3. Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
> should allow the cluster administrator to optionally specify the storage type
> for each local directories. Example:
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default
> storage type.
> ** StorageLocation separates storage type and directory path, used by
> LocalDirAllocator to aware the types of local dirs, the default storage type
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
> local directory of the specified storage type, and will fallback to not care
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch.
> All application frameworks can set the environment variables
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage
> type of local/log directories, and choose to not launch container if fallback
> through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and
> ENSURE_LOG_STORAGE_TYPE).
> * Allow specified storage type for various frameworks (Take MapReduce as an
> example)
> ** Add new configurations should allow application administrator to
> optionally specify the storage type of local/log directories and fallback
> strategy (MapReduce configurations: mapreduce.job.local-storage-type,
> mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and
> mapreduce.job.ensure-log-storage-type).
> ** Support for container work directories. Set the environment variables
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local
> directories of frameworks (such as shuffle directories for MapReduce).
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce-2.png!
> h3. Further Discussion
> * Scheduling : The requirement of storage type for local/log directories may
> not be satisfied for a part of nodes on heterogeneous clusters. To achieve
> global optimum, scheduler should aware and manage disk resources.
> ** Approach-1: Based on node attributes (YARN-3409), Scheduler
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575899#comment-16575899 ] Yechao Chen commented on YARN-5683: --- any updating？ > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: oct16-hard > Attachments: YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch, > flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories, and choose to not launch container if fallback > through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and > ENSURE_LOG_STORAGE_TYPE). > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories and fallback > strategy (MapReduce configurations: mapreduce.job.local-storage-type, > mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and > mapreduce.job.ensure-log-storage-type). > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce-2.png! > h3. Further Discussion > * Scheduling : The requirement of storage type for local/log directories may > not be satisfied for a part of nodes on heterogeneous clusters. To achieve > global optimum, scheduler should aware and manage disk resources. > ** Approach-1: Based on node attributes (YARN-3409), Scheduler can allocate > containers which have SSD requirement on nodes with attribute:ssd=true. > ** Approach-2: Based on extended resource model (YARN-3926), it's easy to > support scheduling through extending resource models like vdisk and vssd > using this feature, but hard to measure for applications and isolate for > non-CFQ based disks. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? We have implemented a fallback strategy that fail > to launch container when none of desired storage type disk are available. Is > there some better methods? > This feature has been used for half a
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184925#comment-16184925
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-5683 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17690/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
>Assignee: Tao Yang
> Labels: oct16-hard
> Attachments: flow_diagram_for_MapReduce-2.png,
> flow_diagram_for_MapReduce.png, YARN-5683-1.patch, YARN-5683-2.patch,
> YARN-5683-3.patch
>
>
> h3. Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc)
> using local storage (checkpoint, shuffle etc) might require high IO
> performance. It's useful to allocate local directories to high performance
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications
> cannot selectively use storage media with different performance
> characteristics. Adding awareness of storage media can allow YARN to make
> better decisions about the placement of local directories.
> h3. Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
> should allow the cluster administrator to optionally specify the storage type
> for each local directories. Example:
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default
> storage type.
> ** StorageLocation separates storage type and directory path, used by
> LocalDirAllocator to aware the types of local dirs, the default storage type
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
> local directory of the specified storage type, and will fallback to not care
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch.
> All application frameworks can set the environment variables
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage
> type of local/log directories, and choose to not launch container if fallback
> through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and
> ENSURE_LOG_STORAGE_TYPE).
> * Allow specified storage type for various frameworks (Take MapReduce as an
> example)
> ** Add new configurations should allow application administrator to
> optionally specify the storage type of local/log directories and fallback
> strategy (MapReduce configurations: mapreduce.job.local-storage-type,
> mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and
> mapreduce.job.ensure-log-storage-type).
> ** Support for container work directories. Set the environment variables
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local
> directories of frameworks (such as shuffle directories for MapReduce).
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce-2.png!
> h3. Further Discussion
> * Scheduling : The requirement of storage type for local/log directories may
> not be satisfied for a part of nodes on heterogeneous clusters. To achieve
> global optimum, scheduler should aware and manage disk resources.
> ** Approach-1: Based on node attributes (YARN-3409), Scheduler can allocate
> containers
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069372#comment-16069372
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} YARN-5683 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16278/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
>Assignee: Tao Yang
> Labels: oct16-hard
> Attachments: flow_diagram_for_MapReduce-2.png,
> flow_diagram_for_MapReduce.png, YARN-5683-1.patch, YARN-5683-2.patch,
> YARN-5683-3.patch
>
>
> h3. Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc)
> using local storage (checkpoint, shuffle etc) might require high IO
> performance. It's useful to allocate local directories to high performance
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications
> cannot selectively use storage media with different performance
> characteristics. Adding awareness of storage media can allow YARN to make
> better decisions about the placement of local directories.
> h3. Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
> should allow the cluster administrator to optionally specify the storage type
> for each local directories. Example:
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default
> storage type.
> ** StorageLocation separates storage type and directory path, used by
> LocalDirAllocator to aware the types of local dirs, the default storage type
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
> local directory of the specified storage type, and will fallback to not care
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch.
> All application frameworks can set the environment variables
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage
> type of local/log directories, and choose to not launch container if fallback
> through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and
> ENSURE_LOG_STORAGE_TYPE).
> * Allow specified storage type for various frameworks (Take MapReduce as an
> example)
> ** Add new configurations should allow application administrator to
> optionally specify the storage type of local/log directories and fallback
> strategy (MapReduce configurations: mapreduce.job.local-storage-type,
> mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and
> mapreduce.job.ensure-log-storage-type).
> ** Support for container work directories. Set the environment variables
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local
> directories of frameworks (such as shuffle directories for MapReduce).
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce-2.png!
> h3. Further Discussion
> * The requirement of storage type for local/log directories may not be
> satisfied on heterogeneous clusters. To achieve global optimum, scheduler
> should aware and manage disk resources.
> [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that
> but seems not support multiple storage
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831196#comment-15831196
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 57s{color} | {color:orange} root: The patch generated 7 new + 939 unchanged
- 9 fixed = 946 total (was 948) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 49s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
34s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
11s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
59s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}108m
26s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}249m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ca5f9793e4cb 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568971#comment-15568971
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 47s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 7s
{color} | {color:red} root: The patch generated 7 new + 940 unchanged - 9 fixed
= 947 total (was 949) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 11s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 49s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 33s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 57s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 34s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 205m 0s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch |
| JIRA Issue | YARN-5683 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 76b8f8dfda17 3.13.0-36-lowlatency #63-Ubuntu SMP
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565323#comment-15565323
]
Hadoop QA commented on YARN-5683:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 41s
{color} | {color:red} root: The patch generated 7 new + 940 unchanged - 9 fixed
= 947 total (was 949) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s
{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 15s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 19s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 31s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 46s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 45s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 197m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| |
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathWithStorageTypePrefix(String)
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565030#comment-15565030 ] Tao Yang commented on YARN-5683: Add YARN-5683-2.patch and update the description. Updates: (1) Fix problem: not choosing specified storage medium when local directories turned good. (2) Support fallback strategy that fail to launch container if none of desired storage type disk are available. > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang >Assignee: Tao Yang > Attachments: YARN-5683-1.patch, YARN-5683-2.patch, > flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories, and choose to not launch container if fallback > through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and > ENSURE_LOG_STORAGE_TYPE). > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories and fallback > strategy (MapReduce configurations: mapreduce.job.local-storage-type, > mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type and > mapreduce.job.ensure-log-storage-type). > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce-2.png! > h3. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and manage disk resources. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints > ([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409]) can also make a > higher chance to satisfy the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? We have implemented a fallback strategy that fail > to launch container when none of
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531569#comment-15531569 ] Tao Yang commented on YARN-5683: Thanks for comments, [~grey]. We think [YARN-3409|https://issues.apache.org/jira/browse/YARN-3409] is helpful for applications to be allocated on the required nodes and will meet the demands of some other requirements except this. We also pay attention to [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139]. SSD is tight on heterogeneous clusters and might be over-allocated without constraints. Take it as a resource may be a good solution and we will discuss the support of storage type in this feature later. > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories. > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories. (MapReduce add > configurations: mapreduce.job.local-storage-type and > mapreduce.job.log-storage-type) > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce.png! > h3. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and manage disk resources. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints > ([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409]) can also make a > higher chance to satisfy the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529614#comment-15529614 ] Lei Guo commented on YARN-5683: --- This is related the constraint label YARN-3409, but different at detail level. This will rely on the constraint label for initial scheduling to find server with proper resource, and then node manager to help to set up proper environment for the workload. It's not just storage, we will also face similar issue on GPU or other hardware accelerator. > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories. > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories. (MapReduce add > configurations: mapreduce.job.local-storage-type and > mapreduce.job.log-storage-type) > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce.png! > h3. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and management disk resources to. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints can also make a higher chance to satisfy > the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
