This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch YARN-8200.branch3
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard 4d88992 YARN-7033. Add support for NM Recovery of assigned resources
(e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan)
discard edcaaf2 YARN-7223. Document GPU isolation feature. Contributed by
Wangda Tan.
discard 9a66cb0 YARN-6620. Add support in NodeManager to isolate GPU devices
by using CGroups. Contributed by Wangda Tan.
add eb1f44e HADOOP-16025. Update the year to 2019.
add f17ef6f HADOOP-15323. AliyunOSS: Improve copy file performance for
AliyunOSSFileSystemStore. Contributed wujinhu.
add c4e39e3 YARN-9164. Shutdown NM may cause NPE when opportunistic
container scheduling is enabled. Contributed by lujie.
add a0291a0 YARN-9175. Null resources check in ResourceInfo for branch-3.0
new 1628177 YARN-6620. Add support in NodeManager to isolate GPU devices
by using CGroups. Contributed by Wangda Tan.
new 37434c2 YARN-7223. Document GPU isolation feature. Contributed by
Wangda Tan.
new d513823 YARN-7033. Add support for NM Recovery of assigned resources
(e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan)
new 1efe958 Partial commit of YARN-7224 to get GpuDevice class and changes
new 51ca26e YARN-7330. Add support to show GPU in UI including metrics.
Contributed by Wangda Tan.
new d5d9795 YARN-7573. Gpu Information page could be empty for nodes
without GPU. (Sunil G via wangda)
new 99dc1bc YARN-7637. GPU volume creation command fails when work
preserving is disabled at NM. Contributed by Zian Chen.
new 0d150c6 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on
trunk. Contributed by Gergely Novák.
new 5f500f2 YARN-7396. NPE when accessing container logs due to null
dirsHandler. Contributed by Jonathan Hung
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4d88992)
\
N -- N -- N refs/heads/YARN-8200.branch3 (5f500f2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
hadoop-project/pom.xml | 2 +-
.../fs/aliyun/oss/AliyunOSSCopyFileTask.java | 7 +-
.../hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java | 10 +-
.../fs/aliyun/oss/AliyunOSSFileSystemStore.java | 31 +---
.../src/site/markdown/tools/hadoop-aliyun/index.md | 4 +-
.../aliyun/oss/TestAliyunOSSBlockOutputStream.java | 1 -
.../oss/TestAliyunOSSFileSystemContract.java | 14 +-
.../aliyun/oss/TestAliyunOSSFileSystemStore.java | 10 +-
.../oss/contract/TestAliyunOSSContractDistCp.java | 1 -
.../hadoop-yarn/dev-support/findbugs-exclude.xml | 8 +
.../apache/hadoop/yarn/api/records/Resource.java | 21 +++
.../yarn/server/nodemanager/NodeManager.java | 3 +-
.../container/ResourceMappings.java | 2 +-
.../linux/resources/gpu/GpuResourceAllocator.java | 119 ++++++------
.../resources/gpu/GpuResourceHandlerImpl.java | 27 ++-
.../resourceplugin/ResourcePlugin.java | 11 ++
.../resourceplugin/gpu/AssignedGpuDevice.java | 79 ++++++++
.../resourceplugin/gpu/GpuDevice.java | 78 ++++++++
.../resourceplugin/gpu/GpuDiscoverer.java | 30 ++-
.../gpu/GpuNodeResourceUpdateHandler.java | 10 +-
.../resourceplugin/gpu/GpuResourcePlugin.java | 24 ++-
.../recovery/NMLeveldbStateStoreService.java | 67 ++++---
.../recovery/NMNullStateStoreService.java | 6 +-
.../nodemanager/recovery/NMStateStoreService.java | 17 +-
.../server/nodemanager/webapp/NMWebServices.java | 26 +++
.../nodemanager/webapp/dao/NMResourceInfo.java} | 14 +-
.../webapp/dao/gpu/GpuDeviceInformation.java | 2 +-
.../webapp/dao/gpu/NMGpuResourceInfo.java | 71 +++++++
.../webapp/dao/gpu/PerGpuDeviceInformation.java | 2 +-
.../webapp/dao/gpu/PerGpuMemoryUsage.java | 2 +-
.../TestContainerManagerRecovery.java | 15 +-
.../resources/gpu/TestGpuResourceHandler.java | 204 +++++++++++++++------
.../resourceplugin/gpu/TestGpuDiscoverer.java | 34 +++-
.../recovery/NMMemoryStateStoreService.java | 7 +-
.../recovery/TestNMLeveldbStateStoreService.java | 28 ++-
.../nodemanager/webapp/TestNMWebServices.java | 181 ++++++++++++++----
.../dao/gpu/TestGpuDeviceInformationParser.java | 2 +-
.../OpportunisticContainerAllocatorAMService.java | 8 +-
.../scheduler/AbstractYarnScheduler.java | 12 +-
.../resourcemanager/scheduler/SchedulerUtils.java | 8 +-
.../resourcemanager/webapp/dao/ResourceInfo.java | 5 +-
...stOpportunisticContainerAllocatorAMService.java | 78 ++++++++
.../app/adapters/{yarn-node.js => yarn-nm-gpu.js} | 4 +-
.../src/main/webapp/app/components/donut-chart.js | 18 +-
.../main/webapp/app/components/gpu-donut-chart.js | 66 +++++++
.../src/main/webapp/app/constants.js | 13 ++
.../webapp/app/controllers/yarn-nodes/table.js | 2 +-
.../src/main/webapp/app/models/cluster-metric.js | 69 +++++++
.../app/models/{yarn-entity.js => yarn-nm-gpu.js} | 9 +-
.../webapp/app/models/yarn-queue/capacity-queue.js | 3 +-
.../src/main/webapp/app/models/yarn-rm-node.js | 37 ++++
.../hadoop-yarn-ui/src/main/webapp/app/router.js | 5 +-
.../src/main/webapp/app/routes/cluster-overview.js | 2 +-
.../src/main/webapp/app/routes/yarn-node.js | 2 +
.../table.js => yarn-node/yarn-nm-gpu.js} | 0
.../{yarn-container-log.js => yarn-nm-gpu.js} | 25 ++-
.../app/serializers/yarn-queue/capacity-queue.js | 1 +
.../main/webapp/app/serializers/yarn-rm-node.js | 4 +-
.../main/webapp/app/templates/cluster-overview.hbs | 88 ++++++---
.../app/templates/components/node-menu-panel.hbs | 10 +-
.../app/templates/components/yarn-nm-gpu-info.hbs | 69 +++++++
.../main/webapp/app/templates/yarn-node-apps.hbs | 2 +-
.../webapp/app/templates/yarn-node-containers.hbs | 2 +-
.../src/main/webapp/app/templates/yarn-node.hbs | 125 -------------
.../main/webapp/app/templates/yarn-node/info.hbs | 154 ++++++++++++++++
.../yarn-nm-gpu.hbs} | 52 +++---
.../src/main/webapp/app/utils/converter.js | 51 ++++++
67 files changed, 1590 insertions(+), 504 deletions(-)
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/AssignedGpuDevice.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDevice.java
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/{hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationDeleteResponseInfo.java
=>
hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/NMResourceInfo.java}
(77%)
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/gpu/NMGpuResourceInfo.java
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/{yarn-node.js
=> yarn-nm-gpu.js} (93%)
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/gpu-donut-chart.js
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/{yarn-entity.js
=> yarn-nm-gpu.js} (88%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/{yarn-nodes/table.js
=> yarn-node/yarn-nm-gpu.js} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/{yarn-container-log.js
=> yarn-nm-gpu.js} (73%)
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/components/yarn-nm-gpu-info.hbs
delete mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-node.hbs
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-node/info.hbs
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/{yarn-node-apps.hbs
=> yarn-node/yarn-nm-gpu.hbs} (50%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
