[
https://issues.apache.org/jira/browse/YARN-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18061113#comment-18061113
]
ASF GitHub Bot commented on YARN-11897:
---------------------------------------
Hean-Chhinling commented on code in PR #8227:
URL: https://github.com/apache/hadoop/pull/8227#discussion_r2854183081
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/NMResourceInfo.java:
##########
@@ -18,24 +18,15 @@
package org.apache.hadoop.yarn.server.nodemanager.webapp.dao;
+import
org.apache.hadoop.yarn.server.nodemanager.webapp.dao.gpu.NMGpuResourceInfo;
+
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlRootElement;
+import javax.xml.bind.annotation.XmlSeeAlso;
@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
+@XmlSeeAlso({NMGpuResourceInfo.class, NMDeviceResourceInfo.class})
public class NMResourceInfo {
-
- private long resourceValue;
Review Comment:
As I could see
[here](https://github.com/apache/hadoop/blob/db80c3a93760f6dbcd1421808c717cefcd93d63d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/ResourcePluginManager.java#L182)
when it tries to initialise a custom class, it has to implment the
DevicePlugin interface
Then it will create the
[DevicePluginAdapter](https://github.com/apache/hadoop/blob/db80c3a93760f6dbcd1421808c717cefcd93d63d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/ResourcePluginManager.java#L217),
when the `/resource/{resourcename}` endpoints call
[here](https://github.com/apache/hadoop/blob/db80c3a93760f6dbcd1421808c717cefcd93d63d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java#L548)
it will call the `DevicePluginAdapter.getNMResourceInf`o and as we could see
[here](https://github.com/apache/hadoop/blob/db80c3a93760f6dbcd1421808c717cefcd93d63d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/deviceframework/DevicePluginAdapter.java#L132)
it will return only `NMDeviceResourceInfo`.
I tested with the built in support `yarn.io/gpu` resource as well and it
returns with `NMGpuResourceInfo` wrap outside:
```
<nmGpuResourceInfo>
<gpuDeviceInformation>
<driver_version>Fake-Driver-99.99</driver_version>
<gpu>
<minor_number>0</minor_number>
<product_name>N/A</product_name>
<uuid>GPU-fake-1234-5678-9abc-def012345678</uuid>
</gpu>
</gpuDeviceInformation>
</nmGpuResourceInfo>
```
I tried to prove it by calling the `resource/{newresource}` but unable to
do so with the following errors.
I suspect with pluggable custom class it will return with
`NMDeviceResourceInfo` based on the source code mentioned above.
```
4:14:14.754 PM | INFO | ResourcePluginManager | New resource type:
custom.com/magic-device registered successfully by
org.apache.hadoop.yarn.server.nodemanager.webapp.dao.MagicDevicePlugin
4:14:14.757 PM | INFO | ResourcePluginManager | Adapter of
org.apache.hadoop.yarn.server.nodemanager.webapp.dao.MagicDevicePlugin created.
Initializing..
4:14:14.758 PM | INFO | DevicePluginAdapter | custom.com/magic-device plugin
adapter initialized
4:14:14.758 PM | INFO | ResourcePluginManager | Adapter of
org.apache.hadoop.yarn.server.nodemanager.webapp.dao.MagicDevicePlugin init
success!
```
```
SERVICE_UNAVAILABLE
java.lang.NullPointerException: Cannot invoke
"java.util.Collection.toArray()" because "c" is null
at java.base/java.util.ArrayList.<init>(ArrayList.java:181)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.deviceframework.DevicePluginAdapter.getNMResourceInfo(DevicePluginAdapter.java:124)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices.getNMResourceInfo(NMWebServices.java:551)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
```
In conclusion, the `/resource/{resourcename}` endpoint could only return
`NMGpuResourceInfo` for the built-in resource and `NMDeviceResourceInfo` for a
custom pluggable class.
> NodeManager REST API backward compatibility with Jersey1
> --------------------------------------------------------
>
> Key: YARN-11897
> URL: https://issues.apache.org/jira/browse/YARN-11897
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.5.0
> Reporter: Peter Szucs
> Assignee: chhinlinghean
> Priority: Major
> Labels: pull-request-available
>
> Jersey 2 upgrade broke NM REST API, so nodes applications page is not
> rendering in yarn UI v2 and also logs CLI cannot retrieve container logs.
> To fix this we need to:
> * introduce moxy in NM similar to YARN-11874
> * "/containers/\{containerid}/logs" endpoint returns a list, which will
> always be generated as a JSON array with Jersey 2. In Jersey 1 it returned a
> JSON object in case the list had a single element, and logs CLI expects a
> JSON object as well when calling this endpoint. That should be changed to
> keep backward compatibility. With returning a wrapper object instead of a
> list we could achieve the same behaviour.
> * Fix unit tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
