[
https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908822#comment-16908822
]
Szilard Nemeth edited comment on YARN-9217 at 8/21/19 1:40 PM:
---------------------------------------------------------------
Hi [~pbacsko]!
Some comments on the latest patch:
1.
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer#getGpuDeviceInformation:
You added IOException to the javadoc and to the signature of the method but
this kind of exception is not thrown, apparently. Could you please verify this?
2. There's a message:
{code:java}
LOG.warn("Cannot properly parse GPU device numbers: {}", device);
{code}
I think this should be rephrased to "Cannot parse..." so the word "properly" is
superfluous.
3. In GpuDiscoverer#parseGpuDevice: Parameters "device" and "allowedDevicesStr"
is unused. Please check this code part!
4. In GpuResourcePlugin#getNMResourceInfo: catch blocks contain the same code
for YarnException and IOException. Please use multicatch.
5. For NvidiaBinaryHelper#MAX_EXEC_TIMEOUT_MS: Please use javadoc instead of
simple comment above.
6. NvidiaBinaryHelper#getGpuDeviceInformation can be package-private instead of
public.
7. In NvidiaBinaryHelper#getGpuDeviceInformation: We could use if
(pathOfGpuBinary == null) form for the null check.
was (Author: snemeth):
Hi [~pbacsko]!
Some comments on the latest patch:
1.
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer#getGpuDeviceInformation:
You added IOException to the javadoc and to the signature of the method but
this kind of exception is not thrown, apparently. Could you please verify this?
2. There's a message:
{code:java}
LOG.warn("Cannot properly parse GPU device numbers: {}", device);
{code}
I think this should be rephrased to "Cannot parse..." so the word "properly" is
superfluous.
3.
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer#parseGpuDevice:
Parameters "device" and "allowedDevicesStr" is unused. Please check this code
part!
4. In
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo:
catch blocks contain the same code for YarnException and IOException. Please
use multicatch.
5. For
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.NvidiaBinaryHelper#MAX_EXEC_TIMEOUT_MS:
Please use javadoc instead of simple comment above.
6.
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.NvidiaBinaryHelper#getGpuDeviceInformation
can be package-private instead of public.
7. In
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.NvidiaBinaryHelper#getGpuDeviceInformation:
We could use if (pathOfGpuBinary == null) form for the null check.
> Nodemanager will fail to start if GPU is misconfigured on the node or GPU
> drivers missing
> -----------------------------------------------------------------------------------------
>
> Key: YARN-9217
> URL: https://issues.apache.org/jira/browse/YARN-9217
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 3.0.0, 3.1.0
> Reporter: Antal Bálint Steinbach
> Assignee: Peter Bacsko
> Priority: Major
> Attachments: YARN-9217.001.patch, YARN-9217.002.patch,
> YARN-9217.003.patch, YARN-9217.004.patch, YARN-9217.005.patch,
> YARN-9217.006.patch, YARN-9217.007.patch, YARN-9217.008.patch,
> YARN-9217.009.patch, YARN-9217.010.patch, YARN-9217.011.patch,
> YARN-9217.012.patch, YARN-9217.branch-3.2.001.patch
>
>
> Nodemanager will not start
> 1. If Autodiscovery is enabled:
> * If nvidia-smi path is misconfigured or the file does not exist.
> * There is 0 GPU found
> * If the file exists but it is not pointing to an nvidia-smi
> * if the binary is ok but there is an IOException
> 2. If the manually configured GPU devices are misconfigured
> * Any index:minor number format failure will cause a problem
> * 0 configured device will cause a problem
> * NumberFormatException is not handled
> It would be a better option to add warnings about the configuration, set 0
> available GPUs and let the node work and run non-gpu jobs.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
