[ 
https://issues.apache.org/jira/browse/MESOS-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575584#comment-16575584
 ] 

Gilbert Song commented on MESOS-8038:
-------------------------------------

[~bmahler], adding a configurable timeout does not address this issue though, 
the stucking time is really based on kernel (e.g., disk I/O). By configuring 
the timeout (say, hanging forever), would make the container process destroy 
stuck there, which is not good but at least better than leaking GPU resources 
to the next container who could not actually consume.

[~zhitao], I would like to discuss about how you guys expect to use this flag. 
Do we want to make it an optional duration? If NONE, no timeout. If we decide 
to change the default behavior, we should document it in upgrade.md and send an 
email to the dev/usr list.

> Launching GPU task sporadically fails.
> --------------------------------------
>
>                 Key: MESOS-8038
>                 URL: https://issues.apache.org/jira/browse/MESOS-8038
>             Project: Mesos
>          Issue Type: Bug
>          Components: allocation, containerization, gpu
>    Affects Versions: 1.4.0
>            Reporter: Sai Teja Ranuva
>            Assignee: Zhitao Li
>            Priority: Critical
>         Attachments: mesos-master.log, mesos-slave-with-issue-uber.txt, 
> mesos-slave.INFO.log
>
>
> I was running a job which uses GPUs. It runs fine most of the time. 
> But occasionally I see the following message in the mesos log.
> "Collect failed: Requested 1 but only 0 available"
> Followed by executor getting killed and the tasks getting lost. This happens 
> even before the the job starts. A little search in the code base points me to 
> something related to GPU resource being the probable cause.
> There is no deterministic way that this can be reproduced. It happens 
> occasionally.
> I have attached the slave log for the issue.
> Using 1.4.0 Mesos Master and 1.4.0 Mesos Slave.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to