I can confirm that the suggested steps work on the headless cluster server
with a 3D X server running.
After installing xvfb and virtualgl into the container, I can run apps like
this:
VGL_DISPLAY=:0 singularity exec --nv /path/to/container/image.simg xvfb-run
-a -s '-screen 0 1x1x24' vglrun opengl_app
And that's it, no more hassle. With singularity, the key is to pass the
`--nv` argument which delegates the GPU drivers. X server authentication
doesn't need to be configured when using singularity as the container app
runs with the same privileges and UID as the regular user. I assume that
for docker there would be enough tutorials about setting up X auth.
Dne pondělí 24. srpna 2020 v 23:37:09 UTC+2 uživatel DRC napsal:
> OK. In that case, your general strategy will be:
>
> - On the host, set up the headless 3D X server on Display :0 with each GPU
> attached to a different screen.
>
> - Figure out how to share the 3D X server connection from the host to a
> container instance. My first approach would be to try sharing the files
> related to the 3D X server's Unix domain socket. Again, I have not
> personally experimented with this yet, so I do not yet know what issues
> might be encountered.
> - To launch an application in a container instance:
> - Launch Xvfb
> - export DISPLAY=:{d}.0 # {d} = X display of Xvfb instance
> - export VGL_DISPLAY=:0.{s} # {s} = screen number of desired GPU
> - vglrun {application}
>
>
> On 8/24/20 2:53 PM, Martin Pecka wrote:
>
> Ah, I'm sorry, it isn't evident from my first post - I'm not interested in
> transferring the "X buffer" anywhere. The apps we need to run only do
> offscreen rendering. But they need accelerated OpenGL for that, and can
> only do it through GLX (though EGL would be more appropriate). Does that
> help understanding the issue?
>
> If I get it correctly, people who need to see what X draws to the
> "onscreen buffer", will use TurboVNC, and people who only need offscreen
> rendering are good with Xvfb.
>
> Dne pondělí 24. srpna 2020 v 19:45:26 UTC+2 uživatel DRC napsal:
>
>> I don't understand how you plan to get the rendered pixels from the X
>> proxy to the client machine. Xvfb won't do that. You would need an X
>> proxy that has an image transport layer attached (TurboVNC, as an example,
>> is essentially Xvfb with an attached VNC server.) I also don't understand
>> why you're still mentioning vglclient, since that has no relevance to the
>> server-side configuration. I can almost guarantee you that the technical
>> specifics you listed below are not correct, but in order to correct them, I
>> need a better understanding of what you are ultimately trying to
>> accomplish. Let's bring the discussion up a level and develop a mutual
>> understanding of the proposed solution architecture before we get mired
>> down in the specifics of command lines and environment variables and such.
>>
>> I'm not sure, for instance, what you are expecting in terms of a window
>> popping up. That will never happen on the 3D X server, since VirtualGL is
>> redirecting all OpenGL rendering into off-screen Pbuffers. It would only
>> happen on the 2D X server, but again, if you're trying to use Xvfb as a 2D
>> X server, then I don't understand how you expect to get the pixels from
>> that X server to the client machine without an image transport layer.
>> Also, unless you change the DISPLAY environment variable, the application
>> will not display to the Xvfb instance anyhow.
>> On 8/24/20 10:29 AM, Martin Pecka wrote:
>>
>> Yes, I'm working with containers, only its Singularity HPC and not
>> Docker, which makes some things more complicated (and some less).
>>
>> So, to get back to the original question: If I don't run the vglclient,
>> are the correct steps the following?
>>
>> 1) Get the headless 3D X server running on :0 (should be quite easy once
>> the cluster admin agrees to do that)
>> 2) Each user runs:
>> xvfb-run -a -s "-screen 0 1920x1080x24" env VGL_DISPLAY=:0 vglrun
>> /path/to/app arg1 arg2 ...
>>
>> I tested this on my (non-headless) laptop and it seemed to work:
>>
>> No window popped up, so apparently :0 wasn't used. And nvidia-smi showed
>> that the GPU is being fully utilized. In glxgears, I get around 700 FPS on
>> my GeForce GTX 1050 via VGL compared to 3000 FPS with direct on-screen
>> rendering, but that's probably okay, we'll see when I test the full-blown
>> apps.
>>
>> If this approach is correct for this use-case, could I help making a part
>> of documentation from it? We could also add the server-side setup part
>> Jason posted.
>>
>> Dne pondělí 24. srpna 2020 v 16:57:49 UTC+2 uživatel DRC napsal:
>>
>>> Yes, that is exactly what VirtualGL does, but the VirtualGL Client is
>>> not a required component of VirtualGL. The VirtualGL Client is only used
>>> if you use the built-in VGL Transport, which is only useful in a remote X
>>> environment (i.e. when the 2D X server is on the client machine.) Most
>>> users use VirtualGL with an X proxy these days, in which case the VirtualGL
>>> Client is not used.
>>>
>>> Apart from that, it sounds like what you are trying to accomplish is the
>>> same as https://github.com/VirtualGL/virtualgl/issues/98: sharing the
>>> 3D X server connection from host to guest in a container environment such
>>> as Docker.
>>> On 8/24/20 9:15 AM, Martin Pecka wrote:
>>>
>>> As the GPUs are shared among more users, I find it useful to give a
>>> separate (but still accelerated) X display to each user. I suppose telling
>>> everybody to use :0 wouldn't end up very well (as long as everyone would be
>>> rendering offscreen, it'd work, but I can't guarantee that). So I thought
>>> that VirtualGL could be the thing that would guarantee that nobody will be
>>> rendering onscreen.
>>>
>>> Dne pondělí 24. srpna 2020 v 16:02:20 UTC+2 uživatel DRC napsal:
>>>
>>>> I don't fully understand what you're proposing. The 3D X server part
>>>> of
>>>> your proposal should be no problem, as long as you connect each GPU to
>>>> a
>>>> separate screen on that X server (presumably, the 3D X server would be
>>>> headless.) But why is the VirtualGL Client involved?
>>>>
>>>> Conceptually, it should be possible to share the 3D X server connection
>>>> with, say, a Docker container, but given the extremely limited
>>>> resources
>>>> of this project, I have thus far been unable to dedicate the time
>>>> toward
>>>> researching how best to accomplish that
>>>> (https://github.com/VirtualGL/virtualgl/issues/98).
>>>>
>>>> On 8/21/20 7:44 AM, Martin Pecka wrote:
>>>> > Hi, we're thinking about getting GLX support on our HPC cluster which
>>>> > (currently) is completely headless. The idea is that users should be
>>>> > able to run virtual containers which would be given access to HW
>>>> > rendering with OpenGL. EGL would be better, but we're stuck with OGRE
>>>> > rendering engine which doesn't have proper support for the nvidia
>>>> EGL.
>>>> >
>>>> > Could you comment on my idea? Is it a supported scenario?
>>>> >
>>>> > The multi-GPU server would run a single "3D X server", probably Xorg.
>>>> > It would also run the virtualgl client. Containers that want to do
>>>> > some OpenGL stuff would call a combination of xvfb and vglrun. I.e.
>>>> > the whole setup only works with a single machine, not a pair
>>>> connected
>>>> > via ssh -X.
>>>> >
>>>> > Is that possible? Is there a tutorial for this kind of setup?
>>>>
>>> --
>>>
>>> You received this message because you are subscribed to the Google
>>> Groups "VirtualGL User Discussion/Support" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/virtualgl-users/3973b3c2-f7ef-41b6-a580-14941b846b96n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/virtualgl-users/3973b3c2-f7ef-41b6-a580-14941b846b96n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "VirtualGL User Discussion/Support" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/virtualgl-users/d4c5899e-c602-4dc6-bf14-4a9d695255fan%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/virtualgl-users/d4c5899e-c602-4dc6-bf14-4a9d695255fan%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/virtualgl-users/d915ad0f-010b-4155-98f3-10147f589256n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/virtualgl-users/d915ad0f-010b-4155-98f3-10147f589256n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/virtualgl-users/e44c3005-08b1-4f1c-803e-ed0a3ba785d8n%40googlegroups.com.
