I can confirm that the suggested steps work on the headless cluster server with a 3D X server running.
After installing xvfb and virtualgl into the container, I can run apps like this: VGL_DISPLAY=:0 singularity exec --nv /path/to/container/image.simg xvfb-run -a -s '-screen 0 1x1x24' vglrun opengl_app And that's it, no more hassle. With singularity, the key is to pass the `--nv` argument which delegates the GPU drivers. X server authentication doesn't need to be configured when using singularity as the container app runs with the same privileges and UID as the regular user. I assume that for docker there would be enough tutorials about setting up X auth. Dne pondělí 24. srpna 2020 v 23:37:09 UTC+2 uživatel DRC napsal: > OK. In that case, your general strategy will be: > > - On the host, set up the headless 3D X server on Display :0 with each GPU > attached to a different screen. > > - Figure out how to share the 3D X server connection from the host to a > container instance. My first approach would be to try sharing the files > related to the 3D X server's Unix domain socket. Again, I have not > personally experimented with this yet, so I do not yet know what issues > might be encountered. > - To launch an application in a container instance: > - Launch Xvfb > - export DISPLAY=:{d}.0 # {d} = X display of Xvfb instance > - export VGL_DISPLAY=:0.{s} # {s} = screen number of desired GPU > - vglrun {application} > > > On 8/24/20 2:53 PM, Martin Pecka wrote: > > Ah, I'm sorry, it isn't evident from my first post - I'm not interested in > transferring the "X buffer" anywhere. The apps we need to run only do > offscreen rendering. But they need accelerated OpenGL for that, and can > only do it through GLX (though EGL would be more appropriate). Does that > help understanding the issue? > > If I get it correctly, people who need to see what X draws to the > "onscreen buffer", will use TurboVNC, and people who only need offscreen > rendering are good with Xvfb. > > Dne pondělí 24. srpna 2020 v 19:45:26 UTC+2 uživatel DRC napsal: > >> I don't understand how you plan to get the rendered pixels from the X >> proxy to the client machine. Xvfb won't do that. You would need an X >> proxy that has an image transport layer attached (TurboVNC, as an example, >> is essentially Xvfb with an attached VNC server.) I also don't understand >> why you're still mentioning vglclient, since that has no relevance to the >> server-side configuration. I can almost guarantee you that the technical >> specifics you listed below are not correct, but in order to correct them, I >> need a better understanding of what you are ultimately trying to >> accomplish. Let's bring the discussion up a level and develop a mutual >> understanding of the proposed solution architecture before we get mired >> down in the specifics of command lines and environment variables and such. >> >> I'm not sure, for instance, what you are expecting in terms of a window >> popping up. That will never happen on the 3D X server, since VirtualGL is >> redirecting all OpenGL rendering into off-screen Pbuffers. It would only >> happen on the 2D X server, but again, if you're trying to use Xvfb as a 2D >> X server, then I don't understand how you expect to get the pixels from >> that X server to the client machine without an image transport layer. >> Also, unless you change the DISPLAY environment variable, the application >> will not display to the Xvfb instance anyhow. >> On 8/24/20 10:29 AM, Martin Pecka wrote: >> >> Yes, I'm working with containers, only its Singularity HPC and not >> Docker, which makes some things more complicated (and some less). >> >> So, to get back to the original question: If I don't run the vglclient, >> are the correct steps the following? >> >> 1) Get the headless 3D X server running on :0 (should be quite easy once >> the cluster admin agrees to do that) >> 2) Each user runs: >> xvfb-run -a -s "-screen 0 1920x1080x24" env VGL_DISPLAY=:0 vglrun >> /path/to/app arg1 arg2 ... >> >> I tested this on my (non-headless) laptop and it seemed to work: >> >> No window popped up, so apparently :0 wasn't used. And nvidia-smi showed >> that the GPU is being fully utilized. In glxgears, I get around 700 FPS on >> my GeForce GTX 1050 via VGL compared to 3000 FPS with direct on-screen >> rendering, but that's probably okay, we'll see when I test the full-blown >> apps. >> >> If this approach is correct for this use-case, could I help making a part >> of documentation from it? We could also add the server-side setup part >> Jason posted. >> >> Dne pondělí 24. srpna 2020 v 16:57:49 UTC+2 uživatel DRC napsal: >> >>> Yes, that is exactly what VirtualGL does, but the VirtualGL Client is >>> not a required component of VirtualGL. The VirtualGL Client is only used >>> if you use the built-in VGL Transport, which is only useful in a remote X >>> environment (i.e. when the 2D X server is on the client machine.) Most >>> users use VirtualGL with an X proxy these days, in which case the VirtualGL >>> Client is not used. >>> >>> Apart from that, it sounds like what you are trying to accomplish is the >>> same as https://github.com/VirtualGL/virtualgl/issues/98: sharing the >>> 3D X server connection from host to guest in a container environment such >>> as Docker. >>> On 8/24/20 9:15 AM, Martin Pecka wrote: >>> >>> As the GPUs are shared among more users, I find it useful to give a >>> separate (but still accelerated) X display to each user. I suppose telling >>> everybody to use :0 wouldn't end up very well (as long as everyone would be >>> rendering offscreen, it'd work, but I can't guarantee that). So I thought >>> that VirtualGL could be the thing that would guarantee that nobody will be >>> rendering onscreen. >>> >>> Dne pondělí 24. srpna 2020 v 16:02:20 UTC+2 uživatel DRC napsal: >>> >>>> I don't fully understand what you're proposing. The 3D X server part >>>> of >>>> your proposal should be no problem, as long as you connect each GPU to >>>> a >>>> separate screen on that X server (presumably, the 3D X server would be >>>> headless.) But why is the VirtualGL Client involved? >>>> >>>> Conceptually, it should be possible to share the 3D X server connection >>>> with, say, a Docker container, but given the extremely limited >>>> resources >>>> of this project, I have thus far been unable to dedicate the time >>>> toward >>>> researching how best to accomplish that >>>> (https://github.com/VirtualGL/virtualgl/issues/98). >>>> >>>> On 8/21/20 7:44 AM, Martin Pecka wrote: >>>> > Hi, we're thinking about getting GLX support on our HPC cluster which >>>> > (currently) is completely headless. The idea is that users should be >>>> > able to run virtual containers which would be given access to HW >>>> > rendering with OpenGL. EGL would be better, but we're stuck with OGRE >>>> > rendering engine which doesn't have proper support for the nvidia >>>> EGL. >>>> > >>>> > Could you comment on my idea? Is it a supported scenario? >>>> > >>>> > The multi-GPU server would run a single "3D X server", probably Xorg. >>>> > It would also run the virtualgl client. Containers that want to do >>>> > some OpenGL stuff would call a combination of xvfb and vglrun. I.e. >>>> > the whole setup only works with a single machine, not a pair >>>> connected >>>> > via ssh -X. >>>> > >>>> > Is that possible? Is there a tutorial for this kind of setup? >>>> >>> -- >>> >>> You received this message because you are subscribed to the Google >>> Groups "VirtualGL User Discussion/Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to virtualgl-use...@googlegroups.com. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/virtualgl-users/3973b3c2-f7ef-41b6-a580-14941b846b96n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/virtualgl-users/3973b3c2-f7ef-41b6-a580-14941b846b96n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "VirtualGL User Discussion/Support" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to virtualgl-use...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/virtualgl-users/d4c5899e-c602-4dc6-bf14-4a9d695255fan%40googlegroups.com >> >> <https://groups.google.com/d/msgid/virtualgl-users/d4c5899e-c602-4dc6-bf14-4a9d695255fan%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "VirtualGL User Discussion/Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to virtualgl-use...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/virtualgl-users/d915ad0f-010b-4155-98f3-10147f589256n%40googlegroups.com > > <https://groups.google.com/d/msgid/virtualgl-users/d915ad0f-010b-4155-98f3-10147f589256n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/e44c3005-08b1-4f1c-803e-ed0a3ba785d8n%40googlegroups.com.