I've setup a small number of systems to run Selenium servers headless and
didn't had much problems.
This was some time ago using selenium 1.0b and Xvfb and not via Jenkins.
Can you split the problem, into what is going with Jenkins and what is going on
with Selenium?
Is there some sort of log from the Selenium scripting as to what is going on?
To understand what is going on with the browser, configure Xvfb to use a
framebuffer ("-fbdir file") and then use xwd to see snapshots of what the
browser is doing. It could be that the browser is taking too long to start, go
to a certain page, SSL cert issue, etc.
For the Jenkins, side what the build job running selenium say (i.e. Console
Output from the build)?
Another option is to dedicate a VM to run the browser and selenium server and
leave it at that - then you can watch what is going on.
