I've just discovered that the service also falls over on Whimsy vm5. It appeared to be OK because it was not being used, but curl killed it the same as on vm6.
I have just run the following on vm5 gem [un]install eventmachine and curl no longer kills the process, so it's looking good I'll do the same on vm6 On Thu, 24 Sep 2020 at 16:53, sebb <seb...@gmail.com> wrote: > > On Thu, 24 Sep 2020 at 16:36, Sam Ruby <ru...@intertwingly.net> wrote: > > > > On Thu, Sep 24, 2020 at 11:12 AM sebb <seb...@gmail.com> wrote: > > > > > > On Thu, 24 Sep 2020 at 16:06, Sam Ruby <ru...@intertwingly.net> wrote: > > > > > > > > On Thu, Sep 24, 2020 at 10:58 AM sebb <seb...@gmail.com> wrote: > > > > > > > > > > I tried running the following as root: > > > > > > > > > > /usr/local/bin/ruby /srv/whimsy/www/board/agenda/daemon/wss.rb -p > > > > > 34243 > > > > > > > > > > This falls over when contacted by: > > > > > > > > > > curl localhost:34243 > > > > > > > > > > However, curiously it does not fall over when run as myself or > > > > > www-data > > > > > > > > > > One obvious difference is that only root has access to > > > > > /etc/letsencrypt/live/*/ > > > > > which is used to set options.privkey and options.chain. > > > > > > > > Perhaps consistent with the theory that the gem is not compiled with > > > > access to the encryption libraries. > > > > > > The letsencrypt certificates won't be available on initial Puppet load > > > - maybe that affects the compilation options? > > > > > > Worth trying to reinstall now that the certs are up and running. > > > > While anything is possible, I doubt the existence of the certs matters > > at install time. At install time, even if the certs were installed, > > it wouldn't know which ones you were intending to use. The paths to > > the certs are only provided at runtime. > > > > Again, my theory is that the puppet installation requests that both > > libssl-dev and eventmachine need to be installed, but does not specify > > in which order these packages need to be installed. And in this case, > > the order is important. If eventmachine is installed first, it will > > be installed without encryption support. If libssl-dev is installed > > first, eventmachine will be installed with encryption support. > > Could be. > > libssl-dev is not mentioned in the whimsy5 puppet packages, but AFAICT > it is installed. > Maybe it comes pre-loaded in Ubuntu 18.04 which would explain how it > worked there. > > I think we should try re-installing eventmachine. > If that works, note the work-round until such time as the cause can be > identified and fixed. > That would require quite a lot of experimenting with Puppet and Docker... > > BTW, I'm not sure what you mean by the agenda console. > > > - Sam Ruby > > > > > > If you run without enabling options.privkey / options.chain, things > > > > will work. If you enable those options, things will fail. > > > > > > > > - Sam Ruby > > > > > > > > > On Thu, 24 Sep 2020 at 15:30, sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > The login message said to reboot, so I have done this. > > > > > > > > > > > > However it does not appear to have helped. > > > > > > > > > > > > On Thu, 24 Sep 2020 at 14:28, sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > > > On Thu, 24 Sep 2020 at 14:14, Sam Ruby <ru...@intertwingly.net> > > > > > > > wrote: > > > > > > > > > > > > > > > > If you go into the board agenda tool and open the console, you > > > > > > > > will > > > > > > > > see messages that it can't connect to the websocket. Things > > > > > > > > appear > > > > > > > > to mostly be set up correctly: > > > > > > > > > > > > > > > > $ grep websocket > > > > > > > > /etc/apache2/sites-available/10-whimsy-vm-443.conf > > > > > > > > ProxyPass "/board/agenda/websocket/" "wss://localhost:34234/" > > > > > > > > > > > > > > > > $ sudo lsof -i :34234 > > > > > > > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > > > > > > > > ruby 3430562 root 16u IPv4 160482206 0t0 TCP > > > > > > > > *:34234 (LISTEN) > > > > > > > > > > > > > > > > $ sudo systemctl status board-agenda-websocket.service > > > > > > > > ● board-agenda-websocket.service - Whimsy Board Agenda > > > > > > > > WebSocket service > > > > > > > > Loaded: loaded > > > > > > > > (/etc/systemd/system/board-agenda-websocket.service; static; > > > > > > > > vendor > > > > > > > > preset: enabled) > > > > > > > > Active: active (running) since Thu 2020-09-24 12:56:18 > > > > > > > > UTC; 9s ago > > > > > > > > Main PID: 3430562 (ruby) > > > > > > > > Tasks: 8 (limit: 9541) > > > > > > > > Memory: 35.5M > > > > > > > > CGroup: /system.slice/board-agenda-websocket.service > > > > > > > > └─3430562 /usr/local/bin/ruby > > > > > > > > /srv/whimsy/www/board/agenda/daemon/wss.rb > > > > > > > > > > > > > > > > Sep 24 12:56:18 whimsy-vm6 systemd[1]: Started Whimsy Board > > > > > > > > Agenda > > > > > > > > WebSocket service. > > > > > > > > > > > > > > > > Note, however, how long the websocket has been running (9s). > > > > > > > > This > > > > > > > > indicates that it is dying and was restarted. Looking at the > > > > > > > > syslog > > > > > > > > confirms this: > > > > > > > > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 ruby[3435205]: terminate called after > > > > > > > > throwing an instance of 'std::runtime_error' > > > > > > > > Sep 24 13:09:07 whimsy-vm6 ruby[3435205]: what(): Encryption > > > > > > > > not > > > > > > > > available on this event-machine > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: > > > > > > > > board-agenda-websocket.service: > > > > > > > > Main process exited, code=dumped, status=6/ABRT > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: > > > > > > > > board-agenda-websocket.service: > > > > > > > > Failed with result 'core-dump'. > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: > > > > > > > > board-agenda-websocket.service: > > > > > > > > Scheduled restart job, restart counter is at 17895. > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: Stopped Whimsy Board > > > > > > > > Agenda > > > > > > > > WebSocket service. > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: Started Whimsy Board > > > > > > > > Agenda > > > > > > > > WebSocket service. > > > > > > > > > > > > > > > > Random googling turned up: > > > > > > > > > > > > > > > > https://github.com/huginn/huginn/issues/386 > > > > > > > > > > > > > > > > Perhaps it is a puppet sequence error, and the eventmachine gem > > > > > > > > needs > > > > > > > > to be installed AFTER libssl-dev? > > > > > > > > > > > > > > Looks like the code is working on vm5, which has a very similar > > > > > > > Puppet setup. > > > > > > > The main differences are some software versions. > > > > > > > > > > > > > > > - Sam Ruby
