OK. I was able to reboot the GZ and tried again, this time with just the lxunsup.d script running.
I have recorded the output here: https://pastebin.com/raw/cnkX3p24 beam.smp is CouchDB and the getsockopt(1, 12)\0 line continues to repeat. Our java application runs under Jetty and seemed to have a few lines in there too. I'm not sure if that is going to be problematic too. -- Amos On 13 April 2018 at 21:24, Amos Hayes <aha...@gcrc.carleton.ca> wrote: > Thank you very much for your reply, John. > > I tried to make some sense of enabling lx_detrace in the lx zone as > describe in the "Enabling lx_debug probes" section of that doc. After > following those instructions (copying libs and mounting lofs) I ran the > "dtrace -Zqn lx*:::debug'{printf("%s", copyinstr(arg0))}'" command and > started up the zone in another terminal. Sure enough, I started to get lots > of output that I don't understand. :) > > So then in that other terminal I tried running the lxunsup.d script shown > in that doc. When I first tried it for kicks before enabling lx_debug it > printed the "PID NAME CALL" line and then sat there. This time, it didn't > even print that line. So I tried to CTRL-C the other dtrace and it stopped > spewing stuff but did not return me to the prompt. Then I tried it in the > lxunsup.d terminal and it wouldn't exit either. Now they are both hung and > I can't SSH into the global zone or the LX zone. The connection is > established but the SSH handshake seems to hang. > > Did I do something wrong? Anybody have any ideas on how to recover from > this? I am a long way from this machine. I may be able to make iDRAC access > work for powering the GZ off/on or seeing the console but am hoping there > is some way to kill dtrace and recover from those existing terminals. > > -- > Amos > > > On 7 April 2018 at 16:01, John E. Barfield <john.barfi...@bissinc.com> > wrote: > >> Ive had some of my own headaches with LX branded zones and found it >> interesting to enable dtrace debugging of LX in the GZ following the hints >> and using some of the dtrace scripts in this guide: >> >> https://wiki.smartos.org/display/DOC/LX+Branded+Zones >> >> Its pretty simple to replace the lx binary with the debug version and >> then monitor the output using dtrace from the GZ. >> >> I think that the most important thing to determine is whether or not your >> application is using unsupported syscalls. >> >> We have found a lot of apps that run great in LX while others just simply >> need to run in a VM. Then there are those that we’ve simply ported to a >> native SmartOS zone to get the best experience. >> >> John Barfield >> >> On Apr 7, 2018, at 2:31 PM, smartos-discuss < >> smartos-discuss@lists.smartos.org> wrote: >> >> This is a digest of messages to smartos-discuss. >> Digest Contents >> >> 1. CouchDB failing in LX zone >> >> <#m_4123811116471168243_m_-2767831536118299497_20180406215237:57414E4E-3A06-11E8-A790-DF9910533F21> >> >> CouchDB failing in LX zone >> <https://www.listbox.com/member/archive/184463/2018/04/20180406215237:57414E4E-3A06-11E8-A790-DF9910533F21> >> >> *Sent by Amos Hayes <aha...@gcrc.carleton.ca <aha...@gcrc.carleton.ca>>* >> at Fri, 6 Apr 2018 21:52:30 -0400 >> Hello SmartOS folks. I have been running SmartOS at home for years now >> with only one issue with a GZ upgrade way back in 2014 so thank you for a >> rock solid system! Plex on an LX zone is my media server. Recently I >> deployed a new SmartOS server in a remote location to host an instance of >> the application we develop at work. We run & develop on Ubuntu in a VSphere >> environment. It has prerequisites like Java/Jetty, CouchDB, and ffmpeg. >> Everything seemed happy in an LX zone at first. But as it turns out, Apache >> CouchDB on an Ubuntu LX zone seems to have trouble remaining responsive. I >> have the same application setup on many VMWare Ubuntu guests and do not >> have any stability problems with CouchDB. I am using this Ubuntu 16.04 >> image (with apt update and full-upgrade): https://docs.joyent.com/public >> -cloud/instances/infrastructure/images/ubuntu#ubuntu-1604-20170403 The >> rest of the setup is identical to my other VMWare guest setups with CouchDB >> 1.6.0 from the default repositories and our application and its >> dependancies such as openjdk, ffmpeg, etc. The tricky thing is that >> everything works for a while and then at some point within hours/days >> couchdb will become unresponsive. Looking at top shows its beam.smp process >> locked at 100%. There are various errors in the logs that seem to point to >> resource problems encountered somewhere in the Erlang code. An example >> snippit would be: {error_info, {exit, {timeout, {gen_server,call, >> [<0.2715.5>,{open_ref_count,<0.4245.5>}]}}, [{gen_server,terminate,7, >> [{file,"gen_server.erl"},{line,826}]}, {proc_lib,init_p_do_apply,3, >> [{file,"proc_lib.erl"},{line,240}]}]}}, I'm not informed enough to know >> how to debug this, although I did try some basics like making sure the zone >> was assigned sufficient RAM (I upped it to 16GB out of 32GB total on host) >> and that quotas were set to 0 (although this was adjusted after creating >> the zone.) Restarting couchdb brings it back up and it behaves normally >> again for a while with no errors or warnings during use. I haven't taken >> this up with couchdb folks yet because the only thing different in the >> environment is that I'm running it in an Ubuntu LX zone vs. an Ubuntu >> VMWare guest. If anyone could point me to some things to try or something I >> can set up to try to catch more details when it happens again, I'd >> appreciate it. I am completely unfamiliar with dtrace but I gather this >> type of problem is where it shines. I have changed the couchdb logging to >> "warning" so it will only log warnings and errors so the logs will be a bit >> more manageable. I imagine it will break again within a day and then maybe >> I'll have some focused logs to post somewhere. Thanks for reading through. >> I welcome any thoughts. >> >> *smartos-discuss* | Archives >> <https://www.listbox.com/member/archive/184463/=now> | Modify >> <https://www.listbox.com/member/?&;> >> Your Subscription <http://www.listbox.com> >> > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
