OK. I was able to reboot the GZ and tried again, this time with just the
lxunsup.d script running.

I have recorded the output here: https://pastebin.com/raw/cnkX3p24

beam.smp is CouchDB and the getsockopt(1, 12)\0 line continues to repeat.

Our java application runs under Jetty and seemed to have a few lines in
there too. I'm not sure if that is going to be problematic too.


On 13 April 2018 at 21:24, Amos Hayes <aha...@gcrc.carleton.ca> wrote:

> Thank you very much for your reply, John.
> I tried to make some sense of enabling lx_detrace in the lx zone as
> describe in the "Enabling lx_debug probes" section of that doc. After
> following those instructions (copying libs and mounting lofs) I ran the
> "dtrace -Zqn lx*:::debug'{printf("%s", copyinstr(arg0))}'" command and
> started up the zone in another terminal. Sure enough, I started to get lots
> of output that I don't understand. :)
> So then in that other terminal I tried running the lxunsup.d script shown
> in that doc. When I first tried it for kicks before enabling lx_debug it
> printed the "PID NAME CALL" line and then sat there. This time, it didn't
> even print that line. So I tried to CTRL-C the other dtrace and it stopped
> spewing stuff but did not return me to the prompt. Then I tried it in the
> lxunsup.d terminal and it wouldn't exit either. Now they are both hung and
> I can't SSH into the global zone or the LX zone. The connection is
> established but the SSH handshake seems to hang.
> Did I do something wrong? Anybody have any ideas on how to recover from
> this? I am a long way from this machine. I may be able to make iDRAC access
> work for powering the GZ off/on or seeing the console but am hoping there
> is some way to kill dtrace and recover from those existing terminals.
> --
> Amos
> On 7 April 2018 at 16:01, John E. Barfield <john.barfi...@bissinc.com>
> wrote:
>> Ive had some of my own headaches with LX branded zones and found it
>> interesting to enable dtrace debugging of LX in the GZ following the hints
>> and using some of the dtrace scripts in this guide:
>> https://wiki.smartos.org/display/DOC/LX+Branded+Zones
>> Its pretty simple to replace the lx binary with the debug version and
>> then monitor the output using dtrace from the GZ.
>> I think that the most important thing to determine is whether or not your
>> application is using unsupported syscalls.
>> We have found a lot of apps that run great in LX while others just simply
>> need to run in a VM. Then there are those that we’ve simply ported to a
>> native SmartOS zone to get the best experience.
>> John Barfield
>> On Apr 7, 2018, at 2:31 PM, smartos-discuss <
>> smartos-discuss@lists.smartos.org> wrote:
>> This is a digest of messages to smartos-discuss.
>> Digest Contents
>>    1. CouchDB failing in LX zone
>> <#m_4123811116471168243_m_-2767831536118299497_20180406215237:57414E4E-3A06-11E8-A790-DF9910533F21>
>> CouchDB failing in LX zone
>> <https://www.listbox.com/member/archive/184463/2018/04/20180406215237:57414E4E-3A06-11E8-A790-DF9910533F21>
>> *Sent by Amos Hayes <aha...@gcrc.carleton.ca <aha...@gcrc.carleton.ca>>*
>> at Fri, 6 Apr 2018 21:52:30 -0400
>> Hello SmartOS folks. I have been running SmartOS at home for years now
>> with only one issue with a GZ upgrade way back in 2014 so thank you for a
>> rock solid system! Plex on an LX zone is my media server. Recently I
>> deployed a new SmartOS server in a remote location to host an instance of
>> the application we develop at work. We run & develop on Ubuntu in a VSphere
>> environment. It has prerequisites like Java/Jetty, CouchDB, and ffmpeg.
>> Everything seemed happy in an LX zone at first. But as it turns out, Apache
>> CouchDB on an Ubuntu LX zone seems to have trouble remaining responsive. I
>> have the same application setup on many VMWare Ubuntu guests and do not
>> have any stability problems with CouchDB. I am using this Ubuntu 16.04
>> image (with apt update and full-upgrade): https://docs.joyent.com/public
>> -cloud/instances/infrastructure/images/ubuntu#ubuntu-1604-20170403 The
>> rest of the setup is identical to my other VMWare guest setups with CouchDB
>> 1.6.0 from the default repositories and our application and its
>> dependancies such as openjdk, ffmpeg, etc. The tricky thing is that
>> everything works for a while and then at some point within hours/days
>> couchdb will become unresponsive. Looking at top shows its beam.smp process
>> locked at 100%. There are various errors in the logs that seem to point to
>> resource problems encountered somewhere in the Erlang code. An example
>> snippit would be: {error_info, {exit, {timeout, {gen_server,call,
>> [<0.2715.5>,{open_ref_count,<0.4245.5>}]}}, [{gen_server,terminate,7,
>> [{file,"gen_server.erl"},{line,826}]}, {proc_lib,init_p_do_apply,3,
>> [{file,"proc_lib.erl"},{line,240}]}]}}, I'm not informed enough to know
>> how to debug this, although I did try some basics like making sure the zone
>> was assigned sufficient RAM (I upped it to 16GB out of 32GB total on host)
>> and that quotas were set to 0 (although this was adjusted after creating
>> the zone.) Restarting couchdb brings it back up and it behaves normally
>> again for a while with no errors or warnings during use. I haven't taken
>> this up with couchdb folks yet because the only thing different in the
>> environment is that I'm running it in an Ubuntu LX zone vs. an Ubuntu
>> VMWare guest. If anyone could point me to some things to try or something I
>> can set up to try to catch more details when it happens again, I'd
>> appreciate it. I am completely unfamiliar with dtrace but I gather this
>> type of problem is where it shines. I have changed the couchdb logging to
>> "warning" so it will only log warnings and errors so the logs will be a bit
>> more manageable. I imagine it will break again within a day and then maybe
>> I'll have some focused logs to post somewhere. Thanks for reading through.
>> I welcome any thoughts.
>> *smartos-discuss* | Archives
>> <https://www.listbox.com/member/archive/184463/=now> | Modify
>> <https://www.listbox.com/member/?&;>
>> Your Subscription <http://www.listbox.com>

Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: 
Powered by Listbox: http://www.listbox.com

Reply via email to