On 3/22/17 9:36 , Mark Creamer wrote: > Robert, the primary VM having an issue is a database server (MySQL). The > service will only go from disabled to offline. I think there's a dependency > issue but don't know where to start of if this is related to the varpd > issue in the GZ. svcs on the host returns these services offline: > offline 12:14:52 svc:/milestone/network:default > offline 12:14:52 svc:/milestone/single-user:default > offline 12:14:52 svc:/system/filesystem/local:default > offline 12:14:53 svc:/system/sysidtool:net > offline 12:14:53 svc:/network/initial:default > offline 12:14:53 svc:/system/sysidtool:system > offline 12:14:53 svc:/milestone/sysconfig:default > offline 12:14:53 svc:/network/service:default > offline 12:14:53 svc:/network/dns/client:default > offline 12:14:53 svc:/milestone/name-services:default > offline 12:14:53 svc:/network/inetd:default > offline 12:14:53 svc:/system/system-log:default > offline 12:14:53 svc:/system/utmp:default > offline 12:14:53 svc:/system/cron:default > offline 12:14:53 svc:/milestone/multi-user:default > offline 12:14:53 svc:/system/console-login:default > offline 12:14:54 svc:/milestone/multi-user-server:default > offline 12:14:54 svc:/network/ssh:default > offline 12:14:54 svc:/network/shares/group:default > offline 12:14:54 svc:/network/shares/group:zfs > offline 12:14:54 svc:/system/sac:default > offline 12:14:54 svc:/network/netmask:default > offline 12:14:54 svc:/smartdc/mdata:execute > offline 12:14:54 svc:/zabbix/agent:default > offline* 12:16:07 svc:/network/routing-setup:default > offline 12:33:24 svc:/pkgsrc/mysql:default
So, the first thing I'd do here is look at the service you care about with svcs -xv and see why it's offline. In general, varpd should not be a dependent service of anything else, meaning that even if varpd is having problems, everything else should work. While we should definitely figure out why it's not working, it's not clear that it's related to the issue you're currently seeing. So, there are a few things that could be going on here. The module could not be loaded, or something could have gone wrong with devfsadm that causes us not to be able to have the symlink. Here are a few other things to look at: mdb -ke 'overlay_thdl_list::whatis' This will see if the overlay module is present and loaded more or less. Can you also run: ls -l /devices/pseudo/overlay@0:overlay That's the file that the /dev/overlay symlink will point to. Finally, is the devfsadmd process running? You could figure that out by running something like pargs $(pgrep devfsadm). Robert > On Wed, Mar 22, 2017 at 12:20 PM, Mark Creamer <[email protected]> wrote: > >> First I entered the dtrace command in one window. >> >> Then opened another window and with the service in maintenance, did svcadm >> clear varpd, then svcadm disable varpd, then svcadm enable varpd. >> >> Nothing new in the log except the notation about clear. Same error as >> originally. I did also check, and /dev/overlay does not exist but it does >> on my other server. Can that be recreated or copied over if that's the >> issue? It looks like it's a symlink to another file so I don't know. >> >> On Wed, Mar 22, 2017 at 12:13 PM, Robert Mustacchi <[email protected]> wrote: >> >>> On 3/22/17 9:11 , Mark Creamer wrote: >>>> Robert, I did that but nothing happens. I don't have any dtrace >>> experience >>>> so I'm not sure what to expect. Should I have seen any output in the >>> dtrace >>>> command window? Thank you >>> >>> Can you relate the exact steps you took? But yes, you should have seen >>> something in the DTrace command window. Did you see additional entries >>> in the varpd service log? >>> >>> Robert >>> >>>> On Wed, Mar 22, 2017 at 11:53 AM, Robert Mustacchi <[email protected]> >>> wrote: >>>> >>>>> On 3/22/17 8:25 , Mark Creamer wrote: >>>>>> I have a host with a service in maintenance after a reboot, and >>> several >>>>>> services on critical VMs will not start. The service in maintenance is >>>>>> varpd. Following is the log. I can't find anything on Google to help >>> with >>>>>> the error "varpd: failed to open a libvarpd handle: No such file or >>>>>> directory". I appreciate any suggestions. If it might be just a >>> matter of >>>>>> reinstalling something or copying a file over from a working host, I >>> just >>>>>> need to know what to try. >>>>>> Thanks >>>>> >>>>> Hi Mark, >>>>> >>>>> Sorry to hear that you're having trouble. While varpd being in >>>>> maintenance is something we should understand, it should not be >>> blocking >>>>> VMs from starting up unless this is Triton and not standalone SmartOS. >>>>> Probably worth understanding why they're not starting up. >>>>> >>>>>> [root@00-25-90-e0-dd-2c ~]# cat /var/svc/log/network-varpd\:de >>> fault.log >>>>>> [ May 9 19:03:16 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ May 9 19:03:17 Method "start" exited with status 0. ] >>>>>> [ Aug 25 23:40:33 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ Aug 25 23:40:35 Method "start" exited with status 0. ] >>>>>> [ Mar 19 17:39:35 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ Mar 19 17:39:36 Method "start" exited with status 0. ] >>>>>> [ Jun 30 03:46:10 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ Jun 30 03:46:11 Method "start" exited with status 0. ] >>>>>> [ Aug 28 23:39:18 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ Aug 28 23:39:20 Method "start" exited with status 0. ] >>>>>> [ Mar 22 12:38:48 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ Mar 22 12:39:49 Method or service exit timed out. Killing contract >>>>> 44. ] >>>>>> [ Mar 22 14:28:26 Leaving maintenance because disable requested. ] >>>>>> [ Mar 22 14:28:26 Disabled. ] >>>>>> [ Mar 22 14:28:48 Enabled. ] >>>>>> [ Mar 22 14:28:48 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> varpd: failed to open a libvarpd handle: No such file or directory >>>>>> [ Mar 22 14:28:48 Method "start" exited with status 95. ] >>>>>> [ Mar 22 14:30:31 Leaving maintenance because clear requested. ] >>>>>> [ Mar 22 14:30:31 Enabled. ] >>>>>> [ Mar 22 14:30:32 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> varpd: failed to open a libvarpd handle: No such file or directory >>>>>> [ Mar 22 14:30:32 Method "start" exited with status 95. ] >>>>>> [ Mar 22 14:56:28 Executing start method >>> ("/lib/svc/method/svc-varpd"). ] >>>>>> [ Mar 22 14:57:29 Method or service exit timed out. Killing contract >>>>> 40. ] >>>>>> [root@00-25-90-e0-dd-2c ~]# >>>>>> >>>>> >>>>> This means that we're dying relatively early in the library >>>>> initialization -- before we can even open up a library handle to allow >>>>> the library to log more. I suspect this means that it's failing to open >>>>> the /dev/overlay file. >>>>> >>>>> I'd recommend confirming that with something like the following: >>>>> >>>>> dtrace -qn 'syscall::open:entry/execname == "varpd"/{ self->p = arg0; >>> }' >>>>> -n 'syscall::open:return/self->p/{ printf("%s: %d %d\n", >>>>> copyinstr(self->p), arg1, errno); self->p = NULL; }' >>>>> >>>>> And then in another window restart / clear varpd. >>>>> >>>>> Robert >>>>> >>>> >>>> >>>> >>> >>> >> >> >> >> -- >> Mark >> > > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
