As a follow-up, last night I replaced the USB flash drive in the server with a new one running the latest SmartOS and the server and all VMs, along with all services came right up. Thank you all for your assistance on this issue! -Mark
On Wed, Mar 22, 2017 at 5:02 PM, Robert Mustacchi <[email protected]> wrote: > On 3/22/17 9:36 , Mark Creamer wrote: > > Robert, the primary VM having an issue is a database server (MySQL). The > > service will only go from disabled to offline. I think there's a > dependency > > issue but don't know where to start of if this is related to the varpd > > issue in the GZ. svcs on the host returns these services offline: > > offline 12:14:52 svc:/milestone/network:default > > offline 12:14:52 svc:/milestone/single-user:default > > offline 12:14:52 svc:/system/filesystem/local:default > > offline 12:14:53 svc:/system/sysidtool:net > > offline 12:14:53 svc:/network/initial:default > > offline 12:14:53 svc:/system/sysidtool:system > > offline 12:14:53 svc:/milestone/sysconfig:default > > offline 12:14:53 svc:/network/service:default > > offline 12:14:53 svc:/network/dns/client:default > > offline 12:14:53 svc:/milestone/name-services:default > > offline 12:14:53 svc:/network/inetd:default > > offline 12:14:53 svc:/system/system-log:default > > offline 12:14:53 svc:/system/utmp:default > > offline 12:14:53 svc:/system/cron:default > > offline 12:14:53 svc:/milestone/multi-user:default > > offline 12:14:53 svc:/system/console-login:default > > offline 12:14:54 svc:/milestone/multi-user-server:default > > offline 12:14:54 svc:/network/ssh:default > > offline 12:14:54 svc:/network/shares/group:default > > offline 12:14:54 svc:/network/shares/group:zfs > > offline 12:14:54 svc:/system/sac:default > > offline 12:14:54 svc:/network/netmask:default > > offline 12:14:54 svc:/smartdc/mdata:execute > > offline 12:14:54 svc:/zabbix/agent:default > > offline* 12:16:07 svc:/network/routing-setup:default > > offline 12:33:24 svc:/pkgsrc/mysql:default > > So, the first thing I'd do here is look at the service you care about > with svcs -xv and see why it's offline. In general, varpd should not be > a dependent service of anything else, meaning that even if varpd is > having problems, everything else should work. While we should definitely > figure out why it's not working, it's not clear that it's related to the > issue you're currently seeing. > > So, there are a few things that could be going on here. The module could > not be loaded, or something could have gone wrong with devfsadm that > causes us not to be able to have the symlink. Here are a few other > things to look at: > > mdb -ke 'overlay_thdl_list::whatis' > > This will see if the overlay module is present and loaded more or less. > Can you also run: > > ls -l /devices/pseudo/overlay@0:overlay > > That's the file that the /dev/overlay symlink will point to. > > Finally, is the devfsadmd process running? You could figure that out by > running something like pargs $(pgrep devfsadm). > > Robert > > > On Wed, Mar 22, 2017 at 12:20 PM, Mark Creamer <[email protected]> > wrote: > > > >> First I entered the dtrace command in one window. > >> > >> Then opened another window and with the service in maintenance, did > svcadm > >> clear varpd, then svcadm disable varpd, then svcadm enable varpd. > >> > >> Nothing new in the log except the notation about clear. Same error as > >> originally. I did also check, and /dev/overlay does not exist but it > does > >> on my other server. Can that be recreated or copied over if that's the > >> issue? It looks like it's a symlink to another file so I don't know. > >> > >> On Wed, Mar 22, 2017 at 12:13 PM, Robert Mustacchi <[email protected]> > wrote: > >> > >>> On 3/22/17 9:11 , Mark Creamer wrote: > >>>> Robert, I did that but nothing happens. I don't have any dtrace > >>> experience > >>>> so I'm not sure what to expect. Should I have seen any output in the > >>> dtrace > >>>> command window? Thank you > >>> > >>> Can you relate the exact steps you took? But yes, you should have seen > >>> something in the DTrace command window. Did you see additional entries > >>> in the varpd service log? > >>> > >>> Robert > >>> > >>>> On Wed, Mar 22, 2017 at 11:53 AM, Robert Mustacchi <[email protected]> > >>> wrote: > >>>> > >>>>> On 3/22/17 8:25 , Mark Creamer wrote: > >>>>>> I have a host with a service in maintenance after a reboot, and > >>> several > >>>>>> services on critical VMs will not start. The service in maintenance > is > >>>>>> varpd. Following is the log. I can't find anything on Google to help > >>> with > >>>>>> the error "varpd: failed to open a libvarpd handle: No such file or > >>>>>> directory". I appreciate any suggestions. If it might be just a > >>> matter of > >>>>>> reinstalling something or copying a file over from a working host, I > >>> just > >>>>>> need to know what to try. > >>>>>> Thanks > >>>>> > >>>>> Hi Mark, > >>>>> > >>>>> Sorry to hear that you're having trouble. While varpd being in > >>>>> maintenance is something we should understand, it should not be > >>> blocking > >>>>> VMs from starting up unless this is Triton and not standalone > SmartOS. > >>>>> Probably worth understanding why they're not starting up. > >>>>> > >>>>>> [root@00-25-90-e0-dd-2c ~]# cat /var/svc/log/network-varpd\:de > >>> fault.log > >>>>>> [ May 9 19:03:16 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ May 9 19:03:17 Method "start" exited with status 0. ] > >>>>>> [ Aug 25 23:40:33 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ Aug 25 23:40:35 Method "start" exited with status 0. ] > >>>>>> [ Mar 19 17:39:35 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ Mar 19 17:39:36 Method "start" exited with status 0. ] > >>>>>> [ Jun 30 03:46:10 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ Jun 30 03:46:11 Method "start" exited with status 0. ] > >>>>>> [ Aug 28 23:39:18 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ Aug 28 23:39:20 Method "start" exited with status 0. ] > >>>>>> [ Mar 22 12:38:48 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ Mar 22 12:39:49 Method or service exit timed out. Killing > contract > >>>>> 44. ] > >>>>>> [ Mar 22 14:28:26 Leaving maintenance because disable requested. ] > >>>>>> [ Mar 22 14:28:26 Disabled. ] > >>>>>> [ Mar 22 14:28:48 Enabled. ] > >>>>>> [ Mar 22 14:28:48 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> varpd: failed to open a libvarpd handle: No such file or directory > >>>>>> [ Mar 22 14:28:48 Method "start" exited with status 95. ] > >>>>>> [ Mar 22 14:30:31 Leaving maintenance because clear requested. ] > >>>>>> [ Mar 22 14:30:31 Enabled. ] > >>>>>> [ Mar 22 14:30:32 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> varpd: failed to open a libvarpd handle: No such file or directory > >>>>>> [ Mar 22 14:30:32 Method "start" exited with status 95. ] > >>>>>> [ Mar 22 14:56:28 Executing start method > >>> ("/lib/svc/method/svc-varpd"). ] > >>>>>> [ Mar 22 14:57:29 Method or service exit timed out. Killing > contract > >>>>> 40. ] > >>>>>> [root@00-25-90-e0-dd-2c ~]# > >>>>>> > >>>>> > >>>>> This means that we're dying relatively early in the library > >>>>> initialization -- before we can even open up a library handle to > allow > >>>>> the library to log more. I suspect this means that it's failing to > open > >>>>> the /dev/overlay file. > >>>>> > >>>>> I'd recommend confirming that with something like the following: > >>>>> > >>>>> dtrace -qn 'syscall::open:entry/execname == "varpd"/{ self->p = arg0; > >>> }' > >>>>> -n 'syscall::open:return/self->p/{ printf("%s: %d %d\n", > >>>>> copyinstr(self->p), arg1, errno); self->p = NULL; }' > >>>>> > >>>>> And then in another window restart / clear varpd. > >>>>> > >>>>> Robert > >>>>> > >>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> -- > >> Mark > >> > > > > > > > > -- Mark ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
