Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang 
Hi Dan,

It's the following open call that failed:

23724:  open("/dev/overlay", O_RDWR|O_EXCL) Err#2 ENOENT

For some reason 'add_drv overlay' didn't load the driver successfully.
Running rem_drv and then add_drv resolved the issue. After that, I rebooted
the host several times, unfortunately the issue was not reproduced again. I
should have tried more digging before rebooting it.

This is the first time we had this varpd issue. If you happen to have more
about the possible cause of this failure, it would be nice to know.

If it happens again, definitely I will keep you guys updated.

Thanks,
-Youzhong

# add_drv overlay
Driver (overlay) is already installed.
# modinfo | grep overlay
# ls -l /dev/overlay
/dev/overlay: No such file or directory
# rem_drv overlay
Invalid argument
Cannot remove major number binding for 196
# modinfo | grep overlay
# add_drv overlay
# ls -l /dev/overlay
lrwxrwxrwx   1 root root  35 May 18 22:08 /dev/overlay ->
../devices/pseudo/overlay@0:overlay
# grep 196 name_to_major
overlay 196
# ls -l name_to_major
-rw-r--r--   1 root sys 2952 May 18 20:33 name_to_major
# svcs -xv
svc:/network/varpd:default (virtual ARP daemon)
 State: maintenance since May 18, 2018 at 03:53:23 PM EDT
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://illumos.org/msg/SMF-8000-KS
   See: /var/svc/log/network-varpd:default.log
Impact: This service is not running.
# svcadm clear network/varpd
# svcs -xv
# svcs varpd
STATE  STIMEFMRI
online 20:34:28 svc:/network/varpd:default
# reboot
# svcs -xv
# modinfo | grep overlay
279 f8712000   7488 196   1  overlay (Overlay Network Driver)


On Fri, May 18, 2018 at 5:16 PM, Dan McDonald  wrote:

>
>
> > On May 18, 2018, at 5:11 PM, Youzhong Yang  wrote:
> >
> > Thanks Jason. In my environment, I've disabled KPTI through boot var
> 'kpti=0', not sure if it contributes to the issue.
> 
> kpti shouldn't affect anything.
> 
> Could you:
> 
> 1. ) ```truss -f -o /tmp/truss-out.txt /usr/lib/varpd/varpd```
> 
> 2. ) Assuming it's not too big, send the output to the list?
> 
> I'm curious what precisely failed with ESEARCH.
> 
> Thanks,
> Dan
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Dan McDonald 


> On May 18, 2018, at 5:11 PM, Youzhong Yang  wrote:
> 
> Thanks Jason. In my environment, I've disabled KPTI through boot var 
> 'kpti=0', not sure if it contributes to the issue.

kpti shouldn't affect anything.

Could you:

1.) ```truss -f -o /tmp/truss-out.txt /usr/lib/varpd/varpd```

2.) Assuming it's not too big, send the output to the list?

I'm curious what precisely failed with ESEARCH.

Thanks,
Dan



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang 
Thanks Jason. In my environment, I've disabled KPTI through boot var
'kpti=0', not sure if it contributes to the issue.

On Fri, May 18, 2018 at 5:06 PM, Jason King <jason.brian.k...@gmail.com>
wrote:

> I’m building an image from this commit and will get it loaded on a test
> Triton setup.  I’ll update the list once I’ve figured out what’s going on.
>
>
> From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
> Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.
> smartos.org> <smartos-discuss@lists.smartos.org>
> Date: May 18, 2018 at 3:46:31 PM
> To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
> <smartos-discuss@lists.smartos.org>
> Subject:  Re: [smartos-discuss] service varpd not working
>
> illumos-joyent: OS-6963 add_fbuf should also use is_env_string
> smartos-live:  OS-6958 bad comparison introduced with OS-6630
>
> On Fri, May 18, 2018 at 4:11 PM, Jason King <jason.brian.k...@gmail.com>
> wrote:
>
>> What is the last commit in your branch that you built?   I’ll try to
>> recreate this locally.
>>
>>
>> From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
>> Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos
>> .org> <smartos-discuss@lists.smartos.org>
>> Date: May 18, 2018 at 3:04:31 PM
>> To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
>> <smartos-discuss@lists.smartos.org>
>> Subject:  [smartos-discuss] service varpd not working
>>
>> Hi,
>>
>> Today I built SmartOS from the master branch. After rebooting the server,
>> varpd did not start up:
>>
>> # svcs varpd
>> STATE  STIMEFMRI
>> maintenance15:53:23 svc:/network/varpd:default
>>
>> # tail /var/svc/log/network-varpd\:default.log
>> [ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
>> varpd: failed to open a libvarpd handle: No such file or directory
>> [ May 18 15:53:23 Method "start" exited with status 95. ]
>>
>> # /usr/lib/varpd/varpd
>> varpd: failed to open a libvarpd handle: No such file or directory
>>
>> # find /usr/lib -name 'libvarpd.so*' | xargs file
>> /usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64 Version 1, 
>> dynamically linked, not stripped, no debugging information available
>> /usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1, 
>> dynamically linked, not stripped, no debugging information available
>>
>> Not sure why, just wanna ask.
>>
>> Thanks,
>> -Youzhong
>>
>>
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now> | Modify
> <https://www.listbox.com/member/?> Your Subscription
> <http://www.listbox.com>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Jason King 
I’m building an image from this commit and will get it loaded on a test
Triton setup.  I’ll update the list once I’ve figured out what’s going on.


From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
<smartos-discuss@lists.smartos.org>
Date: May 18, 2018 at 3:46:31 PM
To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
<smartos-discuss@lists.smartos.org>
Subject:  Re: [smartos-discuss] service varpd not working

illumos-joyent: OS-6963 add_fbuf should also use is_env_string
smartos-live:  OS-6958 bad comparison introduced with OS-6630

On Fri, May 18, 2018 at 4:11 PM, Jason King <jason.brian.k...@gmail.com>
wrote:

> What is the last commit in your branch that you built?   I’ll try to
> recreate this locally.
>
>
> From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
> Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.
> smartos.org> <smartos-discuss@lists.smartos.org>
> Date: May 18, 2018 at 3:04:31 PM
> To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
> <smartos-discuss@lists.smartos.org>
> Subject:  [smartos-discuss] service varpd not working
>
> Hi,
>
> Today I built SmartOS from the master branch. After rebooting the server,
> varpd did not start up:
>
> # svcs varpd
> STATE  STIMEFMRI
> maintenance15:53:23 svc:/network/varpd:default
>
> # tail /var/svc/log/network-varpd\:default.log
> [ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
> varpd: failed to open a libvarpd handle: No such file or directory
> [ May 18 15:53:23 Method "start" exited with status 95. ]
>
> # /usr/lib/varpd/varpd
> varpd: failed to open a libvarpd handle: No such file or directory
>
> # find /usr/lib -name 'libvarpd.so*' | xargs file
> /usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64 Version 1, 
> dynamically linked, not stripped, no debugging information available
> /usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1, 
> dynamically linked, not stripped, no debugging information available
>
> Not sure why, just wanna ask.
>
> Thanks,
> -Youzhong
>
>
*smartos-discuss* | Archives
<https://www.listbox.com/member/archive/184463/=now> | Modify
<https://www.listbox.com/member/?> Your Subscription
<http://www.listbox.com>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Youzhong Yang 
 illumos-joyent: OS-6963 add_fbuf should also use is_env_string
smartos-live:  OS-6958 bad comparison introduced with OS-6630

On Fri, May 18, 2018 at 4:11 PM, Jason King 
wrote:

> What is the last commit in your branch that you built?   I’ll try to
> recreate this locally.
>
>
> From: Youzhong Yang  
> Reply: smartos-discuss@lists.smartos.org  smartos.org> 
> Date: May 18, 2018 at 3:04:31 PM
> To: smartos-discuss@lists.smartos.org 
> 
> Subject:  [smartos-discuss] service varpd not working
>
> Hi,
>
> Today I built SmartOS from the master branch. After rebooting the server,
> varpd did not start up:
>
> # svcs varpd
> STATE  STIMEFMRI
> maintenance15:53:23 svc:/network/varpd:default
>
> # tail /var/svc/log/network-varpd\:default.log
> [ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
> varpd: failed to open a libvarpd handle: No such file or directory
> [ May 18 15:53:23 Method "start" exited with status 95. ]
>
> # /usr/lib/varpd/varpd
> varpd: failed to open a libvarpd handle: No such file or directory
>
> # find /usr/lib -name 'libvarpd.so*' | xargs file
> /usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64 Version 1, 
> dynamically linked, not stripped, no debugging information available
> /usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1, 
> dynamically linked, not stripped, no debugging information available
>
> Not sure why, just wanna ask.
>
> Thanks,
> -Youzhong
>
> *smartos-discuss* | Archives
>  | Modify
>  Your Subscription
> 
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

2018-05-18 Thread Jason King 
What is the last commit in your branch that you built?   I’ll try to
recreate this locally.


From: Youzhong Yang  
Reply: smartos-discuss@lists.smartos.org 

Date: May 18, 2018 at 3:04:31 PM
To: smartos-discuss@lists.smartos.org 

Subject:  [smartos-discuss] service varpd not working

Hi,

Today I built SmartOS from the master branch. After rebooting the server,
varpd did not start up:

# svcs varpd
STATE  STIMEFMRI
maintenance15:53:23 svc:/network/varpd:default

# tail /var/svc/log/network-varpd\:default.log
[ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
varpd: failed to open a libvarpd handle: No such file or directory
[ May 18 15:53:23 Method "start" exited with status 95. ]

# /usr/lib/varpd/varpd
varpd: failed to open a libvarpd handle: No such file or directory

# find /usr/lib -name 'libvarpd.so*' | xargs file
/usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64
Version 1, dynamically linked, not stripped, no debugging information
available
/usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1,
dynamically linked, not stripped, no debugging information available

Not sure why, just wanna ask.

Thanks,
-Youzhong

*smartos-discuss* | Archives
 | Modify
 Your Subscription




---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com