[smartos-discuss] zpool import hangs

[Youzhong Yang]

We have 3 test servers running SmartOS, host B and C have zpools zp01 and
zp02 respectively, host A can see disks of zp01 and zp02 (disks of zp01 are
connected to HBA ports of A and B; disks of zp02 are connected to HBA ports
of A and C).

And I have a loop to test zpool import/export:

for i in {1..100}
do
on B: zpool export -f zp01
on A: zpool import -o cachefile=none zp01
on A: zpool export -f zp01
on B: zpool import -o cachefile=none zp01
on C: zpool export -f zp02
on A: zpool import -o cachefile=none zp02
on A: zpool export -f zp02
on C: zpool import -o cachefile=none zp02
done

After a few iterations, zpool import hangs. This can be reproduced
consistently. Is this a known issue? Should I file a bug report?

I have captured the stack of the hanging zpool import process.

Thanks,

Youzhong

# pstack 29207
29207:  zpool import -o cachefile=none zp01

# mdb -ke '0t29207::pid2proc | ::walk thread | ::findstack -v'
stack pointer for thread ff3289f39040: ff01ed3041e0
[ ff01ed3041e0 _resume_from_idle+0xf4() ]
  ff01ed304210 swtch+0x141()
  ff01ed304250 cv_wait+0x70(ff32330f901e, ff32330f9020)
  ff01ed304380 vmem_xalloc+0x630(ff32330f9000, 6000, 1000, 0, 0, 0,
0,
  0100)
  ff01ed3043f0 vmem_alloc+0x137(ff32330f9000, 6000, 100)
  ff01ed304510 segkp_get_internal+0x11b(fbc33760, 5000, e,
  ff01ed304528, 0)
  ff01ed304570 segkp_cache_get+0x103(1)
  ff01ed304610 thread_create+0x544(0, 0, fbaf35b0,
ff32c9520b40
  , 0, fbc30540, ff010002, 003c)
  ff01ed304660 taskq_thread_create+0x108(ff32c9520b40)
  ff01ed304710 taskq_create_common+0x1a7(f7e7e238, 0, 32, 3c, a,
  7fff, fbc30540, ff32, ff0100040008)
  ff01ed304770 taskq_create+0x50(f7e7e238, 32, 3c, a, 7fff,
8)
  ff01ed3047b0 metaslab_group_create+0x96(ff36fbca70a8,
ff328784d000
  )
  ff01ed304860 vdev_alloc+0x54a(ff32a78fe000, ff01ed304928,
  ff3285910a80, ff3284e74540, a, 0)
  ff01ed304900 spa_config_parse+0x48(ff32a78fe000, ff01ed304928,
  ff3285910a80, ff3284e74540, a, 0)
  ff01ed3049a0 spa_config_parse+0xda(ff32a78fe000, ff01ed304a18,
  ff36fbca7f88, 0, 0, 0)
  ff01ed304a90 spa_load_impl+0xf4(ff32a78fe000, d5c8b305012c90c8,
  ff32d3417d30, 3, 0, 1, ff01ed304ad8)
  ff01ed304b30 spa_load+0x14e(ff32a78fe000, 3, 0, 1)
  ff01ed304b80 spa_tryimport+0xaa(ff3286740180)
  ff01ed304bd0 zfs_ioc_pool_tryimport+0x51(ff335c22a000)
  ff01ed304c80 zfsdev_ioctl+0x4a7(5a, 5a06, 804258c, 13,
  ff32578b3458, ff01ed304e68)
  ff01ed304cc0 cdev_ioctl+0x39(5a, 5a06, 804258c, 13,
  ff32578b3458, ff01ed304e68)
  ff01ed304d10 spec_ioctl+0x60(ff3284335d80, 5a06, 804258c, 13,
  ff32578b3458, ff01ed304e68, 0)
  ff01ed304da0 fop_ioctl+0x55(ff3284335d80, 5a06, 804258c, 13,
  ff32578b3458, ff01ed304e68, 0)
  ff01ed304ec0 ioctl+0x9b(3, 5a06, 804258c)
  ff01ed304f10 _sys_sysenter_post_swapgs+0x149()



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: [developer] zpool import hangs

[Youzhong Yang]

Good, here is the bug report I just filed:

https://www.illumos.org/issues/4730



On Fri, Apr 4, 2014 at 5:17 PM, Irek Szczesniak iszczesn...@gmail.comwrote:

 On Fri, Apr 4, 2014 at 11:08 PM, Youzhong Yang youzh...@gmail.com wrote:
  Thanks Robert, surya and everyone, you guys rock!
 
  I built a new image with the following fix, zpool import works very well,
  'kstat vmem:35:segkp:*' also shows no sign of memory leaks after 100
 times
  zpool export/import.
 
  https://github.com/dweeezil/zfs/commit/69b0687
 
  Would you need me to file an illumos bug report so that it won't be
  forgotten?

 Seriously: Is this really a question? If there is a bug for which no
 bug report exists then file one. Otherwise things get lost or
 forgotten and haunt Illumos forever.

 Irek




---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] How to use mdb to view process stack?

[Youzhong Yang]

I am interested in what a running process is doing, i.e. its user stack. I
know pstack can be used to print process stack, but sometimes it may not
work as you expected.

Is it possible to use mdb to view process stack? There's a blog article
'use mdb to view process stack' created by some ex-Sun/Oracle engineer but
it's no longer available.

Thanks,

Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

[Youzhong Yang]

Thanks Robert. Yes the driver failed to attach:

# rem_drv nvme
# add_drv nvme
devfsadm: driver failed to attach: nvme
Warning: Driver (nvme) successfully added to system but failed to attach

# modinfo | grep nvme
237 f8081000   6088 294   1  nvme (NVMe v1.0e)


On Tue, Jun 21, 2016 at 8:29 PM, Robert Mustacchi <r...@joyent.com> wrote:

> On 6/21/16 6:12 , Youzhong wrote:
> > Any thoughts?
>
> From my perspective, the thing that I would look at is why are we seeing
> these PCI express errors. Is this happening when we access the NVMe
> device from the driver somehow?
>
> If it's reproducible as part of attaching, it might be interesting to
> see if we can figure out what accesses to these devices is causing that,
> which may be an issue we're not properly handling with the device or a
> driver bug somewhere.
>
> Robert
>
> >> On Jun 17, 2016, at 1:17 PM, Youzhong Yang <youzh...@gmail.com> wrote:
> >>
> >> Hi Hans,
> >>
> >> The drivers won't show up in command output of 'diskinfo' or 'format'.
> >>
> >> Please refer to the attachment:
> >>
> >> - ls /dev/dsk and /dev/rdsk reported broken sym links
> >> - once I ran devfsadm -C, those broken links disappeared, then iostat
> -En reported those drives as blkdev*
> >>
> >> Thanks for your quick reply.
> >>
> >> -- Youzhong
> >>
> >>> On Fri, Jun 17, 2016 at 12:49 PM, Hans Rosenfeld <
> hans.rosenf...@nexenta.com> wrote:
> >>> Hi,
> >>>
> >>> On Fri, Jun 17, 2016 at 12:01:18PM -0400, Youzhong Yang wrote:
> >>>> Since NVMe driver has been in illumos for quite a while, so we
> decided to
> >>>> get an evaluation unit and see how it goes. Here is what we got:
> >>>>
> >>>> Baseboad: Supermicro X10DRU-i+ (System SYS-2028U-TN24R4T+)
> >>>> Memory: 768G
> >>>> CPU: 2 x Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz, each with 16
> physical
> >>>> cores, 32 hyper-threaded cores
> >>>> SATA SSDs: 2 x Intel DC S3100 1TB (SSDSC2KI010X601)
> >>>> NVMe SSDs: 2 x Intel DC P3700 800GB (SSDPE2MD800G4)
> >>>> NVMe SSDs: 22 x Intel DC P3600 400GB (SSDPE2ME400G4)
> >>>> Ethernet: Ethernet Controller X710 for 10GbE SFP+
> >>>
> >>>> Attached are some files with information of the system such as
> prtconf,
> >>>> nvme.conf etc. (we pulled out some drives, left only two INTEL DC 3100
> >>>> SSDs, one Intel DC P3700, three Intel DC P3600 SSDs in place).
> >>>
> >>>> c2t1d0   Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
> >>>> Model: INTEL SSDPE2ME400G4  Revision: 8DV10171
> Serial No: CVMD50510009400FGN
> >>>> Size: 400.09GB <400088457216 bytes>
> >>>> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
> >>>> Illegal Request: 0 Predictive Failure Analysis: 0
> >>>> c0t1d0   Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
> >>>> Model: INTEL SSDPE2MD800G4  Revision: 8DV10171
> Serial No: CVFT4222002Q800HGN
> >>>> Size: 800.17GB <800166076416 bytes>
> >>>> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
> >>>> Illegal Request: 0 Predictive Failure Analysis: 0
> >>>> c1t1d0   Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
> >>>> Model: INTEL SSDPE2ME400G4  Revision: 8DV10171
> Serial No: CVMD5051000Z400FGN
> >>>> Size: 400.09GB <400088457216 bytes>
> >>>> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
> >>>> Illegal Request: 0 Predictive Failure Analysis: 0
> >>>> c3t1d0   Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
> >>>> Model: INTEL SSDPE2ME400G4  Revision: 8DV10171
> Serial No: CVMD4450003A400FGN
> >>>> Size: 400.09GB <400088457216 bytes>
> >>>> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
> >>>> Illegal Request: 0 Predictive Failure Analysis: 0
> >>>
> >>> Looks like iostat -En shows the remaining 4 NVMe devices correctly.
> They
> >>> also show up correctly in messages and prtconf.
> >>>
> >>> What did you do to check whether they "show up"? format(1M) or some
> >>> other tool?
> >>>
> >>> Hans
> >>>
> >>> --
> >>> %SYSTEM-F-ANARCHISM, The operating system has been overthrown
> >>>
> >>
> >> 
> >
> >
> >
> > ---
> > illumos-developer
> > Archives: https://www.listbox.com/member/archive/182179/=now
> > RSS Feed:
> https://www.listbox.com/member/archive/rss/182179/21175256-4d0bc9dc
> > Modify Your Subscription:
> https://www.listbox.com/member/?;
> > Powered by Listbox: http://www.listbox.com
> >
>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: Is it possible to roll back a ZPOOL(which cannot be imported) to its last known good state?

[Youzhong Yang]

Thanks Richard and Matthew.

After applying fix for https://www.illumos.org/issues/5770, I was able to
run zdb and zpool import with -X, -F, -T and etc. But unfortunately I am no
luck to successfully import the zpool. -F or -T returned "cannot import
'zp13': one or more devices is currently unavailable" (which seems to fail
at the following code block), -X seems to run forever so I just killed it.

/*
* Find the best uberblock.
*/
vdev_uberblock_load(rvd, ub, );

/*
* If we weren't able to find a single valid uberblock, return failure.
*/
if (ub->ub_txg == 0) {
nvlist_free(label);
return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ENXIO));
}

Fortunately we had backup of the zpool available so we just restored from
the backup.

Thanks again for the tips which may be useful in the future (I hope we will
not make such mistake again).

-Youzhong


On Mon, Jan 25, 2016 at 3:50 PM, Youzhong Yang <youzh...@gmail.com> wrote:

> Hi all,
>
> Just wondering if anyone has done similar recovery using txg stuff.
>
> We have a zpool attached to two hosts physically, ideally at any time only
> one host imports this zpool. Due to some operational mistake this zpool was
> corrupted when the two hosts tried to have access to it. Here is the crash
> stack:
>
> Jan 25 10:07:17 batfs0346 genunix: [ID 403854 kern.notice] assertion
> failed: 0 == dmu_bonus_hold(spa->spa_meta_objset, obj, FTAG, ), file:
> ../../common/fs/zfs/spa.c, line: 1549
> Jan 25 10:07:17 batfs0346 unix: [ID 10 kern.notice]
> Jan 25 10:07:17 batfs0346 genunix: [ID 802836 kern.notice]
> ff017495c920 fba6b1f8 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495c9a0 zfs:load_nvlist+e8 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495ca90 zfs:spa_load_impl+10bb ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cb30 zfs:spa_load+14e ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cb80 zfs:spa_tryimport+aa ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cbd0 zfs:zfs_ioc_pool_tryimport+51 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cc80 zfs:zfsdev_ioctl+4a7 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495ccc0 genunix:cdev_ioctl+39 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cd10 specfs:spec_ioctl+60 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cda0 genunix:fop_ioctl+55 ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cec0 genunix:ioctl+9b ()
> Jan 25 10:07:17 batfs0346 genunix: [ID 655072 kern.notice]
> ff017495cf10 unix:brand_sys_sysenter+1c9 ()
>
> Is it possible to roll back the zpool to its last known good txg? We know
> when the zpool should be in good state.
>
> Any suggestion would be very much appreciated. We can build a kernel if
> needed.
>
> Thanks,
>
> - Youzhong
>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

[Youzhong Yang]

This does not make sense to me, not sure if it's relevant to the issue I am
seeing:

http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1b.pdf
page 42 defines 'CSTS -Controller Status',

but our code defines it as

typedef union {
struct {
uint32_t csts_rdy:1; /* Ready */
uint32_t csts_cfs:1; /* Controller Fatal Status */
uint32_t csts_shst:2; /* Shutdown Status */
uint32_t csts_nssro:1; /* NVM Subsystem Reset Occured */
uint32_t csts_rsvd:28;
} b;
uint32_t r;
} nvme_reg_csts_t;
(
http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/nvme/nvme_reg.h#108
)

Shouldn't "uint32_t csts_rsvd:28;" be "uint32_t csts_rsvd:27;"?

Thanks!

On Mon, Aug 1, 2016 at 2:59 PM, Youzhong Yang <youzh...@gmail.com> wrote:

> I'm not sure what sector size it uses. Randomly failing some devices
> suggests that the NVMe driver is not doing the right thing. I am not going
> to blame the hardware, because everything looks good under Solaris 11.3 and
> Centos.
>
> Thanks!
>
>
> On Mon, Aug 1, 2016 at 2:39 PM, Michael Loftis <mlof...@wgops.com> wrote:
>
>> One random question, are the affected SSDs LBA#3/4K sector formatted?
>>
>> On Mon, Aug 1, 2016 at 10:41 AM, Youzhong Yang <youzh...@gmail.com>
>> wrote:
>>
>>> Hello again,
>>>
>>> Thanks Robert for the advises. I've spent some time struggling with why
>>> NVMe SSDs were retired but there's no error reported by NVMe driver, it
>>> turns out to be a victim of fmd_asru_hash_replay_asru(), i.e. if we don't
>>> tell fmd a fault is repaired, next time when the host is rebooted, it tries
>>> to replay the event.
>>>
>>> I plugged in all the 24 NVMe SSDs, the driver reported errors like these
>>> (see attached txt file for additional info):
>>>
>>> 2016-07-30T23:11:53.468013-04:00 batfs9995 nvme: [ID 265585
>>> kern.warning] WARNING: nvme3: command timeout, OPC = 6, CFS = 0
>>> 2016-07-30T23:11:53.468018-04:00 batfs9995 nvme: [ID 265585
>>> kern.warning] WARNING: nvme3: command timeout, OPC = 8, CFS = 0
>>> 2016-07-30T23:11:53.468024-04:00 batfs9995 nvme: [ID 176450
>>> kern.warning] WARNING: nvme3: nvme_admin_cmd failed for ABORT
>>> 2016-07-30T23:11:53.468032-04:00 batfs9995 nvme: [ID 366983
>>> kern.warning] WARNING: nvme3: nvme_admin_cmd failed for IDENTIFY
>>> 2016-07-30T23:11:53.468038-04:00 batfs9995 nvme: [ID 318795
>>> kern.warning] WARNING: nvme3: failed to identify controller
>>> 2016-07-30T23:11:53.468045-04:00 batfs9995 genunix: [ID 408114 kern.info]
>>> /pci@6d,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7/pci8086,370a@0
>>> (nvme3) down
>>>
>>> Here is my understanding of what happened after NVMe driver reported the
>>> above errors:
>>>
>>> - NVMe driver called ddi_fm_service_impact(nvme->n_dip,
>>> DDI_SERVICE_LOST) to report the error for device /pci@6d
>>> ,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7/pci8086,370a@0
>>>
>>> - fmd received ereport.io.service.lost event with device-path = /pci@6d
>>> ,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7/pci8086,370a@0
>>>
>>> - fmd decided the event affects the following devs:
>>>dev:pci@6d,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7
>>> /pci8086,370a@0
>>>dev:pci@6d,0/pci8086,6f04@2/pci10b5,9765@0
>>>dev:pci@6d,0/pci8086,6f04@2
>>>
>>> - fmd sent requests to retire the above devs, which caused all the SSDs
>>> under /pci@6d,0/pci8086,6f04@2 to be retired.
>>>
>>> Why fmd decides to retire the ancestors of the problematic device is a
>>> different issue, the issue here is why NVMe driver failed to execute some
>>> of its commands during nvme_attach(). Every time I rebooted the host, it
>>> just randomly failed some of the 24 devices, and rarely sometimes there's
>>> no error at all.
>>>
>>> Just an update about what I am up to, hopefully you guys can shed some
>>> light on what can be done next.
>>>
>>> Thanks,
>>>
>>> -Youzhong
>>>
>>>
>>> On Fri, Jun 24, 2016 at 8:13 PM, Robert Mustacchi <r...@joyent.com> wrote:
>>>
>>>> On 6/24/16 11:05 , Youzhong Yang wrote:
>>>>
>>>> > I panicked the host when e_ddi_retire_device() is called, here is
>>>> what I
>>>> > found:
>>>> >
>>>> > it is /usr/lib/fm/fmd/fmd who calls modctl -> modctl_retire
>>>> > -> e_ddi_retire_device to retire /pci@0,0/pci8086,6f08@3.
>>&g

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

[Youzhong Yang]

Thanks Hans. I built a new kernel and tested it, now it works better, but
can still report errors - less frequently. Out of 5 reboots, it failed
once, so something is still there to be found.

On Mon, Aug 1, 2016 at 3:42 PM, Hans Rosenfeld <hans.rosenf...@nexenta.com>
wrote:

> On Mon, Aug 01, 2016 at 03:32:50PM -0400, Youzhong Yang wrote:
> > This does not make sense to me, not sure if it's relevant to the issue I
> am
> > seeing:
> >
> > http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1b.pdf
> > page 42 defines 'CSTS -Controller Status',
> >
> > but our code defines it as
> >
> > typedef union {
> > struct {
> > uint32_t csts_rdy:1; /* Ready */
> > uint32_t csts_cfs:1; /* Controller Fatal Status */
> > uint32_t csts_shst:2; /* Shutdown Status */
> > uint32_t csts_nssro:1; /* NVM Subsystem Reset Occured */
> > uint32_t csts_rsvd:28;
> > } b;
> > uint32_t r;
> > } nvme_reg_csts_t;
> > (
> >
> http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/nvme/nvme_reg.h#108
> > )
> >
> > Shouldn't "uint32_t csts_rsvd:28;" be "uint32_t csts_rsvd:27;"?
> 
> Yes, it should be. Thanks for spotting this, I'll prepare a fix
> tomorrow.
> 
> Hans
> 
> --
> %SYSTEM-F-ANARCHISM, The operating system has been overthrown
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

[Youzhong Yang]

Hi Robert,

Thanks for looking into this issue.

I tried MSI interrupt type on my own but it didn't work, but I will try
your patch again and then report back.

I've studied the nvme driver in Solaris 11.3, it seems they do the same
thing as Linux - MSI-X first, then MSI, finally FIXED, see attached file
for the assembly code and my comments. By the way, if I
set nvmex_enable_msi and nvmex_enable_msix to 0 (false) in /etc/system,
Solaris crashed immediately upon reboot.

MSI-X works well on our host with a minor issue:

In nvme_var.h, NVME_ADMIN_CMD_TIMEOUT is defined as 10, i.e. 100ms I
think. It's too small. One of the INTEL NVMe SSDs took 366ms to execute GET
LOG PAGE command. Once I bumped up the value to 1000,000, nvme driver
happily attached all the 24 SSDs in many reboots.

Here comes new issues/lack of functionality:

- The INTEL drives are formatted to use 512 bytes block size, but they
advertise 4096 bytes block size as the best performing one (see the data in
issue report https://www.illumos.org/issues/7279). So far we don't have the
ability to perform such low level FORMAT and I had to install other OS such
as Linux and use their tool (nvme-cli).

- With NVMe SSD formatted to use 4096 block size, many things don't work.
It seems our blkdev driver never intended to support device with block size
larger than 512 bytes. I tried hacking bd_strategy() function to modify
bp->b_lblkno (in 512 bytes size, passed all the way down from zfs layer) to
be in 4096 block size, it appeared to be working for most I/O ops, but I
know it's just a hack and I will open a new thread discussing this
particular issue.

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/blkdev/blkdev.c#1142
http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/vdev_disk.c#783

- Using 512-bytes block size, I was able to create zpool, do filesystem
ops, everything appears to be great. But, once I do zpool scrub, it reports
checksum errors. It can be reproduced easily with one drive. I tried the
same think using the same drive in Solaris, it worked so I don't think it's
a hardware issue.

The checksum errors issue just bothered me, I don't know where to start
with. I've done some dtracing in nvme driver and found no error returned
for the read/write cmds.

Thanks,

-Youzhong




On Wed, Aug 10, 2016 at 10:41 PM, Robert Mustacchi <r...@joyent.com> wrote:

> On 8/4/16 11:25 , Youzhong Yang wrote:
> > Thanks for the input Robert.
> >
> > I believe the issue is now resolved by using MSI-X (instead of FIXED)
> > interrupt type inside nvme_init() for the admin queue.
> >
> > Here is the issue report I just filed:
> >
> > https://www.illumos.org/issues/7273
> >
> > I don't know why FIXED interrupt would cause issue, probably because we
> > have too many NVMe SSDs?
> 
> Following up on this aspect, I've put together the following:
> 
> https://us-east.manta.joyent.com/rmustacc/public/webrevs/7273/index.html
> 
> Youzhong, would it be possible for you to use this? We've had success
> with this with someone who was seeing issues. I'll hopefully get some
> time to look at the offlining issue, but it's still a bit out, sorry.
> 
> Hans, can you review this and take a look?
> 
> Thanks,
> Robert
> 

int nvmex_register_intrs(nvmex_t *);
int nvmex_setup_intr(nvmex_t *, int);

nvmex_register_intrs:   pushq  %rbp
nvmex_register_intrs+1: movq   %rsp,%rbp
nvmex_register_intrs+4: subq   $0x8,%rsp
nvmex_register_intrs+8: movq   %rdi,-0x8(%rbp)
nvmex_register_intrs+0xc:   pushq  %rbx
nvmex_register_intrs+0xd:   pushq  %r12
nvmex_register_intrs+0xf:   pushq  %r13
nvmex_register_intrs+0x11:  pushq  %r14
nvmex_register_intrs+0x13:  subq   $0x18,%rsp
nvmex_register_intrs+0x17:  movq   %rdi,%rbx < nvmex_t pointer
nvmex_register_intrs+0x1a:  movl   $0x,%eax
nvmex_register_intrs+0x1f:  movl   %eax,0x514(%rbx)  < int nv_intr_type 
of nvmex_t
nvmex_register_intrs+0x25:  movl   $0x1,%r12d
nvmex_register_intrs+0x2b:  movl   %r12d,0x51c(%rbx) < boolean_t 
nv_intr_numa
nvmex_register_intrs+0x32:  movq   (%rbx),%rdi   < dev_info_t 
*nv_dip 
nvmex_register_intrs+0x35:  leaq   -0x34(%rbp),%rsi  < int *typesp arg 
nvmex_register_intrs+0x39:  call   +0x3b42fae   

nvmex_register_intrs+0x3e:  testl  %eax,%eax
nvmex_register_intrs+0x40:  jne+0x141   <nvmex_register_intrs+0x187>  
<--- in case of error
nvmex_register_intrs+0x46:  cmpl   $0x0,-0x37e1db75(%rip)   

nvmex_register_intrs+0x4d:  je +0x1b<nvmex_register_intrs+0x6a>
nvmex_register_intrs+0x4f:  movl   -0x34(%rbp),%eax  < supported intr 
types
nvmex_register_intrs+0x52:  testl  $0x4,%eax < 
DDI_INTR_TYPE_MSIX = 0x4
nvmex_register_intrs+0x57:  je

[smartos-discuss] TSX-NI support in illumos?

[Youzhong Yang]

Just curious, is TSX supported in illumos?

"*Transactional Synchronization Extensions* (*TSX*) is an extension to the
x86  instruction set architecture
 (ISA) that
adds hardware transactional memory
 support, speeding up
execution of multi-threaded software through lock elision. According to
different benchmarks, TSX can provide around 40% faster applications
execution in specific workloads, and 4–5 times more database transactions
per second  (TPS)."

https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions

-youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [developer] Re: [smartos-discuss] TSX-NI support in illumos?

[Youzhong Yang]

Thanks! Definitely it's something that I'd like to explore in our userland
application such as Samba.

On Thu, Jul 21, 2016 at 11:08 AM, Robert Mustacchi <r...@joyent.com> wrote:

> On 7/21/16 7:36 , Garrett D'Amore wrote:
> > No.
>
> Strictly speaking, they don't require kernel support, so you should be
> able to use them; however, they don't have entries in isainfo or in the
> dis tables at this time.
>
> Given how few SKUs work with them and many of the gotchas that can cause
> things to abort (say the use of the pid provider), not sure that they're
> something I'd turn to.
>
> Robert
>
> >> On Jul 21, 2016, at 7:34 AM, Youzhong Yang <youzh...@gmail.com> wrote:
> >>
> >> Just curious, is TSX supported in illumos?
> >>
> >> "Transactional Synchronization Extensions (TSX) is an extension to the
> x86 instruction set architecture (ISA) that adds hardware transactional
> memory support, speeding up execution of multi-threaded software through
> lock elision. According to different benchmarks, TSX can provide around 40%
> faster applications execution in specific workloads, and 4–5 times more
> database transactions per second (TPS)."
> >>
> >> https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions
> >>
> >> -youzhong
> >>
> >> smartos-discuss | Archives  | Modify Your Subscription
> >
> >
> 
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] clock_gettime() causes sporadic segmentation fault

[Youzhong Yang]

I verified that the issue is fixed now. Thank you!

On Tue, Jul 19, 2016 at 5:19 PM, Patrick Mooney <patrick.moo...@joyent.com>
wrote:

> With the fix for OS-5515 merged, this should be solved.
>
> Thanks for bringing it to our attention.
>
> On 18 July 2016 at 08:50, Youzhong Yang <youzh...@gmail.com> wrote:
>
>> Thanks for the quick turnaround ..
>>
>>
>> On Sat, Jul 16, 2016 at 9:18 PM, Patrick Mooney <
>> patrick.moo...@joyent.com> wrote:
>>
>>> I was able to reproduce the issue locally and have located the issue.
>>> The initial analysis is here: https://smartos.org/bugview/OS-5515.
>>> More detail will be added as the problem is addressed.
>>>
>>> On 16 July 2016 at 20:00, Patrick Mooney <patrick.moo...@joyent.com>
>>> wrote:
>>>
>>>> What version of the platform image do you have installed (uname -v)?
>>>> Would you be willing to make the core file available?
>>>>
>>>> On 16 July 2016 at 19:54, Youzhong Yang <youzh...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We've been using mbuffer for years, recently after upgrading SmartOS
>>>>> to the latest version, mbuffer crashes sporadically inside 
>>>>> clock_gettime():
>>>>>
>>>>> # mdb core.mbuffer.28346
>>>>> Loading modules: [ libc.so.1 ld.so.1 ]
>>>>> > ::status
>>>>> debugging core file of mbuffer (64-bit) from batfs9920
>>>>> initial argv: ./mbuffer
>>>>> threading model: native threads
>>>>> status: process terminated by SIGSEGV (Segmentation Fault),
>>>>> addr=fd7fff390018
>>>>> > $C
>>>>> fd7ffe63dec0 libc.so.1`__cp_can_gettime+0xd(fd7fff39)
>>>>> fd7ffe63df00 libc.so.1`__clock_gettime+0x3a(4, fd7ffe63df50)
>>>>> fd7ffe63df20 libc.so.1`clock_gettime+0x15(4, fd7ffe63df50)
>>>>> fd7ffe63dfb0 inputThread+0x3d()
>>>>> fd7ffe63dfe0 libc.so.1`_thrp_setup+0x8a(fd7ffe7e0a40)
>>>>> fd7ffe63dff0 libc.so.1`_lwp_start()
>>>>> > ::walk thread | ::findstack -v
>>>>> stack pointer for thread 1: fd7fffdfd190
>>>>> [ fd7fffdfd190 libc.so.1`__pollsys+0xa() ]
>>>>>   fd7fffdfd2b0 libc.so.1`pselect+0x1cb(5, fd7fffdfd480, 0, 0,
>>>>> fd7fffdfd2c0, 0)
>>>>>   fd7fffdfd300 libc.so.1`select+0x5a(5, fd7fffdfd480, 0, 0,
>>>>> fd7fffdfd370)
>>>>>   fd7fffdff4b0 statusThread+0xe6()
>>>>>   fd7fffdffb90 main+0xdbb()
>>>>>   fd7fffdffba0 _start+0x6c()
>>>>> stack pointer for thread 2: fd7ffe9eeec0
>>>>> [ fd7ffe9eeec0 libc.so.1`__lwp_park+0x17() ]
>>>>>   fd7ffe90 libc.so.1`sema_wait+0x13(421a20)
>>>>>   fd7ffe9eefb0 outputThread+0x222()
>>>>>   fd7ffe9eefe0 libc.so.1`_thrp_setup+0x8a(fd7ffe7e0240)
>>>>>   fd7ffe9eeff0 libc.so.1`_lwp_start()
>>>>> stack pointer for thread 3: fd7ffe63dec0
>>>>> [ fd7ffe63dec0 libc.so.1`__cp_can_gettime+0xd() ]
>>>>>   fd7ffe63df00 libc.so.1`__clock_gettime+0x3a(4, fd7ffe63df50)
>>>>>   fd7ffe63df20 libc.so.1`clock_gettime+0x15(4, fd7ffe63df50)
>>>>>   fd7ffe63dfb0 inputThread+0x3d()
>>>>>   fd7ffe63dfe0 libc.so.1`_thrp_setup+0x8a(fd7ffe7e0a40)
>>>>>   fd7ffe63dff0 libc.so.1`_lwp_start()
>>>>> >
>>>>>
>>>>> To reproduce, just run ,/mbuffer, sometimes it works, sometimes it
>>>>> crashes.
>>>>>
>>>>> Is this a bug?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Youzhong
>>>>>
>>>>>
>>>>
>>> *smartos-discuss* | Archives
>>> <https://www.listbox.com/member/archive/184463/=now>
>>> <https://www.listbox.com/member/archive/rss/184463/25077300-734ee1ca> |
>>> Modify
>>> <https://www.listbox.com/member/?;>
>>> Your Subscription <http://www.listbox.com>
>>>
>>
>>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] [developer] NVMe SSDs (Intel DC P3700, P3600) won't show up

[Youzhong Yang]

Hello again,

Thanks Robert for the advises. I've spent some time struggling with why
NVMe SSDs were retired but there's no error reported by NVMe driver, it
turns out to be a victim of fmd_asru_hash_replay_asru(), i.e. if we don't
tell fmd a fault is repaired, next time when the host is rebooted, it tries
to replay the event.

I plugged in all the 24 NVMe SSDs, the driver reported errors like these
(see attached txt file for additional info):

2016-07-30T23:11:53.468013-04:00 batfs9995 nvme: [ID 265585 kern.warning]
WARNING: nvme3: command timeout, OPC = 6, CFS = 0
2016-07-30T23:11:53.468018-04:00 batfs9995 nvme: [ID 265585 kern.warning]
WARNING: nvme3: command timeout, OPC = 8, CFS = 0
2016-07-30T23:11:53.468024-04:00 batfs9995 nvme: [ID 176450 kern.warning]
WARNING: nvme3: nvme_admin_cmd failed for ABORT
2016-07-30T23:11:53.468032-04:00 batfs9995 nvme: [ID 366983 kern.warning]
WARNING: nvme3: nvme_admin_cmd failed for IDENTIFY
2016-07-30T23:11:53.468038-04:00 batfs9995 nvme: [ID 318795 kern.warning]
WARNING: nvme3: failed to identify controller
2016-07-30T23:11:53.468045-04:00 batfs9995 genunix: [ID 408114 kern.info]
/pci@6d,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7/pci8086,370a@0
(nvme3) down

Here is my understanding of what happened after NVMe driver reported the
above errors:

- NVMe driver called ddi_fm_service_impact(nvme->n_dip, DDI_SERVICE_LOST)
to report the error for device /pci@6d,0/pci8086,6f04@2/pci10b5,9765@0
/pci10b5,9765@7/pci8086,370a@0

- fmd received ereport.io.service.lost event with device-path = /pci@6d
,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7/pci8086,370a@0

- fmd decided the event affects the following devs:
   dev:pci@6d,0/pci8086,6f04@2/pci10b5,9765@0/pci10b5,9765@7
/pci8086,370a@0
   dev:pci@6d,0/pci8086,6f04@2/pci10b5,9765@0
   dev:pci@6d,0/pci8086,6f04@2

- fmd sent requests to retire the above devs, which caused all the SSDs
under /pci@6d,0/pci8086,6f04@2 to be retired.

Why fmd decides to retire the ancestors of the problematic device is a
different issue, the issue here is why NVMe driver failed to execute some
of its commands during nvme_attach(). Every time I rebooted the host, it
just randomly failed some of the 24 devices, and rarely sometimes there's
no error at all.

Just an update about what I am up to, hopefully you guys can shed some
light on what can be done next.

Thanks,

-Youzhong


On Fri, Jun 24, 2016 at 8:13 PM, Robert Mustacchi <r...@joyent.com> wrote:

> On 6/24/16 11:05 , Youzhong Yang wrote:
>
> > I panicked the host when e_ddi_retire_device() is called, here is what I
> > found:
> >
> > it is /usr/lib/fm/fmd/fmd who calls modctl -> modctl_retire
> > -> e_ddi_retire_device to retire /pci@0,0/pci8086,6f08@3.
>
> Okay, this makes some amount of sense, we're seeing various FM ereports
> being generated at a rate which causes us to eventually offline the device.
>
> > Attached is a file with some entries produced by fmdump. It's weird that
> > sometimes I got those fm entries but sometimes the system generated
> nothing
> > but still retired the drives.
> >
> > I don't know how to interpret those entries, maybe someone on the list
> can
> > shed some light?
>
> So, these are errors that are based on the PCI express specification and
> the various entries usually refer to parts of the advanced error
> reporting capabilities. So, what I do here is I go through and look at
> the correctable and uncorrectable error status members which correspond
> to the registers.
>
> So the first one starting at line 11 indicates that a receive error was
> encountered. Note that the entry that generated it is not the device,
> but what seems like the non-transparent bridge.
>
> It's also worth calling out what the general ereports are talking about.
> You'll note there are basically three different classes there:
>
> - ereport.io.pci.fabric
> - ereport.io.pciex.rc.ce-msg
> - ereport.io.pciex.pl.re
>
> So, the pl.re are issues that indicate receiver errors. Which if I'm
> reading this correctly indicates issues in some of the decoding of data?
>
> The rc.ce-msg means that the root complex has been informed of
> correctable errors.
>
> That said, some of the messages that have arrive at the root port seem a
> bit odd.
>
> > Device 8086:6f08 is "Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3
> > v4/Xeon D PCI Express Root Port 3" and seems to use "PCIe bridge/switch
> > driver" (pcieb). Is it possible the pcieb driver in illumos does not work
> > properly with this device?
> 
> It looks like the actual NVMe devices may be connected to a
> non-transparent bridge. So it's highly likely that that device is
> failing which is also what's directly connected to that port. I have
> seen something sim

[smartos-discuss] vioblk driver

[Youzhong Yang]

Hi All,

A quick question - how can I make use of vioblk device driver? Creating a
SmartOS VM using Linux KVM?

The reason I am asking this is that I want to test a change in blkdev, to
exercise some code path in it which can't be done using NVMe devices, such
as the 'dump' functionality.

Thanks,

Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] vioblk driver

[Youzhong Yang]

Thanks, it works.

However, I am having difficulty setting the block size to anything other
than 512:

vmadm tells the block size is 8192,

# vmadm get b175850d-92fd-cf81-d352-90d40e5076ce |json disks
[
  {
"path":
"/dev/zvol/rdsk/zones/b175850d-92fd-cf81-d352-90d40e5076ce-disk1",
"boot": false,
"model": "virtio",
"media": "disk",
"zfs_filesystem": "zones/b175850d-92fd-cf81-d352-90d40e5076ce-disk1",
"zpool": "zones",
"size": 65536,
"compression": "on",
"refreservation": 65536,
"block_size": 8192
  }
]

but inside the VM, it says the block size is 512:

> ff03d8df9700::print -t struct vioblk_softc sc_virtio.sc_features
sc_blk_size sc_pblk_size
uint32_t sc_virtio.sc_features = 0x1454
int sc_blk_size = 0x200
int sc_pblk_size = 0x200

Quite confused ..

Any thoughts?

Thanks,

--Youzhong



On Mon, Jan 30, 2017 at 10:10 AM, Nahum Shalman <nshal...@omniti.com> wrote:

> You can also create a SmartOS KVM VM on a SmartOS machine with a virtio
> virtual disk.
>
> -Nahum
>
> On Mon, Jan 30, 2017 at 9:38 AM, Youzhong Yang <youzh...@gmail.com> wrote:
>
>> Hi All,
>>
>> A quick question - how can I make use of vioblk device driver? Creating a
>> SmartOS VM using Linux KVM?
>>
>> The reason I am asking this is that I want to test a change in blkdev, to
>> exercise some code path in it which can't be done using NVMe devices, such
>> as the 'dump' functionality.
>>
>> Thanks,
>>
>> Youzhong
>>
>>
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now>
> <https://www.listbox.com/member/archive/rss/184463/25077300-734ee1ca> |
> Modify
> <https://www.listbox.com/member/?;>
> Your Subscription <http://www.listbox.com>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] cannot open '...': dataset does not exist

[Youzhong Yang]

Really weird. I am seeing this issue with the latest image, but not with
the OS image built in last November. On a freshly installed VM, there's no
problem.

# zfs create clusters/vmgr/test
cannot open 'vmgr/test': dataset does not exist
# echo $?
0
# zfs list clusters/vmgr/test
NAME USED  AVAIL  REFER  MOUNTPOINT
clusters/vmgr/test29K  2.30T29K  /vmgr/test

Any clue what could have caused this?

Thanks,

-Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] smartos ssd disk question

[Youzhong Yang]

it is power safe and we've tested it here.

https://toshiba.semicon-storage.com/us/product/storage-products/enterprise-ssd/px02smb-px02smfxxx.html?sug=1


On Mon, Jan 23, 2017 at 12:01 PM, Robert Mustacchi <r...@joyent.com> wrote:

> On 1/23/17 6:29 , Youzhong Yang wrote:
> > Add something like this to /kernel/drv/sd.conf:
> >
> > "TOSHIBA PX02SMF020  ", "cache-nonvolatile:true",
> >
> > I don't think the sd.conf comes with smartos image has it, so you need to
> > build your own image.
> 
> In general, you should _never_ set this value. You have basically told
> the system that this device is power safe and never requires a
> synchronize cache command. This is not true for most devices and a
> poorly timed panic will result in data loss on the one device whose
> purpose is to protect its data: the slog.
> 
> Note, when I generally talk about an SSD being power safe, that does not
> mean that this can be set to true. The devices generally only guarantee
> that data is safe after a synchronize cache command.
> 
> I don't have as much experience with these Toshiba drives, so it may be
> that their datasheet tells you something else in this case.
> 
> Robert
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] smartos ssd disk question

[Youzhong Yang]

I don't know. What we know is that the sync write performance of zil drive
is awful if cache-nonvolatile:true is not set.

On Mon, Jan 23, 2017 at 12:30 PM, Robert Mustacchi <r...@joyent.com> wrote:

> On 1/23/17 9:20 , Youzhong Yang wrote:
> > it is power safe and we've tested it here.
> >
> > https://toshiba.semicon-storage.com/us/product/
> storage-products/enterprise-ssd/px02smb-px02smfxxx.html?sug=1
>
> Sure, it does say it's power safe. Are you sure that means you don't
> need to issue synchronize cache commands to the device? For some
> devices, you still need to issue synchronize cache commands even if
> they're power safe. If it works, great. Hopefully that just means
> synchronize cache commands are a no-op.
>
> Robert
>
> > On Mon, Jan 23, 2017 at 12:01 PM, Robert Mustacchi <r...@joyent.com>
> wrote:
> >
> >> On 1/23/17 6:29 , Youzhong Yang wrote:
> >>> Add something like this to /kernel/drv/sd.conf:
> >>>
> >>> "TOSHIBA PX02SMF020  ", "cache-nonvolatile:true",
> >>>
> >>> I don't think the sd.conf comes with smartos image has it, so you need
> to
> >>> build your own image.
> >>
> >> In general, you should _never_ set this value. You have basically told
> >> the system that this device is power safe and never requires a
> >> synchronize cache command. This is not true for most devices and a
> >> poorly timed panic will result in data loss on the one device whose
> >> purpose is to protect its data: the slog.
> >>
> >> Note, when I generally talk about an SSD being power safe, that does not
> >> mean that this can be set to true. The devices generally only guarantee
> >> that data is safe after a synchronize cache command.
> >>
> >> I don't have as much experience with these Toshiba drives, so it may be
> >> that their datasheet tells you something else in this case.
> >>
> >> Robert
> >>
> >
> >
> 
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] how to block init 6 on global zone

[Youzhong Yang]

In smartos-live/overlay/generic/lib/svc/method/fs-joyent, if you add

+ if [[ -d /usbkey/crud ]]; then
+ (cd /usbkey/crud;find . -type f -print|xargs -i sh -c 'cd /;mkdir -p
`dirname "{}"`;touch "{}";mount -F lofs /usbkey/crud/"{}" "{}"')
+ fi

then all files under /usbkey/crud will be lofs mouted under /.

The credit goes to Ira Cooper.

On Thu, Sep 29, 2016 at 9:45 AM, Jonathan Perkin  wrote:

> * On 2016-09-29 at 14:33 BST, Jorge Schrauwen wrote:
>
> > Something like this would be really nice to have upstream,
> >
> > [ -d /usbkey/root ] &&  mount -O -F lofs /usbkey/root /root
> >
> > Or whatever in the local-fs service? Is it worth opening a issue for
> this,
> > or a PR even?
> 
> Yeh, I've wondered about adding a root_home_overlay or whatever to
> /usbkey/config for this.  It's worth a PR at least so we can gather
> input about what might be affected by this (e.g. how it interacts with
> root_authorized_keys).
> 
> --
> Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] zrlock issue: in LOCKED state and never released

[Youzhong Yang]

Hi,

One of our servers had hundreds of threads waiting for a zrlock to be
released, but it never did:

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/zrlock.c#96

I am just wondering how this could happen, the only place trying to lock it
seems to be this one:

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dnode.c#916

As you can see, zrl_exit() is used to release the lock.

I found this 3-years-old issue:

https://www.illumos.org/issues/3746 - ZRLs are racy

Any thoughts?

Thanks,

-- Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: build failure related to python version

[Youzhong Yang]

Sorry, the 'head' output is after I manually modified hdrchk script. Its
first line was referring to /usr/bin/python2.6.

And here is the list of files still referring to /usr/bin/python2.6:

# grep '#!/usr/bin/python2.6'
/root/smartos-live/projects/illumos/usr/src/tools/scripts/*
/root/smartos-live/projects/illumos/usr/src/tools/scripts/cddlchk:#!/usr/bin/python2.6
/root/smartos-live/projects/illumos/usr/src/tools/scripts/copyrightchk:#!/usr/bin/python2.6
/root/smartos-live/projects/illumos/usr/src/tools/scripts/git-pbchk:#!/usr/bin/python2.6
/root/smartos-live/projects/illumos/usr/src/tools/scripts/hg-active:#!/usr/bin/python2.6
/root/smartos-live/projects/illumos/usr/src/tools/scripts/mapfilechk:#!/usr/bin/python2.6
/root/smartos-live/projects/illumos/usr/src/tools/scripts/validate_pkg:#!/usr/bin/python2.6
/root/smartos-live/projects/illumos/usr/src/tools/scripts/wsdiff:#!/usr/bin/python2.6


On Fri, Sep 30, 2016 at 4:01 PM, Youzhong Yang <youzh...@gmail.com> wrote:

> I am trying to build SmartOS today, pulling illumos-joyent from github.com,
> here is what I encountered:
>
> nightly.log:
> sh: line 1: /root/smartos-live/projects/illumos/usr/src/tools/proto/
> root_i386-nd/opt/onbld/bin/hdrchk: not found
> *** Error code 127
>
> # head -1 /root/smartos-live/projects/illumos/usr/src/tools/proto/
> root_i386-nd/opt/onbld/bin/hdrchk
> #!/opt/local/bin/python2.6
>
> It seems the two OS-5688 commits didn't fix the issue.
>
> Thanks,
>
> --Youzhong
>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] build failure related to python version

[Youzhong Yang]

I am trying to build SmartOS today, pulling illumos-joyent from github.com,
here is what I encountered:

nightly.log:
sh: line 1:
/root/smartos-live/projects/illumos/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/hdrchk:
not found
*** Error code 127

# head -1
/root/smartos-live/projects/illumos/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/hdrchk
#!/opt/local/bin/python2.6

It seems the two OS-5688 commits didn't fix the issue.

Thanks,

--Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] Re: build failure related to python version

[Youzhong Yang]

Now it works. Thanks!

On Fri, Sep 30, 2016 at 4:39 PM, Robert Mustacchi <r...@joyent.com> wrote:

> On 9/30/16 13:05 , Youzhong Yang wrote:
> > Sorry, the 'head' output is after I manually modified hdrchk script. Its
> > first line was referring to /usr/bin/python2.6.
>
> There were various bugs and issues with the merge from upstream,
> including some breakage there. These should all be fixed in the commit
> that was just pushed.
>
> There will be future updates now that this has landed to get build zones
> and python updated, but those haven't come yet.
>
> Robert
>
> > On Fri, Sep 30, 2016 at 4:01 PM, Youzhong Yang <youzh...@gmail.com>
> wrote:
> >
> >> I am trying to build SmartOS today, pulling illumos-joyent from
> github.com,
> >> here is what I encountered:
> >>
> >> nightly.log:
> >> sh: line 1: /root/smartos-live/projects/illumos/usr/src/tools/proto/
> >> root_i386-nd/opt/onbld/bin/hdrchk: not found
> >> *** Error code 127
> >>
> >> # head -1 /root/smartos-live/projects/illumos/usr/src/tools/proto/
> >> root_i386-nd/opt/onbld/bin/hdrchk
> >> #!/opt/local/bin/python2.6
> >>
> >> It seems the two OS-5688 commits didn't fix the issue.
> >>
> >> Thanks,
> >>
> >> --Youzhong
> >>
> >>
> >
> >
> 
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] build: 00000000T000000Z

[Youzhong Yang]

Hi,

I noticed something strange today:

# ssh host uname -a
SunOS batfs9920 5.11 joyent_20170406T161321Z i86pc i386 i86pc

# ssh host
- SmartOS (build: T00Z)

By the way, I built the image today after merging latest stuff from
illumos-joyent, smartos-live etc.

Thanks,

--Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] To the OmniOS, OI and SmartOS community

[Youzhong Yang]

I would recommend Plan C. We've used SmartOS for years, with hundreds of
servers as storage backend, serving thousands of Linux/Mac/Windows clients.
Joyent has the best talented engineers and they are always willing to help.

By the way, we don't care much about packages, a stable and reliable kernel
is our requirement.

On Mon, May 15, 2017 at 7:58 AM, Guenther Alka  wrote:

> *Its time to consider pla**n** B/C ??*
> to: omnios-discuss, openindiana-discuss, smartos-discuss
>
> The announcement of OmniTi to cancel OmniOS from now to then is a real
> disaster not only for OmniOS users but for the whole Illumos platform. Many
> users who want a free Solaris based OS especially in production
> environments selected OmniOS as their preferred Illumos platform mainly
> with use cases storage and general server applications.
>
> The reasons:* OmniOS=**Up to date Illumos*
> + commercial support option (although way too expensive)
> + own developments like LX zones integration from SmartOS or drivers
> + stables/long term stables with very experienced full time staff (thanks
> to Dan and Dale again)
>
> As OmniTi has released a new stable 151022, I/we have some time maybe to
> the end of the year unless OmniOS is out of sync with Illumos in a non
> tolerable amount.  Bugfixes of serious problems may be the case until then
> (hope so).
>
> What are the/my options
>
>
> *Plan A*
> Hope for a continuation of OmniOS as a well maintained
> community/commercial project with further development, ongoing stables and
> bugfixes optionally with some paid contributions under the umbrella of a
> firm or at least with some experienced members that were already resonsible
> for OmniOS or an Illumos distribution and that can be trusted for next
> years.
>
> While I hope for this, I doubt that this is a serious option. I switched
> from OpenIndiana to OmniOS three years ago as the OI community was too weak
> and development nearly stalled at that time. I am not interested in a new
> weak OmniOS community for a distribution that should be used as a
> production system. The OmniOS community will be propably too small forever
> as we already have the Illumos community project OpenIndiana nearly
> identical to OmniOS from distribution, features and use cases. And a very
> important thing: The brand OmniOS has already a very bad name as a
> dead/failed project in the press mostly affecting Illumos as well.
>
>
> *Plan B*
> OpenIndiana is a quite established community project for an up to date
> Illumos distribution. I would say its nearly identical to OmniOS beside the
> missing LX improvements from OmniOS but with an additional GUI option. I
> hope to see LX zones upstreamed to Illumos. OpenIndiana currently offers a
> rolling development of newest Illumos bits with snapshots every 6 months
> but without an additional stable repository with backported security fixes.
> Every update give you the newest Illumos fixes and features but also the
> newest bugs (ongoing dev, unstable).
>
> If OmniOS has to become a community project, I undoubtly would prefer a
> merge of the two distributions up from next releases. OpenIndiana with a
> stable repo for every snapshot and with a repo as development path would
> give me what was the main advantage of OmniOS beside commercial support.
> Access to such a stable repo optionally under an OmniOS brand may be even a
> paid (if affordable) option. Such a merge would strengthen Illumos at first
> place but also free OpenSource distributions like OmniOS and OpenIndiana.
>
>
> *Plan C*
> There is another free Illumos distribution with an enterprise background
> suited for datacenter use: SmartOS. It even adds unique Cloud and
> virtualisation features like KVM, Solaris zones, Linux zones and Docker
> support. As it is running from RAM with everything important on a datapool
> it is a very stable/ easy recoverable option but it lacks some features in
> the global zone that are required for a storage server.  An additional plus
> is the pkgin repo with lots of supported long term stable services.
>
> Using SmartOS would require a mechanism to allow storage services like
> SSH, Crossbow, iSCSI, NFS and SMB on the global zone with an option to
> save/restore settings from a datapool to be persistent. To be honest, a
> SmartOS that is capable to act as a storage appliance would be my dream
> option, at least as an additional option. This would require that SmartOS
> is not actively hindering this or preferable is helping to implement a
> save/restore option for global zone settings
>
>
> *Discuss*
> But whatever option is coming, end the Illumos fragmentation for the sake
> of one strong free community distribution with a solid number of
> contributors and one freely available with a commercial background like
> SmartOS/ Samsung owned.
>
>
> Any comments from OmniOS, OI, SmartOS (omnios-discuss,
> openindiana-discuss, smartos-discuss) communities?
> I have send this to all three lists, so 

[smartos-discuss] rpcbind memory leaks

[Youzhong Yang]

Hi all,

I am just wondering if you have seen this same issue. On one of our
servers, the rpcbind daemon used 3.6G memory; on another server, it
reported out of memory error:

 svc_vc_create: makefd_xprt  : out of memory
 do_accept : out of memory
xdrrec_create: out of memory
svc_fd_xprtcopy: strdup failed

It seems lots of 16-byte buffers are allocated but never released. I am
still struggling to find where it is.

Just want to ask around.

Thanks,

--Youzhong

# pmap 5840
5840:   /usr/sbin/rpcbind
08045000  12K rw---[ stack ]
0805  52K r-x--  /usr/sbin/rpcbind
0806D000   4K rw---  /usr/sbin/rpcbind
0806E000 3828440K rw---[ heap ]

 total   3831748K

# mdb core.5840
Loading modules: [ libumem.so.1 libc.so.1 libuutil.so.1 ld.so.1 ]
> ::umem_cache
ADDR NAME  FLAGCFLAG  BUFSIZE  BUFTOTL
.
0809a010 umem_alloc_16 0800 8000   16 226746054
...

> ::findleaks

*** mdb: received signal SEGV at:
[1] libumem.so`leaky_mtabcmp+0xa()
[2] libumem.so`findleaks+0x177()
[3] mdb`dcmd_invoke+0x40()
[4] mdb`mdb_call_idcmd+0x128()
[5] mdb`mdb_call+0x325()
[6] mdb`yyparse+0x3f7()
[7] mdb`mdb_run+0x26d()
[8] mdb`main+0x154c()
[9] mdb`_start+0x83()



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

[Youzhong Yang]

Hi Dan,

It's the following open call that failed:

23724:  open("/dev/overlay", O_RDWR|O_EXCL) Err#2 ENOENT

For some reason 'add_drv overlay' didn't load the driver successfully.
Running rem_drv and then add_drv resolved the issue. After that, I rebooted
the host several times, unfortunately the issue was not reproduced again. I
should have tried more digging before rebooting it.

This is the first time we had this varpd issue. If you happen to have more
about the possible cause of this failure, it would be nice to know.

If it happens again, definitely I will keep you guys updated.

Thanks,
-Youzhong

# add_drv overlay
Driver (overlay) is already installed.
# modinfo | grep overlay
# ls -l /dev/overlay
/dev/overlay: No such file or directory
# rem_drv overlay
Invalid argument
Cannot remove major number binding for 196
# modinfo | grep overlay
# add_drv overlay
# ls -l /dev/overlay
lrwxrwxrwx   1 root root  35 May 18 22:08 /dev/overlay ->
../devices/pseudo/overlay@0:overlay
# grep 196 name_to_major
overlay 196
# ls -l name_to_major
-rw-r--r--   1 root sys 2952 May 18 20:33 name_to_major
# svcs -xv
svc:/network/varpd:default (virtual ARP daemon)
 State: maintenance since May 18, 2018 at 03:53:23 PM EDT
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://illumos.org/msg/SMF-8000-KS
   See: /var/svc/log/network-varpd:default.log
Impact: This service is not running.
# svcadm clear network/varpd
# svcs -xv
# svcs varpd
STATE  STIMEFMRI
online 20:34:28 svc:/network/varpd:default
# reboot
# svcs -xv
# modinfo | grep overlay
279 f8712000   7488 196   1  overlay (Overlay Network Driver)


On Fri, May 18, 2018 at 5:16 PM, Dan McDonald <dan...@joyent.com> wrote:

>
>
> > On May 18, 2018, at 5:11 PM, Youzhong Yang <youzh...@gmail.com> wrote:
> >
> > Thanks Jason. In my environment, I've disabled KPTI through boot var
> 'kpti=0', not sure if it contributes to the issue.
> 
> kpti shouldn't affect anything.
> 
> Could you:
> 
> 1. ) ```truss -f -o /tmp/truss-out.txt /usr/lib/varpd/varpd```
> 
> 2. ) Assuming it's not too big, send the output to the list?
> 
> I'm curious what precisely failed with ESEARCH.
> 
> Thanks,
> Dan
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

[Youzhong Yang]

 illumos-joyent: OS-6963 add_fbuf should also use is_env_string
smartos-live:  OS-6958 bad comparison introduced with OS-6630

On Fri, May 18, 2018 at 4:11 PM, Jason King <jason.brian.k...@gmail.com>
wrote:

> What is the last commit in your branch that you built?   I’ll try to
> recreate this locally.
>
>
> From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
> Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.
> smartos.org> <smartos-discuss@lists.smartos.org>
> Date: May 18, 2018 at 3:04:31 PM
> To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
> <smartos-discuss@lists.smartos.org>
> Subject:  [smartos-discuss] service varpd not working
>
> Hi,
>
> Today I built SmartOS from the master branch. After rebooting the server,
> varpd did not start up:
>
> # svcs varpd
> STATE  STIMEFMRI
> maintenance15:53:23 svc:/network/varpd:default
>
> # tail /var/svc/log/network-varpd\:default.log
> [ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
> varpd: failed to open a libvarpd handle: No such file or directory
> [ May 18 15:53:23 Method "start" exited with status 95. ]
>
> # /usr/lib/varpd/varpd
> varpd: failed to open a libvarpd handle: No such file or directory
>
> # find /usr/lib -name 'libvarpd.so*' | xargs file
> /usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64 Version 1, 
> dynamically linked, not stripped, no debugging information available
> /usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1, 
> dynamically linked, not stripped, no debugging information available
>
> Not sure why, just wanna ask.
>
> Thanks,
> -Youzhong
>
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now> | Modify
> <https://www.listbox.com/member/?> Your Subscription
> <http://www.listbox.com>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] service varpd not working

[Youzhong Yang]

Thanks Jason. In my environment, I've disabled KPTI through boot var
'kpti=0', not sure if it contributes to the issue.

On Fri, May 18, 2018 at 5:06 PM, Jason King <jason.brian.k...@gmail.com>
wrote:

> I’m building an image from this commit and will get it loaded on a test
> Triton setup.  I’ll update the list once I’ve figured out what’s going on.
>
>
> From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
> Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.
> smartos.org> <smartos-discuss@lists.smartos.org>
> Date: May 18, 2018 at 3:46:31 PM
> To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
> <smartos-discuss@lists.smartos.org>
> Subject:  Re: [smartos-discuss] service varpd not working
>
> illumos-joyent: OS-6963 add_fbuf should also use is_env_string
> smartos-live:  OS-6958 bad comparison introduced with OS-6630
>
> On Fri, May 18, 2018 at 4:11 PM, Jason King <jason.brian.k...@gmail.com>
> wrote:
>
>> What is the last commit in your branch that you built?   I’ll try to
>> recreate this locally.
>>
>>
>> From: Youzhong Yang <youzh...@gmail.com> <youzh...@gmail.com>
>> Reply: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos
>> .org> <smartos-discuss@lists.smartos.org>
>> Date: May 18, 2018 at 3:04:31 PM
>> To: smartos-discuss@lists.smartos.org <smartos-discuss@lists.smartos.org>
>> <smartos-discuss@lists.smartos.org>
>> Subject:  [smartos-discuss] service varpd not working
>>
>> Hi,
>>
>> Today I built SmartOS from the master branch. After rebooting the server,
>> varpd did not start up:
>>
>> # svcs varpd
>> STATE  STIMEFMRI
>> maintenance15:53:23 svc:/network/varpd:default
>>
>> # tail /var/svc/log/network-varpd\:default.log
>> [ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
>> varpd: failed to open a libvarpd handle: No such file or directory
>> [ May 18 15:53:23 Method "start" exited with status 95. ]
>>
>> # /usr/lib/varpd/varpd
>> varpd: failed to open a libvarpd handle: No such file or directory
>>
>> # find /usr/lib -name 'libvarpd.so*' | xargs file
>> /usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64 Version 1, 
>> dynamically linked, not stripped, no debugging information available
>> /usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1, 
>> dynamically linked, not stripped, no debugging information available
>>
>> Not sure why, just wanna ask.
>>
>> Thanks,
>> -Youzhong
>>
>>
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now> | Modify
> <https://www.listbox.com/member/?> Your Subscription
> <http://www.listbox.com>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

[smartos-discuss] service varpd not working

[Youzhong Yang]

Hi,

Today I built SmartOS from the master branch. After rebooting the server,
varpd did not start up:

# svcs varpd
STATE  STIMEFMRI
maintenance15:53:23 svc:/network/varpd:default

# tail /var/svc/log/network-varpd\:default.log
[ May 18 15:53:23 Executing start method ("/lib/svc/method/svc-varpd"). ]
varpd: failed to open a libvarpd handle: No such file or directory
[ May 18 15:53:23 Method "start" exited with status 95. ]

# /usr/lib/varpd/varpd
varpd: failed to open a libvarpd handle: No such file or directory

# find /usr/lib -name 'libvarpd.so*' | xargs file
/usr/lib/amd64/libvarpd.so.1:   ELF 64-bit LSB dynamic lib AMD64
Version 1, dynamically linked, not stripped, no debugging information
available
/usr/lib/libvarpd.so.1: ELF 32-bit LSB dynamic lib 80386 Version 1,
dynamically linked, not stripped, no debugging information available

Not sure why, just wanna ask.

Thanks,
-Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

[smartos-discuss] diet too much = suicide ?

[Youzhong Yang]

Last few days I started evaluating the most recent build which has
KPTI/PCID fix.

Thanks to the excellent work done by Joyent. So far I've seen no issue,
except the expected performance degradation.

But there's one thing that caught my eye. A dozen of our perl scripts which
depend on /usr/perl5 now fail to run, for obvious reason described here:

https://smartos.org/bugview/OS-6563
put perl on a diet

The /usr/perl5 is pretty skinny now. It's so skinny that it's useless.

Oh, by the way, I tried to build SmartOS image under the latest, it failed
too:

Can't locate File/Find.pm in @INC (@INC contains:
/usr/perl5/site_perl/5.12/i86pc-solaris-64int /usr/perl5/site_perl/5.12
/usr/
perl5/5.12/lib/i86pc-solaris-64int /usr/perl5/5.12/lib .) at
./extract-copyright line 25.
BEGIN failed--compilation aborted at ./extract-copyright line 25.

Our scripts heavily depend on the module Sun::Solaris::Kstat, not sure if
there's any easy workaround except reverting the OS-6563 commit.

I just finished another build which reverted OS-6563, those scripts work as
before, no breakage.

Thanks.



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] fall out due to OS-5726 (bump pid max to a reasonable modern value)

[Youzhong Yang]

We set pidmax in /etc/system to 9, and our application allocates shared
memory based on the pidmax.

Due to OS-5726 ( bump pid max to a reasonable modern value):

https://smartos.org/bugview/OS-5726

https://github.com/joyent/illumos-joyent/commit/43b3bab58788e30af95d6dd211800182624ede96

It overrides our setting on a 512G mem system and set pidmax to 99.
Too bad, our apps kept crashing.

Setting our pidmax to 8 should work, but does it ring a bell that
OS-5726 does not respect user's setting in /etc/system?

Thanks.



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

[Youzhong Yang]

Hi Jason,

The image that had the issue was a full build, but last night I played git
reset --hard, git pull etc.., the image I got didn't reproduce the hang. So
Now I am starting all over again and see if I can reproduce.

Thanks.

On Mon, Aug 13, 2018 at 10:20 AM, Jason King 
wrote:

> That is strange — the same git repo that built that image was the one that
> was pushed to Gerrit and merged with master (i.e. it is the _same_
> commit).  Looking at the current master (as of a few minutes ago), the
> change doesn’t appear to be stepped on (it’s a very small change — it just
> moves the setting of a bit mask indicating the CPU has finished starting up
> to the last thing in the per-cpu startup thread (minus some diagnostic
> messages after startup and a call to thread_exit()).
>
> Did you try doing a 'gmake clobber’ in your SmartOS repo before building?
> Unfortunately, incremental (even just rebuilding illumos-joyent) doesn’t
> always work and can sometimes cause strange behavior.
>
> In the meantime, I’ll try doing a full build of the current master and
> installing it on my server here at home — it was very good at tripping the
> bug in OS-7079, so I’ll see if I can get it to hang (though it’ll take a
> bit to do a full build).
>
>
> From: Youzhong Yang  
> Reply: Youzhong Yang  
> Date: August 13, 2018 at 12:27:52 AM
>
> To: Jason King  
> Cc: smartos-discuss@lists.smartos.org 
> 
> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
> mp_startup_common races itself
>
> So your image booted up. Interesting ... Maybe something else messed up
> your fix?
> Anyway I am now building my image and see what I can get from ::cpustack.
>
> On Mon, Aug 13, 2018 at 1:14 AM, Jason King 
> wrote:
>
>> Doh.. the problems of it being late :) .. there should be a ‘public’ in
>> there.
>>
>> Try
>>
>> https://us-east.manta.joyent.com/jbk/public/OS-7079/platform
>> -20180719T001516Z.iso
>>
>>
>> From: Youzhong Yang  
>> Reply: Youzhong Yang  
>> Date: August 13, 2018 at 12:12:52 AM
>>
>> To: Jason King  
>> Cc: smartos-discuss@lists.smartos.org 
>> 
>> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
>> mp_startup_common races itself
>>
>> I got this:
>>
>> {"code":"ResourceNotFound","message":"/jbk/OS-7079/platform-20180719T001516Z.iso
>> does not exist"}
>>
>> In our /etc/system, I have
>> set pcplusmp:apic_panic_on_nmi=1
>> set apix:apic_panic_on_nmi=1
>>
>> If I set them to 0, and boot with -k, a NMI should drop into kmdb, right?
>> I will build an image now and test.
>>
>>
>> On Mon, Aug 13, 2018 at 1:04 AM, Jason King 
>> wrote:
>>
>>> There’s a couple of ways — you can boot -kd and set a breakpoint to set
>>> it.  You can also set it in etc/system in the proto area when building an
>>> image.
>>>
>>> If you want, I do have an image of 20180719 w/ OS-7079 applied and kmdb
>>> on NMI already set (you’d still want to boot -k)  — you can grab it at
>>> https://us-east.manta.joyent.com/jbk/OS-7079/platform-201807
>>> 19T001516Z.{iso,tgz,usb.bz2}
>>>
>>> If you do, it’d be interesting to see ::cpustack on each core looks like.
>>>
>>>
>>> From: Youzhong Yang  
>>> Reply: Youzhong Yang  
>>> Date: August 12, 2018 at 11:58:48 PM
>>> To: Jason King  
>>> Cc: smartos-discuss@lists.smartos.org >> .org> 
>>> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
>>> mp_startup_common races itself
>>>
>>> I sent NMI, but it printed out a stack trace plus a message "no dump
>>> device" or something then rebooted. I tried -v on my old supermicro system,
>>> on the console I saw message about sd## devices, then it hung. The console
>>> still responded to keyboard, but just stayed that way forever.
>>>
>>> What change is needed to drop into kmdb when the OS receives NMI?
>>>
>>> On Mon, Aug 13, 2018 at 12:06 AM, Jason King >> > wrote:
>>>
>>>> Was that with boot -v?  Are you able to send the system an NMI after it
>>>> hangs (or get the boot -v output up to the hang)?
>>>>
>>>> Prior to OS-7079, the system would start to startup the next CPU before
>>>> it had completely finished initializing the ‘current’ CPU (which could
>>>> deadlock depending on which CPU obtained a particular lock first), the
>>>> change makes it wait until the current CPU is finished starting up before
>>>

[smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

[Youzhong Yang]

Today I built a smartos image (with all git repos synced to master) and
rebooted the host with that image. It hung after the banner message + one
more line about power management or something.

Then I reverted OS-7079, built an image, rebooted, it worked perfectly.

So does it mean OS-7079 fixed one issue, but caused another? My host is an
old Supermicro X8DAH, Intel(R) Xeon(R) CPU X5570  @ 2.93GHz. Tomorrow I
will try on a new all NVMe system and see if it works.

Thanks.



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

[Youzhong Yang]

So I tried on our new Supermicro X11DPU system ( Intel(R) Xeon(R) Gold 6140
CPU @ 2.30GHz ), same issue, hung at boot, but with OS-7079 reverted, it
booted up successfully.

I can try NMI tomorrow and update.

On Mon, Aug 13, 2018 at 12:06 AM, Jason King 
wrote:

> Was that with boot -v?  Are you able to send the system an NMI after it
> hangs (or get the boot -v output up to the hang)?
>
> Prior to OS-7079, the system would start to startup the next CPU before it
> had completely finished initializing the ‘current’ CPU (which could
> deadlock depending on which CPU obtained a particular lock first), the
> change makes it wait until the current CPU is finished starting up before
> proceeding to the next CPU.
>
> It’s certainly possible it could have revealed another bug — OS-7079
> itself was introduced almost 10 years ago, but didn’t seem to be easy to
> trigger until recent CPUs.
>
>
> From: Youzhong Yang  
> Reply: smartos-discuss@lists.smartos.org  smartos.org> 
> Date: August 12, 2018 at 10:46:05 PM
> To: smartos-discuss@lists.smartos.org 
> 
> Subject:  [smartos-discuss] still hang at boot - OS-7079
> mp_startup_common races itself
>
> Today I built a smartos image (with all git repos synced to master) and
> rebooted the host with that image. It hung after the banner message + one
> more line about power management or something.
>
> Then I reverted OS-7079, built an image, rebooted, it worked perfectly.
>
> So does it mean OS-7079 fixed one issue, but caused another? My host is an
> old Supermicro X8DAH, Intel(R) Xeon(R) CPU X5570  @ 2.93GHz. Tomorrow I
> will try on a new all NVMe system and see if it works.
>
> Thanks.
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now> | Modify
> <https://www.listbox.com/member/?> Your Subscription
> <https://www.listbox.com>
>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

[Youzhong Yang]

I sent NMI, but it printed out a stack trace plus a message "no dump
device" or something then rebooted. I tried -v on my old supermicro system,
on the console I saw message about sd## devices, then it hung. The console
still responded to keyboard, but just stayed that way forever.

What change is needed to drop into kmdb when the OS receives NMI?

On Mon, Aug 13, 2018 at 12:06 AM, Jason King 
wrote:

> Was that with boot -v?  Are you able to send the system an NMI after it
> hangs (or get the boot -v output up to the hang)?
>
> Prior to OS-7079, the system would start to startup the next CPU before it
> had completely finished initializing the ‘current’ CPU (which could
> deadlock depending on which CPU obtained a particular lock first), the
> change makes it wait until the current CPU is finished starting up before
> proceeding to the next CPU.
>
> It’s certainly possible it could have revealed another bug — OS-7079
> itself was introduced almost 10 years ago, but didn’t seem to be easy to
> trigger until recent CPUs.
>
>
> From: Youzhong Yang  
> Reply: smartos-discuss@lists.smartos.org  smartos.org> 
> Date: August 12, 2018 at 10:46:05 PM
> To: smartos-discuss@lists.smartos.org 
> 
> Subject:  [smartos-discuss] still hang at boot - OS-7079
> mp_startup_common races itself
>
> Today I built a smartos image (with all git repos synced to master) and
> rebooted the host with that image. It hung after the banner message + one
> more line about power management or something.
>
> Then I reverted OS-7079, built an image, rebooted, it worked perfectly.
>
> So does it mean OS-7079 fixed one issue, but caused another? My host is an
> old Supermicro X8DAH, Intel(R) Xeon(R) CPU X5570  @ 2.93GHz. Tomorrow I
> will try on a new all NVMe system and see if it works.
>
> Thanks.
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now> | Modify
> <https://www.listbox.com/member/?> Your Subscription
> <https://www.listbox.com>
>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

[Youzhong Yang]

So your image booted up. Interesting ... Maybe something else messed up
your fix?
Anyway I am now building my image and see what I can get from ::cpustack.

On Mon, Aug 13, 2018 at 1:14 AM, Jason King 
wrote:

> Doh.. the problems of it being late :) .. there should be a ‘public’ in
> there.
>
> Try
>
> https://us-east.manta.joyent.com/jbk/public/OS-7079/
> platform-20180719T001516Z.iso
>
>
> From: Youzhong Yang  
> Reply: Youzhong Yang  
> Date: August 13, 2018 at 12:12:52 AM
>
> To: Jason King  
> Cc: smartos-discuss@lists.smartos.org 
> 
> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
> mp_startup_common races itself
>
> I got this:
>
> {"code":"ResourceNotFound","message":"/jbk/OS-7079/platform-20180719T001516Z.iso
> does not exist"}
>
> In our /etc/system, I have
> set pcplusmp:apic_panic_on_nmi=1
> set apix:apic_panic_on_nmi=1
>
> If I set them to 0, and boot with -k, a NMI should drop into kmdb, right?
> I will build an image now and test.
>
>
> On Mon, Aug 13, 2018 at 1:04 AM, Jason King 
> wrote:
>
>> There’s a couple of ways — you can boot -kd and set a breakpoint to set
>> it.  You can also set it in etc/system in the proto area when building an
>> image.
>>
>> If you want, I do have an image of 20180719 w/ OS-7079 applied and kmdb
>> on NMI already set (you’d still want to boot -k)  — you can grab it at
>> https://us-east.manta.joyent.com/jbk/OS-7079/platform-201807
>> 19T001516Z.{iso,tgz,usb.bz2}
>>
>> If you do, it’d be interesting to see ::cpustack on each core looks like.
>>
>>
>> From: Youzhong Yang  
>> Reply: Youzhong Yang  
>> Date: August 12, 2018 at 11:58:48 PM
>> To: Jason King  
>> Cc: smartos-discuss@lists.smartos.org 
>> 
>> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
>> mp_startup_common races itself
>>
>> I sent NMI, but it printed out a stack trace plus a message "no dump
>> device" or something then rebooted. I tried -v on my old supermicro system,
>> on the console I saw message about sd## devices, then it hung. The console
>> still responded to keyboard, but just stayed that way forever.
>>
>> What change is needed to drop into kmdb when the OS receives NMI?
>>
>> On Mon, Aug 13, 2018 at 12:06 AM, Jason King 
>> wrote:
>>
>>> Was that with boot -v?  Are you able to send the system an NMI after it
>>> hangs (or get the boot -v output up to the hang)?
>>>
>>> Prior to OS-7079, the system would start to startup the next CPU before
>>> it had completely finished initializing the ‘current’ CPU (which could
>>> deadlock depending on which CPU obtained a particular lock first), the
>>> change makes it wait until the current CPU is finished starting up before
>>> proceeding to the next CPU.
>>>
>>> It’s certainly possible it could have revealed another bug — OS-7079
>>> itself was introduced almost 10 years ago, but didn’t seem to be easy to
>>> trigger until recent CPUs.
>>>
>>>
>>> From: Youzhong Yang  
>>> Reply: smartos-discuss@lists.smartos.org >> .org> 
>>> Date: August 12, 2018 at 10:46:05 PM
>>> To: smartos-discuss@lists.smartos.org >> .org> 
>>> Subject:  [smartos-discuss] still hang at boot - OS-7079
>>> mp_startup_common races itself
>>>
>>> Today I built a smartos image (with all git repos synced to master) and
>>> rebooted the host with that image. It hung after the banner message + one
>>> more line about power management or something.
>>>
>>> Then I reverted OS-7079, built an image, rebooted, it worked perfectly.
>>>
>>> So does it mean OS-7079 fixed one issue, but caused another? My host is
>>> an old Supermicro X8DAH, Intel(R) Xeon(R) CPU X5570  @ 2.93GHz. Tomorrow I
>>> will try on a new all NVMe system and see if it works.
>>>
>>> Thanks.
>>> *smartos-discuss* | Archives
>>> <https://www.listbox.com/member/archive/184463/=now> | Modify
>>> <https://www.listbox.com/member/?> Your Subscription
>>> <https://www.listbox.com>
>>>
>>>
>>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

[smartos-discuss] zvol image on image server

[Youzhong Yang]

If I need to store our own kvm image on our server, what format of the
image file should be? zfs send stream and compressed by gzip? where to put
the file? together with the manifest file?

The following image on joyent server does not say anything about the file
location. I have no idea what it is:

https://images.joyent.com/images/0ed8e3fa-154e-4302-8d4e-d74f877dbf37

  "files": [
{
  "sha1": "55719bec7dcde369ca8bfdff8e289335d798216e",
  "size": 329064188,
  "compression": "gzip"
}
  ],

Just want to ask before digging in.

Thanks,

--Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] zvol image on image server

[Youzhong Yang]

Thanks everyone.

linux-prepare-image rocks! I was able to modify and use it based on our
needs.

Once the manifest and zvol stream file are created, I was able to upload
them to our image repository and use it on other smartos servers through
imgadm import.

On Wed, Jun 27, 2018 at 5:50 PM, Mike Gerdts  wrote:

> The easiest way to create an image from an existing kvm instance is with:
>
> src_vm_uuid=FIXME
> name=my-first-image
> version=1
> imgadm create -i $src_vm_uuid -s linux-prepare-image version=$version
> name=$name
>
> You can find linux-prepare-image at
> https://raw.githubusercontent.com/joyent/sdc-imgapi/master/
> tools/prepare-image/linux-prepare-image
>
> You can then install that image on the server with
>
> imgadm install -f $name-$version.zvol -m $name-$version.imgmanifest
>
> HTH,
> Mike
>
>
> On Wed, Jun 27, 2018 at 1:53 PM, Youzhong Yang  wrote:
>
>> If I need to store our own kvm image on our server, what format of the
>> image file should be? zfs send stream and compressed by gzip? where to put
>> the file? together with the manifest file?
>>
>> The following image on joyent server does not say anything about the file
>> location. I have no idea what it is:
>>
>> https://images.joyent.com/images/0ed8e3fa-154e-4302-8d4e-d74f877dbf37
>>
>>   "files": [
>> {
>>   "sha1": "55719bec7dcde369ca8bfdff8e289335d798216e",
>>   "size": 329064188,
>>   "compression": "gzip"
>> }
>>   ],
>>
>> Just want to ask before digging in.
>>
>> Thanks,
>>
>> --Youzhong
>>
>>
> *smartos-discuss* | Archives
> <https://www.listbox.com/member/archive/184463/=now> | Modify
> <https://www.listbox.com/member/?> Your Subscription
> <http://www.listbox.com>
>



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] still hang at boot - OS-7079 mp_startup_common races itself

[Youzhong Yang]

I finally figured out it's my issue. My branch has a fix for
https://www.illumos.org/issues/7395, when merging the master branch, the
upstream stuff stepped over my fix and caused the boot to hang. Sorry for
the fuss, and thanks a lot for your attention and kindness.


On Mon, Aug 13, 2018 at 1:43 PM, Jason King 
wrote:

> I also built master from this morning 
> (745f0bf63f3e20e5c1f0b83d85eaa4e99efcf441)
> and so far has been ok.
>
> If you do need to do kmdb on nmi, the easiest way I’ve found is something
> similar to this diff (in smartos-live):
>
> diff --git a/overlay/generic/etc/system b/overlay/generic/etc/system
> index 62b96ef..2fded09 100644
> --- a/overlay/generic/etc/system
> +++ b/overlay/generic/etc/system
> @@ -113,8 +113,8 @@ set ip:ip_squeue_fanout=1
>  *
>  * Machines should take a crash dump and reboot when receiving an NMI
>  *
> -set pcplusmp:apic_panic_on_nmi=1
> -set apix:apic_panic_on_nmi=1
> +set pcplusmp:apic_kmdb_on_nmi=1
> +set apix:apic_kmdb_on_nmi=1
>
>  *
>  * Don't use multi-threaded fast crash dump or a high compression level
>
>
> You can boot -k and do some kmdb tricks to set a breakpoint at the right
> spot and set those in kmdb (you need to set a breakpoint some point after
> the modules are loaded, but before it hits the hang), but I found it was a
> bigger hassle to get it right than just producing a special-purpose image
> (I just keep a branch with just the above change I can rebase/checkout/etc
> as needed) — though maybe those with better mdb-fu than me have better luck
> with that approach.
>
>
> From: Youzhong Yang  
> Reply: Youzhong Yang  
> Date: August 13, 2018 at 9:29:07 AM
>
> To: Jason King  
> Cc: smartos-discuss@lists.smartos.org 
> 
> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
> mp_startup_common races itself
>
> Hi Jason,
>
> The image that had the issue was a full build, but last night I played git
> reset --hard, git pull etc.., the image I got didn't reproduce the hang. So
> Now I am starting all over again and see if I can reproduce.
>
> Thanks.
>
> On Mon, Aug 13, 2018 at 10:20 AM, Jason King 
> wrote:
>
>> That is strange — the same git repo that built that image was the one
>> that was pushed to Gerrit and merged with master (i.e. it is the _same_
>> commit).  Looking at the current master (as of a few minutes ago), the
>> change doesn’t appear to be stepped on (it’s a very small change — it just
>> moves the setting of a bit mask indicating the CPU has finished starting up
>> to the last thing in the per-cpu startup thread (minus some diagnostic
>> messages after startup and a call to thread_exit()).
>>
>> Did you try doing a 'gmake clobber’ in your SmartOS repo before
>> building?  Unfortunately, incremental (even just rebuilding illumos-joyent)
>> doesn’t always work and can sometimes cause strange behavior.
>>
>> In the meantime, I’ll try doing a full build of the current master and
>> installing it on my server here at home — it was very good at tripping the
>> bug in OS-7079, so I’ll see if I can get it to hang (though it’ll take a
>> bit to do a full build).
>>
>>
>> From: Youzhong Yang  
>> Reply: Youzhong Yang  
>> Date: August 13, 2018 at 12:27:52 AM
>>
>> To: Jason King  
>> Cc: smartos-discuss@lists.smartos.org 
>> 
>> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
>> mp_startup_common races itself
>>
>> So your image booted up. Interesting ... Maybe something else messed up
>> your fix?
>> Anyway I am now building my image and see what I can get from ::cpustack.
>>
>> On Mon, Aug 13, 2018 at 1:14 AM, Jason King 
>> wrote:
>>
>>> Doh.. the problems of it being late :) .. there should be a ‘public’ in
>>> there.
>>>
>>> Try
>>>
>>> https://us-east.manta.joyent.com/jbk/public/OS-7079/platform
>>> -20180719T001516Z.iso
>>>
>>>
>>> From: Youzhong Yang  
>>> Reply: Youzhong Yang  
>>> Date: August 13, 2018 at 12:12:52 AM
>>>
>>> To: Jason King  
>>> Cc: smartos-discuss@lists.smartos.org >> .org> 
>>> Subject:  Re: [smartos-discuss] still hang at boot - OS-7079
>>> mp_startup_common races itself
>>>
>>> I got this:
>>>
>>> {"code":"ResourceNotFound","message":"/jbk/OS-7079/platform-20180719T001516Z.iso
>>> does not exist"}
>>>
>>> In our /etc/system, I have
>>> set pcplusmp:apic_panic_on_nmi=1
>>> set apix:apic_panic_on_nmi=1
>>>
>>> If I set t

Re: [smartos-discuss] ACPI Testing Request

[Youzhong Yang]

Supermicro X11DPU, BIOS 2.1, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz

No issue.

On Thu, Aug 16, 2018 at 12:01 PM, Robert Mustacchi  wrote:

> Hi All,
> 
> There have been a number of issues with boot hangs on some of the more
> recent Kaby Lake processors and a few Skylake SKUs. We were able to root
> cause this to a deadlock in the core ACPICA code. For the full details
> see https://github.com/joyent/smartos-live/issues/727 and
> https://smartos.org/bugview/OS-7093.
> 
> Since ACPI changes can be a bit gritty, I would like to ask for a bit
> more help in testing this across a variety of platforms -- in particular
> Desktop platforms. I've put together a series of test images that have
> the newer ACPI and also end up logging substantially more ACPI related
> information to the console in case something goes wrong (particularly on
> debug bits).
> 
> If you could test this and ensure that you can boot and reboot OK, I
> would greatly appreciate it. I have both debug and non-debug media. If
> you'd like to build this yourself, the changes to illumos-joyent that
> we've made are available at
> https://github.com/rmustacc/illumos-gate/tree/acpi-dev-smartos. I'd also
> like to thank Mike Gerdts who wrote a bunch of tools for updating the
> ACPI tree in illumos which has made this effort substantially easier.
> Based on that we're now able to better track how we're handling changes
> and revisions to ACPI. That's available at
> https://github.com/joyent/acpica/tree/joyent/20180629-wip.
> 
> Please note that these images are based on a platform from last week.
> 
> non-debug raw platform:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/platform-20180807T230146Z.tgz
> 
> non-debug ISO vga:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/acpi-nd-vga.iso
> 
> non-debug ISO ttya:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/acpi-nd-ttya.iso
> 
> non-debug ISO ttyb:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/acpi-nd-ttyb.iso
> 
> non-debug USB vga:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/acpi-nd-vga.usb.bz2
> 
> non-debug USB ttya:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/acpi-nd-ttya.usb.bz2
> 
> non-debug USB ttyb:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/non-debug/acpi-nd-ttyb.usb.bz2
> 
> debug raw platform:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/platform-20180807T223604Z.tgz
> 
> debug ISO vga:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/acpi-debug-vga.iso
> 
> debug ISO ttya:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/acpi-debug-ttya.iso
> 
> debug ISO ttyb:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/acpi-debug-ttyb.iso
> 
> debug USB vga:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/acpi-debug-vga.usb.bz2
> 
> debug USB ttya:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/acpi-debug-ttya.usb.bz2
> 
> debug USB ttyb:
> https://us-east.manta.joyent.com/rmustacc/public/preview/
> acpi-201808/debug/acpi-debug-ttyb.usb.bz2
> 
> Again, thank you in advance for giving this a shot. Whether it works and
> especially if it does not for some reason, if you test this, can you
> please reply and let me know what the motherboard, processor, and BIOS
> revision that you're using are?
> 
> Thanks,
> Robert
> 



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125
Powered by Listbox: https://www.listbox.com

[smartos-discuss] iostat hangs

[Youzhong Yang via smartos-discuss]

Hi All,

On one of our servers running SmartOS, the 'iostat' hung and there're a
bunch of stuck 'iostat' commands.

It seems everyone of them has the same kernel stack:

stack pointer for thread ffbe78d0f3a0: ff0173d84380
[ ff0173d84380 _resume_from_idle+0xf4() ]
  ff0173d843b0 swtch+0x141()
  ff0173d843f0 cv_wait+0x70(ffbe78d0f58e, ffbe78d0f590)
  ff0173d84450 delay_common+0xb8(1)
  ff0173d84490 delay+0x30(1)
  ff0173d845a0 ufs_dirlook+0xbd(ff3284c2b778, ff0173d847a0,
  ff0173d845d0, ffc18e7bf918, 1, 0)
  ff0173d84640 ufs_lookup+0x191(ff3284b5da00, ff0173d847a0,
  ff0173d848e8, ff0173d84a70, 0, ff3257d44840,
ffc18e7bf918, 0,
  0, 0)
  ff0173d846f0 fop_lookup+0xa2(ff3284b5da00, ff0173d847a0,
  ff0173d848e8, ff0173d84a70, 0, ff3257d44840,
ffc18e7bf918, 0,
  0, 0)
  ff0173d84950 lookuppnvp+0x1f6(ff0173d84a70, 0, 1, 0,
ff0173d84c88
  , ff3257d44840, ff3257d44840, ffc18e7bf918)
  ff0173d849f0 lookuppnatcred+0x15e(ff0173d84a70, 0, 1, 0,
  ff0173d84c88, 0, ffc18e7bf918)
  ff0173d84af0 lookupnameatcred+0xad(8046e68, 0, 1, 0,
ff0173d84c88, 0,
  ffc18e7bf918)
  ff0173d84b40 lookupnameat+0x39(8046e68, 0, 1, 0, ff0173d84c88, 0)
  ff0173d84cf0 vn_openat+0x315(8046e68, 0, 1, 1a4, ff0173d84e00, 0,
  ff010012, 0, 4)
  ff0173d84e60 copen+0x20c(ffd19553, 8046e68, 1, 1a4)
  ff0173d84e90 openat32+0x27(ffd19553, 8046e68, 0, 1a4)
  ff0173d84ec0 open32+0x25(8046e68, 0, 1a4)
  ff0173d84f10 _sys_sysenter_post_swapgs+0x149()

Is this a known issue? how can I further debug this issue on the server? is
there a way to break out of this stuckness?

Thanks,

-youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: [developer] Issue 4965: NLM: Mac OS X client can be stuck at locking a NFS file on an illumos server (SmartOS, Nexenta or OmniOS etc.) running open source lockd

[Youzhong Yang via smartos-discuss]

Thanks for helping us solve this issue.



In our use case, we need the NULL_RPC timeout to be at least lower than the
client LOCK request timeout…otherwise our client Mac processes stall/fail
–that’s the real pain point we need to solve.



We don’t see any use that in which the illumos server must wait/recover
across a Mac reboot. While it certainly seems like a valid use-case, it’s
not one that we expect to encounter. But please note, we’d be more than
happy to test/validate any such scenario if helps.



After thinking more about it, what I’d like to do is make the timeout value
configurable (via mdb) – that way, it can be defaulted to whatever value
you feel serves best for the most affected, and we will be able to adjust
it to our use-case pain points.


Below is our proposed patch, your comments/advices are appreciated.


Thank you,


- Youzhong


-

diff --git a/usr/src/uts/common/klm/nlm_impl.c
b/usr/src/uts/common/klm/nlm_impl.c

index 7daa30d..c2f7178 100644

--- a/usr/src/uts/common/klm/nlm_impl.c

+++ b/usr/src/uts/common/klm/nlm_impl.c

@@ -124,6 +124,11 @@ krwlock_t lm_lck;

 static const struct timeval nlm_rpctv_zero = { 0,  0 };


 /*

+ * Initial timeout for NLM NULL RPC

+ */

+static volatile struct timeval nlm_nullrpc_wait = { 0, 20 };

+

+/*

  * List of all Zone globals nlm_globals instences

  * linked together.

  */

@@ -527,6 +532,20 @@ nlm_clnt_call(CLIENT *clnt, rpcproc_t procnum,
xdrproc_t xdr_args,

wait = nlm_rpctv_zero;


/*

+* Default timeout value of 25 seconds can take

+* nlm_null_rpc() 150 seconds to return RPC_TIMEDOUT

+* if it uses UDP and the destination port is

+* unreachable.

+*

+* A shorter timeout value, e.g. 200 milliseconds,

+* will cause nlm_null_rpc() to time out after

+* 200 * (1 + 2 + 4 + 8 + 16 + 32) = 12.6 seconds

+* (with retries set to 5)

+*/

+   if (procnum == NLM_NULL)

+   wait = nlm_nullrpc_wait;

+

+   /*

 * We need to block signals in case of NLM_CANCEL RPC

 * in order to prevent interruption of network RPC

 * calls.

diff --git a/usr/src/uts/common/klm/nlm_rpc_handle.c
b/usr/src/uts/common/klm/nlm_rpc_handle.c

index 9ddf568..26397b3 100644

--- a/usr/src/uts/common/klm/nlm_rpc_handle.c

+++ b/usr/src/uts/common/klm/nlm_rpc_handle.c

@@ -55,6 +55,7 @@

(_status) == RPC_PROGVERSMISMATCH ||\

(_status) == RPC_PROCUNAVAIL || \

(_status) == RPC_CANTCONNECT || \

+   (_status) == RPC_TIMEDOUT ||\

(_status) == RPC_XPRTFAILED)


 static struct kmem_cache *nlm_rpch_cache = NULL;




On Tue, Jul 8, 2014 at 2:44 PM, Marcel Telka mar...@telka.sk wrote:

 Hi,

 Nice work. Please find my comments inline.

 On Tue, Jul 08, 2014 at 01:43:36PM -0400, Youzhong Yang via
 illumos-developer wrote:
  I made the following two changes and built a new image, the issue goes
 away.
 
  --- a/usr/src/uts/common/klm/nlm_impl.c
  +++ b/usr/src/uts/common/klm/nlm_impl.c
  @@ -525,6 +525,12 @@ nlm_clnt_call(CLIENT *clnt, rpcproc_t procnum,
  xdrproc_t xdr_args,
   */
  if (procnum = NLM_TEST_RES  procnum = NLM_GRANTED_RES)
  wait = nlm_rpctv_zero;
  +   if (procnum == NLM_NULL) {
  +   wait.tv_sec = 0;
  +   wait.tv_usec = 25000;
  +   }
  +
 
  --- a/usr/src/uts/common/klm/nlm_rpc_handle.c
  +++ b/usr/src/uts/common/klm/nlm_rpc_handle.c
  @@ -55,6 +55,7 @@
  (_status) == RPC_PROGVERSMISMATCH ||\
  (_status) == RPC_PROCUNAVAIL || \
  (_status) == RPC_CANTCONNECT || \
  +   (_status) == RPC_TIMEDOUT ||\
  (_status) == RPC_XPRTFAILED)
 

 Please add the following (or similar) note as a comment close to your
 modification in nlm_clnt_call() so the explanation why we do need to
 adjust the
 default timeout is retained for future.

  Setting timeout value of NULL rpc to 25 milliseconds instead of the
 default
  25 seconds can make nlm_null_rpc() returns RPC_TIMEDOUT after 1575
  milliseconds when the UDP port is not reachable:
 1575 = 25 + 50 + 100 + 200 + 400 + 800 = 5 retries

 In general, this should work, but I'm not sure your suggestion of 25 ms
 (for
 the initial timoeut) and 1.5 sec (as the total timeout) is long enough.
 Assuming the client (Mac OS) rebooted.  The reboot is slow operation, so
 there
 is no need to be so fast (1.5 sec) in detecting it.  I think we should
 push the
 total timeout to ca 30 seconds (with the initial timeout at 0.5 sec) or
 maybe
 even more to avoid spurious stale clients detection.


 Thank you.

 --
 +---+
 | Marcel Telka   e-mail:   mar...@telka.sk  |
 |homepage: http://telka.sk/ |
 |jabber:   mar...@jabber.sk

[smartos-discuss] Issue with LSI 3108 MegaRAID ROMB card

[Youzhong Yang via smartos-discuss]

Hi All,

We have problem using the LSI 3108 card, just wondering if anyone here has
any success story using this card in production.

Here is the FM version info and error we got in /var/adm/messages:

BIOS Version   : 6.13.00_4.14.05.00_0x06010600
Ctrl-R Version : 5.01-0004
FW Version : 4.210.10-2910
NVDATA Version : 3.1310.00-0054
Boot Block Version : 3.00.00.00-0009

Jul 15 21:01:56  mr_sas: [ID 270009 kern.warning] WARNING:
io_timeout_checker: FW Fault, calling reset adapter
Jul 15 21:01:56  mr_sas: [ID 643100 kern.notice]
io_timeout_checker: fw_outstanding 0x17 max_fw_cmds 0x39F
Jul 15 21:01:59  mr_sas: [ID 347913 kern.warning] WARNING:
mrsas_tbolt_reset_ppc: FW is in fault after OCR count 1 Retry Reset
Jul 15 21:02:09  mr_sas: [ID 887724 kern.warning] WARNING:
mrsas_tbolt_reset_ppc:resetadapter bit is set already check retry
count 101

Thanks,

-Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: [OmniOS-discuss] Issue with LSI 3108 MegaRAID ROMB card

[Youzhong Yang via smartos-discuss]

Thanks Dan.

We ordered these Supermicro X9DRW-CF/CTF boxes which have ROMB LSI 3108 on
the motherboard and got stuck. We will probably add 9211-8i HBA cards to
the machines and get them move forward.

Thanks,
Youzhong



On Thu, Jul 17, 2014 at 11:15 AM, Dan McDonald dan...@omniti.com wrote:


 On Jul 17, 2014, at 11:07 AM, Youzhong Yang youzh...@gmail.com wrote:

  Hi All,
 
  We have problem using the LSI 3108 card, just wondering if anyone here
 has any success story using this card in production.

 SNIP!

 When mr_sas(7d) was updated for 2208, it included untested 3108 support.
  3108 was untested because people didn't have 3108 cards at the time it
 went back.

 The messages you're seeing indicate the card's timing-out an IO operation,
 followed by a reset-the-card failure.

 Beyond that, I can't help you much right now.  I have no such card
 available to me.  For the record, are you stuck using this, or did you
 choose a 3108?  I'd recommend choosing something else if it was your
 choice.  If it's not, please tell me what platform stuck you with a 3108,
 as it may be a harbinger of future complaints.

 Thanks,
 Dan





---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] Re: [OmniOS-discuss] Issue with LSI 3108 MegaRAID ROMB card

[Youzhong Yang via smartos-discuss]

Thanks Keith.

We upgraded the firmware to its latest but still no luck, so likely we'll
give up.

BIOS Version   : 6.17.04.0_4.16.08.00_0x06060A01

Ctrl-R Version : 5.04-0002

Preboot CLI Version: 01.07-05:#%

FW Version : 4.230.20-3532

NVDATA Version : 3.1403.00-0079

Boot Block Version : 3.02.00.00-0001



Jul 17 13:08:35 batfs0388 mr_sas: [ID 643100 kern.notice]
io_timeout_checker: fw_outstanding 0x17 max_fw_cmds 0x39F

Jul 17 13:08:38 batfs0388 mr_sas: [ID 347913 kern.warning] WARNING:
mrsas_tbolt_reset_ppc: FW is in fault after OCR count 1 Retry Reset

Jul 17 13:08:48 batfs0388 mr_sas: [ID 887724 kern.warning] WARNING:
mrsas_tbolt_reset_ppc:resetadapter bit is set already check retry count 101

Jul 17 13:08:49 batfs0388 mr_sas: [ID 270009 kern.warning] WARNING:
io_timeout_checker: FW Fault, calling reset adapter



On Thu, Jul 17, 2014 at 1:01 PM, Keith Wesolowski 
keith.wesolow...@joyent.com wrote:

 On Thu, Jul 17, 2014 at 11:59:16AM -0400, Youzhong Yang via
 smartos-discuss wrote:

  We ordered these Supermicro X9DRW-CF/CTF boxes which have ROMB LSI 3108
 on
  the motherboard and got stuck. We will probably add 9211-8i HBA cards to
  the machines and get them move forward.

 The 9207-8i is likely a better fit; it comes standard with IT firmware
 vs IR in the 9211-8i.  Both do work, however.  SMCI makes a similar
 board, the X9DRD-7LN4F-JBOD, which has the same 2308-IT on board as the
 9207-8i.  I recommend using that instead of the X9DRW unless you're
 wedded to the WIO form factor.




---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] epoll_ctl and epoll_wait on SmartOS

[Youzhong Yang via smartos-discuss]

Hi All,

The man page of epoll_ctl says:

 The data field specifies the datum  to  be  associated  with  the
 event  and  will be returned via epoll_wait(3C). The events field
 denotes both the desired events (when specified via  epoll_ctl())
 and   the   events   that   have   occurred  (when  returned  via
 epoll_wait(3C)).

This does not seem to be the case and is not compatible with Linux's
implementation. Recently I tried to compile Samba 4.1.10 on smartos and
'net ads join' just crashes. Samba uses epoll_event.data.ptr to keep track
of its tevent_fd data but epoll_ctl() on smartos simply ignores it and does
not keep a copy of the 'data' structure and returns it when epoll_wait is
called.

Can I assume this is an implementation specific bug?

Thanks,

-Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] epoll_ctl and epoll_wait on SmartOS

[Youzhong Yang via smartos-discuss]

Thanks Bryan.

I take back what I said. epoll_* does not ignore epoll_event.data passed in.

I managed to reproduce the issue with a program rewritten from yours:

# gcc myepoll.c
# ./a.out
EPOLL_CTL_ADD: fd = 4, ptr = fd7fffdffba0
writing to pipe fd 5
written 6 bytes to fd 5
Got an event on fd 4, events = 1, ptr = fd7fffdffba0
got 6 bytes from fd 5
EPOLL_CTL_DEL: fd = 4, ptr = fd7fffdffba0
EPOLL_CTL_ADD: fd = 4, ptr = fd7fffdffb80
writing to pipe fd 5
written 6 bytes to fd 5
Got an event on fd 4, events = 1, ptr = fd7fffdffba0
got 6 bytes from fd 5

As it indicates, the issue here is that after EPOLL_CTL_DEL, and then
EPOLL_CTL_ADD of the same file descriptor with a new data.ptr, epoll_wait
still returns an old pointer.

C code is attached for your reference.

Thanks very much,

-Youzhong


On Wed, Jul 30, 2014 at 5:38 PM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 Seems like a bug, but not sure I entirely understand:  there may be some
 edge conditions where data specified in the data field of the epoll_event
 structure via epoll_ctl() is not properly returned, but it's not true that
 SmartOS simply ignores it.  Take, for example, the following program:

   https://us-east.manta.joyent.com/bcantrill/public/epoll_add.c

 (It's essentially the most basic possible use of epoll(5).)  Compiling
 that program and running it yields (for me, anyway) the same result on
 SmartOS as it does on Linux:

   $ ./epoll_add
   writing to pipe
   adding event...
   parent: got an event on fd 4

 I certainly don't doubt that there's a bug here -- I just don't think it's
 quite as broad as you're implying; could you point me to the core file for
 Samba and/or instructions as to how to reproduce the issue?

 - Bryan



 On Wed, Jul 30, 2014 at 2:08 PM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hi All,

 The man page of epoll_ctl says:

  The data field specifies the datum  to  be  associated  with  the
  event  and  will be returned via epoll_wait(3C). The events field
  denotes both the desired events (when specified via  epoll_ctl())
  and   the   events   that   have   occurred  (when  returned  via
  epoll_wait(3C)).

 This does not seem to be the case and is not compatible with Linux's
 implementation. Recently I tried to compile Samba 4.1.10 on smartos and
 'net ads join' just crashes. Samba uses epoll_event.data.ptr to keep track
 of its tevent_fd data but epoll_ctl() on smartos simply ignores it and does
 not keep a copy of the 'data' structure and returns it when epoll_wait is
 called.

 Can I assume this is an implementation specific bug?

 Thanks,

 -Youzhong
  *smartos-discuss* | Archives
 https://www.listbox.com/member/archive/184463/=now
 https://www.listbox.com/member/archive/rss/184463/21493597-819305a5 |
 Modify
 https://www.listbox.com/member/?;
 Your Subscription http://www.listbox.com






---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
#include unistd.h
#include stdio.h
#include stdlib.h
#include string.h
#include errno.h
#include signal.h
#include limits.h
#include pthread.h
#include strings.h
#include sys/epoll.h

struct tevent_fd {
	int fd;
	void *private_data;
	void *additional_data;
};

void add_or_del(int epoll_fd, int fd, int op, int events, void *ptr)
{
	struct epoll_event evt;

	if(op != EPOLL_CTL_ADD  op != EPOLL_CTL_DEL){
		printf(only handle EPOLL_CTL_ADD or EPOLL_CTL_DEL\n);
		return;
	}

	bzero(evt, sizeof (evt));
	evt.events = events; 
	evt.data.ptr = ptr;

	if (epoll_ctl(epoll_fd, op, fd, evt)  0)
		perror(epoll_ctl);
	else
		printf(%s: fd = %d, ptr = %p\n, 
			(op == EPOLL_CTL_ADD) ? EPOLL_CTL_ADD : EPOLL_CTL_DEL, fd, ptr);

	return;
}

void wait_event(int epoll_fd, int timeout)
{
	struct epoll_event evt;
	int ret;

	bzero(evt, sizeof(evt));
	ret = epoll_wait(epoll_fd, evt, 1, timeout);
	if(ret == 1) {
		printf(Got an event on fd %d, events = %d, ptr = %p\n, 
			((struct tevent_fd*)evt.data.ptr)-fd, evt.events, evt.data.ptr);
	}
	else if(ret == -1) {
		perror(epoll_wait);
	}
	else {
		printf(No event!\n);
	}
}

int
main(int argc, char **argv)
{
	int pipefds[2];
	int epoll_fd;
	struct tevent_fd fde1, fde2;
	char buf[64];
	int nbytes;

	if ((epoll_fd = epoll_create1(EPOLL_CLOEXEC)) == -1) {
		perror(epoll_create1);
		return (1);
	}

	if (pipe(pipefds)  0) {
		perror(pipe());
		return (1);
	}

	bzero(fde1, sizeof(fde1));
	fde1.fd = pipefds[0];
	add_or_del(epoll_fd, pipefds[0], EPOLL_CTL_ADD, EPOLLIN, fde1);
	printf(writing to pipe fd %d\n, pipefds[1]);
	nbytes = write(pipefds[1], foo!\n, 6);
	printf(written %d bytes to fd %d\n, nbytes, pipefds[1]);
	wait_event(epoll_fd, 5000);
	nbytes = read(pipefds[0

Re: [smartos-discuss] epoll_ctl and epoll_wait on SmartOS

[Youzhong Yang via smartos-discuss]

Thanks very much for the quick fix.

I'll build a new image and also samba, then report back after testing.

-Youzhong


On Fri, Aug 1, 2014 at 4:56 AM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 Perfect -- thank you!  As it turns out, this was -- as you surmised -- a
 bug in our epoll implementation.  I have pushed the fix:


 https://github.com/joyent/illumos-joyent/commit/471f1f3ef02e25a84d7c36c857ed260ae2b7ac99

 Thanks for taking the time for the clean test case -- and apologies for
 the bug!

  - Bryan


 On Thu, Jul 31, 2014 at 7:28 AM, Youzhong Yang youzh...@gmail.com wrote:

 Thanks Bryan.

 I take back what I said. epoll_* does not ignore epoll_event.data passed
 in.

 I managed to reproduce the issue with a program rewritten from yours:

 # gcc myepoll.c
 # ./a.out
 EPOLL_CTL_ADD: fd = 4, ptr = fd7fffdffba0
 writing to pipe fd 5
 written 6 bytes to fd 5
 Got an event on fd 4, events = 1, ptr = fd7fffdffba0
 got 6 bytes from fd 5
 EPOLL_CTL_DEL: fd = 4, ptr = fd7fffdffba0
 EPOLL_CTL_ADD: fd = 4, ptr = fd7fffdffb80
 writing to pipe fd 5
 written 6 bytes to fd 5
 Got an event on fd 4, events = 1, ptr = fd7fffdffba0
 got 6 bytes from fd 5

 As it indicates, the issue here is that after EPOLL_CTL_DEL, and then
 EPOLL_CTL_ADD of the same file descriptor with a new data.ptr, epoll_wait
 still returns an old pointer.

 C code is attached for your reference.

 Thanks very much,

 -Youzhong


 On Wed, Jul 30, 2014 at 5:38 PM, Bryan Cantrill bryancantr...@gmail.com
 wrote:


 Seems like a bug, but not sure I entirely understand:  there may be some
 edge conditions where data specified in the data field of the epoll_event
 structure via epoll_ctl() is not properly returned, but it's not true that
 SmartOS simply ignores it.  Take, for example, the following program:

   https://us-east.manta.joyent.com/bcantrill/public/epoll_add.c

 (It's essentially the most basic possible use of epoll(5).)  Compiling
 that program and running it yields (for me, anyway) the same result on
 SmartOS as it does on Linux:

   $ ./epoll_add
   writing to pipe
   adding event...
   parent: got an event on fd 4

 I certainly don't doubt that there's a bug here -- I just don't think
 it's quite as broad as you're implying; could you point me to the core file
 for Samba and/or instructions as to how to reproduce the issue?

 - Bryan



 On Wed, Jul 30, 2014 at 2:08 PM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hi All,

 The man page of epoll_ctl says:

  The data field specifies the datum  to  be  associated  with  the
  event  and  will be returned via epoll_wait(3C). The events field
  denotes both the desired events (when specified via  epoll_ctl())
  and   the   events   that   have   occurred  (when  returned  via
  epoll_wait(3C)).

 This does not seem to be the case and is not compatible with Linux's
 implementation. Recently I tried to compile Samba 4.1.10 on smartos and
 'net ads join' just crashes. Samba uses epoll_event.data.ptr to keep track
 of its tevent_fd data but epoll_ctl() on smartos simply ignores it and does
 not keep a copy of the 'data' structure and returns it when epoll_wait is
 called.

 Can I assume this is an implementation specific bug?

 Thanks,

 -Youzhong
  *smartos-discuss* | Archives
 https://www.listbox.com/member/archive/184463/=now
 https://www.listbox.com/member/archive/rss/184463/21493597-819305a5
 | Modify
 https://www.listbox.com/member/?;
 Your Subscription http://www.listbox.com








---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] epoll_ctl and epoll_wait on SmartOS

[Youzhong Yang via smartos-discuss]

I got my new smartos image and alos samba 4.1.10 built, it works, no more
'net ads join' crashes.

Your help is absolutely appreciated.

Thanks,

 -Youzhong



On Fri, Aug 1, 2014 at 9:42 AM, Youzhong Yang youzh...@gmail.com wrote:

 Thanks very much for the quick fix.

 I'll build a new image and also samba, then report back after testing.

 -Youzhong


 On Fri, Aug 1, 2014 at 4:56 AM, Bryan Cantrill bryancantr...@gmail.com
 wrote:


 Perfect -- thank you!  As it turns out, this was -- as you surmised -- a
 bug in our epoll implementation.  I have pushed the fix:


 https://github.com/joyent/illumos-joyent/commit/471f1f3ef02e25a84d7c36c857ed260ae2b7ac99

 Thanks for taking the time for the clean test case -- and apologies for
 the bug!

  - Bryan


 On Thu, Jul 31, 2014 at 7:28 AM, Youzhong Yang youzh...@gmail.com
 wrote:

 Thanks Bryan.

 I take back what I said. epoll_* does not ignore epoll_event.data passed
 in.

 I managed to reproduce the issue with a program rewritten from yours:

 # gcc myepoll.c
 # ./a.out
 EPOLL_CTL_ADD: fd = 4, ptr = fd7fffdffba0
 writing to pipe fd 5
 written 6 bytes to fd 5
 Got an event on fd 4, events = 1, ptr = fd7fffdffba0
 got 6 bytes from fd 5
 EPOLL_CTL_DEL: fd = 4, ptr = fd7fffdffba0
 EPOLL_CTL_ADD: fd = 4, ptr = fd7fffdffb80
 writing to pipe fd 5
 written 6 bytes to fd 5
 Got an event on fd 4, events = 1, ptr = fd7fffdffba0
 got 6 bytes from fd 5

 As it indicates, the issue here is that after EPOLL_CTL_DEL, and then
 EPOLL_CTL_ADD of the same file descriptor with a new data.ptr, epoll_wait
 still returns an old pointer.

 C code is attached for your reference.

 Thanks very much,

 -Youzhong


 On Wed, Jul 30, 2014 at 5:38 PM, Bryan Cantrill bryancantr...@gmail.com
  wrote:


 Seems like a bug, but not sure I entirely understand:  there may be
 some edge conditions where data specified in the data field of the
 epoll_event structure via epoll_ctl() is not properly returned, but it's
 not true that SmartOS simply ignores it.  Take, for example, the following
 program:

   https://us-east.manta.joyent.com/bcantrill/public/epoll_add.c

 (It's essentially the most basic possible use of epoll(5).)  Compiling
 that program and running it yields (for me, anyway) the same result on
 SmartOS as it does on Linux:

   $ ./epoll_add
   writing to pipe
   adding event...
   parent: got an event on fd 4

 I certainly don't doubt that there's a bug here -- I just don't think
 it's quite as broad as you're implying; could you point me to the core file
 for Samba and/or instructions as to how to reproduce the issue?

 - Bryan



 On Wed, Jul 30, 2014 at 2:08 PM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hi All,

 The man page of epoll_ctl says:

  The data field specifies the datum  to  be  associated  with  the
  event  and  will be returned via epoll_wait(3C). The events field
  denotes both the desired events (when specified via  epoll_ctl())
  and   the   events   that   have   occurred  (when  returned  via
  epoll_wait(3C)).

 This does not seem to be the case and is not compatible with Linux's
 implementation. Recently I tried to compile Samba 4.1.10 on smartos and
 'net ads join' just crashes. Samba uses epoll_event.data.ptr to keep track
 of its tevent_fd data but epoll_ctl() on smartos simply ignores it and 
 does
 not keep a copy of the 'data' structure and returns it when epoll_wait is
 called.

 Can I assume this is an implementation specific bug?

 Thanks,

 -Youzhong
  *smartos-discuss* | Archives
 https://www.listbox.com/member/archive/184463/=now
 https://www.listbox.com/member/archive/rss/184463/21493597-819305a5
 | Modify
 https://www.listbox.com/member/?;
 Your Subscription http://www.listbox.com









---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] How to release reserved TCP port binding in BOUND state

[Youzhong Yang via smartos-discuss]

Hello,

We are having a very strange issue on one of servers. The issue is that
fcntl locking over NFS returns 'no locks available' immediately.

dtrace shows that bindresvport() returns error code 125 (EADDRINUSE):

# dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack();
printf(ret = %d, arg1);}'
  9  52692  bindresvport:return
  rpcmod`connmgr_get+0x560
  rpcmod`connmgr_wrapget+0x63
  rpcmod`clnt_cots_kcallit+0x198
  rpcmod`rpcbind_getaddr+0x245
  klmmod`update_host_rpcbinding+0x4f
  klmmod`nlm_host_get_rpc+0x6d
  klmmod`nlm_do_lock+0x10d
  klmmod`nlm4_lock_4_svc+0x2a
  klmmod`nlm_dispatch+0xe6
  klmmod`nlm_prog_4+0x34
  rpcmod`svc_getreq+0x1c1
  rpcmod`svc_run+0x146
  rpcmod`svc_do_run+0x8e
  nfs`nfssys+0xf1
  unix`_sys_sysenter_post_swapgs+0x149
ret = 125

netstat shows that 501 reserved ports are in BOUND state:

# netstat -an | grep BOUND
  *.935*.*0  0 1049740  0 BOUND
  *.801*.*0  0 1049740  0 BOUND
  *.798*.*0  0 1049740  0 BOUND
  *.561*.*0  0 1049740  0 BOUND
  *.613*.*0  0 1049740  0 BOUND
  
# netstat -an | grep BOUND | wc -l
 501

Has anyone seen this similar issue? is it possible to unbind those reserved
ports? Rebooting the server is our last resort.

Any advice would be very much appreciated.

Thanks,

-Youzhong



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] Re: [developer] How to release reserved TCP port binding in BOUND state

[Youzhong Yang via smartos-discuss]

Hi Marcel,

I really appreciate your input. We will try to find out what is wrong in
connmgr_get() which caused port leaking.

In the meantime, we want to change clnt_cots_do_bindresvport from 1 to 0
through mdb so that new connection will start using non-reserved port,

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/rpc/clnt_cots.c#508

Will dynamically changing this variable have any side effect?

Thanks,
-Youzhong

On Mon, Nov 3, 2014 at 4:50 PM, Marcel Telka mar...@telka.sk wrote:

 Hi Youzhong,

 On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via
 illumos-developer wrote:
  Hello,
 
  We are having a very strange issue on one of servers. The issue is that
  fcntl locking over NFS returns 'no locks available' immediately.
 
  dtrace shows that bindresvport() returns error code 125 (EADDRINUSE):
 
  # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack();
  printf(ret = %d, arg1);}'
9  52692  bindresvport:return
rpcmod`connmgr_get+0x560
rpcmod`connmgr_wrapget+0x63
rpcmod`clnt_cots_kcallit+0x198
rpcmod`rpcbind_getaddr+0x245
klmmod`update_host_rpcbinding+0x4f
klmmod`nlm_host_get_rpc+0x6d
klmmod`nlm_do_lock+0x10d
klmmod`nlm4_lock_4_svc+0x2a
klmmod`nlm_dispatch+0xe6
klmmod`nlm_prog_4+0x34
rpcmod`svc_getreq+0x1c1
rpcmod`svc_run+0x146
rpcmod`svc_do_run+0x8e
nfs`nfssys+0xf1
unix`_sys_sysenter_post_swapgs+0x149
  ret = 125
 
  netstat shows that 501 reserved ports are in BOUND state:
 
  # netstat -an | grep BOUND
*.935*.*0  0 1049740  0
 BOUND
*.801*.*0  0 1049740  0
 BOUND
*.798*.*0  0 1049740  0
 BOUND
*.561*.*0  0 1049740  0
 BOUND
*.613*.*0  0 1049740  0
 BOUND

  # netstat -an | grep BOUND | wc -l
   501
 
  Has anyone seen this similar issue? is it possible to unbind those
 reserved
  ports? Rebooting the server is our last resort.
 
  Any advice would be very much appreciated.

 I faced similar issue in connmgr_get().  It is filed as #1616 and the
 problem
 is that the dead connection is not properly closed (there seems to be
 missing
 connmgr_cancelconn() call somewhere), so the client could properly
 reconnect.
 Unfortunately, I had no time to finish the analysis of this bug.


 HTH

 --
 +---+
 | Marcel Telka   e-mail:   mar...@telka.sk  |
 |homepage: http://telka.sk/ |
 |jabber:   mar...@jabber.sk |
 +---+




---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] inotify driver not loaded automatically

[Youzhong Yang via smartos-discuss]

Yeah, that's what I need. Thanks Robert.

On Tue, Dec 9, 2014 at 11:04 AM, Robert Mustacchi r...@joyent.com wrote:

 On 12/9/14 8:02 , Youzhong Yang via smartos-discuss wrote:
  Hi All,
 
  I cherry-picked this cool inotify implementation (
 
 https://github.com/joyent/illumos-joyent/commit/a9a246c0c49e192616e7499eaa2362b21fde8f5e
 )
  and built my own image. After the machine reboot, inotify is not loaded
 so
  I have to manually load it through add_drv command.
 
  Is there a way to have it loaded automatically? or did I miss something
 or
  did something wrong?

 Yes, you missed a change in smartos-live. See

 https://github.com/joyent/smartos-live/commit/553afee1763bc36a7f7d6a14366abfe4f1998d71
 .

 Robert




---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] inotify driver not loaded automatically

[Youzhong Yang via smartos-discuss]

We have a stable base platform and don't want to introduce too many changes
which could potentially break something.

I've gone through the history of all files involved in this inotify
implementation and also picked up a few other bug fixes. We will run
thorough qualification to make sure the new commits/changes do not
introduce any issues.

Thanks Keith.

On Tue, Dec 9, 2014 at 11:06 AM, Keith Wesolowski 
keith.wesolow...@joyent.com wrote:

 On Tue, Dec 09, 2014 at 11:02:03AM -0500, Youzhong Yang via
 smartos-discuss wrote:

  I cherry-picked this cool inotify implementation (
 
 https://github.com/joyent/illumos-joyent/commit/a9a246c0c49e192616e7499eaa2362b21fde8f5e
 )
  and built my own image. After the machine reboot, inotify is not loaded
 so
  I have to manually load it through add_drv command.
 
  Is there a way to have it loaded automatically? or did I miss something
 or
  did something wrong?

 You can't just cherry-pick changes unless you understand them (and
 SmartOS) completely.  At minimum, you missed smartos-live changeset
 553afee1763bc36a7f7d6a14366abfe4f1998d71.  It's possible you will need
 others as well.  In general, there is no support for building platforms
 from mismatched repos, so always use the most recent release tag for all
 of them unless you are going to look in detail at every single change.




---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] inotify issues

[Youzhong Yang via smartos-discuss]

Hello again,

I downloaded smartos latest image (joyent_20141127T173954Z) and tested
inotify using it.

Two issues were observed:

1. ioctl(fd, FIONREAD, ...) does not work as expected. It does not return
the number of bytes that are available to be read on an inotify instance.

2. In an OS zone, touching a file always generates an IN_CREATE event, even
though the file already exists. This only happens in an OS zone, not in
global zone.

Attached is the test program I used for testing. Just compile and run it,
then in another terminal window, 'touch /tmp/file' twice. Please look at
the details at the end of this e-mail.

Thanks,

-Youzhong

--- In global zone ---
[root@batfs5999 /var/tmp]# ./inotify
ioctl = 64, errno = 0, bufsize = 0
got 48 bytes
   :: mask 256
   - New file file created
   :: mask 4
   - file ATTRIB
ioctl = 32, errno = 0, bufsize = 0
got 24 bytes
   :: mask 4
   - file ATTRIB

--- In OS zone ---
[root@batfs5999-cifs0 /var/tmp]# ./inotify
ioctl = 64, errno = 0, bufsize = 0
got 48 bytes
   :: mask 256
   - New file file created
   :: mask 4
   - file ATTRIB
ioctl = 64, errno = 0, bufsize = 0
got 48 bytes
   :: mask 256
   - New file file created
   :: mask 4
   - file ATTRIB

--- On Linux  ---
% ./inotify
ioctl = 0, errno = 0, bufsize = 32
got 32 bytes
   :: mask 256
   - New file file created
ioctl = 0, errno = 0, bufsize = 32
got 32 bytes
   :: mask 4
   - file ATTRIB
ioctl = 0, errno = 0, bufsize = 32
got 32 bytes
   :: mask 4
   - file ATTRIB



---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
/*
	This is the sample program to test inotify.
	
	- compile this program: gcc -o inotify inotify.c
	- run it by ./inotify or ./inotify direcotory to monitor
	
	If no argument is given, it will monitor /tmp.	  
*/
#include stdio.h
#include stdlib.h
#include errno.h
#include sys/types.h
#include sys/ioctl.h
#include sys/inotify.h
#ifdef __sun__
#include sys/filio.h
#endif

#include unistd.h
#include stropts.h
#include poll.h

#define EVENT_SIZE  ( sizeof (struct inotify_event) )
#define EVENT_BUF_LEN ( 1024 * ( EVENT_SIZE + 64 ) )

void print_inotify_event(struct inotify_event *event)
{
	if ( event-len ) {
		printf(   :: mask %d\n, event-mask);
		if ( event-mask  IN_CREATE ) {
			if ( event-mask  IN_ISDIR ) {
printf(- New directory %s created\n, event-name );
			}
			else {
printf(- New file %s created\n, event-name );
			}
		}
		else if ( event-mask  IN_DELETE ) {
			if ( event-mask  IN_ISDIR ) {
printf(- Directory %s deleted\n, event-name );
			}
			else {
printf(- File %s deleted\n, event-name );
			}
		}
		else if ( event-mask  IN_MOVED_FROM ) {
			printf(- %s MOVED_FROM\n, event-name );
		}
		else if ( event-mask  IN_MOVED_TO ) {
			printf(- %s MOVED_TO\n, event-name );
		}
		else if ( event-mask  IN_ATTRIB ) {
			printf(- %s ATTRIB\n, event-name );
		}	  
		else if ( event-mask  IN_MODIFY ) {
			printf(- %s MODIFY\n, event-name );
		}	  	 
		else {
			printf(- MASK %d - %s\n, event-mask, event-name);
		}
	}
}

int main(int argc, char *argv[] )
{
	int length, i = 0;
	int fd;
	int wd;
	char buffer[EVENT_BUF_LEN];
	int bufsize;
	int ret;
  
	struct pollfd fds[1];

	/*creating the INOTIFY instance*/
	fd = inotify_init();
	/*checking for error*/
	if ( fd  0 ) {
		perror( inotify_init );
		exit(1);
	}

	/* monitor /tmp or the given directory */
	if(argc == 1)
		wd = inotify_add_watch( fd, /tmp, IN_CREATE | IN_DELETE | IN_MOVED_FROM | IN_MOVED_TO | IN_ATTRIB | IN_MODIFY | IN_ONLYDIR);
	else
		wd = inotify_add_watch( fd, argv[1], IN_CREATE | IN_DELETE | IN_MOVED_FROM | IN_MOVED_TO | IN_ATTRIB | IN_MODIFY | IN_ONLYDIR);

	fds[0].fd = fd;
	fds[0].events = POLLIN;
  
	while(1)
	{
		/* check if there's data available for reading */
		ret = poll(fds, 1, 1000);
		if(ret = 0) continue;

		bufsize = 0;		
		/* check how many bytes available for reading */
		ret = ioctl(fd, FIONREAD, bufsize); /* FIONREAD = 0x4004667f */
		printf(ioctl = %d, errno = %d, bufsize = %d\n, ret, errno, bufsize);
	  
		/* Read the data. Normally the program can allocate a buffer based
		   on the size returned by above iotcl call. Here we just use a static
		   buffer 
		*/
		length = read( fd, buffer, EVENT_BUF_LEN ); 

		/* checking for error */
		if ( length  0 ) {
			perror( read );
			break;
		}  
		printf(got %d bytes\n, length);
	  
		/*
			Actually read the list of change events. 
			Here, read the change event one by one and process it accordingly.
		*/
		i = 0;
		while ( i  length ) { 
			struct inotify_event *event = ( struct inotify_event * ) buffer[ i ]; 
			print_inotify_event(event);
			i += EVENT_SIZE + event-len;
		}	  
	}

	inotify_rm_watch( fd, wd );

	

Re: [smartos-discuss] inotify issues

[Youzhong Yang via smartos-discuss]

Thanks Bryan. Indeed in the zone /opt/local/bin/touch is used.

But I still see the issue:

open(filename, O_WRONLY|O_NONBLOCK|O_CREAT|O_NOCTTY, 0666) is a file open
operation and should result in no IN_CREATE event if the file exists
(O_CREAT - Create the file if it does not exist).

I used the following C program to test this file open behavior on Linux and
SmartOS, as you can see, on SmartOS, it causes a IN_CREATE inotify event,
while on Linux, it triggers nothing. This happens both in global zone and
OS zone.

I am confused.

Thanks,

--Youzhong

---
#include sys/types.h
#include sys/stat.h
#include fcntl.h
#include stdio.h
#include stdlib.h

int main(int argc, char *argv[])
{
int fd;
char *filename;
 if(argc  1) {
filename = argv[1];
} else {
printf(No filename specified!\n);
exit(1);
}
 fd = open(filename, O_WRONLY|O_NONBLOCK|O_CREAT|O_NOCTTY, 0666);
if(fd == -1) {
perror(open);
exit(1);
}
close(fd);
return 0;
}


On Wed, Dec 10, 2014 at 1:05 PM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 The touch differences can be explained by the touch(1) that you're using:
  in the global zone, you're using /usr/bin/touch, but in the non-global, I
 imagine that you're using /opt/local/bin/touch.  These two implementations
 differ, so not a huge surprise that their implementation details differ.

 As for FIONREAD: indeed, that seems screwed up -- it seems to be returning
 the number of bytes instead of storing in in the specified argument.  I'll
 get that fixed today...

  - Bryan



 On Wed, Dec 10, 2014 at 8:00 AM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hello again,

 I downloaded smartos latest image (joyent_20141127T173954Z) and tested
 inotify using it.

 Two issues were observed:

 1. ioctl(fd, FIONREAD, ...) does not work as expected. It does not return
 the number of bytes that are available to be read on an inotify instance.

 2. In an OS zone, touching a file always generates an IN_CREATE event,
 even though the file already exists. This only happens in an OS zone, not
 in global zone.

 Attached is the test program I used for testing. Just compile and run it,
 then in another terminal window, 'touch /tmp/file' twice. Please look at
 the details at the end of this e-mail.

 Thanks,

 -Youzhong

 --- In global zone ---
 [root@batfs5999 /var/tmp]# ./inotify
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB
 ioctl = 32, errno = 0, bufsize = 0
 got 24 bytes
:: mask 4
- file ATTRIB

 --- In OS zone ---
 [root@batfs5999-cifs0 /var/tmp]# ./inotify
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB

 --- On Linux  ---
 % ./inotify
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 256
- New file file created
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 4
- file ATTRIB
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 4
- file ATTRIB

 *smartos-discuss* | Archives
 https://www.listbox.com/member/archive/184463/=now
 https://www.listbox.com/member/archive/rss/184463/21493597-819305a5 |
 Modify
 https://www.listbox.com/member/?;
 Your Subscription http://www.listbox.com






---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] inotify issues

[Youzhong Yang via smartos-discuss]

Hi Bryan,

I saw the bug in the function inotify_ioctl() and tried to fix it. Now with
my new image ioctl(FIONREAD) returns state-ins_size as the bytes available
to be read.

[root@batfs5901 ~]# /var/tmp/inotify
ioctl = 0, errno = 0, bufsize = 64
got 48 bytes
   - New file yyang created
   - yyang ATTRIB

this state-ins_size seems larger than the number of bytes read by read()
call, i.e. state-ins_size does not tell the exact number of bytes
available for read.

This can be a problem when the program expects to read that many bytes. I
just confirmed it causes issue in Samba.

Thanks,

-- Youzhong


On Wed, Dec 10, 2014 at 1:05 PM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 The touch differences can be explained by the touch(1) that you're using:
  in the global zone, you're using /usr/bin/touch, but in the non-global, I
 imagine that you're using /opt/local/bin/touch.  These two implementations
 differ, so not a huge surprise that their implementation details differ.

 As for FIONREAD: indeed, that seems screwed up -- it seems to be returning
 the number of bytes instead of storing in in the specified argument.  I'll
 get that fixed today...

  - Bryan



 On Wed, Dec 10, 2014 at 8:00 AM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hello again,

 I downloaded smartos latest image (joyent_20141127T173954Z) and tested
 inotify using it.

 Two issues were observed:

 1. ioctl(fd, FIONREAD, ...) does not work as expected. It does not return
 the number of bytes that are available to be read on an inotify instance.

 2. In an OS zone, touching a file always generates an IN_CREATE event,
 even though the file already exists. This only happens in an OS zone, not
 in global zone.

 Attached is the test program I used for testing. Just compile and run it,
 then in another terminal window, 'touch /tmp/file' twice. Please look at
 the details at the end of this e-mail.

 Thanks,

 -Youzhong

 --- In global zone ---
 [root@batfs5999 /var/tmp]# ./inotify
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB
 ioctl = 32, errno = 0, bufsize = 0
 got 24 bytes
:: mask 4
- file ATTRIB

 --- In OS zone ---
 [root@batfs5999-cifs0 /var/tmp]# ./inotify
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB

 --- On Linux  ---
 % ./inotify
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 256
- New file file created
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 4
- file ATTRIB
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 4
- file ATTRIB

 *smartos-discuss* | Archives
 https://www.listbox.com/member/archive/184463/=now
 https://www.listbox.com/member/archive/rss/184463/21493597-819305a5 |
 Modify
 https://www.listbox.com/member/?;
 Your Subscription http://www.listbox.com






---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Re: [smartos-discuss] inotify issues

[Youzhong Yang via smartos-discuss]

Thanks Bryan for your attentions to these issues and providing fixes so
quickly. I really appreciate it.

This morning I built new image with your fix and confirmed that now samba
works smoothly providing change notification using inotify in a smartos OS
zone. It's the first time we no longer depends on the buggy gam_server.

Regarding the IN_CREATE issue, probably we don't care much about it, but
for the sake of correctness, it would be nice if it can be fixed in the
future.

Sincerely,

-- Youzhong


On Thu, Dec 11, 2014 at 3:26 AM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 Just wanted to let you know that I fixed this; commit is here:


 https://github.com/joyent/illumos-joyent/commit/af6c46b5dae7a37d78d71546df5045236cd48994

 This doesn't address the IN_CREATE issue, which is a much more involved
 issue -- but should solve the FIONREAD problems...

  - Bryan


 On Wed, Dec 10, 2014 at 4:07 PM, Bryan Cantrill bryancantr...@gmail.com
 wrote:


 Yeah, that's busted, and I see the bug (we're calculating the size
 incorrectly); I'll fix that when I fix FIONREAD -- with my apologies...

  - Bryan


 On Wed, Dec 10, 2014 at 12:37 PM, Youzhong Yang youzh...@gmail.com
 wrote:

 Hi Bryan,

 I saw the bug in the function inotify_ioctl() and tried to fix it. Now
 with my new image ioctl(FIONREAD) returns state-ins_size as the bytes
 available to be read.

 [root@batfs5901 ~]# /var/tmp/inotify
 ioctl = 0, errno = 0, bufsize = 64
 got 48 bytes
- New file yyang created
- yyang ATTRIB

 this state-ins_size seems larger than the number of bytes read by
 read() call, i.e. state-ins_size does not tell the exact number of bytes
 available for read.

 This can be a problem when the program expects to read that many bytes.
 I just confirmed it causes issue in Samba.

 Thanks,

 -- Youzhong


 On Wed, Dec 10, 2014 at 1:05 PM, Bryan Cantrill bryancantr...@gmail.com
  wrote:


 The touch differences can be explained by the touch(1) that you're
 using:  in the global zone, you're using /usr/bin/touch, but in the
 non-global, I imagine that you're using /opt/local/bin/touch.  These two
 implementations differ, so not a huge surprise that their implementation
 details differ.

 As for FIONREAD: indeed, that seems screwed up -- it seems to be
 returning the number of bytes instead of storing in in the specified
 argument.  I'll get that fixed today...

  - Bryan



 On Wed, Dec 10, 2014 at 8:00 AM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hello again,

 I downloaded smartos latest image (joyent_20141127T173954Z) and tested
 inotify using it.

 Two issues were observed:

 1. ioctl(fd, FIONREAD, ...) does not work as expected. It does not
 return the number of bytes that are available to be read on an inotify
 instance.

 2. In an OS zone, touching a file always generates an IN_CREATE event,
 even though the file already exists. This only happens in an OS zone, not
 in global zone.

 Attached is the test program I used for testing. Just compile and run
 it, then in another terminal window, 'touch /tmp/file' twice. Please look
 at the details at the end of this e-mail.

 Thanks,

 -Youzhong

 --- In global zone ---
 [root@batfs5999 /var/tmp]# ./inotify
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB
 ioctl = 32, errno = 0, bufsize = 0
 got 24 bytes
:: mask 4
- file ATTRIB

 --- In OS zone ---
 [root@batfs5999-cifs0 /var/tmp]# ./inotify
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB
 ioctl = 64, errno = 0, bufsize = 0
 got 48 bytes
:: mask 256
- New file file created
:: mask 4
- file ATTRIB

 --- On Linux  ---
 % ./inotify
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 256
- New file file created
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 4
- file ATTRIB
 ioctl = 0, errno = 0, bufsize = 32
 got 32 bytes
:: mask 4
- file ATTRIB

 *smartos-discuss* | Archives
 https://www.listbox.com/member/archive/184463/=now
 https://www.listbox.com/member/archive/rss/184463/21493597-819305a5
 | Modify
 https://www.listbox.com/member/?;
 Your Subscription http://www.listbox.com









---
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

[smartos-discuss] inotify panic

[Youzhong Yang via smartos-discuss]

Hello again,

I tried to test samba with inotify under relatively heavy load, it panicked
the machine.

Does the stack give a hint about what's going wrong? I have the crash dump
but I cannot upload it since it may contain some production data. If
further analysis is needed, please advise.

Thanks,

-Youzhong

 $C
ff0091179030 cpu_idle_exit+0xc()
ff0091179090 do_interrupt+0x41(ff00911790a0, 0)
ff00911790a0 _interrupt+0xba()
ff00911791d0 fakesoftint_return()
ff0091179200 disp_lock_exit_nopreempt+0x42(ff136441b2c8)
ff0091179220 preempt+0xe7()
ff0091179250 kpreempt+0x98(1)
ff0091179280 sys_rtt_common+0x1ba(ff0091179290)
ff0091179290 _sys_rtt_ints_disabled+8()
ff00911793f0 mutex_enter+9()
ff00911794a0 dbuf_prefetch+0x65(ff21994217d8, 253e, 2)
ff00911794f0 dmu_zfetch_fetch+0x55(ff21994217d8, 24f1, d4)
ff0091179530 dmu_zfetch_dofetch+0xbb(ff2199421a10, ff1aac35c440)
ff00911795a0 dmu_zfetch_find+0x3d6(ff2199421a10, ff00911795c0,
1)
ff0091179670 dmu_zfetch+0xe0(ff2199421a10, 9b4c000, 4000, 1)
ff00911796e0 dbuf_read+0x301(ff3d9e911238, 0, 2)
ff0091179790 dnode_hold_impl+0xe4(ff2d0cfca700, 4da74, 1,
f7e8bb8d, ff00911797e8)
ff00911797c0 dnode_hold+0x28(ff2d0cfca700, 4da74, f7e8bb8d,
ff00911797e8)
ff0091179840 dmu_buf_hold+0x5c(ff2d0cfca700, 4da74, 0, 0,
ff0091179888, 1)
ff00911798f0 zap_lockdir+0x6d(ff2d0cfca700, 4da74, 0, 1, 1, 0,
ff0091179b78)
ff00911799a0 zap_cursor_retrieve+0x16c(ff0091179b70,
ff0091179a50)
ff0091179c20 zfs_readdir+0x33d(ff2d2fe46340, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179c60 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179cc0 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179d00 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179d60 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179da0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179e00 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179e40 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179ea0 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179ee0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179f40 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179f80 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff0091179fe0 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a020 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a080 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a0c0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a120 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a160 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a1c0 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a200 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a260 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a2a0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a300 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a340 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a3a0 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a3e0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a440 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a480 vnext_readdir+0x84(ff009117dd40, ff009117de60,
ff2b6f35ab30, ff009117deac, 0, 0)
ff009117a4e0 inotify_fop_readdir+0x30(ff009117dd40,
ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)

Re: [smartos-discuss] inotify panic

[Youzhong Yang via smartos-discuss]

Thanks Bryan. The issue was reproduced using our automated build and test
system. As usual, I will try to simplify the reproduction (hopefully) and
get back to you.

   - Youzhong

On Tue, Dec 16, 2014 at 6:20 PM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 Youzhong,

 Interesting. No, we haven't seen it, and it's not immediately obvious
 what's going on; the dump is going to be essential here.  Can you perhaps
 upload the dump to a private Manta account and then sign a URL and send it
 to me privately to use?  (Or otherwise make it available to me privately?)
 Alternatively, is this reproducible?  Could you reproduce it on a machine
 for which you could share the dump?  Or perhaps tell me how you reproduced
 it?  To answer your question: yes, further analysis is needed. ;)

 - Bryan



 On Tue, Dec 16, 2014 at 3:10 PM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hello again,

 I tried to test samba with inotify under relatively heavy load, it
 panicked the machine.

 Does the stack give a hint about what's going wrong? I have the crash
 dump but I cannot upload it since it may contain some production data. If
 further analysis is needed, please advise.

 Thanks,

 -Youzhong

  $C
 ff0091179030 cpu_idle_exit+0xc()
 ff0091179090 do_interrupt+0x41(ff00911790a0, 0)
 ff00911790a0 _interrupt+0xba()
 ff00911791d0 fakesoftint_return()
 ff0091179200 disp_lock_exit_nopreempt+0x42(ff136441b2c8)
 ff0091179220 preempt+0xe7()
 ff0091179250 kpreempt+0x98(1)
 ff0091179280 sys_rtt_common+0x1ba(ff0091179290)
 ff0091179290 _sys_rtt_ints_disabled+8()
 ff00911793f0 mutex_enter+9()
 ff00911794a0 dbuf_prefetch+0x65(ff21994217d8, 253e, 2)
 ff00911794f0 dmu_zfetch_fetch+0x55(ff21994217d8, 24f1, d4)
 ff0091179530 dmu_zfetch_dofetch+0xbb(ff2199421a10,
 ff1aac35c440)
 ff00911795a0 dmu_zfetch_find+0x3d6(ff2199421a10,
 ff00911795c0, 1)
 ff0091179670 dmu_zfetch+0xe0(ff2199421a10, 9b4c000, 4000, 1)
 ff00911796e0 dbuf_read+0x301(ff3d9e911238, 0, 2)
 ff0091179790 dnode_hold_impl+0xe4(ff2d0cfca700, 4da74, 1,
 f7e8bb8d, ff00911797e8)
 ff00911797c0 dnode_hold+0x28(ff2d0cfca700, 4da74,
 f7e8bb8d, ff00911797e8)
 ff0091179840 dmu_buf_hold+0x5c(ff2d0cfca700, 4da74, 0, 0,
 ff0091179888, 1)
 ff00911798f0 zap_lockdir+0x6d(ff2d0cfca700, 4da74, 0, 1, 1, 0,
 ff0091179b78)
 ff00911799a0 zap_cursor_retrieve+0x16c(ff0091179b70,
 ff0091179a50)
 ff0091179c20 zfs_readdir+0x33d(ff2d2fe46340, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179c60 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179cc0 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179d00 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179d60 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179da0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179e00 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179e40 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179ea0 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179ee0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179f40 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179f80 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179fe0 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a020 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a080 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a0c0 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a120 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a160 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a1c0 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a200 vnext_readdir+0x84(ff009117dd40, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff009117a260 inotify_fop_readdir+0x30

Re: [smartos-discuss] inotify panic

[Youzhong Yang via smartos-discuss]

Thank you Bryan for the fix. I will try it out.

On Mon, Feb 2, 2015 at 12:49 PM, Bryan Cantrill bryancantr...@gmail.com
wrote:


 Just wanted to let you know that I fixed this a little while ago (with my
 apologies for not following up here earlier).  The problem was a tad nasty,
 and the fix took a little while to get right; the diffs go into the details:


 https://github.com/joyent/illumos-joyent/commit/0173c3089b5d2cc973f4453ad6410855e13ad28a

 Youzhong: any recent build of SmartOS should have that fix; please let me
 know if you see any other issues, with my thanks again for reporting this!

 - Bryan


 On Wed, Dec 17, 2014 at 2:51 PM, Bryan Cantrill bryancantr...@gmail.com
 wrote:


 Thanks for this.  I see what's going on, but it's going to take me a
 little while to noodle on a fix; I'll let you know when I have something.

 - Bryan


 On Wed, Dec 17, 2014 at 9:56 AM, Youzhong Yang youzh...@gmail.com
 wrote:

 Hi Bryan,

 I managed to reproduce the issue using the attached C program.

 - compile: gcc -o inotify_stress inotify_stress.c
 - run: ./inotify_stress -f /var/log -n 120

 If 120 does not cause crash, please use a larger number.

 It seems a stack overflow issue. In all the crash dumps we saw those
 recursive inotify_fop_readdir/vnext_readdir calls used almost 20K stack
 space and reached its limit.

 Thanks,

  -Youzhong


 On Tue, Dec 16, 2014 at 6:20 PM, Bryan Cantrill bryancantr...@gmail.com
  wrote:


 Youzhong,

 Interesting. No, we haven't seen it, and it's not immediately obvious
 what's going on; the dump is going to be essential here.  Can you perhaps
 upload the dump to a private Manta account and then sign a URL and send it
 to me privately to use?  (Or otherwise make it available to me privately?)
 Alternatively, is this reproducible?  Could you reproduce it on a machine
 for which you could share the dump?  Or perhaps tell me how you reproduced
 it?  To answer your question: yes, further analysis is needed. ;)

 - Bryan



 On Tue, Dec 16, 2014 at 3:10 PM, Youzhong Yang via smartos-discuss 
 smartos-discuss@lists.smartos.org wrote:

 Hello again,

 I tried to test samba with inotify under relatively heavy load, it
 panicked the machine.

 Does the stack give a hint about what's going wrong? I have the crash
 dump but I cannot upload it since it may contain some production data. If
 further analysis is needed, please advise.

 Thanks,

 -Youzhong

  $C
 ff0091179030 cpu_idle_exit+0xc()
 ff0091179090 do_interrupt+0x41(ff00911790a0, 0)
 ff00911790a0 _interrupt+0xba()
 ff00911791d0 fakesoftint_return()
 ff0091179200 disp_lock_exit_nopreempt+0x42(ff136441b2c8)
 ff0091179220 preempt+0xe7()
 ff0091179250 kpreempt+0x98(1)
 ff0091179280 sys_rtt_common+0x1ba(ff0091179290)
 ff0091179290 _sys_rtt_ints_disabled+8()
 ff00911793f0 mutex_enter+9()
 ff00911794a0 dbuf_prefetch+0x65(ff21994217d8, 253e, 2)
 ff00911794f0 dmu_zfetch_fetch+0x55(ff21994217d8, 24f1, d4)
 ff0091179530 dmu_zfetch_dofetch+0xbb(ff2199421a10,
 ff1aac35c440)
 ff00911795a0 dmu_zfetch_find+0x3d6(ff2199421a10,
 ff00911795c0, 1)
 ff0091179670 dmu_zfetch+0xe0(ff2199421a10, 9b4c000, 4000, 1)
 ff00911796e0 dbuf_read+0x301(ff3d9e911238, 0, 2)
 ff0091179790 dnode_hold_impl+0xe4(ff2d0cfca700, 4da74, 1,
 f7e8bb8d, ff00911797e8)
 ff00911797c0 dnode_hold+0x28(ff2d0cfca700, 4da74,
 f7e8bb8d, ff00911797e8)
 ff0091179840 dmu_buf_hold+0x5c(ff2d0cfca700, 4da74, 0, 0,
 ff0091179888, 1)
 ff00911798f0 zap_lockdir+0x6d(ff2d0cfca700, 4da74, 0, 1, 1, 0,
 ff0091179b78)
 ff00911799a0 zap_cursor_retrieve+0x16c(ff0091179b70,
 ff0091179a50)
 ff0091179c20 zfs_readdir+0x33d(ff2d2fe46340, ff009117de60,
 ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179c60 vnext_readdir+0x84(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179cc0 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179d00 vnext_readdir+0x84(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179d60 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179da0 vnext_readdir+0x84(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179e00 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179e40 vnext_readdir+0x84(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179ea0 inotify_fop_readdir+0x30(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179ee0 vnext_readdir+0x84(ff009117dd40,
 ff009117de60, ff2b6f35ab30, ff009117deac, 0, 0)
 ff0091179f40 inotify_fop_readdir