Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-28 Thread Fred Liu 


> -Original Message-
> From: Fred Liu
> Sent: 星期四, 三月 24, 2016 18:26
> To: 'Chris Siebenmann'; Richard Jahnel
> Cc: omnios-discuss@lists.omniti.com
> Subject: RE: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
> 
> > -Original Message-
> > From: Chris Siebenmann [mailto:c...@cs.toronto.edu]
> > Sent: 星期三, 三月 23, 2016 23:33
> > To: Richard Jahnel
> > Cc: Chris Siebenmann; Fred Liu; omnios-discuss@lists.omniti.com
> > Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> >
> > > It should be noted that using a 512e disk as a 512n disk subjects
> > > you to a significant risk of silent corruption in the event of power loss.
> > > Because 512e disks does a read>modify>write operation to modify
> > > 512byte chunk of a 4k sector, zfs won't know about the other
> > > 7 corrupted 512e sectors in the event of a power loss during a write
> > > operation. So when discards the incomplete txg on reboot, it won't
> > > do anything about the other 7 512e sectors it doesn't know were affected.
> >
> >  This is true; under normal circumstances you do not want to use a
> > 512e drive in an ashift=9 vdev. However, if you have a dead 512n drive
> > and you have no remaining 512n spares, your choices are to run without
> > redundancy, to wedge in a 512e drive and accept the potential problems
> > on power failure (problems that can likely be fixed by scrubbing the
> > pool afterwards), or obtain enough additional drives (and perhaps
> > server(s)) to entirely rebuild the pool on 512e drives with ashift=12.
> >
> >  In this situation, running with a 512e drive and accepting the
> > performance issues and potential exposure to power failures is
> > basically the lesser evil. (I wish ZFS was willing to accept this, but
> > it isn't.)
> >
> [Fred Liu]: I have a similar test here:
> 
> [root@00-25-90-74-f5-04 ~]# zpool status
>   pool: tank
>  state: ONLINE
>   scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16 2015
> config:
> 
> NAME STATE READ WRITE CKSUM
> tank ONLINE   0 0 0
>   raidz2-0   ONLINE   0 0 0
> c2t45d0  ONLINE   0 0 0
> c2t46d0  ONLINE   0 0 0
> c2t47d0  ONLINE   0 0 0
> c2t48d0  ONLINE   0 0 0
> c2t49d0  ONLINE   0 0 0
> c2t52d0  ONLINE   0 0 0
> c2t53d0  ONLINE   0 0 0
> c2t44d0  ONLINE   0 0 0
> spares
>   c0t5000CCA6A0C791CBd0  AVAIL
> 
> errors: No known data errors
> 
>   pool: zones
>  state: ONLINE
>   scan: scrub repaired 0 in 2h45m with 0 errors on Tue Aug 12 20:24:30 2014
> config:
> 
> NAME   STATE READ WRITE CKSUM
> zones  ONLINE   0 0 0
>   raidz2-0 ONLINE   0 0 0
> c0t5000C500584AC07Bd0  ONLINE   0 0 0
> c0t5000C500584AC557d0  ONLINE   0 0 0
> c0t5000C500584ACB1Fd0  ONLINE   0 0 0
> c0t5000C500584AD7B3d0  ONLINE   0 0 0
> c0t5000C500584C30DBd0  ONLINE   0 0 0
> c0t5000C500586E54A3d0  ONLINE   0 0 0
> c0t5000C500586EF0CBd0  ONLINE   0 0 0
> c0t5000C50058426A0Fd0  ONLINE   0 0 0
> logs
>   c4t0d0   ONLINE   0 0 0
>   c4t1d0   ONLINE   0 0 0
> cache
>   c0t55CD2E404BE9CB7Ed0ONLINE   0 0 0
> 
> errors: No known data errors
> 
> [root@00-25-90-74-f5-04 ~]# format
> Searching for disks...done
> 
> 
> AVAILABLE DISK SELECTIONS:
>0. c0t55CD2E404BE9CB7Ed0  SSDSC2BW18-DC32-167.68GB>
>   /scsi_vhci/disk@g55cd2e404be9cb7e
>1. c0t5000C500584AC07Bd0
> 
>   /scsi_vhci/disk@g5000c500584ac07b
>2. c0t5000C500584AC557d0
> 
>   /scsi_vhci/disk@g5000c500584ac557
>3. c0t5000C500584ACB1Fd0
> 
>   /scsi_vhci/disk@g5000c500584acb1f
>4. c0t5000C500584AD7B3d0
> 
>   /scsi_vhci/disk@g5000c500584ad7b3
>5. c0t5000C500584C30DBd0
> 
>   /scsi_vhci/disk@g5000c500584c30db
>6. c0t5000C500586E54A3d0
> 
>   /scsi_vhci/disk@g5000c500586e54a3

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-28 Thread Fred Liu 


> -Original Message-
> From: Fred Liu
> Sent: 星期一, 三月 28, 2016 16:57
> To: 'Chris Siebenmann'; 'Richard Jahnel'
> Cc: 'omnios-discuss@lists.omniti.com'
> Subject: RE: [OmniOS-discuss] 4kn or 512e with ashift=12
>
>
>
> > -Original Message-
> > From: Fred Liu
> > Sent: 星期四, 三月 24, 2016 18:26
> > To: 'Chris Siebenmann'; Richard Jahnel
> > Cc: omnios-discuss@lists.omniti.com
> > Subject: RE: [OmniOS-discuss] 4kn or 512e with ashift=12
> >
> >
> >
> > > -Original Message-
> > > From: Chris Siebenmann [mailto:c...@cs.toronto.edu]
> > > Sent: 星期三, 三月 23, 2016 23:33
> > > To: Richard Jahnel
> > > Cc: Chris Siebenmann; Fred Liu; omnios-discuss@lists.omniti.com
> > > Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> > >
> > > > It should be noted that using a 512e disk as a 512n disk subjects
> > > > you to a significant risk of silent corruption in the event of power 
> > > > loss.
> > > > Because 512e disks does a read>modify>write operation to modify
> > > > 512byte chunk of a 4k sector, zfs won't know about the other
> > > > 7 corrupted 512e sectors in the event of a power loss during a
> > > > write operation. So when discards the incomplete txg on reboot, it
> > > > won't do anything about the other 7 512e sectors it doesn't know were
> affected.
> > >
> > >  This is true; under normal circumstances you do not want to use a
> > > 512e drive in an ashift=9 vdev. However, if you have a dead 512n
> > > drive and you have no remaining 512n spares, your choices are to run
> > > without redundancy, to wedge in a 512e drive and accept the
> > > potential problems on power failure (problems that can likely be
> > > fixed by scrubbing the pool afterwards), or obtain enough additional
> > > drives (and perhaps
> > > server(s)) to entirely rebuild the pool on 512e drives with ashift=12.
> > >
> > >  In this situation, running with a 512e drive and accepting the
> > > performance issues and potential exposure to power failures is
> > > basically the lesser evil. (I wish ZFS was willing to accept this,
> > > but it isn't.)
> > >
> > [Fred Liu]: I have a similar test here:
> >
> > [root@00-25-90-74-f5-04 ~]# zpool status
> >   pool: tank
> >  state: ONLINE
> >   scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16
> > 2015
> > config:
> >
> > NAME STATE READ WRITE CKSUM
> > tank ONLINE   0 0 0
> >   raidz2-0   ONLINE   0 0 0
> > c2t45d0  ONLINE   0 0 0
> > c2t46d0  ONLINE   0 0 0
> > c2t47d0  ONLINE   0 0 0
> > c2t48d0  ONLINE   0 0 0
> > c2t49d0  ONLINE   0 0 0
> > c2t52d0  ONLINE   0 0 0
> > c2t53d0  ONLINE   0 0 0
> > c2t44d0  ONLINE   0 0 0
> > spares
> >   c0t5000CCA6A0C791CBd0  AVAIL
> >
> > errors: No known data errors
> >
> >   pool: zones
> >  state: ONLINE
> >   scan: scrub repaired 0 in 2h45m with 0 errors on Tue Aug 12 20:24:30
> > 2014
> > config:
> >
> > NAME   STATE READ WRITE
> CKSUM
> > zones  ONLINE   0 0 0
> >   raidz2-0 ONLINE   0 0 0
> > c0t5000C500584AC07Bd0  ONLINE   0 0 0
> > c0t5000C500584AC557d0  ONLINE   0 0 0
> > c0t5000C500584ACB1Fd0  ONLINE   0 0 0
> > c0t5000C500584AD7B3d0  ONLINE   0 0 0
> > c0t5000C500584C30DBd0  ONLINE   0 0 0
> > c0t5000C500586E54A3d0  ONLINE   0 0 0
> > c0t5000C500586EF0CBd0  ONLINE   0 0 0
> > c0t5000C50058426A0Fd0  ONLINE   0 0 0
> > logs
> >   c4t0d0   ONLINE   0 0 0
> >   c4t1d0   ONLINE   0 0 0
> > cache
> >   c0t55CD2E404BE9CB7Ed0ONLINE   0 0 0
> >
> > errors: No known data errors
> >
> > [root@00-25-90-74-f5-04 ~]# format
>

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-24 Thread Richard Elling 

> On Mar 23, 2016, at 6:37 PM, Bob Friesenhahn  
> wrote:
> 
> On Wed, 23 Mar 2016, Richard Elling wrote:
> 
>> 
>>> On Mar 23, 2016, at 7:49 AM, Richard Jahnel  wrote:
>>> 
>>> It should be noted that using a 512e disk as a 512n disk subjects you to a 
>>> significant risk of silent corruption in the event of power loss. Because 
>>> 512e disks does a read>modify>write operation to modify 512byte chunk of a 
>>> 4k sector, zfs won't know about the other 7 corrupted 512e sectors in the 
>>> event of a power loss during a write operation. So when discards the 
>>> incomplete txg on reboot, it won't do anything about the other 7 512e 
>>> sectors it doesn't know were affected.
>> 
>> Disagree. The risk is no greater than HDDs today with their volatile write 
>> caches.
> 
> If the data unrelated to the current transaction group is read and then 
> partially modifed (possibly with data corruption due to loss of power during 
> write), this would seem to be worse than loss due to a volatile write cache 
> (assuming drives which observe cache sync requests) since data unrelated to 
> the current transaction group may have been modified.  The end result would 
> be checksum errors during a scrub.

The old data is not modified. This is not read-destroy-modify-write.
 -- richard

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-24 Thread Fred Liu 


> -Original Message-
> From: Chris Siebenmann [mailto:c...@cs.toronto.edu]
> Sent: 星期三, 三月 23, 2016 23:33
> To: Richard Jahnel
> Cc: Chris Siebenmann; Fred Liu; omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> > It should be noted that using a 512e disk as a 512n disk subjects you
> > to a significant risk of silent corruption in the event of power loss.
> > Because 512e disks does a read>modify>write operation to modify
> > 512byte chunk of a 4k sector, zfs won't know about the other
> > 7 corrupted 512e sectors in the event of a power loss during a write
> > operation. So when discards the incomplete txg on reboot, it won't do
> > anything about the other 7 512e sectors it doesn't know were affected.
> 
>  This is true; under normal circumstances you do not want to use a 512e drive
> in an ashift=9 vdev. However, if you have a dead 512n drive and you have no
> remaining 512n spares, your choices are to run without redundancy, to wedge
> in a 512e drive and accept the potential problems on power failure (problems
> that can likely be fixed by scrubbing the pool afterwards), or obtain enough
> additional drives (and perhaps
> server(s)) to entirely rebuild the pool on 512e drives with ashift=12.
> 
>  In this situation, running with a 512e drive and accepting the performance
> issues and potential exposure to power failures is basically the lesser evil. 
> (I
> wish ZFS was willing to accept this, but it isn't.)
> 
[Fred Liu]: I have a similar test here:

[root@00-25-90-74-f5-04 ~]# zpool status
  pool: tank
 state: ONLINE
  scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16 2015
config:

NAME STATE READ WRITE CKSUM
tank ONLINE   0 0 0
  raidz2-0   ONLINE   0 0 0
c2t45d0  ONLINE   0 0 0
c2t46d0  ONLINE   0 0 0
c2t47d0  ONLINE   0 0 0
c2t48d0  ONLINE   0 0 0
c2t49d0  ONLINE   0 0 0
c2t52d0  ONLINE   0 0 0
c2t53d0  ONLINE   0 0 0
c2t44d0  ONLINE   0 0 0
spares
  c0t5000CCA6A0C791CBd0  AVAIL

errors: No known data errors

  pool: zones
 state: ONLINE
  scan: scrub repaired 0 in 2h45m with 0 errors on Tue Aug 12 20:24:30 2014
config:

NAME   STATE READ WRITE CKSUM
zones  ONLINE   0 0 0
  raidz2-0 ONLINE   0 0 0
c0t5000C500584AC07Bd0  ONLINE   0 0 0
c0t5000C500584AC557d0  ONLINE   0 0 0
c0t5000C500584ACB1Fd0  ONLINE   0 0 0
c0t5000C500584AD7B3d0  ONLINE   0 0 0
c0t5000C500584C30DBd0  ONLINE   0 0 0
c0t5000C500586E54A3d0  ONLINE   0 0 0
c0t5000C500586EF0CBd0  ONLINE   0 0 0
c0t5000C50058426A0Fd0  ONLINE   0 0 0
logs
  c4t0d0   ONLINE   0 0 0
  c4t1d0   ONLINE   0 0 0
cache
  c0t55CD2E404BE9CB7Ed0ONLINE   0 0 0

errors: No known data errors

[root@00-25-90-74-f5-04 ~]# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
   0. c0t55CD2E404BE9CB7Ed0 
  /scsi_vhci/disk@g55cd2e404be9cb7e
   1. c0t5000C500584AC07Bd0 
  /scsi_vhci/disk@g5000c500584ac07b
   2. c0t5000C500584AC557d0 
  /scsi_vhci/disk@g5000c500584ac557
   3. c0t5000C500584ACB1Fd0 
  /scsi_vhci/disk@g5000c500584acb1f
   4. c0t5000C500584AD7B3d0 
  /scsi_vhci/disk@g5000c500584ad7b3
   5. c0t5000C500584C30DBd0 
  /scsi_vhci/disk@g5000c500584c30db
   6. c0t5000C500586E54A3d0 
  /scsi_vhci/disk@g5000c500586e54a3
   7. c0t5000C500586EF0CBd0 
  /scsi_vhci/disk@g5000c500586ef0cb
   8. c0t5000C50058426A0Fd0 
  /scsi_vhci/disk@g5000c50058426a0f
   9. c0t5000CCA6A0C791CBd0 
  /scsi_vhci/disk@g5000cca6a0c791cb
  10. c0t5F0056425331d0 
  /scsi_vhci/disk@g5f0056425331
  11. c2t44d0 
  /pci@0,0/pci8086,1c10@1c/pci1000,3140@0/sd@2c,0
  12. c2t45d0 
  /pci@0,0/pci8086,1c10@1c/pci1000,3140@0/sd@2d,0
  13. c2t46d0 
  /pci@0,0/pci8086,1c10@1c/pci1000,3140@0/sd@2e,0
  14. c2t47d0 
  /pci@0,0/pci8086,1c10@1c/pci1000,3140@0/sd@2f,0
  15. c2t48d0 
  /pci@0,0/pci8086,1c10@1c/pci1000,3140@0/sd@30,0
  16. c2t49d0 
  /pci@0,0/pci8086,1c10@1c/pci1000,3140@0/sd@31,0

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-23 Thread Richard Elling 

> On Mar 23, 2016, at 7:49 AM, Richard Jahnel <rjah...@ellipseinc.com> wrote:
> 
> It should be noted that using a 512e disk as a 512n disk subjects you to a 
> significant risk of silent corruption in the event of power loss. Because 
> 512e disks does a read>modify>write operation to modify 512byte chunk of a 4k 
> sector, zfs won't know about the other 7 corrupted 512e sectors in the event 
> of a power loss during a write operation. So when discards the incomplete txg 
> on reboot, it won't do anything about the other 7 512e sectors it doesn't 
> know were affected.

Disagree. The risk is no greater than HDDs today with their volatile write 
caches.
 -- richard

> 
> Richard Jahnel
> Network Engineer
> On-Site.com | Ellipse Design
> 866-266-7483 ext. 4408
> Direct: 669-800-6270
> 
> 
> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On 
> Behalf Of Chris Siebenmann
> Sent: Wednesday, March 23, 2016 9:36 AM
> To: Fred Liu <fred_...@issi.com>
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
>>> The sd.conf whitelist also requires a reboot to activate if you need
>>> to add a new entry, as far as I know.
>>> 
>>>(Nor do I know what happens if you have some 512n disks and
>>>some 512e disks, both correctly recognized and in different
>>>pools, and now you need to replace a 512n disk with a spare 512e
>>>disk so you change sd.conf to claim that all of the 512e disks
>>>are 512n. I'd like to think that ZFS will carry on as normal,
>>>but I'm not sure.  This makes it somewhat dangerous to change
>>>sd.conf on a live system.)
>> 
>> There are two cases if we don't use the remedy (whitelist in illumos 
>> or -o ashift in ZoL) here:
>> a): 512n <---> 512e. This replacement should work *in theory* if the 
>> lie works *correctly*.
> 
> This will not work without the sd.conf workaround in Illumos.
> 
> All 512e disks that I know of correctly report their actual physical disk 
> size to Illumos (and to Linux/ZoL). When a disk reports a 4K physical sector 
> size, ZFS will refuse to allow it into an ashift=9 vdev *regardless* of the 
> fact that it is 512e and will accept reads and writes in 512-byte sectors.
> 
> In Illumos, you can use sd.conf to lie to the system and claim that this is 
> not a 512e but a 512n disk (ie, it has a 512 byte physical sector size). I 
> don't believe there's an equivalent on ZoL, but I haven't looked.
> 
> This absolute insistence on ZFS's part is what makes ashift=9 vdevs so 
> dangerous today, because you cannot replace existing 512n disks in them with 
> 512e disks without (significant) hackery.
> 
> (Perhaps I'm misunderstanding what people mean by '512e' here; I've been 
> assuming it means a disk which reports 512 byte logical sectors and 4k 
> physical sectors. Such disks are what you commonly get today.)
> 
>   - cks
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-23 Thread Richard Jahnel 
It should be noted that using a 512e disk as a 512n disk subjects you to a 
significant risk of silent corruption in the event of power loss. Because 512e 
disks does a read>modify>write operation to modify 512byte chunk of a 4k 
sector, zfs won't know about the other 7 corrupted 512e sectors in the event of 
a power loss during a write operation. So when discards the incomplete txg on 
reboot, it won't do anything about the other 7 512e sectors it doesn't know 
were affected.

Richard Jahnel
Network Engineer
On-Site.com | Ellipse Design
866-266-7483 ext. 4408
Direct: 669-800-6270


-Original Message-
From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On Behalf 
Of Chris Siebenmann
Sent: Wednesday, March 23, 2016 9:36 AM
To: Fred Liu <fred_...@issi.com>
Cc: omnios-discuss@lists.omniti.com
Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12

> > The sd.conf whitelist also requires a reboot to activate if you need
> > to add a new entry, as far as I know.
> > 
> > (Nor do I know what happens if you have some 512n disks and
> > some 512e disks, both correctly recognized and in different
> > pools, and now you need to replace a 512n disk with a spare 512e
> > disk so you change sd.conf to claim that all of the 512e disks
> > are 512n. I'd like to think that ZFS will carry on as normal,
> > but I'm not sure.  This makes it somewhat dangerous to change
> > sd.conf on a live system.)
> 
> There are two cases if we don't use the remedy (whitelist in illumos 
> or -o ashift in ZoL) here:
> a): 512n <---> 512e. This replacement should work *in theory* if the 
> lie works *correctly*.

 This will not work without the sd.conf workaround in Illumos.

 All 512e disks that I know of correctly report their actual physical disk size 
to Illumos (and to Linux/ZoL). When a disk reports a 4K physical sector size, 
ZFS will refuse to allow it into an ashift=9 vdev *regardless* of the fact that 
it is 512e and will accept reads and writes in 512-byte sectors.

 In Illumos, you can use sd.conf to lie to the system and claim that this is 
not a 512e but a 512n disk (ie, it has a 512 byte physical sector size). I 
don't believe there's an equivalent on ZoL, but I haven't looked.

 This absolute insistence on ZFS's part is what makes ashift=9 vdevs so 
dangerous today, because you cannot replace existing 512n disks in them with 
512e disks without (significant) hackery.

(Perhaps I'm misunderstanding what people mean by '512e' here; I've been 
assuming it means a disk which reports 512 byte logical sectors and 4k physical 
sectors. Such disks are what you commonly get today.)

- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-23 Thread Chris Siebenmann 
> > The sd.conf whitelist also requires a reboot to activate if you need
> > to add a new entry, as far as I know.
> > 
> > (Nor do I know what happens if you have some 512n disks and
> > some 512e disks, both correctly recognized and in different
> > pools, and now you need to replace a 512n disk with a spare 512e
> > disk so you change sd.conf to claim that all of the 512e disks
> > are 512n. I'd like to think that ZFS will carry on as normal,
> > but I'm not sure.  This makes it somewhat dangerous to change
> > sd.conf on a live system.)
> 
> There are two cases if we don't use the remedy (whitelist in illumos
> or -o ashift in ZoL) here:
> a): 512n <---> 512e. This replacement should work *in theory* if the
> lie works *correctly*.

 This will not work without the sd.conf workaround in Illumos.

 All 512e disks that I know of correctly report their actual physical
disk size to Illumos (and to Linux/ZoL). When a disk reports a 4K
physical sector size, ZFS will refuse to allow it into an ashift=9
vdev *regardless* of the fact that it is 512e and will accept reads
and writes in 512-byte sectors.

 In Illumos, you can use sd.conf to lie to the system and claim that
this is not a 512e but a 512n disk (ie, it has a 512 byte physical
sector size). I don't believe there's an equivalent on ZoL, but I
haven't looked.

 This absolute insistence on ZFS's part is what makes ashift=9 vdevs so
dangerous today, because you cannot replace existing 512n disks in them
with 512e disks without (significant) hackery.

(Perhaps I'm misunderstanding what people mean by '512e' here; I've
been assuming it means a disk which reports 512 byte logical sectors and
4k physical sectors. Such disks are what you commonly get today.)

- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-23 Thread Fred Liu 


> -Original Message-
> From: Richard Elling [mailto:richard.ell...@richardelling.com]
> Sent: 星期三, 三月 23, 2016 4:53
> To: Chris Siebenmann
> Cc: Fred Liu; omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
>   On Mar 22, 2016, at 7:41 AM, Chris Siebenmann <c...@cs.toronto.edu
> <mailto:c...@cs.toronto.edu> > wrote:
> 
> 
>   This implicitly assumes that the only reason to set 
> ashift=12 is
>   if you are currently using one or more drives that 
> require it. I
>   strongly disagree with this view. Since ZFS cannot 
> currently
> replace
>   a 512n drive with a 512e one, I feel [...]
> 
> 
> 
>   *In theory* this replacement should work well if the lie works
> *correctly*.
>   In ZoL, for the "-o ashift" is supported in "zpool replace", the
>   replacement should also work in mixed sector sizes.
>   And in illumos the whitelist will do the same.
>   What errors have you ever seen?
> 
> 
> 
>   We have seen devices that changed between (claimed) 512n and
>   (claimed) 512e/4k *within the same model number*; the only thing that
>   distinguished the two was firmware version (which is not something that
>   you can match in sd.conf). This came as a complete surprise to us the
>   first time we needed to replace an old (512n) one of these with a new
>   (512e) one.
> 
>   The sd.conf whitelist also requires a reboot to activate if you need
>   to add a new entry, as far as I know.
> 
>   (Nor do I know what happens if you have some 512n disks and some
>   512e disks, both correctly recognized and in different pools, and
>   now you need to replace a 512n disk with a spare 512e disk so you
>   change sd.conf to claim that all of the 512e disks are 512n. I'd
>   like to think that ZFS will carry on as normal, but I'm not sure.
>   This makes it somewhat dangerous to change sd.conf on a live system.)

There are two cases if we don't use the remedy (whitelist in illumos or -o 
ashift in ZoL) here:
a): 512n <---> 512e. This replacement should work *in theory* if the lie works 
*correctly*.
b): 512n <-x-> 4kn.  This replacement may not work for the different physical 
sector sizes.

Your surprise may come from case b.
> 
> 
> 
> What is missing from
> http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks
> is:
> 
> 1. how to change the un_phy_blocksize for any or all uns 2. how to set a 
> default
> setting for all drives in sd.conf by setting attributes to
> the "<vid+pid>" of ""  (see sd(7d))
> 
> I am aware of no new HDDs with 512n, so this problem will go away for HDDs.
> However, there are many SSDs that work better with un_phy_blocksize = 8192
> and some vendors set sd.conf or source appropriately.
>  -- richard
> 
> 
> 
>   For many usage cases, somewhat more space usage and
> perhaps
>   somewhat slower pools are vastly preferable to a loss 
> of pool
>   redundancy over time. I feel that OmniOS should at 
> least give
> you
>   the option here (in a less crude way than simply 
> telling it that
>   absolutely all of your drives are 4k drives, partly 
> because such
>   general lies are problematic in various situations).
> 
> 
> 
>   The whitelist (sd.conf) should fit into this consideration. But 
> not
>   sure how mixed sector sizes impact the performance.
> 
> 
> 
>   Oh, 512e disks in a 512n pool will probably have not great performance.
>   ZFS does a lot of unaligned reads and writes, unlike other filesystems;
>   if you say your disks are 512n, it really believes you and behaves
>   accordingly.

I am just curious about if the mixed sector sizes(512n+4kn) will impact 
performance.

Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Richard Elling 

> On Mar 22, 2016, at 7:41 AM, Chris Siebenmann  wrote:
> 
>>> This implicitly assumes that the only reason to set ashift=12 is
>>> if you are currently using one or more drives that require it. I
>>> strongly disagree with this view. Since ZFS cannot currently replace
>>> a 512n drive with a 512e one, I feel [...]
>> 
>> *In theory* this replacement should work well if the lie works *correctly*.
>> In ZoL, for the "-o ashift" is supported in "zpool replace", the
>> replacement should also work in mixed sector sizes.
>> And in illumos the whitelist will do the same.
>> What errors have you ever seen?
> 
> We have seen devices that changed between (claimed) 512n and
> (claimed) 512e/4k *within the same model number*; the only thing that
> distinguished the two was firmware version (which is not something that
> you can match in sd.conf). This came as a complete surprise to us the
> first time we needed to replace an old (512n) one of these with a new
> (512e) one.
> 
> The sd.conf whitelist also requires a reboot to activate if you need
> to add a new entry, as far as I know.
> 
> (Nor do I know what happens if you have some 512n disks and some
> 512e disks, both correctly recognized and in different pools, and
> now you need to replace a 512n disk with a spare 512e disk so you
> change sd.conf to claim that all of the 512e disks are 512n. I'd
> like to think that ZFS will carry on as normal, but I'm not sure.
> This makes it somewhat dangerous to change sd.conf on a live system.)

What is missing from
http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks 

is:

1. how to change the un_phy_blocksize for any or all uns
2. how to set a default setting for all drives in sd.conf by setting attributes 
to
the "" of ""  (see sd(7d))

I am aware of no new HDDs with 512n, so this problem will go away for HDDs.
However, there are many SSDs that work better with un_phy_blocksize = 8192
and some vendors set sd.conf or source appropriately.
 -- richard

> 
>>> For many usage cases, somewhat more space usage and perhaps
>>> somewhat slower pools are vastly preferable to a loss of pool
>>> redundancy over time. I feel that OmniOS should at least give you
>>> the option here (in a less crude way than simply telling it that
>>> absolutely all of your drives are 4k drives, partly because such
>>> general lies are problematic in various situations).
>> 
>> The whitelist (sd.conf) should fit into this consideration. But not
>> sure how mixed sector sizes impact the performance.
> 
> Oh, 512e disks in a 512n pool will probably have not great performance.
> ZFS does a lot of unaligned reads and writes, unlike other filesystems;
> if you say your disks are 512n, it really believes you and behaves
> accordingly.
> 
>   - cks

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Chris Siebenmann 
> >  This implicitly assumes that the only reason to set ashift=12 is
> > if you are currently using one or more drives that require it. I
> > strongly disagree with this view. Since ZFS cannot currently replace
> > a 512n drive with a 512e one, I feel [...]
> 
> *In theory* this replacement should work well if the lie works *correctly*.
>  In ZoL, for the "-o ashift" is supported in "zpool replace", the
> replacement should also work in mixed sector sizes.
>  And in illumos the whitelist will do the same.
>  What errors have you ever seen?

 We have seen devices that changed between (claimed) 512n and
(claimed) 512e/4k *within the same model number*; the only thing that
distinguished the two was firmware version (which is not something that
you can match in sd.conf). This came as a complete surprise to us the
first time we needed to replace an old (512n) one of these with a new
(512e) one.

 The sd.conf whitelist also requires a reboot to activate if you need
to add a new entry, as far as I know.

(Nor do I know what happens if you have some 512n disks and some
512e disks, both correctly recognized and in different pools, and
now you need to replace a 512n disk with a spare 512e disk so you
change sd.conf to claim that all of the 512e disks are 512n. I'd
like to think that ZFS will carry on as normal, but I'm not sure.
This makes it somewhat dangerous to change sd.conf on a live system.)

> >  For many usage cases, somewhat more space usage and perhaps
> > somewhat slower pools are vastly preferable to a loss of pool
> > redundancy over time. I feel that OmniOS should at least give you
> > the option here (in a less crude way than simply telling it that
> > absolutely all of your drives are 4k drives, partly because such
> > general lies are problematic in various situations).
>
> The whitelist (sd.conf) should fit into this consideration. But not
> sure how mixed sector sizes impact the performance.

 Oh, 512e disks in a 512n pool will probably have not great performance.
ZFS does a lot of unaligned reads and writes, unlike other filesystems;
if you say your disks are 512n, it really believes you and behaves
accordingly.

- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Fred Liu 


> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Richard Elling
> Sent: 星期二, 三月 22, 2016 3:19
> To: Richard Jahnel
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
> > On Mar 21, 2016, at 12:00 PM, Richard Jahnel <rjah...@ellipseinc.com>
> wrote:
> >
> > Both approaches have their error points.
> >
> > FWIW I would very very much like to be able to force my new pools into
> ashift=12. It would make drive purchasing and replacement a lot more straight
> forward and future resistant.
> 
> The fundamental problem is that this is a vdev-settable, not a pool settable.
> Today, it is very common for pools to have multiple ashifts active. Recently,
> per-vdev ZAP objects have been proposed and that code is working its way
> through the review and integration process.

Not sure how multiple-ashifts impact the performace.
> 
> Meanwhile, use one or more of the dozens of methods documented for doing
> this.
> 
> FWIW, most people with HDDs want space efficiency, because HDDs lost the
> performance race to SSDs long ago. In general, forcing everything to 4k 
> reduces
> space efficiency, so it is unlikely to be a good default.

Agree.

Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Fred Liu 


> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Richard Elling
> Sent: 星期二, 三月 22, 2016 3:21
> To: Bob Friesenhahn
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
> > On Mar 21, 2016, at 12:19 PM, Bob Friesenhahn
> <bfrie...@simple.dallas.tx.us> wrote:
> >
> > On Mon, 21 Mar 2016, Richard Elling wrote:
> >>>
> >>> Adding the ashift argument to zpool was discussed every few years and so
> far was always deemed not enterprisey enough for the Solaris heritage, so the
> setup to tweak sd driver reports and properly rely on that layer was pushed
> instead.
> >>
> >> The issue is that once a drive model lies, then the Solaris approach
> >> is to encode the lie into a whitelist, so that the lie is always
> >> handled properly. The whitelist is in the sd.conf file.
> >
> > Does this approach require that Illumos users only use drive hardware much
> older than the version of Illumos they happen running since outdated whitelist
> won't know about the new lies?
> 
> Forunately, lies are becoming less common. But this raises a good point: if 
> your
> drive doesn't lie, then you don't need to workaround.

That means 512n and 4kn are handled correctly by default, right?


Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Fred Liu 


> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Hanno Hirschberger
> Sent: 星期一, 三月 21, 2016 17:02
> To: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> On 21.03.2016 08:00, Fred Liu wrote:
> > So that means illumos can handle 512n and 4kn automatically and properly?
> 
> Not necessarily as far as I know. Sometime drives are emulating 512 blocks and
> don't properly tell the OS about that and Illumos ZFS is aligning the drives 
> with
> ashift=9 which leads to enormous performance issues. Also forcing the system

That is the goal(ashift=9) that 512e wants to realize. Isn't it? Emulation is 
to guarantee
compatiblity not performace, right?

> to handle drives with a specific sector size with the sd.conf doesn't turn 
> out to
> be reliable in some cases (at least on my workstations). Here's what I do to

What errors have you ever seen?

> ensure ashift=12 values:
> 
> Reboot the system with a Linux live disk of your choice and install ZoL in 
> the live
> session. Then create the ZFS pool, export it and reboot the machine. OmniOS /
> Illumos can import the new pool without problems and the ashift value is

This operation is driver-dependant. I have the similar workaround. It works for 
avago HBAs.
But it doesn't apply in NVMe drivers. The zpool comprised of NVMe SSDs created 
under ZoL can't be imported under Illumos
in my tests.

> correctly set. There was a fixed zpool binary (Solaris 11 binary) flying 
> around the
> internet which can handle the "-o shift=12" parameter and works with OmniOS
> but unfortunately I can't find it again right now. This would make the reboot
> into a live session obsolete.
> 
> Does anyone know if the "ashift" parameter will be implemented in the OmniOS
> / Illumos zpool binary in the near future?

I think the whitelist(sd.conf) is the choice by Illumos. But we may try build 
it ourselves if the source is available.

Thanks.

Fred 
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Fred Liu 


> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Jim Klimov
> Sent: 星期二, 三月 22, 2016 2:11
> To: Hanno Hirschberger; omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 21 марта 2016 г. 10:02:03 CET, Hanno Hirschberger
> <hannohirschber...@googlemail.com> пишет:
> >On 21.03.2016 08:00, Fred Liu wrote:
> >> So that means illumos can handle 512n and 4kn automatically and
> >properly?
> >
> >Not necessarily as far as I know. Sometime drives are emulating 512
> >blocks and don't properly tell the OS about that and Illumos ZFS is
> >aligning the drives with ashift=9 which leads to enormous performance
> >issues. Also forcing the system to handle drives with a specific sector
> >
> >size with the sd.conf doesn't turn out to be reliable in some cases (at
> >
> >least on my workstations). Here's what I do to ensure ashift=12 values:
> >
> >Reboot the system with a Linux live disk of your choice and install ZoL
> >
> >in the live session. Then create the ZFS pool, export it and reboot the
> >
> >machine. OmniOS / Illumos can import the new pool without problems and
> >the ashift value is correctly set. There was a fixed zpool binary
> >(Solaris 11 binary) flying around the internet which can handle the "-o
> >
> >shift=12" parameter and works with OmniOS but unfortunately I can't
> >find it again right now. This would make the reboot into a live session
> >obsolete.
> >
> >Does anyone know if the "ashift" parameter will be implemented in the
> >OmniOS / Illumos zpool binary in the near future?
> >
> >Best regards
> >
> >Hanno
> >___
> >OmniOS-discuss mailing list
> >OmniOS-discuss@lists.omniti.com
> >http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
> Adding the ashift argument to zpool was discussed every few years and so far
> was always deemed not enterprisey enough for the Solaris heritage, so the
> setup to tweak sd driver reports and properly rely on that layer was pushed
> instead.
> 
> That said, the old tweaked binary came with a blog post detailing the source
> changes; you're welcome to try a d port and rti it (I'd say there is enough 
> user
> demand to back the non-enterprisey fix to be on par with other OpenZFS
> siblings). At worst, you can publish the modernized binary as the original
> blogger did ;)

Is the post/source still accessible?


Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-22 Thread Fred Liu 


> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Chris Siebenmann
> Sent: 星期二, 三月 22, 2016 3:27
> To: Richard Elling
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> > > Adding the ashift argument to zpool was discussed every few years
> > > and so far was always deemed not enterprisey enough for the Solaris
> > > heritage, so the setup to tweak sd driver reports and properly rely
> > > on that layer was pushed instead.
> >
> > The issue is that once a drive model lies, then the Solaris approach
> > is to encode the lie into a whitelist, so that the lie is always
> > handled properly. The whitelist is in the sd.conf file.
> >
> > By contrast, the ZFSonLinux approach requires that the sysadmin knows
> > there is a lie and manually corrects for every invocation. This is
> > unfortunately a very error-prone approach.
> 
>  This implicitly assumes that the only reason to set ashift=12 is if you are
> currently using one or more drives that require it. I strongly disagree with 
> this
> view. Since ZFS cannot currently replace a 512n drive with a 512e one, I feel

*In theory* this replacement should work well if the lie works *correctly*.
 In ZoL, for the "-o ashift" is supported in "zpool replace", the replacement 
should also work in mixed sector sizes.
 And in illumos the whitelist will do the same.
 What errors have you ever seen?

> that you really want to force-create all pools with ashift=12 even on 512n 
> drives
> unless you're absolutely confident that you will be able to obtain replacement
> 512n drives over the lifetime of your pool (or unless the impact of using
> ashift=12 is so drastic on your usage case that you will need to totally 
> rethink
> your pool if you become unable to get 512n replacement drives).

Same as my comments above.
> 
>  For many usage cases, somewhat more space usage and perhaps somewhat
> slower pools are vastly preferable to a loss of pool redundancy over time. I 
> feel
> that OmniOS should at least give you the option here (in a less crude way than
> simply telling it that absolutely all of your drives are 4k drives, partly 
> because
> such general lies are problematic in various situations).
> 

The whitelist (sd.conf) should fit into this consideration. But not sure how 
mixed sector sizes impact the performance.


Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Chris Siebenmann 
> > Adding the ashift argument to zpool was discussed every few years
> > and so far was always deemed not enterprisey enough for the Solaris
> > heritage, so the setup to tweak sd driver reports and properly rely
> > on that layer was pushed instead.
>
> The issue is that once a drive model lies, then the Solaris approach
> is to encode the lie into a whitelist, so that the lie is always
> handled properly. The whitelist is in the sd.conf file.
>
> By contrast, the ZFSonLinux approach requires that the sysadmin knows
> there is a lie and manually corrects for every invocation. This is
> unfortunately a very error-prone approach.

 This implicitly assumes that the only reason to set ashift=12 is if
you are currently using one or more drives that require it. I strongly
disagree with this view. Since ZFS cannot currently replace a 512n
drive with a 512e one, I feel that you really want to force-create
all pools with ashift=12 even on 512n drives unless you're absolutely
confident that you will be able to obtain replacement 512n drives over
the lifetime of your pool (or unless the impact of using ashift=12 is so
drastic on your usage case that you will need to totally rethink your
pool if you become unable to get 512n replacement drives).

 For many usage cases, somewhat more space usage and perhaps somewhat
slower pools are vastly preferable to a loss of pool redundancy over
time. I feel that OmniOS should at least give you the option here (in
a less crude way than simply telling it that absolutely all of your
drives are 4k drives, partly because such general lies are problematic
in various situations).

- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Richard Elling 

> On Mar 21, 2016, at 12:00 PM, Richard Jahnel <rjah...@ellipseinc.com> wrote:
> 
> Both approaches have their error points.
> 
> FWIW I would very very much like to be able to force my new pools into 
> ashift=12. It would make drive purchasing and replacement a lot more straight 
> forward and future resistant.

The fundamental problem is that this is a vdev-settable, not a pool settable. 
Today, it is very common
for pools to have multiple ashifts active. Recently, per-vdev ZAP objects have 
been proposed and that
code is working its way through the review and integration process.

Meanwhile, use one or more of the dozens of methods documented for doing this.

FWIW, most people with HDDs want space efficiency, because HDDs lost the 
performance race to
SSDs long ago. In general, forcing everything to 4k reduces space efficiency, 
so it is unlikely to be
a good default.
 -- richard

> 
> Regards
> 
> Richard Jahnel
> Network Engineer
> On-Site.com | Ellipse Design
> 866-266-7483 ext. 4408
> Direct: 669-800-6270
> 
> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On 
> Behalf Of Richard Elling
> Sent: Monday, March 21, 2016 1:54 PM
> To: Jim Klimov <jimkli...@cos.ru>
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
>> On Mar 21, 2016, at 11:11 AM, Jim Klimov <jimkli...@cos.ru> wrote:
>> 
>> 21 марта 2016 г. 10:02:03 CET, Hanno Hirschberger 
>> <hannohirschber...@googlemail.com> пишет:
>>> On 21.03.2016 08:00, Fred Liu wrote:
>>>> So that means illumos can handle 512n and 4kn automatically and
>>> properly?
>>> 
>>> Not necessarily as far as I know. Sometime drives are emulating 512 
>>> blocks and don't properly tell the OS about that and Illumos ZFS is 
>>> aligning the drives with ashift=9 which leads to enormous performance 
>>> issues. Also forcing the system to handle drives with a specific 
>>> sector
>>> 
>>> size with the sd.conf doesn't turn out to be reliable in some cases 
>>> (at
>>> 
>>> least on my workstations). Here's what I do to ensure ashift=12 values:
>>> 
>>> Reboot the system with a Linux live disk of your choice and install 
>>> ZoL
>>> 
>>> in the live session. Then create the ZFS pool, export it and reboot 
>>> the
>>> 
>>> machine. OmniOS / Illumos can import the new pool without problems 
>>> and the ashift value is correctly set. There was a fixed zpool binary 
>>> (Solaris 11 binary) flying around the internet which can handle the 
>>> "-o
>>> 
>>> shift=12" parameter and works with OmniOS but unfortunately I can't 
>>> find it again right now. This would make the reboot into a live 
>>> session obsolete.
>>> 
>>> Does anyone know if the "ashift" parameter will be implemented in the 
>>> OmniOS / Illumos zpool binary in the near future?
>>> 
>>> Best regards
>>> 
>>> Hanno
>>> ___
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss@lists.omniti.com
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>> 
>> Adding the ashift argument to zpool was discussed every few years and so far 
>> was always deemed not enterprisey enough for the Solaris heritage, so the 
>> setup to tweak sd driver reports and properly rely on that layer was pushed 
>> instead.
> 
> The issue is that once a drive model lies, then the Solaris approach is to 
> encode the lie into a whitelist, so that the lie is always handled properly. 
> The whitelist is in the sd.conf file.
> 
> By contrast, the ZFSonLinux approach requires that the sysadmin knows there 
> is a lie and manually corrects for every invocation. This is unfortunately a 
> very error-prone approach.
> -- richard
> 
>> 
>> That said, the old tweaked binary came with a blog post detailing the 
>> source changes; you're welcome to try a d port and rti it (I'd say 
>> there is enough user demand to back the non-enterprisey fix to be on 
>> par with other OpenZFS siblings). At worst, you can publish the 
>> modernized binary as the original blogger did ;)
>> 
>> Jim
>> --
>> Typos courtesy of K-9 Mail on my Samsung Android 
>> ___
>> OmniOS-discuss mailing list
>> OmniOS-discuss@lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Richard Elling 

> On Mar 21, 2016, at 12:19 PM, Bob Friesenhahn  
> wrote:
> 
> On Mon, 21 Mar 2016, Richard Elling wrote:
>>> 
>>> Adding the ashift argument to zpool was discussed every few years and so 
>>> far was always deemed not enterprisey enough for the Solaris heritage, so 
>>> the setup to tweak sd driver reports and properly rely on that layer was 
>>> pushed instead.
>> 
>> The issue is that once a drive model lies, then the Solaris approach is to 
>> encode
>> the lie into a whitelist, so that the lie is always handled properly. The 
>> whitelist is in the
>> sd.conf file.
> 
> Does this approach require that Illumos users only use drive hardware much 
> older than the version of Illumos they happen running since outdated 
> whitelist won't know about the new lies?

Forunately, lies are becoming less common. But this raises a good point: if 
your drive doesn't lie,
then you don't need to workaround.

> 
> What if a user is using classic drives but wants to be prepared to install 
> newer drives which require ashift=12?

See the bazillion other posts on this topic.
 -- richard

> 
> Bob
> -- 
> Bob Friesenhahn
> bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Bob Friesenhahn 

On Mon, 21 Mar 2016, Richard Elling wrote:


Adding the ashift argument to zpool was discussed every few years and so far 
was always deemed not enterprisey enough for the Solaris heritage, so the setup 
to tweak sd driver reports and properly rely on that layer was pushed instead.


The issue is that once a drive model lies, then the Solaris approach is to 
encode
the lie into a whitelist, so that the lie is always handled properly. The 
whitelist is in the
sd.conf file.


Does this approach require that Illumos users only use drive hardware 
much older than the version of Illumos they happen running since 
outdated whitelist won't know about the new lies?


What if a user is using classic drives but wants to be prepared to 
install newer drives which require ashift=12?


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Richard Elling 

> On Mar 21, 2016, at 11:11 AM, Jim Klimov  wrote:
> 
> 21 марта 2016 г. 10:02:03 CET, Hanno Hirschberger 
>  пишет:
>> On 21.03.2016 08:00, Fred Liu wrote:
>>> So that means illumos can handle 512n and 4kn automatically and
>> properly?
>> 
>> Not necessarily as far as I know. Sometime drives are emulating 512 
>> blocks and don't properly tell the OS about that and Illumos ZFS is 
>> aligning the drives with ashift=9 which leads to enormous performance 
>> issues. Also forcing the system to handle drives with a specific sector
>> 
>> size with the sd.conf doesn't turn out to be reliable in some cases (at
>> 
>> least on my workstations). Here's what I do to ensure ashift=12 values:
>> 
>> Reboot the system with a Linux live disk of your choice and install ZoL
>> 
>> in the live session. Then create the ZFS pool, export it and reboot the
>> 
>> machine. OmniOS / Illumos can import the new pool without problems and 
>> the ashift value is correctly set. There was a fixed zpool binary 
>> (Solaris 11 binary) flying around the internet which can handle the "-o
>> 
>> shift=12" parameter and works with OmniOS but unfortunately I can't
>> find 
>> it again right now. This would make the reboot into a live session
>> obsolete.
>> 
>> Does anyone know if the "ashift" parameter will be implemented in the 
>> OmniOS / Illumos zpool binary in the near future?
>> 
>> Best regards
>> 
>> Hanno
>> ___
>> OmniOS-discuss mailing list
>> OmniOS-discuss@lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
> Adding the ashift argument to zpool was discussed every few years and so far 
> was always deemed not enterprisey enough for the Solaris heritage, so the 
> setup to tweak sd driver reports and properly rely on that layer was pushed 
> instead.

The issue is that once a drive model lies, then the Solaris approach is to 
encode
the lie into a whitelist, so that the lie is always handled properly. The 
whitelist is in the
sd.conf file.

By contrast, the ZFSonLinux approach requires that the sysadmin knows there is a
lie and manually corrects for every invocation. This is unfortunately a very 
error-prone
approach.
 -- richard

> 
> That said, the old tweaked binary came with a blog post detailing the source 
> changes; you're welcome to try a d port and rti it (I'd say there is enough 
> user demand to back the non-enterprisey fix to be on par with other OpenZFS 
> siblings). At worst, you can publish the modernized binary as the original 
> blogger did ;)
> 
> Jim
> --
> Typos courtesy of K-9 Mail on my Samsung Android
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Jim Klimov 
21 марта 2016 г. 10:02:03 CET, Hanno Hirschberger 
 пишет:
>On 21.03.2016 08:00, Fred Liu wrote:
>> So that means illumos can handle 512n and 4kn automatically and
>properly?
>
>Not necessarily as far as I know. Sometime drives are emulating 512 
>blocks and don't properly tell the OS about that and Illumos ZFS is 
>aligning the drives with ashift=9 which leads to enormous performance 
>issues. Also forcing the system to handle drives with a specific sector
>
>size with the sd.conf doesn't turn out to be reliable in some cases (at
>
>least on my workstations). Here's what I do to ensure ashift=12 values:
>
>Reboot the system with a Linux live disk of your choice and install ZoL
>
>in the live session. Then create the ZFS pool, export it and reboot the
>
>machine. OmniOS / Illumos can import the new pool without problems and 
>the ashift value is correctly set. There was a fixed zpool binary 
>(Solaris 11 binary) flying around the internet which can handle the "-o
>
>shift=12" parameter and works with OmniOS but unfortunately I can't
>find 
>it again right now. This would make the reboot into a live session
>obsolete.
>
>Does anyone know if the "ashift" parameter will be implemented in the 
>OmniOS / Illumos zpool binary in the near future?
>
>Best regards
>
>Hanno
>___
>OmniOS-discuss mailing list
>OmniOS-discuss@lists.omniti.com
>http://lists.omniti.com/mailman/listinfo/omnios-discuss

Adding the ashift argument to zpool was discussed every few years and so far 
was always deemed not enterprisey enough for the Solaris heritage, so the setup 
to tweak sd driver reports and properly rely on that layer was pushed instead.

That said, the old tweaked binary came with a blog post detailing the source 
changes; you're welcome to try a d port and rti it (I'd say there is enough 
user demand to back the non-enterprisey fix to be on par with other OpenZFS 
siblings). At worst, you can publish the modernized binary as the original 
blogger did ;)

Jim
--
Typos courtesy of K-9 Mail on my Samsung Android
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Hanno Hirschberger 

On 21.03.2016 08:00, Fred Liu wrote:

So that means illumos can handle 512n and 4kn automatically and properly?


Not necessarily as far as I know. Sometime drives are emulating 512 
blocks and don't properly tell the OS about that and Illumos ZFS is 
aligning the drives with ashift=9 which leads to enormous performance 
issues. Also forcing the system to handle drives with a specific sector 
size with the sd.conf doesn't turn out to be reliable in some cases (at 
least on my workstations). Here's what I do to ensure ashift=12 values:


Reboot the system with a Linux live disk of your choice and install ZoL 
in the live session. Then create the ZFS pool, export it and reboot the 
machine. OmniOS / Illumos can import the new pool without problems and 
the ashift value is correctly set. There was a fixed zpool binary 
(Solaris 11 binary) flying around the internet which can handle the "-o 
shift=12" parameter and works with OmniOS but unfortunately I can't find 
it again right now. This would make the reboot into a live session obsolete.


Does anyone know if the "ashift" parameter will be implemented in the 
OmniOS / Illumos zpool binary in the near future?


Best regards

Hanno
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Fred Liu 
So that means illumos can handle 512n and 4kn automatically and properly?

Thanks.

Fred

From: Matej Žerovnik [mailto:ma...@zunaj.si]
Sent: 星期一, 三月 21, 2016 14:20
To: Fred Liu
Cc: geo...@gnaa.net; omnios-discuss; zfs-disc...@list.zfsonlinux.org
Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12

You can’t do that, but you can force system to recognize hard drive as if it 
was 4Kn. Just like we do it for SSDs: 
http://wiki.illumos.org/display/illumos/List+of+sd-config-list+entries+for+Advanced-Format+drives

lp, Matej


On 21 Mar 2016, at 06:00, Fred Liu 
<fred_...@issi.com<mailto:fred_...@issi.com>> wrote:

So, you can “zpool create –o ashift=12” in illumos? I can’t do that in smartos 
at least….

From: Geoff Nordli [mailto:geoff.nor...@gmail.com] On Behalf Of Geoff Nordli
Sent: 星期一, 三月 21, 2016 10:59
To: Fred Liu; omnios-discuss
Cc: zfs-disc...@list.zfsonlinux.org<mailto:zfs-disc...@list.zfsonlinux.org>
Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12



Hi Fred.

Definitely, the ashift parameter has been around on the illumos for a couple of 
years.

thanks,

Geoff

On 16-03-20 05:21 PM, Fred Liu wrote:
The ashift parameter doesn't apply in illumos if my memory serves me well. It 
was introduced by ZoL.

Thanks.

Fred






On Sun, Mar 20, 2016 at 1:37 PM -0700, "Geoff Nordli" 
<geo...@gnaa.net<mailto:geo...@gnaa.net>> wrote:
Quick, question.

Any performance differences between 4Kn and 512e with ashift=12?

I am thinking there should not be, since they will both write the full
4K block size.

The workload will be virtual machines using a zvol.

thanks,

Geoff
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com<mailto:OmniOS-discuss@lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com<mailto:OmniOS-discuss@lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-21 Thread Matej Žerovnik 
You can’t do that, but you can force system to recognize hard drive as if it 
was 4Kn. Just like we do it for SSDs: 
http://wiki.illumos.org/display/illumos/List+of+sd-config-list+entries+for+Advanced-Format+drives

lp, Matej


> On 21 Mar 2016, at 06:00, Fred Liu <fred_...@issi.com> wrote:
> 
> So, you can “zpool create –o ashift=12” in illumos? I can’t do that in 
> smartos at least….
>  
> From: Geoff Nordli [mailto:geoff.nor...@gmail.com 
> <mailto:geoff.nor...@gmail.com>] On Behalf Of Geoff Nordli
> Sent: 星期一, 三月 21, 2016 10:59
> To: Fred Liu; omnios-discuss
> Cc: zfs-disc...@list.zfsonlinux.org <mailto:zfs-disc...@list.zfsonlinux.org>
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
>  
> 
> 
> Hi Fred. 
> 
> Definitely, the ashift parameter has been around on the illumos for a couple 
> of years.  
> 
> thanks,
> 
> Geoff 
> 
> On 16-03-20 05:21 PM, Fred Liu wrote:
> The ashift parameter doesn't apply in illumos if my memory serves me well. It 
> was introduced by ZoL.
>  
> Thanks.
>  
> Fred
>  
> 
>  
>  
> 
> 
> 
> On Sun, Mar 20, 2016 at 1:37 PM -0700, "Geoff Nordli" <geo...@gnaa.net 
> <mailto:geo...@gnaa.net>> wrote:
> 
> Quick, question.
> 
> Any performance differences between 4Kn and 512e with ashift=12?
> 
> I am thinking there should not be, since they will both write the full 
> 4K block size.
> 
> The workload will be virtual machines using a zvol.
> 
> thanks,
> 
> Geoff
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com <mailto:OmniOS-discuss@lists.omniti.com>
> http://lists.omniti.com/mailman/listinfo/omnios-discuss 
> <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>  
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com <mailto:OmniOS-discuss@lists.omniti.com>
> http://lists.omniti.com/mailman/listinfo/omnios-discuss 
> <http://lists.omniti.com/mailman/listinfo/omnios-discuss>



smime.p7s
Description: S/MIME cryptographic signature
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-20 Thread Fred Liu 
So, you can “zpool create –o ashift=12” in illumos? I can’t do that in smartos 
at least….

From: Geoff Nordli [mailto:geoff.nor...@gmail.com] On Behalf Of Geoff Nordli
Sent: 星期一, 三月 21, 2016 10:59
To: Fred Liu; omnios-discuss
Cc: zfs-disc...@list.zfsonlinux.org
Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12



Hi Fred.

Definitely, the ashift parameter has been around on the illumos for a couple of 
years.

thanks,

Geoff

On 16-03-20 05:21 PM, Fred Liu wrote:
The ashift parameter doesn't apply in illumos if my memory serves me well. It 
was introduced by ZoL.

Thanks.

Fred





On Sun, Mar 20, 2016 at 1:37 PM -0700, "Geoff Nordli" 
<geo...@gnaa.net<mailto:geo...@gnaa.net>> wrote:
Quick, question.

Any performance differences between 4Kn and 512e with ashift=12?

I am thinking there should not be, since they will both write the full
4K block size.

The workload will be virtual machines using a zvol.

thanks,

Geoff
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com<mailto:OmniOS-discuss@lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-20 Thread Fred Liu 
The ashift parameter doesn't apply in illumos if my memory serves me well. It 
was introduced by ZoL.

Thanks.

Fred







On Sun, Mar 20, 2016 at 1:37 PM -0700, "Geoff Nordli" 
> wrote:

Quick, question.

Any performance differences between 4Kn and 512e with ashift=12?

I am thinking there should not be, since they will both write the full
4K block size.

The workload will be virtual machines using a zvol.

thanks,

Geoff
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

[OmniOS-discuss] 4kn or 512e with ashift=12

2016-03-20 Thread Geoff Nordli 

Quick, question.

Any performance differences between 4Kn and 512e with ashift=12?

I am thinking there should not be, since they will both write the full 
4K block size.


The workload will be virtual machines using a zvol.

thanks,

Geoff
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss