Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target

[Christopher Forgeron]

I just submitted this as a bug:

( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204641 )

..but I thought I should bring it to the list's attention for more exposure
- If that's a no-no, let me know, as I have a few others that are related
to this that I'd like to discuss.

- - - -


Consider this scenario:

Virtual FreeBSD Machine, with a zpool created out of iSCSI disks.
Physical FreeBSD Machine, with a zpool holding a sparse file that is the
target for the iSCSI disk.

This setup works in an environment with all 10.1 machines, doesn't with all
10.2 machines.

- The 10.2 Machines are 10.2-p7 RELEASE, updated via freebsd-update, no
custom.
- The 10.1 Machine are 10.1-p24 RELEASE, updated via freebsd-update, no
custom.
- iSCSI is all CAM iSCSI, not the old istgt platform.
- The iSCSI Target is a sparse file, stored on a zpool (not a vdev Target)

The target machine is the same physical machine, with the same zpools - I
either boot 10.1 or 10.2 for testing, and use the same zpool/disks

to ensure nothing is changing.

If I have a 10.2 iSCSI Initiator (client) connected to a 10.2 iSCSI Target,
TRIM doesn't work (shows as NONE below).
If I have a 10.2 iSCSI Initiator (client) connected to a 10.1 iSCSI Target,
TRIM does work.

(There is another bug with that last scenario as well, but I will open it
separately)

...for clarity, a 10.1 iSCSI Initiator connected to a 10.1 iSCSI Target
also works perfectly. I have ~20 of these in the field.

On the 10.1 / 10.2 Targets, the ctl.conf file is identical. Zpools are
identical, because they are shared between reboots of the same iSCSI

target machine.



On the 10.2 initiator machine, connected to a 10.2 Target machine:

# sysctl -a | grep cam.da

kern.cam.da.2.minimum_cmd_size: 6
kern.cam.da.2.delete_max: 131072
kern.cam.da.2.delete_method: NONE
kern.cam.da.1.error_inject: 0
kern.cam.da.1.sort_io_queue: 0
kern.cam.da.1.minimum_cmd_size: 6
kern.cam.da.1.delete_max: 131072
kern.cam.da.1.delete_method: NONE
kern.cam.da.0.error_inject: 0
kern.cam.da.0.sort_io_queue: -1
kern.cam.da.0.minimum_cmd_size: 6
kern.cam.da.0.delete_max: 131072
kern.cam.da.0.delete_method: NONE

Note the delete_method is NONE


# sysctl -a | grep trim
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vdev.trim_max_pending: 1
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_on_init: 1
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 181
kstat.zfs.misc.zio_trim.success: 0
kstat.zfs.misc.zio_trim.bytes: 0

Note no trimmed bytes.


On the target machine, 10.1 and 10.2 share the same config file:
/etc/ctl.conf

portal-group pg0 {
discovery-auth-group no-authentication
listen 0.0.0.0
listen [::]
}

lun 0 {
path /pool92/iscsi/iscsi.zvol
blocksize 4K
size 5T
option unmap "on"
option scsiname "pool92"
option vendor "pool92"
option insecure_tpc "on"
}
}


target iqn.iscsi1.zvol {
auth-group no-authentication
portal-group pg0

lun 0 {
path /pool92_1/iscsi/iscsi.zvol
blocksize 4K
size 5T
option unmap "on"
option scsiname "pool92_1"
option vendor "pool92_1"
option insecure_tpc "on"
}
}


When I boot a 10.1 Target server, the 10.2 initiator connects, and we do
see proper UNMAP ability:


kern.cam.da.2.minimum_cmd_size: 6
kern.cam.da.2.delete_max: 5497558138880
kern.cam.da.2.delete_method: UNMAP
kern.cam.da.1.error_inject: 0
kern.cam.da.1.sort_io_queue: 0
kern.cam.da.1.minimum_cmd_size: 6
kern.cam.da.1.delete_max: 5497558138880
kern.cam.da.1.delete_method: UNMAP
kern.cam.da.0.error_inject: 0
kern.cam.da.0.sort_io_queue: -1
kern.cam.da.0.minimum_cmd_size: 6
kern.cam.da.0.delete_max: 131072
kern.cam.da.0.delete_method: NONE


Please let me know what you'd like to know next.

Thanks.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Some filesystem performance numbers

[Christopher Forgeron]

Sounds interesting. I'd love to see your results when you're ready to
share, or even the 'work in progress' if you want to share privately .



On Tue, Jan 13, 2015 at 6:30 PM, Garrett Wollman 
wrote:

> I recently bought a copy of the SPECsfs2014 benchmark, and I've been
> using it to test out our NFS server platform.  One scenario of
> interest to me is identifying where the limits are in terms of the
> local CAM/storage/filesystem implementation versus bottlenecks unique
> to the NFS server, and to that end I've been running the benchmark
> suite directly on the server's local disk.  (This is of course also
> the way you'd benchmark for shared-nothing container-based
> virtualization.)
>
> I have found a few interesting results on my test platform:
>
> 1) I can quantify the cost of using SHA256 vs. fletcher4 as the ZFS
> checksum algorithm.  On the VDA workload (essentially a simulated
> video streaming/recording application), my server can do about half as
> many "streams" with SHA256 as it can with fletcher4.
>
> 2) Both L2ARC and separate ZIL have small but measurable performance
> impacts.  I haven't examined the differences closely.
>
> 3) LZ4 compression also makes a small performance impact, but as
> advertised, less than LZJB for mostly-incompressible data.
>
> I hope to be able to present the actual benchmark results at some
> point, as well as some results for the other three workloads.
>
> -GAWollman
> ___
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ix(intel) vs mlxen(mellanox) 10Gb performance

[Christopher Forgeron]

FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
10.1. Before 10.1 it was less.

I used to tweak the card settings, but now it's just stock. You may want to
check your settings, the Mellanox may just have better defaults for your
switch.

On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov s...@zxy.spb.ru wrote:

 On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:

  hi,
I have a host (Dell R730) with both cards, connected to an HP8200
 switch at 10Gb.
when writing to the same storage (netapp) this is what I get:
ix0:~130MGB/s
mlxen0  ~330MGB/s
this is via nfs/tcpv3
 
I can get similar (bad) performance with the mellanox if I
 increase the file size
to 512MGB.

 Look like mellanox have internal beffer for caching and do ACK acclerating.

so at face value, it seems the mlxen does a better use of
 resources than the intel.
Any ideas how to improve ix/intel's performance?

 Are you sure about netapp performance?
 ___
 freebsd-...@freebsd.org mailing list
 https://lists.freebsd.org/mailman/listinfo/freebsd-net
 To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Why does my installworld keep breaking here?

[Christopher Forgeron]

Hi, I've been fighting with this for a while;

I do a make installworld from my newly d/led CVS source tree.  Here is the
tail-end of my output.

=== lib/libcrypt
install -c -o root -g wheel -m 444   libscrypt.a /usr/lib
install -c -s -o root -g wheel -m 444   -fschg  libscrypt.so.2 /usr/lib
/usr/libexec/ld-elf.so.1: install: Undefined symbol "strtofflags"
*** Error code 1

Stop in /usr/src/lib/libcrypt.
*** Error code 1

Stop in /usr/src/lib.
*** Error code 1

Stop in /usr/src.
*** Error code 1

 I have 4.0-RELEASE, and am trying to upgrade to 4.0-STABLE. I got the
complete src/all from CVS (RELENG4), and I have, and have not gotten all the
crypto, depending on my mood.

 I've done these before, and they worked. On this box, it's dying.

 Anything blindingly stupid that I'm forgetting?

--
  Christopher Forgeron
  Aardvark Computer Solutions, Inc.
  2070 Oxford Street, Suite 200, Halifax, Nova Scotia. B3L-2T2
  Tel: (902) 425-2686  Fax: (902) 425-1550
  [EMAIL PROTECTED]




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-stable" in the body of the message