Re: some general zfs tuning (for iSCSI)

[Eugene M. Zheganin]

Hi.

On 02.08.2017 17:43, Ronald Klop wrote:
On Fri, 28 Jul 2017 12:56:11 +0200, Eugene M. Zheganin 
 wrote:



Hi,


I'm using several FreeBSD zfs installations as the iSCSI production 
systems, they basically consist of an LSI HBA, and a JBOD with a 
bunch of SSD disks (12-24, Intel, Toshiba or Sandisk (avoid Sandisks 
btw)). And I observe a problem very often: gstat shows 20-30% of disk 
load, but the system reacts very slowly: cloning a dataset takes 10 
seconds, similar operations aren't lightspeeding too. To my 
knowledge, until the disks are 90-100% busy, this shouldn't happen. 
My systems are equipped with 32-64 gigs of RAM, and the only tuning I 
use is limiting the ARC size (in a very tender manner - at least to 
16 gigs) and playing with TRIM. The number of datasets is high enough 
- hundreds of clones, dozens of snapshots, most of teh data ovjects 
are zvols. Pools aren't overfilled, most are filled up to 60-70% (no 
questions about low space pools, but even in this case the situation 
is clearer - %busy goes up in the sky).


So, my question is - is there some obvious zfs tuning not mentioned 
in the Handbook ? On the other side - handbook isn't much clear on 
how to tune zfs, it's written mostly in the manner of "these are 
sysctl iods you can play with". Of course I have seen several ZFS 
tuning guides. Like Opensolaris one, but they are mostly file- and 
application-specific. Is there some special approach to tune ZFS in 
the environment with loads of disks ? I don't know like tuning 
the vdev cache or something simllar. ?





What version of FreeBSD are you running?

Well, different ones. Mostly some versions of 11.0-RELEASE-pX and 11-STABLE.


What is the system doing during all this?
What do you meant by "what" ? Nothing else except serving iSCSI - it's 
the main purpose of every one of these servers.



How are your pools setup (raidz1/2/3, mirror, 3mirror)?
zroot is a mirrored two-disk pool, others are raidz, mostly spans of 
multiple 5-disk radizs.



How is your iSCSI configured and what are the clients doing with it?
Using the kernel ctld of course. As you may know ctl.conf dosn't suppose 
any performance tweaks, it's just a way of organizing the authorization 
layer. Clients are the VMWare ESX hypervisors, using iSCSI as disk 
devices, as for ESX SRs, and as direct iSCSI disks in Windows VMs.



Is the data distributed evenly on all disks?

It's not. Does it ever ditrubute evenly anywhere ?


Do the clients write a lot of sync data?

What do you exactly mean by "sync data" ?

Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

panic: dva_get_dsize_sync(): bad DVA on 2016 11-STABLE.

[Eugene M. Zheganin]

Hi,


today I got the following panic on the December 2016 11-STABLE:


FreeBSD san02.playkey.net 11.0-STABLE FreeBSD 11.0-STABLE #0 r310734M: 
Thu Dec 29 19:22:30 UTC 2016 emz@san02:/usr/obj/usr/src/sys/GENERIC  amd64


panic: dva_get_dsize_sync(): bad DVA 4294967295:2086400

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.

Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: dva_get_dsize_sync(): bad DVA 4294967295:2086400
cpuid = 2
KDB: stack backtrace:
#0 0x80b023a7 at kdb_backtrace+0x67
#1 0x80ab88e6 at vpanic+0x186
#2 0x80ab8753 at panic+0x43
#3 0x8226a148 at bp_get_dsize+0x128
#4 0x8222cc29 at dmu_tx_count_write+0x589
#5 0x8222c675 at dmu_tx_hold_write+0x35
#6 0x822c573e at zvol_strategy+0x21e
#7 0x809f6a10 at g_io_request+0x4a0
#8 0x8283a45f at ctl_be_block_dispatch_dev+0x20f
#9 0x8283bddc at ctl_be_block_worker+0x6c
#10 0x80b1484a at taskqueue_run_locked+0x14a
#11 0x80b15a38 at taskqueue_thread_loop+0xe8
#12 0x80a70785 at fork_exit+0x85
#13 0x80f55f2e at fork_trampoline+0xe
Uptime: 78d7h43m31s


My question is (since I din't find much on this) what does this "bad 
DVA" mean ? I've read that this may indicate the on-disk zfs corrupton, 
but I'm not suer about it. Is this fixable in any way ? Do I have to 
prepare to recreate the pool (btw I have three pools) from scratch, and 
how do I determine which one has the corruption.



Some [useless ?] zfs info:


# zpool status
  pool: data
 state: ONLINE
  scan: none requested
config:

NAMESTATE READ WRITE CKSUM
dataONLINE   0 0 0
  raidz1-0  ONLINE   0 0 0
da2 ONLINE   0 0 0
da3 ONLINE   0 0 0
da4 ONLINE   0 0 0
da5 ONLINE   0 0 0
  raidz1-1  ONLINE   0 0 0
da6 ONLINE   0 0 0
da7 ONLINE   0 0 0
da8 ONLINE   0 0 0
da9 ONLINE   0 0 0
  raidz1-2  ONLINE   0 0 0
da10ONLINE   0 0 0
da11ONLINE   0 0 0
da12ONLINE   0 0 0
da13ONLINE   0 0 0
  raidz1-3  ONLINE   0 0 0
da14ONLINE   0 0 0
da15ONLINE   0 0 0
da16ONLINE   0 0 0
da17ONLINE   0 0 0

errors: No known data errors

  pool: userdata
 state: ONLINE
  scan: none requested
config:

NAME   STATE READ WRITE CKSUM
userdata   ONLINE   0 0 0
  mirror-0 ONLINE   0 0 0
gpt/userdata0  ONLINE   0 0 0
gpt/userdata1  ONLINE   0 0 0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: none requested
config:

NAMESTATE READ WRITE CKSUM
zroot   ONLINE   0 0 0
  mirror-0  ONLINE   0 0 0
gpt/zroot0  ONLINE   0 0 0
gpt/zroot1  ONLINE   0 0 0

errors: No known data errors


Hardware:


# camcontrol devlist
   at scbus4 target 0 lun 0 (pass0,ses0)
   at scbus11 target 0 lun 0 (pass1,ses1)
at scbus12 target 4 lun 0 (pass2,da0)
at scbus12 target 5 lun 0 (pass3,da1)
at scbus12 target 8 lun 0 (pass4,da2)
at scbus12 target 9 lun 0 (pass5,da3)
at scbus12 target 10 lun 0 (pass6,da4)
at scbus12 target 11 lun 0 (pass7,da5)
at scbus12 target 12 lun 0 (pass8,da6)
at scbus12 target 13 lun 0 (pass9,da7)
at scbus12 target 14 lun 0 (pass10,da8)
at scbus12 target 15 lun 0 (pass11,da9)
at scbus12 target 16 lun 0 (pass12,da10)
at scbus12 target 17 lun 0 (pass13,da11)
at scbus12 target 18 lun 0 (pass14,da12)
at scbus12 target 19 lun 0 (pass15,da13)
at scbus12 target 20 lun 0 (pass16,da14)
at scbus12 target 21 lun 0 (pass17,da15)
at scbus12 target 22 lun 0 (pass18,da16)
at scbus12 target 23 lun 0 (pass19,da17)
at scbus12 target 24 lun 0 (pass20,da18)
 at scbus12 target 32 lun 0 (pass21,ses2)


I also have the zdb -uuumdC for each pool, here they are in case someone 
needs them:



https://enaza.ru/stub-data/zdb-uuumdC-data.txt

https://enaza.ru/stub-data/zdb-uuumdC-userdata.txt

[Bug 221146] [LAGG] Problem with second laggport

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221146

--- Comment #7 from Eric Joyner  ---
Do we know who to assign this to, then? If it is actually a lagg problem and
not a driver-specific one.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

[Bug 221146] [LAGG] Problem with second laggport

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221146

--- Comment #8 from Cassiano Peixoto  ---
(In reply to Eric Joyner from comment #7)
So should i open a new PR related to driver update problem and netmap?

-- 
You are receiving this mail because:
You are on the CC list for the bug.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"