Re: ZFS secondarycache on SSD problem on r255173

2014-06-06 Thread Matthew D. Fuller
On Fri, Apr 25, 2014 at 08:26:52PM -0500 I heard the voice of Matthew D. Fuller, and lo! it spake thus: On Mon, Mar 03, 2014 at 11:17:05AM +0200 I heard the voice of Andriy Gapon, and lo! it spake thus: I noticed that on some of our systems we were getting a clearly abnormal number of

Re: ZFS secondarycache on SSD problem on r255173

2014-04-25 Thread Matthew D. Fuller
On Mon, Mar 03, 2014 at 11:17:05AM +0200 I heard the voice of Andriy Gapon, and lo! it spake thus: I noticed that on some of our systems we were getting a clearly abnormal number of l2arc checksum errors accounted in l2_cksum_bad. [...] I propose the following patch which has been tested and

Re: ZFS secondarycache on SSD problem on r255173

2014-03-11 Thread Matthew D. Fuller
On Mon, Mar 03, 2014 at 11:17:05AM +0200 I heard the voice of Andriy Gapon, and lo! it spake thus: I noticed that on some of our systems we were getting a clearly abnormal number of l2arc checksum errors accounted in l2_cksum_bad. The hardware appeared to be in good health. FWIW, I have a

Re: ZFS secondarycache on SSD problem on r255173

2014-03-03 Thread Andriy Gapon
on 18/10/2013 17:57 Steven Hartland said the following: I think we we may well need the following patch to set the minblock size based on the vdev ashift and not SPA_MINBLOCKSIZE. svn diff -x -p sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Index:

Re: ZFS secondarycache on SSD problem on r255173

2014-03-03 Thread Steven Hartland
- Original Message - From: Andriy Gapon a...@freebsd.org on 18/10/2013 17:57 Steven Hartland said the following: I think we we may well need the following patch to set the minblock size based on the vdev ashift and not SPA_MINBLOCKSIZE. svn diff -x -p

Re: ZFS secondarycache on SSD problem on r255173

2013-10-22 Thread Vitalij Satanivskij
Ок, just up to now no error on l2arc L2 ARC Summary: (HEALTHY) Passed Headroom:1.99m Tried Lock Failures:144.53m IO In Progress: 130.15k Low Memory Aborts: 7 Free on

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Steven Hartland
Hows things looking Vitalij? - Original Message - From: Vitalij Satanivskij sa...@ukr.net Ok. Just right now system rebooted with you patch. Trim enabled again. WIll wait some time untile size of used cache grow's. Steven Hartland wrote: SH Looking at the l2arc compression code

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Vitalij Satanivskij
Just now I cannot say, as to triger problem we need at last 200+gb size on l2arc wich usually grow in one production day. But for some reason today in the morning server was rebooted so cache was flushed and now only 100Gb. Need to wait some more time. At last for now none error on l2.

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Steven Hartland
So previously you only started seeing l2 errors after there was a significant amount of data in l2arc? Thats interesting in itself if thats the case. I wonder if its the type of data, or something similar. Do you run compression on any of your volumes? zfs get compression Regards Steve

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Vitalij Satanivskij
Steven Hartland wrote: SH So previously you only started seeing l2 errors after there was SH a significant amount of data in l2arc? Thats interesting in itself SH if thats the case. Yes someting arround 200+gb SH I wonder if its the type of data, or something similar. Do you SH run compression

Re: ZFS secondarycache on SSD problem on r255173

2013-10-19 Thread Vitalij Satanivskij
suppor...@ukr.net; Justin T. Gibbs gi...@freebsd.org; Borja SH Marcos bor...@sarenet.es; freebsd-current@freebsd.org SH Sent: Friday, October 18, 2013 3:45 PM SH Subject: Re: ZFS secondarycache on SSD problem on r255173 SH SH SH SH Just right now stats not to actual because of some another test

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Vitalij Satanivskij
Hello. Yesterday system was rebooted with vfs.zfs.trim.enabled=0 System version 10.0-BETA1 FreeBSD 10.0-BETA1 #6 r256669, without any changes in code Uptime 10:51 up 16:41 sysctl vfs.zfs.trim.enabled vfs.zfs.trim.enabled: 0 Around 2 hours ago errors counter's

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Steven Hartland
Hartland kill...@multiplay.co.uk Cc: sa...@ukr.net; Justin T. Gibbs gi...@freebsd.org; freebsd-current@freebsd.org; Borja Marcos bor...@sarenet.es; Dmitriy Makarov suppor...@ukr.net Sent: Friday, October 18, 2013 9:01 AM Subject: Re: ZFS secondarycache on SSD problem on r255173 Hello. Yesterday

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Vitalij Satanivskij
Subject: Re: ZFS secondarycache on SSD problem on r255173 SH SH SH Hello. SH SH Yesterday system was rebooted with vfs.zfs.trim.enabled=0 SH SH System version 10.0-BETA1 FreeBSD 10.0-BETA1 #6 r256669, without any changes in code SH SH Uptime 10:51 up 16:41 SH SH sysctl vfs.zfs.trim.enabled

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Steven Hartland
Satanivskij sa...@ukr.net; Dmitriy Makarov suppor...@ukr.net; Justin T. Gibbs gi...@freebsd.org; Borja Marcos bor...@sarenet.es; freebsd-current@freebsd.org Sent: Friday, October 18, 2013 3:45 PM Subject: Re: ZFS secondarycache on SSD problem on r255173 Just right now stats not to actual because

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Vitalij Satanivskij
Hello. Problem description is in - http://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html As we find later first begin's problem with errors counter in arcstats than size of l2 grows abnormal. After patch rollback everything is ok. Justin T. Gibbs wrote: JTG You'll

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Vitalij Satanivskij
Hello. SSD is Intel SSD 530 series (INTEL SSDSC2BW180A4 DC12) Controler is onboard intel sata controler, motherboard is Supermicro X9SRL-F so it's Intel C602 chipset All cache ssd connected to sata 2 ports. System has LSI MPS controler (SAS2308) with firmware version - 16.00.00.00, but only

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Steven Hartland
...@multiplay.co.uk Cc: Justin T. Gibbs gi...@freebsd.org; Vitalij Satanivskij sa...@ukr.net; freebsd-current@freebsd.org; Borja Marcos bor...@sarenet.es; Dmitriy Makarov suppor...@ukr.net Sent: Thursday, October 17, 2013 7:12 AM Subject: Re: ZFS secondarycache on SSD problem on r255173 Hello. SSD

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Vitalij Satanivskij
secondarycache on SSD problem on r255173 SH SH SH Hello. SH SH SSD is Intel SSD 530 series (INTEL SSDSC2BW180A4 DC12) SH SH Controler is onboard intel sata controler, motherboard is Supermicro X9SRL-F so it's Intel C602 chipset SH SH All cache ssd connected to sata 2 ports. SH SH System has LSI MPS

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Steven Hartland
Correct. - Original Message - From: Vitalij Satanivskij sa...@ukr.net Just to be sure I understand you clearly, I need to test next configuration: 1) System with ashift patch eg. just latest stable/10 revision 2) vfs.zfs.trim.enabled=0 in /boot/loader.conf So realy only diferens

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Hello. Patch brocke cache functionality. Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300 With subject ZFS L2ARC - incorrect size and abnormal system load on r255173 As patch alredy in head and BETA it's not good. Yesterday we update one machine up to beta1 and forgot about

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
: Vitalij Satanivskij sa...@ukr.net To: Dmitriy Makarov suppor...@ukr.net Cc: Steven Hartland kill...@multiplay.co.uk; Justin T. Gibbs gi...@freebsd.org; Borja Marcos bor...@sarenet.es; freebsd-current@freebsd.org Sent: Wednesday, October 16, 2013 9:01 AM Subject: Re: ZFS secondarycache on SSD problem

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Marcos bor...@sarenet.es; SH freebsd-current@freebsd.org SH Sent: Wednesday, October 16, 2013 9:01 AM SH Subject: Re: ZFS secondarycache on SSD problem on r255173 SH SH SH Hello. SH SH Patch brocke cache functionality. SH SH Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300 SH

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
-current@freebsd.org Sent: Wednesday, October 16, 2013 3:10 PM Subject: Re: ZFS secondarycache on SSD problem on r255173 Yes We have 15 servers, all of them have problem while using with patch fo ashift, sh we rollback path (for r255173) and all of them works for a week without that's problem's

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Steven Hartland wrote: SH I'm not clear what you rolled back there as r255173 has ntothing to do SH with this. Could you clarify r255173 with you patch from email dated Tue, 17 Sep 2013 23:53:12 +0100 with subject Re: ZFS secondarycache on SSD problem on r255173 Errors wich we gets

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Justin T. Gibbs
You'll have to be more specific. I don't have that email or know what list on which to search. Thanks, Justin On Oct 16, 2013, at 2:01 AM, Vitalij Satanivskij sa...@ukr.net wrote: Hello. Patch brocke cache functionality. Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Justin T. Gibbs
I took a quick look at arc.c and see that the trim_map_free() calls don't take into account ashift. I don't know if that has anything to do with your problem though. I would expect this would just make the trim less efficient, but I need to dig further. -- Justin On Oct 16, 2013, at 4:42

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
- Original Message - From: Justin T. Gibbs gi...@freebsd.org I took a quick look at arc.c and see that the trim_map_free() calls don't take into account ashift. I don't know if that has anything to do with your problem though. I would expect this would just make the trim less

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
Ohh stupid question what hardware are you running this on, specifically what SSD's and what controller and if relavent what controller Firmware version? I wonder if you might have bad HW / FW, such as older LSI mps Firmware, which is know to causing corruption with some delete methods. Without

Re: ZFS secondarycache on SSD problem on r255173

2013-09-20 Thread Justin T. Gibbs
On Sep 17, 2013, at 4:53 PM, Steven Hartland kill...@multiplay.co.uk wrote: - Original Message - From: Justin T. Gibbs gi...@freebsd.org Sorry for being slow to chime in on this thread. I live in Boulder, CO and we've had a bit of rain. :-) Hope all is well your side, everyone

Re: ZFS secondarycache on SSD problem on r255173

2013-09-17 Thread Steven Hartland
- Original Message - From: Justin T. Gibbs gi...@freebsd.org Sorry for being slow to chime in on this thread. I live in Boulder, CO and we've had a bit of rain. :-) Hope all is well your side, everyone safe and sound if may be little wetter than usual. As Steven pointed out,

Re: ZFS secondarycache on SSD problem on r255173

2013-09-16 Thread Borja Marcos
On Sep 13, 2013, at 2:18 PM, Steven Hartland wrote: This is a recent bit of code by Justin cc'ed, so he's likely the best person to investigate this one. Hmm. There is still a lot of confusion surrounding all this, and it's a time bomb waiting to explode. A friend had serious problems on

Re: ZFS secondarycache on SSD problem on r255173

2013-09-16 Thread Steven Hartland
: Dmitryy Makarov suppor...@ukr.net; freebsd-current@freebsd.org; Justin T. Gibbs gi...@freebsd.org Sent: Monday, September 16, 2013 12:06 PM Subject: Re: ZFS secondarycache on SSD problem on r255173 On Sep 13, 2013, at 2:18 PM, Steven Hartland wrote: This is a recent bit of code by Justin cc'ed, so

Re: ZFS secondarycache on SSD problem on r255173

2013-09-16 Thread Justin T. Gibbs
Sorry for being slow to chime in on this thread. I live in Boulder, CO and we've had a bit of rain. :-) As Steven pointed out, the warning is benign, but does show that the code I committed to -current is not optimizing the allocation size for L2ARC devices. The fix for this is to find the

Re: ZFS secondarycache on SSD problem on r255173

2013-09-13 Thread Steven Hartland
This is a recent bit of code by Justin cc'ed, so he's likely the best person to investigate this one. Regards Steve - Original Message - From: Dmitryy Makarov suppor...@ukr.net To: freebsd-current@freebsd.org Sent: Friday, September 13, 2013 12:16 PM Subject: ZFS secondarycache