Re: patch which implements ZFS LZ4 compression
Hi, On Sat, Feb 09, 2013 at 12:44:36PM -0800, Jeremy Chadwick wrote: > Bottom line: people enable compression on an fs, issue large amounts of > write I/O to that fs (say hundreds of megabytes, or gigabytes), and > start to see the entire system intermittently stalling hard (for > multiple seconds at a time). This affects everything from switching VTs > on physical console to packets going across SSH. The stalls vary in > duration depending on what compression type is used (lzjb vs. gzip-1 -- > I cannot even imagine what gzip-9 would be like). I described it as > verbosely as I could, including going back and "re-testing" because > people felt the "ZFSv28 import might have addressed it" (it did not): > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html > > The exact same behaviour happens if dedup is used. There is no relation > between compression (the feature) and dedup (the feature), obviously, > but the symptom I've described matches Bob's explanation perfectly. > > If you want to provide the aforementioned instructions, I'll happily > follow them. Did you try using 4BSD instead of ULE at some point? I had similar problems and that completely fixed it for me. Which would mean, that there's some interactio between scheduler and ZFS code. - Oliver -- | Oliver Brandmueller http://sysadm.in/ o...@sysadm.in | |Ich bin das Internet. Sowahr ich Gott helfe. | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: patch which implements ZFS LZ4 compression
Jeremy Chadwick wrote: > On Sat, Feb 09, 2013 at 03:19:18PM +0100, Fabian Keil wrote: > > Jeremy Chadwick wrote: > > > If you want a PR for it, I'll file one, but all it's going to contain is > > > the contents of this Email. > > > > My impression is that your emails describe symptoms and contain > > some speculation about what the cause might be. I didn't see any > > sched traces, so it's unclear (to me) that priorities are actual > > the problem. > > They contain no speculation. > > Bob Friesenhahn, who has a lot of experience and familiarity with ZFS on > Solaris, seemed to know exactly the behaviour I described. Others on > FreeBSD have reported the same behaviour as well, just not in that > thread circa 2011. Similar symptoms can have different causes. > Regarding "sched traces", please expand and include instructions. I'm referring to the stuff that is fed into: /usr/src/tools/sched/schedgraph.py It can be created with ktrace and dtrace and I believe the "documentation" is buried in the various "the scheduler sucks" threads. > > It's also unclear to me why the dedup and compression issues should > > be related. There are lots of dedup performance issues reported for > > Solaris as well and I doubt that they can be fixed for FreeBSD without > > significantly deviating from the ZFS upstream. > > What part of Bob's statement did you not understand? Here, let me > repeat it verbatim: > > "Solaris solved the problem by putting the zfs writer threads into a > special scheduling class so that they are usually lower priority than > normal processing. Before this change, a desktop system would become > almost unusable (intermittent loss of keyboard/mouse) while writing > lots of data with compression enabled. Some NFS servers encountered > severe enough issues that NFS clients reported NFS timeouts." My impression from reading zfs-discuss@ is that dedup performance and some interactivity issues actually still exist in Illumos and that they are completely unrelated to zfs writer threads. As I can't use dedup on my systems I don't really pay attention to them, though. > > I'm not saying a PR would be useless, but in my experience PRs > > with insufficient information just stay open and if the problem > > isn't important enough for you to provide additional information > > filing a PR is unlikely to have a great impact: > > http://www.freebsd.org/cgi/query-pr-summary.cgi?category=&text=zfs > > Then someone in the know needs to explain exactly *what* data would help > and (more importantly) *how* to go about providing it (i.e. what to > enable in the kernel, what commands to issue, etc.). Eidan has > repeatedly insisted that PRs are a Good Thing(tm) because they allow for > an official way to track issues vs. mailing list threads that start and > turn into tumbleweeds (just like the one I've referenced). And how many of those PRs are actually solved? This is a rhetoric question and I don't expect you to look it up. I'm not saying that PRs are a bad thing, but filing PRs is the easy part and in my experience issues that don't spark developer interest on the mailing lists are usually also ignored when filed as PR, especially when they don't contain 100% of the information that may be relevant. Even if you provide "proof" that the priorities are indeed the cause of the problem there's a fair chance that the PR gets ignored anyway. I currently have four somewhat ZFS-related PRs open, the first was filed in 2007. I still don't think that the solution is that nobody works on ZFS improvements until my PRs are solved. I'm looking forward to using LZ4 which promises better compression than lzjb with less interactivity impact than gzip. It might even work for your /dev/random test as it's supposed to better deal with poorly compressible data. > Without those necessary instructions, in effect what you're asking me to > do is "prove" that the problem exists, which I have already done so. > You just don't like the data I've provided. I don't expect you to "prove" that the problem exists. My impression is that the interactivity issues with gzip have been well known for years and exist since the ZFS import. I also don't dislike your data, all I'm saying is that there could be other explanations. > Bottom line: people enable compression on an fs, issue large amounts of > write I/O to that fs (say hundreds of megabytes, or gigabytes), and > start to see the entire system intermittently stalling hard (for > multiple seconds at a time). This affects everything from switching VTs > on physical console to packets going across SSH. The stalls vary in > duration depending on what compression type is used (lzjb vs. gzip-1 -- > I cannot even imagine what gzip-9 would be like). I described it as > verbosely as I could, including going back and "re-testing" because > people felt the "ZFSv28 import might have addressed it" (it did not): > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-Octo
Re: patch which implements ZFS LZ4 compression
On Sat, Feb 09, 2013 at 03:19:18PM +0100, Fabian Keil wrote: > Jeremy Chadwick wrote: > > > On Fri, Feb 08, 2013 at 02:52:57PM -0800, Xin Li wrote: > > > On 02/08/13 14:29, Dan Langille wrote: > > > > Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 > > > > compression. > > > > > > > > https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM > > > > > > > > short link: http://bpaste.net/show/76095 > > > > > > Please DO NOT use this patch! It will ruin your data silently. > > > > > > As I already posted on Ivan's Google+ post, I'm doing final universe > > > builds to make sure that there is no regression and will merge my > > > changes to -HEAD later today. > > > > Another compression algorithm, this time 50%+ faster than lzjb. Great, > > fine, wonderful, awesome, kudos, huzzah, blah blah blah. > > > > So when is someone going to step up to the plate and fix how compression > > (as well as dedup) destroys interactivity on FreeBSD? Do I need to > > remind folks of this issue once again? Here you have it, dated October > > 2011, including the root cause and how it was fixed in Solaris et al: > > > > Description: > > > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012718.html > > > > Explanation and how Solaris et al fixed it, and how on Solaris the > > problem was major enough that it even caused NFS timeouts (sound > > familiar to anyone?): > > > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012726.html > > > > Further testing showing gzip-1 vs. lzjb and interactivity stalls: > > > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html > > > > This is still a problem with base/stable/9. And as I have said > > elsewhere on lists, do not ask me to run CURRENT -- it will be a cold > > day in hell before I ever do that. I assume this same problem exists in > > CURRENT unless I have some key developer/committer say "I backported > > this fix in CURRENT, absolutely 100% sure". > > > > I'm also wondering why iXSystems hasn't stepped up to the plate to > > contribute to making this happen, given their business focus. I do not > > have the knowledge of the kernel (or of threading) to fix this myself, > > and for that I do apologise. > > > > But every time I see compression or dedup mentioned, I use the > > opportunity to bring up this subject. STOP ADDING FEATURES AND FIX > > STUFF LIKE THIS INSTEAD -- while new algorithms are neat/fun toys, they > > do not truly fix issues like this. How this problem has continually > > gotten overlooked is beyond me. > > Did you consider that other people may have different priorities > than you do? I see what you did there. > > If you want a PR for it, I'll file one, but all it's going to contain is > > the contents of this Email. > > My impression is that your emails describe symptoms and contain > some speculation about what the cause might be. I didn't see any > sched traces, so it's unclear (to me) that priorities are actual > the problem. They contain no speculation. Bob Friesenhahn, who has a lot of experience and familiarity with ZFS on Solaris, seemed to know exactly the behaviour I described. Others on FreeBSD have reported the same behaviour as well, just not in that thread circa 2011. Regarding "sched traces", please expand and include instructions. > It's also unclear to me why the dedup and compression issues should > be related. There are lots of dedup performance issues reported for > Solaris as well and I doubt that they can be fixed for FreeBSD without > significantly deviating from the ZFS upstream. What part of Bob's statement did you not understand? Here, let me repeat it verbatim: "Solaris solved the problem by putting the zfs writer threads into a special scheduling class so that they are usually lower priority than normal processing. Before this change, a desktop system would become almost unusable (intermittent loss of keyboard/mouse) while writing lots of data with compression enabled. Some NFS servers encountered severe enough issues that NFS clients reported NFS timeouts." > I'm not saying a PR would be useless, but in my experience PRs > with insufficient information just stay open and if the problem > isn't important enough for you to provide additional information > filing a PR is unlikely to have a great impact: > http://www.freebsd.org/cgi/query-pr-summary.cgi?category=&text=zfs Then someone in the know needs to explain exactly *what* data would help and (more importantly) *how* to go about providing it (i.e. what to enable in the kernel, what commands to issue, etc.). Eidan has repeatedly insisted that PRs are a Good Thing(tm) because they allow for an official way to track issues vs. mailing list threads that start and turn into tumbleweeds (just like the one I've referenced). Without those necessary instructions, in effect what you're asking me to do is "prove" that the problem exists, which I have already done so. You just d
Re: patch which implements ZFS LZ4 compression
Jeremy Chadwick wrote: > On Fri, Feb 08, 2013 at 02:52:57PM -0800, Xin Li wrote: > > On 02/08/13 14:29, Dan Langille wrote: > > > Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 > > > compression. > > > > > > https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM > > > > > > short link: http://bpaste.net/show/76095 > > > > Please DO NOT use this patch! It will ruin your data silently. > > > > As I already posted on Ivan's Google+ post, I'm doing final universe > > builds to make sure that there is no regression and will merge my > > changes to -HEAD later today. > > Another compression algorithm, this time 50%+ faster than lzjb. Great, > fine, wonderful, awesome, kudos, huzzah, blah blah blah. > > So when is someone going to step up to the plate and fix how compression > (as well as dedup) destroys interactivity on FreeBSD? Do I need to > remind folks of this issue once again? Here you have it, dated October > 2011, including the root cause and how it was fixed in Solaris et al: > > Description: > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012718.html > > Explanation and how Solaris et al fixed it, and how on Solaris the > problem was major enough that it even caused NFS timeouts (sound > familiar to anyone?): > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012726.html > > Further testing showing gzip-1 vs. lzjb and interactivity stalls: > > http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html > > This is still a problem with base/stable/9. And as I have said > elsewhere on lists, do not ask me to run CURRENT -- it will be a cold > day in hell before I ever do that. I assume this same problem exists in > CURRENT unless I have some key developer/committer say "I backported > this fix in CURRENT, absolutely 100% sure". > > I'm also wondering why iXSystems hasn't stepped up to the plate to > contribute to making this happen, given their business focus. I do not > have the knowledge of the kernel (or of threading) to fix this myself, > and for that I do apologise. > > But every time I see compression or dedup mentioned, I use the > opportunity to bring up this subject. STOP ADDING FEATURES AND FIX > STUFF LIKE THIS INSTEAD -- while new algorithms are neat/fun toys, they > do not truly fix issues like this. How this problem has continually > gotten overlooked is beyond me. Did you consider that other people may have different priorities than you do? > If you want a PR for it, I'll file one, but all it's going to contain is > the contents of this Email. My impression is that your emails describe symptoms and contain some speculation about what the cause might be. I didn't see any sched traces, so it's unclear (to me) that priorities are actual the problem. It's also unclear to me why the dedup and compression issues should be related. There are lots of dedup performance issues reported for Solaris as well and I doubt that they can be fixed for FreeBSD without significantly deviating from the ZFS upstream. I'm not saying a PR would be useless, but in my experience PRs with insufficient information just stay open and if the problem isn't important enough for you to provide additional information filing a PR is unlikely to have a great impact: http://www.freebsd.org/cgi/query-pr-summary.cgi?category=&text=zfs Fabian signature.asc Description: PGP signature
Re: patch which implements ZFS LZ4 compression
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 For those who are inpatient, here is an overlay for LZ4 for 9-STABLE. http://people.freebsd.org/~delphij/misc/lz4.tar.xz Overwrite /usr/src and recompile ZFS module. Note: before using this in any serious environment, please be sure to test if there is any data corruption by comparing the files copied to a LZ4 dataset after export and re-import the pool, or reboot. I have only tested the code on amd64 and would like to see if it works on big endian environment, etc. Cheers, -BEGIN PGP SIGNATURE- iQEcBAEBCAAGBQJRFfwgAAoJEG80Jeu8UPuzjKgH/0i+PShTX3kyLvEjORHjmEp+ T0YxLiTEssmpz8IhUwbjN+FLoQZlV+DCL5MUBzIsA5kM3p1aUgffrNnO1RvZmbhT y+pnKxEDay30QhMs48jWLO6oyHA07bb8q5xyut+noK6X1y4epqwrWid44nqlzPZQ w6XKhCuz8DXm4TIKQpeB4ncFNfUv+rlImmn5XToC6DaKn/2XtYVLVE73pjP3qANL mxZA/RqJNQy1cBp0gougDcw6AoNMBkjfxaNe0V073nEffyXgI/1BeCWsxKmWFNWb VNGGjxLzmpP4MGz2SAABfaUvRb306pHJtfYWtoeIxpZLM0A9h9wnDGUWo9w+lLo= =eVP8 -END PGP SIGNATURE- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: patch which implements ZFS LZ4 compression
On Fri, Feb 08, 2013 at 02:52:57PM -0800, Xin Li wrote: > On 02/08/13 14:29, Dan Langille wrote: > > Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 > > compression. > > > > https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM > > > > short link: http://bpaste.net/show/76095 > > Please DO NOT use this patch! It will ruin your data silently. > > As I already posted on Ivan's Google+ post, I'm doing final universe > builds to make sure that there is no regression and will merge my > changes to -HEAD later today. Another compression algorithm, this time 50%+ faster than lzjb. Great, fine, wonderful, awesome, kudos, huzzah, blah blah blah. So when is someone going to step up to the plate and fix how compression (as well as dedup) destroys interactivity on FreeBSD? Do I need to remind folks of this issue once again? Here you have it, dated October 2011, including the root cause and how it was fixed in Solaris et al: Description: http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012718.html Explanation and how Solaris et al fixed it, and how on Solaris the problem was major enough that it even caused NFS timeouts (sound familiar to anyone?): http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012726.html Further testing showing gzip-1 vs. lzjb and interactivity stalls: http://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012752.html This is still a problem with base/stable/9. And as I have said elsewhere on lists, do not ask me to run CURRENT -- it will be a cold day in hell before I ever do that. I assume this same problem exists in CURRENT unless I have some key developer/committer say "I backported this fix in CURRENT, absolutely 100% sure". I'm also wondering why iXSystems hasn't stepped up to the plate to contribute to making this happen, given their business focus. I do not have the knowledge of the kernel (or of threading) to fix this myself, and for that I do apologise. But every time I see compression or dedup mentioned, I use the opportunity to bring up this subject. STOP ADDING FEATURES AND FIX STUFF LIKE THIS INSTEAD -- while new algorithms are neat/fun toys, they do not truly fix issues like this. How this problem has continually gotten overlooked is beyond me. If you want a PR for it, I'll file one, but all it's going to contain is the contents of this Email. -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Mountain View, CA, US| | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: patch which implements ZFS LZ4 compression
On Feb 8, 2013, at 5:52 PM, Xin Li wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 02/08/13 14:29, Dan Langille wrote: >> Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 >> compression. >> >> https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM >> >> short link: http://bpaste.net/show/76095 > > Please DO NOT use this patch! It will ruin your data silently. I'm sorry, I will remove the post. > > As I already posted on Ivan's Google+ post, I'm doing final universe > builds to make sure that there is no regression and will merge my > changes to -HEAD later today. My apologies. :) Thank you. -- Dan Langille - http://langille.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: patch which implements ZFS LZ4 compression
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 02/08/13 14:29, Dan Langille wrote: > Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 > compression. > > https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM > > short link: http://bpaste.net/show/76095 Please DO NOT use this patch! It will ruin your data silently. As I already posted on Ivan's Google+ post, I'm doing final universe builds to make sure that there is no regression and will merge my changes to -HEAD later today. Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -BEGIN PGP SIGNATURE- iQEcBAEBCgAGBQJRFYHJAAoJEG80Jeu8UPuz3e8IALAcbjZo063casLry/zyBjLe jbat7AmmWTQ5ndBtfx5PxAV1of6ePCiQQS/RQVd001c5M32hzfoqb7524th3giyx q4RRuVPIc6YswJOe3Gz/vRyHdBRfJlownxZy1Glv480qK9dzeUtd2ziZhoi+FZ1X PfigRwg/9OtyDbmUhEqDvq2qeyAakuiE6yL9NVlGGKSe8GysYWPQFisVfDWn3WYH a+Rt+O3P/QHK0mTWBYTs/r92AU9J0Sv4plhNNQZSZWpAlQLuWRhMit/2e2W/0TPx XnXXuzdr01/cE/+ou3MMnq+uvO1FiqKBUXN6/oJzZnl/VkvvL795iT/C5KGq+2s= =mVAM -END PGP SIGNATURE- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
patch which implements ZFS LZ4 compression
Here is a patch against FreeBSD 9.1 STABLE which implements ZFS LZ4 compression. https://plus.google.com/106386350930626759085/posts/PLbkNfndPiM short link: http://bpaste.net/show/76095 HTH -- Dan Langille - http://langille.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
