Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On Mon, Dec 30, 2019 at 02:55:16PM -0500, Alexander Motin wrote: > On 30.12.2019 12:02, Alexey Dokuchaev wrote: > > ... > > Well, hard drives essentially didn't change since then, still being > > the same rotation media. :) > > At least some papers about gsched I read mention adX devices, which > means old ATA stack and no NCQ. It can be quite a significant change to > let HDD to do its own scheduling. Also about a year ago in r335066 > Warner added sysctl debug.bioq_batchsize, which if set to non-zero value > may, I think, improve fairness between several processes, just not sure > why it was never enabled. Thanks for the pointer, I'll remember to play with it. > > Admittedly, I've only did some tests no later than in 8.4 times when I > > first started using it. Fair point, though, I should redo them again. > > I'm sorry to create a regression for you, if there is really one. As I > have written I don't have so much against the scheduler part itself, as > against the accumulated technical debt and the way integration is done, > such as mechanism of live insertion, etc. Without unmapped I/O and > direct dispatch I bet it must be quite slow on bigger systems, that is > why I doubted anybody really use it. It's OK, no need to be sorry, I understand the reasons behind the move and it certainly looks justified enough. > > Is there a planned replacement, or I'd better create a port for the > > GEOM_SCHED class and gsched(8) tool? > > I wasn't planning replacement. And moving it to ports would be a > problem, since in process I removed few capabilities critical for it: > nstart/nend for live insertion and BIO classification for scheduling. > But the last I don't mind to return if there appear to be a need. It > is only the first I am strongly against. But if somebody would like to > reimplement it, may be it would be better to consider merging it with > CAM I/O scheduler by Warner? The one at least knows about device queue > depth, etc. We could return the BIO classification to be used by CAM > scheduler instead, if needed. Make sense; thanks everyone who replied in the thread, it was quite fruitful. ./danfe ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
In message , Warner Losh writes: >I never enabled it because I never had a good car size as the default. I'm >guessing it's somewhere on the order of 2 times the queue size in >hardware, but with modern drives I think phk might be right and that >disabling disksort entirely might be optimal, or close to optimal. I think that is a given for SSDs. For disks I fear it would be a model-by-model determination. The situation is quite different for "traditional" and shingled drives for instance, or if, God forbid, the rumours are true and we'll see IBM3380-style dual head assemblies in the market again. I guess the kernel could turn the disksort on/off a few times and only leave it on if it improves things. But first, Somebody Should™ benchmark to see if disksort *ever* is an improvement on contemporary disks. Poul-Henning PS: If somebody really want to improve disk- and ssd- performance, the low-hanging fruit is to write a log-structured storage engine under UFS, to make life easier for flash-adaptation layers and shingling drives. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On Mon, 30 Dec 2019, Warner Losh wrote: On Mon, Dec 30, 2019 at 12:55 PM Alexander Motin wrote: On 30.12.2019 12:02, Alexey Dokuchaev wrote: > On Mon, Dec 30, 2019 at 08:55:14AM -0700, Warner Losh wrote: >> On Mon, Dec 30, 2019, 5:32 AM Alexey Dokuchaev wrote: >>> On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: New Revision: 356185 URL: https://svnweb.freebsd.org/changeset/base/356185 Log: ? ?Remove GEOM_SCHED class and gsched tool. ? ?[...] >>> >>> Wow, that was unexpected, I use it on all my machines' HDD drives. >>> Is there a planned replacement, or I'd better create a port for the >>> GEOM_SCHED class and gsched(8) tool? >> >> How much of a performance improvement do you see with it? >> >> There has been no tweaks to this geom in years and years. It was tuned >> to 10 year old hard drives and never retuned for anything newer. > > Well, hard drives essentially didn't change since then, still being the > same roration media. :) At least some papers about gsched I read mention adX devices, which means old ATA stack and no NCQ.? It can be quite a significant change to let HDD to do its own scheduling.? Also about a year ago in r335066 Warner added sysctl debug.bioq_batchsize, which if set to non-zero value may, I think, improve fairness between several processes, just not sure why it was never enabled. I never?enabled it because I never had a good?car size as the default. I'm guessing? it's somewhere?on the order of 2 times the queue size in hardware, but with modern drives I think phk might be right and that disabling disksort entirely might be optimal, or close to optimal. ? >> And when I played with it a few years ago, I saw no improvements... > > Admittedly, I've only did some tests no later than in 8.4 times when I > first started using it.? Fair point, though, I should redo them again. I'm sorry to create a regression for you, if there is really one.? As I have written I don't have so much against the scheduler part itself, as against the accumulated technical debt and the way integration is done, such as mechanism of live insertion, etc.? Without unmapped I/O and direct dispatch I bet it must be quite slow on bigger systems, that is why I doubted anybody really use it. > Is there a planned replacement, or I'd better create a port for the > GEOM_SCHED class and gsched(8) tool? I wasn't planning replacement.? And moving it to ports would be a problem, since in process I removed few capabilities critical for it: nstart/nend for live insertion and BIO classification for scheduling. But the last I don't mind to return if there appear to be a need.? It is only the first I am strongly against.? But if somebody would like to reimplement it, may be it would be better to consider merging it with CAM I/O scheduler by Warner?? The one at least knows about device queue depth, etc.? We could return the BIO classification to be used by CAM scheduler instead, if needed. I'd be keen on helping anybody that wants to experiment with hard disk drive optmizations in iosched. My doodles to make it better showed no early improvements, so Iv'e not tried to bring them into the tree. However, our workload is basically 'large block random' which isn't the same as others and others might have a workload that could benefit. I've found a marginal improvement from the read over writes bias in our workload, and another?marginal improvement for favoring metadata reads over normal reads (because?for us, sendfile blocks for some of these reads, but others may see no improvement). I'm working to clean up the metadata read stuff to get it into the tree. I've not tested it on ZFS, though, so there will be no ZFS metadata labeling in the initial commit. So I like the idea, and would love to work with someone that needs it and/or whose work loads can be improved by it. The biggest issue I have found with drive sorting and traditional elevator algorithms is that it is not latency limiting. We have other problems at higher layers where we scheduling too many writes simultaneously that contribute substantially to I/O latency. Also read-after-writes are blocked in the buffer cache while a senseless number of buffers are queued and locked. An algorithm I have found effective and implemented at least twice is to estimate I/O time and then give a maximum sort latency. For many drives you have to go further and starve them for I/O until they complete a particularly long running operation or they can continue to decide to
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On Mon, Dec 30, 2019 at 12:55 PM Alexander Motin wrote: > On 30.12.2019 12:02, Alexey Dokuchaev wrote: > > On Mon, Dec 30, 2019 at 08:55:14AM -0700, Warner Losh wrote: > >> On Mon, Dec 30, 2019, 5:32 AM Alexey Dokuchaev wrote: > >>> On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: > New Revision: 356185 > URL: https://svnweb.freebsd.org/changeset/base/356185 > > Log: > Remove GEOM_SCHED class and gsched tool. > [...] > >>> > >>> Wow, that was unexpected, I use it on all my machines' HDD drives. > >>> Is there a planned replacement, or I'd better create a port for the > >>> GEOM_SCHED class and gsched(8) tool? > >> > >> How much of a performance improvement do you see with it? > >> > >> There has been no tweaks to this geom in years and years. It was tuned > >> to 10 year old hard drives and never retuned for anything newer. > > > > Well, hard drives essentially didn't change since then, still being the > > same roration media. :) > > At least some papers about gsched I read mention adX devices, which > means old ATA stack and no NCQ. It can be quite a significant change to > let HDD to do its own scheduling. Also about a year ago in r335066 > Warner added sysctl debug.bioq_batchsize, which if set to non-zero value > may, I think, improve fairness between several processes, just not sure > why it was never enabled. > I never enabled it because I never had a good car size as the default. I'm guessing it's somewhere on the order of 2 times the queue size in hardware, but with modern drives I think phk might be right and that disabling disksort entirely might be optimal, or close to optimal. > >> And when I played with it a few years ago, I saw no improvements... > > > > Admittedly, I've only did some tests no later than in 8.4 times when I > > first started using it. Fair point, though, I should redo them again. > > I'm sorry to create a regression for you, if there is really one. As I > have written I don't have so much against the scheduler part itself, as > against the accumulated technical debt and the way integration is done, > such as mechanism of live insertion, etc. Without unmapped I/O and > direct dispatch I bet it must be quite slow on bigger systems, that is > why I doubted anybody really use it. > > > Is there a planned replacement, or I'd better create a port for the > > GEOM_SCHED class and gsched(8) tool? > > I wasn't planning replacement. And moving it to ports would be a > problem, since in process I removed few capabilities critical for it: > nstart/nend for live insertion and BIO classification for scheduling. > But the last I don't mind to return if there appear to be a need. It is > only the first I am strongly against. But if somebody would like to > reimplement it, may be it would be better to consider merging it with > CAM I/O scheduler by Warner? The one at least knows about device queue > depth, etc. We could return the BIO classification to be used by CAM > scheduler instead, if needed. > I'd be keen on helping anybody that wants to experiment with hard disk drive optmizations in iosched. My doodles to make it better showed no early improvements, so Iv'e not tried to bring them into the tree. However, our workload is basically 'large block random' which isn't the same as others and others might have a workload that could benefit. I've found a marginal improvement from the read over writes bias in our workload, and another marginal improvement for favoring metadata reads over normal reads (because for us, sendfile blocks for some of these reads, but others may see no improvement). I'm working to clean up the metadata read stuff to get it into the tree. I've not tested it on ZFS, though, so there will be no ZFS metadata labeling in the initial commit. So I like the idea, and would love to work with someone that needs it and/or whose work loads can be improved by it. Warner -- > Alexander Motin > ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On 30.12.2019 12:02, Alexey Dokuchaev wrote: > On Mon, Dec 30, 2019 at 08:55:14AM -0700, Warner Losh wrote: >> On Mon, Dec 30, 2019, 5:32 AM Alexey Dokuchaev wrote: >>> On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: New Revision: 356185 URL: https://svnweb.freebsd.org/changeset/base/356185 Log: Remove GEOM_SCHED class and gsched tool. [...] >>> >>> Wow, that was unexpected, I use it on all my machines' HDD drives. >>> Is there a planned replacement, or I'd better create a port for the >>> GEOM_SCHED class and gsched(8) tool? >> >> How much of a performance improvement do you see with it? >> >> There has been no tweaks to this geom in years and years. It was tuned >> to 10 year old hard drives and never retuned for anything newer. > > Well, hard drives essentially didn't change since then, still being the > same roration media. :) At least some papers about gsched I read mention adX devices, which means old ATA stack and no NCQ. It can be quite a significant change to let HDD to do its own scheduling. Also about a year ago in r335066 Warner added sysctl debug.bioq_batchsize, which if set to non-zero value may, I think, improve fairness between several processes, just not sure why it was never enabled. >> And when I played with it a few years ago, I saw no improvements... > > Admittedly, I've only did some tests no later than in 8.4 times when I > first started using it. Fair point, though, I should redo them again. I'm sorry to create a regression for you, if there is really one. As I have written I don't have so much against the scheduler part itself, as against the accumulated technical debt and the way integration is done, such as mechanism of live insertion, etc. Without unmapped I/O and direct dispatch I bet it must be quite slow on bigger systems, that is why I doubted anybody really use it. > Is there a planned replacement, or I'd better create a port for the > GEOM_SCHED class and gsched(8) tool? I wasn't planning replacement. And moving it to ports would be a problem, since in process I removed few capabilities critical for it: nstart/nend for live insertion and BIO classification for scheduling. But the last I don't mind to return if there appear to be a need. It is only the first I am strongly against. But if somebody would like to reimplement it, may be it would be better to consider merging it with CAM I/O scheduler by Warner? The one at least knows about device queue depth, etc. We could return the BIO classification to be used by CAM scheduler instead, if needed. -- Alexander Motin ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On Mon, Dec 30, 2019 at 08:55:14AM -0700, Warner Losh wrote: > On Mon, Dec 30, 2019, 5:32 AM Alexey Dokuchaev wrote: > > On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: > > > New Revision: 356185 > > > URL: https://svnweb.freebsd.org/changeset/base/356185 > > > > > > Log: > > > Remove GEOM_SCHED class and gsched tool. > > > [...] > > > > Wow, that was unexpected, I use it on all my machines' HDD drives. > > Is there a planned replacement, or I'd better create a port for the > > GEOM_SCHED class and gsched(8) tool? > > How much of a performance improvement do you see with it? > > There has been no tweaks to this geom in years and years. It was tuned > to 10 year old hard drives and never retuned for anything newer. Well, hard drives essentially didn't change since then, still being the same roration media. :) > And when I played with it a few years ago, I saw no improvements... Admittedly, I've only did some tests no later than in 8.4 times when I first started using it. Fair point, though, I should redo them again. ./danfe ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On Mon, Dec 30, 2019, 5:32 AM Alexey Dokuchaev wrote: > On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: > > New Revision: 356185 > > URL: https://svnweb.freebsd.org/changeset/base/356185 > > > > Log: > > Remove GEOM_SCHED class and gsched tool. > > > > This code was not actively maintained since it was introduced 10 years > ago. > > It lacks support for many later GEOM features, such as direct dispatch, > > unmapped I/O, stripesize/stripeoffset, resize, etc. Plus it is the > only > > remaining use of GEOM nstart/nend request counters, used there to > implement > > live insertion/removal, questionable by itself. > > Wow, that was unexpected, I use it on all my machines' HDD drives. > Is there a planned replacement, or I'd better create a port for the > GEOM_SCHED class and gsched(8) tool? How much of a performance improvement do you see with it? There has been no tweaks to this geom in years and years. It was tuned to 10 year old hard drives and never retuned for anything newer. And when I played with it a few years ago, I saw no improvements... Warner > ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On 2019-12-30 06:32, Alexey Dokuchaev wrote: > On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: >> New Revision: 356185 >> URL: https://svnweb.freebsd.org/changeset/base/356185 >> >> Log: >> Remove GEOM_SCHED class and gsched tool. >> >> This code was not actively maintained since it was introduced 10 years ago. >> It lacks support for many later GEOM features, such as direct dispatch, >> unmapped I/O, stripesize/stripeoffset, resize, etc. Plus it is the only >> remaining use of GEOM nstart/nend request counters, used there to implement >> live insertion/removal, questionable by itself. > Wow, that was unexpected, I use it on all my machines' HDD drives. There was a posting by mav@ on the -arch list : " gsched: modernize or remove?". It was a short notice (Dec 27), but there was strong agreement from the author. > Is there a planned replacement, or I'd better create a port for the > GEOM_SCHED class and gsched(8) tool? If you can convince someone to update it with the enhancements suggested in the -arch post, I guess it could be resurrected, otherwise a port. Pedro. ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
On Sun, Dec 29, 2019 at 09:16:04PM +, Alexander Motin wrote: > New Revision: 356185 > URL: https://svnweb.freebsd.org/changeset/base/356185 > > Log: > Remove GEOM_SCHED class and gsched tool. > > This code was not actively maintained since it was introduced 10 years ago. > It lacks support for many later GEOM features, such as direct dispatch, > unmapped I/O, stripesize/stripeoffset, resize, etc. Plus it is the only > remaining use of GEOM nstart/nend request counters, used there to implement > live insertion/removal, questionable by itself. Wow, that was unexpected, I use it on all my machines' HDD drives. Is there a planned replacement, or I'd better create a port for the GEOM_SCHED class and gsched(8) tool? ./danfe ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r356185 - in head: lib/geom lib/geom/sched sys/geom sys/geom/sched sys/modules/geom sys/modules/geom/geom_sched sys/sys
Author: mav
Date: Sun Dec 29 21:16:03 2019
New Revision: 356185
URL: https://svnweb.freebsd.org/changeset/base/356185
Log:
Remove GEOM_SCHED class and gsched tool.
This code was not actively maintained since it was introduced 10 years ago.
It lacks support for many later GEOM features, such as direct dispatch,
unmapped I/O, stripesize/stripeoffset, resize, etc. Plus it is the only
remaining use of GEOM nstart/nend request counters, used there to implement
live insertion/removal, questionable by itself. Plus, as number of people
commented, GEOM is not the best place for I/O scheduler, since it has
limited information about layers both above and below it, required for
efficient scheduling. Plus with the modern shift to SSDs there is just no
more significant need for this kind of scheduling.
Approved by: imp, phk, luigi
Relnotes: yes
Deleted:
head/lib/geom/sched/
head/sys/geom/sched/
head/sys/modules/geom/geom_sched/
Modified:
head/lib/geom/Makefile.classes
head/sys/geom/geom.h
head/sys/geom/geom_io.c
head/sys/modules/geom/Makefile
head/sys/sys/bio.h
head/sys/sys/ktr_class.h
head/sys/sys/param.h
Modified: head/lib/geom/Makefile.classes
==
--- head/lib/geom/Makefile.classes Sun Dec 29 20:57:49 2019
(r356184)
+++ head/lib/geom/Makefile.classes Sun Dec 29 21:16:03 2019
(r356185)
@@ -20,7 +20,6 @@ GEOM_CLASSES+=nop
GEOM_CLASSES+= part
GEOM_CLASSES+= raid
GEOM_CLASSES+= raid3
-GEOM_CLASSES+= sched
GEOM_CLASSES+= shsec
GEOM_CLASSES+= stripe
GEOM_CLASSES+= virstor
Modified: head/sys/geom/geom.h
==
--- head/sys/geom/geom.hSun Dec 29 20:57:49 2019(r356184)
+++ head/sys/geom/geom.hSun Dec 29 21:16:03 2019(r356185)
@@ -231,17 +231,6 @@ struct g_provider {
u_int index;
};
-/*
- * Descriptor of a classifier. We can register a function and
- * an argument, which is called by g_io_request() on bio's
- * that are not previously classified.
- */
-struct g_classifier_hook {
- TAILQ_ENTRY(g_classifier_hook) link;
- int (*func)(void *arg, struct bio *bp);
- void*arg;
-};
-
/* BIO_GETATTR("GEOM::setstate") argument values. */
#define G_STATE_FAILED 0
#define G_STATE_REBUILD1
@@ -344,8 +333,6 @@ int g_io_getattr(const char *attr, struct g_consumer *
int g_io_zonecmd(struct disk_zone_args *zone_args, struct g_consumer *cp);
int g_io_flush(struct g_consumer *cp);
int g_io_speedup(size_t shortage, u_int flags, size_t *resid, struct
g_consumer *cp);
-int g_register_classifier(struct g_classifier_hook *hook);
-void g_unregister_classifier(struct g_classifier_hook *hook);
void g_io_request(struct bio *bp, struct g_consumer *cp);
struct bio *g_new_bio(void);
struct bio *g_alloc_bio(void);
Modified: head/sys/geom/geom_io.c
==
--- head/sys/geom/geom_io.c Sun Dec 29 20:57:49 2019(r356184)
+++ head/sys/geom/geom_io.c Sun Dec 29 21:16:03 2019(r356185)
@@ -87,15 +87,6 @@ static volatile u_int __read_mostly pace;
static uma_zone_t __read_mostly biozone;
-/*
- * The head of the list of classifiers used in g_io_request.
- * Use g_register_classifier() and g_unregister_classifier()
- * to add/remove entries to the list.
- * Classifiers are invoked in registration order.
- */
-static TAILQ_HEAD(, g_classifier_hook) g_classifier_tailq __read_mostly =
-TAILQ_HEAD_INITIALIZER(g_classifier_tailq);
-
#include
static void
@@ -224,9 +215,6 @@ g_clone_bio(struct bio *bp)
if (bp->bio_cmd == BIO_ZONE)
bcopy(>bio_zone, >bio_zone,
sizeof(bp->bio_zone));
- /* Inherit classification info from the parent */
- bp2->bio_classifier1 = bp->bio_classifier1;
- bp2->bio_classifier2 = bp->bio_classifier2;
#if defined(BUF_TRACKING) || defined(FULL_BUF_TRACKING)
bp2->bio_track_bp = bp->bio_track_bp;
#endif
@@ -498,66 +486,7 @@ g_io_check(struct bio *bp)
return (EJUSTRETURN);
}
-/*
- * bio classification support.
- *
- * g_register_classifier() and g_unregister_classifier()
- * are used to add/remove a classifier from the list.
- * The list is protected using the g_bio_run_down lock,
- * because the classifiers are called in this path.
- *
- * g_io_request() passes bio's that are not already classified
- * (i.e. those with bio_classifier1 == NULL) to g_run_classifiers().
- * Classifiers can store their result in the two fields
- * bio_classifier1 and bio_classifier2.
- * A classifier that updates one of the fields should
- * return a non-zero value.
- * If no classifier updates the field, g_run_classifiers() sets
- *
