Re: [PATCH] scsi/sata write barrier support
On Tue, Feb 22 2005, Greg Stark wrote: Jens Axboe [EMAIL PROTECTED] writes: fsync has been working all along, since the initial barrier support for ide. only ext3 and reiserfs support it. Really? That's huge news. Since what kernel version(s) is that? Since 2.6.9. What about a non-journaled fs, or at least a meta-data-only-journaled fs? Journaled FS's don't mix well with transaction based databases since they're basically doing their own journaling anyways. Only works on ext3 and reiserfs currently. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
Jens Axboe [EMAIL PROTECTED] writes: fsync has been working all along, since the initial barrier support for ide. only ext3 and reiserfs support it. Really? That's huge news. Since what kernel version(s) is that? What about a non-journaled fs, or at least a meta-data-only-journaled fs? Journaled FS's don't mix well with transaction based databases since they're basically doing their own journaling anyways. -- greg - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
On Mon, Feb 21 2005, Greg Stark wrote: Jens Axboe [EMAIL PROTECTED] writes: For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. What about for fsync(2)? One of the most frequent sources of data loss on the postgres mailing list has to do with users with IDE drives where fsync returns even though the data hasn't actually reached the disk. A power outage can cause lost data or a corrupted database. Is there any progress getting fsync to use this new infrastructure so it can actually satisfy its contract? fsync has been working all along, since the initial barrier support for ide. only ext3 and reiserfs support it. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
Jens Axboe wrote: On Thu, Jan 27 2005, Jeff Garzik wrote: Doug Maxey wrote: On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote: Hi, For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. This patch adds support for the same type of cache flushing barriers that PATA uses for SCSI, to be utilized with libata. What, if any mechanism supports changing the underlying write cache? That is, assuming this is common across PATA and SCSI drives, and it is possible to turn the cache off on the IDE drives, would switching the cache underneath require completing the inflight IO? [ignoring your question, but it made me think...] I am thinking the barrier support should know if the write cache is disabled (some datacenters do this), and avoid flushing if so? Ehm it does, read the code :) I did. I see nowhere that handles the case where the user uses a util (hdparm or blktool) to switch off write cache after sd.c has probed the disk. sd only sets its WCE bit at probe time, and doesn't appear to notice state changes. Since nobody snoops the MODE SELECT on the caching page, nobody knows past probe the state of write caching. Thus my comment... I think barrier support should know about that sort of thing :) Jeff - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
On Fri, Jan 28 2005, Jeff Garzik wrote: Jens Axboe wrote: On Thu, Jan 27 2005, Jeff Garzik wrote: Doug Maxey wrote: On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote: Hi, For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. This patch adds support for the same type of cache flushing barriers that PATA uses for SCSI, to be utilized with libata. What, if any mechanism supports changing the underlying write cache? That is, assuming this is common across PATA and SCSI drives, and it is possible to turn the cache off on the IDE drives, would switching the cache underneath require completing the inflight IO? [ignoring your question, but it made me think...] I am thinking the barrier support should know if the write cache is disabled (some datacenters do this), and avoid flushing if so? Ehm it does, read the code :) I did. I see nowhere that handles the case where the user uses a util (hdparm or blktool) to switch off write cache after sd.c has probed the disk. sd only sets its WCE bit at probe time, and doesn't appear to notice state changes. WCE bit should change then, like -wcache does for ide. It's handled inside sd, there's nothing more it can do. sd_shutdown() and sd_issue_flush() have the same issue, they all rely on -WCE being correct. Can't say I'm too fond of command snooping, but I guess it's the only solution. Since nobody snoops the MODE SELECT on the caching page, nobody knows past probe the state of write caching. Thus my comment... I think barrier support should know about that sort of thing :) So this would mainly be a problem if you boot with write caching disabled, but later turn it on. The other way around will still work safely, at the cost of two noop commands on each write barrier (which I doubt you would notice). -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
On Fri, Jan 28 2005, Jens Axboe wrote:
On Fri, Jan 28 2005, Jeff Garzik wrote:
Jens Axboe wrote:
On Thu, Jan 27 2005, Jeff Garzik wrote:
Doug Maxey wrote:
On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote:
Hi,
For the longest time, only the old PATA drivers supported barrier writes
with journalled file systems. This patch adds support for the same type
of cache flushing barriers that PATA uses for SCSI, to be utilized with
libata.
What, if any mechanism supports changing the underlying write cache?
That is, assuming this is common across PATA and SCSI drives, and it is
possible to turn the cache off on the IDE drives, would switching the
cache underneath require completing the inflight IO?
[ignoring your question, but it made me think...]
I am thinking the barrier support should know if the write cache is
disabled (some datacenters do this), and avoid flushing if so?
Ehm it does, read the code :)
I did. I see nowhere that handles the case where the user uses a util
(hdparm or blktool) to switch off write cache after sd.c has probed the
disk. sd only sets its WCE bit at probe time, and doesn't appear to
notice state changes.
WCE bit should change then, like -wcache does for ide. It's handled
inside sd, there's nothing more it can do. sd_shutdown() and
sd_issue_flush() have the same issue, they all rely on -WCE being
correct.
Can't say I'm too fond of command snooping, but I guess it's the only
solution.
Here's an update patch that attempts to snoop successful completion of a
mode select that updates the caching page. Untested, what do you think?
diff -urp -X /home/axboe/cdrom/exclude
/opt/kernel/linux-2.6.10/drivers/block/cfq-iosched.c
linux-2.6.10/drivers/block/cfq-iosched.c
--- /opt/kernel/linux-2.6.10/drivers/block/cfq-iosched.c2005-01-26
21:18:37.0 +0100
+++ linux-2.6.10/drivers/block/cfq-iosched.c2005-01-26 21:19:05.0
+0100
@@ -1285,19 +1285,19 @@ static int cfq_queue_empty(request_queue
static void cfq_completed_request(request_queue_t *q, struct request *rq)
{
struct cfq_rq *crq = RQ_DATA(rq);
+ struct cfq_queue *cfqq;
if (unlikely(!blk_fs_request(rq)))
return;
- if (crq-in_flight) {
- struct cfq_queue *cfqq = crq-cfq_queue;
+ cfqq = crq-cfq_queue;
+ if (crq-in_flight) {
WARN_ON(!cfqq-in_flight);
cfqq-in_flight--;
-
- cfq_account_completion(cfqq, crq);
}
+ cfq_account_completion(cfqq, crq);
}
static struct request *
diff -urp -X /home/axboe/cdrom/exclude
/opt/kernel/linux-2.6.10/drivers/block/elevator.c
linux-2.6.10/drivers/block/elevator.c
--- /opt/kernel/linux-2.6.10/drivers/block/elevator.c 2005-01-26
21:18:37.0 +0100
+++ linux-2.6.10/drivers/block/elevator.c 2005-01-26 21:19:05.0
+0100
@@ -320,7 +320,21 @@ void elv_add_request(request_queue_t *q,
static inline struct request *__elv_next_request(request_queue_t *q)
{
- return q-elevator-ops-elevator_next_req_fn(q);
+ struct request *rq = q-elevator-ops-elevator_next_req_fn(q);
+
+ /*
+* if this is a barrier write and the device has to issue a
+* flush sequence to support it, check how far we are
+*/
+ if (rq blk_fs_request(rq) blk_barrier_rq(rq)) {
+ BUG_ON(q-ordered == QUEUE_ORDERED_NONE);
+
+ if (q-ordered == QUEUE_ORDERED_FLUSH
+ !blk_barrier_preflush(rq))
+ rq = blk_start_pre_flush(q, rq);
+ }
+
+ return rq;
}
struct request *elv_next_request(request_queue_t *q)
diff -urp -X /home/axboe/cdrom/exclude
/opt/kernel/linux-2.6.10/drivers/block/ll_rw_blk.c
linux-2.6.10/drivers/block/ll_rw_blk.c
--- /opt/kernel/linux-2.6.10/drivers/block/ll_rw_blk.c 2005-01-26
21:18:37.0 +0100
+++ linux-2.6.10/drivers/block/ll_rw_blk.c 2005-01-28 10:35:04.0
+0100
@@ -267,6 +267,24 @@ void blk_queue_make_request(request_queu
EXPORT_SYMBOL(blk_queue_make_request);
+static inline void rq_init(request_queue_t *q, struct request *rq)
+{
+ INIT_LIST_HEAD(rq-queuelist);
+
+ rq-errors = 0;
+ rq-rq_status = RQ_ACTIVE;
+ rq-bio = rq-biotail = NULL;
+ rq-buffer = NULL;
+ rq-ref_count = 1;
+ rq-q = q;
+ rq-waiting = NULL;
+ rq-special = NULL;
+ rq-data_len = 0;
+ rq-data = NULL;
+ rq-sense = NULL;
+ rq-end_io = NULL;
+}
+
/**
* blk_queue_ordered - does this queue support ordered writes
* @q: the request queue
@@ -281,10 +299,26 @@ EXPORT_SYMBOL(blk_queue_make_request);
**/
void blk_queue_ordered(request_queue_t *q, int flag)
{
- if (flag)
- set_bit(QUEUE_FLAG_ORDERED, q-queue_flags);
- else
- clear_bit(QUEUE_FLAG_ORDERED, q-queue_flags);
+ switch (flag) {
+
Re: [PATCH] scsi/sata write barrier support
On Fri, 2005-01-28 at 10:38 +0100, Jens Axboe wrote:
+/*
+ * snoop succesfull completion of mode select commands that update the
+ * write back cache state
+ */
+#define MS_CACHE_PAGE0x08
+static void sd_snoop_cmd(struct scsi_cmnd *cmd)
+{
+ struct scsi_disk *sdpk;
+ char *page;
+
+ if (cmd-result)
+ return;
+
+ switch (cmd-cmnd[0]) {
+ case MODE_SELECT:
+ case MODE_SELECT_10:
+ page = cmd-request_buffer;
+ if (!page)
+ break;
+ if ((page[0] 0x3f) != MS_CACHE_PAGE)
+ break;
+ sdpk = dev_get_drvdata(cmd-device-sdev_gendev);
+ sdpk-WCE = (page[2] 0x04) != 0;
+ break;
+ }
+}
+
/**
* sd_rw_intr - bottom half handler: called when the lower level
* driver has completed (successfully or otherwise) a scsi command.
@@ -773,6 +831,9 @@ static void sd_rw_intr(struct scsi_cmnd
SCpnt-sense_buffer[13]));
}
#endif
+
+ sd_snoop_cmd(SCpnt);
+
Good grief no!
If you're going to try something like this, it needs to be a separate
patch over the scsi-list for one thing. And to save time:
1) The patch is actually wrong. There's more than one caching mode page
and not all of them affect current behaviour.
2) We have a current interface to update the WCE bit: You twiddle all
the disc parameters and then trigger a device rescan via sysfs (I'll
check that this updates the cache bits, I think it does, but if it
doesn't I'll make it).
3) If we think this is a quantity the users would like to see and alter,
then reading and setting it should be exported via sysfs.
4) Snooping SCSI commands is really bad ... it can get you into all
sorts of trouble which is why we prefer asking the device what state
it's in to trying to guess ourselves.
James
-
To unsubscribe from this list: send the line unsubscribe linux-scsi in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
On Fri, 2005-01-28 at 10:38 +0100, Jens Axboe wrote:
+/*
+ * snoop succesfull completion of mode select commands that update the
+ * write back cache state
+ */
+#define MS_CACHE_PAGE0x08
+static void sd_snoop_cmd(struct scsi_cmnd *cmd)
+{
+ struct scsi_disk *sdpk;
+ char *page;
+
+ if (cmd-result)
+ return;
+
+ switch (cmd-cmnd[0]) {
+ case MODE_SELECT:
+ case MODE_SELECT_10:
+ page = cmd-request_buffer;
+ if (!page)
+ break;
+ if ((page[0] 0x3f) != MS_CACHE_PAGE)
+ break;
+ sdpk = dev_get_drvdata(cmd-device-sdev_gendev);
+ sdpk-WCE = (page[2] 0x04) != 0;
+ break;
+ }
+}
+
/**
* sd_rw_intr - bottom half handler: called when the lower level
* driver has completed (successfully or otherwise) a scsi command.
@@ -773,6 +831,9 @@ static void sd_rw_intr(struct scsi_cmnd
SCpnt-sense_buffer[13]));
}
#endif
+
+ sd_snoop_cmd(SCpnt);
+
Good grief no!
If you're going to try something like this, it needs to be a separate
patch over the scsi-list for one thing. And to save time:
1) The patch is actually wrong. There's more than one caching mode page
and not all of them affect current behaviour.
2) We have a current interface to update the WCE bit: You twiddle all
the disc parameters and then trigger a device rescan via sysfs (I'll
check that this updates the cache bits, I think it does, but if it
doesn't I'll make it).
3) If we think this is a quantity the users would like to see and alter,
then reading and setting it should be exported via sysfs.
4) Snooping SCSI commands is really bad ... it can get you into all
sorts of trouble which is why we prefer asking the device what state
it's in to trying to guess ourselves.
James
-
To unsubscribe from this list: send the line unsubscribe linux-scsi in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] scsi/sata write barrier support
Hi,
For the longest time, only the old PATA drivers supported barrier writes
with journalled file systems. This patch adds support for the same type
of cache flushing barriers that PATA uses for SCSI, to be utilized with
libata. It prepares to support barriers with 'real' SCSI hardware that
can do ordered tags as well, but it isn't done. The remaining work is to
check error handling for reordering on requeue (probably fine), and add
the few lines of support to each hba lld to set -ordered_tag in the
host template and be able to set the ordered tag bit on commands that
are blk_barrier_rq().
A note on this patch. For write barriers, the PATA drivers issued a pre
and post flush when seeing a barrier write. This ensures that dirty data
in cache is on platter before the barrier, and that later writers don't
go to platter before the barrier. The handling of this was done in
drivers/ide core, which was a little nasty because some of it is a bit
tricky block layer code. This patch moves the handling of this into the
block layer, the only real change needed in drivers (scsi mid layer and
drivers/ide) is to delay completion of the real barrier write until the
post flush has completed. See scsi_io_completion() and
ide_end_request().
So the patch removes a lot of drivers/ide cruft and special casing,
which is always good. It relies on a seperate patch that adds an end_io
function to struct request, it's folded into this one right now. bsg
needs this as well, and the multipath folks need it too. I might submit
that seperately first.
Please test! Patch is against current BK.
diff -urp -X /home/axboe/cdrom/exclude
/opt/kernel/linux-2.6.10/drivers/block/cfq-iosched.c
linux-2.6.10/drivers/block/cfq-iosched.c
--- /opt/kernel/linux-2.6.10/drivers/block/cfq-iosched.c2005-01-26
21:18:37.0 +0100
+++ linux-2.6.10/drivers/block/cfq-iosched.c2005-01-26 21:19:05.0
+0100
@@ -1285,19 +1285,19 @@ static int cfq_queue_empty(request_queue
static void cfq_completed_request(request_queue_t *q, struct request *rq)
{
struct cfq_rq *crq = RQ_DATA(rq);
+ struct cfq_queue *cfqq;
if (unlikely(!blk_fs_request(rq)))
return;
- if (crq-in_flight) {
- struct cfq_queue *cfqq = crq-cfq_queue;
+ cfqq = crq-cfq_queue;
+ if (crq-in_flight) {
WARN_ON(!cfqq-in_flight);
cfqq-in_flight--;
-
- cfq_account_completion(cfqq, crq);
}
+ cfq_account_completion(cfqq, crq);
}
static struct request *
diff -urp -X /home/axboe/cdrom/exclude
/opt/kernel/linux-2.6.10/drivers/block/elevator.c
linux-2.6.10/drivers/block/elevator.c
--- /opt/kernel/linux-2.6.10/drivers/block/elevator.c 2005-01-26
21:18:37.0 +0100
+++ linux-2.6.10/drivers/block/elevator.c 2005-01-26 21:19:05.0
+0100
@@ -320,7 +320,21 @@ void elv_add_request(request_queue_t *q,
static inline struct request *__elv_next_request(request_queue_t *q)
{
- return q-elevator-ops-elevator_next_req_fn(q);
+ struct request *rq = q-elevator-ops-elevator_next_req_fn(q);
+
+ /*
+* if this is a barrier write and the device has to issue a
+* flush sequence to support it, check how far we are
+*/
+ if (rq blk_fs_request(rq) blk_barrier_rq(rq)) {
+ BUG_ON(q-ordered == QUEUE_ORDERED_NONE);
+
+ if (q-ordered == QUEUE_ORDERED_FLUSH
+ !blk_barrier_preflush(rq))
+ rq = blk_start_pre_flush(q, rq);
+ }
+
+ return rq;
}
struct request *elv_next_request(request_queue_t *q)
diff -urp -X /home/axboe/cdrom/exclude
/opt/kernel/linux-2.6.10/drivers/block/ll_rw_blk.c
linux-2.6.10/drivers/block/ll_rw_blk.c
--- /opt/kernel/linux-2.6.10/drivers/block/ll_rw_blk.c 2005-01-26
21:18:37.0 +0100
+++ linux-2.6.10/drivers/block/ll_rw_blk.c 2005-01-27 12:56:17.0
+0100
@@ -267,6 +267,24 @@ void blk_queue_make_request(request_queu
EXPORT_SYMBOL(blk_queue_make_request);
+static inline void rq_init(request_queue_t *q, struct request *rq)
+{
+ INIT_LIST_HEAD(rq-queuelist);
+
+ rq-errors = 0;
+ rq-rq_status = RQ_ACTIVE;
+ rq-bio = rq-biotail = NULL;
+ rq-buffer = NULL;
+ rq-ref_count = 1;
+ rq-q = q;
+ rq-waiting = NULL;
+ rq-special = NULL;
+ rq-data_len = 0;
+ rq-data = NULL;
+ rq-sense = NULL;
+ rq-end_io = NULL;
+}
+
/**
* blk_queue_ordered - does this queue support ordered writes
* @q: the request queue
@@ -281,10 +299,26 @@ EXPORT_SYMBOL(blk_queue_make_request);
**/
void blk_queue_ordered(request_queue_t *q, int flag)
{
- if (flag)
- set_bit(QUEUE_FLAG_ORDERED, q-queue_flags);
- else
- clear_bit(QUEUE_FLAG_ORDERED, q-queue_flags);
+ switch (flag) {
+ case QUEUE_ORDERED_NONE:
+ if (q-flush_rq)
+
Re: [PATCH] scsi/sata write barrier support
On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote: Hi, For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. This patch adds support for the same type of cache flushing barriers that PATA uses for SCSI, to be utilized with libata. What, if any mechanism supports changing the underlying write cache? That is, assuming this is common across PATA and SCSI drives, and it is possible to turn the cache off on the IDE drives, would switching the cache underneath require completing the inflight IO? ++doug - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
Doug Maxey wrote: On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote: Hi, For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. This patch adds support for the same type of cache flushing barriers that PATA uses for SCSI, to be utilized with libata. What, if any mechanism supports changing the underlying write cache? That is, assuming this is common across PATA and SCSI drives, and it is possible to turn the cache off on the IDE drives, would switching the cache underneath require completing the inflight IO? [ignoring your question, but it made me think...] I am thinking the barrier support should know if the write cache is disabled (some datacenters do this), and avoid flushing if so? Jeff - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support #2
Jens Axboe wrote: Hi, A few changes: - Cleanup up the driver additions even more, blk_complete_barrier_rq() does all the work now. - Fixed up the exports - Comment functions - Fixed a bug with SCSI and write back caching disabled - Rename blk_queue_flush() to blk_queue_flushing() to indicate it's a state Tiny nit for SATA drivers: Define a ATA_ORDERED_IO constant in libata.h, rather than hardcoding the 1 in each driver. - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
On Thu, Jan 27 2005, Jeff Garzik wrote: Doug Maxey wrote: On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote: Hi, For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. This patch adds support for the same type of cache flushing barriers that PATA uses for SCSI, to be utilized with libata. What, if any mechanism supports changing the underlying write cache? That is, assuming this is common across PATA and SCSI drives, and it is possible to turn the cache off on the IDE drives, would switching the cache underneath require completing the inflight IO? [ignoring your question, but it made me think...] I am thinking the barrier support should know if the write cache is disabled (some datacenters do this), and avoid flushing if so? Ehm it does, read the code :) -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] scsi/sata write barrier support
On Thu, Jan 27 2005, Doug Maxey wrote: On Thu, 27 Jan 2005 13:02:48 +0100, Jens Axboe wrote: Hi, For the longest time, only the old PATA drivers supported barrier writes with journalled file systems. This patch adds support for the same type of cache flushing barriers that PATA uses for SCSI, to be utilized with libata. What, if any mechanism supports changing the underlying write cache? That is, assuming this is common across PATA and SCSI drives, and it is possible to turn the cache off on the IDE drives, would switching the cache underneath require completing the inflight IO? Not sure what you mean, switching it off while the io is in flight? You cannot do that, either the cache state will change right before starting the io or right after. The cache state at the start of the barrier write will determine the action taken. -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-scsi in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
