Re: [Nbd] [Qemu-devel] [RFC 1/1] nbd (specification): add NBD_CMD_WRITE_ZEROES command

2016-02-17 Thread Eric Blake
. Do we have the converse operation, that is, an easy way to query if a block of data will read as all zeroes, and therefore the client can bypass reading that portion of the disk (in other words, an equivalent to lseek(SEEK_HOLE/SEEK_DATA))? -- Eric Blake ebl

Re: [Nbd] [Qemu-devel] [RFC 1/1] nbd (specification): add NBD_CMD_WRITE_ZEROES command

2016-02-18 Thread Eric Blake
On 02/18/2016 02:18 AM, Roman Kagan wrote: > On Wed, Feb 17, 2016 at 01:58:47PM -0700, Eric Blake wrote: >> On 02/17/2016 11:10 AM, Denis V. Lunev wrote: >>> @@ -446,6 +448,11 @@ The following request types exist: >>> about the contents of the export affe

Re: [Nbd] SUMMARY: Re: [Qemu-devel] [RFC 1/1] nbd (specification): add NBD_CMD_WRITE_ZEROES command

2016-02-18 Thread Eric Blake
ring the entire read request (no compression). > > If there are no objections I'll sum this up and come with a > specification draft. Good luck! I'm sure you'll get good reviews. > > Den > > P.S. I have added here all parties which have participated in > c

Re: [Nbd] [Qemu-devel] [PATCH 1/2] NBD proto: add WRITE_ZEROES extension

2016-03-23 Thread Eric Blake
client issues the command in spite of not negotiating it? Should we enhance the documentation in the "Error values" heading to mention that EINVAL should be used in general for any client command not expected by the server?

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-03-23 Thread Eric Blake
ives a `GET_LBA_STATUS` request > +including one or more sectors beyond the size of the device. As mentioned in the previous mail, should we also recommend an EINVAL if NBD_CMD_GET_LBA_STATUS was not negotiated in the export options but the client sends the command anyways; and/or a requirement th

Re: [Nbd] [Qemu-devel] [PATCH 1/2] NBD proto: add WRITE_ZEROES extension

2016-03-24 Thread Eric Blake
ssages is a bug. > I should probably just fix my implementation instead. While I agree that you should fix your implementation, I am strongly in favor of a new command, so that we can blindly state that the NBD_CMD_WRITE always sends a payload of length bytes, independent of flag value.

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-03-24 Thread Eric Blake
and with EINVAL, or mark all blocks as > dirty in the descriptor that it returns. Is it feasible to return zero/allocated/dirty status all at the same time, or do we want to strictly require two different modes of operation? That is, if we are returning zero and allocated as two bits, c

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-03-24 Thread Eric Blake
t no guarantee on contents >> 0x2 - LBA extent not present, but client can treat the extent as zeroes >> and reads will succeed >> 0x3 - LBA extent present, client can treat the extent as zeroes and >> reads will succeed > > I'm not sure that clients need this level

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-03-24 Thread Eric Blake
On 03/24/2016 09:53 AM, Wouter Verhelst wrote: > On Thu, Mar 24, 2016 at 04:33:42PM +0100, Paolo Bonzini wrote: >> On 24/03/2016 16:25, Eric Blake wrote: >>>> However, let's make these bits, so that >>>> >>>> NBD_STATE_ALLOCATED (0x1), LBA extent

[Nbd] [PATCH] Fix doc typos

2016-03-24 Thread Eric Blake
Fix the spelling of "beef", and the case of an "NBD_REP" constant. Signed-off-by: Eric Blake --- I noticed these while reviewing the recent proposed spec additions for efficient sparse file handling doc/proto.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-03-24 Thread Eric Blake
egotiated flag for structured length replies, it would be possible to allow for short reads and/or returning an error with 0 bytes of payload but still keeping the connection to the client open, without having to send wasted bytes over the wire. I could write up a negotiation of global flags f

Re: [Nbd] [PATCH 1/3] NBD proto: forbid TRIM command without negotiation

2016-03-28 Thread Eric Blake
ommand without prior > successful negotiation. > > Signed-off-by: Pavel Borzenkov > Reviewed-by: Roman Kagan > Signed-off-by: Denis V. Lunev > CC: Wouter Verhelst > CC: Eric Blake > CC: Alex Bligh > --- > doc/proto.md | 3 +++ > 1 file changed, 3 insertions(+)

Re: [Nbd] [PATCH 2/3] NBD proto: document additional error conditions

2016-03-28 Thread Eric Blake
l such cases. I'd go one step further and document that for backwards compatibility, clients should not rely on this particular error. > > Signed-off-by: Pavel Borzenkov > Reviewed-by: Roman Kagan > Signed-off-by: Denis V. Lunev > CC: Wouter Verhelst > CC: Eric Blake &g

Re: [Nbd] [PATCH 3/3] NBD proto: add "Command flags" section

2016-03-28 Thread Eric Blake
f-by: Denis V. Lunev > CC: Wouter Verhelst > CC: Eric Blake > CC: Alex Bligh > --- > doc/proto.md | 10 ++ > 1 file changed, 10 insertions(+) > > diff --git a/doc/proto.md b/doc/proto.md > index 036d6d9..662f741 100644 > --- a/doc/proto.md > +++ b/do

[Nbd] [PATCH] doc: Mention proper use of handle

2016-03-28 Thread Eric Blake
order. Signed-off-by: Eric Blake --- doc/proto.md | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/proto.md b/doc/proto.md index 6d1cb34..d0102e0 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -200,7 +200,11 @@ S: 64 bits, handle S: (*length* bytes of data if the

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-03-28 Thread Eric Blake
it is allowed or forbidden from advertising structured reply status during the handshake phase (including the case where the ioctl() is not present being treated as the client must not enable structured replies). > > Markus? Markus is on vacation for a bit, so we'll just have to

[Nbd] [PATCH 2/1] doc: More details on flag negotiation

2016-03-28 Thread Eric Blake
sed during oldstyle negotiation since a client has no chance to respond with the corresponding client flag. Signed-off-by: Eric Blake --- doc/proto.md | 17 ++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index d0102e0..44579fc 100644

[Nbd] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-28 Thread Eric Blake
: Eric Blake --- doc/proto.md | 123 +++ 1 file changed, 123 insertions(+) diff --git a/doc/proto.md b/doc/proto.md index 44579fc..f687e3e 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -209,6 +209,10 @@ same value for handle as was sent by the

Re: [Nbd] [PATCH 1/3] NBD proto: forbid TRIM command without negotiation

2016-03-29 Thread Eric Blake
more, a server is compliant if it accepts and ignores NBD_CMD_TRIM (as that command is only a hint); which is different from NBD_CMD_FLUSH (which must ensure a barrier). -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description

Re: [Nbd] [PATCH] doc: Mention proper use of handle

2016-03-29 Thread Eric Blake
On 03/29/2016 01:11 AM, Wouter Verhelst wrote: > On Mon, Mar 28, 2016 at 07:59:15AM -0600, Eric Blake wrote: >> Although the proper use of the handle field during transmission >> phase was implied, it never hurts to make it more explicit that >> clients should alter the handle

Re: [Nbd] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
On 03/29/2016 02:24 AM, Alex Bligh wrote: > > On 29 Mar 2016, at 04:56, Eric Blake wrote: > >> The existing transmission phase protocol is difficult to sniff, >> because correct interpretation of the server stream requires >> context from the client stream (or risks

Re: [Nbd] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
make the sending of the chunk flags dependent on the client's choice of command flags (that is, we already argued that wire format should never be changed based merely on command flags, as it makes the server stream harder to decipher in isolation). Thanks for the good

Re: [Nbd] [PATCH 3/3] NBD proto: add "Command flags" section

2016-03-29 Thread Eric Blake
f-by: Denis V. Lunev > CC: Wouter Verhelst > CC: Eric Blake > CC: Alex Bligh > --- > doc/proto.md | 10 ++ > 1 file changed, 10 insertions(+) > > diff --git a/doc/proto.md b/doc/proto.md > index 036d6d9..662f741 100644 > --- a/doc/proto.md > +++ b/do

Re: [Nbd] [PATCH 3/3] NBD proto: add "Command flags" section

2016-03-29 Thread Eric Blake
On 03/29/2016 10:01 AM, Eric Blake wrote: > On 03/28/2016 04:43 AM, Denis V. Lunev wrote: >> From: Pavel Borzenkov >> >> Add separate "Command flags" section to make it clear which flags are >> currently defined by the protocol. >> >> Signed-off

Re: [Nbd] [Qemu-devel] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
On 03/29/2016 11:34 AM, Alex Bligh wrote: > > On 29 Mar 2016, at 16:12, Eric Blake wrote: >>> >>> More a way of guaranteeing avoiding a fragmentation on 'simple' reads. >>> Perhaps a 'DF' bit (don't fragment)! If the server doesn't

Re: [Nbd] [Qemu-devel] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
On 03/29/2016 12:03 PM, Wouter Verhelst wrote: > On Tue, Mar 29, 2016 at 11:45:45AM -0600, Eric Blake wrote: >> On 03/29/2016 11:34 AM, Alex Bligh wrote: >>> I would agree. I think if it supports the structured reply semantics, >>> it should also support 'DF'

Re: [Nbd] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
On 03/29/2016 11:53 AM, Wouter Verhelst wrote: > Hi Eric, > > Having read this in more detail now: > > On Mon, Mar 28, 2016 at 09:56:36PM -0600, Eric Blake wrote: >> + The server MUST ensure that each read chunk lies within the original >> + offset and length of

Re: [Nbd] [Qemu-devel] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
On 03/29/2016 12:19 PM, Wouter Verhelst wrote: > On Tue, Mar 29, 2016 at 12:07:59PM -0600, Eric Blake wrote: >> On 03/29/2016 12:03 PM, Wouter Verhelst wrote: >>> On Tue, Mar 29, 2016 at 11:45:45AM -0600, Eric Blake wrote: >>>> Supporting DF merely transfers the burde

Re: [Nbd] [PATCH 3/1] doc: Propose Structured Replies extension

2016-03-29 Thread Eric Blake
that command can always return a single structured reply where the id of the reply says whether the payload will be length 0 or not, but only new commands should get that treatment. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.a

Re: [Nbd] [PATCH] Strawman proposal for NBD structured replies

2016-03-29 Thread Eric Blake
> detailed information", or something along those lines. Except that offset 0 IS a valid location, and there is no unsigned number before 0 if the read error occurs while reading the head of the file. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library ht

[Nbd] [PATCH v2 1/3] NBD proto: add "Command flags" section

2016-03-29 Thread Eric Blake
20-4-git-send-email-...@openvz.org> [rearrange subsections to parallel Handshake phase, add more details] Signed-off-by: Eric Blake --- doc/proto.md | 14 ++ 1 file changed, 14 insertions(+) diff --git a/doc/proto.md b/doc/proto.md index aaae0a2..3d49162 100644 --- a/doc/proto.md ++

[Nbd] [PATCH v2 2/3] doc: Mention proper use of handle

2016-03-29 Thread Eric Blake
server is sending replies out of order. Make it clear that the client does not have to follow any particular order of handles, and can reuse values. Signed-off-by: Eric Blake --- doc/proto.md | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/proto.md b/doc/proto.md index

[Nbd] [PATCH v2 3/3] doc: Propose Structured Read extension

2016-03-29 Thread Eric Blake
they carry a data payload. This proposal does NOT permit structured replies to anything other than NBD_CMD_READ, although a future proposal may wish to make that valid (so that a server could be written that only returns structured replies). Signed-off-by: Eric Blake --- doc/proto.md | 260

[Nbd] [PATCH v2 0/3] NBD Structured Read

2016-03-29 Thread Eric Blake
th no offset). Eric Blake (2): doc: Mention proper use of handle doc: Propose Structured Read extension Pavel Borzenkov (1): NBD proto: add "Command flags" section doc/proto.md | 282 +-- 1 file changed, 277 insertions(

Re: [Nbd] [PATCHv2] Strawman proposal for NBD structured replies

2016-03-29 Thread Eric Blake
nder such circumstances the server MAY error the command > +with `ETOOBIG` if the length read exceeds [65,536 bytes | the `ETOOBIG` is not a standard error; my v2 went with the POSIX EOVERFLOW and defined it to it's Linux value of 75. > + > +If more than one data chunk containing

Re: [Nbd] [Qemu-devel] [PATCH v2 3/3] doc: Propose Structured Read extension

2016-03-29 Thread Eric Blake
On 03/29/2016 05:01 PM, Eric Blake wrote: > The existing transmission phase protocol is difficult to sniff, > because correct interpretation of the server stream requires > context from the client stream (or risks false positives if > data payloads happen to contain the protocol m

Re: [Nbd] [Qemu-devel] [PATCH v2 3/3] doc: Propose Structured Read extension

2016-03-30 Thread Eric Blake
hink there are bits I've got that you haven't (as well as vice versa). > But I'm inclined to use yours as a base unless you particularly > like mine. > > Comments inline below. > > Alex > > On 30 Mar 2016, at 00:01, Eric Blake wrote: > ... &g

Re: [Nbd] [PATCHv2] Strawman proposal for NBD structured replies

2016-03-30 Thread Eric Blake
On 03/30/2016 01:33 AM, Wouter Verhelst wrote: > Morning, > > On Wed, Mar 30, 2016 at 07:59:15AM +0100, Alex Bligh wrote: >> On 30 Mar 2016, at 00:17, Eric Blake wrote: >>>> >>>> -The server replies with: >>>> +Replies take one of two forms.

[Nbd] NBD_REP_SERVER layout [was: [Qemu-devel] [PATCHv2] Strawman proposal for NBD structured replies]

2016-03-30 Thread Eric Blake
On 03/30/2016 12:43 PM, Eric Blake wrote: > On that tangent, I found SELECT slightly ambiguous (particularly since > I'm also considering a proposal to modify NBD_REP_SERVER to expose > alignment details, so it would have to play nicely with SELECT): > > Based on normative

Re: [Nbd] [Qemu-devel] [PATCH v2 3/3] doc: Propose Structured Read extension

2016-03-30 Thread Eric Blake
d of the command, all the bytes will have been covered); while a client using pwritev() will have to construct an iovec that visits the chunks in the correct order (not necessarily in the order received). Clients that really don't want to do much work have the DF flag to forbid fragmentati

Re: [Nbd] [PATCH v2 3/3] doc: Propose Structured Read extension

2016-03-30 Thread Eric Blake
e it makes sense to rename the "global" > flags field as the "negotiation flags field" which signals incompatible Maybe "handshake flags" instead, since we document handshake/transmission phases? > changes in the negotiation phase, and the "per-expo

Re: [Nbd] [Qemu-devel] [PATCH v2 3/3] doc: Propose Structured Read extension

2016-03-30 Thread Eric Blake
chnically, I don't see how haggling structured replies before encryption is a data leak (I _do_ see how NBD_OPT_LIST could be a security leak on an unencrypted connection, but that's different) - so maybe we need to revisit the wording on when it is fair game for a server to re

[Nbd] [PATCH v3 2/5] doc: Clean up wording on UTF-8 and REP_SERVER usage

2016-03-30 Thread Eric Blake
about SELECT changing the semantics of the extra data is not a field within the data). Signed-off-by: Eric Blake --- doc/proto.md | 46 -- 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index ea7a94f..04d4a13

[Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-30 Thread Eric Blake
able to observe as option haggling progresses. Signed-off-by: Eric Blake --- doc/proto.md | 391 --- 1 file changed, 374 insertions(+), 17 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index c1e05c5..cd59d81 100644 --- a/doc

[Nbd] [PATCH v3 3/5] doc: Reformat description of Transmission phase

2016-03-30 Thread Eric Blake
Rearrange a paragraph and add some headings, so that the next patch adding a Structured Reply has less churn. Signed-off-by: Eric Blake --- doc/proto.md | 29 ++--- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index 04d4a13

[Nbd] [PATCH v3 5/5] RFC: doc: Promote structured reply out of experimental

2016-03-30 Thread Eric Blake
indentation). Signed-off-by: Eric Blake --- doc/proto.md | 621 ++- 1 file changed, 273 insertions(+), 348 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index cd59d81..7bc65f8 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -182,29 +182,32

[Nbd] [PATCH v3 1/5] doc: Minor consistency issues

2016-03-30 Thread Eric Blake
- Fix a typo - consistently use trailing spacing at the end of layout descriptions - drop other unneeded trailing spacing - add a missing sentence for consistency on what a client should do with unknown flags - fix a terminology change missed in a prior patch Signed-off-by: Eric Blake --- doc

[Nbd] [PATCH v3 0/5] Structured replies

2016-03-30 Thread Eric Blake
a message within a larger structured reply Eric Blake (5): doc: Minor consistency issues doc: Clean up wording on UTF-8 and REP_SERVER usage doc: Reformat description of Transmission phase doc: Propose STRUCTURED_REPLY extension RFC: doc: Promote structured reply out of experimental do

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
lex had some good comments that will also affect a v4, and we're still waiting for Markus to get back and have a chance to chime in, so this patch is not quite ready for inclusion yet, but I'm liking the progress we've made. -- Eric Blake eblake redhat com+1-919-301-3266 Li

Re: [Nbd] [PATCH v2 1/1] NBD proto: add WRITE_ZEROES extension

2016-03-31 Thread Eric Blake
llocation requests are less common, and also less efficient; so having the default state of the flag geared towards efficiency (both in the sense that punching holes can be faster than writing zeroes, and that most people LIKE the storage savings of sparse files). > I suspect there are already som

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
On 03/31/2016 05:23 AM, Alex Bligh wrote: > Eric, > > We're getting there! I'm only going to comment on this one rather than > the 'final form' patch, so we get this one right to start off with. Agreed. > > On 31 Mar 2016, at 07:06, Eric Blake wrote: &

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
d DATA) > * non-content chunks (NONE and ERROR) > > and define everything explicitly as a chunk. Reformulated slightly: There are 5 chunk types currently defined. Of those 5, 2 are content chunks (HOLE, DATA) (3 are non-content, but other non-

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
on; I've started a strawman locally, but it's not polished enough to post yet, with all the churn on structured replies first. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature -

Re: [Nbd] [PATCH 1/2] proto.md: Clearly set out NBDMAGIC is the actual value

2016-03-31 Thread Eric Blake
rest of this file is less than 80 columns, making this a long line. You can wrap it (see how I wrapped the line regarding 124 bytes of zeroes in response to NBD_OPT_EXPORT_NAME). > S: 64 bits, `0x49484156454F5054` (note different magic number) As long as we are spelling out ASCII counterpar

Re: [Nbd] [PATCH 2/2] Correct definition of NBD_CMD_FLAG_FUA

2016-03-31 Thread Eric Blake
-client library for embedding in larger programs. > > Should we produce a new name for it (and future command flags) > that aren't shifted left 16 places, and just maintain the > current value for compatibility? I don't see the point. Your fix looks correct. -- Er

[Nbd] Is NBD_CMD_FLAG_FUA valid during NBD_CMD_FLUSH?

2016-03-31 Thread Eric Blake
semantics would the flag add, and we need to fix the NBD docs as well as relax the reference implementation to allow the flag), or is it a bug in qemu (and the recent tightening of NBD to throw EINVAL on unsupported flags will trip up qemu)? -- Eric Blake eblake redhat com+1-919-301-32

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
On 03/31/2016 01:34 PM, Alex Bligh wrote: > > On 31 Mar 2016, at 19:50, Eric Blake wrote: > >>> I'd also like to say "I'd like my offsets / lengths to be a multiple of 4k" >>> so I can open O_DIRECT. Or at least say "can you cope with only of

Re: [Nbd] Is NBD_CMD_FLAG_FUA valid during NBD_CMD_FLUSH?

2016-03-31 Thread Eric Blake
On 03/31/2016 01:41 PM, Alex Bligh wrote: > > On 31 Mar 2016, at 20:33, Eric Blake wrote: > >> Qemu's nbd-client is setting NBD_CMD_FLAG_FUA during a flush command, >> but the official NBD protocol documentation doesn't describe this as >> valid (it mer

Re: [Nbd] [PATCH 2/2] Correct definition of NBD_CMD_FLAG_FUA

2016-03-31 Thread Eric Blake
On 03/31/2016 01:25 PM, Alex Bligh wrote: > > On 31 Mar 2016, at 20:14, Eric Blake wrote: > >>> >>> >>> Should we produce a new name for it (and future command flags) >>> that aren't shifted left 16 places, and just maintain the >>> c

Re: [Nbd] Is NBD_CMD_FLAG_FUA valid during NBD_CMD_FLUSH?

2016-03-31 Thread Eric Blake
just barely documented that servers SHOULD reject invalid flags with EINVAL; and that clients MUST NOT send FUA on commands where it is not documented; I don't know if we have an adequate discovery system in place to learn _which_ commands support FUA, especially if yo

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
e reserved a bit in the protocol means that userspace could still reconstruct the correct flag to pass to the kernel based on the result of structured reply negotiation, rather than relying on the state of the bit returned from OPT_SELECT). -- Eric

Re: [Nbd] [PATCH v2 1/1] NBD proto: add WRITE_ZEROES extension

2016-03-31 Thread Eric Blake
e server supports >`NBD_CMD_TRIM` commands > +- bit 6, `NBD_FLAG_SEND_WRITE_ZEROES`; should be set to 1 if the server > + supports `NBD_CMD_WRITE_ZEROES` commands Hmm, we've picked overlapping bits between your proposal and mine for `NBD_FLAG_SEND_DF`. Obviously, whoever goes in second get

Re: [Nbd] [PATCH v3 4/5] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
On 03/31/2016 05:39 PM, Alex Bligh wrote: > > On 1 Apr 2016, at 00:26, Eric Blake wrote: > >> Hmm. If OPT_LIST reports 0, that's okay; we'll send transmission flags >> again later during OPT_EXPORT_NAME/OPT_SELECT. But if OPT_SELECT >> reports 0, and then

[Nbd] [PATCH] doc: Fix shortcoming of NBD_OPT_GO before it gets implemented

2016-03-31 Thread Eric Blake
). Also, the trailing zeroes only occur after NBD_OPT_EXPORT_NAME (or oldstyle handshake), and should never appear after NBD_OPT_GO. Signed-off-by: Eric Blake --- doc/proto.md | 32 +++- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/doc/proto.md b/doc

Re: [Nbd] [PATCH] Improve documentation of FUA and FLUSH

2016-03-31 Thread Eric Blake
reply until it has ensured that any data referred to > + by this request (i.e. written data on a write or trim, read data on > + a read) has reached permanent storage. There will be certain commands > + (e.g. `NBD_CMD_DISC`) for which this flag will thus not alter behaviour > + (as th

[Nbd] [PATCH v4] doc: Propose STRUCTURED_REPLY extension

2016-03-31 Thread Eric Blake
they carry a data payload. It also makes it possible to return UTF-8 human-readable messages alongside an error code; and therefore structured replies are permitted for all commands. Signed-off-by: Eric Blake --- In v4: rearrange paragraphs a bit, document 'structured reply chunk message'

Re: [Nbd] [PATCH v4] doc: Propose STRUCTURED_REPLY extension

2016-04-01 Thread Eric Blake
On 04/01/2016 02:57 AM, Wouter Verhelst wrote: > Did a few minor changes, though: > > On Thu, Mar 31, 2016 at 11:29:48PM -0600, Eric Blake wrote: >> -S: 32 bits, 0x67446698, magic (`NBD_REPLY_MAGIC`) >> -S: 32 bits, error >> +S: 32 bits, 0x67446698, mag

Re: [Nbd] [PATCHv2 2/2] Correct definition of NBD_CMD_FLAG_FUA

2016-04-01 Thread Eric Blake
up, make it obvious this isn't really bit 16. > > Signed-off-by: Alex Bligh > --- > nbd.h | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) Reviewed-by: Eric Blake -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt

Re: [Nbd] [PATCHv2 1/2] proto.md: Clearly set out NBDMAGIC is the actual value

2016-04-01 Thread Eric Blake
12 > 1 file changed, 8 insertions(+), 4 deletions(-) Reviewed-by: Eric Blake -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc D

Re: [Nbd] [PATCH] doc/proto.md: restore formatting

2016-04-01 Thread Eric Blake
On 04/01/2016 04:34 AM, Alex Bligh wrote: > Restore formatting and correct name of 'length' > > Signed-off-by: Alex Bligh > --- > doc/proto.md | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) Reviewed-by: Eric Blake -- Eric Blake eblake redhat co

Re: [Nbd] [PATCH] doc/proto.md: Clients MUST not set unknown client flags

2016-04-01 Thread Eric Blake
o close the connection. > > Signed-off-by: Alex Bligh > --- > doc/proto.md | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Reviewed-by: Eric Blake -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc D

Re: [Nbd] Is NBD_CMD_FLAG_FUA valid during NBD_CMD_FLUSH?

2016-04-01 Thread Eric Blake
bey FUA semantics, but buggy servers MAY ignore the flag instead. NBD_CMD_TRIM: If NBD_FLAG_SEND_FUA is set, the client SHOULD use NBD_OPT_QUERY_FUA to determine whether using NBD_CMD_FLAG_FUA will have any effect. Since the command is advisory, the server is free to ignore the flag. Thoughts?

Re: [Nbd] Is NBD_CMD_FLAG_FUA valid during NBD_CMD_FLUSH?

2016-04-01 Thread Eric Blake
an older server can't reliably tell if the behavior will be guaranteed. > I suppose I am going have to try another lkml message to get > to the bottom of the first one. On the other hand if we > take route (a) above, we can relatively easily add a > 'mea

Re: [Nbd] Is NBD_CMD_FLAG_FUA valid during NBD_CMD_FLUSH?

2016-04-01 Thread Eric Blake
ime out to fix NBD server to silently ignore FUA on flush (fix the breakage between the pairing). -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature

Re: [Nbd] [PATCH v4] doc: Propose STRUCTURED_REPLY extension

2016-04-01 Thread Eric Blake
g content chunk are not valid. >> + >> +A client MAY close the connection if it detects that the server > > MUST? > >> +has sent invalid chunks (such as overlapping data, or not enough >> +data before claiming success)

Re: [Nbd] [PATCH] doc/proto.md: restore formatting

2016-04-01 Thread Eric Blake
On 04/01/2016 07:30 AM, Eric Blake wrote: > On 04/01/2016 04:34 AM, Alex Bligh wrote: >> Restore formatting and correct name of 'length' >> >> Signed-off-by: Alex Bligh >> --- >> doc/proto.md | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deleti

[Nbd] [PATCH] doc: Cleanups to structured reads

2016-04-01 Thread Eric Blake
A couple of typos, odd formatting, and missing words made it into the structured read spec, and several potential ambiguous situations were worth rewording for clarity. Signed-off-by: Eric Blake Signed-off-by: Alex Bligh --- doc/proto.md | 112

Re: [Nbd] [PATCH v2 1/1] NBD proto: add WRITE_ZEROES extension

2016-04-01 Thread Eric Blake
as to write the data onto disk, effectively denying >> the sparseness. >> >> + >> +- bit 1, `NBD_CMD_MAY_TRIM`; defined by the experimental `WRITE_ZEROES` >> + extension; see below. Hmm, we had an unfinished conversation about whether the default sense of this bit should

[Nbd] [PATCH] doc: Flip bit sense for allowing trim during WRITE_ZEROES

2016-04-01 Thread Eric Blake
no dependency on whether NBD_CMD_TRIM is supported. Also tweak a couple of formatting issues for consistency (for example, only reserve a bit number in one place). Signed-off-by: Eric Blake --- doc/proto.md | 30 +- 1 file changed, 17 insertions(+), 13 deletions

Re: [Nbd] [PATCH] Fix tests on OS-X

2016-04-04 Thread Eric Blake
orrect. > FOLLOW the non-option part of the command line. This is unnecessary. > Correct three tests that do this to use natural command-line > ordering. > > Signed-off-by: Alex Bligh > --- -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization li

Re: [Nbd] [PATCH] doc: Flip bit sense for allowing trim during WRITE_ZEROES

2016-04-04 Thread Eric Blake
On 04/01/2016 03:29 PM, Eric Blake wrote: > Rather than requiring allocation by default and allowing trims > only on request during WRITE_ZEROES, it seems like a better > default is to allow server optimizations by default and require > full allocation by specific request. Since WRI

[Nbd] [PATCH] doc: Allow NBD_CMD_FLAG_NO_HOLE during NBD_CMD_WRITE

2016-04-04 Thread Eric Blake
wanted to write zeroes rather than a hole. So it makes sense to let the new NBD_CMD_FLAG_NO_HOLE work for WRITE as well as WRITE_ZEROES. Signed-off-by: Eric Blake --- doc/proto.md | 33 +++-- 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/doc/proto.md

Re: [Nbd] [PATCH] doc: Allow NBD_CMD_FLAG_NO_HOLE during NBD_CMD_WRITE

2016-04-04 Thread Eric Blake
On 04/04/2016 08:47 AM, Denis V. Lunev wrote: > On 04/04/2016 05:15 PM, Eric Blake wrote: >> qemu already has an existing server implementation option that will >> explicitly search the payload of NBD_CMD_WRITE for large blocks of >> zeroes, and punch holes in the underlying f

[Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
CC: Wouter Verhelst CC: Paolo Bonzini CC: Kevin Wolf CC: Stefan Hajnoczi Signed-off-by: Eric Blake --- v1 was: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg05574.html Since then, we've added the STRUCTURED_REPLY extension, which necessitates a rather larger rebase; I've al

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-04-04 Thread Eric Blake
+++ > 1 file changed, 82 insertions(+) I've posted a v2 of this proposal under a new title, rebased on top of the recent work to add structured replies, and trying to take into account a number of the suggestions that occurred i

Re: [Nbd] [Qemu-devel] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-04-04 Thread Eric Blake
oding up the work on a qemu implementation, and will report on anything that I run into during that process). -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature ---

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
On 04/04/2016 12:06 PM, Alex Bligh wrote: > > On 4 Apr 2016, at 17:39, Eric Blake wrote: > >> +This command is meant to operate in tandem with other (non-NBD) >> +channels to the server. Generally, a "dirty" block is a block >> +that has been

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
" and "this part of the file is dirty". Even for an implementation that DOES track generation id on every sector, you still have the case that you need out-of-band communication to make use of it. So the client would first send an out-of-band message "set generation 123 as my l

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
which is going to be a pain. No, the code for reporting allocation does NOT have to be run at the same time as when reporting dirtiness. > > If these were two different commands, they could each run at their > natural block size. Both modes of operation already can run at their natura

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
fully skip over the variable-length return size reply to NBD_CMD_BLOCK_STATUS, and any other extension command that we might add later. But right now, I'm not seeing a compelling reason to add structured commands to the N

Re: [Nbd] [PATCH 2/2] NBD proto: add GET_LBA_STATUS extension

2016-04-04 Thread Eric Blake
all 0), rather than a set of 3 states, since it really is possible to have all four combinations of those two orthognal status information. That was one of the topics already hashed out in the v1 conversation, and fixed in v2. -- Eric Blake eblake redhat com+1-919-301-3

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
o direct bearing on whether it is dirty or not; and it is feasible that a server could report the act of NBD_CMD_TRIM as causing a portion of the file to become dirty. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Descripti

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
ted state is not replied to command with NBD_CMD_FLAG_STATUS_DIRTY > then why to have different meaning of bits. Because we still have room - no need to overlap the meaning of bits as long as we have more bits to choose from. -- Eric Blake eblake redhat c

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
On 04/04/2016 03:07 PM, Alex Bligh wrote: > > On 4 Apr 2016, at 21:26, Eric Blake wrote: > >> Huh? The current proposal _requires_ these to be separate queries. You >> cannot query dirtiness at the same time as allocation, because the value >> of NBD_CMD_FLAG_DIRTY

[Nbd] [PATCH v2] doc: Allow NBD_CMD_FLAG_NO_HOLE during NBD_CMD_WRITE

2016-04-04 Thread Eric Blake
wanted to write zeroes rather than a hole. So it makes sense to let the new NBD_CMD_FLAG_NO_HOLE work for WRITE as well as WRITE_ZEROES. Signed-off-by: Eric Blake --- v2: fix some typos, add a sentence about server MUST support NBD_CMD_FLAG_NO_HOLE if it advertises NBD_FLAG_SEND_WRITE_ZEROES do

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
rmation separate (NBD_STATUS_HOLE is bit 0, NBD_STATUS_CLEAN is bit 2). I also see that I failed to reserve a bit for NBD_CMD_FLAG_STATUS_DIRTY. Looks like there will still be some more conversation and at least a v3 needed, but I'll wait a couple days for that to happen so that more rev

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-04 Thread Eric Blake
occupy 2 layers of depth: one if dirty, and another if allocated; then deeper layers are determined by backing files). But I'm also worried that it may be more complicated than the original question at hand (qemu wants to know, in advance of a read, which portions of a file are worth read

Re: [Nbd] NBD_OPT_GO

2016-04-05 Thread Eric Blake
y handle the empty name as selecting the default export, but NBD_OPT_GO would be ambiguous as to whether you are using it without a name option, or if you are using the optional name option to explicitly select the default export. -- Eric Blake eblake redhat com+1-919-301-3266 Libvir

Re: [Nbd] [PATCH v2] doc: Add NBD_CMD_BLOCK_STATUS extension

2016-04-05 Thread Eric Blake
ed in > the backend and so on. Is it safe to cache this stuff? If the client is the only thing modifying the drive, maybe we want to make that additional constraint on the server. But how best to word it, or is it too tight of a specification? > > Until now something like READ and WRITE w

Re: [Nbd] [PATCHv3] Improve the documentation of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA.

2016-04-05 Thread Eric Blake
orce Unit Access") was set in the > -transmission flags field, the client MAY set the flag `NBD_CMD_FLAG_FUA` > in > -the command flags field. If this flag was set, the server MUST NOT send > -the reply until it has ensured that the newly-written data has reached >

  1   2   3   4   >