[Qemu-block] [RFC 0/5] block: Generic file truncation/creation fallbacks

2019-07-11 Thread Max Reitz
Hi, Some protocol drivers do not really support file truncation but still implement .bdrv_co_truncate(): They just don’t do anything when asked to shrink a file. This is reflected by qemu-img, which warns if you resize a file and it has the exact same length afterwards as it had before. We can

[Qemu-block] [RFC 1/5] block/nbd: Fix hang in .bdrv_close()

2019-07-11 Thread Max Reitz
When nbd_close() is called from a coroutine, the connection_co never gets to run, and thus nbd_teardown_connection() hangs. This is because aio_co_enter() only puts the connection_co into the main coroutine's wake-up queue, so this main coroutine needs to yield and reschedule itself to let the

[Qemu-block] [RFC 5/5] iotests: Add test for fallback truncate/create

2019-07-11 Thread Max Reitz
Signed-off-by: Max Reitz --- tests/qemu-iotests/259 | 71 ++ tests/qemu-iotests/259.out | 20 +++ tests/qemu-iotests/group | 1 + 3 files changed, 92 insertions(+) create mode 100755 tests/qemu-iotests/259 create mode 100644

[Qemu-block] [RFC 4/5] block: Generic file creation fallback

2019-07-11 Thread Max Reitz
If a protocol driver does not support image creation, we can see whether maybe the file exists already. If so, just truncating it will be sufficient. Signed-off-by: Max Reitz --- block.c | 77 - 1 file changed, 65 insertions(+), 12

[Qemu-block] [RFC 3/5] block: Fall back to fallback truncate function

2019-07-11 Thread Max Reitz
file-posix does not need to basically duplicate our fallback truncate implementation; and sheepdog can fall back to it for "shrinking" files. Signed-off-by: Max Reitz --- block/file-posix.c | 21 + block/sheepdog.c | 2 +- 2 files changed, 2 insertions(+), 21 deletions(-)

[Qemu-block] [RFC 2/5] block: Generic truncation fallback

2019-07-11 Thread Max Reitz
If a protocol driver does not support truncation, we call fall back to effectively not doing anything if the new size is less than the actual file size. This is what we have been doing for some host device drivers already. The only caveat is that we have to zero out everything in the first

Re: [Qemu-block] [PATCH 8/8] iotests/257: test traditional sync modes

2019-07-11 Thread John Snow
On 7/11/19 8:37 AM, Max Reitz wrote: > On 11.07.19 05:21, John Snow wrote: >> >> On 7/10/19 4:46 PM, Max Reitz wrote: >>> On 10.07.19 21:00, John Snow wrote: On 7/10/19 1:14 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: > > Hm. How useful is bitmap support for

Re: [Qemu-block] [PATCH-for-4.1 v4 2/5] hw/block/pflash_cfi01: Use the correct READ_ARRAY value

2019-07-11 Thread Dr. David Alan Gilbert
* Philippe Mathieu-Daudé (phi...@redhat.com) wrote: > In the "Read Array Flowchart" the command has a value of 0xFF. > > In the document [*] the "Read Array Flowchart", the READ_ARRAY > command has a value of 0xff. > > Use the correct value in the pflash model. > > There is no change of

Re: [Qemu-block] [PATCH v3] LUKS: support preallocation

2019-07-11 Thread Stefano Garzarella
On Thu, Jul 11, 2019 at 06:09:40PM +0300, Maxim Levitsky wrote: > preallocation=off and preallocation=metadata > both allocate luks header only, and preallocation=falloc/full > is passed to underlying file. > > Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1534951 > > Signed-off-by: Maxim

[Qemu-block] [PATCH-for-4.1 v4 5/5] hw/block/pflash_cfi01: Add the DeviceReset() handler

2019-07-11 Thread Philippe Mathieu-Daudé
A "system reset" sets the device state machine in READ_ARRAY mode and, after some delay, set the SR.7 READY bit. We do not model timings, so we set the SR.7 bit directly. The TYPE_DEVICE interface provides a DeviceReset handler. This pflash device is a subclass of TYPE_SYS_BUS_DEVICE (which is a

[Qemu-block] [PATCH-for-4.1 v4 3/5] hw/block/pflash_cfi01: Extract pflash_mode_read_array()

2019-07-11 Thread Philippe Mathieu-Daudé
The same pattern is used when setting the flash in READ_ARRAY mode: - Set the state machine command to READ_ARRAY - Reset the write_cycle counter - Reset the memory region in ROMD Refactor the current code by extracting this pattern. It is used twice: - On a write access (on command failure,

[Qemu-block] [PATCH-for-4.1 v4 4/5] hw/block/pflash_cfi01: Start state machine as READY to accept commands

2019-07-11 Thread Philippe Mathieu-Daudé
When the state machine is ready to accept command, the bit 7 of the status register (SR) is set to 1. The guest polls the status register and check this bit before writting command to the internal 'Write State Machine' (WSM). Set SR.7 bit to 1 when the device is created. There is no migration

[Qemu-block] [PATCH-for-4.1 v4 2/5] hw/block/pflash_cfi01: Use the correct READ_ARRAY value

2019-07-11 Thread Philippe Mathieu-Daudé
In the "Read Array Flowchart" the command has a value of 0xFF. In the document [*] the "Read Array Flowchart", the READ_ARRAY command has a value of 0xff. Use the correct value in the pflash model. There is no change of behavior in the guest, because: - when the guest were sending 0xFF, the

[Qemu-block] [PATCH-for-4.1 v4 0/5] hw/block/pflash_cfi01: Add DeviceReset() handler

2019-07-11 Thread Philippe Mathieu-Daudé
The pflash device lacks a reset() function. When a machine is resetted, the flash might be in an inconsistent state, leading to unexpected behavior: https://bugzilla.redhat.com/show_bug.cgi?id=1678713 Resolve this issue by adding a DeviceReset() handler. Fix also two minor issues, and clean a bit

[Qemu-block] [PATCH-for-4.1 v4 1/5] hw/block/pflash_cfi01: Removed an unused timer

2019-07-11 Thread Philippe Mathieu-Daudé
The 'CFI02' NOR flash was introduced in commit 29133e9a0fff, with timing modelled. One year later, the CFI01 model was introduced (commit 05ee37ebf630) based on the CFI02 model. As noted in the header, "It does not support timings". 12 years later, we never had to model the device timings. Time to

[Qemu-block] [PATCH v3] LUKS: support preallocation

2019-07-11 Thread Maxim Levitsky
preallocation=off and preallocation=metadata both allocate luks header only, and preallocation=falloc/full is passed to underlying file. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1534951 Signed-off-by: Maxim Levitsky --- Note that QMP support was only compile tested, since I am still

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Alberto Garcia
On Thu 11 Jul 2019 04:32:34 PM CEST, Kevin Wolf wrote: >> - It is possible to configure very easily the number of subclusters per >> cluster. It is now hardcoded to 32 in qcow2_do_open() but any power of >> 2 would work (just change the number there if you want to test >> it). Would an

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Kevin Wolf
Am 11.07.2019 um 16:08 hat Alberto Garcia geschrieben: > Some questions that are still open: > > - It is possible to configure very easily the number of subclusters per > cluster. It is now hardcoded to 32 in qcow2_do_open() but any power of > 2 would work (just change the number there if you

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Kevin Wolf
Am 11.07.2019 um 15:50 hat Eric Blake geschrieben: > On 7/11/19 7:24 AM, Max Reitz wrote: > > >>> So it isn’t just me who expects these to pre-initialize the image to 0. > >>> Hm, although... I suppose @falloc technically does not specify whether > >>> the data reads as zeroes. I kind of find

Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images

2019-07-11 Thread Alberto Garcia
llocated" and 32 for "subcluster is all zeroes"). I also fixed a few bugs on the way and started to clean the code a bit so it is more readable. You can get it here: https://github.com/bertogg/qemu/releases/tag/subcluster-allocation-prototype-20190711 The idea is that yo

Re: [Qemu-block] [PATCH v2] LUKS: support preallocation in qemu-img

2019-07-11 Thread Maxim Levitsky
On Thu, 2019-07-11 at 15:43 +0200, Max Reitz wrote: > On 11.07.19 11:11, Maxim Levitsky wrote: > > preallocation=off and preallocation=metadata > > both allocate luks header only, and preallocation=falloc/full > > is passed to underlying file. > > > > Fixes:

Re: [Qemu-block] [PATCH] doc: Preallocation does not require writing zeroes

2019-07-11 Thread Maxim Levitsky
On Thu, 2019-07-11 at 15:29 +0200, Max Reitz wrote: > When preallocating an encrypted qcow2 image, it just lets the protocol > driver write data and then does not mark the clusters as zero. > Therefore, reading this image will yield effectively random data. > > As such, we have not fulfilled the

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Daniel P . Berrangé
On Thu, Jul 11, 2019 at 08:50:56AM -0500, Eric Blake wrote: > On 7/11/19 7:24 AM, Max Reitz wrote: > > >>> So it isn’t just me who expects these to pre-initialize the image to 0. > >>> Hm, although... I suppose @falloc technically does not specify whether > >>> the data reads as zeroes. I kind

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Eric Blake
On 7/11/19 7:24 AM, Max Reitz wrote: >>> So it isn’t just me who expects these to pre-initialize the image to 0. >>> Hm, although... I suppose @falloc technically does not specify whether >>> the data reads as zeroes. I kind of find it to be implied, but, well... >> >> I personally don't

Re: [Qemu-block] [PATCH v2] LUKS: support preallocation in qemu-img

2019-07-11 Thread Max Reitz
On 11.07.19 11:11, Maxim Levitsky wrote: > preallocation=off and preallocation=metadata > both allocate luks header only, and preallocation=falloc/full > is passed to underlying file. > > Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1534951 > > Signed-off-by: Maxim Levitsky > --- >

Re: [Qemu-block] [Qemu-devel] [PATCH] doc: Preallocation does not require writing zeroes

2019-07-11 Thread Daniel P . Berrangé
On Thu, Jul 11, 2019 at 03:29:35PM +0200, Max Reitz wrote: > When preallocating an encrypted qcow2 image, it just lets the protocol > driver write data and then does not mark the clusters as zero. > Therefore, reading this image will yield effectively random data. > > As such, we have not

Re: [Qemu-block] [PATCH] doc: Preallocation does not require writing zeroes

2019-07-11 Thread Eric Blake
On 7/11/19 8:29 AM, Max Reitz wrote: > When preallocating an encrypted qcow2 image, it just lets the protocol > driver write data and then does not mark the clusters as zero. > Therefore, reading this image will yield effectively random data. > > As such, we have not fulfilled the promise of

[Qemu-block] [PATCH] doc: Preallocation does not require writing zeroes

2019-07-11 Thread Max Reitz
When preallocating an encrypted qcow2 image, it just lets the protocol driver write data and then does not mark the clusters as zero. Therefore, reading this image will yield effectively random data. As such, we have not fulfilled the promise of always writing zeroes when preallocating an image

Re: [Qemu-block] [PATCH-for-4.1] hw/block/pflash_cfi02: Explicit switch fallthrough for ERASE commands

2019-07-11 Thread Peter Maydell
On Thu, 11 Jul 2019 at 14:08, Philippe Mathieu-Daudé wrote: > > Previous to commit ddb6f2254, the DQ2 bit was incorrectly set > during PROGRAM command (0xA0). The commit reordered the switch > cases to only set the DQ2 bit for the ERASE commands using a > fallthrough, but did not explicit the

[Qemu-block] [PATCH-for-4.1] hw/block/pflash_cfi02: Explicit switch fallthrough for ERASE commands

2019-07-11 Thread Philippe Mathieu-Daudé
Previous to commit ddb6f2254, the DQ2 bit was incorrectly set during PROGRAM command (0xA0). The commit reordered the switch cases to only set the DQ2 bit for the ERASE commands using a fallthrough, but did not explicit the fallthrough is intentional. Mark the switch fallthrough with a comment

Re: [Qemu-block] [Qemu-devel] [PATCH v6] qemu-io: add pattern file for write command

2019-07-11 Thread Max Reitz
On 27.06.19 17:26, Denis Plotnikov wrote: > > > On 19.06.2019 13:09, Vladimir Sementsov-Ogievskiy wrote: >> 10.06.2019 16:21, Denis Plotnikov wrote: >>> The patch allows to provide a pattern file for write >>> command. There was no similar ability before. >>> >>> Signed-off-by: Denis Plotnikov

Re: [Qemu-block] [Qemu-devel] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Daniel P . Berrangé
On Thu, Jul 11, 2019 at 02:23:55PM +0200, Max Reitz wrote: > On 11.07.19 11:20, Daniel P. Berrangé wrote: > > On Wed, Jul 10, 2019 at 11:24:46PM +0200, Max Reitz wrote: > >> On 10.07.19 19:03, Maxim Levitsky wrote: > >>> preallocation=off and preallocation=metadata > >>> both allocate luks header

Re: [Qemu-block] [Qemu-devel] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Max Reitz
On 11.07.19 14:23, Max Reitz wrote: > On 11.07.19 11:20, Daniel P. Berrangé wrote: >> On Wed, Jul 10, 2019 at 11:24:46PM +0200, Max Reitz wrote: [...] >>> Hm. I would expect a preallocated image to read 0. But if you just >>> pass this through to the protocol layer, it won’t read 0. >> >> Yes,

Re: [Qemu-block] [Qemu-devel] [PULL 24/27] hw/block/pflash_cfi02: Implement erase suspend/resume

2019-07-11 Thread Philippe Mathieu-Daudé
On 7/11/19 2:35 PM, Peter Maydell wrote: > On Tue, 2 Jul 2019 at 04:29, Philippe Mathieu-Daudé wrote: >> >> From: Stephen Checkoway >> >> During a sector erase (but not a chip erase), the embeded erase program >> can be suspended. Once suspended, the sectors not selected for erasure >> may be

Re: [Qemu-block] [PATCH 8/8] iotests/257: test traditional sync modes

2019-07-11 Thread Max Reitz
On 11.07.19 05:21, John Snow wrote: > > On 7/10/19 4:46 PM, Max Reitz wrote: >> On 10.07.19 21:00, John Snow wrote: >>> On 7/10/19 1:14 PM, Max Reitz wrote: On 10.07.19 03:05, John Snow wrote: Hm. How useful is bitmap support for 'top' then, anyway? That means that if you

Re: [Qemu-block] [Qemu-devel] [PULL 24/27] hw/block/pflash_cfi02: Implement erase suspend/resume

2019-07-11 Thread Peter Maydell
On Tue, 2 Jul 2019 at 04:29, Philippe Mathieu-Daudé wrote: > > From: Stephen Checkoway > > During a sector erase (but not a chip erase), the embeded erase program > can be suspended. Once suspended, the sectors not selected for erasure > may be read and programmed. Autoselect mode is allowed

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Max Reitz
On 11.07.19 10:39, Maxim Levitsky wrote: > On Wed, 2019-07-10 at 23:52 +0200, Max Reitz wrote: >> On 10.07.19 23:24, Max Reitz wrote: >>> On 10.07.19 19:03, Maxim Levitsky wrote: preallocation=off and preallocation=metadata both allocate luks header only, and preallocation=falloc/full

Re: [Qemu-block] [Qemu-devel] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Max Reitz
On 11.07.19 11:20, Daniel P. Berrangé wrote: > On Wed, Jul 10, 2019 at 11:24:46PM +0200, Max Reitz wrote: >> On 10.07.19 19:03, Maxim Levitsky wrote: >>> preallocation=off and preallocation=metadata >>> both allocate luks header only, and preallocation=falloc/full >>> is passed to underlying file,

Re: [Qemu-block] [PULL 0/3] Bitmaps patches

2019-07-11 Thread Peter Maydell
On Wed, 10 Jul 2019 at 20:23, John Snow wrote: > > The following changes since commit 6df2cdf44a82426f7a59dcb03f0dd2181ed7fdfa: > > Update version for v4.1.0-rc0 release (2019-07-09 17:21:53 +0100) > > are available in the Git repository at: > > https://github.com/jnsnow/qemu.git

Re: [Qemu-block] [Qemu-devel] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Daniel P . Berrangé
On Wed, Jul 10, 2019 at 11:24:46PM +0200, Max Reitz wrote: > On 10.07.19 19:03, Maxim Levitsky wrote: > > preallocation=off and preallocation=metadata > > both allocate luks header only, and preallocation=falloc/full > > is passed to underlying file, with the given image size. > > > > Note that

[Qemu-block] [PATCH v2] LUKS: support preallocation in qemu-img

2019-07-11 Thread Maxim Levitsky
preallocation=off and preallocation=metadata both allocate luks header only, and preallocation=falloc/full is passed to underlying file. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1534951 Signed-off-by: Maxim Levitsky --- block/crypto.c | 25 ++--- 1 file changed,

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-11 Thread Maxim Levitsky
On Wed, 2019-07-10 at 23:52 +0200, Max Reitz wrote: > On 10.07.19 23:24, Max Reitz wrote: > > On 10.07.19 19:03, Maxim Levitsky wrote: > > > preallocation=off and preallocation=metadata > > > both allocate luks header only, and preallocation=falloc/full > > > is passed to underlying file, with the

Re: [Qemu-block] [PATCH 0/4] virtio: handle zoned backing devices

2019-07-11 Thread Paolo Bonzini
On 11/07/19 02:52, Dmitry Fomichev wrote: > Paolo, > WRT to Host Aware drives, these MAY work, but we don't have any of these > available for testing and are not able to verify which drivers do work > with them and which do not. This is the reason for not letting them pass > thru. If you prefer, I