From: Alberto Garcia
This function is only used by qcow2_expand_zero_clusters() to
downgrade a qcow2 image to a previous version. This would require
transforming all extended L2 entries into normal L2 entries but this
is not a simple task and there are no plans to implement this at the
moment.
From: Alberto Garcia
This function returns an integer that can be either an error code or a
cluster type (a value from the QCow2ClusterType enum).
We are going to start using subcluster types instead of cluster types
in some functions so it's better to use the exact data types instead
of
From: Alberto Garcia
Compressed clusters always have the bitmap part of the extended L2
entry set to 0.
Signed-off-by: Alberto Garcia
Reviewed-by: Max Reitz
Message-Id:
<04455b3de5dfeb9d1cfe1fc7b02d7060a6e09710.1594396418.git.be...@igalia.com>
Signed-off-by: Max Reitz
---
On 21.08.20 01:54, Nir Soffer wrote:
> When this class was extracted from FilePath, the docstring was not
> updated for generating multiple files, and the example usage was
> referencing unrelated file.
>
> Fixes: de263986b5dc
> Signed-off-by: Nir Soffer
> ---
> tests/qemu-iotests/iotests.py |
On Wed, Aug 19, 2020 at 08:12:08PM -0400, Eduardo Habkost wrote:
> The TYPE_* constants and the typedefs are defined in ahci.h, so
> we can move the type checking macros there too.
>
> This will make future conversion to OBJECT_DECLARE* easier.
>
> Signed-off-by: Eduardo Habkost
> ---
> Changes
Am 21.08.2020 um 01:54 hat Nir Soffer geschrieben:
> Fix some issues introduced when iotests.FilePaths was added and merge it back
> into FilePath keeping the option to create multiple file names.
Reviewed-by: Kevin Wolf
From: Alberto Garcia
Signed-off-by: Alberto Garcia
Reviewed-by: Max Reitz
Message-Id:
Signed-off-by: Max Reitz
---
tests/qemu-iotests/271 | 901 +
tests/qemu-iotests/271.out | 726 ++
tests/qemu-iotests/group | 1 +
3
From: Alberto Garcia
In order to support extended L2 entries some functions of the qcow2
driver need to start dealing with subclusters instead of clusters.
qcow2_get_host_offset() is modified to return the subcluster type
instead of the cluster type, and all callers are updated to replace
all
From: Alberto Garcia
Traditional qcow2 images don't allow preallocation if a backing file
is set. This is because once a cluster is allocated there is no way to
tell that its data should be read from the backing file.
Extended L2 entries have individual allocation bits for each
subcluster, and
From: Alberto Garcia
The bdrv_co_pwrite_zeroes() call here fills complete clusters with
zeroes, but it can happen that some subclusters are not part of the
write request or the copy-on-write. This patch makes sure that only
the affected subclusters are overwritten.
A potential improvement would
From: Alberto Garcia
The QCOW_OFLAG_ZERO bit that indicates that a cluster reads as
zeroes is only used in standard L2 entries. Extended L2 entries use
individual 'all zeroes' bits for each subcluster.
This must be taken into account when updating the L2 entry and also
when deciding that an
From: Alberto Garcia
This field allows us to indicate that the L2 metadata update does not
come from a write request with actual data but from a preallocation
request.
For traditional images this does not make any difference, but for
images with extended L2 entries this means that the clusters
On 21.08.20 01:54, Nir Soffer wrote:
> When this class was extracted from FilePath, the docstring was not
> updated for generating multiple files, and the example usage was
> referencing unrelated file.
>
> Fixes: de263986b5dc
> Signed-off-by: Nir Soffer
> ---
> tests/qemu-iotests/iotests.py |
On 21.08.20 01:54, Nir Soffer wrote:
> FilePath creates now one temporary file:
>
> with FilePath("a") as a:
>
> Or more:
>
> with FilePath("a", "b", "c") as (a, b, c):
>
> This is also the behavior of the file_path() helper, used by some of the
> tests. Now we have only 2 helpers for
From: Alberto Garcia
This patch adds QCow2SubclusterType, which is the subcluster-level
version of QCow2ClusterType. All QCOW2_SUBCLUSTER_* values have the
the same meaning as their QCOW2_CLUSTER_* equivalents (when they
exist). See below for details and caveats.
In images without extended L2
From: Alberto Garcia
Extended L2 entries are 128-bit wide: 64 bits for the entry itself and
64 bits for the subcluster allocation bitmap.
In order to support them correctly get/set_l2_entry() need to be
updated so they take the entry width into account in order to
calculate the correct offset.
From: Alberto Garcia
This function will be used by the qcow2 code to check if an image has
subclusters or not.
At the moment this simply returns false. Once all patches needed for
subcluster support are ready then QEMU will be able to create and
read images with subclusters and this function
On 21.08.20 01:54, Nir Soffer wrote:
> If os.remove() fails to remove one of the paths, for example if the file
> was removed by the test, the cleanup loop would exit silently, without
> removing the rest of the files.
>
> Fixes: de263986b5dc
> Signed-off-by: Nir Soffer
> ---
>
On 25.08.2020 09:37, Markus Armbruster wrote:
Andrey Shinkevich writes:
Provide the possibility to pass the 'filter-node-name' parameter to the
block-stream job as it is done for the commit block job.
Signed-off-by: Andrey Shinkevich
Reviewed-by: Vladimir Sementsov-Ogievskiy
[...]
diff
On 21.08.20 01:54, Nir Soffer wrote:
> Use list comprehension instead of append loop.
>
> Signed-off-by: Nir Soffer
> ---
> tests/qemu-iotests/iotests.py | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
Reviewed-by: Max Reitz
signature.asc
Description: OpenPGP digital signature
On Wed, Aug 19, 2020 at 08:12:10PM -0400, Eduardo Habkost wrote:
> Move the ALLWINNER_AHCI macro close to the TYPE_ALLWINNER_AHCI
> define.
>
> This will make future conversion to OBJECT_DECLARE* easier.
>
> Signed-off-by: Eduardo Habkost
> ---
> Changes series v1 -> v2: new patch in series v2
From: Alberto Garcia
qcow2 images with subclusters have 128-bit L2 entries. The first 64
bits contain the same information as traditional images and the last
64 bits form a bitmap with the status of each individual subcluster.
Because of that we cannot assume that L2 entries are
From: Alberto Garcia
For a given offset, return the subcluster number within its cluster
(i.e. with 32 subclusters per cluster it returns a number between 0
and 31).
Signed-off-by: Alberto Garcia
Reviewed-by: Max Reitz
Reviewed-by: Vladimir Sementsov-Ogievskiy
Message-Id:
From: Alberto Garcia
This patch adds the following new fields to BDRVQcow2State:
- subclusters_per_cluster: Number of subclusters in a cluster
- subcluster_size: The size of each subcluster, in bytes
- subcluster_bits: No. of bits so 1 << subcluster_bits = subcluster_size
Images without
On 19.08.20 16:19, David Edmondson wrote:
> On Wednesday, 2020-08-19 at 15:11:37 +01, Stefan Hajnoczi wrote:
>
>> On Tue, Aug 18, 2020 at 12:08:36PM +0100, David Edmondson wrote:
>>> When using qemu-img to convert an image that is hosted on an HTTP
>>> server to some faster local (or
On Mon, Aug 24, 2020 at 12:12:53PM -0500, Eric Blake wrote:
> On 8/24/20 12:02 PM, Daniel P. Berrangé wrote:
> > Disabling these parts are sufficient to get the qemu-nbd program
> > compiling in a Windows build.
> >
> > Signed-off-by: Daniel P. Berrangé
> > ---
> > meson.build | 7 ++-
> >
On Tue, Aug 25, 2020 at 1:48 PM Max Reitz wrote:
>
> On 21.08.20 01:54, Nir Soffer wrote:
> > When this class was extracted from FilePath, the docstring was not
> > updated for generating multiple files, and the example usage was
> > referencing unrelated file.
> >
> > Fixes: de263986b5dc
> >
From: Alberto Garcia
Like offset_into_cluster() and size_to_clusters(), but for
subclusters.
Signed-off-by: Alberto Garcia
Reviewed-by: Eric Blake
Reviewed-by: Max Reitz
Message-Id:
<3cc2390dcdef3d234d47c741b708bd8734490862.1594396418.git.be...@igalia.com>
Signed-off-by: Max Reitz
---
From: Alberto Garcia
If an image has subclusters then there are more copy-on-write
scenarios that we need to consider. Let's say we have a write request
from the middle of subcluster #3 until the end of the cluster:
1) If we are writing to a newly allocated cluster then we need
From: Alberto Garcia
Two things need to be taken into account here:
1) With full_discard == true the L2 entry must be cleared completely.
This also includes the L2 bitmap if the image has extended L2
entries.
2) With full_discard == false we have to make the discarded cluster
read
From: Alberto Garcia
The L2 bitmap needs to be updated after each write to indicate what
new subclusters are now allocated. This needs to happen even if the
cluster was already allocated and the L2 entry was otherwise valid.
In some cases however a write operation doesn't need change the L2
From: Alberto Garcia
This helper function tells us if a cluster is allocated (that is,
there is an associated host offset for it).
Signed-off-by: Alberto Garcia
Reviewed-by: Eric Blake
Reviewed-by: Max Reitz
Message-Id:
Am 23.08.2020 um 16:46 hat Nir Soffer geschrieben:
> Using -o preallocation falloc works great on NFS 4.2 and local file system,
> when fallocate() is supported, but when it is not, posix_fallocate falls back
> to very inefficient way:
>
Any tool that uses sockets needs to call socket_init() in order to work
on the Windows platform.
Reviewed-by: Eric Blake
Signed-off-by: Daniel P. Berrangé
---
qemu-img.c | 2 ++
qemu-io.c | 2 ++
qemu-nbd.c | 1 +
3 files changed, 5 insertions(+)
diff --git a/qemu-img.c b/qemu-img.c
index
We are already building the NBD client and server on Windows when it is
used via the main system emulator binaries. This demonstrates there is
no fundamental blocker to buildig the qemu-nbd binary too.
Changed in v2:
- Split second patch into two parts
- Use HAVE_NBD_DEVICE condition to
Andrey Shinkevich writes:
> Provide the possibility to pass the 'filter-node-name' parameter to the
> block-stream job as it is done for the commit block job.
>
> Signed-off-by: Andrey Shinkevich
> Reviewed-by: Vladimir Sementsov-Ogievskiy
[...]
> diff --git a/qapi/block-core.json
From: Alberto Garcia
The file_cluster_offset field of Qcow2AioTask stores a cluster-aligned
host offset. In practice this is not very useful because all users(*)
of this structure need the final host offset into the cluster, which
they calculate using
host_offset = file_cluster_offset +
From: Alberto Garcia
Subcluster allocation in qcow2 is implemented by extending the
existing L2 table entries and adding additional information to
indicate the allocation status of each subcluster.
This patch documents the changes to the qcow2 format and how they
affect the calculation of the
From: Alberto Garcia
The size of an L2 entry is 64 bits, but if we want to have subclusters
we need extended L2 entries. This means that we have to access L2
tables and slices differently depending on whether an image has
extended L2 entries or not.
This patch replaces all l2_slice[] accesses
The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1:
Merge remote-tracking branch 'remotes/cschoenebeck/tags/pull-9p-20200812'
into staging (2020-08-24 16:39:53 +0100)
are available in the Git repository at:
https://github.com/XanClic/qemu.git
From: Alberto Garcia
qcow2_get_cluster_offset() takes an (unaligned) guest offset and
returns the (aligned) offset of the corresponding cluster in the qcow2
image.
In practice none of the callers need to know where the cluster starts
so this patch makes the function calculate and return the
From: Alberto Garcia
handle_alloc() creates a QCowL2Meta structure in order to update the
image metadata and perform the necessary copy-on-write operations.
This patch moves that code to a separate function so it can be used
from other places.
Signed-off-by: Alberto Garcia
Reviewed-by: Max
From: Alberto Garcia
This works now at the subcluster level and pwrite_zeroes_alignment is
updated accordingly.
qcow2_cluster_zeroize() is turned into qcow2_subcluster_zeroize() with
the following changes:
- The request can now be subcluster-aligned.
- The cluster-aligned body of the
From: Alberto Garcia
Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with this feature,
which we call "extended L2 entries".
Signed-off-by: Alberto Garcia
Reviewed-by: Eric Blake
Reviewed-by: Max Reitz
Message-Id:
From: Alberto Garcia
When dealing with subcluster types there is a new value called
QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in
QCow2ClusterType.
This patch handles that value in all places where subcluster types
are processed.
Signed-off-by: Alberto Garcia
Reviewed-by: Max
From: Alberto Garcia
There are situations in which we want to know how many contiguous
subclusters of the same type there are in a given cluster. This can be
done by simply iterating over the subclusters and repeatedly calling
qcow2_get_subcluster_type() for each one of them.
However once we
From: Alberto Garcia
When writing to a qcow2 file there are two functions that take a
virtual offset and return a host offset, possibly allocating new
clusters if necessary:
- handle_copied() looks for normal data clusters that are already
allocated and have a reference count of 1. In
From: Alberto Garcia
We are going to need it in other places.
Signed-off-by: Alberto Garcia
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Max Reitz
Message-Id:
<65e5d9627ca2ebe7e62deaeddf60949c33067d9d.1594396418.git.be...@igalia.com>
Signed-off-by: Max Reitz
---
From: Alberto Garcia
The logic of this function remains pretty much the same, except that
it uses count_contiguous_subclusters(), which combines the logic of
count_contiguous_clusters() / count_contiguous_clusters_unallocated()
and checks individual subclusters.
From: Alberto Garcia
Extended L2 entries are bigger than normal L2 entries so this has an
impact on the amount of metadata needed for a qcow2 file.
Signed-off-by: Alberto Garcia
Reviewed-by: Max Reitz
Message-Id:
<7efae2efd5e36b42d2570743a12576d68ce53685.1594396418.git.be...@igalia.com>
On 21.08.20 01:54, Nir Soffer wrote:
> Accept variable number of names instead of a sequence:
>
> with FilePaths("a", "b", "c") as (a, b, c):
>
> The disadvantage is that base_dir must be used as kwarg:
>
> with FilePaths("a", "b", base_dir=soc_dir) as (sock1, sock2):
>
> But this is
From: Alberto Garcia
The offset field of an uncompressed cluster's L2 entry must be aligned
to the cluster size, otherwise it is invalid. If the cluster has no
data then it means that the offset points to a preallocation, so we
can clear the offset field without affecting the guest-visible data.
The termsig_handler function is used by the client thread handling the
host NBD device connection to do a graceful shutdown. IOW, if we have
disabled NBD device support at compile time, we don't need the SIGTERM
handler. This fixes a build issue for Windows.
Signed-off-by: Daniel P. Berrangé
---
Disabling these parts are sufficient to get the qemu-nbd program
compiling in a Windows build.
Signed-off-by: Daniel P. Berrangé
---
meson.build | 7 ++-
qemu-nbd.c | 5 +
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/meson.build b/meson.build
index
On Sat, Aug 22, 2020 at 11:10 AM Philippe Mathieu-Daudé wrote:
>
> Most uses of qemu_hexdump() do not take an array of char
> as input, forcing use of cast. Since we can use this
> helper to dump any kind of buffer, use a pointer to void
> argument instead.
>
> Signed-off-by: Philippe
On Tue, Aug 25, 2020 at 02:24:58PM +0200, Alberto Garcia wrote:
> On Fri 21 Aug 2020 07:02:32 PM CEST, Brian Foster wrote:
> >> I was running fio with --ramp_time=5 which ignores the first 5 seconds
> >> of data in order to let performance settle, but if I remove that I can
> >> see the effect
On Tue 25 Aug 2020 06:54:15 PM CEST, Brian Foster wrote:
> If I compare this 5m fio test between XFS and ext4 on a couple of my
> systems (with either no prealloc or full file prealloc), I end up seeing
> ext4 run slightly faster on my vm and XFS slightly faster on bare metal.
> Either way, I
Daniel P. Berrangé writes:
> This simple refactoring prepares for future patches. The variadic args
> handling is split from the main bulk of the open logic. The duplicated
> calls to open() are removed in favour of updating the "flags" variable
> to have O_CLOEXEC.
>
> Signed-off-by: Daniel P.
Daniel P. Berrangé writes:
> qemu_open_old() works like open(): set errno and return -1 on failure.
> It has even more failure modes, though. Reporting the error clearly
> to users is basically impossible for many of them.
>
> Our standard cure for "errno is too coarse" is the Error object.
>
Daniel P. Berrangé writes:
> Currently at startup if using cache=none on a filesystem lacking
> O_DIRECT such as tmpfs, at startup QEMU prints
>
> qemu-system-x86_64: -drive file=/tmp/foo.img,cache=none: file system may not
> support O_DIRECT
> qemu-system-x86_64: -drive
On Tue, 25 Aug 2020 at 09:33, Max Reitz wrote:
>
> The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1:
>
> Merge remote-tracking branch 'remotes/cschoenebeck/tags/pull-9p-20200812'
> into staging (2020-08-24 16:39:53 +0100)
>
> are available in the Git repository at:
>
On Tue, Aug 25, 2020 at 04:56:40PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé writes:
>
> > This simple refactoring prepares for future patches. The variadic args
> > handling is split from the main bulk of the open logic. The duplicated
> > calls to open() are removed in favour of
Daniel P. Berrangé writes:
> Instead of relying on the limited information from errno, we can now
> also provide detailed error messages.
The more detailed error messages are currently always ignored, but the
next patches will fix that.
> Signed-off-by: Daniel P. Berrangé
> ---
>
On Mon, Aug 24, 2020 at 06:02:16PM +0100, Daniel P. Berrangé wrote:
> We are already building the NBD client and server on Windows when it is
> used via the main system emulator binaries. This demonstrates there is
> no fundamental blocker to buildig the qemu-nbd binary too.
>
>
> In testing
Daniel P. Berrangé writes:
> A common error scenario is to tell QEMU to use O_DIRECT in combination
> with a filesystem that doesn't support it. To aid users to diagnosing
> their mistake we want to provide a clear error message when this happens.
>
> Reviewed-by: Eric Blake
> Signed-off-by:
On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote:
> We'll need a separate function, which will only "mark" request
> serialising with specified align but not wait for conflicting
> requests. So, it will be like old bdrv_mark_request_serialising(),
> before merging
On Tue 25 Aug 2020 04:01:14 PM CEST, Peter Maydell
wrote:
> On Tue, 25 Aug 2020 at 09:33, Max Reitz wrote:
>>
>> The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1:
>>
>> Merge remote-tracking branch 'remotes/cschoenebeck/tags/pull-9p-20200812'
>> into staging
On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote:
> Add flag to make serialising request no wait: if there are conflicting
> requests, just return error immediately. It's will be used in upcoming
> preallocate filter.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy
> ---
>
On 25.08.20 16:22, Alberto Garcia wrote:
> On Tue 25 Aug 2020 04:01:14 PM CEST, Peter Maydell
> wrote:
>> On Tue, 25 Aug 2020 at 09:33, Max Reitz wrote:
>>>
>>> The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1:
>>>
>>> Merge remote-tracking branch
On Fri 21 Aug 2020 07:02:32 PM CEST, Brian Foster wrote:
>> I was running fio with --ramp_time=5 which ignores the first 5 seconds
>> of data in order to let performance settle, but if I remove that I can
>> see the effect more clearly. I can observe it with raw files (in 'off'
>> and 'prealloc'
On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote:
> It's intended to be inserted between format and protocol nodes to
> preallocate additional space (expanding protocol file) on writes
> crossing EOF. It improves performance for file-systems with slow
> allocation.
>
> Signed-off-by:
On Tue, Aug 25, 2020 at 05:19:53PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé writes:
>
> > A common error scenario is to tell QEMU to use O_DIRECT in combination
> > with a filesystem that doesn't support it. To aid users to diagnosing
> > their mistake we want to provide a clear
On Tue, Aug 25, 2020 at 05:14:21PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé writes:
>
> > Instead of relying on the limited information from errno, we can now
> > also provide detailed error messages.
>
> The more detailed error messages are currently always ignored, but the
> next
This will remove instance_size/class_size fields from TypeInfo
variables when the value is exactly the same as the one in the
parent class.
Generated by:
$ ./scripts/codeconverter/converter.py -i \
--pattern=RedundantTypeSizes $(git grep -l TypeInfo -- '*.[ch]')
Signed-off-by: Eduardo
On 8/13/20 8:47 AM, Connor Kuehl wrote:
Providing an empty string for the backing file parameter like so:
qemu-img create -f qcow2 -b '' /tmp/foo
allows the flow of control to reach and subsequently fail an assert
statement because passing an empty string to
Move typedef closer to the type check macros, to make it easier
to convert the code to OBJECT_DEFINE_TYPE() in the future.
Reviewed-by: Philippe Mathieu-Daudé
Reviewed-by: Daniel P. Berrangé
Signed-off-by: Eduardo Habkost
---
Changes v2 -> v3: none
Changes v1 -> v2: none
---
Cc: Alberto
The TYPE_* constants and the typedefs are defined in ahci.h, so
we can move the type checking macros there too.
This will make future conversion to OBJECT_DECLARE* easier.
Reviewed-by: Daniel P. Berrangé
Signed-off-by: Eduardo Habkost
---
Changes v2 -> v3: none
Changes series v1 -> v2: new
Separate run of the TypeCheckMacro converter using the --force
flag, for the cases where typedefs weren't found in the same
header nor in typedefs.h.
Generated initially using:
$ ./scripts/codeconverter/converter.py --force -i \
--pattern=TypeCheckMacro $(git grep -l '' -- '*.[ch]')
Then
Replace DECLARE_OBJ_CHECKERS with OBJECT_DECLARE_TYPE where the
typedefs can be safely removed.
Generated running:
$ ./scripts/codeconverter/converter.py -i \
--pattern=DeclareObjCheckers $(git grep -l '' -- '*.[ch]')
Reviewed-by: Daniel P. Berrangé
Signed-off-by: Eduardo Habkost
---
Currently we have a SWIM typedef and a SWIM type checking macro,
but OBJECT_DECLARE* would transform the SWIM macro into a
function, and the function name would conflict with the SWIM
typedef name.
Rename the struct and typedef to "Swim". This will make future
conversion to OBJECT_DECLARE*
On Tue, Aug 25, 2020 at 07:18:19PM +0200, Alberto Garcia wrote:
> On Tue 25 Aug 2020 06:54:15 PM CEST, Brian Foster wrote:
> > If I compare this 5m fio test between XFS and ext4 on a couple of my
> > systems (with either no prealloc or full file prealloc), I end up seeing
> > ext4 run slightly
I have been measuring the performance of qcow2 overlays, and I am hoping to
get some help in understanding the data I collected. In my experiments, I
created a VM and attached a 16G qcow2 disk to it using "qemu-img create"
and "virsh attach-disk". I use fio to fill it. I create some number of
Rename the MEGASAS_DEVICE_CLASS() and MEGASAS_DEVICE_GET_CLASS()
macros to be consistent with the MEGASAS() instance cast macro.
This will allow us to register the type cast macros using
OBJECT_DECLARE_TYPE later.
Reviewed-by: Daniel P. Berrangé
Signed-off-by: Eduardo Habkost
---
Changes v2 ->
Move the ALLWINNER_AHCI macro close to the TYPE_ALLWINNER_AHCI
define.
This will make future conversion to OBJECT_DECLARE* easier.
Reviewed-by: Daniel P. Berrangé
Signed-off-by: Eduardo Habkost
---
Changes v2 -> v3: none
Changes series v1 -> v2: new patch in series v2
Cc: John Snow
Cc:
84 matches
Mail list logo
