Re: [PATCH v8 6/7] block-stream: freeze link to base node during stream job

2020-09-24 Thread Max Reitz
On 07.09.20 14:17, Vladimir Sementsov-Ogievskiy wrote:
> 07.09.2020 14:44, Max Reitz wrote:
>> On 04.09.20 15:48, Vladimir Sementsov-Ogievskiy wrote:
>>> 04.09.2020 16:21, Max Reitz wrote:
 On 28.08.20 18:52, Andrey Shinkevich wrote:
> To keep the base node unchanged during the block-stream operation,
> freeze it as the other part of the backing chain with the intermediate
> nodes related to the job.
> This patch revers the change made with the commit c624b015bf as the
> correct base file name and its format have to be written down to the
> QCOW2 header on the disk when the backing file is being changed after
> the stream job completes.
> This reversion incurs changes in the tests 030, 245 and discards the
> test 258 where concurrent stream/commit jobs are tested. When the link
> to a base node is frozen, the concurrent job cannot change the common
> backing chain.
>
> Signed-off-by: Andrey Shinkevich 
> ---
>    block/stream.c |  29 ++--
>    tests/qemu-iotests/030 |  10 +--
>    tests/qemu-iotests/245 |   2 +-
>    tests/qemu-iotests/258 | 161
> -
>    tests/qemu-iotests/258.out |  33 --
>    5 files changed, 14 insertions(+), 221 deletions(-)
>    delete mode 100755 tests/qemu-iotests/258
>    delete mode 100644 tests/qemu-iotests/258.out

 Does this need to be in this series?  (I’m not entirely sure, based on
 what I can see in patch 7.)

 When doing this, should we introduce a @bottom-node option
 alongside, so
 that we can make all the tests that are deleted here pass still, just
 with changes?

>>>
>>> That's a question to discuss, and I asked Andrey to make this patch
>>> in this
>>> simple way to see, how much damage we have with this change.
>>>
>>> Honestly, I doubt that we need the new option. Previously, we decided
>>> that
>>> we can make this change without a deprecation. If we still going to
>>> do it,
>>> we shouldn't care about these use cases. So, if we freeze base again
>>> wituhout
>>> a depreaction and:
>>>
>>> 1. add bottom-node
>>>
>>>   - we keep the iotests
>>>   - If (unlikely) someone will came and say: hey, you've broken my
>>> working scenario, we will say "use bottom-node option, sorry"
>>>   - Most likely we'll have unused option and corresponding unused logic,
>>> making code more complex for nothing (and we'll have to say "sorry"
>>> anyway)
>>>
>>> 2. without option
>>>
>>>   - we loose the iotests. this looks scary, but it is honest: we drop
>>> use-cases and corresponding iotests
>>>   - If (unlikely) someone will came and say: hey, you've broken my
>>> working scenario, he will have to wait for next release / package
>>> version / or update the downstream himself
>>>   - Most likely all will be OK.
>>
>> Well, yes, we’ll disrupt either way, but it is a difference whether we
>> can tell people immediately that there’s an alternative now, or whether
>> we’ll have to make them wait for the next release.
>>
>> Basically, the whole argument hinges on the question of whether anyone
>> uses this right now or not, and we just don’t know.
>>
>> The question remains whether this patch is necessary for this series.
> 
> Otherwise iotests fail :)
> 
>> We also have the option of introducing @bottom-node, leaving @base’s
>> behavior as-is
> 
> You mean not make it freeze base again, but just don't care?

Yes.  I think the only problem with that would be that it’s unintuitive
in case the graph is modified while the job is running, but I can’t find
that worse than forbidding that case completely.

(And I think it would be easier to explain if we introduced @bottom-node.)

>> and explaining it as a legacy option from which
>> @bottom-node is inferred.  Then specifying @base just becomes weird and
>> problem-prone when the graph is reconfigured while the job is active,
>> but you can get around that by simply using the non-legacy option.
> 
> Hmm. Last time, I thought that bottom-node was a bad idea, as we have a
> lot of problems with it,

Hm, did we?  Off the top of my head, I can’t remember any.  Besides the
fact that it would require users to use a different parameter and us to
support two parameters unless we decide to deprecate @base.

> but you think it should be kept as preferred
> behavior? But this sounds as working idea.
> 
> Then, we'll probably want to set skip_filters(bottom->backing) as
> backing of top in qcow2 metadata, and direct bottom->backing as new
> backing of top in block node graph.

I’m not sure whether I agree with skipping filters for the qcow2
metadata, just because then it’s different from the runtime state.  But
OTOH I would expect that any application that seriously cares about
filters would override the qcow2 metadata anyway, so I think I do agree
after all.

Yeah, I think skipping filters for the backing file name in the qcow2
header is right.

Re: [PATCH v8 6/7] block-stream: freeze link to base node during stream job

2020-09-07 Thread Vladimir Sementsov-Ogievskiy

07.09.2020 14:44, Max Reitz wrote:

On 04.09.20 15:48, Vladimir Sementsov-Ogievskiy wrote:

04.09.2020 16:21, Max Reitz wrote:

On 28.08.20 18:52, Andrey Shinkevich wrote:

To keep the base node unchanged during the block-stream operation,
freeze it as the other part of the backing chain with the intermediate
nodes related to the job.
This patch revers the change made with the commit c624b015bf as the
correct base file name and its format have to be written down to the
QCOW2 header on the disk when the backing file is being changed after
the stream job completes.
This reversion incurs changes in the tests 030, 245 and discards the
test 258 where concurrent stream/commit jobs are tested. When the link
to a base node is frozen, the concurrent job cannot change the common
backing chain.

Signed-off-by: Andrey Shinkevich 
---
   block/stream.c |  29 ++--
   tests/qemu-iotests/030 |  10 +--
   tests/qemu-iotests/245 |   2 +-
   tests/qemu-iotests/258 | 161
-
   tests/qemu-iotests/258.out |  33 --
   5 files changed, 14 insertions(+), 221 deletions(-)
   delete mode 100755 tests/qemu-iotests/258
   delete mode 100644 tests/qemu-iotests/258.out


Does this need to be in this series?  (I’m not entirely sure, based on
what I can see in patch 7.)

When doing this, should we introduce a @bottom-node option alongside, so
that we can make all the tests that are deleted here pass still, just
with changes?



That's a question to discuss, and I asked Andrey to make this patch in this
simple way to see, how much damage we have with this change.

Honestly, I doubt that we need the new option. Previously, we decided that
we can make this change without a deprecation. If we still going to do it,
we shouldn't care about these use cases. So, if we freeze base again
wituhout
a depreaction and:

1. add bottom-node

  - we keep the iotests
  - If (unlikely) someone will came and say: hey, you've broken my
working scenario, we will say "use bottom-node option, sorry"
  - Most likely we'll have unused option and corresponding unused logic,
making code more complex for nothing (and we'll have to say "sorry" anyway)

2. without option

  - we loose the iotests. this looks scary, but it is honest: we drop
use-cases and corresponding iotests
  - If (unlikely) someone will came and say: hey, you've broken my
working scenario, he will have to wait for next release / package
version / or update the downstream himself
  - Most likely all will be OK.


Well, yes, we’ll disrupt either way, but it is a difference whether we
can tell people immediately that there’s an alternative now, or whether
we’ll have to make them wait for the next release.

Basically, the whole argument hinges on the question of whether anyone
uses this right now or not, and we just don’t know.

The question remains whether this patch is necessary for this series.


Otherwise iotests fail :)


We also have the option of introducing @bottom-node, leaving @base’s
behavior as-is


You mean not make it freeze base again, but just don't care?


and explaining it as a legacy option from which
@bottom-node is inferred.  Then specifying @base just becomes weird and
problem-prone when the graph is reconfigured while the job is active,
but you can get around that by simply using the non-legacy option.


Hmm. Last time, I thought that bottom-node was a bad idea, as we have a lot of 
problems with it, but you think it should be kept as preferred behavior? But 
this sounds as working idea.

Then, we'll probably want to set skip_filters(bottom->backing) as backing of top 
in qcow2 metadata, and direct bottom->backing as new backing of top in block node 
graph.

Anyway, I like the idea to deprecate filename-based interfaces wherever we can.

PS: Sorry for my decreased attention to the list for last weeks, I have to 
finish necessary work for Virtuozzo release.



Max


Hmm. OK, and the hard-way:

3. Enable all the new logic (filter insertion, freezing base, etc.) only
when filter-node-name option specified. And immediately deprecate
not-specifying the option.
  [Note, that in way [3] we don't need bottom-node option]









--
Best regards,
Vladimir



Re: [PATCH v8 6/7] block-stream: freeze link to base node during stream job

2020-09-07 Thread Max Reitz
On 04.09.20 15:48, Vladimir Sementsov-Ogievskiy wrote:
> 04.09.2020 16:21, Max Reitz wrote:
>> On 28.08.20 18:52, Andrey Shinkevich wrote:
>>> To keep the base node unchanged during the block-stream operation,
>>> freeze it as the other part of the backing chain with the intermediate
>>> nodes related to the job.
>>> This patch revers the change made with the commit c624b015bf as the
>>> correct base file name and its format have to be written down to the
>>> QCOW2 header on the disk when the backing file is being changed after
>>> the stream job completes.
>>> This reversion incurs changes in the tests 030, 245 and discards the
>>> test 258 where concurrent stream/commit jobs are tested. When the link
>>> to a base node is frozen, the concurrent job cannot change the common
>>> backing chain.
>>>
>>> Signed-off-by: Andrey Shinkevich 
>>> ---
>>>   block/stream.c |  29 ++--
>>>   tests/qemu-iotests/030 |  10 +--
>>>   tests/qemu-iotests/245 |   2 +-
>>>   tests/qemu-iotests/258 | 161
>>> -
>>>   tests/qemu-iotests/258.out |  33 --
>>>   5 files changed, 14 insertions(+), 221 deletions(-)
>>>   delete mode 100755 tests/qemu-iotests/258
>>>   delete mode 100644 tests/qemu-iotests/258.out
>>
>> Does this need to be in this series?  (I’m not entirely sure, based on
>> what I can see in patch 7.)
>>
>> When doing this, should we introduce a @bottom-node option alongside, so
>> that we can make all the tests that are deleted here pass still, just
>> with changes?
>>
> 
> That's a question to discuss, and I asked Andrey to make this patch in this
> simple way to see, how much damage we have with this change.
> 
> Honestly, I doubt that we need the new option. Previously, we decided that
> we can make this change without a deprecation. If we still going to do it,
> we shouldn't care about these use cases. So, if we freeze base again
> wituhout
> a depreaction and:
> 
> 1. add bottom-node
> 
>  - we keep the iotests
>  - If (unlikely) someone will came and say: hey, you've broken my
> working scenario, we will say "use bottom-node option, sorry"
>  - Most likely we'll have unused option and corresponding unused logic,
> making code more complex for nothing (and we'll have to say "sorry" anyway)
> 
> 2. without option
> 
>  - we loose the iotests. this looks scary, but it is honest: we drop
> use-cases and corresponding iotests
>  - If (unlikely) someone will came and say: hey, you've broken my
> working scenario, he will have to wait for next release / package
> version / or update the downstream himself
>  - Most likely all will be OK.

Well, yes, we’ll disrupt either way, but it is a difference whether we
can tell people immediately that there’s an alternative now, or whether
we’ll have to make them wait for the next release.

Basically, the whole argument hinges on the question of whether anyone
uses this right now or not, and we just don’t know.

The question remains whether this patch is necessary for this series.
We also have the option of introducing @bottom-node, leaving @base’s
behavior as-is and explaining it as a legacy option from which
@bottom-node is inferred.  Then specifying @base just becomes weird and
problem-prone when the graph is reconfigured while the job is active,
but you can get around that by simply using the non-legacy option.

Max

> Hmm. OK, and the hard-way:
> 
> 3. Enable all the new logic (filter insertion, freezing base, etc.) only
> when filter-node-name option specified. And immediately deprecate
> not-specifying the option.
>  [Note, that in way [3] we don't need bottom-node option]
> 
> 
> 




signature.asc
Description: OpenPGP digital signature


Re: [PATCH v8 6/7] block-stream: freeze link to base node during stream job

2020-09-04 Thread Vladimir Sementsov-Ogievskiy

04.09.2020 16:21, Max Reitz wrote:

On 28.08.20 18:52, Andrey Shinkevich wrote:

To keep the base node unchanged during the block-stream operation,
freeze it as the other part of the backing chain with the intermediate
nodes related to the job.
This patch revers the change made with the commit c624b015bf as the
correct base file name and its format have to be written down to the
QCOW2 header on the disk when the backing file is being changed after
the stream job completes.
This reversion incurs changes in the tests 030, 245 and discards the
test 258 where concurrent stream/commit jobs are tested. When the link
to a base node is frozen, the concurrent job cannot change the common
backing chain.

Signed-off-by: Andrey Shinkevich 
---
  block/stream.c |  29 ++--
  tests/qemu-iotests/030 |  10 +--
  tests/qemu-iotests/245 |   2 +-
  tests/qemu-iotests/258 | 161 -
  tests/qemu-iotests/258.out |  33 --
  5 files changed, 14 insertions(+), 221 deletions(-)
  delete mode 100755 tests/qemu-iotests/258
  delete mode 100644 tests/qemu-iotests/258.out


Does this need to be in this series?  (I’m not entirely sure, based on
what I can see in patch 7.)

When doing this, should we introduce a @bottom-node option alongside, so
that we can make all the tests that are deleted here pass still, just
with changes?



That's a question to discuss, and I asked Andrey to make this patch in this
simple way to see, how much damage we have with this change.

Honestly, I doubt that we need the new option. Previously, we decided that
we can make this change without a deprecation. If we still going to do it,
we shouldn't care about these use cases. So, if we freeze base again wituhout
a depreaction and:

1. add bottom-node

 - we keep the iotests
 - If (unlikely) someone will came and say: hey, you've broken my working scenario, we 
will say "use bottom-node option, sorry"
 - Most likely we'll have unused option and corresponding unused logic, making code more 
complex for nothing (and we'll have to say "sorry" anyway)

2. without option

 - we loose the iotests. this looks scary, but it is honest: we drop use-cases 
and corresponding iotests
 - If (unlikely) someone will came and say: hey, you've broken my working 
scenario, he will have to wait for next release / package version / or update 
the downstream himself
 - Most likely all will be OK.


Hmm. OK, and the hard-way:

3. Enable all the new logic (filter insertion, freezing base, etc.) only when 
filter-node-name option specified. And immediately deprecate not-specifying the 
option.
 [Note, that in way [3] we don't need bottom-node option]



--
Best regards,
Vladimir



[PATCH v8 6/7] block-stream: freeze link to base node during stream job

2020-08-28 Thread Andrey Shinkevich via
To keep the base node unchanged during the block-stream operation,
freeze it as the other part of the backing chain with the intermediate
nodes related to the job.
This patch revers the change made with the commit c624b015bf as the
correct base file name and its format have to be written down to the
QCOW2 header on the disk when the backing file is being changed after
the stream job completes.
This reversion incurs changes in the tests 030, 245 and discards the
test 258 where concurrent stream/commit jobs are tested. When the link
to a base node is frozen, the concurrent job cannot change the common
backing chain.

Signed-off-by: Andrey Shinkevich 
---
 block/stream.c |  29 ++--
 tests/qemu-iotests/030 |  10 +--
 tests/qemu-iotests/245 |   2 +-
 tests/qemu-iotests/258 | 161 -
 tests/qemu-iotests/258.out |  33 --
 5 files changed, 14 insertions(+), 221 deletions(-)
 delete mode 100755 tests/qemu-iotests/258
 delete mode 100644 tests/qemu-iotests/258.out

diff --git a/block/stream.c b/block/stream.c
index 8bf6b6d..fee4117 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -32,7 +32,7 @@ enum {
 typedef struct StreamBlockJob {
 BlockJob common;
 BlockDriverState *base_overlay; /* COW overlay (stream from this) */
-BlockDriverState *above_base;   /* Node directly above the base */
+BlockDriverState *base;   /* The base node */
 BlockdevOnError on_error;
 char *backing_file_str;
 bool bs_read_only;
@@ -54,7 +54,7 @@ static void stream_abort(Job *job)
 
 if (s->chain_frozen) {
 BlockJob *bjob = >common;
-bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
+bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->base);
 }
 }
 
@@ -64,11 +64,11 @@ static int stream_prepare(Job *job)
 BlockJob *bjob = >common;
 BlockDriverState *bs = blk_bs(bjob->blk);
 BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
-BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);
+BlockDriverState *base = s->base;
 Error *local_err = NULL;
 int ret = 0;
 
-bdrv_unfreeze_backing_chain(bs, s->above_base);
+bdrv_unfreeze_backing_chain(bs, s->base);
 s->chain_frozen = false;
 
 if (bdrv_cow_child(unfiltered_bs)) {
@@ -230,7 +230,6 @@ void stream_start(const char *job_id, BlockDriverState *bs,
 bool bs_read_only;
 int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
 BlockDriverState *base_overlay = bdrv_find_overlay(bs, base);
-BlockDriverState *above_base;
 
 if (!base_overlay) {
 error_setg(errp, "'%s' is not in the backing chain of '%s'",
@@ -238,20 +237,7 @@ void stream_start(const char *job_id, BlockDriverState *bs,
 return;
 }
 
-/*
- * Find the node directly above @base.  @base_overlay is a COW overlay, so
- * it must have a bdrv_cow_child(), but it is the immediate overlay of
- * @base, so between the two there can only be filters.
- */
-above_base = base_overlay;
-if (bdrv_cow_bs(above_base) != base) {
-above_base = bdrv_cow_bs(above_base);
-while (bdrv_filter_bs(above_base) != base) {
-above_base = bdrv_filter_bs(above_base);
-}
-}
-
-if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) {
+if (bdrv_freeze_backing_chain(bs, base, errp) < 0) {
 return;
 }
 
@@ -284,7 +270,6 @@ void stream_start(const char *job_id, BlockDriverState *bs,
  * above_base node might change after the call to
  * bdrv_reopen_set_read_only() due to parallel block jobs running.
  */
-base = bdrv_filter_or_cow_bs(above_base);
 for (iter = bdrv_filter_or_cow_bs(bs); iter != base;
  iter = bdrv_filter_or_cow_bs(iter))
 {
@@ -293,7 +278,7 @@ void stream_start(const char *job_id, BlockDriverState *bs,
 }
 
 s->base_overlay = base_overlay;
-s->above_base = above_base;
+s->base = base;
 s->backing_file_str = g_strdup(backing_file_str);
 s->bs_read_only = bs_read_only;
 s->chain_frozen = true;
@@ -307,5 +292,5 @@ fail:
 if (bs_read_only) {
 bdrv_reopen_set_read_only(bs, true, NULL);
 }
-bdrv_unfreeze_backing_chain(bs, above_base);
+bdrv_unfreeze_backing_chain(bs, base);
 }
diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 1cdd7e2..c565e76 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -382,7 +382,7 @@ class TestParallelOps(iotests.QMPTestCase):
 # Stream from node2 into node4
 result = self.vm.qmp('block-stream', device='node4', 
base_node='node2', job_id='node4')
 self.assert_qmp(result, 'error/desc',
-"Cannot freeze 'backing' link to 'commit-filter'")
+"Cannot change 'backing' link from 'commit-filter' to 'node2'")
 
 result = self.vm.qmp('block-job-set-speed', device='drive0', speed=0)
 self.assert_qmp(result, 'return', {})
@@