Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 4/9/21 5:57 AM, Max Reitz wrote: Just as a PS, in a reply to one of Vladimir’s mails (da048f58-43a6-6811-6ad2-0d7899737...@redhat.com) I was wondering whether it even makes sense for mirror to do all the stuff it does in mirror_complete() to do it there. Aren’t all of those things that should really be done in job-finalize (i.e. mirror_exit_common())? Max Yes, I think so -- admittedly, when I added that finalize logic, I was just very confused about what was safe to move where in the mirror code and never got my patches off the ground to do a more vigorous refactoring. We've got, I think, three different user-initiated "This job should finish now" mechanisms: - Cancelling the mirror job after it reaches READY - Issuing "complete" to the mirror job after it reaches READY - Issuing "finalize" to a job Maybe these could all be integrated into a single mechanism somehow. I think I just lack the knowledge of the draining/threading/aio models to do it safely myself, and we'd need some compatibility shims for a while, etc. Would have to look at this stuff again to know for certain what we'd be able to change compatibly. --js
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 09.04.21 12:07, Vladimir Sementsov-Ogievskiy wrote:
09.04.2021 12:51, Max Reitz wrote:
On 08.04.21 19:26, Vladimir Sementsov-Ogievskiy wrote:
08.04.2021 20:04, John Snow wrote:
On 4/8/21 12:58 PM, Vladimir Sementsov-Ogievskiy wrote:
job-complete command is async. Can we instead just add a boolean
like job->completion_requested, and set it if job-complete called
in STANDBY state, and on job_resume job_complete will be called
automatically if this boolean is true?
job_complete has a synchronous setup, though -- we lose out on a lot
of synchronous error checking in that circumstance.
yes, that's a problem..
I was not able to audit it to determine that it'd be safe to attempt
that setup during a drained section -- I imagine it won't work and
will fail, though.
So I thought we'd have to signal completion and run the setup
*later*, but what do we do if we get an error then? Does the entire
job fail? Do we emit some new event? ("BLOCK_JOB_COMPLETION_FAILED"
?) Is it recoverable?
Isn't it possible even now, that after successful job-complete job
still fails and we report BLOCK_JOB_COMPLETED with error?
And actually, how much benefit user get from the fact that
job-complete may fail?
We can make job-complete a simple always-success boolean flag setter
like job-pause.
I wanted to say the following:
But job-pause does always succeed, in contrast to block-job-complete.
block-job-complete is more akin to job-finalize, which too is a
synchronous operation.
But when I wrote that last sentence, I asked myself whether what
mirror_complete() does isn’t actually a remnant of what we had to do
when we didn’t have job-finalize yet. Shouldn’t that all be in
mirror_exit_common()? What’s the advantage of opening the backing
chain or putting blockers on the to-replace node in
block-job-complete? Aren’t that all graph-changing operation,
basically, i.e. stuff that should be done in job-finalize?
If we move everything to mirror_exit_common(), all that remains to do
is basically set some should_complete flag (could even be part of the
Job struct), and then the whole problem disappears.
Thoughts?
Sounds good.. ButI want to understand first one simple thing: can job
fail even after block-job-complete succeeded?
Sure, if you get an I/O error afterwards.
As I understand current users think that it can't. And
block-job-complete is documented as "This command completes an active
background block operation synchronously". So it's assumed that if
block-job-complete succeeded we are totally done.
I think the only thing that block-job-complete does is signal to the job
it should exit once source and target have converged again. (The READY
event just says that source and target have converged once already.)
(Only in write-blocking copy mode is there a guarantee of source and
target remaining converged after READY.)
Well, and of course mirror_complete() also does a couple of stuff that
prepares replacing the source by the target.
But maybe, it's wrong? Can mirror_prepare fail after mirror_complete
success?
Oh definitely. For example, mirror_prepare replaces the source by the
target, which can definitely fail. (See mirror_exit_common().)
And user must check job status after job is finalized? Or check
error in BLOCK_JOB_COMPLETED event?
If the BLOCK_JOB_COMPLETED event shows an error, then the job doesn’t
even try to complete. If there is an error on job-finalize, source and
target have converged (so the target is consistent), but the source most
likely couldn’t be replaced by the target.
I suppose in practice if anything goes wrong libvirt just shows an error
and that’s it. No matter where the error occurs exactly.
Max
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
Am 09.04.2021 um 11:31 hat Max Reitz geschrieben:
> On 08.04.21 18:55, John Snow wrote:
> > On 4/8/21 12:20 PM, Max Reitz wrote:
> > > + /* Similarly, if the job is still drained, waiting will not
> > > help either */
> > > + if (job->pause_count > 0) {
> > > + error_setg(errp, "Job '%s' is blocked and cannot be
> > > unpaused", job->id);
> > > + return -EBUSY;
> > > + }
> > > +
> >
> > This leaks an internal state detail out to the caller. In which
> > circumstances does this happen?
>
> Hm. Now that you ask it.
>
> The circumstance would be a concurrent drain in some other IO thread.
> Probably the IO thread the job runs in? I don’t know any other thread that
> could concurrently drain, because this function runs in the main thread, and
> there shouldn’t be any drain in the background.
>
> If it is another IO thread, waiting would indeed help, so there would not be
> a need to error out.
>
> Perhaps it’s possible to have a background drain in the main thread? I
> don’t think so, though...
Hm... Maybe like this:
1. bdrv_do_drained_begin() in the main thread
2. BDRV_POLL_WHILE() reenters the QMP dispatcher coroutine
3. qmp_job_complete()
4. Deadlock because we poll here until the job is undrained
This is why nested event loops are evil...
I guess a way to fix this would be making qmp_job_complete() a coroutine
handler and yielding instead of actively polling. Then job_pause_point()
would have to wake the waiting coroutine rather than calling
aio_wait_kick().
On the other hand, this fix would be a lot more complex for -rc3 than
what you posted here.
So maybe take this one for -rc3 and do the coroutine thing for 6.1?
> > Looks about right to me, but you'll want Kevin's look-see for the finer
> > details, of course.
> >
> > My concern is that this adds a wait of an indefinite period to the
> > job_complete command. We mitigate this by checking for some other
> > internal state criteria first, and then by process of elimination deduce
> > that it's safe to wait, as it will (likely) be very quick.
> >
> > Do we open the door for ourselves to get into trouble here, either by a
> > state we are forgetting to rule out (You'd have added it if you know the
> > answer to this) or a hypothetical future change where we forget to
> > update this function?
>
> Well. Who knows.
>
> The alternatives I see are:
>
> (A) Let drained_end wait for the block jobs to be resumed. There are some
> details to consider there, I had some discussion around this with Kevin on
> Tuesday. For example, should every drained_end wait for all jobs involved
> to be resumed? (That would mean waiting for concurrent drained_ends, too.)
> Would the drained_end_counter be the right tool for the job? (None of this
> is unsolvable, I guess, but it would mean having another discussion.)
The advantage there would be that you don't have to deal with new nested
event loops.
I think the problem you brought up was that we can't wait for the job
resuming if it is at the same time paused externally. But we could just
check that and not increase the drained_end_counter in this case.
This would probably result in a fix that is simple enough for -rc3.
> It would also mean that you basically just move the wait to wherever the
> drained_end occurs, for example to qmp_transaction(). Now, every
> drained_end is preceded by a drained_begin that always has to wait, so it
> probably isn’t bad. OTOH, if qmp_transaction() would be allowed to wait for
> a job to be resumed, I think we can allow the same for
> qmp_block_job_complete().
> (And there’s the fact that most of the time not having the block job running
> after drained_end poses no problem. This is the first time I’m aware of a
> problem, so I think it would be preferable to wait only on the rare occasion
> where we have to.)
Yeah, I can see your point, but on the other hand, waiting in
drained_end until we can be sure that the effect of drained_begin has
been undone is a single place that covers all occasions.
I'm not overly confident that we would catch all occasions where we have
to if we try attacking them one by one. As you noticed, this is not some
simple case where things just fail all the time when you get it wrong,
but results in race conditions that are hard to reproduce.
> (B) Have block-job-complete be kind of asynchronous. We talked about that
> on IRC yesterday, and the main problem seems to be that we don’t know what
> we’d do with errors. We could only emit them via an event, or let the whole
> job fail, both of which seem like bad solutions.
I guess my suggestion to make it a coroutine QMP handlers is similar,
except that it blocks QMP, so you could still deliver an error.
On the other hand, this wouldn't be able to address the concern that we
might be waiting for too long (but I'm not convinced that this is a
problem anyway).
> (C) Perhaps mirror’s completion function works just fine when the job is
> paused (we just would have to
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
09.04.2021 12:51, Max Reitz wrote:
On 08.04.21 19:26, Vladimir Sementsov-Ogievskiy wrote:
08.04.2021 20:04, John Snow wrote:
On 4/8/21 12:58 PM, Vladimir Sementsov-Ogievskiy wrote:
job-complete command is async. Can we instead just add a boolean like
job->completion_requested, and set it if job-complete called in STANDBY state,
and on job_resume job_complete will be called automatically if this boolean is
true?
job_complete has a synchronous setup, though -- we lose out on a lot of
synchronous error checking in that circumstance.
yes, that's a problem..
I was not able to audit it to determine that it'd be safe to attempt that setup
during a drained section -- I imagine it won't work and will fail, though.
So I thought we'd have to signal completion and run the setup *later*, but what do we do
if we get an error then? Does the entire job fail? Do we emit some new event?
("BLOCK_JOB_COMPLETION_FAILED" ?) Is it recoverable?
Isn't it possible even now, that after successful job-complete job still fails
and we report BLOCK_JOB_COMPLETED with error?
And actually, how much benefit user get from the fact that job-complete may
fail?
We can make job-complete a simple always-success boolean flag setter like
job-pause.
I wanted to say the following:
But job-pause does always succeed, in contrast to block-job-complete.
block-job-complete is more akin to job-finalize, which too is a
synchronous operation.
But when I wrote that last sentence, I asked myself whether what
mirror_complete() does isn’t actually a remnant of what we had to do when we
didn’t have job-finalize yet. Shouldn’t that all be in mirror_exit_common()?
What’s the advantage of opening the backing chain or putting blockers on the
to-replace node in block-job-complete? Aren’t that all graph-changing
operation, basically, i.e. stuff that should be done in job-finalize?
If we move everything to mirror_exit_common(), all that remains to do is
basically set some should_complete flag (could even be part of the Job struct),
and then the whole problem disappears.
Thoughts?
Sounds good.. ButI want to understand first one simple thing: can job fail even
after block-job-complete succeeded?
As I understand current users think that it can't. And block-job-complete is documented
as "This command completes an active background block operation synchronously".
So it's assumed that if block-job-complete succeeded we are totally done.
But maybe, it's wrong? Can mirror_prepare fail after mirror_complete success?
And user must check job status after job is finalized? Or check error in
BLOCK_JOB_COMPLETED event?
--
Best regards,
Vladimir
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 09.04.21 11:44, Kevin Wolf wrote:
Am 08.04.2021 um 18:55 hat John Snow geschrieben:
On 4/8/21 12:20 PM, Max Reitz wrote:
block-job-complete can only be applied when the job is READY, not when
it is on STANDBY (ready, but paused). Draining a job technically pauses
it (which makes a READY job enter STANDBY), and ending the drained
section does not synchronously resume it, but only schedules the job,
which will then be resumed. So attempting to complete a job immediately
after a drained section may sometimes fail.
That is bad at least because users cannot really work nicely around
this: A job may be paused and resumed at any time, so waiting for the
job to be in the READY state and then issuing a block-job-complete poses
a TOCTTOU problem. The only way around it would be to issue
block-job-complete until it no longer fails due to the job being in the
STANDBY state, but that would not be nice.
We can solve the problem by allowing block-job-complete to be invoked on
jobs that are on STANDBY, if that status is the result of a drained
section (not because the user has paused the job), and that section has
ended. That is, if the job is on STANDBY, but scheduled to be resumed.
Perhaps we could actually just directly allow this, seeing that mirror
is the only user of ready/complete, and that mirror_complete() could
probably work under the given circumstances, but there may be many side
effects to consider.
It is simpler to add a function job_wait_unpaused() that waits for the
job to be resumed (under said circumstances), and to make
qmp_block_job_complete() use it to delay job_complete() until then.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1945635
Signed-off-by: Max Reitz
---
include/qemu/job.h | 15 +++
blockdev.c | 3 +++
job.c | 42 ++
3 files changed, 60 insertions(+)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index efc6fa7544..cf3082b6d7 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -563,4 +563,19 @@ void job_dismiss(Job **job, Error **errp);
*/
int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error
**errp);
+/**
+ * If the job has been paused because of a drained section, and that
+ * section has ended, wait until the job is resumed.
+ *
+ * Return 0 if the job is not paused, or if it has been successfully
+ * resumed.
+ * Return an error if the job has been paused in such a way that
+ * waiting will not resume it, i.e. if it has been paused by the user,
+ * or if it is still drained.
+ *
+ * Callers must be in the home AioContext and hold the AioContext lock
+ * of job->aio_context.
+ */
+int job_wait_unpaused(Job *job, Error **errp);
+
#endif
diff --git a/blockdev.c b/blockdev.c
index a57590aae4..c0cc2fa364 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3414,6 +3414,9 @@ void qmp_block_job_complete(const char *device, Error
**errp)
return;
}
+if (job_wait_unpaused(>job, errp) < 0) {
+return;
+}
After which point, we assume we've transitioned back to either RUNNING or
READY, and
trace_qmp_block_job_complete(job);
job_complete(>job, errp);
This function checks the usual state table for permission to deliver/perform
the verb.
aio_context_release(aio_context);
diff --git a/job.c b/job.c
index 289edee143..1ea30fd294 100644
--- a/job.c
+++ b/job.c
@@ -1023,3 +1023,45 @@ int job_finish_sync(Job *job, void (*finish)(Job *,
Error **errp), Error **errp)
job_unref(job);
return ret;
}
+
+int job_wait_unpaused(Job *job, Error **errp)
+{
+/*
+ * Only run this function from the main context, because this is
+ * what we need, and this way we do not have to think about what
+ * happens if the user concurrently pauses the job from the main
+ * monitor.
+ */
+assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+
+/*
+ * Quick path (e.g. so we do not get an error if pause_count > 0
+ * but the job is not even paused)
+ */
+if (!job->paused) {
+return 0;
+}
+
+/* If the user has paused the job, waiting will not help */
+if (job->user_paused) {
+error_setg(errp, "Job '%s' has been paused by the user", job->id);
+return -EBUSY;
+}
+
Or the job has encountered an error if that error policy is set. It is maybe
more accurate to say that the job is currently paused/halted (for some
reason) and is awaiting the explicit unpause instruction.
"Job '%s' has been paused and needs to be explicitly resumed with
job-resume", maybe?
Sounds good to me.
Job '%s' has been paused and needs to be [explicitly] resumed
[by the user] [with job-resume]
Some combo of those runes.
+/* Similarly, if the job is still drained, waiting will not help either */
+if (job->pause_count > 0) {
+error_setg(errp, "Job '%s' is blocked and cannot be unpaused",
job->id);
+return -EBUSY;
+}
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 08.04.21 19:26, Vladimir Sementsov-Ogievskiy wrote:
08.04.2021 20:04, John Snow wrote:
On 4/8/21 12:58 PM, Vladimir Sementsov-Ogievskiy wrote:
job-complete command is async. Can we instead just add a boolean like
job->completion_requested, and set it if job-complete called in
STANDBY state, and on job_resume job_complete will be called
automatically if this boolean is true?
job_complete has a synchronous setup, though -- we lose out on a lot
of synchronous error checking in that circumstance.
yes, that's a problem..
I was not able to audit it to determine that it'd be safe to attempt
that setup during a drained section -- I imagine it won't work and
will fail, though.
So I thought we'd have to signal completion and run the setup *later*,
but what do we do if we get an error then? Does the entire job fail?
Do we emit some new event? ("BLOCK_JOB_COMPLETION_FAILED" ?) Is it
recoverable?
Isn't it possible even now, that after successful job-complete job still
fails and we report BLOCK_JOB_COMPLETED with error?
And actually, how much benefit user get from the fact that job-complete
may fail?
We can make job-complete a simple always-success boolean flag setter
like job-pause.
I wanted to say the following:
But job-pause does always succeed, in contrast to block-job-complete.
block-job-complete is more akin to job-finalize, which too is a
synchronous operation.
But when I wrote that last sentence, I asked myself whether what
mirror_complete() does isn’t actually a remnant of what we had to do
when we didn’t have job-finalize yet. Shouldn’t that all be in
mirror_exit_common()? What’s the advantage of opening the backing chain
or putting blockers on the to-replace node in block-job-complete?
Aren’t that all graph-changing operation, basically, i.e. stuff that
should be done in job-finalize?
If we move everything to mirror_exit_common(), all that remains to do is
basically set some should_complete flag (could even be part of the Job
struct), and then the whole problem disappears.
Thoughts?
Max
And actual completion will be done in background, when possible. And if
it fail, job just fails, like it does for any background io error. And
user have to check error/success status of final BLOCK_JOB_COMPLETED
anyway.
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
Am 08.04.2021 um 18:55 hat John Snow geschrieben:
> On 4/8/21 12:20 PM, Max Reitz wrote:
> > block-job-complete can only be applied when the job is READY, not when
> > it is on STANDBY (ready, but paused). Draining a job technically pauses
> > it (which makes a READY job enter STANDBY), and ending the drained
> > section does not synchronously resume it, but only schedules the job,
> > which will then be resumed. So attempting to complete a job immediately
> > after a drained section may sometimes fail.
> >
> > That is bad at least because users cannot really work nicely around
> > this: A job may be paused and resumed at any time, so waiting for the
> > job to be in the READY state and then issuing a block-job-complete poses
> > a TOCTTOU problem. The only way around it would be to issue
> > block-job-complete until it no longer fails due to the job being in the
> > STANDBY state, but that would not be nice.
> >
> > We can solve the problem by allowing block-job-complete to be invoked on
> > jobs that are on STANDBY, if that status is the result of a drained
> > section (not because the user has paused the job), and that section has
> > ended. That is, if the job is on STANDBY, but scheduled to be resumed.
> >
> > Perhaps we could actually just directly allow this, seeing that mirror
> > is the only user of ready/complete, and that mirror_complete() could
> > probably work under the given circumstances, but there may be many side
> > effects to consider.
> >
> > It is simpler to add a function job_wait_unpaused() that waits for the
> > job to be resumed (under said circumstances), and to make
> > qmp_block_job_complete() use it to delay job_complete() until then.
> >
> > Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1945635
> > Signed-off-by: Max Reitz
> > ---
> > include/qemu/job.h | 15 +++
> > blockdev.c | 3 +++
> > job.c | 42 ++
> > 3 files changed, 60 insertions(+)
> >
> > diff --git a/include/qemu/job.h b/include/qemu/job.h
> > index efc6fa7544..cf3082b6d7 100644
> > --- a/include/qemu/job.h
> > +++ b/include/qemu/job.h
> > @@ -563,4 +563,19 @@ void job_dismiss(Job **job, Error **errp);
> >*/
> > int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error
> > **errp);
> > +/**
> > + * If the job has been paused because of a drained section, and that
> > + * section has ended, wait until the job is resumed.
> > + *
> > + * Return 0 if the job is not paused, or if it has been successfully
> > + * resumed.
> > + * Return an error if the job has been paused in such a way that
> > + * waiting will not resume it, i.e. if it has been paused by the user,
> > + * or if it is still drained.
> > + *
> > + * Callers must be in the home AioContext and hold the AioContext lock
> > + * of job->aio_context.
> > + */
> > +int job_wait_unpaused(Job *job, Error **errp);
> > +
> > #endif
> > diff --git a/blockdev.c b/blockdev.c
> > index a57590aae4..c0cc2fa364 100644
> > --- a/blockdev.c
> > +++ b/blockdev.c
> > @@ -3414,6 +3414,9 @@ void qmp_block_job_complete(const char *device, Error
> > **errp)
> > return;
> > }
> > +if (job_wait_unpaused(>job, errp) < 0) {
> > +return;
> > +}
>
> After which point, we assume we've transitioned back to either RUNNING or
> READY, and
>
> > trace_qmp_block_job_complete(job);
> > job_complete(>job, errp);
>
> This function checks the usual state table for permission to deliver/perform
> the verb.
>
> > aio_context_release(aio_context);
> > diff --git a/job.c b/job.c
> > index 289edee143..1ea30fd294 100644
> > --- a/job.c
> > +++ b/job.c
> > @@ -1023,3 +1023,45 @@ int job_finish_sync(Job *job, void (*finish)(Job *,
> > Error **errp), Error **errp)
> > job_unref(job);
> > return ret;
> > }
> > +
> > +int job_wait_unpaused(Job *job, Error **errp)
> > +{
> > +/*
> > + * Only run this function from the main context, because this is
> > + * what we need, and this way we do not have to think about what
> > + * happens if the user concurrently pauses the job from the main
> > + * monitor.
> > + */
> > +assert(qemu_get_current_aio_context() == qemu_get_aio_context());
> > +
> > +/*
> > + * Quick path (e.g. so we do not get an error if pause_count > 0
> > + * but the job is not even paused)
> > + */
> > +if (!job->paused) {
> > +return 0;
> > +}
> > +
> > +/* If the user has paused the job, waiting will not help */
> > +if (job->user_paused) {
> > +error_setg(errp, "Job '%s' has been paused by the user", job->id);
> > +return -EBUSY;
> > +}
> > +
>
> Or the job has encountered an error if that error policy is set. It is maybe
> more accurate to say that the job is currently paused/halted (for some
> reason) and is awaiting the explicit unpause instruction.
>
> "Job '%s' has been paused and needs to be explicitly
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 08.04.21 18:58, Vladimir Sementsov-Ogievskiy wrote:
08.04.2021 19:20, Max Reitz wrote:
block-job-complete can only be applied when the job is READY, not when
it is on STANDBY (ready, but paused). Draining a job technically pauses
it (which makes a READY job enter STANDBY), and ending the drained
section does not synchronously resume it, but only schedules the job,
which will then be resumed. So attempting to complete a job immediately
after a drained section may sometimes fail.
That is bad at least because users cannot really work nicely around
this: A job may be paused and resumed at any time, so waiting for the
job to be in the READY state and then issuing a block-job-complete poses
a TOCTTOU problem. The only way around it would be to issue
block-job-complete until it no longer fails due to the job being in the
STANDBY state, but that would not be nice.
We can solve the problem by allowing block-job-complete to be invoked on
jobs that are on STANDBY, if that status is the result of a drained
section (not because the user has paused the job), and that section has
ended. That is, if the job is on STANDBY, but scheduled to be resumed.
Perhaps we could actually just directly allow this, seeing that mirror
is the only user of ready/complete, and that mirror_complete() could
probably work under the given circumstances, but there may be many side
effects to consider.
It is simpler to add a function job_wait_unpaused() that waits for the
job to be resumed (under said circumstances), and to make
qmp_block_job_complete() use it to delay job_complete() until then.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1945635
Signed-off-by: Max Reitz
---
include/qemu/job.h | 15 +++
blockdev.c | 3 +++
job.c | 42 ++
3 files changed, 60 insertions(+)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index efc6fa7544..cf3082b6d7 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -563,4 +563,19 @@ void job_dismiss(Job **job, Error **errp);
*/
int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
Error **errp);
+/**
+ * If the job has been paused because of a drained section, and that
+ * section has ended, wait until the job is resumed.
+ *
+ * Return 0 if the job is not paused, or if it has been successfully
+ * resumed.
+ * Return an error if the job has been paused in such a way that
+ * waiting will not resume it, i.e. if it has been paused by the user,
+ * or if it is still drained.
+ *
+ * Callers must be in the home AioContext and hold the AioContext lock
+ * of job->aio_context.
+ */
+int job_wait_unpaused(Job *job, Error **errp);
+
#endif
diff --git a/blockdev.c b/blockdev.c
index a57590aae4..c0cc2fa364 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3414,6 +3414,9 @@ void qmp_block_job_complete(const char *device,
Error **errp)
return;
}
+ if (job_wait_unpaused(>job, errp) < 0) {
+ return;
+ }
trace_qmp_block_job_complete(job);
job_complete(>job, errp);
aio_context_release(aio_context);
diff --git a/job.c b/job.c
index 289edee143..1ea30fd294 100644
--- a/job.c
+++ b/job.c
@@ -1023,3 +1023,45 @@ int job_finish_sync(Job *job, void
(*finish)(Job *, Error **errp), Error **errp)
job_unref(job);
return ret;
}
+
+int job_wait_unpaused(Job *job, Error **errp)
+{
+ /*
+ * Only run this function from the main context, because this is
+ * what we need, and this way we do not have to think about what
+ * happens if the user concurrently pauses the job from the main
+ * monitor.
+ */
+ assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+
+ /*
+ * Quick path (e.g. so we do not get an error if pause_count > 0
+ * but the job is not even paused)
+ */
+ if (!job->paused) {
+ return 0;
+ }
+
+ /* If the user has paused the job, waiting will not help */
+ if (job->user_paused) {
+ error_setg(errp, "Job '%s' has been paused by the user",
job->id);
+ return -EBUSY;
+ }
+
+ /* Similarly, if the job is still drained, waiting will not help
either */
+ if (job->pause_count > 0) {
+ error_setg(errp, "Job '%s' is blocked and cannot be
unpaused", job->id);
+ return -EBUSY;
+ }
+
+ /*
+ * This function is specifically for waiting for a job to be
+ * resumed after a drained section. Ending the drained section
+ * includes a job_enter(), which schedules the job loop to be run,
+ * and once it does, job->paused will be cleared. Therefore, we
+ * do not need to invoke job_enter() here.
+ */
+ AIO_WAIT_WHILE(job->aio_context, job->paused);
+
+ return 0;
+}
Hmm.. It seems that when job->pause_count becomes 0, job_enter is
called, and the period when pause_count is 0 but paused is still true
should be relatively shot.
Yes.
And patch doesn't help if user call
job-complete during drained
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 08.04.21 18:55, John Snow wrote:
On 4/8/21 12:20 PM, Max Reitz wrote:
block-job-complete can only be applied when the job is READY, not when
it is on STANDBY (ready, but paused). Draining a job technically pauses
it (which makes a READY job enter STANDBY), and ending the drained
section does not synchronously resume it, but only schedules the job,
which will then be resumed. So attempting to complete a job immediately
after a drained section may sometimes fail.
That is bad at least because users cannot really work nicely around
this: A job may be paused and resumed at any time, so waiting for the
job to be in the READY state and then issuing a block-job-complete poses
a TOCTTOU problem. The only way around it would be to issue
block-job-complete until it no longer fails due to the job being in the
STANDBY state, but that would not be nice.
We can solve the problem by allowing block-job-complete to be invoked on
jobs that are on STANDBY, if that status is the result of a drained
section (not because the user has paused the job), and that section has
ended. That is, if the job is on STANDBY, but scheduled to be resumed.
Perhaps we could actually just directly allow this, seeing that mirror
is the only user of ready/complete, and that mirror_complete() could
probably work under the given circumstances, but there may be many side
effects to consider.
It is simpler to add a function job_wait_unpaused() that waits for the
job to be resumed (under said circumstances), and to make
qmp_block_job_complete() use it to delay job_complete() until then.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1945635
Signed-off-by: Max Reitz
---
include/qemu/job.h | 15 +++
blockdev.c | 3 +++
job.c | 42 ++
3 files changed, 60 insertions(+)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index efc6fa7544..cf3082b6d7 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -563,4 +563,19 @@ void job_dismiss(Job **job, Error **errp);
*/
int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
Error **errp);
+/**
+ * If the job has been paused because of a drained section, and that
+ * section has ended, wait until the job is resumed.
+ *
+ * Return 0 if the job is not paused, or if it has been successfully
+ * resumed.
+ * Return an error if the job has been paused in such a way that
+ * waiting will not resume it, i.e. if it has been paused by the user,
+ * or if it is still drained.
+ *
+ * Callers must be in the home AioContext and hold the AioContext lock
+ * of job->aio_context.
+ */
+int job_wait_unpaused(Job *job, Error **errp);
+
#endif
diff --git a/blockdev.c b/blockdev.c
index a57590aae4..c0cc2fa364 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3414,6 +3414,9 @@ void qmp_block_job_complete(const char *device,
Error **errp)
return;
}
+ if (job_wait_unpaused(>job, errp) < 0) {
+ return;
+ }
After which point, we assume we've transitioned back to either RUNNING
or READY, and
trace_qmp_block_job_complete(job);
job_complete(>job, errp);
This function checks the usual state table for permission to
deliver/perform the verb.
aio_context_release(aio_context);
diff --git a/job.c b/job.c
index 289edee143..1ea30fd294 100644
--- a/job.c
+++ b/job.c
@@ -1023,3 +1023,45 @@ int job_finish_sync(Job *job, void
(*finish)(Job *, Error **errp), Error **errp)
job_unref(job);
return ret;
}
+
+int job_wait_unpaused(Job *job, Error **errp)
+{
+ /*
+ * Only run this function from the main context, because this is
+ * what we need, and this way we do not have to think about what
+ * happens if the user concurrently pauses the job from the main
+ * monitor.
+ */
+ assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+
+ /*
+ * Quick path (e.g. so we do not get an error if pause_count > 0
+ * but the job is not even paused)
+ */
+ if (!job->paused) {
+ return 0;
+ }
+
+ /* If the user has paused the job, waiting will not help */
+ if (job->user_paused) {
+ error_setg(errp, "Job '%s' has been paused by the user",
job->id);
+ return -EBUSY;
+ }
+
Or the job has encountered an error if that error policy is set. It is
maybe more accurate to say that the job is currently paused/halted (for
some reason) and is awaiting the explicit unpause instruction.
"Job '%s' has been paused and needs to be explicitly resumed with
job-resume", maybe?
Job '%s' has been paused and needs to be [explicitly] resumed
[by the user] [with job-resume]
Some combo of those runes.
Sounds good. I think I’ll go for “Job '%s' has been paused and needs to
be explicitly resumed”.
+ /* Similarly, if the job is still drained, waiting will not help
either */
+ if (job->pause_count > 0) {
+ error_setg(errp, "Job '%s' is blocked and cannot be
unpaused", job->id);
+
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
08.04.2021 20:04, John Snow wrote:
On 4/8/21 12:58 PM, Vladimir Sementsov-Ogievskiy wrote:
job-complete command is async. Can we instead just add a boolean like
job->completion_requested, and set it if job-complete called in STANDBY state,
and on job_resume job_complete will be called automatically if this boolean is
true?
job_complete has a synchronous setup, though -- we lose out on a lot of
synchronous error checking in that circumstance.
yes, that's a problem..
I was not able to audit it to determine that it'd be safe to attempt that setup
during a drained section -- I imagine it won't work and will fail, though.
So I thought we'd have to signal completion and run the setup *later*, but what do we do
if we get an error then? Does the entire job fail? Do we emit some new event?
("BLOCK_JOB_COMPLETION_FAILED" ?) Is it recoverable?
Isn't it possible even now, that after successful job-complete job still fails
and we report BLOCK_JOB_COMPLETED with error?
And actually, how much benefit user get from the fact that job-complete may
fail?
We can make job-complete a simple always-success boolean flag setter like
job-pause.
And actual completion will be done in background, when possible. And if it
fail, job just fails, like it does for any background io error. And user have
to check error/success status of final BLOCK_JOB_COMPLETED anyway.
--
Best regards,
Vladimir
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 4/8/21 12:58 PM, Vladimir Sementsov-Ogievskiy wrote:
job-complete command is async. Can we instead just add a boolean like
job->completion_requested, and set it if job-complete called in STANDBY
state, and on job_resume job_complete will be called automatically if
this boolean is true?
job_complete has a synchronous setup, though -- we lose out on a lot of
synchronous error checking in that circumstance.
I was not able to audit it to determine that it'd be safe to attempt
that setup during a drained section -- I imagine it won't work and will
fail, though.
So I thought we'd have to signal completion and run the setup *later*,
but what do we do if we get an error then? Does the entire job fail? Do
we emit some new event? ("BLOCK_JOB_COMPLETION_FAILED" ?) Is it recoverable?
So on and so forth. Seems like a lot of things to consider, unless I am
making a giant fuss about nothing again, not like that's ever happened. O:-)
--js
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
08.04.2021 19:20, Max Reitz wrote:
block-job-complete can only be applied when the job is READY, not when
it is on STANDBY (ready, but paused). Draining a job technically pauses
it (which makes a READY job enter STANDBY), and ending the drained
section does not synchronously resume it, but only schedules the job,
which will then be resumed. So attempting to complete a job immediately
after a drained section may sometimes fail.
That is bad at least because users cannot really work nicely around
this: A job may be paused and resumed at any time, so waiting for the
job to be in the READY state and then issuing a block-job-complete poses
a TOCTTOU problem. The only way around it would be to issue
block-job-complete until it no longer fails due to the job being in the
STANDBY state, but that would not be nice.
We can solve the problem by allowing block-job-complete to be invoked on
jobs that are on STANDBY, if that status is the result of a drained
section (not because the user has paused the job), and that section has
ended. That is, if the job is on STANDBY, but scheduled to be resumed.
Perhaps we could actually just directly allow this, seeing that mirror
is the only user of ready/complete, and that mirror_complete() could
probably work under the given circumstances, but there may be many side
effects to consider.
It is simpler to add a function job_wait_unpaused() that waits for the
job to be resumed (under said circumstances), and to make
qmp_block_job_complete() use it to delay job_complete() until then.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1945635
Signed-off-by: Max Reitz
---
include/qemu/job.h | 15 +++
blockdev.c | 3 +++
job.c | 42 ++
3 files changed, 60 insertions(+)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index efc6fa7544..cf3082b6d7 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -563,4 +563,19 @@ void job_dismiss(Job **job, Error **errp);
*/
int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error
**errp);
+/**
+ * If the job has been paused because of a drained section, and that
+ * section has ended, wait until the job is resumed.
+ *
+ * Return 0 if the job is not paused, or if it has been successfully
+ * resumed.
+ * Return an error if the job has been paused in such a way that
+ * waiting will not resume it, i.e. if it has been paused by the user,
+ * or if it is still drained.
+ *
+ * Callers must be in the home AioContext and hold the AioContext lock
+ * of job->aio_context.
+ */
+int job_wait_unpaused(Job *job, Error **errp);
+
#endif
diff --git a/blockdev.c b/blockdev.c
index a57590aae4..c0cc2fa364 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3414,6 +3414,9 @@ void qmp_block_job_complete(const char *device, Error
**errp)
return;
}
+if (job_wait_unpaused(>job, errp) < 0) {
+return;
+}
trace_qmp_block_job_complete(job);
job_complete(>job, errp);
aio_context_release(aio_context);
diff --git a/job.c b/job.c
index 289edee143..1ea30fd294 100644
--- a/job.c
+++ b/job.c
@@ -1023,3 +1023,45 @@ int job_finish_sync(Job *job, void (*finish)(Job *,
Error **errp), Error **errp)
job_unref(job);
return ret;
}
+
+int job_wait_unpaused(Job *job, Error **errp)
+{
+/*
+ * Only run this function from the main context, because this is
+ * what we need, and this way we do not have to think about what
+ * happens if the user concurrently pauses the job from the main
+ * monitor.
+ */
+assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+
+/*
+ * Quick path (e.g. so we do not get an error if pause_count > 0
+ * but the job is not even paused)
+ */
+if (!job->paused) {
+return 0;
+}
+
+/* If the user has paused the job, waiting will not help */
+if (job->user_paused) {
+error_setg(errp, "Job '%s' has been paused by the user", job->id);
+return -EBUSY;
+}
+
+/* Similarly, if the job is still drained, waiting will not help either */
+if (job->pause_count > 0) {
+error_setg(errp, "Job '%s' is blocked and cannot be unpaused",
job->id);
+return -EBUSY;
+}
+
+/*
+ * This function is specifically for waiting for a job to be
+ * resumed after a drained section. Ending the drained section
+ * includes a job_enter(), which schedules the job loop to be run,
+ * and once it does, job->paused will be cleared. Therefore, we
+ * do not need to invoke job_enter() here.
+ */
+AIO_WAIT_WHILE(job->aio_context, job->paused);
+
+return 0;
+}
Hmm.. It seems that when job->pause_count becomes 0, job_enter is called, and
the period when pause_count is 0 but paused is still true should be relatively
shot. And patch doesn't help if user call job-complete during drained section. So
it looks like the patch will help relatively seldom.. Or
Re: [PATCH for-6.0? 1/3] job: Add job_wait_unpaused() for block-job-complete
On 4/8/21 12:20 PM, Max Reitz wrote:
block-job-complete can only be applied when the job is READY, not when
it is on STANDBY (ready, but paused). Draining a job technically pauses
it (which makes a READY job enter STANDBY), and ending the drained
section does not synchronously resume it, but only schedules the job,
which will then be resumed. So attempting to complete a job immediately
after a drained section may sometimes fail.
That is bad at least because users cannot really work nicely around
this: A job may be paused and resumed at any time, so waiting for the
job to be in the READY state and then issuing a block-job-complete poses
a TOCTTOU problem. The only way around it would be to issue
block-job-complete until it no longer fails due to the job being in the
STANDBY state, but that would not be nice.
We can solve the problem by allowing block-job-complete to be invoked on
jobs that are on STANDBY, if that status is the result of a drained
section (not because the user has paused the job), and that section has
ended. That is, if the job is on STANDBY, but scheduled to be resumed.
Perhaps we could actually just directly allow this, seeing that mirror
is the only user of ready/complete, and that mirror_complete() could
probably work under the given circumstances, but there may be many side
effects to consider.
It is simpler to add a function job_wait_unpaused() that waits for the
job to be resumed (under said circumstances), and to make
qmp_block_job_complete() use it to delay job_complete() until then.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1945635
Signed-off-by: Max Reitz
---
include/qemu/job.h | 15 +++
blockdev.c | 3 +++
job.c | 42 ++
3 files changed, 60 insertions(+)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index efc6fa7544..cf3082b6d7 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -563,4 +563,19 @@ void job_dismiss(Job **job, Error **errp);
*/
int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error
**errp);
+/**
+ * If the job has been paused because of a drained section, and that
+ * section has ended, wait until the job is resumed.
+ *
+ * Return 0 if the job is not paused, or if it has been successfully
+ * resumed.
+ * Return an error if the job has been paused in such a way that
+ * waiting will not resume it, i.e. if it has been paused by the user,
+ * or if it is still drained.
+ *
+ * Callers must be in the home AioContext and hold the AioContext lock
+ * of job->aio_context.
+ */
+int job_wait_unpaused(Job *job, Error **errp);
+
#endif
diff --git a/blockdev.c b/blockdev.c
index a57590aae4..c0cc2fa364 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3414,6 +3414,9 @@ void qmp_block_job_complete(const char *device, Error
**errp)
return;
}
+if (job_wait_unpaused(>job, errp) < 0) {
+return;
+}
After which point, we assume we've transitioned back to either RUNNING
or READY, and
trace_qmp_block_job_complete(job);
job_complete(>job, errp);
This function checks the usual state table for permission to
deliver/perform the verb.
aio_context_release(aio_context);
diff --git a/job.c b/job.c
index 289edee143..1ea30fd294 100644
--- a/job.c
+++ b/job.c
@@ -1023,3 +1023,45 @@ int job_finish_sync(Job *job, void (*finish)(Job *,
Error **errp), Error **errp)
job_unref(job);
return ret;
}
+
+int job_wait_unpaused(Job *job, Error **errp)
+{
+/*
+ * Only run this function from the main context, because this is
+ * what we need, and this way we do not have to think about what
+ * happens if the user concurrently pauses the job from the main
+ * monitor.
+ */
+assert(qemu_get_current_aio_context() == qemu_get_aio_context());
+
+/*
+ * Quick path (e.g. so we do not get an error if pause_count > 0
+ * but the job is not even paused)
+ */
+if (!job->paused) {
+return 0;
+}
+
+/* If the user has paused the job, waiting will not help */
+if (job->user_paused) {
+error_setg(errp, "Job '%s' has been paused by the user", job->id);
+return -EBUSY;
+}
+
Or the job has encountered an error if that error policy is set. It is
maybe more accurate to say that the job is currently paused/halted (for
some reason) and is awaiting the explicit unpause instruction.
"Job '%s' has been paused and needs to be explicitly resumed with
job-resume", maybe?
Job '%s' has been paused and needs to be [explicitly] resumed
[by the user] [with job-resume]
Some combo of those runes.
+/* Similarly, if the job is still drained, waiting will not help either */
+if (job->pause_count > 0) {
+error_setg(errp, "Job '%s' is blocked and cannot be unpaused",
job->id);
+return -EBUSY;
+}
+
This leaks an internal state detail out to the caller. In which
circumstances does this happen? Do we
