Tomas Jelinek has uploaded a new change for review.
Change subject: migrations: change convergence schedule from time to iterations
......................................................................
migrations: change convergence schedule from time to iterations
Currently the convergence schedule reacted to specific number of seconds of
stalling. It turned out to be incorrect, because the algorithm which detected
stalling was not detecting it properly and fundamentally can not.
The reason is that the alg remembers tha last progress time and than if the
data_remaining is bigger it considers the migration to be stalling.
This does not work properly, because the qemu works in iterations and each
iteration takes significant amount of time.
In the first iteration there can be no stalling, qemu only copies memory and
does not look at how much of it changed.
After this period of time it checks if it can move the rest of the VMs memory
when pauses it for the dowtime. If not, it starts a new iteration. This new
iteration starts copying all the dirtyed memory which can be a lot. And the
whole second iteration the lowmark of the memory will be lower than the current
remaining thus being threated as stalling. But this does not actually mean we
want to enlarge the downtime during the iteration since we don't know if qemu
can or can not migrate the last part. This information can be found only
between two iterations - e.g. if it is detected that the copying is in new
iteration it means the current downtime was not enough so the enlargement of
the downtime may make sense.
So, the current patch changes the meaning of the "limit" from num of seconds
the migration can be stalling to num of iterations the migration can be
stalling (it may make sense to wait for more than one iteration to give qemu
couple of tries before enlarging the downtime).
The detection of new iteration is done this way:
- in each monitoring cycle the current amount of remaining memory is remembered
- in the next cycle the remembered remaining memory is compared to the current
one
- the result can be:
- the current amount is smaller than the remembered => migration either
progressing inside one iteration or the next iteration but it was so fast
that it went under the remembered value. In this case we can fail to
recognize the next iteration but it does not really matter since the
migration progressed in between.
- the current amount is equal or higher than the remembered => it is a new
iteration because inside of one iteration the amount of memory has to
slowly go down and can not stay at one point even less grow
The detection is not perfect but is the best we can have until libvirt 1.3 when
we will get this info reported.
Change-Id: I6f87c954031842c35c99888c228a34ec7f19d800
Signed-off-by: Tomas Jelinek <[email protected]>
---
M lib/api/vdsmapi-schema.json
M vdsm/virt/migration.py
2 files changed, 18 insertions(+), 12 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/58/56558/1
diff --git a/lib/api/vdsmapi-schema.json b/lib/api/vdsmapi-schema.json
index 4153f86..7d255fe 100644
--- a/lib/api/vdsmapi-schema.json
+++ b/lib/api/vdsmapi-schema.json
@@ -7581,7 +7581,7 @@
# A convergence action which will be executed when the migration
# is stalling longer than the first argument.
#
-# @limit: If the migration is stalling for this amount of time,
+# @limit: If the migration is stalling for this amount of iterations,
# execute the action.
#
# @action: The specific action which has to be executed.
diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py
index f74f194..b5228e8 100644
--- a/vdsm/virt/migration.py
+++ b/vdsm/virt/migration.py
@@ -572,6 +572,7 @@
maxTimePerGiB = config.getint('vars',
'migration_max_time_per_gib_mem')
migrationMaxTime = (maxTimePerGiB * memSize + 1023) / 1024
+ progress_timeout = config.getint('vars', 'migration_progress_timeout')
lastProgressTime = time.time()
lowmark = None
lastDataRemaining = None
@@ -583,14 +584,6 @@
self._stop.wait(self._MIGRATION_MONITOR_INTERVAL)
prog = Progress.from_job_stats(self._vm._dom.jobStats())
now = time.time()
- if lastDataRemaining is not None and\
- lastDataRemaining < prog.data_remaining:
- self._vm.log.debug('new iteration detected')
- iterationCount += 1
- if self._use_conv_schedule:
- self._next_action(iterationCount)
-
- lastDataRemaining = prog.data_remaining
if not self._use_conv_schedule and\
(0 < migrationMaxTime < now - self._startTime):
@@ -613,10 +606,23 @@
' Refer to RHBZ#919201.',
prog.data_remaining / Mbytes, lowmark / Mbytes)
- if not self._use_conv_schedule and\
- progress_timeout < now - lastProgressTime:
+ if lastDataRemaining is not None and\
+ lastDataRemaining < prog.data_remaining:
+ iterationCount += 1
+ self._vm.log.debug('new iteration detected: %i',
iterationCount)
+ if self._use_conv_schedule:
+ self._next_action(iterationCount)
- self._next_action(now - lastProgressTime)
+ lastDataRemaining = prog.data_remaining
+
+ if not self._use_conv_schedule and\
+ (now - lastProgressTime) > progress_timeout:
+ # Migration is stuck, abort
+ self._vm.log.warn(
+ 'Migration is stuck: Hasn\'t progressed in %s seconds. '
+ 'Aborting.' % (now - lastProgressTime))
+ self._vm._dom.abortJob()
+ self.stop()
if self._stop.isSet():
break
--
To view, visit https://gerrit.ovirt.org/56558
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6f87c954031842c35c99888c228a34ec7f19d800
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Tomas Jelinek <[email protected]>
_______________________________________________
vdsm-patches mailing list
[email protected]
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
