Vinzenz Feenstra has uploaded a new change for review. Change subject: Introduce a maximum time limit a migration may take ......................................................................
Introduce a maximum time limit a migration may take There are cases where migrations can take too much time. To set a finite end to this we're now introducing a new configuration value migration_max_time with a default value of 21600 seconds (6 hours) This time tracking is now handled in the MigrationMonitorThread on the source host side of the migration. Change-Id: Ifd2f76b9334fcb7d2db24c081cccae15e8fd0b0c Bug-Url: https://bugzilla.redhat.com/970645 Signed-off-by: Vinzenz Feenstra <[email protected]> --- M lib/vdsm/config.py.in M vdsm/vm.py 2 files changed, 18 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/08/21708/1 diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in index a0a081a..7cc0ef0 100644 --- a/lib/vdsm/config.py.in +++ b/lib/vdsm/config.py.in @@ -55,6 +55,10 @@ 'Please note, that this is not overall migration timeout. ' 'Source waits twice as long (to avoid races).'), + ('migration_max_time', '21600', + 'The maximum time in seconds a migration may take before the ' + 'migration will be aborted.'), + ('migration_listener_timeout', '30', 'Time to wait (in seconds) for migration destination to start ' 'listening before migration begins.'), diff --git a/vdsm/vm.py b/vdsm/vm.py index 4e3650d..9b8db68 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -751,6 +751,8 @@ self._vm.log.debug('starting migration monitor thread') + startTime = time.time() + migrationMaxTime = config.getint('vars', 'migration_max_time') lastProgressTime = time.time() smallest_dataRemaining = None @@ -761,7 +763,15 @@ memTotal, memProcessed, memRemaining, fileTotal, fileProcessed, _) = self._vm._dom.jobInfo() - if (smallest_dataRemaining is None or + abort = False + if time.time() - startTime > migrationMaxTime: + self._vm.log.warn('The migration took %d seconds which is ' + 'exceeding the configured maximum time ' + 'for migrations of %d seconds. The ' + 'migration has been aborted.', + time.time() - startTime, migrationMaxTime) + abort = True + elif (smallest_dataRemaining is None or smallest_dataRemaining > dataRemaining): smallest_dataRemaining = dataRemaining lastProgressTime = time.time() @@ -771,6 +781,9 @@ self._vm.log.warn( 'Migration is stuck: Hasn\'t progressed in %s seconds. ' 'Aborting.' % (time.time() - lastProgressTime)) + abort = True + + if abort: self._vm._dom.abortJob() self.stop() break -- To view, visit http://gerrit.ovirt.org/21708 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifd2f76b9334fcb7d2db24c081cccae15e8fd0b0c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Vinzenz Feenstra <[email protected]> _______________________________________________ vdsm-patches mailing list [email protected] https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
