[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Chris J Arges]

Hello Pavel, or anyone else affected,

Accepted qemu into trusty-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/qemu/2.0.0+dfsg-
2ubuntu1.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: qemu (Ubuntu Trusty)
   Status: Confirmed => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Serge Hallyn]

** No longer affects: qemu (Ubuntu Vivid)

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Launchpad Bug Tracker]

This bug was fixed in the package qemu - 1:2.3+dfsg-5ubuntu6

---
qemu (1:2.3+dfsg-5ubuntu6) wily; urgency=medium

  * Make qemu-system-common and qemu-utils depend on qemu-block-extra
to fix errors with missing block backends. (LP: #1495895)
  * Cherry pick fixes for vmdk stream-optimized subformat (LP: #1006655)
  * Apply fix for memory corruption during live-migration in tcg mode
(LP: #1493049)
  * Apply tracing patch to remove use of custom vtable in newer glibc
(LP: #1491972)

 -- Ryan Harper   Tue, 15 Sep 2015 09:37:23
-0500

** Changed in: qemu (Ubuntu)
   Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: qemu (Ubuntu Vivid)
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: qemu (Ubuntu Trusty)
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Pavel Boldin]

Here is the updated patch for the bug.

All it does is merely setting `tcg_commit' function as
`log_global_start' callback. `tcg_commit' is then flushes all the
appropriate TLBs on `memory_log_global_start' call.

** Patch added: "more clean solution"
   
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+attachment/4459905/+files/tcg-commit-on-log-global-start.patch

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Pavel Boldin]

Applying the attached patch I was able to do around 300 migrations back
and forth successfully.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Pavel Boldin]

@serge-hallyn,

This patch is exactly what fixes the problem for me (was able to do
around 150 successful migrations with it).

However, this should be rewritten to only contain exactly calls to
tlb_flush when in TCG mode so there is no extra code and no unrelated
changes.


The reason patches are different is quite reworked migration and dirtying 
mechanisms in new QEMU.

In exactly, new QEMU KVM has no `log_global_start' handler and the
regions are to be marked as DIRTY_MEMORY_MIGRATE by the `log_start'
handlers which are called only starting with the referenced patch in
upstream. This call is done deeply in memory_region_transaction_commit
-> address_space_update_topology -> *_pass.

Regarding the DIRTY_MEMORY_CODE, in the 2.0.0 it is used only inside the
TCG-related code. Every time a TranslationBlock code is generated for a
given VM code the pages of VM code are removed from TLB cache and marked
as 'clean'. So, on next write to the VM code pages the TLB will be
missed and the TranslationBlock will be updated accordingly.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Ubuntu Foundations Team Bug Bot]

The attachment "backported solution" seems to be a patch.  If it isn't,
please remove the "patch" flag from the attachment, remove the "patch"
tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the
team.

[This is an automated message performed by a Launchpad user owned by
~brian-murray, for any issues please contact him.]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Pavel Boldin]

There is no memory_region_get_dirty_log_mask in the 2.0.0, this was only
introduced later. Strictly speaking, 2.0.0 is quite different from 2.4.0
in this matter but I checked the code logic here and it should be good.

Yes, I was not sure what versions it affects so I only referenced
Trusty. If this affects anything else please make appropriate edits.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Serge Hallyn]

thanks for reporting this bug.

Your backported patch skips the part in render_memory_region() where:

-fr.dirty_log_mask = mr->dirty_log_mask;
+fr.dirty_log_mask = memory_region_get_dirty_log_mask(mr);

Was that on purpose?

Also, in the NOVA bug you said you had filed a 'trusty' bug.  From the
description it seems as though wily should also be affected.  That means
I should fix in wily and SRU to trusty.  Is there a reason why it need
not be fixed in wily?

** Changed in: qemu (Ubuntu)
   Status: New => Incomplete

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Serge Hallyn]

@pboldin,

the patch you cited was one of several (7?) which appear to be related,
including

commit 677e7805cf95f3b2bca8baf0888d1ebed7f0c606
Author: Paolo Bonzini 
Date:   Mon Mar 23 10:53:21 2015 +0100
memory: track DIRTY_MEMORY_CODE in mr->dirty_log_mask
DIRTY_MEMORY_CODE is only needed for TCG.  By adding it directly to
mr->dirty_log_mask, we avoid testing for TCG everywhere a region is
checked for the enabled/disabled state of dirty logging.

Are you certain only that one patch is needed?

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1493049] Re: memory corruption during live-migration in TCG mode

[Serge Hallyn]

Great, thank you.

** Changed in: qemu (Ubuntu)
   Status: Incomplete => Triaged

** Changed in: qemu (Ubuntu)
   Importance: Undecided => High

** Also affects: qemu (Ubuntu Vivid)
   Importance: Undecided
   Status: New

** Also affects: qemu (Ubuntu Trusty)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu in Ubuntu.
https://bugs.launchpad.net/bugs/1493049

Title:
  memory corruption during live-migration in TCG mode

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1493049/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs