Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-17 Thread Stefano Stabellini
On Tue, 9 Jun 2015, Ian Campbell wrote: On Tue, 2015-06-09 at 11:14 +0100, Stefano Stabellini wrote: I don't think that the current solution is inherently racy. If we are really interested in an honest evaluation of the current solution in terms of races, I would be happy to do so.

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-16 Thread Stefano Stabellini
On Tue, 9 Jun 2015, George Dunlap wrote: On 06/09/2015 11:14 AM, Stefano Stabellini wrote: On Mon, 8 Jun 2015, George Dunlap wrote: I think that qemu needs to tell libxl how much memory it is using for all of its needs -- including option ROMs. (See my example below.) For older qemus we

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread Wei Liu
On Tue, Jun 09, 2015 at 11:00:12AM +0100, George Dunlap wrote: I think that qemu needs to tell libxl how much memory it is using for all of its needs -- including option ROMs. (See my example below.) For older qemus we can just make some assumptions like we always have. I am happy

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread George Dunlap
On 06/08/2015 05:06 PM, Don Slutz wrote: On 06/08/15 11:37, George Dunlap wrote: We already have the problem that the balloon driver at the moment doesn't actually know how big the guest RAM is, nor , but is being told to make a balloon exactly big enough to bring the total RAM down to a

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread George Dunlap
On 06/09/2015 11:14 AM, Stefano Stabellini wrote: On Mon, 8 Jun 2015, George Dunlap wrote: I think that qemu needs to tell libxl how much memory it is using for all of its needs -- including option ROMs. (See my example below.) For older qemus we can just make some assumptions like we always

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread Ian Campbell
On Tue, 2015-06-09 at 11:14 +0100, Stefano Stabellini wrote: I don't think that the current solution is inherently racy. If we are really interested in an honest evaluation of the current solution in terms of races, I would be happy to do so. Consider a domain with 1M of RAM (==target and

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Andrew Cooper
On 08/06/15 12:39, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Ian Campbell
On Mon, 2015-06-08 at 14:22 +0100, Stefano Stabellini wrote: On Mon, 8 Jun 2015, Ian Campbell wrote: On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: I disagree that libxl should be the arbitrator of a property that is stored, maintained and enforced by Xen. Xen should

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/05/2015 06:10 PM, Stefano Stabellini wrote: 5. Add a user configurable field in current libxl JSON structure to record how much more memory this domain needs. Admin is required to fill in that value manually. In the mean time we revert the change in QEMU and declare QEMU with

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/08/2015 02:22 PM, Stefano Stabellini wrote: 3. A group of entities which operate in isolation by only ever increasing or descreasing the max pages according to their own requirements, without reference to anyone else. When QEMU entered the fray, and with the

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Andrew Cooper
On 08/06/15 14:38, Stefano Stabellini wrote: Also device-mode/$domid/state is writable by QEMU so we can't trust the content as indicator either. We can because the write happens before we unpause the guest Only when creating the domain fresh. On resume, the guest has possibly had the chance

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Ian Campbell wrote: On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: I disagree that libxl should be the arbitrator of a property that is stored, maintained and enforced by Xen. Xen should be the arbitrator. That's not what arbitrator means, an

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Andrew Cooper wrote: On 08/06/15 14:38, Stefano Stabellini wrote: Also device-mode/$domid/state is writable by QEMU so we can't trust the content as indicator either. We can because the write happens before we unpause the guest Only when creating the domain fresh.

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Wei Liu wrote: On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same (George's translated version):

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Ian Campbell wrote: On Fri, 2015-06-05 at 18:10 +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same (George's translated version):

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Wei Liu
On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: [...] 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2. Having a libxl layer that's not part of migration v2 is a waste of effort. There are several

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Wei Liu
On Mon, Jun 08, 2015 at 02:27:11PM +0100, Stefano Stabellini wrote: On Mon, 8 Jun 2015, Wei Liu wrote: On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: [...] 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2.

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Jan Beulich
On 08.06.15 at 15:01, stefano.stabell...@eu.citrix.com wrote: On Mon, 8 Jun 2015, Andrew Cooper wrote: At the moment, set_max_mem is the only method the toolstack has of putting a hard upper bound on a domains memory usage (along with a few others like the shadow mem size, and PoD cache.)

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Wei Liu wrote: On Mon, Jun 08, 2015 at 02:27:11PM +0100, Stefano Stabellini wrote: On Mon, 8 Jun 2015, Wei Liu wrote: On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: [...] 3. Add a libxl layer that wraps necessary information, take over

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Ian Campbell
On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: I disagree that libxl should be the arbitrator of a property that is stored, maintained and enforced by Xen. Xen should be the arbitrator. That's not what arbitrator means, an arbitrator decides what the value should be,

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Andrew Cooper wrote: On 08/06/15 12:39, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release.

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Wei Liu wrote: On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: [...] 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2. Having a libxl layer that's not part of migration v2 is a

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Don Slutz
On 06/08/15 10:20, George Dunlap wrote: On 06/08/2015 02:22 PM, Stefano Stabellini wrote: 3. A group of entities which operate in isolation by only ever increasing or descreasing the max pages according to their own requirements, without reference to anyone else. When QEMU

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/08/2015 04:01 PM, Don Slutz wrote: On 06/08/15 10:20, George Dunlap wrote: And at the moment, pages in the p2m are allocated by a number of entities: * In the libxc domain builder. * In the guest balloon driver * And now, in qemu, to allocate extra memory for virtual ROMs. This is

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Konrad Rzeszutek Wilk
On Mon, Jun 08, 2015 at 04:20:48PM +0100, George Dunlap wrote: On 06/08/2015 03:53 PM, Konrad Rzeszutek Wilk wrote: On Mon, Jun 08, 2015 at 01:11:38PM +0100, Ian Campbell wrote: On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: I disagree that libxl should be the arbitrator of

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Konrad Rzeszutek Wilk
On Mon, Jun 08, 2015 at 01:11:38PM +0100, Ian Campbell wrote: On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: I disagree that libxl should be the arbitrator of a property that is stored, maintained and enforced by Xen. Xen should be the arbitrator. That's not what

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/08/2015 03:53 PM, Konrad Rzeszutek Wilk wrote: On Mon, Jun 08, 2015 at 01:11:38PM +0100, Ian Campbell wrote: On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: I disagree that libxl should be the arbitrator of a property that is stored, maintained and enforced by Xen. Xen

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Don Slutz
On 06/08/15 11:37, George Dunlap wrote: On 06/08/2015 04:01 PM, Don Slutz wrote: On 06/08/15 10:20, George Dunlap wrote: And at the moment, pages in the p2m are allocated by a number of entities: * In the libxc domain builder. * In the guest balloon driver * And now, in qemu, to allocate

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
On Fri, Jun 05, 2015 at 05:58:11PM +0100, Ian Campbell wrote: On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2. Having a libxl layer that's not part of migration v2 is a waste of

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
On Fri, Jun 05, 2015 at 06:13:01PM +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Ian Campbell wrote: On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2. Having a libxl layer

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same (George's translated version): 1. QEMU may need extra pages from Xen

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Ian Campbell
On Fri, 2015-06-05 at 18:10 +0100, Stefano Stabellini wrote: On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same (George's translated version): 1. QEMU may need extra pages from Xen to

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Ian Campbell wrote: On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2. Having a libxl layer that's not part of migration v2 is a waste of effort. There are

[Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same (George's translated version): 1. QEMU may need extra pages from Xen to implement option ROMS, and so at the moment it calls set_max_mem() to increase max_pages so that it can allocate

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Andrew Cooper
On 05/06/15 17:58, Ian Campbell wrote: On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: 3. Add a libxl layer that wraps necessary information, take over Andrew's work on libxl migration v2. Having a libxl layer that's not part of migration v2 is a waste of effort. There are several

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the problem remain the same (George's translated version): 1. QEMU may need extra pages from Xen to implement option ROMS, and so at the moment it calls set_max_mem() to