Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Tue, 6 Oct 2015, Roger Pau Monné wrote: > Do KVM guests also need such extensive set of modifications in order to > run using 64KB pages? I know it's not fair to compare KVM and Xen in > this regard, because the interfaces are very different, but that's what > developers are going to look at. QEMU doesn't need extensive modifications to run 64K guests on a 4K host, but on the other hand KVM needs its own set of non-trivial modifications to do it.
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Tue, 2015-10-06 at 11:28 +0200, Roger Pau Monné wrote: > El 22/09/15 a les 12.59, Ian Campbell ha escrit: > > On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote: > > > I'm not saying that we shouldn't take those patches, I'm just saying > > > that IMHO this is a workaround, and I would like to see a plan and > > > somebody committed to have it fixed in a proper way, by introducing a > > > 64KB PV block protocol. > > > > In my view the basic unit of operation for all Xen interfaces (on x86 > > and > > arm at least[0]) is 4K. The peer at either end of a PV protocol should > > therefore be entitled to assume that at a minimum the other end > > supports > > operation in 4K mode (i.e. 4K is the baseline). > > > > Operations in larger sizes (which would necessarily be multiples of 4K) > > are > > then an extension which is subject to a negotiation between the two > > ends, > > it doesn't really matter if those larger sizes arise because of the use > > of > > superpages or because the guest is internally using some larger basic > > page > > size (which ARM calls a "granule", and where 64K comes from here). > > > > I think this line of reasoning applies just as strongly to the > > hypercall > > ABI as well BTW, they all use 4K as their basic unit, but might support > > extensions to operation on multiples of that (negotiated either via a > > specific error return and fallback or via the use of XENFEAT). > > Yes, I completely agree that the current Xen interface is based on 4KB > chunks. What I'm trying to say is that the approach taken, which is > "let's not modify Xen" puts all the burden on the guest and it is going > to hurt us in the long run. > > Are we expecting all guests that want to use 64KB page to implement all > the modifications that are needed? IMHO, those modifications are very > far from trivial, and we are putting the bar for Xen guest support very > high, and that is wrong. We are already struggling to have a decent set > of PV drivers on guests different than Linux, and this is certainly not > making things easier. I think you are underestimating the impact/burden of not having a common lowest common denominator interface which all front and backends should be expected to provide in order to interact successfully with each other at a basic level. With your approach of requiring a 64KB guest to use native 64K interfaces only you end up with guests containing front and backends which simply cannot communicate at all because they do not have a common denominator or a combinatorial explosion of guests supporting different sets of mutually incompatible interfaces with no common ground. I think that fragmentation is a far worse danger than of developers being unwilling or unable to provide helpers which iterate over pages in whatever size the guest happens to have (either larger granule or superpages) and performing however many of whatever ops size has been negotiated with the peer. I'm also not convinced that Julien's patches are any less trivial than the equivalent code to select the correct operations based on the kernels page size without breaking them up, the developer still needs to be aware of the 4K vs large distinction if they want to support larger than 4K pages. Developers can still choose to only support 4K pages in their OS, if they feel this scheme of supporting larger pages is too intrusive or hard I doubt they are going to want to implement a scheme based on an alternative large-page protocol either. > Do KVM guests also need such extensive set of modifications in order to > run using 64KB pages? I know it's not fair to compare KVM and Xen in > this regard, because the interfaces are very different, but that's what > developers are going to look at. I'm not sure that virtio cares about pages at all, just addresses which have no particular alignment or granularity constraints, since they don't have grant tables nor any requirement to mediate accesses to guest memory by the host at any granularity at all. Ian. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 22/09/15 a les 12.59, Ian Campbell ha escrit: > On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote: >> I'm not saying that we shouldn't take those patches, I'm just saying >> that IMHO this is a workaround, and I would like to see a plan and >> somebody committed to have it fixed in a proper way, by introducing a >> 64KB PV block protocol. > > In my view the basic unit of operation for all Xen interfaces (on x86 and > arm at least[0]) is 4K. The peer at either end of a PV protocol should > therefore be entitled to assume that at a minimum the other end supports > operation in 4K mode (i.e. 4K is the baseline). > > Operations in larger sizes (which would necessarily be multiples of 4K) are > then an extension which is subject to a negotiation between the two ends, > it doesn't really matter if those larger sizes arise because of the use of > superpages or because the guest is internally using some larger basic page > size (which ARM calls a "granule", and where 64K comes from here). > > I think this line of reasoning applies just as strongly to the hypercall > ABI as well BTW, they all use 4K as their basic unit, but might support > extensions to operation on multiples of that (negotiated either via a > specific error return and fallback or via the use of XENFEAT). Yes, I completely agree that the current Xen interface is based on 4KB chunks. What I'm trying to say is that the approach taken, which is "let's not modify Xen" puts all the burden on the guest and it is going to hurt us in the long run. Are we expecting all guests that want to use 64KB page to implement all the modifications that are needed? IMHO, those modifications are very far from trivial, and we are putting the bar for Xen guest support very high, and that is wrong. We are already struggling to have a decent set of PV drivers on guests different than Linux, and this is certainly not making things easier. Do KVM guests also need such extensive set of modifications in order to run using 64KB pages? I know it's not fair to compare KVM and Xen in this regard, because the interfaces are very different, but that's what developers are going to look at. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 22/09/15 a les 12.59, Ian Campbell ha escrit: > On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote: >> I'm not saying that we shouldn't take those patches, I'm just saying >> that IMHO this is a workaround, and I would like to see a plan and >> somebody committed to have it fixed in a proper way, by introducing a >> 64KB PV block protocol. > > In my view the basic unit of operation for all Xen interfaces (on x86 and > arm at least[0]) is 4K. The peer at either end of a PV protocol should > therefore be entitled to assume that at a minimum the other end supports > operation in 4K mode (i.e. 4K is the baseline). > > Operations in larger sizes (which would necessarily be multiples of 4K) are > then an extension which is subject to a negotiation between the two ends, > it doesn't really matter if those larger sizes arise because of the use of > superpages or because the guest is internally using some larger basic page > size (which ARM calls a "granule", and where 64K comes from here). > > I think this line of reasoning applies just as strongly to the hypercall > ABI as well BTW, they all use 4K as their basic unit, but might support > extensions to operation on multiples of that (negotiated either via a > specific error return and fallback or via the use of XENFEAT). Yes, I completely agree that the current Xen interface is based on 4KB chunks. What I'm trying to say is that the approach taken, which is "let's not modify Xen" puts all the burden on the guest and it is going to hurt us in the long run. Are we expecting all guests that want to use 64KB page to implement all the modifications that are needed? IMHO, those modifications are very far from trivial, and we are putting the bar for Xen guest support very high, and that is wrong. We are already struggling to have a decent set of PV drivers on guests different than Linux, and this is certainly not making things easier. Do KVM guests also need such extensive set of modifications in order to run using 64KB pages? I know it's not fair to compare KVM and Xen in this regard, because the interfaces are very different, but that's what developers are going to look at. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Tue, 2015-10-06 at 11:28 +0200, Roger Pau Monné wrote: > El 22/09/15 a les 12.59, Ian Campbell ha escrit: > > On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote: > > > I'm not saying that we shouldn't take those patches, I'm just saying > > > that IMHO this is a workaround, and I would like to see a plan and > > > somebody committed to have it fixed in a proper way, by introducing a > > > 64KB PV block protocol. > > > > In my view the basic unit of operation for all Xen interfaces (on x86 > > and > > arm at least[0]) is 4K. The peer at either end of a PV protocol should > > therefore be entitled to assume that at a minimum the other end > > supports > > operation in 4K mode (i.e. 4K is the baseline). > > > > Operations in larger sizes (which would necessarily be multiples of 4K) > > are > > then an extension which is subject to a negotiation between the two > > ends, > > it doesn't really matter if those larger sizes arise because of the use > > of > > superpages or because the guest is internally using some larger basic > > page > > size (which ARM calls a "granule", and where 64K comes from here). > > > > I think this line of reasoning applies just as strongly to the > > hypercall > > ABI as well BTW, they all use 4K as their basic unit, but might support > > extensions to operation on multiples of that (negotiated either via a > > specific error return and fallback or via the use of XENFEAT). > > Yes, I completely agree that the current Xen interface is based on 4KB > chunks. What I'm trying to say is that the approach taken, which is > "let's not modify Xen" puts all the burden on the guest and it is going > to hurt us in the long run. > > Are we expecting all guests that want to use 64KB page to implement all > the modifications that are needed? IMHO, those modifications are very > far from trivial, and we are putting the bar for Xen guest support very > high, and that is wrong. We are already struggling to have a decent set > of PV drivers on guests different than Linux, and this is certainly not > making things easier. I think you are underestimating the impact/burden of not having a common lowest common denominator interface which all front and backends should be expected to provide in order to interact successfully with each other at a basic level. With your approach of requiring a 64KB guest to use native 64K interfaces only you end up with guests containing front and backends which simply cannot communicate at all because they do not have a common denominator or a combinatorial explosion of guests supporting different sets of mutually incompatible interfaces with no common ground. I think that fragmentation is a far worse danger than of developers being unwilling or unable to provide helpers which iterate over pages in whatever size the guest happens to have (either larger granule or superpages) and performing however many of whatever ops size has been negotiated with the peer. I'm also not convinced that Julien's patches are any less trivial than the equivalent code to select the correct operations based on the kernels page size without breaking them up, the developer still needs to be aware of the 4K vs large distinction if they want to support larger than 4K pages. Developers can still choose to only support 4K pages in their OS, if they feel this scheme of supporting larger pages is too intrusive or hard I doubt they are going to want to implement a scheme based on an alternative large-page protocol either. > Do KVM guests also need such extensive set of modifications in order to > run using 64KB pages? I know it's not fair to compare KVM and Xen in > this regard, because the interfaces are very different, but that's what > developers are going to look at. I'm not sure that virtio cares about pages at all, just addresses which have no particular alignment or granularity constraints, since they don't have grant tables nor any requirement to mediate accesses to guest memory by the host at any granularity at all. Ian. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Tue, 6 Oct 2015, Roger Pau Monné wrote: > Do KVM guests also need such extensive set of modifications in order to > run using 64KB pages? I know it's not fair to compare KVM and Xen in > this regard, because the interfaces are very different, but that's what > developers are going to look at. QEMU doesn't need extensive modifications to run 64K guests on a 4K host, but on the other hand KVM needs its own set of non-trivial modifications to do it.
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 29/09/15 17:33, Julien Grall wrote: > Hi David, > > On 29/09/15 17:27, David Vrabel wrote: >> On 07/09/15 16:33, Julien Grall wrote: >>> >>> A branch based on the latest xentip/for-linus-4.3 can be found here: >>> >>> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 >> >> This has too many conflicts with the >> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4 >> branch. >> >> Please rebase. > > I was about to send a new version rebased (no changes in it so far > except for rebasing purpose). I will do it on top of for-linus-4.4. > I will also include the swiotlb changes at the end as it was reviewed by > Stefano. Thanks. > BTW, some of the patches should, strictly speaking, go via the block and > network subsystem. Although, they heavily depends on other patches in > this series. Would it be fine to send all through xentip? Yes, if they've been acked by the appropriate maintainer. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi David, On 29/09/15 17:27, David Vrabel wrote: > On 07/09/15 16:33, Julien Grall wrote: >> >> A branch based on the latest xentip/for-linus-4.3 can be found here: >> >> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 > > This has too many conflicts with the > git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4 > branch. > > Please rebase. I was about to send a new version rebased (no changes in it so far except for rebasing purpose). I will do it on top of for-linus-4.4. I will also include the swiotlb changes at the end as it was reviewed by Stefano. BTW, some of the patches should, strictly speaking, go via the block and network subsystem. Although, they heavily depends on other patches in this series. Would it be fine to send all through xentip? Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 07/09/15 16:33, Julien Grall wrote: > > A branch based on the latest xentip/for-linus-4.3 can be found here: > > git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 This has too many conflicts with the git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4 branch. Please rebase. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 07/09/15 16:33, Julien Grall wrote: > > A branch based on the latest xentip/for-linus-4.3 can be found here: > > git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 This has too many conflicts with the git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4 branch. Please rebase. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 29/09/15 17:33, Julien Grall wrote: > Hi David, > > On 29/09/15 17:27, David Vrabel wrote: >> On 07/09/15 16:33, Julien Grall wrote: >>> >>> A branch based on the latest xentip/for-linus-4.3 can be found here: >>> >>> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 >> >> This has too many conflicts with the >> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4 >> branch. >> >> Please rebase. > > I was about to send a new version rebased (no changes in it so far > except for rebasing purpose). I will do it on top of for-linus-4.4. > I will also include the swiotlb changes at the end as it was reviewed by > Stefano. Thanks. > BTW, some of the patches should, strictly speaking, go via the block and > network subsystem. Although, they heavily depends on other patches in > this series. Would it be fine to send all through xentip? Yes, if they've been acked by the appropriate maintainer. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi David, On 29/09/15 17:27, David Vrabel wrote: > On 07/09/15 16:33, Julien Grall wrote: >> >> A branch based on the latest xentip/for-linus-4.3 can be found here: >> >> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 > > This has too many conflicts with the > git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4 > branch. > > Please rebase. I was about to send a new version rebased (no changes in it so far except for rebasing purpose). I will do it on top of for-linus-4.4. I will also include the swiotlb changes at the end as it was reviewed by Stefano. BTW, some of the patches should, strictly speaking, go via the block and network subsystem. Although, they heavily depends on other patches in this series. Would it be fine to send all through xentip? Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote: > I'm not saying that we shouldn't take those patches, I'm just saying > that IMHO this is a workaround, and I would like to see a plan and > somebody committed to have it fixed in a proper way, by introducing a > 64KB PV block protocol. In my view the basic unit of operation for all Xen interfaces (on x86 and arm at least[0]) is 4K. The peer at either end of a PV protocol should therefore be entitled to assume that at a minimum the other end supports operation in 4K mode (i.e. 4K is the baseline). Operations in larger sizes (which would necessarily be multiples of 4K) are then an extension which is subject to a negotiation between the two ends, it doesn't really matter if those larger sizes arise because of the use of superpages or because the guest is internally using some larger basic page size (which ARM calls a "granule", and where 64K comes from here). I think this line of reasoning applies just as strongly to the hypercall ABI as well BTW, they all use 4K as their basic unit, but might support extensions to operation on multiples of that (negotiated either via a specific error return and fallback or via the use of XENFEAT). Ian. [0] It's not implausible that some other architecture may exsit which doesn't use 4K as the basic page szie) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote: > I'm not saying that we shouldn't take those patches, I'm just saying > that IMHO this is a workaround, and I would like to see a plan and > somebody committed to have it fixed in a proper way, by introducing a > 64KB PV block protocol. In my view the basic unit of operation for all Xen interfaces (on x86 and arm at least[0]) is 4K. The peer at either end of a PV protocol should therefore be entitled to assume that at a minimum the other end supports operation in 4K mode (i.e. 4K is the baseline). Operations in larger sizes (which would necessarily be multiples of 4K) are then an extension which is subject to a negotiation between the two ends, it doesn't really matter if those larger sizes arise because of the use of superpages or because the guest is internally using some larger basic page size (which ARM calls a "granule", and where 64K comes from here). I think this line of reasoning applies just as strongly to the hypercall ABI as well BTW, they all use 4K as their basic unit, but might support extensions to operation on multiples of that (negotiated either via a specific error return and fallback or via the use of XENFEAT). Ian. [0] It's not implausible that some other architecture may exsit which doesn't use 4K as the basic page szie) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi Roger, On 14/09/15 13:08, Roger Pau Monné wrote: > Well, absolute numbers together with the standard deviation are IMHO the > best way to provide those figures (ie: see ministat(1) output for > example), but percentages should also be fine. > > I'm just interested in knowing the performance difference between having > this patches applied or not when using 4KB pages on the frontend and the > backend. I did some benchmark: DOM0: 1 VCPU, 4G of RAM based on 4.3-rc1 without this series GUEST: 4 VPUs, 4G of RAM, second disk associate to a nullbk device I used fio with the following options 42sh> fio --name=test --ioengine=libaio --rw=read --numjobs=8 \ --iodepth=32 --time_based=1 --runtime=30 --bs=4KB \ --filename=/dev/xvdb--direct=1 --group_reporting=1 --iodepth_batch=16 The guest is also based on 4.3-rc1 with and without the series. The overhead with my series is about 0.56%. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi Roger, On 14/09/15 13:08, Roger Pau Monné wrote: > Well, absolute numbers together with the standard deviation are IMHO the > best way to provide those figures (ie: see ministat(1) output for > example), but percentages should also be fine. > > I'm just interested in knowing the performance difference between having > this patches applied or not when using 4KB pages on the frontend and the > backend. I did some benchmark: DOM0: 1 VCPU, 4G of RAM based on 4.3-rc1 without this series GUEST: 4 VPUs, 4G of RAM, second disk associate to a nullbk device I used fio with the following options 42sh> fio --name=test --ioengine=libaio --rw=read --numjobs=8 \ --iodepth=32 --time_based=1 --runtime=30 --bs=4KB \ --filename=/dev/xvdb--direct=1 --group_reporting=1 --iodepth_batch=16 The guest is also based on 4.3-rc1 with and without the series. The overhead with my series is about 0.56%. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Tuesday 15 September 2015 14:14:09 David Vrabel wrote: > On 14/09/15 12:32, Arnd Bergmann wrote: > > On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote: > >>> TBH, I'm expecting a small impact to the performance. It would be hard > >>> to get the exactly the same performance as today if we keep the helpers > >>> to avoid the backend dealing himself with the splitting and page > >>> granularity. > >>> > >>> Although, if the performance impact is not acceptable, it may be > >>> possible to optimize gnttab_foreach_grant_in_range by moving the > >>> function inline. The current way to the loop is the fastest I've found > >>> (I've wrote a small program to test different way) and we will need it > >>> when different of size will be supported. > >> > >> I don't expect the performance to drop massively with this patches > >> applied, but it would be good to al least have an idea of the impact. > > > > Note that using 64kb pages in Linux tends to destroy performance > > in Linux in any case, as the memory consumption for most workloads > > explodes. In a virtualized environment you already tend to be > > memory constrained, so any measurement should take that into account > > and put the extra overhead into perspective to the massive overhead > > of running 64kb pages when RAM is tight. > > If this is the case, why are some distros using 64 KiB pages then? I believe that IBM started this on PowerPC64 when they found that it performs better in microbenchmarks and for certain database workloads when you do have enough memory and do not work with small files or many processes. I also believe that the AIX memory management code is designed differently enough that it suffers less from large pages (or suffers more from small pages), so the hardware designers pushed Linux to do the same as AIX. Why you'd do it on ARM64 I have no idea at all, other than copying what IBM did on PowerPC. If your TLB is too small, hardware designers may of course try to push the OS to using pages as large as possible for the low-level benchmarks to suck less, but in general, you'd be better off with transparent hugepages and 4kb or 8kb pages for real workloads. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 12:32, Arnd Bergmann wrote: > On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote: >>> TBH, I'm expecting a small impact to the performance. It would be hard >>> to get the exactly the same performance as today if we keep the helpers >>> to avoid the backend dealing himself with the splitting and page >>> granularity. >>> >>> Although, if the performance impact is not acceptable, it may be >>> possible to optimize gnttab_foreach_grant_in_range by moving the >>> function inline. The current way to the loop is the fastest I've found >>> (I've wrote a small program to test different way) and we will need it >>> when different of size will be supported. >> >> I don't expect the performance to drop massively with this patches >> applied, but it would be good to al least have an idea of the impact. > > Note that using 64kb pages in Linux tends to destroy performance > in Linux in any case, as the memory consumption for most workloads > explodes. In a virtualized environment you already tend to be > memory constrained, so any measurement should take that into account > and put the extra overhead into perspective to the massive overhead > of running 64kb pages when RAM is tight. If this is the case, why are some distros using 64 KiB pages then? David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 12:32, Arnd Bergmann wrote: > On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote: >>> TBH, I'm expecting a small impact to the performance. It would be hard >>> to get the exactly the same performance as today if we keep the helpers >>> to avoid the backend dealing himself with the splitting and page >>> granularity. >>> >>> Although, if the performance impact is not acceptable, it may be >>> possible to optimize gnttab_foreach_grant_in_range by moving the >>> function inline. The current way to the loop is the fastest I've found >>> (I've wrote a small program to test different way) and we will need it >>> when different of size will be supported. >> >> I don't expect the performance to drop massively with this patches >> applied, but it would be good to al least have an idea of the impact. > > Note that using 64kb pages in Linux tends to destroy performance > in Linux in any case, as the memory consumption for most workloads > explodes. In a virtualized environment you already tend to be > memory constrained, so any measurement should take that into account > and put the extra overhead into perspective to the massive overhead > of running 64kb pages when RAM is tight. If this is the case, why are some distros using 64 KiB pages then? David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Tuesday 15 September 2015 14:14:09 David Vrabel wrote: > On 14/09/15 12:32, Arnd Bergmann wrote: > > On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote: > >>> TBH, I'm expecting a small impact to the performance. It would be hard > >>> to get the exactly the same performance as today if we keep the helpers > >>> to avoid the backend dealing himself with the splitting and page > >>> granularity. > >>> > >>> Although, if the performance impact is not acceptable, it may be > >>> possible to optimize gnttab_foreach_grant_in_range by moving the > >>> function inline. The current way to the loop is the fastest I've found > >>> (I've wrote a small program to test different way) and we will need it > >>> when different of size will be supported. > >> > >> I don't expect the performance to drop massively with this patches > >> applied, but it would be good to al least have an idea of the impact. > > > > Note that using 64kb pages in Linux tends to destroy performance > > in Linux in any case, as the memory consumption for most workloads > > explodes. In a virtualized environment you already tend to be > > memory constrained, so any measurement should take that into account > > and put the extra overhead into perspective to the massive overhead > > of running 64kb pages when RAM is tight. > > If this is the case, why are some distros using 64 KiB pages then? I believe that IBM started this on PowerPC64 when they found that it performs better in microbenchmarks and for certain database workloads when you do have enough memory and do not work with small files or many processes. I also believe that the AIX memory management code is designed differently enough that it suffers less from large pages (or suffers more from small pages), so the hardware designers pushed Linux to do the same as AIX. Why you'd do it on ARM64 I have no idea at all, other than copying what IBM did on PowerPC. If your TLB is too small, hardware designers may of course try to push the OS to using pages as large as possible for the low-level benchmarks to suck less, but in general, you'd be better off with transparent hugepages and 4kb or 8kb pages for real workloads. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 14/09/15 a les 16.54, Stefano Stabellini ha escrit: > On Mon, 14 Sep 2015, Roger Pau Monné wrote: >> IMHO this splitting is just a workaround for the fact that we don't have >> a 64KB PV block protocol, and this is the real problem that should be >> solved. > > 64K is a pure one guest kernel configuration option, not a platform wide > option. The hypervisor interfaces are still the same, the ABI is the > same and all the other guests are still the same, the Xen binary is > still the same. Yes, I understand that, but the PV block protocol is missing 64KB page support, and that's a fact that cannot be ignored. To put an example, is there a hardware SATA controller on ARM that doesn't support 64KB pages and needs a similar workaround? > A 64K block protocol could be a good performance imprevement, but should > not be required to run kernels which have different config options. > >> In the long term this will put a burden on all blkfronts (if 64KB pages >> are also used by other OSes), while introducing a 64KB PV block protocol >> will make the blkfront implementation in all OSes very similar to what >> we have now, without replicating the splitting code amongst all the >> possible blkfront implementations. >> >> Granted that some changes to blkback will be needed in order to support >> mapping 64KB grants, but there are much fewer blkback implementations >> out there than blkfronts. > > I don't think we can rely on blkback having something in order to run > new guests, otherwise we break compatibility: new guests won't run on > old hypervisors. I agree that this is far from ideal, but I don't think it's so outrageous. For example Linux PVOPS Dom0 kernels require Xen 4.0.1 at least in order to run, because previous versions lack the necessary IOAPIC setup hypercall. Also, it won't prevent guests from booting, it would just prevent them from using blkfront, but you can still get a root filesystem using iSCSI, NFS or other means. I'm not saying that we shouldn't take those patches, I'm just saying that IMHO this is a workaround, and I would like to see a plan and somebody committed to have it fixed in a proper way, by introducing a 64KB PV block protocol. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Mon, 14 Sep 2015, Roger Pau Monné wrote: > IMHO this splitting is just a workaround for the fact that we don't have > a 64KB PV block protocol, and this is the real problem that should be > solved. 64K is a pure one guest kernel configuration option, not a platform wide option. The hypervisor interfaces are still the same, the ABI is the same and all the other guests are still the same, the Xen binary is still the same. A 64K block protocol could be a good performance imprevement, but should not be required to run kernels which have different config options. > In the long term this will put a burden on all blkfronts (if 64KB pages > are also used by other OSes), while introducing a 64KB PV block protocol > will make the blkfront implementation in all OSes very similar to what > we have now, without replicating the splitting code amongst all the > possible blkfront implementations. > > Granted that some changes to blkback will be needed in order to support > mapping 64KB grants, but there are much fewer blkback implementations > out there than blkfronts. I don't think we can rely on blkback having something in order to run new guests, otherwise we break compatibility: new guests won't run on old hypervisors.
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 15:29, Roger Pau Monné wrote: >> To give you an example, Centos 7, which will support Xen and only 64KB >> page granularity, will be supported for years. Dropping any splitting in >> a short future (3-5 years) will just break those guests to boot on Xen. > > Can't the patches to support 64KB grants be backported to CentOS? In the > past you said that distros had no problem in picking the needed Xen > patches for 64KB support, I don't see how that would be different. Centos 7 is not yet released so it's more easy to get patch in. I don't know what will be position when the guest is shipped. I bet they will still want to support 64KB with older Xen (i.e where 64KB grant is not supported). Anyway, I don't expect to see the 64KB grants support before next year as my TODO list is already pretty full. So let's talk about it when I (or someone else) will send an RFC to extend the grant size. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hello, El 14/09/15 a les 14.47, Julien Grall ha escrit: > On 14/09/15 13:08, Roger Pau Monné wrote: >>> I'd like to see a basic support of 64KB page granularity upstream before >>> starting to think about performance improvement. And there is still a >>> lot to do. >> >> I wasn't actually thinking of this as a performance improvement, but >> rather as a way of fixing this properly and removing the complexity >> added to xen-blkfront, thus providing a PV block protocol that natively >> supports 64KB pages. This would be like hardware components already do >> AFAIK, because I don't think there are other drivers in the Linux tree >> that do this kind of splitting. >> >>> Although, having 64KB grants won't remove the splitting as we would >>> still have to support frontend/backend which only handle 4KB grant. >> >> IMHO, given enough time I would just drop those. > > The PV protocol has always been backward compatible. > > Introducing the support of 64KB grant will require some changes in Xen > which won't surely be backported to older version. We still have to be > able to run new Linux guest on older Xen and vice versa. OK, so let's wait a couple of Xen and Linux releases before dropping this from the Linux kernel once 64KB grant support is added. Guests should still be able to boot, it's just the disk that won't be attached. IMHO this splitting is just a workaround for the fact that we don't have a 64KB PV block protocol, and this is the real problem that should be solved. In the long term this will put a burden on all blkfronts (if 64KB pages are also used by other OSes), while introducing a 64KB PV block protocol will make the blkfront implementation in all OSes very similar to what we have now, without replicating the splitting code amongst all the possible blkfront implementations. Granted that some changes to blkback will be needed in order to support mapping 64KB grants, but there are much fewer blkback implementations out there than blkfronts. > To give you an example, Centos 7, which will support Xen and only 64KB > page granularity, will be supported for years. Dropping any splitting in > a short future (3-5 years) will just break those guests to boot on Xen. Can't the patches to support 64KB grants be backported to CentOS? In the past you said that distros had no problem in picking the needed Xen patches for 64KB support, I don't see how that would be different. Also, a blkfront with the splitting code should also be able to work perfectly fine once we have a 64KB PV block protocol, it will just be treated like it's using 4KB pages, that's all. > AFAIK, we never took a such radical decision on Xen based on the > complexity of the code. We never had to deal with a problem like this I'm afraid (mixing guests with different page sizes), so I don't think we can make statements about that. And it's not just about the complexity, it's because as stated above this IMHO is just sweeping the real problem under the carpet instead of solving it. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 13:08, Roger Pau Monné wrote: >> I'd like to see a basic support of 64KB page granularity upstream before >> starting to think about performance improvement. And there is still a >> lot to do. > > I wasn't actually thinking of this as a performance improvement, but > rather as a way of fixing this properly and removing the complexity > added to xen-blkfront, thus providing a PV block protocol that natively > supports 64KB pages. This would be like hardware components already do > AFAIK, because I don't think there are other drivers in the Linux tree > that do this kind of splitting. > >> Although, having 64KB grants won't remove the splitting as we would >> still have to support frontend/backend which only handle 4KB grant. > > IMHO, given enough time I would just drop those. The PV protocol has always been backward compatible. Introducing the support of 64KB grant will require some changes in Xen which won't surely be backported to older version. We still have to be able to run new Linux guest on older Xen and vice versa. To give you an example, Centos 7, which will support Xen and only 64KB page granularity, will be supported for years. Dropping any splitting in a short future (3-5 years) will just break those guests to boot on Xen. AFAIK, we never took a such radical decision on Xen based on the complexity of the code. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 14/09/15 a les 13.21, Julien Grall ha escrit: > On 14/09/15 12:04, Roger Pau Monné wrote: >> Hello, >> >> El 14/09/15 a les 12.40, Julien Grall ha escrit: >>> Hi Roger, >>> >>> On 14/09/15 09:56, Roger Pau Monné wrote: El 07/09/15 a les 17.33, Julien Grall ha escrit: > Hi all, > > ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, > Xen > hypercall interface and PV protocol are always based on 4KB page > granularity. > > Any attempt to boot a Linux guest with 64KB pages enabled will result to a > guest crash. > > This series is a first attempt to allow those Linux running with the > current > hypercall interface and PV protocol. > > This solution has been chosen because we want to run Linux 64KB in > released > Xen ARM version or/and platform using an old version of Linux DOM0. > > There is room for improvement, such as support of 64KB grant, modification > of PV protocol to support different page size... They will be explored in > a > separate patch series later. > > TODO list: > - Convert swiotlb to 64KB > - Convert xenfb to 64KB > - Support for multiple page ring support > - Support for 64KB in gnttdev > - Support of non-indirect grant with 64KB frontend > - It may be possible to move some common define between > netback/netfront and blkfront/blkback in an header > > I've got most of the patches for the TODO items. I'm planning to send > them as > a follow-up as it's not a requirement for a basic guests. > > All patches has been built tested for ARM32, ARM64, x86. But I haven't > tested > to run it on x86 as I don't have a box with Xen x86 running. I would be > happy if someone give a try and see possible regression for x86. Do you have figures regarding if/how much performance penalty do the blkfront/blkback patches add to the traditional 4KB scenario (frontend and backend running on guests using 4KB pages)? >>> >>> Which benchmark do you advice? Although, I don't have SSD on this >>> platform so it may be possible that the bottleneck will be the hard drive. >> >> I've normally used a ramdisk in order to test performance, but it seems >> nullblk would be a better option (it wasn't available before) and it >> doesn't reduce the size of RAM available to the system: >> >> https://www.kernel.org/doc/Documentation/block/null_blk.txt > > I will give a look to this. > Since there's been no design document about this and the TODO list doesn't contain it, I would like to know which plans do we have in order to fix this properly. >>> >>> Can you explain what kind of design document you were expecting? The >>> support of 64KB page granularity is pretty straight-forward and there is >>> not many way to do it. We have to split the page in 4KB chunk to feed >>> the ring. >> >> I was thinking about adding support for 64KB grants with the aim of >> eventually removing the splitting done in xen-blkfront and the >> grant-table code in general. > > I'd like to see a basic support of 64KB page granularity upstream before > starting to think about performance improvement. And there is still a > lot to do. I wasn't actually thinking of this as a performance improvement, but rather as a way of fixing this properly and removing the complexity added to xen-blkfront, thus providing a PV block protocol that natively supports 64KB pages. This would be like hardware components already do AFAIK, because I don't think there are other drivers in the Linux tree that do this kind of splitting. > Although, having 64KB grants won't remove the splitting as we would > still have to support frontend/backend which only handle 4KB grant. IMHO, given enough time I would just drop those. >>> TBH, I'm expecting a small impact to the performance. It would be hard >>> to get the exactly the same performance as today if we keep the helpers >>> to avoid the backend dealing himself with the splitting and page >>> granularity. >>> >>> Although, if the performance impact is not acceptable, it may be >>> possible to optimize gnttab_foreach_grant_in_range by moving the >>> function inline. The current way to the loop is the fastest I've found >>> (I've wrote a small program to test different way) and we will need it >>> when different of size will be supported. >> >> I don't expect the performance to drop massively with this patches >> applied, but it would be good to al least have an idea of the impact. > > I will only be able to give percentage. I guess this would be sufficient > for you? Well, absolute numbers together with the standard deviation are IMHO the best way to provide those figures (ie: see ministat(1) output for example), but percentages should also be fine. I'm just interested in knowing the performance difference between having this patches applied or not when using
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote: > > TBH, I'm expecting a small impact to the performance. It would be hard > > to get the exactly the same performance as today if we keep the helpers > > to avoid the backend dealing himself with the splitting and page > > granularity. > > > > Although, if the performance impact is not acceptable, it may be > > possible to optimize gnttab_foreach_grant_in_range by moving the > > function inline. The current way to the loop is the fastest I've found > > (I've wrote a small program to test different way) and we will need it > > when different of size will be supported. > > I don't expect the performance to drop massively with this patches > applied, but it would be good to al least have an idea of the impact. Note that using 64kb pages in Linux tends to destroy performance in Linux in any case, as the memory consumption for most workloads explodes. In a virtualized environment you already tend to be memory constrained, so any measurement should take that into account and put the extra overhead into perspective to the massive overhead of running 64kb pages when RAM is tight. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 12:04, Roger Pau Monné wrote: > Hello, > > El 14/09/15 a les 12.40, Julien Grall ha escrit: >> Hi Roger, >> >> On 14/09/15 09:56, Roger Pau Monné wrote: >>> El 07/09/15 a les 17.33, Julien Grall ha escrit: Hi all, ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen hypercall interface and PV protocol are always based on 4KB page granularity. Any attempt to boot a Linux guest with 64KB pages enabled will result to a guest crash. This series is a first attempt to allow those Linux running with the current hypercall interface and PV protocol. This solution has been chosen because we want to run Linux 64KB in released Xen ARM version or/and platform using an old version of Linux DOM0. There is room for improvement, such as support of 64KB grant, modification of PV protocol to support different page size... They will be explored in a separate patch series later. TODO list: - Convert swiotlb to 64KB - Convert xenfb to 64KB - Support for multiple page ring support - Support for 64KB in gnttdev - Support of non-indirect grant with 64KB frontend - It may be possible to move some common define between netback/netfront and blkfront/blkback in an header I've got most of the patches for the TODO items. I'm planning to send them as a follow-up as it's not a requirement for a basic guests. All patches has been built tested for ARM32, ARM64, x86. But I haven't tested to run it on x86 as I don't have a box with Xen x86 running. I would be happy if someone give a try and see possible regression for x86. >>> >>> Do you have figures regarding if/how much performance penalty do the >>> blkfront/blkback patches add to the traditional 4KB scenario (frontend >>> and backend running on guests using 4KB pages)? >> >> Which benchmark do you advice? Although, I don't have SSD on this >> platform so it may be possible that the bottleneck will be the hard drive. > > I've normally used a ramdisk in order to test performance, but it seems > nullblk would be a better option (it wasn't available before) and it > doesn't reduce the size of RAM available to the system: > > https://www.kernel.org/doc/Documentation/block/null_blk.txt I will give a look to this. >>> >>> Since there's been no design document about this and the TODO list >>> doesn't contain it, I would like to know which plans do we have in order >>> to fix this properly. >> >> Can you explain what kind of design document you were expecting? The >> support of 64KB page granularity is pretty straight-forward and there is >> not many way to do it. We have to split the page in 4KB chunk to feed >> the ring. > > I was thinking about adding support for 64KB grants with the aim of > eventually removing the splitting done in xen-blkfront and the > grant-table code in general. I'd like to see a basic support of 64KB page granularity upstream before starting to think about performance improvement. And there is still a lot to do. Although, having 64KB grants won't remove the splitting as we would still have to support frontend/backend which only handle 4KB grant. >> TBH, I'm expecting a small impact to the performance. It would be hard >> to get the exactly the same performance as today if we keep the helpers >> to avoid the backend dealing himself with the splitting and page >> granularity. >> >> Although, if the performance impact is not acceptable, it may be >> possible to optimize gnttab_foreach_grant_in_range by moving the >> function inline. The current way to the loop is the fastest I've found >> (I've wrote a small program to test different way) and we will need it >> when different of size will be supported. > > I don't expect the performance to drop massively with this patches > applied, but it would be good to al least have an idea of the impact. I will only be able to give percentage. I guess this would be sufficient for you? Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hello, El 14/09/15 a les 12.40, Julien Grall ha escrit: > Hi Roger, > > On 14/09/15 09:56, Roger Pau Monné wrote: >> El 07/09/15 a les 17.33, Julien Grall ha escrit: >>> Hi all, >>> >>> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen >>> hypercall interface and PV protocol are always based on 4KB page >>> granularity. >>> >>> Any attempt to boot a Linux guest with 64KB pages enabled will result to a >>> guest crash. >>> >>> This series is a first attempt to allow those Linux running with the current >>> hypercall interface and PV protocol. >>> >>> This solution has been chosen because we want to run Linux 64KB in released >>> Xen ARM version or/and platform using an old version of Linux DOM0. >>> >>> There is room for improvement, such as support of 64KB grant, modification >>> of PV protocol to support different page size... They will be explored in a >>> separate patch series later. >>> >>> TODO list: >>> - Convert swiotlb to 64KB >>> - Convert xenfb to 64KB >>> - Support for multiple page ring support >>> - Support for 64KB in gnttdev >>> - Support of non-indirect grant with 64KB frontend >>> - It may be possible to move some common define between >>> netback/netfront and blkfront/blkback in an header >>> >>> I've got most of the patches for the TODO items. I'm planning to send them >>> as >>> a follow-up as it's not a requirement for a basic guests. >>> >>> All patches has been built tested for ARM32, ARM64, x86. But I haven't >>> tested >>> to run it on x86 as I don't have a box with Xen x86 running. I would be >>> happy if someone give a try and see possible regression for x86. >> >> Do you have figures regarding if/how much performance penalty do the >> blkfront/blkback patches add to the traditional 4KB scenario (frontend >> and backend running on guests using 4KB pages)? > > Which benchmark do you advice? Although, I don't have SSD on this > platform so it may be possible that the bottleneck will be the hard drive. I've normally used a ramdisk in order to test performance, but it seems nullblk would be a better option (it wasn't available before) and it doesn't reduce the size of RAM available to the system: https://www.kernel.org/doc/Documentation/block/null_blk.txt >> >> Since there's been no design document about this and the TODO list >> doesn't contain it, I would like to know which plans do we have in order >> to fix this properly. > > Can you explain what kind of design document you were expecting? The > support of 64KB page granularity is pretty straight-forward and there is > not many way to do it. We have to split the page in 4KB chunk to feed > the ring. I was thinking about adding support for 64KB grants with the aim of eventually removing the splitting done in xen-blkfront and the grant-table code in general. Some modifications would be needed in xen-blkback in order to support mapping 64KB grants, but this could also be used by x86 if we ever enable 2MB grants and allow them to be used in the block PV protocol, while the current code can only be used by ARM. > TBH, I'm expecting a small impact to the performance. It would be hard > to get the exactly the same performance as today if we keep the helpers > to avoid the backend dealing himself with the splitting and page > granularity. > > Although, if the performance impact is not acceptable, it may be > possible to optimize gnttab_foreach_grant_in_range by moving the > function inline. The current way to the loop is the fastest I've found > (I've wrote a small program to test different way) and we will need it > when different of size will be supported. I don't expect the performance to drop massively with this patches applied, but it would be good to al least have an idea of the impact. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi Roger, On 14/09/15 09:56, Roger Pau Monné wrote: > El 07/09/15 a les 17.33, Julien Grall ha escrit: >> Hi all, >> >> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen >> hypercall interface and PV protocol are always based on 4KB page granularity. >> >> Any attempt to boot a Linux guest with 64KB pages enabled will result to a >> guest crash. >> >> This series is a first attempt to allow those Linux running with the current >> hypercall interface and PV protocol. >> >> This solution has been chosen because we want to run Linux 64KB in released >> Xen ARM version or/and platform using an old version of Linux DOM0. >> >> There is room for improvement, such as support of 64KB grant, modification >> of PV protocol to support different page size... They will be explored in a >> separate patch series later. >> >> TODO list: >> - Convert swiotlb to 64KB >> - Convert xenfb to 64KB >> - Support for multiple page ring support >> - Support for 64KB in gnttdev >> - Support of non-indirect grant with 64KB frontend >> - It may be possible to move some common define between >> netback/netfront and blkfront/blkback in an header >> >> I've got most of the patches for the TODO items. I'm planning to send them as >> a follow-up as it's not a requirement for a basic guests. >> >> All patches has been built tested for ARM32, ARM64, x86. But I haven't tested >> to run it on x86 as I don't have a box with Xen x86 running. I would be >> happy if someone give a try and see possible regression for x86. > > Do you have figures regarding if/how much performance penalty do the > blkfront/blkback patches add to the traditional 4KB scenario (frontend > and backend running on guests using 4KB pages)? Which benchmark do you advice? Although, I don't have SSD on this platform so it may be possible that the bottleneck will be the hard drive. > > Since there's been no design document about this and the TODO list > doesn't contain it, I would like to know which plans do we have in order > to fix this properly. Can you explain what kind of design document you were expecting? The support of 64KB page granularity is pretty straight-forward and there is not many way to do it. We have to split the page in 4KB chunk to feed the ring. TBH, I'm expecting a small impact to the performance. It would be hard to get the exactly the same performance as today if we keep the helpers to avoid the backend dealing himself with the splitting and page granularity. Although, if the performance impact is not acceptable, it may be possible to optimize gnttab_foreach_grant_in_range by moving the function inline. The current way to the loop is the fastest I've found (I've wrote a small program to test different way) and we will need it when different of size will be supported. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 07/09/15 a les 17.33, Julien Grall ha escrit: > Hi all, > > ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen > hypercall interface and PV protocol are always based on 4KB page granularity. > > Any attempt to boot a Linux guest with 64KB pages enabled will result to a > guest crash. > > This series is a first attempt to allow those Linux running with the current > hypercall interface and PV protocol. > > This solution has been chosen because we want to run Linux 64KB in released > Xen ARM version or/and platform using an old version of Linux DOM0. > > There is room for improvement, such as support of 64KB grant, modification > of PV protocol to support different page size... They will be explored in a > separate patch series later. > > TODO list: > - Convert swiotlb to 64KB > - Convert xenfb to 64KB > - Support for multiple page ring support > - Support for 64KB in gnttdev > - Support of non-indirect grant with 64KB frontend > - It may be possible to move some common define between > netback/netfront and blkfront/blkback in an header > > I've got most of the patches for the TODO items. I'm planning to send them as > a follow-up as it's not a requirement for a basic guests. > > All patches has been built tested for ARM32, ARM64, x86. But I haven't tested > to run it on x86 as I don't have a box with Xen x86 running. I would be > happy if someone give a try and see possible regression for x86. Do you have figures regarding if/how much performance penalty do the blkfront/blkback patches add to the traditional 4KB scenario (frontend and backend running on guests using 4KB pages)? Since there's been no design document about this and the TODO list doesn't contain it, I would like to know which plans do we have in order to fix this properly. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 12:04, Roger Pau Monné wrote: > Hello, > > El 14/09/15 a les 12.40, Julien Grall ha escrit: >> Hi Roger, >> >> On 14/09/15 09:56, Roger Pau Monné wrote: >>> El 07/09/15 a les 17.33, Julien Grall ha escrit: Hi all, ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen hypercall interface and PV protocol are always based on 4KB page granularity. Any attempt to boot a Linux guest with 64KB pages enabled will result to a guest crash. This series is a first attempt to allow those Linux running with the current hypercall interface and PV protocol. This solution has been chosen because we want to run Linux 64KB in released Xen ARM version or/and platform using an old version of Linux DOM0. There is room for improvement, such as support of 64KB grant, modification of PV protocol to support different page size... They will be explored in a separate patch series later. TODO list: - Convert swiotlb to 64KB - Convert xenfb to 64KB - Support for multiple page ring support - Support for 64KB in gnttdev - Support of non-indirect grant with 64KB frontend - It may be possible to move some common define between netback/netfront and blkfront/blkback in an header I've got most of the patches for the TODO items. I'm planning to send them as a follow-up as it's not a requirement for a basic guests. All patches has been built tested for ARM32, ARM64, x86. But I haven't tested to run it on x86 as I don't have a box with Xen x86 running. I would be happy if someone give a try and see possible regression for x86. >>> >>> Do you have figures regarding if/how much performance penalty do the >>> blkfront/blkback patches add to the traditional 4KB scenario (frontend >>> and backend running on guests using 4KB pages)? >> >> Which benchmark do you advice? Although, I don't have SSD on this >> platform so it may be possible that the bottleneck will be the hard drive. > > I've normally used a ramdisk in order to test performance, but it seems > nullblk would be a better option (it wasn't available before) and it > doesn't reduce the size of RAM available to the system: > > https://www.kernel.org/doc/Documentation/block/null_blk.txt I will give a look to this. >>> >>> Since there's been no design document about this and the TODO list >>> doesn't contain it, I would like to know which plans do we have in order >>> to fix this properly. >> >> Can you explain what kind of design document you were expecting? The >> support of 64KB page granularity is pretty straight-forward and there is >> not many way to do it. We have to split the page in 4KB chunk to feed >> the ring. > > I was thinking about adding support for 64KB grants with the aim of > eventually removing the splitting done in xen-blkfront and the > grant-table code in general. I'd like to see a basic support of 64KB page granularity upstream before starting to think about performance improvement. And there is still a lot to do. Although, having 64KB grants won't remove the splitting as we would still have to support frontend/backend which only handle 4KB grant. >> TBH, I'm expecting a small impact to the performance. It would be hard >> to get the exactly the same performance as today if we keep the helpers >> to avoid the backend dealing himself with the splitting and page >> granularity. >> >> Although, if the performance impact is not acceptable, it may be >> possible to optimize gnttab_foreach_grant_in_range by moving the >> function inline. The current way to the loop is the fastest I've found >> (I've wrote a small program to test different way) and we will need it >> when different of size will be supported. > > I don't expect the performance to drop massively with this patches > applied, but it would be good to al least have an idea of the impact. I will only be able to give percentage. I guess this would be sufficient for you? Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hello, El 14/09/15 a les 12.40, Julien Grall ha escrit: > Hi Roger, > > On 14/09/15 09:56, Roger Pau Monné wrote: >> El 07/09/15 a les 17.33, Julien Grall ha escrit: >>> Hi all, >>> >>> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen >>> hypercall interface and PV protocol are always based on 4KB page >>> granularity. >>> >>> Any attempt to boot a Linux guest with 64KB pages enabled will result to a >>> guest crash. >>> >>> This series is a first attempt to allow those Linux running with the current >>> hypercall interface and PV protocol. >>> >>> This solution has been chosen because we want to run Linux 64KB in released >>> Xen ARM version or/and platform using an old version of Linux DOM0. >>> >>> There is room for improvement, such as support of 64KB grant, modification >>> of PV protocol to support different page size... They will be explored in a >>> separate patch series later. >>> >>> TODO list: >>> - Convert swiotlb to 64KB >>> - Convert xenfb to 64KB >>> - Support for multiple page ring support >>> - Support for 64KB in gnttdev >>> - Support of non-indirect grant with 64KB frontend >>> - It may be possible to move some common define between >>> netback/netfront and blkfront/blkback in an header >>> >>> I've got most of the patches for the TODO items. I'm planning to send them >>> as >>> a follow-up as it's not a requirement for a basic guests. >>> >>> All patches has been built tested for ARM32, ARM64, x86. But I haven't >>> tested >>> to run it on x86 as I don't have a box with Xen x86 running. I would be >>> happy if someone give a try and see possible regression for x86. >> >> Do you have figures regarding if/how much performance penalty do the >> blkfront/blkback patches add to the traditional 4KB scenario (frontend >> and backend running on guests using 4KB pages)? > > Which benchmark do you advice? Although, I don't have SSD on this > platform so it may be possible that the bottleneck will be the hard drive. I've normally used a ramdisk in order to test performance, but it seems nullblk would be a better option (it wasn't available before) and it doesn't reduce the size of RAM available to the system: https://www.kernel.org/doc/Documentation/block/null_blk.txt >> >> Since there's been no design document about this and the TODO list >> doesn't contain it, I would like to know which plans do we have in order >> to fix this properly. > > Can you explain what kind of design document you were expecting? The > support of 64KB page granularity is pretty straight-forward and there is > not many way to do it. We have to split the page in 4KB chunk to feed > the ring. I was thinking about adding support for 64KB grants with the aim of eventually removing the splitting done in xen-blkfront and the grant-table code in general. Some modifications would be needed in xen-blkback in order to support mapping 64KB grants, but this could also be used by x86 if we ever enable 2MB grants and allow them to be used in the block PV protocol, while the current code can only be used by ARM. > TBH, I'm expecting a small impact to the performance. It would be hard > to get the exactly the same performance as today if we keep the helpers > to avoid the backend dealing himself with the splitting and page > granularity. > > Although, if the performance impact is not acceptable, it may be > possible to optimize gnttab_foreach_grant_in_range by moving the > function inline. The current way to the loop is the fastest I've found > (I've wrote a small program to test different way) and we will need it > when different of size will be supported. I don't expect the performance to drop massively with this patches applied, but it would be good to al least have an idea of the impact. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi Roger, On 14/09/15 09:56, Roger Pau Monné wrote: > El 07/09/15 a les 17.33, Julien Grall ha escrit: >> Hi all, >> >> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen >> hypercall interface and PV protocol are always based on 4KB page granularity. >> >> Any attempt to boot a Linux guest with 64KB pages enabled will result to a >> guest crash. >> >> This series is a first attempt to allow those Linux running with the current >> hypercall interface and PV protocol. >> >> This solution has been chosen because we want to run Linux 64KB in released >> Xen ARM version or/and platform using an old version of Linux DOM0. >> >> There is room for improvement, such as support of 64KB grant, modification >> of PV protocol to support different page size... They will be explored in a >> separate patch series later. >> >> TODO list: >> - Convert swiotlb to 64KB >> - Convert xenfb to 64KB >> - Support for multiple page ring support >> - Support for 64KB in gnttdev >> - Support of non-indirect grant with 64KB frontend >> - It may be possible to move some common define between >> netback/netfront and blkfront/blkback in an header >> >> I've got most of the patches for the TODO items. I'm planning to send them as >> a follow-up as it's not a requirement for a basic guests. >> >> All patches has been built tested for ARM32, ARM64, x86. But I haven't tested >> to run it on x86 as I don't have a box with Xen x86 running. I would be >> happy if someone give a try and see possible regression for x86. > > Do you have figures regarding if/how much performance penalty do the > blkfront/blkback patches add to the traditional 4KB scenario (frontend > and backend running on guests using 4KB pages)? Which benchmark do you advice? Although, I don't have SSD on this platform so it may be possible that the bottleneck will be the hard drive. > > Since there's been no design document about this and the TODO list > doesn't contain it, I would like to know which plans do we have in order > to fix this properly. Can you explain what kind of design document you were expecting? The support of 64KB page granularity is pretty straight-forward and there is not many way to do it. We have to split the page in 4KB chunk to feed the ring. TBH, I'm expecting a small impact to the performance. It would be hard to get the exactly the same performance as today if we keep the helpers to avoid the backend dealing himself with the splitting and page granularity. Although, if the performance impact is not acceptable, it may be possible to optimize gnttab_foreach_grant_in_range by moving the function inline. The current way to the loop is the fastest I've found (I've wrote a small program to test different way) and we will need it when different of size will be supported. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote: > > TBH, I'm expecting a small impact to the performance. It would be hard > > to get the exactly the same performance as today if we keep the helpers > > to avoid the backend dealing himself with the splitting and page > > granularity. > > > > Although, if the performance impact is not acceptable, it may be > > possible to optimize gnttab_foreach_grant_in_range by moving the > > function inline. The current way to the loop is the fastest I've found > > (I've wrote a small program to test different way) and we will need it > > when different of size will be supported. > > I don't expect the performance to drop massively with this patches > applied, but it would be good to al least have an idea of the impact. Note that using 64kb pages in Linux tends to destroy performance in Linux in any case, as the memory consumption for most workloads explodes. In a virtualized environment you already tend to be memory constrained, so any measurement should take that into account and put the extra overhead into perspective to the massive overhead of running 64kb pages when RAM is tight. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 13:08, Roger Pau Monné wrote: >> I'd like to see a basic support of 64KB page granularity upstream before >> starting to think about performance improvement. And there is still a >> lot to do. > > I wasn't actually thinking of this as a performance improvement, but > rather as a way of fixing this properly and removing the complexity > added to xen-blkfront, thus providing a PV block protocol that natively > supports 64KB pages. This would be like hardware components already do > AFAIK, because I don't think there are other drivers in the Linux tree > that do this kind of splitting. > >> Although, having 64KB grants won't remove the splitting as we would >> still have to support frontend/backend which only handle 4KB grant. > > IMHO, given enough time I would just drop those. The PV protocol has always been backward compatible. Introducing the support of 64KB grant will require some changes in Xen which won't surely be backported to older version. We still have to be able to run new Linux guest on older Xen and vice versa. To give you an example, Centos 7, which will support Xen and only 64KB page granularity, will be supported for years. Dropping any splitting in a short future (3-5 years) will just break those guests to boot on Xen. AFAIK, we never took a such radical decision on Xen based on the complexity of the code. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 14/09/15 a les 13.21, Julien Grall ha escrit: > On 14/09/15 12:04, Roger Pau Monné wrote: >> Hello, >> >> El 14/09/15 a les 12.40, Julien Grall ha escrit: >>> Hi Roger, >>> >>> On 14/09/15 09:56, Roger Pau Monné wrote: El 07/09/15 a les 17.33, Julien Grall ha escrit: > Hi all, > > ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, > Xen > hypercall interface and PV protocol are always based on 4KB page > granularity. > > Any attempt to boot a Linux guest with 64KB pages enabled will result to a > guest crash. > > This series is a first attempt to allow those Linux running with the > current > hypercall interface and PV protocol. > > This solution has been chosen because we want to run Linux 64KB in > released > Xen ARM version or/and platform using an old version of Linux DOM0. > > There is room for improvement, such as support of 64KB grant, modification > of PV protocol to support different page size... They will be explored in > a > separate patch series later. > > TODO list: > - Convert swiotlb to 64KB > - Convert xenfb to 64KB > - Support for multiple page ring support > - Support for 64KB in gnttdev > - Support of non-indirect grant with 64KB frontend > - It may be possible to move some common define between > netback/netfront and blkfront/blkback in an header > > I've got most of the patches for the TODO items. I'm planning to send > them as > a follow-up as it's not a requirement for a basic guests. > > All patches has been built tested for ARM32, ARM64, x86. But I haven't > tested > to run it on x86 as I don't have a box with Xen x86 running. I would be > happy if someone give a try and see possible regression for x86. Do you have figures regarding if/how much performance penalty do the blkfront/blkback patches add to the traditional 4KB scenario (frontend and backend running on guests using 4KB pages)? >>> >>> Which benchmark do you advice? Although, I don't have SSD on this >>> platform so it may be possible that the bottleneck will be the hard drive. >> >> I've normally used a ramdisk in order to test performance, but it seems >> nullblk would be a better option (it wasn't available before) and it >> doesn't reduce the size of RAM available to the system: >> >> https://www.kernel.org/doc/Documentation/block/null_blk.txt > > I will give a look to this. > Since there's been no design document about this and the TODO list doesn't contain it, I would like to know which plans do we have in order to fix this properly. >>> >>> Can you explain what kind of design document you were expecting? The >>> support of 64KB page granularity is pretty straight-forward and there is >>> not many way to do it. We have to split the page in 4KB chunk to feed >>> the ring. >> >> I was thinking about adding support for 64KB grants with the aim of >> eventually removing the splitting done in xen-blkfront and the >> grant-table code in general. > > I'd like to see a basic support of 64KB page granularity upstream before > starting to think about performance improvement. And there is still a > lot to do. I wasn't actually thinking of this as a performance improvement, but rather as a way of fixing this properly and removing the complexity added to xen-blkfront, thus providing a PV block protocol that natively supports 64KB pages. This would be like hardware components already do AFAIK, because I don't think there are other drivers in the Linux tree that do this kind of splitting. > Although, having 64KB grants won't remove the splitting as we would > still have to support frontend/backend which only handle 4KB grant. IMHO, given enough time I would just drop those. >>> TBH, I'm expecting a small impact to the performance. It would be hard >>> to get the exactly the same performance as today if we keep the helpers >>> to avoid the backend dealing himself with the splitting and page >>> granularity. >>> >>> Although, if the performance impact is not acceptable, it may be >>> possible to optimize gnttab_foreach_grant_in_range by moving the >>> function inline. The current way to the loop is the fastest I've found >>> (I've wrote a small program to test different way) and we will need it >>> when different of size will be supported. >> >> I don't expect the performance to drop massively with this patches >> applied, but it would be good to al least have an idea of the impact. > > I will only be able to give percentage. I guess this would be sufficient > for you? Well, absolute numbers together with the standard deviation are IMHO the best way to provide those figures (ie: see ministat(1) output for example), but percentages should also be fine. I'm just interested in knowing the performance difference between having this patches applied or not when using
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hello, El 14/09/15 a les 14.47, Julien Grall ha escrit: > On 14/09/15 13:08, Roger Pau Monné wrote: >>> I'd like to see a basic support of 64KB page granularity upstream before >>> starting to think about performance improvement. And there is still a >>> lot to do. >> >> I wasn't actually thinking of this as a performance improvement, but >> rather as a way of fixing this properly and removing the complexity >> added to xen-blkfront, thus providing a PV block protocol that natively >> supports 64KB pages. This would be like hardware components already do >> AFAIK, because I don't think there are other drivers in the Linux tree >> that do this kind of splitting. >> >>> Although, having 64KB grants won't remove the splitting as we would >>> still have to support frontend/backend which only handle 4KB grant. >> >> IMHO, given enough time I would just drop those. > > The PV protocol has always been backward compatible. > > Introducing the support of 64KB grant will require some changes in Xen > which won't surely be backported to older version. We still have to be > able to run new Linux guest on older Xen and vice versa. OK, so let's wait a couple of Xen and Linux releases before dropping this from the Linux kernel once 64KB grant support is added. Guests should still be able to boot, it's just the disk that won't be attached. IMHO this splitting is just a workaround for the fact that we don't have a 64KB PV block protocol, and this is the real problem that should be solved. In the long term this will put a burden on all blkfronts (if 64KB pages are also used by other OSes), while introducing a 64KB PV block protocol will make the blkfront implementation in all OSes very similar to what we have now, without replicating the splitting code amongst all the possible blkfront implementations. Granted that some changes to blkback will be needed in order to support mapping 64KB grants, but there are much fewer blkback implementations out there than blkfronts. > To give you an example, Centos 7, which will support Xen and only 64KB > page granularity, will be supported for years. Dropping any splitting in > a short future (3-5 years) will just break those guests to boot on Xen. Can't the patches to support 64KB grants be backported to CentOS? In the past you said that distros had no problem in picking the needed Xen patches for 64KB support, I don't see how that would be different. Also, a blkfront with the splitting code should also be able to work perfectly fine once we have a 64KB PV block protocol, it will just be treated like it's using 4KB pages, that's all. > AFAIK, we never took a such radical decision on Xen based on the > complexity of the code. We never had to deal with a problem like this I'm afraid (mixing guests with different page sizes), so I don't think we can make statements about that. And it's not just about the complexity, it's because as stated above this IMHO is just sweeping the real problem under the carpet instead of solving it. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On 14/09/15 15:29, Roger Pau Monné wrote: >> To give you an example, Centos 7, which will support Xen and only 64KB >> page granularity, will be supported for years. Dropping any splitting in >> a short future (3-5 years) will just break those guests to boot on Xen. > > Can't the patches to support 64KB grants be backported to CentOS? In the > past you said that distros had no problem in picking the needed Xen > patches for 64KB support, I don't see how that would be different. Centos 7 is not yet released so it's more easy to get patch in. I don't know what will be position when the guest is shipped. I bet they will still want to support 64KB with older Xen (i.e where 64KB grant is not supported). Anyway, I don't expect to see the 64KB grants support before next year as my TODO list is already pretty full. So let's talk about it when I (or someone else) will send an RFC to extend the grant size. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
On Mon, 14 Sep 2015, Roger Pau Monné wrote: > IMHO this splitting is just a workaround for the fact that we don't have > a 64KB PV block protocol, and this is the real problem that should be > solved. 64K is a pure one guest kernel configuration option, not a platform wide option. The hypervisor interfaces are still the same, the ABI is the same and all the other guests are still the same, the Xen binary is still the same. A 64K block protocol could be a good performance imprevement, but should not be required to run kernels which have different config options. > In the long term this will put a burden on all blkfronts (if 64KB pages > are also used by other OSes), while introducing a 64KB PV block protocol > will make the blkfront implementation in all OSes very similar to what > we have now, without replicating the splitting code amongst all the > possible blkfront implementations. > > Granted that some changes to blkback will be needed in order to support > mapping 64KB grants, but there are much fewer blkback implementations > out there than blkfronts. I don't think we can rely on blkback having something in order to run new guests, otherwise we break compatibility: new guests won't run on old hypervisors.
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 14/09/15 a les 16.54, Stefano Stabellini ha escrit: > On Mon, 14 Sep 2015, Roger Pau Monné wrote: >> IMHO this splitting is just a workaround for the fact that we don't have >> a 64KB PV block protocol, and this is the real problem that should be >> solved. > > 64K is a pure one guest kernel configuration option, not a platform wide > option. The hypervisor interfaces are still the same, the ABI is the > same and all the other guests are still the same, the Xen binary is > still the same. Yes, I understand that, but the PV block protocol is missing 64KB page support, and that's a fact that cannot be ignored. To put an example, is there a hardware SATA controller on ARM that doesn't support 64KB pages and needs a similar workaround? > A 64K block protocol could be a good performance imprevement, but should > not be required to run kernels which have different config options. > >> In the long term this will put a burden on all blkfronts (if 64KB pages >> are also used by other OSes), while introducing a 64KB PV block protocol >> will make the blkfront implementation in all OSes very similar to what >> we have now, without replicating the splitting code amongst all the >> possible blkfront implementations. >> >> Granted that some changes to blkback will be needed in order to support >> mapping 64KB grants, but there are much fewer blkback implementations >> out there than blkfronts. > > I don't think we can rely on blkback having something in order to run > new guests, otherwise we break compatibility: new guests won't run on > old hypervisors. I agree that this is far from ideal, but I don't think it's so outrageous. For example Linux PVOPS Dom0 kernels require Xen 4.0.1 at least in order to run, because previous versions lack the necessary IOAPIC setup hypercall. Also, it won't prevent guests from booting, it would just prevent them from using blkfront, but you can still get a root filesystem using iSCSI, NFS or other means. I'm not saying that we shouldn't take those patches, I'm just saying that IMHO this is a workaround, and I would like to see a plan and somebody committed to have it fixed in a proper way, by introducing a 64KB PV block protocol. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
El 07/09/15 a les 17.33, Julien Grall ha escrit: > Hi all, > > ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen > hypercall interface and PV protocol are always based on 4KB page granularity. > > Any attempt to boot a Linux guest with 64KB pages enabled will result to a > guest crash. > > This series is a first attempt to allow those Linux running with the current > hypercall interface and PV protocol. > > This solution has been chosen because we want to run Linux 64KB in released > Xen ARM version or/and platform using an old version of Linux DOM0. > > There is room for improvement, such as support of 64KB grant, modification > of PV protocol to support different page size... They will be explored in a > separate patch series later. > > TODO list: > - Convert swiotlb to 64KB > - Convert xenfb to 64KB > - Support for multiple page ring support > - Support for 64KB in gnttdev > - Support of non-indirect grant with 64KB frontend > - It may be possible to move some common define between > netback/netfront and blkfront/blkback in an header > > I've got most of the patches for the TODO items. I'm planning to send them as > a follow-up as it's not a requirement for a basic guests. > > All patches has been built tested for ARM32, ARM64, x86. But I haven't tested > to run it on x86 as I don't have a box with Xen x86 running. I would be > happy if someone give a try and see possible regression for x86. Do you have figures regarding if/how much performance penalty do the blkfront/blkback patches add to the traditional 4KB scenario (frontend and backend running on guests using 4KB pages)? Since there's been no design document about this and the TODO list doesn't contain it, I would like to know which plans do we have in order to fix this properly. Roger. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi, A quick update on the TODO. On 07/09/15 16:33, Julien Grall wrote: > ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen > hypercall interface and PV protocol are always based on 4KB page granularity. > > Any attempt to boot a Linux guest with 64KB pages enabled will result to a > guest crash. > > This series is a first attempt to allow those Linux running with the current > hypercall interface and PV protocol. > > This solution has been chosen because we want to run Linux 64KB in released > Xen ARM version or/and platform using an old version of Linux DOM0. > > There is room for improvement, such as support of 64KB grant, modification > of PV protocol to support different page size... They will be explored in a > separate patch series later. > > TODO list: > - Convert swiotlb to 64KB Sent http://lists.xen.org/archives/html/xen-devel/2015-09/msg01292.html > - Convert xenfb to 64KB > - Support for multiple page ring support > - Support for 64KB in gnttdev > - Support of non-indirect grant with 64KB frontend Sent http://lists.xen.org/archives/html/xen-devel/2015-09/msg01577.html > - It may be possible to move some common define between > netback/netfront and blkfront/blkback in an header > > I've got most of the patches for the TODO items. I'm planning to send them as > a follow-up as it's not a requirement for a basic guests. > > All patches has been built tested for ARM32, ARM64, x86. But I haven't tested > to run it on x86 as I don't have a box with Xen x86 running. I would be > happy if someone give a try and see possible regression for x86. > > I know that Konrad as a test-suite for x86. Konrand, would it be possible to > give a run to for this series? > > A branch based on the latest xentip/for-linus-4.3 can be found here: > > git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 I will resend a new version when 4.3-rc1 is out in order to fix any possible conflict with linux/master. I already know that the netback patch (#18) is conflicting with 1d5d48523900a4b0f25d6b52f1a93c84bd671186 "xen-netback: require fewer guest Rx slots when not using GSO". If the 2 series above are completely acked/reviewed, I will fold them in this series. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi, A quick update on the TODO. On 07/09/15 16:33, Julien Grall wrote: > ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen > hypercall interface and PV protocol are always based on 4KB page granularity. > > Any attempt to boot a Linux guest with 64KB pages enabled will result to a > guest crash. > > This series is a first attempt to allow those Linux running with the current > hypercall interface and PV protocol. > > This solution has been chosen because we want to run Linux 64KB in released > Xen ARM version or/and platform using an old version of Linux DOM0. > > There is room for improvement, such as support of 64KB grant, modification > of PV protocol to support different page size... They will be explored in a > separate patch series later. > > TODO list: > - Convert swiotlb to 64KB Sent http://lists.xen.org/archives/html/xen-devel/2015-09/msg01292.html > - Convert xenfb to 64KB > - Support for multiple page ring support > - Support for 64KB in gnttdev > - Support of non-indirect grant with 64KB frontend Sent http://lists.xen.org/archives/html/xen-devel/2015-09/msg01577.html > - It may be possible to move some common define between > netback/netfront and blkfront/blkback in an header > > I've got most of the patches for the TODO items. I'm planning to send them as > a follow-up as it's not a requirement for a basic guests. > > All patches has been built tested for ARM32, ARM64, x86. But I haven't tested > to run it on x86 as I don't have a box with Xen x86 running. I would be > happy if someone give a try and see possible regression for x86. > > I know that Konrad as a test-suite for x86. Konrand, would it be possible to > give a run to for this series? > > A branch based on the latest xentip/for-linus-4.3 can be found here: > > git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 I will resend a new version when 4.3-rc1 is out in order to fix any possible conflict with linux/master. I already know that the netback patch (#18) is conflicting with 1d5d48523900a4b0f25d6b52f1a93c84bd671186 "xen-netback: require fewer guest Rx slots when not using GSO". If the 2 series above are completely acked/reviewed, I will fold them in this series. Regards, -- Julien Grall -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi all, ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen hypercall interface and PV protocol are always based on 4KB page granularity. Any attempt to boot a Linux guest with 64KB pages enabled will result to a guest crash. This series is a first attempt to allow those Linux running with the current hypercall interface and PV protocol. This solution has been chosen because we want to run Linux 64KB in released Xen ARM version or/and platform using an old version of Linux DOM0. There is room for improvement, such as support of 64KB grant, modification of PV protocol to support different page size... They will be explored in a separate patch series later. TODO list: - Convert swiotlb to 64KB - Convert xenfb to 64KB - Support for multiple page ring support - Support for 64KB in gnttdev - Support of non-indirect grant with 64KB frontend - It may be possible to move some common define between netback/netfront and blkfront/blkback in an header I've got most of the patches for the TODO items. I'm planning to send them as a follow-up as it's not a requirement for a basic guests. All patches has been built tested for ARM32, ARM64, x86. But I haven't tested to run it on x86 as I don't have a box with Xen x86 running. I would be happy if someone give a try and see possible regression for x86. I know that Konrad as a test-suite for x86. Konrand, would it be possible to give a run to for this series? A branch based on the latest xentip/for-linus-4.3 can be found here: git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 Comments, suggestions are welcomed. Sincerely yours, Cc: david.vra...@citrix.com Cc: konrad.w...@oracle.com Cc: boris.ostrov...@oracle.com Cc: wei.l...@citrix.com Cc: roger@citrix.com Status of each patch: A: Reviewed-by - Acked-by M: Patch modified in this series m: Minor changes in this series (i.e renaming due to previous patches, typoes) L: Missing Acked-by from a Linux maintainers (Boris, David, Konrad) N: Missing Acked-by from a Netback maintainers (Ian or Wei) Julien Grall (20): A net/xen-netback: xenvif_gop_frag_copy: move GSO check out of the loop A arm/xen: Drop pte_mfn and mfn_pte A M L xen: Add Xen specific page definition A M xen/grant: Introduce helpers to split a page into grant A xen/grant: Add helper gnttab_page_grant_foreign_access_ref_one A block/xen-blkfront: Split blkif_queue_request in 2 A m block/xen-blkfront: Store a page rather a pfn in the grant structure A block/xen-blkfront: split get_grant in 2 A m L xen/biomerge: Don't allow biovec's to be merged when Linux is not using 4KB pages A xen/xenbus: Use Xen page definition A m L tty/hvc: xen: Use xen page definition M L xen/balloon: Don't rely on the page granularity is the same for Xen and Linux A xen/events: fifo: Make it running on 64KB granularity A xen/grant-table: Make it running on 64KB granularity A m block/xen-blkfront: Make it running on 64KB page granularity A block/xen-blkback: Make it running on 64KB page granularity A m net/xen-netfront: Make it running on 64KB page granularity m N net/xen-netback: Make it running on 64KB page granularity A m xen/privcmd: Add support for Linux 64KB page granularity A arm/xen: Add support for 64KB page granularity arch/arm/include/asm/xen/page.h | 18 +- arch/arm/xen/enlighten.c| 6 +- arch/arm/xen/p2m.c | 6 +- arch/x86/include/asm/xen/page.h | 2 +- drivers/block/xen-blkback/blkback.c | 5 +- drivers/block/xen-blkback/common.h | 17 +- drivers/block/xen-blkback/xenbus.c | 9 +- drivers/block/xen-blkfront.c| 552 +++- drivers/net/xen-netback/common.h| 18 +- drivers/net/xen-netback/netback.c | 163 +++ drivers/net/xen-netfront.c | 122 +--- drivers/tty/hvc/hvc_xen.c | 4 +- drivers/xen/balloon.c | 59 +++- drivers/xen/biomerge.c | 8 + drivers/xen/events/events_base.c| 2 +- drivers/xen/events/events_fifo.c| 2 +- drivers/xen/grant-table.c | 32 ++- drivers/xen/privcmd.c | 8 +- drivers/xen/xenbus/xenbus_client.c | 6 +- drivers/xen/xenbus/xenbus_probe.c | 3 +- drivers/xen/xlate_mmu.c | 124 +--- include/xen/grant_table.h | 51 include/xen/page.h | 27 +- 23 files changed, 855 insertions(+), 389 deletions(-) -- 2.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux
Hi all, ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen hypercall interface and PV protocol are always based on 4KB page granularity. Any attempt to boot a Linux guest with 64KB pages enabled will result to a guest crash. This series is a first attempt to allow those Linux running with the current hypercall interface and PV protocol. This solution has been chosen because we want to run Linux 64KB in released Xen ARM version or/and platform using an old version of Linux DOM0. There is room for improvement, such as support of 64KB grant, modification of PV protocol to support different page size... They will be explored in a separate patch series later. TODO list: - Convert swiotlb to 64KB - Convert xenfb to 64KB - Support for multiple page ring support - Support for 64KB in gnttdev - Support of non-indirect grant with 64KB frontend - It may be possible to move some common define between netback/netfront and blkfront/blkback in an header I've got most of the patches for the TODO items. I'm planning to send them as a follow-up as it's not a requirement for a basic guests. All patches has been built tested for ARM32, ARM64, x86. But I haven't tested to run it on x86 as I don't have a box with Xen x86 running. I would be happy if someone give a try and see possible regression for x86. I know that Konrad as a test-suite for x86. Konrand, would it be possible to give a run to for this series? A branch based on the latest xentip/for-linus-4.3 can be found here: git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4 Comments, suggestions are welcomed. Sincerely yours, Cc: david.vra...@citrix.com Cc: konrad.w...@oracle.com Cc: boris.ostrov...@oracle.com Cc: wei.l...@citrix.com Cc: roger@citrix.com Status of each patch: A: Reviewed-by - Acked-by M: Patch modified in this series m: Minor changes in this series (i.e renaming due to previous patches, typoes) L: Missing Acked-by from a Linux maintainers (Boris, David, Konrad) N: Missing Acked-by from a Netback maintainers (Ian or Wei) Julien Grall (20): A net/xen-netback: xenvif_gop_frag_copy: move GSO check out of the loop A arm/xen: Drop pte_mfn and mfn_pte A M L xen: Add Xen specific page definition A M xen/grant: Introduce helpers to split a page into grant A xen/grant: Add helper gnttab_page_grant_foreign_access_ref_one A block/xen-blkfront: Split blkif_queue_request in 2 A m block/xen-blkfront: Store a page rather a pfn in the grant structure A block/xen-blkfront: split get_grant in 2 A m L xen/biomerge: Don't allow biovec's to be merged when Linux is not using 4KB pages A xen/xenbus: Use Xen page definition A m L tty/hvc: xen: Use xen page definition M L xen/balloon: Don't rely on the page granularity is the same for Xen and Linux A xen/events: fifo: Make it running on 64KB granularity A xen/grant-table: Make it running on 64KB granularity A m block/xen-blkfront: Make it running on 64KB page granularity A block/xen-blkback: Make it running on 64KB page granularity A m net/xen-netfront: Make it running on 64KB page granularity m N net/xen-netback: Make it running on 64KB page granularity A m xen/privcmd: Add support for Linux 64KB page granularity A arm/xen: Add support for 64KB page granularity arch/arm/include/asm/xen/page.h | 18 +- arch/arm/xen/enlighten.c| 6 +- arch/arm/xen/p2m.c | 6 +- arch/x86/include/asm/xen/page.h | 2 +- drivers/block/xen-blkback/blkback.c | 5 +- drivers/block/xen-blkback/common.h | 17 +- drivers/block/xen-blkback/xenbus.c | 9 +- drivers/block/xen-blkfront.c| 552 +++- drivers/net/xen-netback/common.h| 18 +- drivers/net/xen-netback/netback.c | 163 +++ drivers/net/xen-netfront.c | 122 +--- drivers/tty/hvc/hvc_xen.c | 4 +- drivers/xen/balloon.c | 59 +++- drivers/xen/biomerge.c | 8 + drivers/xen/events/events_base.c| 2 +- drivers/xen/events/events_fifo.c| 2 +- drivers/xen/grant-table.c | 32 ++- drivers/xen/privcmd.c | 8 +- drivers/xen/xenbus/xenbus_client.c | 6 +- drivers/xen/xenbus/xenbus_probe.c | 3 +- drivers/xen/xlate_mmu.c | 124 +--- include/xen/grant_table.h | 51 include/xen/page.h | 27 +- 23 files changed, 855 insertions(+), 389 deletions(-) -- 2.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
