Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: > > On 2019/3/8 上午3:16, Andrea Arcangeli wrote: > > On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: > > > On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: > > > > On Wed, Mar 06, 2019 at 02:18:12AM -

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Fri, Mar 08, 2019 at 07:56:04AM -0500, Michael S. Tsirkin wrote: > On Fri, Mar 08, 2019 at 04:58:44PM +0800, Jason Wang wrote: > > > > On 2019/3/8 上午3:17, Jerome Glisse wrote: > > > On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: > > > > On Thu, Mar 07, 2019 at 10:47:22AM -0

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Fri, Mar 08, 2019 at 02:48:45PM -0500, Andrea Arcangeli wrote: > Hello Jeson, > > On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: > > Just to make sure I understand here. For boosting through huge TLB, do > > you mean we can do that in the future (e.g by mapping more userspace > >

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Thu, Mar 07, 2019 at 10:43:12PM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 07, 2019 at 10:40:53PM -0500, Jerome Glisse wrote: > > On Thu, Mar 07, 2019 at 10:16:00PM -0500, Michael S. Tsirkin wrote: > > > On Thu, Mar 07, 2019 at 09:55:39PM -0500, Jerome Glisse wrote: > > > > On Thu, Mar 07, 2

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Thu, Mar 07, 2019 at 09:21:03PM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 07, 2019 at 02:17:20PM -0500, Jerome Glisse wrote: > > > It's because of all these issues that I preferred just accessing > > > userspace memory and handling faults. Unfortunately there does not > > > appear to exist

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Thu, Mar 07, 2019 at 10:16:00PM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 07, 2019 at 09:55:39PM -0500, Jerome Glisse wrote: > > On Thu, Mar 07, 2019 at 09:21:03PM -0500, Michael S. Tsirkin wrote: > > > On Thu, Mar 07, 2019 at 02:17:20PM -0500, Jerome Glisse wrote: > > > > > It's because of

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: > > On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: > > > +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { > > > + .invalidate_range

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Thu, Mar 07, 2019 at 02:38:38PM -0500, Andrea Arcangeli wrote: > On Thu, Mar 07, 2019 at 02:09:10PM -0500, Jerome Glisse wrote: > > I thought this patch was only for anonymous memory ie not file back ? > > Yes, the other common usages are on hugetlbfs/tmpfs that also don't > need to implement w

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-04-19 Thread Jerome Glisse
On Thu, Mar 07, 2019 at 10:34:39AM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 07, 2019 at 10:45:57AM +0800, Jason Wang wrote: > > > > On 2019/3/7 上午12:31, Michael S. Tsirkin wrote: > > > > +static void vhost_set_vmap_dirty(struct vhost_vmap *used) > > > > +{ > > > > + int i; > > > > + >

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-12 Thread Jason Wang
On 2019/3/12 上午11:50, Michael S. Tsirkin wrote: Using direct mapping (I guess kernel will always try hugepage for that?) should be better and we can even use it for the data transfer not only for the metadata. Thanks We can't really. The big issue is get user pages. Doing that on data path wil

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Michael S. Tsirkin
On Tue, Mar 12, 2019 at 10:56:20AM +0800, Jason Wang wrote: > > On 2019/3/11 下午9:43, Andrea Arcangeli wrote: > > On Mon, Mar 11, 2019 at 08:48:37AM -0400, Michael S. Tsirkin wrote: > > > Using copyXuser is better I guess. > > It certainly would be faster there, but I don't think it's needed if > >

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Michael S. Tsirkin
On Tue, Mar 12, 2019 at 10:52:15AM +0800, Jason Wang wrote: > > On 2019/3/11 下午8:48, Michael S. Tsirkin wrote: > > On Mon, Mar 11, 2019 at 03:40:31PM +0800, Jason Wang wrote: > > > On 2019/3/9 上午3:48, Andrea Arcangeli wrote: > > > > Hello Jeson, > > > > > > > > On Fri, Mar 08, 2019 at 04:50:36PM

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Jason Wang
On 2019/3/11 下午9:43, Andrea Arcangeli wrote: On Mon, Mar 11, 2019 at 08:48:37AM -0400, Michael S. Tsirkin wrote: Using copyXuser is better I guess. It certainly would be faster there, but I don't think it's needed if that would be the only use case left that justifies supporting two different

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Jason Wang
On 2019/3/11 下午8:48, Michael S. Tsirkin wrote: On Mon, Mar 11, 2019 at 03:40:31PM +0800, Jason Wang wrote: On 2019/3/9 上午3:48, Andrea Arcangeli wrote: Hello Jeson, On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: Just to make sure I understand here. For boosting through huge TLB,

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Jan Kara
On Thu 07-03-19 16:27:17, Andrea Arcangeli wrote: > > driver that GUP page for hours/days/weeks/months ... obviously the > > race window is big enough here. It affects many fs (ext4, xfs, ...) > > in different ways. I think ext4 is the most obvious because of the > > kernel log trace it leaves behi

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Andrea Arcangeli
On Mon, Mar 11, 2019 at 08:48:37AM -0400, Michael S. Tsirkin wrote: > Using copyXuser is better I guess. It certainly would be faster there, but I don't think it's needed if that would be the only use case left that justifies supporting two different models. On small 32bit systems with little RAM

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Michael S. Tsirkin
On Mon, Mar 11, 2019 at 03:40:31PM +0800, Jason Wang wrote: > > On 2019/3/9 上午3:48, Andrea Arcangeli wrote: > > Hello Jeson, > > > > On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: > > > Just to make sure I understand here. For boosting through huge TLB, do > > > you mean we can do th

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Jason Wang
On 2019/3/9 上午3:48, Andrea Arcangeli wrote: Hello Jeson, On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: Just to make sure I understand here. For boosting through huge TLB, do you mean we can do that in the future (e.g by mapping more userspace pages to kenrel) or it can be done by

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Jason Wang
On 2019/3/9 上午3:11, Andrea Arcangeli wrote: On Fri, Mar 08, 2019 at 05:13:26PM +0800, Jason Wang wrote: Actually not wrapping around,  the pages for used ring was marked as dirty after a round of virtqueue processing when we're sure vhost wrote something there. Thanks for the clarification. So

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-11 Thread Jason Wang
On 2019/3/8 下午10:58, Jerome Glisse wrote: On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: On 2019/3/8 上午3:16, Andrea Arcangeli wrote: On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: On Wed, Ma

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Andrea Arcangeli
Hello Jeson, On Fri, Mar 08, 2019 at 04:50:36PM +0800, Jason Wang wrote: > Just to make sure I understand here. For boosting through huge TLB, do > you mean we can do that in the future (e.g by mapping more userspace > pages to kenrel) or it can be done by this series (only about three 4K > pag

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Andrea Arcangeli
On Fri, Mar 08, 2019 at 04:58:44PM +0800, Jason Wang wrote: > Can I simply can set_page_dirty() before vunmap() in the mmu notifier > callback, or is there any reason that it must be called within vumap()? I also don't see any problem in doing it before vunmap. As far as the mmu notifier and set_

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Andrea Arcangeli
On Fri, Mar 08, 2019 at 05:13:26PM +0800, Jason Wang wrote: > Actually not wrapping around,  the pages for used ring was marked as > dirty after a round of virtqueue processing when we're sure vhost wrote > something there. Thanks for the clarification. So we need to convert it to set_page_dirty

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Michael S. Tsirkin
On Fri, Mar 08, 2019 at 04:58:44PM +0800, Jason Wang wrote: > > On 2019/3/8 上午3:17, Jerome Glisse wrote: > > On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: > > > On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: > > > > On Wed, Mar 06, 2019 at 02:18:12AM -050

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Jason Wang
On 2019/3/8 上午11:45, Jerome Glisse wrote: On Thu, Mar 07, 2019 at 10:43:12PM -0500, Michael S. Tsirkin wrote: On Thu, Mar 07, 2019 at 10:40:53PM -0500, Jerome Glisse wrote: On Thu, Mar 07, 2019 at 10:16:00PM -0500, Michael S. Tsirkin wrote: On Thu, Mar 07, 2019 at 09:55:39PM -0500, Jerome Gli

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Jason Wang
On 2019/3/8 上午5:27, Andrea Arcangeli wrote: Hello Jerome, On Thu, Mar 07, 2019 at 03:17:22PM -0500, Jerome Glisse wrote: So for the above the easiest thing is to call set_page_dirty() from the mmu notifier callback. It is always safe to use the non locking variant from such callback. Well it i

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Jason Wang
On 2019/3/8 上午3:17, Jerome Glisse wrote: On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: +static const struct mmu_notifier_ops vhost_mmu_notifier

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Jason Wang
On 2019/3/8 上午3:16, Andrea Arcangeli wrote: On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: +static const struct mmu_notifier_ops vhost_mmu_notif

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-08 Thread Jason Wang
On 2019/3/7 下午11:34, Michael S. Tsirkin wrote: On Thu, Mar 07, 2019 at 10:45:57AM +0800, Jason Wang wrote: On 2019/3/7 上午12:31, Michael S. Tsirkin wrote: +static void vhost_set_vmap_dirty(struct vhost_vmap *used) +{ + int i; + + for (i = 0; i < used->npages; i++) + se

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Michael S. Tsirkin
On Thu, Mar 07, 2019 at 10:40:53PM -0500, Jerome Glisse wrote: > On Thu, Mar 07, 2019 at 10:16:00PM -0500, Michael S. Tsirkin wrote: > > On Thu, Mar 07, 2019 at 09:55:39PM -0500, Jerome Glisse wrote: > > > On Thu, Mar 07, 2019 at 09:21:03PM -0500, Michael S. Tsirkin wrote: > > > > On Thu, Mar 07, 2

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Michael S. Tsirkin
On Thu, Mar 07, 2019 at 09:55:39PM -0500, Jerome Glisse wrote: > On Thu, Mar 07, 2019 at 09:21:03PM -0500, Michael S. Tsirkin wrote: > > On Thu, Mar 07, 2019 at 02:17:20PM -0500, Jerome Glisse wrote: > > > > It's because of all these issues that I preferred just accessing > > > > userspace memory a

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Michael S. Tsirkin
On Thu, Mar 07, 2019 at 02:17:20PM -0500, Jerome Glisse wrote: > > It's because of all these issues that I preferred just accessing > > userspace memory and handling faults. Unfortunately there does not > > appear to exist an API that whitelists a specific driver along the lines > > of "I checked t

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Andrea Arcangeli
Hello Jerome, On Thu, Mar 07, 2019 at 03:17:22PM -0500, Jerome Glisse wrote: > So for the above the easiest thing is to call set_page_dirty() from > the mmu notifier callback. It is always safe to use the non locking > variant from such callback. Well it is safe only if the page was > map with wri

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Andrea Arcangeli
On Thu, Mar 07, 2019 at 02:09:10PM -0500, Jerome Glisse wrote: > I thought this patch was only for anonymous memory ie not file back ? Yes, the other common usages are on hugetlbfs/tmpfs that also don't need to implement writeback and are obviously safe too. > If so then set dirty is mostly usele

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Andrea Arcangeli
On Thu, Mar 07, 2019 at 12:56:45PM -0500, Michael S. Tsirkin wrote: > On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: > > On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: > > > +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { > > > + .invalidate_range

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Michael S. Tsirkin
On Thu, Mar 07, 2019 at 10:47:22AM -0500, Michael S. Tsirkin wrote: > On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: > > +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { > > + .invalidate_range = vhost_invalidate_range, > > +}; > > + > > void vhost_dev_init(struct vh

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Michael S. Tsirkin
On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: > +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { > + .invalidate_range = vhost_invalidate_range, > +}; > + > void vhost_dev_init(struct vhost_dev *dev, > struct vhost_virtqueue **vqs, int nvqs, int

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-07 Thread Michael S. Tsirkin
On Thu, Mar 07, 2019 at 10:45:57AM +0800, Jason Wang wrote: > > On 2019/3/7 上午12:31, Michael S. Tsirkin wrote: > > > +static void vhost_set_vmap_dirty(struct vhost_vmap *used) > > > +{ > > > + int i; > > > + > > > + for (i = 0; i < used->npages; i++) > > > + set_page_dirty_lock(used->pages

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-06 Thread Jason Wang
On 2019/3/7 上午12:31, Michael S. Tsirkin wrote: +static void vhost_set_vmap_dirty(struct vhost_vmap *used) +{ + int i; + + for (i = 0; i < used->npages; i++) + set_page_dirty_lock(used->pages[i]); This seems to rely on page lock to mark page dirty. Could it happen that

Re: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-06 Thread Michael S. Tsirkin
On Wed, Mar 06, 2019 at 02:18:12AM -0500, Jason Wang wrote: > It was noticed that the copy_user() friends that was used to access > virtqueue metdata tends to be very expensive for dataplane > implementation like vhost since it involves lots of software checks, > speculation barrier, hardware featu

[RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address

2019-03-05 Thread Jason Wang
It was noticed that the copy_user() friends that was used to access virtqueue metdata tends to be very expensive for dataplane implementation like vhost since it involves lots of software checks, speculation barrier, hardware feature toggling (e.g SMAP). The extra cost will be more obvious when tra