Re: Possible use_mm() mis-uses

[Linus Torvalds]

On Wed, Aug 22, 2018 at 11:16 PM Zhenyu Wang  wrote:
>
> yeah, that's the clear way to fix this imo. We only depend on guest
> life cycle to access guest memory properly. Here's proposed fix, will
> verify and integrate it later.

Thanks, this looks sane to me,

Linus
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Paolo Bonzini]

On 23/08/2018 08:07, Zhenyu Wang wrote:
> On 2018.08.22 20:20:46 +0200, Paolo Bonzini wrote:
>> On 22/08/2018 18:44, Linus Torvalds wrote:
>>> An example of something that *isn't* right, is the i915 kvm interface,
>>> which does
>>>
>>> use_mm(kvm->mm);
>>>
>>> on an mm that was initialized in virt/kvm/kvm_main.c using
>>>
>>> mmgrab(current->mm);
>>> kvm->mm = current->mm;
>>>
>>> which is *not* right. Using "mmgrab()" does indeed guarantee the
>>> lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
>>> the lifetime of the page tables. You need to use "mmget()" and
>>> "mmput()", which get the reference to the actual process address
>>> space!
>>>
>>> Now, it is *possible* that the kvm use is correct too, because kvm
>>> does register a mmu_notifier chain, and in theory you can avoid the
>>> proper refcounting by just making sure the mmu "release" notifier
>>> kills any existing uses, but I don't really see kvm doing that. Kvm
>>> does register a release notifier, but that just flushes the shadow
>>> page tables, it doesn't kill any use_mm() use by some i915 use case.
>>
>> Yes, KVM is correct but the i915 bits are at least fishy.  It's probably
>> as simple as adding a mmget/mmput pair respectively in kvmgt_guest_init
>> and kvmgt_guest_exit, or maybe mmget_not_zero.
>>
> 
> yeah, that's the clear way to fix this imo. We only depend on guest
> life cycle to access guest memory properly. Here's proposed fix, will
> verify and integrate it later.
> 
> Thanks!
> 
> From 5e5a8d0409aa150884adf5a4d0b956fd0b9906b3 Mon Sep 17 00:00:00 2001
> From: Zhenyu Wang 
> Date: Thu, 23 Aug 2018 14:08:06 +0800
> Subject: [PATCH] drm/i915/gvt: Fix life cycle reference on KVM mm
> 
> Handle guest mm access life cycle properly with mmget()/mmput()
> through guest init()/exit(). As noted by Linus, use_mm() depends
> on valid live page table but KVM's mmgrab() doesn't guarantee that.
> As vGPU usage depends on guest VM life cycle, need to make sure to
> use mmget()/mmput() to guarantee VM address access.
> 
> Cc: Linus Torvalds 
> Cc: Paolo Bonzini 
> Cc: Zhi Wang 
> Signed-off-by: Zhenyu Wang 

Reviewed-by: Paolo Bonzini 

> ---
>  drivers/gpu/drm/i915/gvt/kvmgt.c | 12 +++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c 
> b/drivers/gpu/drm/i915/gvt/kvmgt.c
> index 71751be329e3..4a0988747d08 100644
> --- a/drivers/gpu/drm/i915/gvt/kvmgt.c
> +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
> @@ -32,6 +32,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  #include 
>  #include 
>  #include 
> @@ -1614,9 +1615,16 @@ static int kvmgt_guest_init(struct mdev_device *mdev)
>   if (__kvmgt_vgpu_exist(vgpu, kvm))
>   return -EEXIST;
>  
> + if (!mmget_not_zero(kvm->mm)) {
> + gvt_vgpu_err("Can't get KVM mm reference\n");
> + return -EINVAL;
> + }
> +
>   info = vzalloc(sizeof(struct kvmgt_guest_info));
> - if (!info)
> + if (!info) {
> + mmput(kvm->mm);
>   return -ENOMEM;
> + }
>  
>   vgpu->handle = (unsigned long)info;
>   info->vgpu = vgpu;
> @@ -1647,6 +1655,8 @@ static bool kvmgt_guest_exit(struct kvmgt_guest_info 
> *info)
>   debugfs_remove(info->debugfs_cache_entries);
>  
>   kvm_page_track_unregister_notifier(info->kvm, >track_node);
> + if (info->kvm->mm)
> + mmput(info->kvm->mm);
>   kvm_put_kvm(info->kvm);
>   kvmgt_protect_table_destroy(info);
>   gvt_cache_destroy(info->vgpu);
> 




signature.asc
Description: OpenPGP digital signature
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Zhenyu Wang]

On 2018.08.22 20:20:46 +0200, Paolo Bonzini wrote:
> On 22/08/2018 18:44, Linus Torvalds wrote:
> > An example of something that *isn't* right, is the i915 kvm interface,
> > which does
> > 
> > use_mm(kvm->mm);
> > 
> > on an mm that was initialized in virt/kvm/kvm_main.c using
> > 
> > mmgrab(current->mm);
> > kvm->mm = current->mm;
> > 
> > which is *not* right. Using "mmgrab()" does indeed guarantee the
> > lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
> > the lifetime of the page tables. You need to use "mmget()" and
> > "mmput()", which get the reference to the actual process address
> > space!
> > 
> > Now, it is *possible* that the kvm use is correct too, because kvm
> > does register a mmu_notifier chain, and in theory you can avoid the
> > proper refcounting by just making sure the mmu "release" notifier
> > kills any existing uses, but I don't really see kvm doing that. Kvm
> > does register a release notifier, but that just flushes the shadow
> > page tables, it doesn't kill any use_mm() use by some i915 use case.
> 
> Yes, KVM is correct but the i915 bits are at least fishy.  It's probably
> as simple as adding a mmget/mmput pair respectively in kvmgt_guest_init
> and kvmgt_guest_exit, or maybe mmget_not_zero.
> 

yeah, that's the clear way to fix this imo. We only depend on guest
life cycle to access guest memory properly. Here's proposed fix, will
verify and integrate it later.

Thanks!

From 5e5a8d0409aa150884adf5a4d0b956fd0b9906b3 Mon Sep 17 00:00:00 2001
From: Zhenyu Wang 
Date: Thu, 23 Aug 2018 14:08:06 +0800
Subject: [PATCH] drm/i915/gvt: Fix life cycle reference on KVM mm

Handle guest mm access life cycle properly with mmget()/mmput()
through guest init()/exit(). As noted by Linus, use_mm() depends
on valid live page table but KVM's mmgrab() doesn't guarantee that.
As vGPU usage depends on guest VM life cycle, need to make sure to
use mmget()/mmput() to guarantee VM address access.

Cc: Linus Torvalds 
Cc: Paolo Bonzini 
Cc: Zhi Wang 
Signed-off-by: Zhenyu Wang 
---
 drivers/gpu/drm/i915/gvt/kvmgt.c | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 71751be329e3..4a0988747d08 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -32,6 +32,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -1614,9 +1615,16 @@ static int kvmgt_guest_init(struct mdev_device *mdev)
if (__kvmgt_vgpu_exist(vgpu, kvm))
return -EEXIST;
 
+   if (!mmget_not_zero(kvm->mm)) {
+   gvt_vgpu_err("Can't get KVM mm reference\n");
+   return -EINVAL;
+   }
+
info = vzalloc(sizeof(struct kvmgt_guest_info));
-   if (!info)
+   if (!info) {
+   mmput(kvm->mm);
return -ENOMEM;
+   }
 
vgpu->handle = (unsigned long)info;
info->vgpu = vgpu;
@@ -1647,6 +1655,8 @@ static bool kvmgt_guest_exit(struct kvmgt_guest_info 
*info)
debugfs_remove(info->debugfs_cache_entries);
 
kvm_page_track_unregister_notifier(info->kvm, >track_node);
+   if (info->kvm->mm)
+   mmput(info->kvm->mm);
kvm_put_kvm(info->kvm);
kvmgt_protect_table_destroy(info);
gvt_cache_destroy(info->vgpu);
-- 
2.18.0


-- 
Open Source Technology Center, Intel ltd.

$gpg --keyserver wwwkeys.pgp.net --recv-keys 4D781827


signature.asc
Description: PGP signature
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Linus Torvalds]

On Wed, Aug 22, 2018 at 12:44 PM Felix Kuehling  wrote:
>
> You're right, but that's a bit fragile and convoluted. I'll fix KFD to
> handle this more robustly. See the attached (untested) patch.

Yes, this patch that makes the whole "has to use current mm" or uses
"get_task_mm()" looks good from a VM< worry standpoint.

Thanks.

> And
> obviously that opaque pointer didn't work as intended. It just gets
> promoted to an mm_struct * without a warning from the compiler. Maybe I
> should change that to a long to make abuse easier to spot.

Using a "void *" is actually just about the worst possible type for
something that should be a cookie, because it silently translates to
any pointer.

"long" is actually not much better, becuase it will silently convert
to any integer type.

A good fairly type-safe cookie type is actually this:

typedef volatile const struct no_such_struct *cookie_ptr_t;

and now something of type "cookie_ptr_t" is actually very  hard to
convert to other types by mistake.

Note that the "volatile const" is not just random noise - it's so that
it won't even convert without warnings to things that take a "const
void *" as an argument (like, say, the source of 'memcpy()').

So you almost _have_ to explicitly cast it to use it.

   Linus
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Linus Torvalds]

On Wed, Aug 22, 2018 at 12:37 PM Oded Gabbay  wrote:
>
> Having said that, I think we *are* protected by the mmu_notifier
> release because if the process suddenly dies, we will gracefully clean
> the process's data in our driver and on the H/W before returning to
> the mm core code. And before we return to the mm core code, we set the
> mm pointer to NULL. And the graceful cleaning should be serialized
> with the load_hqd uses.

So I'm a bit nervous about the mmu_notifier model (and the largely
equivalent exit_aio() model for the USB gardget AIO uses).

The reason I'm nervous about it is that the mmu_notifier() gets called
only after the mm_users count has already been decremented to zero
(and the exact same thing goes for exit_aio()).

Now that's fine if you actually get rid of all accesses in
mmu_notifier_release() or in exit_aio(), because the page tables still
exist at that point - they are in the process of being torn down, but
they haven't been torn down yet.

But for something like a kernel thread doing use_mm(), the thing that
worries me is a pattern something like this:

  kwork thread  exit thread
    

mmput() ->
  mm_users goes to zero

  use_mm(mmptr);
  ..

  mmu_notifier_release();
  exit_mm() ->
exit_aio()

and the pattern is basically the same regatdless of whether you use
mmu_notifier_release() or depend on some exit_aio() flushing your aio
work: the use_mm() can be called with a mm that has already had its
mm_users count decremented to zero, and that is now scheduled to be
free'd.

Does it "work"? Yes. Kind of. At least if the mmu notifier and/or
exit_aio() actually makes sure to wait for any kwork thread thing. But
it's a bit of a worrisome pattern.

   Linus
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Oded Gabbay]

On Wed, Aug 22, 2018 at 10:58 PM Linus Torvalds
 wrote:
>
> On Wed, Aug 22, 2018 at 12:37 PM Oded Gabbay  wrote:
> >
> > Having said that, I think we *are* protected by the mmu_notifier
> > release because if the process suddenly dies, we will gracefully clean
> > the process's data in our driver and on the H/W before returning to
> > the mm core code. And before we return to the mm core code, we set the
> > mm pointer to NULL. And the graceful cleaning should be serialized
> > with the load_hqd uses.
>
> So I'm a bit nervous about the mmu_notifier model (and the largely
> equivalent exit_aio() model for the USB gardget AIO uses).
>
> The reason I'm nervous about it is that the mmu_notifier() gets called
> only after the mm_users count has already been decremented to zero
> (and the exact same thing goes for exit_aio()).
>
> Now that's fine if you actually get rid of all accesses in
> mmu_notifier_release() or in exit_aio(), because the page tables still
> exist at that point - they are in the process of being torn down, but
> they haven't been torn down yet.
>
> But for something like a kernel thread doing use_mm(), the thing that
> worries me is a pattern something like this:
>
>   kwork thread  exit thread
>     
>
> mmput() ->
>   mm_users goes to zero
>
>   use_mm(mmptr);
>   ..
>
>   mmu_notifier_release();
>   exit_mm() ->
> exit_aio()
>
> and the pattern is basically the same regatdless of whether you use
> mmu_notifier_release() or depend on some exit_aio() flushing your aio
> work: the use_mm() can be called with a mm that has already had its
> mm_users count decremented to zero, and that is now scheduled to be
> free'd.
>
> Does it "work"? Yes. Kind of. At least if the mmu notifier and/or
> exit_aio() actually makes sure to wait for any kwork thread thing. But
> it's a bit of a worrisome pattern.
>
>Linus

Yes, agreed, and that's why we will be on the safe side and eliminate
this pattern from our code and make sure we won't add this pattern in
the future.

Oded
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Felix Kuehling]

On 2018-08-22 02:13 PM, Christian König wrote:
> Adding Felix because the KFD part of amdgpu is actually his
> responsibility.
>
> If I'm not completely mistaken the release callback of the
> mmu_notifier should take care of that for amdgpu.

You're right, but that's a bit fragile and convoluted. I'll fix KFD to
handle this more robustly. See the attached (untested) patch. And
obviously that opaque pointer didn't work as intended. It just gets
promoted to an mm_struct * without a warning from the compiler. Maybe I
should change that to a long to make abuse easier to spot.

Regards,
  Felix

>
> Regards,
> Christian.
>
> Am 22.08.2018 um 18:44 schrieb Linus Torvalds:
>> Guys and gals,
>>   this is a *very* random list of people on the recipients list, but we
>> had a subtle TLB shootdown issue in the VM, and that brought up some
>> issues when people then went through the code more carefully.
>>
>> I think we have a handle on the TLB shootdown bug itself. But when
>> people were discussing all the possible situations, one thing that
>> came up was "use_mm()" that takes a mm, and makes it temporarily the
>> mm for a kernel thread (until "unuse_mm()", duh).
>>
>> And it turns out that some of those uses are definitely wrong, some of
>> them are right, and some of them are suspect or at least so overly
>> complicated that it's hard for the VM people to know if they are ok.
>>
>> Basically, the rule for "use_mm()" is that the mm in question *has* to
>> have a valid page table associated with it over the whole use_mm() ->
>> unuse_mm() sequence. That may sound obvious, and I guess it actually
>> is so obvious that there isn't even a comment about it, but the actual
>> users are showing that it's sadly apparently not so obvious after all.
>>
>> There is one user that uses the "obviously correct" model: the vhost
>> driver does a "mmget()" and "mmput()" pair around its use of it,
>> thanks to vhost_dev_set_owner() doing a
>>
>>  dev->mm = get_task_mm(current);
>>
>> to look up the mm, and then the teardown case does a
>>
>>  if (dev->mm)
>>  mmput(dev->mm);
>>  dev->mm = NULL;
>>
>> This is the *right* sequence. A gold star to the vhost people.
>>
>> Sadly, the vhost people are the only ones who seem to get things
>> unquestionably right. And some of those gold star people are also
>> apparently involved in the cases that didn't get things right.
>>
>> An example of something that *isn't* right, is the i915 kvm interface,
>> which does
>>
>>  use_mm(kvm->mm);
>>
>> on an mm that was initialized in virt/kvm/kvm_main.c using
>>
>>  mmgrab(current->mm);
>>  kvm->mm = current->mm;
>>
>> which is *not* right. Using "mmgrab()" does indeed guarantee the
>> lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
>> the lifetime of the page tables. You need to use "mmget()" and
>> "mmput()", which get the reference to the actual process address
>> space!
>>
>> Now, it is *possible* that the kvm use is correct too, because kvm
>> does register a mmu_notifier chain, and in theory you can avoid the
>> proper refcounting by just making sure the mmu "release" notifier
>> kills any existing uses, but I don't really see kvm doing that. Kvm
>> does register a release notifier, but that just flushes the shadow
>> page tables, it doesn't kill any use_mm() use by some i915 use case.
>>
>> So while the vhost use looks right, the kvm/i915 use looks definitely
>> wrong.
>>
>> The other users of "use_mm()" and "unuse_mm()" are less
>> black-and-white right vs wrong..
>>
>> One of the complex ones is the amdgpu driver. It does a
>> "use_mm(mmptr)" deep deep in the guts of a macro that ends up being
>> used in fa few places, and it's very hard to tell if it's right.
>>
>> It looks almost certainly buggy (there is no mmget/mmput to get the
>> refcount), but there _is_ a "release" mmu_notifier function and that
>> one - unlike the kvm case - looks like it might actually be trying to
>> flush the existing pending users of that mm.
>>
>> But on the whole, I'm suspicious of the amdgpu use. It smells. Jann
>> Horn pointed out that even if it migth be ok due to the mmu_notifier,
>> the comments are garbage:
>>
>>>   Where "process" in the uniquely-named "struct queue" is a "struct
>>>   kfd_process"; that struct's definition has this comment in it:
>>>
>>>     /*
>>>  * Opaque pointer to mm_struct. We don't hold a reference to
>>>  * it so it should never be dereferenced from here. This is
>>>  * only used for looking up processes by their mm.
>>>  */
>>>     void *mm;
>>>
>>>   So I think either that comment is wrong, or their code is wrong?
>> so I'm chalking the amdgpu use up in the "broken" column.
>>
>> It's also actually quite hard to synchronze with some other kernel
>> worker thread correctly, so just on general principles, if you use
>> "use_mm()" it really really should be on something that you've
>> properly gotten a 

Re: Possible use_mm() mis-uses

[Oded Gabbay]

On Wed, Aug 22, 2018 at 7:44 PM Linus Torvalds
 wrote:
> One of the complex ones is the amdgpu driver. It does a
> "use_mm(mmptr)" deep deep in the guts of a macro that ends up being
> used in fa few places, and it's very hard to tell if it's right.
>
> It looks almost certainly buggy (there is no mmget/mmput to get the
> refcount), but there _is_ a "release" mmu_notifier function and that
> one - unlike the kvm case - looks like it might actually be trying to
> flush the existing pending users of that mm.
>
> But on the whole, I'm suspicious of the amdgpu use. It smells. Jann
> Horn pointed out that even if it migth be ok due to the mmu_notifier,
> the comments are garbage:
>
> >  Where "process" in the uniquely-named "struct queue" is a "struct
> >  kfd_process"; that struct's definition has this comment in it:
> >
> >/*
> > * Opaque pointer to mm_struct. We don't hold a reference to
> > * it so it should never be dereferenced from here. This is
> > * only used for looking up processes by their mm.
> > */
> >void *mm;
> >
> >  So I think either that comment is wrong, or their code is wrong?
>
> so I'm chalking the amdgpu use up in the "broken" column.
>
Hello Linus,

I looked at the amdkfd code and indeed the comment does not match the
actual code because the mm pointer is clearly dereferenced directly in
the macro you mentioned (read_user_wptr). That macro is used in the
code path of loading a descriptor to the H/W (load_hqd). That function
is called in several cases, where in some of them we are in the
context of the calling process, but in others we are in a kernel
thread context (hence the use_mm). That's why we check these two
situations inside that macro and only do use_mm if we are in kernel
thread.

We need to fix that behavior and obviously make sure that in future
code we don't cast this pointer to mm_struct* and derereference it
directly.
Actually, the original code had "mm_struct *mm" instead of "void *mm"
in the structure, and I think the reason we changed it to void* is to
"make sure" that we won't dereference it directly, but clearly that
failed :(

Having said that, I think we *are* protected by the mmu_notifier
release because if the process suddenly dies, we will gracefully clean
the process's data in our driver and on the H/W before returning to
the mm core code. And before we return to the mm core code, we set the
mm pointer to NULL. And the graceful cleaning should be serialized
with the load_hqd uses.

Felix, do you have anything to add here that I might have missed ?

Thanks,
Oded
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Possible use_mm() mis-uses

[Linus Torvalds]

Guys and gals,
 this is a *very* random list of people on the recipients list, but we
had a subtle TLB shootdown issue in the VM, and that brought up some
issues when people then went through the code more carefully.

I think we have a handle on the TLB shootdown bug itself. But when
people were discussing all the possible situations, one thing that
came up was "use_mm()" that takes a mm, and makes it temporarily the
mm for a kernel thread (until "unuse_mm()", duh).

And it turns out that some of those uses are definitely wrong, some of
them are right, and some of them are suspect or at least so overly
complicated that it's hard for the VM people to know if they are ok.

Basically, the rule for "use_mm()" is that the mm in question *has* to
have a valid page table associated with it over the whole use_mm() ->
unuse_mm() sequence. That may sound obvious, and I guess it actually
is so obvious that there isn't even a comment about it, but the actual
users are showing that it's sadly apparently not so obvious after all.

There is one user that uses the "obviously correct" model: the vhost
driver does a "mmget()" and "mmput()" pair around its use of it,
thanks to vhost_dev_set_owner() doing a

dev->mm = get_task_mm(current);

to look up the mm, and then the teardown case does a

if (dev->mm)
mmput(dev->mm);
dev->mm = NULL;

This is the *right* sequence. A gold star to the vhost people.

Sadly, the vhost people are the only ones who seem to get things
unquestionably right. And some of those gold star people are also
apparently involved in the cases that didn't get things right.

An example of something that *isn't* right, is the i915 kvm interface,
which does

use_mm(kvm->mm);

on an mm that was initialized in virt/kvm/kvm_main.c using

mmgrab(current->mm);
kvm->mm = current->mm;

which is *not* right. Using "mmgrab()" does indeed guarantee the
lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
the lifetime of the page tables. You need to use "mmget()" and
"mmput()", which get the reference to the actual process address
space!

Now, it is *possible* that the kvm use is correct too, because kvm
does register a mmu_notifier chain, and in theory you can avoid the
proper refcounting by just making sure the mmu "release" notifier
kills any existing uses, but I don't really see kvm doing that. Kvm
does register a release notifier, but that just flushes the shadow
page tables, it doesn't kill any use_mm() use by some i915 use case.

So while the vhost use looks right, the kvm/i915 use looks definitely wrong.

The other users of "use_mm()" and "unuse_mm()" are less
black-and-white right vs wrong..

One of the complex ones is the amdgpu driver. It does a
"use_mm(mmptr)" deep deep in the guts of a macro that ends up being
used in fa few places, and it's very hard to tell if it's right.

It looks almost certainly buggy (there is no mmget/mmput to get the
refcount), but there _is_ a "release" mmu_notifier function and that
one - unlike the kvm case - looks like it might actually be trying to
flush the existing pending users of that mm.

But on the whole, I'm suspicious of the amdgpu use. It smells. Jann
Horn pointed out that even if it migth be ok due to the mmu_notifier,
the comments are garbage:

>  Where "process" in the uniquely-named "struct queue" is a "struct
>  kfd_process"; that struct's definition has this comment in it:
>
>/*
> * Opaque pointer to mm_struct. We don't hold a reference to
> * it so it should never be dereferenced from here. This is
> * only used for looking up processes by their mm.
> */
>void *mm;
>
>  So I think either that comment is wrong, or their code is wrong?

so I'm chalking the amdgpu use up in the "broken" column.

It's also actually quite hard to synchronze with some other kernel
worker thread correctly, so just on general principles, if you use
"use_mm()" it really really should be on something that you've
properly gotten a mm refcount on with mmget(). Really. Even if you try
to synchronize things.

The two final cases are two uses in the USB gadget driver. Again, they
don't have the proper mmget/mmput, but they may br ok simply because
the uses are done for AIO, and the VM teardown is preceded by an AIO
teardown, so the proper serialization may come in from that.

Anyway, sorry for the long email, and the big list of participants and
odd mailing lists, but I'd really like people to look at their
"use_mm()" cases, and ask themselves if they have done enough to
guarantee that the full mm exists. Again, "mmgrab()" is *not* enough
on its own. You need either "mmget()" or some lifetime guarantee.

And if you do have those lifetime guarantees, it would be really nice
to get a good explanatory comment about said lifetime guarantees above
the "use_mm()" call. Ok?

Note that the lifetime rules are very important, because obviously
use_mm() itself is never called 

Re: Possible use_mm() mis-uses

[Linus Torvalds]

On Wed, Aug 22, 2018 at 11:33 AM Linus Torvalds
 wrote:
>
> On Wed, Aug 22, 2018 at 11:21 AM Paolo Bonzini  wrote:
> >
> > Yes, KVM is correct but the i915 bits are at least fishy.  It's probably
> > as simple as adding a mmget/mmput pair respectively in kvmgt_guest_init
> > and kvmgt_guest_exit, or maybe mmget_not_zero.
>
> Definitely mmget_not_zero(). If it was just mmgrab()'ed earlier, the
> actual page tables might already be gone.

Side note: we _could_ do the mmget_not_zero() inside use_mm() itself,
if we just knew that the mm was at least mmgrab()'ed correctly.

But for some of the uses, even that isn't clear. It's not entirely
obvious that the "struct mm_struct" exists _at_all_ at that point, and
that a mmget_not_zero() wouldn't just have some use-after-free access.

Again, independent lifetime rules could show that this isn't the case
(ie "exit_aio() is always called before exit_mmap(), and kill_ioctx()
takes care of it all"), but it would be good to have the users of
"use_mm()" actually verify their lifetime rules are correct and
enforced.

Because quite often, the lifetime rule might nbot be a mmu notifier or
aio_exit at all, but just be "oh, the user won't exit until this is
all done". But do you *control* the user? What if the user is buggy?

 Linus
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Linus Torvalds]

On Wed, Aug 22, 2018 at 11:21 AM Paolo Bonzini  wrote:
>
> Yes, KVM is correct but the i915 bits are at least fishy.  It's probably
> as simple as adding a mmget/mmput pair respectively in kvmgt_guest_init
> and kvmgt_guest_exit, or maybe mmget_not_zero.

Definitely mmget_not_zero(). If it was just mmgrab()'ed earlier, the
actual page tables might already be gone.

  Linus
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Zhi Wang]

Hi Linus:

Thanks for letting us know that. We would fix this ASAP. The kvmgt.c 
module is a part of GVT-g code. It's our fault that we didn't find this 
mis-uses, not i915 or KVM guys. Wish they would feel better after seeing 
this message.


Thanks,
Zhi.

On 08/23/18 00:44, Linus Torvalds wrote:

Guys and gals,
  this is a *very* random list of people on the recipients list, but we
had a subtle TLB shootdown issue in the VM, and that brought up some
issues when people then went through the code more carefully.

I think we have a handle on the TLB shootdown bug itself. But when
people were discussing all the possible situations, one thing that
came up was "use_mm()" that takes a mm, and makes it temporarily the
mm for a kernel thread (until "unuse_mm()", duh).

And it turns out that some of those uses are definitely wrong, some of
them are right, and some of them are suspect or at least so overly
complicated that it's hard for the VM people to know if they are ok.

Basically, the rule for "use_mm()" is that the mm in question *has* to
have a valid page table associated with it over the whole use_mm() ->
unuse_mm() sequence. That may sound obvious, and I guess it actually
is so obvious that there isn't even a comment about it, but the actual
users are showing that it's sadly apparently not so obvious after all.

There is one user that uses the "obviously correct" model: the vhost
driver does a "mmget()" and "mmput()" pair around its use of it,
thanks to vhost_dev_set_owner() doing a

 dev->mm = get_task_mm(current);

to look up the mm, and then the teardown case does a

 if (dev->mm)
 mmput(dev->mm);
 dev->mm = NULL;

This is the *right* sequence. A gold star to the vhost people.

Sadly, the vhost people are the only ones who seem to get things
unquestionably right. And some of those gold star people are also
apparently involved in the cases that didn't get things right.

An example of something that *isn't* right, is the i915 kvm interface,
which does

 use_mm(kvm->mm);

on an mm that was initialized in virt/kvm/kvm_main.c using

 mmgrab(current->mm);
 kvm->mm = current->mm;

which is *not* right. Using "mmgrab()" does indeed guarantee the
lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
the lifetime of the page tables. You need to use "mmget()" and
"mmput()", which get the reference to the actual process address
space!

Now, it is *possible* that the kvm use is correct too, because kvm
does register a mmu_notifier chain, and in theory you can avoid the
proper refcounting by just making sure the mmu "release" notifier
kills any existing uses, but I don't really see kvm doing that. Kvm
does register a release notifier, but that just flushes the shadow
page tables, it doesn't kill any use_mm() use by some i915 use case.

So while the vhost use looks right, the kvm/i915 use looks definitely wrong.

The other users of "use_mm()" and "unuse_mm()" are less
black-and-white right vs wrong..

One of the complex ones is the amdgpu driver. It does a
"use_mm(mmptr)" deep deep in the guts of a macro that ends up being
used in fa few places, and it's very hard to tell if it's right.

It looks almost certainly buggy (there is no mmget/mmput to get the
refcount), but there _is_ a "release" mmu_notifier function and that
one - unlike the kvm case - looks like it might actually be trying to
flush the existing pending users of that mm.

But on the whole, I'm suspicious of the amdgpu use. It smells. Jann
Horn pointed out that even if it migth be ok due to the mmu_notifier,
the comments are garbage:


  Where "process" in the uniquely-named "struct queue" is a "struct
  kfd_process"; that struct's definition has this comment in it:

/*
 * Opaque pointer to mm_struct. We don't hold a reference to
 * it so it should never be dereferenced from here. This is
 * only used for looking up processes by their mm.
 */
void *mm;

  So I think either that comment is wrong, or their code is wrong?


so I'm chalking the amdgpu use up in the "broken" column.

It's also actually quite hard to synchronze with some other kernel
worker thread correctly, so just on general principles, if you use
"use_mm()" it really really should be on something that you've
properly gotten a mm refcount on with mmget(). Really. Even if you try
to synchronize things.

The two final cases are two uses in the USB gadget driver. Again, they
don't have the proper mmget/mmput, but they may br ok simply because
the uses are done for AIO, and the VM teardown is preceded by an AIO
teardown, so the proper serialization may come in from that.

Anyway, sorry for the long email, and the big list of participants and
odd mailing lists, but I'd really like people to look at their
"use_mm()" cases, and ask themselves if they have done enough to
guarantee that the full mm exists. Again, "mmgrab()" is *not* enough
on its own. You need either 

Re: Possible use_mm() mis-uses

[Paolo Bonzini]

On 22/08/2018 18:44, Linus Torvalds wrote:
> An example of something that *isn't* right, is the i915 kvm interface,
> which does
> 
> use_mm(kvm->mm);
> 
> on an mm that was initialized in virt/kvm/kvm_main.c using
> 
> mmgrab(current->mm);
> kvm->mm = current->mm;
> 
> which is *not* right. Using "mmgrab()" does indeed guarantee the
> lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
> the lifetime of the page tables. You need to use "mmget()" and
> "mmput()", which get the reference to the actual process address
> space!
> 
> Now, it is *possible* that the kvm use is correct too, because kvm
> does register a mmu_notifier chain, and in theory you can avoid the
> proper refcounting by just making sure the mmu "release" notifier
> kills any existing uses, but I don't really see kvm doing that. Kvm
> does register a release notifier, but that just flushes the shadow
> page tables, it doesn't kill any use_mm() use by some i915 use case.

Yes, KVM is correct but the i915 bits are at least fishy.  It's probably
as simple as adding a mmget/mmput pair respectively in kvmgt_guest_init
and kvmgt_guest_exit, or maybe mmget_not_zero.

Paolo
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Possible use_mm() mis-uses

[Christian König]

Adding Felix because the KFD part of amdgpu is actually his responsibility.

If I'm not completely mistaken the release callback of the mmu_notifier 
should take care of that for amdgpu.


Regards,
Christian.

Am 22.08.2018 um 18:44 schrieb Linus Torvalds:

Guys and gals,
  this is a *very* random list of people on the recipients list, but we
had a subtle TLB shootdown issue in the VM, and that brought up some
issues when people then went through the code more carefully.

I think we have a handle on the TLB shootdown bug itself. But when
people were discussing all the possible situations, one thing that
came up was "use_mm()" that takes a mm, and makes it temporarily the
mm for a kernel thread (until "unuse_mm()", duh).

And it turns out that some of those uses are definitely wrong, some of
them are right, and some of them are suspect or at least so overly
complicated that it's hard for the VM people to know if they are ok.

Basically, the rule for "use_mm()" is that the mm in question *has* to
have a valid page table associated with it over the whole use_mm() ->
unuse_mm() sequence. That may sound obvious, and I guess it actually
is so obvious that there isn't even a comment about it, but the actual
users are showing that it's sadly apparently not so obvious after all.

There is one user that uses the "obviously correct" model: the vhost
driver does a "mmget()" and "mmput()" pair around its use of it,
thanks to vhost_dev_set_owner() doing a

 dev->mm = get_task_mm(current);

to look up the mm, and then the teardown case does a

 if (dev->mm)
 mmput(dev->mm);
 dev->mm = NULL;

This is the *right* sequence. A gold star to the vhost people.

Sadly, the vhost people are the only ones who seem to get things
unquestionably right. And some of those gold star people are also
apparently involved in the cases that didn't get things right.

An example of something that *isn't* right, is the i915 kvm interface,
which does

 use_mm(kvm->mm);

on an mm that was initialized in virt/kvm/kvm_main.c using

 mmgrab(current->mm);
 kvm->mm = current->mm;

which is *not* right. Using "mmgrab()" does indeed guarantee the
lifetime of the 'struct mm_struct' itself, but it does *not* guarantee
the lifetime of the page tables. You need to use "mmget()" and
"mmput()", which get the reference to the actual process address
space!

Now, it is *possible* that the kvm use is correct too, because kvm
does register a mmu_notifier chain, and in theory you can avoid the
proper refcounting by just making sure the mmu "release" notifier
kills any existing uses, but I don't really see kvm doing that. Kvm
does register a release notifier, but that just flushes the shadow
page tables, it doesn't kill any use_mm() use by some i915 use case.

So while the vhost use looks right, the kvm/i915 use looks definitely wrong.

The other users of "use_mm()" and "unuse_mm()" are less
black-and-white right vs wrong..

One of the complex ones is the amdgpu driver. It does a
"use_mm(mmptr)" deep deep in the guts of a macro that ends up being
used in fa few places, and it's very hard to tell if it's right.

It looks almost certainly buggy (there is no mmget/mmput to get the
refcount), but there _is_ a "release" mmu_notifier function and that
one - unlike the kvm case - looks like it might actually be trying to
flush the existing pending users of that mm.

But on the whole, I'm suspicious of the amdgpu use. It smells. Jann
Horn pointed out that even if it migth be ok due to the mmu_notifier,
the comments are garbage:


  Where "process" in the uniquely-named "struct queue" is a "struct
  kfd_process"; that struct's definition has this comment in it:

/*
 * Opaque pointer to mm_struct. We don't hold a reference to
 * it so it should never be dereferenced from here. This is
 * only used for looking up processes by their mm.
 */
void *mm;

  So I think either that comment is wrong, or their code is wrong?

so I'm chalking the amdgpu use up in the "broken" column.

It's also actually quite hard to synchronze with some other kernel
worker thread correctly, so just on general principles, if you use
"use_mm()" it really really should be on something that you've
properly gotten a mm refcount on with mmget(). Really. Even if you try
to synchronize things.

The two final cases are two uses in the USB gadget driver. Again, they
don't have the proper mmget/mmput, but they may br ok simply because
the uses are done for AIO, and the VM teardown is preceded by an AIO
teardown, so the proper serialization may come in from that.

Anyway, sorry for the long email, and the big list of participants and
odd mailing lists, but I'd really like people to look at their
"use_mm()" cases, and ask themselves if they have done enough to
guarantee that the full mm exists. Again, "mmgrab()" is *not* enough
on its own. You need either "mmget()" or some lifetime guarantee.

And if