Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12/02/2024 16:24, David Hildenbrand wrote: > On 12.02.24 16:34, Ryan Roberts wrote: >> On 12/02/2024 15:26, David Hildenbrand wrote: >>> On 12.02.24 15:45, Ryan Roberts wrote: On 12/02/2024 13:54, David Hildenbrand wrote: >>> If so, I wonder if we could instead do that comparison modulo the >>> access/dirty >>> bits, >> >> I think that would work - but will need to think a bit more on it. >> >>> and leave ptep_get_lockless() only reading a single entry? >> >> I think we will need to do something a bit less fragile. ptep_get() does >> collect >> the access/dirty bits so its confusing if ptep_get_lockless() doesn't >> IMHO. So >> we will likely want to rename the function and make its documentation >> explicit >> that it does not return those bits. >> >> ptep_get_lockless_noyoungdirty()? yuk... Any ideas? >> >> Of course if I could convince you the current implementation is safe, I >> might be >> able to sidestep this optimization until a later date? > > As discussed (and pointed out abive), there might be quite some callsites > where > we don't really care about uptodate accessed/dirty bits -- where > ptep_get() is > used nowadays. > > One way to approach that I had in mind was having an explicit interface: > > ptep_get() > ptep_get_uptodate() > ptep_get_lockless() > ptep_get_lockless_uptodate() Yes, I like the direction of this. I guess we anticipate that call sites requiring the "_uptodate" variant will be the minority so it makes sense to use the current names for the "_not_uptodate" variants? But to do a slow migration, it might be better/safer to have the weaker variant use the new name - that would allow us to downgrade one at a time? >>> >>> Yes, I was primarily struggling with names. Likely it makes sense to either >>> have >>> two completely new function names, or use the new name only for the "faster >>> but >>> less precise" variant. >>> > > Especially the last one might not be needed. I've done a scan through the code and agree with Mark's original conclusions. Additionally, huge_pte_alloc() (which isn't used for arm64) doesn't rely on access/dirty info. So I think I could migrate everything to the weaker variant fairly easily. > > Futher, "uptodate" might not be the best choice because of PageUptodate() > and > friends. But it's better than "youngdirty"/"noyoungdirty" IMHO. Certainly agree with "noyoungdirty" being a horrible name. How about "_sync" / "_nosync"? >>> >>> I could live with >>> >>> ptep_get_sync() >>> ptep_get_nosync() >>> >>> with proper documentation :) >> >> but could you live with: >> >> ptep_get() >> ptep_get_nosync() >> ptep_get_lockless_nosync() >> >> ? >> >> So leave the "slower, more precise" version with the existing name. > > Sure. > I'm just implementing this (as a separate RFC), and had an alternative idea for naming/semantics: ptep_get() ptep_get_norecency() ptep_get_lockless() ptep_get_lockless_norecency() The "_norecency" versions explicitly clear the access/dirty bits. This is useful for the "compare to original pte to check we are not racing" pattern: pte = ptep_get_lockless_norecency(ptep) ... if (!pte_same(pte, ptep_get_norecency(ptep))) // RACE! ... With the "_nosync" semantic, the access/dirty bits may or may not be set, so the user has to explicitly clear them to do the comparison. (although I considered a pte_same_nosync() that would clear the bits for you - but that name is pretty naff). Although the _norecency semantic requires always explicitly clearing the bits, so may be infinitesimally slower, it gives a very clear expectation that the access/dirty bits are always clear and I think that's conveyed well in the name too. Thoughts?
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13/02/2024 14:08, Ard Biesheuvel wrote:
> On Tue, 13 Feb 2024 at 15:05, David Hildenbrand wrote:
>>
>> On 13.02.24 15:02, Ryan Roberts wrote:
>>> On 13/02/2024 13:45, David Hildenbrand wrote:
On 13.02.24 14:33, Ard Biesheuvel wrote:
> On Tue, 13 Feb 2024 at 14:21, Ryan Roberts wrote:
>>
>> On 13/02/2024 13:13, David Hildenbrand wrote:
> ...
>>> Just a thought, you could have a is_efi_mm() function that abstracts
>>> all that.
>>>
>>> diff --git a/include/linux/efi.h b/include/linux/efi.h
>>> index c74f47711f0b..152f5fa66a2a 100644
>>> --- a/include/linux/efi.h
>>> +++ b/include/linux/efi.h
>>> @@ -692,6 +692,15 @@ extern struct efi {
>>>
>>>extern struct mm_struct efi_mm;
>>>
>>> +static inline void is_efi_mm(struct mm_struct *mm)
>>> +{
>>> +#ifdef CONFIG_EFI
>>> + return mm == _mm;
>>> +#else
>>> + return false;
>>> +#endif
>>> +}
>>> +
>>>static inline int
>>>efi_guidcmp (efi_guid_t left, efi_guid_t right)
>>>{
>>>
>>>
>>
>> That would definitely work, but in that case, I might as well just check
>> for it
>> in mm_is_user() (and personally I would change the name to mm_is_efi()):
>>
>>
>> static inline bool mm_is_user(struct mm_struct *mm)
>> {
>> return mm != _mm && !mm_is_efi(mm);
>> }
>>
>> Any objections?
>>
>
> Any reason not to use IS_ENABLED(CONFIG_EFI) in the above? The extern
> declaration is visible to the compiler, and any references should
> disappear before the linker could notice that efi_mm does not exist.
>
Sure, as long as the linker is happy why not. I'll let Ryan mess with that
:)
>>>
>>> I'm not sure if you are suggesting dropping the mm_is_efi() helper and just
>>> use
>>> IS_ENABLED(CONFIG_EFI) in mm_is_user() to guard efi_mm, or if you are
>>> suggesting
>>> using IS_ENABLED(CONFIG_EFI) in mm_is_efi() instead of the ifdefery?
>>>
>>> The former was what I did initially; It works great, but I didn't like that
>>> I
>>> was introducing a new code dependecy between efi and arm64 (nothing else
>>> outside
>>> of efi references efi_mm).
>>>
>>> So then concluded that it is safe to not worry about efi_mm (thanks for your
>>> confirmation). But then David wanted a VM_WARN check, which reintroduces the
>>> code dependency. So he suggested the mm_is_efi() helper to hide that...
>>> This is
>>> all starting to feel circular...
>>
>> I think Ard meant that inside mm_is_efi(), we could avoid the #ifdef and
>> simply use IS_ENABLED().
>>
>
> Yes.
>
> static inline void mm_is_efi(struct mm_struct *mm)
> {
> return IS_ENABLED(CONFIG_EFI) && mm == _mm;
> }
Ahh, got it. Yes, I'll do it like this. Thanks!
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On Tue, 13 Feb 2024 at 15:05, David Hildenbrand wrote:
>
> On 13.02.24 15:02, Ryan Roberts wrote:
> > On 13/02/2024 13:45, David Hildenbrand wrote:
> >> On 13.02.24 14:33, Ard Biesheuvel wrote:
> >>> On Tue, 13 Feb 2024 at 14:21, Ryan Roberts wrote:
>
> On 13/02/2024 13:13, David Hildenbrand wrote:
...
> > Just a thought, you could have a is_efi_mm() function that abstracts
> > all that.
> >
> > diff --git a/include/linux/efi.h b/include/linux/efi.h
> > index c74f47711f0b..152f5fa66a2a 100644
> > --- a/include/linux/efi.h
> > +++ b/include/linux/efi.h
> > @@ -692,6 +692,15 @@ extern struct efi {
> >
> >extern struct mm_struct efi_mm;
> >
> > +static inline void is_efi_mm(struct mm_struct *mm)
> > +{
> > +#ifdef CONFIG_EFI
> > + return mm == _mm;
> > +#else
> > + return false;
> > +#endif
> > +}
> > +
> >static inline int
> >efi_guidcmp (efi_guid_t left, efi_guid_t right)
> >{
> >
> >
>
> That would definitely work, but in that case, I might as well just check
> for it
> in mm_is_user() (and personally I would change the name to mm_is_efi()):
>
>
> static inline bool mm_is_user(struct mm_struct *mm)
> {
> return mm != _mm && !mm_is_efi(mm);
> }
>
> Any objections?
>
> >>>
> >>> Any reason not to use IS_ENABLED(CONFIG_EFI) in the above? The extern
> >>> declaration is visible to the compiler, and any references should
> >>> disappear before the linker could notice that efi_mm does not exist.
> >>>
> >>
> >> Sure, as long as the linker is happy why not. I'll let Ryan mess with that
> >> :)
> >
> > I'm not sure if you are suggesting dropping the mm_is_efi() helper and just
> > use
> > IS_ENABLED(CONFIG_EFI) in mm_is_user() to guard efi_mm, or if you are
> > suggesting
> > using IS_ENABLED(CONFIG_EFI) in mm_is_efi() instead of the ifdefery?
> >
> > The former was what I did initially; It works great, but I didn't like that
> > I
> > was introducing a new code dependecy between efi and arm64 (nothing else
> > outside
> > of efi references efi_mm).
> >
> > So then concluded that it is safe to not worry about efi_mm (thanks for your
> > confirmation). But then David wanted a VM_WARN check, which reintroduces the
> > code dependency. So he suggested the mm_is_efi() helper to hide that...
> > This is
> > all starting to feel circular...
>
> I think Ard meant that inside mm_is_efi(), we could avoid the #ifdef and
> simply use IS_ENABLED().
>
Yes.
static inline void mm_is_efi(struct mm_struct *mm)
{
return IS_ENABLED(CONFIG_EFI) && mm == _mm;
}
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13.02.24 15:02, Ryan Roberts wrote:
On 13/02/2024 13:45, David Hildenbrand wrote:
On 13.02.24 14:33, Ard Biesheuvel wrote:
On Tue, 13 Feb 2024 at 14:21, Ryan Roberts wrote:
On 13/02/2024 13:13, David Hildenbrand wrote:
On 13.02.24 14:06, Ryan Roberts wrote:
On 13/02/2024 12:19, David Hildenbrand wrote:
On 13.02.24 13:06, Ryan Roberts wrote:
On 12/02/2024 20:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+ * Don't attempt to apply the contig bit to kernel mappings,
because
+ * dynamically adding/removing the contig bit can cause page
faults.
+ * These racing faults are ok for user space, since they get
serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+ return mm != _mm;
+}
We also have the efi_mm as a non-user mm, though I don't think we
manipulate
that while it is live, and I'm not sure if that needs any special
handling.
Well we never need this function in the hot (order-0 folio) path, so I
think I
could add a check for efi_mm here with performance implication. It's
probably
safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled I
can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like
userspace.
There are a couple of things that need to be garanteed for it to be safe:
- The PFNs of present ptes either need to have an associated struct
page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause
fold/unfold
while live) or the system must be able to tolerate a temporary fault
Mark suggests efi_mm is not manipulated while live, so that meets the
latter
requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests, the
mappings are indeed static. And additionally, the ptes are populated
using only
the _private_ ptep API, so there is no issue here. As just discussed with
Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
"official" APIs.
We could, but that would lead to the same linkage issue, which I'm trying to
avoid in the first place:
VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
This creates new source code dependencies, which I would rather avoid if
possible.
Just a thought, you could have a is_efi_mm() function that abstracts all that.
diff --git a/include/linux/efi.h b/include/linux/efi.h
index c74f47711f0b..152f5fa66a2a 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -692,6 +692,15 @@ extern struct efi {
extern struct mm_struct efi_mm;
+static inline void is_efi_mm(struct mm_struct *mm)
+{
+#ifdef CONFIG_EFI
+ return mm == _mm;
+#else
+ return false;
+#endif
+}
+
static inline int
efi_guidcmp (efi_guid_t left, efi_guid_t right)
{
That would definitely work, but in that case, I might as well just check for it
in mm_is_user() (and personally I would change the name to mm_is_efi()):
static inline bool mm_is_user(struct mm_struct *mm)
{
return mm != _mm && !mm_is_efi(mm);
}
Any objections?
Any reason not to use IS_ENABLED(CONFIG_EFI) in the above? The extern
declaration is visible to the compiler, and any references should
disappear before the linker could notice that efi_mm does not exist.
Sure, as long as the linker is happy why not. I'll let Ryan mess with that :)
I'm not sure if you are suggesting dropping the mm_is_efi() helper and just use
IS_ENABLED(CONFIG_EFI) in mm_is_user() to guard efi_mm, or if you are suggesting
using IS_ENABLED(CONFIG_EFI) in mm_is_efi() instead of the ifdefery?
The former was what I did initially; It works great, but I didn't like that I
was introducing a new code dependecy between efi and arm64 (nothing else outside
of efi references efi_mm).
So then concluded that it is safe to not worry about efi_mm (thanks for your
confirmation). But then David wanted a VM_WARN check, which reintroduces the
code dependency. So he suggested the mm_is_efi() helper to hide that... This is
all starting to feel circular...
I think Ard meant that inside mm_is_efi(), we could avoid the #ifdef and
simply use
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13/02/2024 13:45, David Hildenbrand wrote:
> On 13.02.24 14:33, Ard Biesheuvel wrote:
>> On Tue, 13 Feb 2024 at 14:21, Ryan Roberts wrote:
>>>
>>> On 13/02/2024 13:13, David Hildenbrand wrote:
On 13.02.24 14:06, Ryan Roberts wrote:
> On 13/02/2024 12:19, David Hildenbrand wrote:
>> On 13.02.24 13:06, Ryan Roberts wrote:
>>> On 12/02/2024 20:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+ * Don't attempt to apply the contig bit to kernel mappings,
because
+ * dynamically adding/removing the contig bit can cause page
faults.
+ * These racing faults are ok for user space, since they get
serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+ return mm != _mm;
+}
>>>
>>> We also have the efi_mm as a non-user mm, though I don't think we
>>> manipulate
>>> that while it is live, and I'm not sure if that needs any special
>>> handling.
>>
>> Well we never need this function in the hot (order-0 folio) path, so
>> I
>> think I
>> could add a check for efi_mm here with performance implication. It's
>> probably
>> safest to explicitly exclude it? What do you think?
>
> Oops: This should have read "I think I could add a check for efi_mm
> here
> *without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled I
can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of
efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like
userspace.
There are a couple of things that need to be garanteed for it to be
safe:
- The PFNs of present ptes either need to have an associated
struct
page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause
fold/unfold
while live) or the system must be able to tolerate a temporary
fault
Mark suggests efi_mm is not manipulated while live, so that meets the
latter
requirement, but I'm not sure about the former?
>>>
>>> I've gone through all the efi code, and conclude that, as Mark
>>> suggests, the
>>> mappings are indeed static. And additionally, the ptes are populated
>>> using only
>>> the _private_ ptep API, so there is no issue here. As just discussed
>>> with
>>> Mark,
>>> my prefereence is to not make any changes to code, and just add a
>>> comment
>>> describing why efi_mm is safe.
>>>
>>> Details:
>>>
>>> * Registered with ptdump
>>> * ptep_get_lockless()
>>> * efi_create_mapping -> create_pgd_mapping … -> init_pte:
>>> * __ptep_get()
>>> * __set_pte()
>>> * efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
>>> set_permissions
>>> * __ptep_get()
>>> * __set_pte()
>>
>> Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via
>> the
>> "official" APIs.
>
> We could, but that would lead to the same linkage issue, which I'm trying
> to
> avoid in the first place:
>
> VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
>
> This creates new source code dependencies, which I would rather avoid if
> possible.
Just a thought, you could have a is_efi_mm() function that abstracts all
that.
diff --git a/include/linux/efi.h b/include/linux/efi.h
index c74f47711f0b..152f5fa66a2a 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -692,6 +692,15 @@ extern struct efi {
extern struct mm_struct efi_mm;
+static inline void is_efi_mm(struct mm_struct *mm)
+{
+#ifdef CONFIG_EFI
+ return mm == _mm;
+#else
+ return false;
+#endif
+}
+
static inline int
efi_guidcmp (efi_guid_t left, efi_guid_t right)
{
>>>
>>> That would definitely work, but in that case, I might as well just check
>>> for it
>>> in mm_is_user() (and personally I would change the name to mm_is_efi()):
>>>
>>>
>>> static inline bool mm_is_user(struct mm_struct *mm)
>>> {
>>> return mm != _mm && !mm_is_efi(mm);
>>> }
>>>
>>> Any objections?
>>>
>>
>> Any reason not to use
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13.02.24 14:33, Ard Biesheuvel wrote:
On Tue, 13 Feb 2024 at 14:21, Ryan Roberts wrote:
On 13/02/2024 13:13, David Hildenbrand wrote:
On 13.02.24 14:06, Ryan Roberts wrote:
On 13/02/2024 12:19, David Hildenbrand wrote:
On 13.02.24 13:06, Ryan Roberts wrote:
On 12/02/2024 20:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+/*
+ * Don't attempt to apply the contig bit to kernel mappings, because
+ * dynamically adding/removing the contig bit can cause page faults.
+ * These racing faults are ok for user space, since they get
serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+return mm != _mm;
+}
We also have the efi_mm as a non-user mm, though I don't think we
manipulate
that while it is live, and I'm not sure if that needs any special handling.
Well we never need this function in the hot (order-0 folio) path, so I
think I
could add a check for efi_mm here with performance implication. It's
probably
safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled I can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like userspace.
There are a couple of things that need to be garanteed for it to be safe:
- The PFNs of present ptes either need to have an associated struct
page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause
fold/unfold
while live) or the system must be able to tolerate a temporary fault
Mark suggests efi_mm is not manipulated while live, so that meets the latter
requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests, the
mappings are indeed static. And additionally, the ptes are populated using only
the _private_ ptep API, so there is no issue here. As just discussed with Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
"official" APIs.
We could, but that would lead to the same linkage issue, which I'm trying to
avoid in the first place:
VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
This creates new source code dependencies, which I would rather avoid if
possible.
Just a thought, you could have a is_efi_mm() function that abstracts all that.
diff --git a/include/linux/efi.h b/include/linux/efi.h
index c74f47711f0b..152f5fa66a2a 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -692,6 +692,15 @@ extern struct efi {
extern struct mm_struct efi_mm;
+static inline void is_efi_mm(struct mm_struct *mm)
+{
+#ifdef CONFIG_EFI
+ return mm == _mm;
+#else
+ return false;
+#endif
+}
+
static inline int
efi_guidcmp (efi_guid_t left, efi_guid_t right)
{
That would definitely work, but in that case, I might as well just check for it
in mm_is_user() (and personally I would change the name to mm_is_efi()):
static inline bool mm_is_user(struct mm_struct *mm)
{
return mm != _mm && !mm_is_efi(mm);
}
Any objections?
Any reason not to use IS_ENABLED(CONFIG_EFI) in the above? The extern
declaration is visible to the compiler, and any references should
disappear before the linker could notice that efi_mm does not exist.
Sure, as long as the linker is happy why not. I'll let Ryan mess with
that :)
In any case, feel free to add
Acked-by: Ard Biesheuvel
Thanks for the review.
--
Cheers,
David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On Tue, 13 Feb 2024 at 14:21, Ryan Roberts wrote:
>
> On 13/02/2024 13:13, David Hildenbrand wrote:
> > On 13.02.24 14:06, Ryan Roberts wrote:
> >> On 13/02/2024 12:19, David Hildenbrand wrote:
> >>> On 13.02.24 13:06, Ryan Roberts wrote:
> On 12/02/2024 20:38, Ryan Roberts wrote:
> > [...]
> >
> > +static inline bool mm_is_user(struct mm_struct *mm)
> > +{
> > +/*
> > + * Don't attempt to apply the contig bit to kernel mappings,
> > because
> > + * dynamically adding/removing the contig bit can cause page
> > faults.
> > + * These racing faults are ok for user space, since they get
> > serialized
> > + * on the PTL. But kernel mappings can't tolerate faults.
> > + */
> > +return mm != _mm;
> > +}
>
> We also have the efi_mm as a non-user mm, though I don't think we
> manipulate
> that while it is live, and I'm not sure if that needs any special
> handling.
> >>>
> >>> Well we never need this function in the hot (order-0 folio) path, so I
> >>> think I
> >>> could add a check for efi_mm here with performance implication. It's
> >>> probably
> >>> safest to explicitly exclude it? What do you think?
> >>
> >> Oops: This should have read "I think I could add a check for efi_mm
> >> here
> >> *without* performance implication"
> >
> > It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I
> > can do
> > this:
> >
> > return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
> >
> > Is that acceptable? This is my preference, but nothing else outside of
> > efi
> > references this symbol currently.
> >
> > Or perhaps I can convince myself that its safe to treat efi_mm like
> > userspace.
> > There are a couple of things that need to be garanteed for it to be
> > safe:
> >
> > - The PFNs of present ptes either need to have an associated struct
> > page or
> > need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
> > pte_mkdevmap())
> >
> > - Live mappings must either be static (no changes that could cause
> > fold/unfold
> > while live) or the system must be able to tolerate a temporary
> > fault
> >
> > Mark suggests efi_mm is not manipulated while live, so that meets the
> > latter
> > requirement, but I'm not sure about the former?
>
> I've gone through all the efi code, and conclude that, as Mark suggests,
> the
> mappings are indeed static. And additionally, the ptes are populated
> using only
> the _private_ ptep API, so there is no issue here. As just discussed
> with Mark,
> my prefereence is to not make any changes to code, and just add a comment
> describing why efi_mm is safe.
>
> Details:
>
> * Registered with ptdump
> * ptep_get_lockless()
> * efi_create_mapping -> create_pgd_mapping … -> init_pte:
> * __ptep_get()
> * __set_pte()
> * efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
> set_permissions
> * __ptep_get()
> * __set_pte()
> >>>
> >>> Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
> >>> "official" APIs.
> >>
> >> We could, but that would lead to the same linkage issue, which I'm trying
> >> to
> >> avoid in the first place:
> >>
> >> VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
> >>
> >> This creates new source code dependencies, which I would rather avoid if
> >> possible.
> >
> > Just a thought, you could have a is_efi_mm() function that abstracts all
> > that.
> >
> > diff --git a/include/linux/efi.h b/include/linux/efi.h
> > index c74f47711f0b..152f5fa66a2a 100644
> > --- a/include/linux/efi.h
> > +++ b/include/linux/efi.h
> > @@ -692,6 +692,15 @@ extern struct efi {
> >
> > extern struct mm_struct efi_mm;
> >
> > +static inline void is_efi_mm(struct mm_struct *mm)
> > +{
> > +#ifdef CONFIG_EFI
> > + return mm == _mm;
> > +#else
> > + return false;
> > +#endif
> > +}
> > +
> > static inline int
> > efi_guidcmp (efi_guid_t left, efi_guid_t right)
> > {
> >
> >
>
> That would definitely work, but in that case, I might as well just check for
> it
> in mm_is_user() (and personally I would change the name to mm_is_efi()):
>
>
> static inline bool mm_is_user(struct mm_struct *mm)
> {
> return mm != _mm && !mm_is_efi(mm);
> }
>
> Any objections?
>
Any reason not to use IS_ENABLED(CONFIG_EFI) in the above? The extern
declaration is visible to the compiler, and any references should
disappear before the linker could notice that efi_mm does not exist.
In any case, feel free to add
Acked-by: Ard Biesheuvel
when you roll a patch based on the above, with or without
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13/02/2024 13:22, David Hildenbrand wrote:
> On 13.02.24 14:20, Ryan Roberts wrote:
>> On 13/02/2024 13:13, David Hildenbrand wrote:
>>> On 13.02.24 14:06, Ryan Roberts wrote:
On 13/02/2024 12:19, David Hildenbrand wrote:
> On 13.02.24 13:06, Ryan Roberts wrote:
>> On 12/02/2024 20:38, Ryan Roberts wrote:
>>> [...]
>>>
>>> +static inline bool mm_is_user(struct mm_struct *mm)
>>> +{
>>> + /*
>>> + * Don't attempt to apply the contig bit to kernel mappings,
>>> because
>>> + * dynamically adding/removing the contig bit can cause page
>>> faults.
>>> + * These racing faults are ok for user space, since they get
>>> serialized
>>> + * on the PTL. But kernel mappings can't tolerate faults.
>>> + */
>>> + return mm != _mm;
>>> +}
>>
>> We also have the efi_mm as a non-user mm, though I don't think we
>> manipulate
>> that while it is live, and I'm not sure if that needs any special
>> handling.
>
> Well we never need this function in the hot (order-0 folio) path, so I
> think I
> could add a check for efi_mm here with performance implication. It's
> probably
> safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm
here
*without* performance implication"
>>>
>>> It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I
>>> can do
>>> this:
>>>
>>> return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
>>>
>>> Is that acceptable? This is my preference, but nothing else outside of
>>> efi
>>> references this symbol currently.
>>>
>>> Or perhaps I can convince myself that its safe to treat efi_mm like
>>> userspace.
>>> There are a couple of things that need to be garanteed for it to be
>>> safe:
>>>
>>> - The PFNs of present ptes either need to have an associated struct
>>> page or
>>> need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
>>> pte_mkdevmap())
>>>
>>> - Live mappings must either be static (no changes that could cause
>>> fold/unfold
>>> while live) or the system must be able to tolerate a temporary
>>> fault
>>>
>>> Mark suggests efi_mm is not manipulated while live, so that meets the
>>> latter
>>> requirement, but I'm not sure about the former?
>>
>> I've gone through all the efi code, and conclude that, as Mark suggests,
>> the
>> mappings are indeed static. And additionally, the ptes are populated
>> using
>> only
>> the _private_ ptep API, so there is no issue here. As just discussed with
>> Mark,
>> my prefereence is to not make any changes to code, and just add a comment
>> describing why efi_mm is safe.
>>
>> Details:
>>
>> * Registered with ptdump
>> * ptep_get_lockless()
>> * efi_create_mapping -> create_pgd_mapping … -> init_pte:
>> * __ptep_get()
>> * __set_pte()
>> * efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
>> set_permissions
>> * __ptep_get()
>> * __set_pte()
>
> Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
> "official" APIs.
We could, but that would lead to the same linkage issue, which I'm trying
to
avoid in the first place:
VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
This creates new source code dependencies, which I would rather avoid if
possible.
>>>
>>> Just a thought, you could have a is_efi_mm() function that abstracts all
>>> that.
>>>
>>> diff --git a/include/linux/efi.h b/include/linux/efi.h
>>> index c74f47711f0b..152f5fa66a2a 100644
>>> --- a/include/linux/efi.h
>>> +++ b/include/linux/efi.h
>>> @@ -692,6 +692,15 @@ extern struct efi {
>>> extern struct mm_struct efi_mm;
>>> +static inline void is_efi_mm(struct mm_struct *mm)
>>> +{
>>> +#ifdef CONFIG_EFI
>>> + return mm == _mm;
>>> +#else
>>> + return false;
>>> +#endif
>>> +}
>>> +
>>> static inline int
>>> efi_guidcmp (efi_guid_t left, efi_guid_t right)
>>> {
>>>
>>>
>>
>> That would definitely work, but in that case, I might as well just check for
>> it
>> in mm_is_user() (and personally I would change the name to mm_is_efi()):
>>
>>
>> static inline bool mm_is_user(struct mm_struct *mm)
>> {
>> return mm != _mm && !mm_is_efi(mm);
>> }
>>
>> Any objections?
>>
>
> Nope :) Maybe slap in an "unlikely()", because efi_mm *is* unlikely to show
> up.
Deal
>
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13.02.24 14:20, Ryan Roberts wrote:
On 13/02/2024 13:13, David Hildenbrand wrote:
On 13.02.24 14:06, Ryan Roberts wrote:
On 13/02/2024 12:19, David Hildenbrand wrote:
On 13.02.24 13:06, Ryan Roberts wrote:
On 12/02/2024 20:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+ * Don't attempt to apply the contig bit to kernel mappings, because
+ * dynamically adding/removing the contig bit can cause page faults.
+ * These racing faults are ok for user space, since they get
serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+ return mm != _mm;
+}
We also have the efi_mm as a non-user mm, though I don't think we
manipulate
that while it is live, and I'm not sure if that needs any special handling.
Well we never need this function in the hot (order-0 folio) path, so I
think I
could add a check for efi_mm here with performance implication. It's
probably
safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like userspace.
There are a couple of things that need to be garanteed for it to be safe:
- The PFNs of present ptes either need to have an associated struct
page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause
fold/unfold
while live) or the system must be able to tolerate a temporary fault
Mark suggests efi_mm is not manipulated while live, so that meets the latter
requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests, the
mappings are indeed static. And additionally, the ptes are populated using only
the _private_ ptep API, so there is no issue here. As just discussed with Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
"official" APIs.
We could, but that would lead to the same linkage issue, which I'm trying to
avoid in the first place:
VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
This creates new source code dependencies, which I would rather avoid if
possible.
Just a thought, you could have a is_efi_mm() function that abstracts all that.
diff --git a/include/linux/efi.h b/include/linux/efi.h
index c74f47711f0b..152f5fa66a2a 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -692,6 +692,15 @@ extern struct efi {
extern struct mm_struct efi_mm;
+static inline void is_efi_mm(struct mm_struct *mm)
+{
+#ifdef CONFIG_EFI
+ return mm == _mm;
+#else
+ return false;
+#endif
+}
+
static inline int
efi_guidcmp (efi_guid_t left, efi_guid_t right)
{
That would definitely work, but in that case, I might as well just check for it
in mm_is_user() (and personally I would change the name to mm_is_efi()):
static inline bool mm_is_user(struct mm_struct *mm)
{
return mm != _mm && !mm_is_efi(mm);
}
Any objections?
Nope :) Maybe slap in an "unlikely()", because efi_mm *is* unlikely to
show up.
--
Cheers,
David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13/02/2024 13:13, David Hildenbrand wrote:
> On 13.02.24 14:06, Ryan Roberts wrote:
>> On 13/02/2024 12:19, David Hildenbrand wrote:
>>> On 13.02.24 13:06, Ryan Roberts wrote:
On 12/02/2024 20:38, Ryan Roberts wrote:
> [...]
>
> +static inline bool mm_is_user(struct mm_struct *mm)
> +{
> + /*
> + * Don't attempt to apply the contig bit to kernel mappings,
> because
> + * dynamically adding/removing the contig bit can cause page
> faults.
> + * These racing faults are ok for user space, since they get
> serialized
> + * on the PTL. But kernel mappings can't tolerate faults.
> + */
> + return mm != _mm;
> +}
We also have the efi_mm as a non-user mm, though I don't think we
manipulate
that while it is live, and I'm not sure if that needs any special
handling.
>>>
>>> Well we never need this function in the hot (order-0 folio) path, so I
>>> think I
>>> could add a check for efi_mm here with performance implication. It's
>>> probably
>>> safest to explicitly exclude it? What do you think?
>>
>> Oops: This should have read "I think I could add a check for efi_mm here
>> *without* performance implication"
>
> It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I
> can do
> this:
>
> return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
>
> Is that acceptable? This is my preference, but nothing else outside of efi
> references this symbol currently.
>
> Or perhaps I can convince myself that its safe to treat efi_mm like
> userspace.
> There are a couple of things that need to be garanteed for it to be safe:
>
> - The PFNs of present ptes either need to have an associated struct
> page or
> need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
> pte_mkdevmap())
>
> - Live mappings must either be static (no changes that could cause
> fold/unfold
> while live) or the system must be able to tolerate a temporary fault
>
> Mark suggests efi_mm is not manipulated while live, so that meets the
> latter
> requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests,
the
mappings are indeed static. And additionally, the ptes are populated using
only
the _private_ ptep API, so there is no issue here. As just discussed with
Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
>>>
>>> Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
>>> "official" APIs.
>>
>> We could, but that would lead to the same linkage issue, which I'm trying to
>> avoid in the first place:
>>
>> VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
>>
>> This creates new source code dependencies, which I would rather avoid if
>> possible.
>
> Just a thought, you could have a is_efi_mm() function that abstracts all that.
>
> diff --git a/include/linux/efi.h b/include/linux/efi.h
> index c74f47711f0b..152f5fa66a2a 100644
> --- a/include/linux/efi.h
> +++ b/include/linux/efi.h
> @@ -692,6 +692,15 @@ extern struct efi {
>
> extern struct mm_struct efi_mm;
>
> +static inline void is_efi_mm(struct mm_struct *mm)
> +{
> +#ifdef CONFIG_EFI
> + return mm == _mm;
> +#else
> + return false;
> +#endif
> +}
> +
> static inline int
> efi_guidcmp (efi_guid_t left, efi_guid_t right)
> {
>
>
That would definitely work, but in that case, I might as well just check for it
in mm_is_user() (and personally I would change the name to mm_is_efi()):
static inline bool mm_is_user(struct mm_struct *mm)
{
return mm != _mm && !mm_is_efi(mm);
}
Any objections?
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13.02.24 14:06, Ryan Roberts wrote:
On 13/02/2024 12:19, David Hildenbrand wrote:
On 13.02.24 13:06, Ryan Roberts wrote:
On 12/02/2024 20:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+ * Don't attempt to apply the contig bit to kernel mappings, because
+ * dynamically adding/removing the contig bit can cause page faults.
+ * These racing faults are ok for user space, since they get serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+ return mm != _mm;
+}
We also have the efi_mm as a non-user mm, though I don't think we manipulate
that while it is live, and I'm not sure if that needs any special handling.
Well we never need this function in the hot (order-0 folio) path, so I think I
could add a check for efi_mm here with performance implication. It's probably
safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like userspace.
There are a couple of things that need to be garanteed for it to be safe:
- The PFNs of present ptes either need to have an associated struct page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause
fold/unfold
while live) or the system must be able to tolerate a temporary fault
Mark suggests efi_mm is not manipulated while live, so that meets the latter
requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests, the
mappings are indeed static. And additionally, the ptes are populated using only
the _private_ ptep API, so there is no issue here. As just discussed with Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
"official" APIs.
We could, but that would lead to the same linkage issue, which I'm trying to
avoid in the first place:
VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
This creates new source code dependencies, which I would rather avoid if
possible.
Just a thought, you could have a is_efi_mm() function that abstracts all that.
diff --git a/include/linux/efi.h b/include/linux/efi.h
index c74f47711f0b..152f5fa66a2a 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -692,6 +692,15 @@ extern struct efi {
extern struct mm_struct efi_mm;
+static inline void is_efi_mm(struct mm_struct *mm)
+{
+#ifdef CONFIG_EFI
+ return mm == _mm;
+#else
+ return false;
+#endif
+}
+
static inline int
efi_guidcmp (efi_guid_t left, efi_guid_t right)
{
--
Cheers,
David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13/02/2024 12:19, David Hildenbrand wrote:
> On 13.02.24 13:06, Ryan Roberts wrote:
>> On 12/02/2024 20:38, Ryan Roberts wrote:
>>> [...]
>>>
>>> +static inline bool mm_is_user(struct mm_struct *mm)
>>> +{
>>> + /*
>>> + * Don't attempt to apply the contig bit to kernel mappings,
>>> because
>>> + * dynamically adding/removing the contig bit can cause page
>>> faults.
>>> + * These racing faults are ok for user space, since they get
>>> serialized
>>> + * on the PTL. But kernel mappings can't tolerate faults.
>>> + */
>>> + return mm != _mm;
>>> +}
>>
>> We also have the efi_mm as a non-user mm, though I don't think we
>> manipulate
>> that while it is live, and I'm not sure if that needs any special
>> handling.
>
> Well we never need this function in the hot (order-0 folio) path, so I
> think I
> could add a check for efi_mm here with performance implication. It's
> probably
> safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
>>>
>>> It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can
>>> do
>>> this:
>>>
>>> return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
>>>
>>> Is that acceptable? This is my preference, but nothing else outside of efi
>>> references this symbol currently.
>>>
>>> Or perhaps I can convince myself that its safe to treat efi_mm like
>>> userspace.
>>> There are a couple of things that need to be garanteed for it to be safe:
>>>
>>> - The PFNs of present ptes either need to have an associated struct page
>>> or
>>> need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
>>> pte_mkdevmap())
>>>
>>> - Live mappings must either be static (no changes that could cause
>>> fold/unfold
>>> while live) or the system must be able to tolerate a temporary fault
>>>
>>> Mark suggests efi_mm is not manipulated while live, so that meets the latter
>>> requirement, but I'm not sure about the former?
>>
>> I've gone through all the efi code, and conclude that, as Mark suggests, the
>> mappings are indeed static. And additionally, the ptes are populated using
>> only
>> the _private_ ptep API, so there is no issue here. As just discussed with
>> Mark,
>> my prefereence is to not make any changes to code, and just add a comment
>> describing why efi_mm is safe.
>>
>> Details:
>>
>> * Registered with ptdump
>> * ptep_get_lockless()
>> * efi_create_mapping -> create_pgd_mapping … -> init_pte:
>> * __ptep_get()
>> * __set_pte()
>> * efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
>> set_permissions
>> * __ptep_get()
>> * __set_pte()
>
> Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via the
> "official" APIs.
We could, but that would lead to the same linkage issue, which I'm trying to
avoid in the first place:
VM_WARN_ON(IS_ENABLED(CONFIG_EFI) && mm == efi_mm);
This creates new source code dependencies, which I would rather avoid if
possible.
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13/02/2024 12:02, Mark Rutland wrote:
> On Mon, Feb 12, 2024 at 12:59:57PM +, Ryan Roberts wrote:
>> On 12/02/2024 12:00, Mark Rutland wrote:
>>> Hi Ryan,
>
> [...]
>
+static inline void set_pte(pte_t *ptep, pte_t pte)
+{
+ /*
+ * We don't have the mm or vaddr so cannot unfold contig entries (since
+ * it requires tlb maintenance). set_pte() is not used in core code, so
+ * this should never even be called. Regardless do our best to service
+ * any call and emit a warning if there is any attempt to set a pte on
+ * top of an existing contig range.
+ */
+ pte_t orig_pte = __ptep_get(ptep);
+
+ WARN_ON_ONCE(pte_valid_cont(orig_pte));
+ __set_pte(ptep, pte_mknoncont(pte));
+}
+
+#define set_ptes set_ptes
+static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr)
+{
+ pte = pte_mknoncont(pte);
>>>
>>> Why do we have to clear the contiguous bit here? Is that for the same
>>> reason as
>>> set_pte(), or do we expect callers to legitimately call this with the
>>> contiguous bit set in 'pte'?
>>>
>>> I think you explained this to me in-person, and IIRC we don't expect
>>> callers to
>>> go set the bit themselves, but since it 'leaks' out to them via
>>> __ptep_get() we
>>> have to clear it here to defer the decision of whether to set/clear it when
>>> modifying entries. It would be nice if we could have a description of
>>> why/when
>>> we need to clear this, e.g. in the 'public API' comment block above.
>>
>> Yes, I think you've got it, but just to ram home the point: The PTE_CONT bit
>> is
>> private to the architecture code and is never set directly by core code. If
>> the
>> public API ever receives a pte that happens to have the PTE_CONT bit set, it
>> would be bad news if we then accidentally set that in the pgtable.
>>
>> Ideally, we would just uncondidtionally clear the bit before a getter returns
>> the pte (e.g. ptep_get(), ptep_get_lockless(), ptep_get_and_clear(), ...).
>> That
>> way, the code code is guarranteed never to see a pte with the PTE_CONT bit
>> set
>> and can therefore never accidentally pass such a pte into a setter function.
>> However, there is existing functionality that relies on being able to get a
>> pte,
>> then pass it to pte_leaf_size(), and arch function that checks the PTE_CONT
>> bit
>> to determine how big the leaf is. This is used in perf_get_pgtable_size().
>>
>> So to allow perf_get_pgtable_size() to continue to see the "real" page size,
>> I
>> decided to allow PTE_CONT to leak through the getters and instead
>> unconditionally clear the bit when a pte is passed to any of the setters.
>>
>> I'll add a (slightly less verbose) comment as you suggest.
>
> Great, thanks!
>
> [...]
>
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+ * Don't attempt to apply the contig bit to kernel mappings, because
+ * dynamically adding/removing the contig bit can cause page faults.
+ * These racing faults are ok for user space, since they get serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+ return mm != _mm;
+}
>>>
>>> We also have the efi_mm as a non-user mm, though I don't think we manipulate
>>> that while it is live, and I'm not sure if that needs any special handling.
>>
>> Well we never need this function in the hot (order-0 folio) path, so I think
>> I
>> could add a check for efi_mm here with performance implication. It's probably
>> safest to explicitly exclude it? What do you think?
>
> That sounds ok to me.
>
> Otherwise, if we (somehow) know that we avoid calling this at all with an EFI
> mm (e.g. because of the way we construct that), I'd be happy with a comment.
We crossed streams - as per my other email, I'm confident that this is safe so
will just add a comment.
>
> Probably best to Cc Ard for whatever we do here.
Ard is already on CC.
>
+static inline pte_t *contpte_align_down(pte_t *ptep)
+{
+ return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3);
>>>
>>> I think this can be:
>>>
>>> static inline pte_t *contpte_align_down(pte_t *ptep)
>>> {
>>> return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES);
>>> }
>>
>> Yep - that's much less ugly - thanks!
>>
>>>
+
+static void contpte_convert(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
+ unsigned long start_addr;
+ pte_t *start_ptep;
+ int i;
+
+ start_ptep = ptep = contpte_align_down(ptep);
+ start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE);
+ pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte));
+
+ for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) {
+
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 13.02.24 13:06, Ryan Roberts wrote:
On 12/02/2024 20:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+* Don't attempt to apply the contig bit to kernel mappings, because
+* dynamically adding/removing the contig bit can cause page faults.
+* These racing faults are ok for user space, since they get serialized
+* on the PTL. But kernel mappings can't tolerate faults.
+*/
+ return mm != _mm;
+}
We also have the efi_mm as a non-user mm, though I don't think we manipulate
that while it is live, and I'm not sure if that needs any special handling.
Well we never need this function in the hot (order-0 folio) path, so I think I
could add a check for efi_mm here with performance implication. It's probably
safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like userspace.
There are a couple of things that need to be garanteed for it to be safe:
- The PFNs of present ptes either need to have an associated struct page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause
fold/unfold
while live) or the system must be able to tolerate a temporary fault
Mark suggests efi_mm is not manipulated while live, so that meets the latter
requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests, the
mappings are indeed static. And additionally, the ptes are populated using only
the _private_ ptep API, so there is no issue here. As just discussed with Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
Sound good. We could add some VM_WARN_ON if we ever get the efi_mm via
the "official" APIs.
--
Cheers,
David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12/02/2024 20:38, Ryan Roberts wrote:
> [...]
>
> +static inline bool mm_is_user(struct mm_struct *mm)
> +{
> + /*
> + * Don't attempt to apply the contig bit to kernel mappings, because
> + * dynamically adding/removing the contig bit can cause page faults.
> + * These racing faults are ok for user space, since they get serialized
> + * on the PTL. But kernel mappings can't tolerate faults.
> + */
> + return mm != _mm;
> +}
We also have the efi_mm as a non-user mm, though I don't think we
manipulate
that while it is live, and I'm not sure if that needs any special handling.
>>>
>>> Well we never need this function in the hot (order-0 folio) path, so I
>>> think I
>>> could add a check for efi_mm here with performance implication. It's
>>> probably
>>> safest to explicitly exclude it? What do you think?
>>
>> Oops: This should have read "I think I could add a check for efi_mm here
>> *without* performance implication"
>
> It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can do
> this:
>
> return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
>
> Is that acceptable? This is my preference, but nothing else outside of efi
> references this symbol currently.
>
> Or perhaps I can convince myself that its safe to treat efi_mm like userspace.
> There are a couple of things that need to be garanteed for it to be safe:
>
> - The PFNs of present ptes either need to have an associated struct page or
> need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
> pte_mkdevmap())
>
> - Live mappings must either be static (no changes that could cause
> fold/unfold
> while live) or the system must be able to tolerate a temporary fault
>
> Mark suggests efi_mm is not manipulated while live, so that meets the latter
> requirement, but I'm not sure about the former?
I've gone through all the efi code, and conclude that, as Mark suggests, the
mappings are indeed static. And additionally, the ptes are populated using only
the _private_ ptep API, so there is no issue here. As just discussed with Mark,
my prefereence is to not make any changes to code, and just add a comment
describing why efi_mm is safe.
Details:
* Registered with ptdump
* ptep_get_lockless()
* efi_create_mapping -> create_pgd_mapping … -> init_pte:
* __ptep_get()
* __set_pte()
* efi_memattr_apply_permissions -> efi_set_mapping_permissions … ->
set_permissions
* __ptep_get()
* __set_pte()
Thanks,
Ryan
>
> Thanks,
> Ryan
>
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On Mon, Feb 12, 2024 at 12:59:57PM +, Ryan Roberts wrote:
> On 12/02/2024 12:00, Mark Rutland wrote:
> > Hi Ryan,
[...]
> >> +static inline void set_pte(pte_t *ptep, pte_t pte)
> >> +{
> >> + /*
> >> + * We don't have the mm or vaddr so cannot unfold contig entries (since
> >> + * it requires tlb maintenance). set_pte() is not used in core code, so
> >> + * this should never even be called. Regardless do our best to service
> >> + * any call and emit a warning if there is any attempt to set a pte on
> >> + * top of an existing contig range.
> >> + */
> >> + pte_t orig_pte = __ptep_get(ptep);
> >> +
> >> + WARN_ON_ONCE(pte_valid_cont(orig_pte));
> >> + __set_pte(ptep, pte_mknoncont(pte));
> >> +}
> >> +
> >> +#define set_ptes set_ptes
> >> +static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
> >> + pte_t *ptep, pte_t pte, unsigned int nr)
> >> +{
> >> + pte = pte_mknoncont(pte);
> >
> > Why do we have to clear the contiguous bit here? Is that for the same
> > reason as
> > set_pte(), or do we expect callers to legitimately call this with the
> > contiguous bit set in 'pte'?
> >
> > I think you explained this to me in-person, and IIRC we don't expect
> > callers to
> > go set the bit themselves, but since it 'leaks' out to them via
> > __ptep_get() we
> > have to clear it here to defer the decision of whether to set/clear it when
> > modifying entries. It would be nice if we could have a description of
> > why/when
> > we need to clear this, e.g. in the 'public API' comment block above.
>
> Yes, I think you've got it, but just to ram home the point: The PTE_CONT bit
> is
> private to the architecture code and is never set directly by core code. If
> the
> public API ever receives a pte that happens to have the PTE_CONT bit set, it
> would be bad news if we then accidentally set that in the pgtable.
>
> Ideally, we would just uncondidtionally clear the bit before a getter returns
> the pte (e.g. ptep_get(), ptep_get_lockless(), ptep_get_and_clear(), ...).
> That
> way, the code code is guarranteed never to see a pte with the PTE_CONT bit set
> and can therefore never accidentally pass such a pte into a setter function.
> However, there is existing functionality that relies on being able to get a
> pte,
> then pass it to pte_leaf_size(), and arch function that checks the PTE_CONT
> bit
> to determine how big the leaf is. This is used in perf_get_pgtable_size().
>
> So to allow perf_get_pgtable_size() to continue to see the "real" page size, I
> decided to allow PTE_CONT to leak through the getters and instead
> unconditionally clear the bit when a pte is passed to any of the setters.
>
> I'll add a (slightly less verbose) comment as you suggest.
Great, thanks!
[...]
> >> +static inline bool mm_is_user(struct mm_struct *mm)
> >> +{
> >> + /*
> >> + * Don't attempt to apply the contig bit to kernel mappings, because
> >> + * dynamically adding/removing the contig bit can cause page faults.
> >> + * These racing faults are ok for user space, since they get serialized
> >> + * on the PTL. But kernel mappings can't tolerate faults.
> >> + */
> >> + return mm != _mm;
> >> +}
> >
> > We also have the efi_mm as a non-user mm, though I don't think we manipulate
> > that while it is live, and I'm not sure if that needs any special handling.
>
> Well we never need this function in the hot (order-0 folio) path, so I think I
> could add a check for efi_mm here with performance implication. It's probably
> safest to explicitly exclude it? What do you think?
That sounds ok to me.
Otherwise, if we (somehow) know that we avoid calling this at all with an EFI
mm (e.g. because of the way we construct that), I'd be happy with a comment.
Probably best to Cc Ard for whatever we do here.
> >> +static inline pte_t *contpte_align_down(pte_t *ptep)
> >> +{
> >> + return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3);
> >
> > I think this can be:
> >
> > static inline pte_t *contpte_align_down(pte_t *ptep)
> > {
> > return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES);
> > }
>
> Yep - that's much less ugly - thanks!
>
> >
> >> +
> >> +static void contpte_convert(struct mm_struct *mm, unsigned long addr,
> >> + pte_t *ptep, pte_t pte)
> >> +{
> >> + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
> >> + unsigned long start_addr;
> >> + pte_t *start_ptep;
> >> + int i;
> >> +
> >> + start_ptep = ptep = contpte_align_down(ptep);
> >> + start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE);
> >> + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte));
> >> +
> >> + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) {
> >> + pte_t ptent = __ptep_get_and_clear(mm, addr, ptep);
> >> +
> >> + if (pte_dirty(ptent))
> >> + pte = pte_mkdirty(pte);
> >> +
> >> + if (pte_young(ptent))
> >> + pte =
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12.02.24 21:38, Ryan Roberts wrote:
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+* Don't attempt to apply the contig bit to kernel mappings, because
+* dynamically adding/removing the contig bit can cause page faults.
+* These racing faults are ok for user space, since they get serialized
+* on the PTL. But kernel mappings can't tolerate faults.
+*/
+ return mm != _mm;
+}
We also have the efi_mm as a non-user mm, though I don't think we manipulate
that while it is live, and I'm not sure if that needs any special handling.
Well we never need this function in the hot (order-0 folio) path, so I think I
could add a check for efi_mm here with performance implication. It's probably
safest to explicitly exclude it? What do you think?
Oops: This should have read "I think I could add a check for efi_mm here
*without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Please use all the lines you need ;)
if (IS_ENABLED(CONFIG_EFI) && unlikely(mm == _mm))
return false;
return mm != _mm;
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
We could also mark MMs in some way to be special.
return mm->is_user;
Then it's easy to extend.
--
Cheers,
David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
[...]
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ /*
+ * Don't attempt to apply the contig bit to kernel mappings, because
+ * dynamically adding/removing the contig bit can cause page faults.
+ * These racing faults are ok for user space, since they get serialized
+ * on the PTL. But kernel mappings can't tolerate faults.
+ */
+ return mm != _mm;
+}
>>>
>>> We also have the efi_mm as a non-user mm, though I don't think we manipulate
>>> that while it is live, and I'm not sure if that needs any special handling.
>>
>> Well we never need this function in the hot (order-0 folio) path, so I think
>> I
>> could add a check for efi_mm here with performance implication. It's probably
>> safest to explicitly exclude it? What do you think?
>
> Oops: This should have read "I think I could add a check for efi_mm here
> *without* performance implication"
It turns out that efi_mm is only defined when CONFIG_EFI is enabled. I can do
this:
return mm != _mm && (!IS_ENABLED(CONFIG_EFI) || mm != _mm);
Is that acceptable? This is my preference, but nothing else outside of efi
references this symbol currently.
Or perhaps I can convince myself that its safe to treat efi_mm like userspace.
There are a couple of things that need to be garanteed for it to be safe:
- The PFNs of present ptes either need to have an associated struct page or
need to have the PTE_SPECIAL bit set (either pte_mkspecial() or
pte_mkdevmap())
- Live mappings must either be static (no changes that could cause fold/unfold
while live) or the system must be able to tolerate a temporary fault
Mark suggests efi_mm is not manipulated while live, so that meets the latter
requirement, but I'm not sure about the former?
Thanks,
Ryan
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12.02.24 16:34, Ryan Roberts wrote: On 12/02/2024 15:26, David Hildenbrand wrote: On 12.02.24 15:45, Ryan Roberts wrote: On 12/02/2024 13:54, David Hildenbrand wrote: If so, I wonder if we could instead do that comparison modulo the access/dirty bits, I think that would work - but will need to think a bit more on it. and leave ptep_get_lockless() only reading a single entry? I think we will need to do something a bit less fragile. ptep_get() does collect the access/dirty bits so its confusing if ptep_get_lockless() doesn't IMHO. So we will likely want to rename the function and make its documentation explicit that it does not return those bits. ptep_get_lockless_noyoungdirty()? yuk... Any ideas? Of course if I could convince you the current implementation is safe, I might be able to sidestep this optimization until a later date? As discussed (and pointed out abive), there might be quite some callsites where we don't really care about uptodate accessed/dirty bits -- where ptep_get() is used nowadays. One way to approach that I had in mind was having an explicit interface: ptep_get() ptep_get_uptodate() ptep_get_lockless() ptep_get_lockless_uptodate() Yes, I like the direction of this. I guess we anticipate that call sites requiring the "_uptodate" variant will be the minority so it makes sense to use the current names for the "_not_uptodate" variants? But to do a slow migration, it might be better/safer to have the weaker variant use the new name - that would allow us to downgrade one at a time? Yes, I was primarily struggling with names. Likely it makes sense to either have two completely new function names, or use the new name only for the "faster but less precise" variant. Especially the last one might not be needed. I've done a scan through the code and agree with Mark's original conclusions. Additionally, huge_pte_alloc() (which isn't used for arm64) doesn't rely on access/dirty info. So I think I could migrate everything to the weaker variant fairly easily. Futher, "uptodate" might not be the best choice because of PageUptodate() and friends. But it's better than "youngdirty"/"noyoungdirty" IMHO. Certainly agree with "noyoungdirty" being a horrible name. How about "_sync" / "_nosync"? I could live with ptep_get_sync() ptep_get_nosync() with proper documentation :) but could you live with: ptep_get() ptep_get_nosync() ptep_get_lockless_nosync() ? So leave the "slower, more precise" version with the existing name. Sure. -- Cheers, David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12/02/2024 15:26, David Hildenbrand wrote: > On 12.02.24 15:45, Ryan Roberts wrote: >> On 12/02/2024 13:54, David Hildenbrand wrote: > If so, I wonder if we could instead do that comparison modulo the > access/dirty > bits, I think that would work - but will need to think a bit more on it. > and leave ptep_get_lockless() only reading a single entry? I think we will need to do something a bit less fragile. ptep_get() does collect the access/dirty bits so its confusing if ptep_get_lockless() doesn't IMHO. So we will likely want to rename the function and make its documentation explicit that it does not return those bits. ptep_get_lockless_noyoungdirty()? yuk... Any ideas? Of course if I could convince you the current implementation is safe, I might be able to sidestep this optimization until a later date? >>> >>> As discussed (and pointed out abive), there might be quite some callsites >>> where >>> we don't really care about uptodate accessed/dirty bits -- where ptep_get() >>> is >>> used nowadays. >>> >>> One way to approach that I had in mind was having an explicit interface: >>> >>> ptep_get() >>> ptep_get_uptodate() >>> ptep_get_lockless() >>> ptep_get_lockless_uptodate() >> >> Yes, I like the direction of this. I guess we anticipate that call sites >> requiring the "_uptodate" variant will be the minority so it makes sense to >> use >> the current names for the "_not_uptodate" variants? But to do a slow >> migration, >> it might be better/safer to have the weaker variant use the new name - that >> would allow us to downgrade one at a time? > > Yes, I was primarily struggling with names. Likely it makes sense to either > have > two completely new function names, or use the new name only for the "faster > but > less precise" variant. > >> >>> >>> Especially the last one might not be needed. >> I've done a scan through the code and agree with Mark's original conclusions. >> Additionally, huge_pte_alloc() (which isn't used for arm64) doesn't rely on >> access/dirty info. So I think I could migrate everything to the weaker >> variant >> fairly easily. >> >>> >>> Futher, "uptodate" might not be the best choice because of PageUptodate() >>> and >>> friends. But it's better than "youngdirty"/"noyoungdirty" IMHO. >> >> Certainly agree with "noyoungdirty" being a horrible name. How about "_sync" >> / >> "_nosync"? > > I could live with > > ptep_get_sync() > ptep_get_nosync() > > with proper documentation :) but could you live with: ptep_get() ptep_get_nosync() ptep_get_lockless_nosync() ? So leave the "slower, more precise" version with the existing name. > > I don't think we use "_sync" / "_nosync" in the context of pte operations yet. > > Well, there seems to be "__arm_v7s_pte_sync" in iommu code, bit at least in > core > code nothing jumped at me. >
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12/02/2024 12:59, Ryan Roberts wrote: > On 12/02/2024 12:00, Mark Rutland wrote: >> Hi Ryan, >> >> Overall this looks pretty good; I have a bunch of minor comments below, and a >> bigger question on the way ptep_get_lockless() works. > > OK great - thanks for the review. Let's see if I can answer them all... > >> >> On Fri, Feb 02, 2024 at 08:07:50AM +, Ryan Roberts wrote: >>> With the ptep API sufficiently refactored, we can now introduce a new >>> "contpte" API layer, which transparently manages the PTE_CONT bit for >>> user mappings. >>> >>> In this initial implementation, only suitable batches of PTEs, set via >>> set_ptes(), are mapped with the PTE_CONT bit. Any subsequent >>> modification of individual PTEs will cause an "unfold" operation to >>> repaint the contpte block as individual PTEs before performing the >>> requested operation. While, a modification of a single PTE could cause >>> the block of PTEs to which it belongs to become eligible for "folding" >>> into a contpte entry, "folding" is not performed in this initial >>> implementation due to the costs of checking the requirements are met. >>> Due to this, contpte mappings will degrade back to normal pte mappings >>> over time if/when protections are changed. This will be solved in a >>> future patch. >>> >>> Since a contpte block only has a single access and dirty bit, the >>> semantic here changes slightly; when getting a pte (e.g. ptep_get()) >>> that is part of a contpte mapping, the access and dirty information are >>> pulled from the block (so all ptes in the block return the same >>> access/dirty info). When changing the access/dirty info on a pte (e.g. >>> ptep_set_access_flags()) that is part of a contpte mapping, this change >>> will affect the whole contpte block. This is works fine in practice >>> since we guarantee that only a single folio is mapped by a contpte >>> block, and the core-mm tracks access/dirty information per folio. >>> >>> In order for the public functions, which used to be pure inline, to >>> continue to be callable by modules, export all the contpte_* symbols >>> that are now called by those public inline functions. >>> >>> The feature is enabled/disabled with the ARM64_CONTPTE Kconfig parameter >>> at build time. It defaults to enabled as long as its dependency, >>> TRANSPARENT_HUGEPAGE is also enabled. The core-mm depends upon >>> TRANSPARENT_HUGEPAGE to be able to allocate large folios, so if its not >>> enabled, then there is no chance of meeting the physical contiguity >>> requirement for contpte mappings. >>> >>> Tested-by: John Hubbard >>> Signed-off-by: Ryan Roberts >>> --- >>> arch/arm64/Kconfig | 9 + >>> arch/arm64/include/asm/pgtable.h | 161 ++ >>> arch/arm64/mm/Makefile | 1 + >>> arch/arm64/mm/contpte.c | 283 +++ >>> 4 files changed, 454 insertions(+) >>> create mode 100644 arch/arm64/mm/contpte.c >>> >>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >>> index d86d7f4758b5..1442e8ed95b6 100644 >>> --- a/arch/arm64/Kconfig >>> +++ b/arch/arm64/Kconfig >>> @@ -2230,6 +2230,15 @@ config UNWIND_PATCH_PAC_INTO_SCS >>> select UNWIND_TABLES >>> select DYNAMIC_SCS >>> >>> +config ARM64_CONTPTE >>> + bool "Contiguous PTE mappings for user memory" if EXPERT >>> + depends on TRANSPARENT_HUGEPAGE >>> + default y >>> + help >>> + When enabled, user mappings are configured using the PTE contiguous >>> + bit, for any mappings that meet the size and alignment requirements. >>> + This reduces TLB pressure and improves performance. >>> + >>> endmenu # "Kernel Features" >>> >>> menu "Boot options" >>> diff --git a/arch/arm64/include/asm/pgtable.h >>> b/arch/arm64/include/asm/pgtable.h >>> index 7dc6b68ee516..34892a95403d 100644 >>> --- a/arch/arm64/include/asm/pgtable.h >>> +++ b/arch/arm64/include/asm/pgtable.h >>> @@ -133,6 +133,10 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t >>> phys) >>> */ >>> #define pte_valid_not_user(pte) \ >>> ((pte_val(pte) & (PTE_VALID | PTE_USER | PTE_UXN)) == (PTE_VALID | >>> PTE_UXN)) >>> +/* >>> + * Returns true if the pte is valid and has the contiguous bit set. >>> + */ >>> +#define pte_valid_cont(pte)(pte_valid(pte) && pte_cont(pte)) >>> /* >>> * Could the pte be present in the TLB? We must check mm_tlb_flush_pending >>> * so that we don't erroneously return false for pages that have been >>> @@ -1135,6 +1139,161 @@ void vmemmap_update_pte(unsigned long addr, pte_t >>> *ptep, pte_t pte); >>> #define vmemmap_update_pte vmemmap_update_pte >>> #endif >>> >>> +#ifdef CONFIG_ARM64_CONTPTE >>> + >>> +/* >>> + * The contpte APIs are used to transparently manage the contiguous bit in >>> ptes >>> + * where it is possible and makes sense to do so. The PTE_CONT bit is >>> considered >>> + * a private implementation detail of the public ptep API (see below). >>> + */ >>> +extern void __contpte_try_unfold(struct
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12.02.24 15:45, Ryan Roberts wrote: On 12/02/2024 13:54, David Hildenbrand wrote: If so, I wonder if we could instead do that comparison modulo the access/dirty bits, I think that would work - but will need to think a bit more on it. and leave ptep_get_lockless() only reading a single entry? I think we will need to do something a bit less fragile. ptep_get() does collect the access/dirty bits so its confusing if ptep_get_lockless() doesn't IMHO. So we will likely want to rename the function and make its documentation explicit that it does not return those bits. ptep_get_lockless_noyoungdirty()? yuk... Any ideas? Of course if I could convince you the current implementation is safe, I might be able to sidestep this optimization until a later date? As discussed (and pointed out abive), there might be quite some callsites where we don't really care about uptodate accessed/dirty bits -- where ptep_get() is used nowadays. One way to approach that I had in mind was having an explicit interface: ptep_get() ptep_get_uptodate() ptep_get_lockless() ptep_get_lockless_uptodate() Yes, I like the direction of this. I guess we anticipate that call sites requiring the "_uptodate" variant will be the minority so it makes sense to use the current names for the "_not_uptodate" variants? But to do a slow migration, it might be better/safer to have the weaker variant use the new name - that would allow us to downgrade one at a time? Yes, I was primarily struggling with names. Likely it makes sense to either have two completely new function names, or use the new name only for the "faster but less precise" variant. Especially the last one might not be needed. I've done a scan through the code and agree with Mark's original conclusions. Additionally, huge_pte_alloc() (which isn't used for arm64) doesn't rely on access/dirty info. So I think I could migrate everything to the weaker variant fairly easily. Futher, "uptodate" might not be the best choice because of PageUptodate() and friends. But it's better than "youngdirty"/"noyoungdirty" IMHO. Certainly agree with "noyoungdirty" being a horrible name. How about "_sync" / "_nosync"? I could live with ptep_get_sync() ptep_get_nosync() with proper documentation :) I don't think we use "_sync" / "_nosync" in the context of pte operations yet. Well, there seems to be "__arm_v7s_pte_sync" in iommu code, bit at least in core code nothing jumped at me. -- Cheers, David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12/02/2024 13:54, David Hildenbrand wrote: >>> If so, I wonder if we could instead do that comparison modulo the >>> access/dirty >>> bits, >> >> I think that would work - but will need to think a bit more on it. >> >>> and leave ptep_get_lockless() only reading a single entry? >> >> I think we will need to do something a bit less fragile. ptep_get() does >> collect >> the access/dirty bits so its confusing if ptep_get_lockless() doesn't IMHO. >> So >> we will likely want to rename the function and make its documentation >> explicit >> that it does not return those bits. >> >> ptep_get_lockless_noyoungdirty()? yuk... Any ideas? >> >> Of course if I could convince you the current implementation is safe, I >> might be >> able to sidestep this optimization until a later date? > > As discussed (and pointed out abive), there might be quite some callsites > where > we don't really care about uptodate accessed/dirty bits -- where ptep_get() is > used nowadays. > > One way to approach that I had in mind was having an explicit interface: > > ptep_get() > ptep_get_uptodate() > ptep_get_lockless() > ptep_get_lockless_uptodate() Yes, I like the direction of this. I guess we anticipate that call sites requiring the "_uptodate" variant will be the minority so it makes sense to use the current names for the "_not_uptodate" variants? But to do a slow migration, it might be better/safer to have the weaker variant use the new name - that would allow us to downgrade one at a time? > > Especially the last one might not be needed. I've done a scan through the code and agree with Mark's original conclusions. Additionally, huge_pte_alloc() (which isn't used for arm64) doesn't rely on access/dirty info. So I think I could migrate everything to the weaker variant fairly easily. > > Futher, "uptodate" might not be the best choice because of PageUptodate() and > friends. But it's better than "youngdirty"/"noyoungdirty" IMHO. Certainly agree with "noyoungdirty" being a horrible name. How about "_sync" / "_nosync"? > > Of course, any such changes require care and are better done one step at at > time > separately. > So I propose to introduce ptep_get_lockless_nosync() (name up for discussion) and migrate all users to it, as part of this series. This will side-step Mark's correctness concerns. We can add ptep_get_nosync() later and migrate slowly. Shout if you think this is a bad plan. Thanks, Ryan
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
If so, I wonder if we could instead do that comparison modulo the access/dirty bits, I think that would work - but will need to think a bit more on it. and leave ptep_get_lockless() only reading a single entry? I think we will need to do something a bit less fragile. ptep_get() does collect the access/dirty bits so its confusing if ptep_get_lockless() doesn't IMHO. So we will likely want to rename the function and make its documentation explicit that it does not return those bits. ptep_get_lockless_noyoungdirty()? yuk... Any ideas? Of course if I could convince you the current implementation is safe, I might be able to sidestep this optimization until a later date? As discussed (and pointed out abive), there might be quite some callsites where we don't really care about uptodate accessed/dirty bits -- where ptep_get() is used nowadays. One way to approach that I had in mind was having an explicit interface: ptep_get() ptep_get_uptodate() ptep_get_lockless() ptep_get_lockless_uptodate() Especially the last one might not be needed. Futher, "uptodate" might not be the best choice because of PageUptodate() and friends. But it's better than "youngdirty"/"noyoungdirty" IMHO. Of course, any such changes require care and are better done one step at at time separately. -- Cheers, David / dhildenb
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
On 12/02/2024 12:00, Mark Rutland wrote: > Hi Ryan, > > Overall this looks pretty good; I have a bunch of minor comments below, and a > bigger question on the way ptep_get_lockless() works. OK great - thanks for the review. Let's see if I can answer them all... > > On Fri, Feb 02, 2024 at 08:07:50AM +, Ryan Roberts wrote: >> With the ptep API sufficiently refactored, we can now introduce a new >> "contpte" API layer, which transparently manages the PTE_CONT bit for >> user mappings. >> >> In this initial implementation, only suitable batches of PTEs, set via >> set_ptes(), are mapped with the PTE_CONT bit. Any subsequent >> modification of individual PTEs will cause an "unfold" operation to >> repaint the contpte block as individual PTEs before performing the >> requested operation. While, a modification of a single PTE could cause >> the block of PTEs to which it belongs to become eligible for "folding" >> into a contpte entry, "folding" is not performed in this initial >> implementation due to the costs of checking the requirements are met. >> Due to this, contpte mappings will degrade back to normal pte mappings >> over time if/when protections are changed. This will be solved in a >> future patch. >> >> Since a contpte block only has a single access and dirty bit, the >> semantic here changes slightly; when getting a pte (e.g. ptep_get()) >> that is part of a contpte mapping, the access and dirty information are >> pulled from the block (so all ptes in the block return the same >> access/dirty info). When changing the access/dirty info on a pte (e.g. >> ptep_set_access_flags()) that is part of a contpte mapping, this change >> will affect the whole contpte block. This is works fine in practice >> since we guarantee that only a single folio is mapped by a contpte >> block, and the core-mm tracks access/dirty information per folio. >> >> In order for the public functions, which used to be pure inline, to >> continue to be callable by modules, export all the contpte_* symbols >> that are now called by those public inline functions. >> >> The feature is enabled/disabled with the ARM64_CONTPTE Kconfig parameter >> at build time. It defaults to enabled as long as its dependency, >> TRANSPARENT_HUGEPAGE is also enabled. The core-mm depends upon >> TRANSPARENT_HUGEPAGE to be able to allocate large folios, so if its not >> enabled, then there is no chance of meeting the physical contiguity >> requirement for contpte mappings. >> >> Tested-by: John Hubbard >> Signed-off-by: Ryan Roberts >> --- >> arch/arm64/Kconfig | 9 + >> arch/arm64/include/asm/pgtable.h | 161 ++ >> arch/arm64/mm/Makefile | 1 + >> arch/arm64/mm/contpte.c | 283 +++ >> 4 files changed, 454 insertions(+) >> create mode 100644 arch/arm64/mm/contpte.c >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index d86d7f4758b5..1442e8ed95b6 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -2230,6 +2230,15 @@ config UNWIND_PATCH_PAC_INTO_SCS >> select UNWIND_TABLES >> select DYNAMIC_SCS >> >> +config ARM64_CONTPTE >> +bool "Contiguous PTE mappings for user memory" if EXPERT >> +depends on TRANSPARENT_HUGEPAGE >> +default y >> +help >> + When enabled, user mappings are configured using the PTE contiguous >> + bit, for any mappings that meet the size and alignment requirements. >> + This reduces TLB pressure and improves performance. >> + >> endmenu # "Kernel Features" >> >> menu "Boot options" >> diff --git a/arch/arm64/include/asm/pgtable.h >> b/arch/arm64/include/asm/pgtable.h >> index 7dc6b68ee516..34892a95403d 100644 >> --- a/arch/arm64/include/asm/pgtable.h >> +++ b/arch/arm64/include/asm/pgtable.h >> @@ -133,6 +133,10 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t >> phys) >> */ >> #define pte_valid_not_user(pte) \ >> ((pte_val(pte) & (PTE_VALID | PTE_USER | PTE_UXN)) == (PTE_VALID | >> PTE_UXN)) >> +/* >> + * Returns true if the pte is valid and has the contiguous bit set. >> + */ >> +#define pte_valid_cont(pte) (pte_valid(pte) && pte_cont(pte)) >> /* >> * Could the pte be present in the TLB? We must check mm_tlb_flush_pending >> * so that we don't erroneously return false for pages that have been >> @@ -1135,6 +1139,161 @@ void vmemmap_update_pte(unsigned long addr, pte_t >> *ptep, pte_t pte); >> #define vmemmap_update_pte vmemmap_update_pte >> #endif >> >> +#ifdef CONFIG_ARM64_CONTPTE >> + >> +/* >> + * The contpte APIs are used to transparently manage the contiguous bit in >> ptes >> + * where it is possible and makes sense to do so. The PTE_CONT bit is >> considered >> + * a private implementation detail of the public ptep API (see below). >> + */ >> +extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, >> +pte_t *ptep, pte_t pte); >> +extern pte_t contpte_ptep_get(pte_t *ptep, pte_t
Re: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
Hi Ryan, Overall this looks pretty good; I have a bunch of minor comments below, and a bigger question on the way ptep_get_lockless() works. On Fri, Feb 02, 2024 at 08:07:50AM +, Ryan Roberts wrote: > With the ptep API sufficiently refactored, we can now introduce a new > "contpte" API layer, which transparently manages the PTE_CONT bit for > user mappings. > > In this initial implementation, only suitable batches of PTEs, set via > set_ptes(), are mapped with the PTE_CONT bit. Any subsequent > modification of individual PTEs will cause an "unfold" operation to > repaint the contpte block as individual PTEs before performing the > requested operation. While, a modification of a single PTE could cause > the block of PTEs to which it belongs to become eligible for "folding" > into a contpte entry, "folding" is not performed in this initial > implementation due to the costs of checking the requirements are met. > Due to this, contpte mappings will degrade back to normal pte mappings > over time if/when protections are changed. This will be solved in a > future patch. > > Since a contpte block only has a single access and dirty bit, the > semantic here changes slightly; when getting a pte (e.g. ptep_get()) > that is part of a contpte mapping, the access and dirty information are > pulled from the block (so all ptes in the block return the same > access/dirty info). When changing the access/dirty info on a pte (e.g. > ptep_set_access_flags()) that is part of a contpte mapping, this change > will affect the whole contpte block. This is works fine in practice > since we guarantee that only a single folio is mapped by a contpte > block, and the core-mm tracks access/dirty information per folio. > > In order for the public functions, which used to be pure inline, to > continue to be callable by modules, export all the contpte_* symbols > that are now called by those public inline functions. > > The feature is enabled/disabled with the ARM64_CONTPTE Kconfig parameter > at build time. It defaults to enabled as long as its dependency, > TRANSPARENT_HUGEPAGE is also enabled. The core-mm depends upon > TRANSPARENT_HUGEPAGE to be able to allocate large folios, so if its not > enabled, then there is no chance of meeting the physical contiguity > requirement for contpte mappings. > > Tested-by: John Hubbard > Signed-off-by: Ryan Roberts > --- > arch/arm64/Kconfig | 9 + > arch/arm64/include/asm/pgtable.h | 161 ++ > arch/arm64/mm/Makefile | 1 + > arch/arm64/mm/contpte.c | 283 +++ > 4 files changed, 454 insertions(+) > create mode 100644 arch/arm64/mm/contpte.c > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index d86d7f4758b5..1442e8ed95b6 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -2230,6 +2230,15 @@ config UNWIND_PATCH_PAC_INTO_SCS > select UNWIND_TABLES > select DYNAMIC_SCS > > +config ARM64_CONTPTE > + bool "Contiguous PTE mappings for user memory" if EXPERT > + depends on TRANSPARENT_HUGEPAGE > + default y > + help > + When enabled, user mappings are configured using the PTE contiguous > + bit, for any mappings that meet the size and alignment requirements. > + This reduces TLB pressure and improves performance. > + > endmenu # "Kernel Features" > > menu "Boot options" > diff --git a/arch/arm64/include/asm/pgtable.h > b/arch/arm64/include/asm/pgtable.h > index 7dc6b68ee516..34892a95403d 100644 > --- a/arch/arm64/include/asm/pgtable.h > +++ b/arch/arm64/include/asm/pgtable.h > @@ -133,6 +133,10 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t > phys) > */ > #define pte_valid_not_user(pte) \ > ((pte_val(pte) & (PTE_VALID | PTE_USER | PTE_UXN)) == (PTE_VALID | > PTE_UXN)) > +/* > + * Returns true if the pte is valid and has the contiguous bit set. > + */ > +#define pte_valid_cont(pte) (pte_valid(pte) && pte_cont(pte)) > /* > * Could the pte be present in the TLB? We must check mm_tlb_flush_pending > * so that we don't erroneously return false for pages that have been > @@ -1135,6 +1139,161 @@ void vmemmap_update_pte(unsigned long addr, pte_t > *ptep, pte_t pte); > #define vmemmap_update_pte vmemmap_update_pte > #endif > > +#ifdef CONFIG_ARM64_CONTPTE > + > +/* > + * The contpte APIs are used to transparently manage the contiguous bit in > ptes > + * where it is possible and makes sense to do so. The PTE_CONT bit is > considered > + * a private implementation detail of the public ptep API (see below). > + */ > +extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, pte_t pte); > +extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); > +extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); > +extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, > + pte_t *ptep, pte_t pte, unsigned int nr);
[PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings
With the ptep API sufficiently refactored, we can now introduce a new "contpte" API layer, which transparently manages the PTE_CONT bit for user mappings. In this initial implementation, only suitable batches of PTEs, set via set_ptes(), are mapped with the PTE_CONT bit. Any subsequent modification of individual PTEs will cause an "unfold" operation to repaint the contpte block as individual PTEs before performing the requested operation. While, a modification of a single PTE could cause the block of PTEs to which it belongs to become eligible for "folding" into a contpte entry, "folding" is not performed in this initial implementation due to the costs of checking the requirements are met. Due to this, contpte mappings will degrade back to normal pte mappings over time if/when protections are changed. This will be solved in a future patch. Since a contpte block only has a single access and dirty bit, the semantic here changes slightly; when getting a pte (e.g. ptep_get()) that is part of a contpte mapping, the access and dirty information are pulled from the block (so all ptes in the block return the same access/dirty info). When changing the access/dirty info on a pte (e.g. ptep_set_access_flags()) that is part of a contpte mapping, this change will affect the whole contpte block. This is works fine in practice since we guarantee that only a single folio is mapped by a contpte block, and the core-mm tracks access/dirty information per folio. In order for the public functions, which used to be pure inline, to continue to be callable by modules, export all the contpte_* symbols that are now called by those public inline functions. The feature is enabled/disabled with the ARM64_CONTPTE Kconfig parameter at build time. It defaults to enabled as long as its dependency, TRANSPARENT_HUGEPAGE is also enabled. The core-mm depends upon TRANSPARENT_HUGEPAGE to be able to allocate large folios, so if its not enabled, then there is no chance of meeting the physical contiguity requirement for contpte mappings. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/Kconfig | 9 + arch/arm64/include/asm/pgtable.h | 161 ++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/contpte.c | 283 +++ 4 files changed, 454 insertions(+) create mode 100644 arch/arm64/mm/contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index d86d7f4758b5..1442e8ed95b6 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2230,6 +2230,15 @@ config UNWIND_PATCH_PAC_INTO_SCS select UNWIND_TABLES select DYNAMIC_SCS +config ARM64_CONTPTE + bool "Contiguous PTE mappings for user memory" if EXPERT + depends on TRANSPARENT_HUGEPAGE + default y + help + When enabled, user mappings are configured using the PTE contiguous + bit, for any mappings that meet the size and alignment requirements. + This reduces TLB pressure and improves performance. + endmenu # "Kernel Features" menu "Boot options" diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7dc6b68ee516..34892a95403d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -133,6 +133,10 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) */ #define pte_valid_not_user(pte) \ ((pte_val(pte) & (PTE_VALID | PTE_USER | PTE_UXN)) == (PTE_VALID | PTE_UXN)) +/* + * Returns true if the pte is valid and has the contiguous bit set. + */ +#define pte_valid_cont(pte)(pte_valid(pte) && pte_cont(pte)) /* * Could the pte be present in the TLB? We must check mm_tlb_flush_pending * so that we don't erroneously return false for pages that have been @@ -1135,6 +1139,161 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define vmemmap_update_pte vmemmap_update_pte #endif +#ifdef CONFIG_ARM64_CONTPTE + +/* + * The contpte APIs are used to transparently manage the contiguous bit in ptes + * where it is possible and makes sense to do so. The PTE_CONT bit is considered + * a private implementation detail of the public ptep API (see below). + */ +extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); +extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); +extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long
