Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

2020-06-05 Thread David Hildenbrand
On 04.06.20 17:27, David Hildenbrand wrote:
> On 04.06.20 17:06, Christoph Hellwig wrote:
>> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
>>> Just a thought: If memory hotplug is applicable as well, you might
>>> either want to always assume data->enable_4GB, or handle memory hotplug
>>> events from the memory notifier, when new memory gets onlined (not sure
>>> how tricky that is).
>>
>> We probably want a highest_pfn_possible() or similar API instead of
>> having drivers poking into random VM internals.
> 
> Well, memory notifiers are a reasonable api used accross the kernel to
> get notified when new memory is onlined to the buddy that could be used
> for allocations.
> 
> highest_pfn_possible() would have to default to something linked to
> MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure
> how helpful that is (IOW, you can just default to enable_4GB=true in
> that case instead in most cases).

Correction: At least on x86-64 we have max_possible_pfn, which will
consult the ACPI SRAT table to figure out the maximum possible PFN.
(Without SRAT, max_possible_pfn will point at the end of initial boot
memory and not consider hotplug memory - something that e.g., newer QEMU
versions work around by creating SRAT tables if memory hotplug might be
possible, even if there is no actual NUMA configuration).

pci-swiotlb.c similarly relies on that to figure out if there are any
!DMA addresses to handle.

-- 
Thanks,

David / dhildenb

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu


Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

2020-06-04 Thread David Hildenbrand
On 04.06.20 17:06, Christoph Hellwig wrote:
> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
>> Just a thought: If memory hotplug is applicable as well, you might
>> either want to always assume data->enable_4GB, or handle memory hotplug
>> events from the memory notifier, when new memory gets onlined (not sure
>> how tricky that is).
> 
> We probably want a highest_pfn_possible() or similar API instead of
> having drivers poking into random VM internals.

Well, memory notifiers are a reasonable api used accross the kernel to
get notified when new memory is onlined to the buddy that could be used
for allocations.

highest_pfn_possible() would have to default to something linked to
MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure
how helpful that is (IOW, you can just default to enable_4GB=true in
that case instead in most cases).

-- 
Thanks,

David / dhildenb

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu


Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

2020-06-04 Thread Christoph Hellwig
On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote:
> Just a thought: If memory hotplug is applicable as well, you might
> either want to always assume data->enable_4GB, or handle memory hotplug
> events from the memory notifier, when new memory gets onlined (not sure
> how tricky that is).

We probably want a highest_pfn_possible() or similar API instead of
having drivers poking into random VM internals.
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu


Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

2020-06-04 Thread David Hildenbrand
On 04.06.20 11:49, Miles Chen wrote:
> On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote:
>> On 04.06.20 10:01, Miles Chen wrote:
>>> To build this driver as a kernel module, we cannot use
>>> the unexported symbol "max_pfn" to setup enable_4GB.
>>>
>>> Use totalram_pages() instead to setup enable_4GB.
>>>
>>> Suggested-by: Mike Rapoport 
>>> Signed-off-by: Miles Chen 
>>> Cc: David Hildenbrand 
>>> Cc: Yong Wu 
>>> Cc: Chao Hao 
>>> ---
>>>  drivers/iommu/mtk_iommu.c | 5 ++---
>>>  1 file changed, 2 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
>>> index 5f4d6df59cf6..c2798a6e0e38 100644
>>> --- a/drivers/iommu/mtk_iommu.c
>>> +++ b/drivers/iommu/mtk_iommu.c
>>> @@ -3,7 +3,6 @@
>>>   * Copyright (c) 2015-2016 MediaTek Inc.
>>>   * Author: Yong Wu 
>>>   */
>>> -#include 
>>>  #include 
>>>  #include 
>>>  #include 
>>> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
>>> return -ENOMEM;
>>> data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
>>>  
>>> -   /* Whether the current dram is over 4GB */
>>> -   data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
>>> +   /* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
>>> +   data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> 
>>> PAGE_SHIFT));
>>
>> A similar thing seems to be done by
>> drivers/media/platform/mtk-vpu/mtk_vpu.c:
>>  vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
>>
>> I do wonder if some weird memory hotplug setups might give you false
>> negatives.
>>
>> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
>> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
>> return something < 2GB.
>>
>> Same can happen when you have a VM and use ballooning to fake-unplug
>> memory, making totalram_pages() return something < 4GB, but leaving
>> usable pfns >= 4GB
> 
> Yes. Yingjoe also told me that this patch is not correct.
> 
> Thanks for pointing this out. totalram_pages() does not work 
> for some cases:
> 

Just a thought: If memory hotplug is applicable as well, you might
either want to always assume data->enable_4GB, or handle memory hotplug
events from the memory notifier, when new memory gets onlined (not sure
how tricky that is).


-- 
Thanks,

David / dhildenb

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu


Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

2020-06-04 Thread Miles Chen
On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote:
> On 04.06.20 10:01, Miles Chen wrote:
> > To build this driver as a kernel module, we cannot use
> > the unexported symbol "max_pfn" to setup enable_4GB.
> > 
> > Use totalram_pages() instead to setup enable_4GB.
> > 
> > Suggested-by: Mike Rapoport 
> > Signed-off-by: Miles Chen 
> > Cc: David Hildenbrand 
> > Cc: Yong Wu 
> > Cc: Chao Hao 
> > ---
> >  drivers/iommu/mtk_iommu.c | 5 ++---
> >  1 file changed, 2 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> > index 5f4d6df59cf6..c2798a6e0e38 100644
> > --- a/drivers/iommu/mtk_iommu.c
> > +++ b/drivers/iommu/mtk_iommu.c
> > @@ -3,7 +3,6 @@
> >   * Copyright (c) 2015-2016 MediaTek Inc.
> >   * Author: Yong Wu 
> >   */
> > -#include 
> >  #include 
> >  #include 
> >  #include 
> > @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> > return -ENOMEM;
> > data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
> >  
> > -   /* Whether the current dram is over 4GB */
> > -   data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> > +   /* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
> > +   data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> 
> > PAGE_SHIFT));
> 
> A similar thing seems to be done by
> drivers/media/platform/mtk-vpu/mtk_vpu.c:
>   vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));
> 
> I do wonder if some weird memory hotplug setups might give you false
> negatives.
> 
> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
> return something < 2GB.
> 
> Same can happen when you have a VM and use ballooning to fake-unplug
> memory, making totalram_pages() return something < 4GB, but leaving
> usable pfns >= 4GB

Yes. Yingjoe also told me that this patch is not correct.

Thanks for pointing this out. totalram_pages() does not work 
for some cases:

e.g., DRAM start @0x4000_ and DRAM size is 0x1__ but we
reserve large amount of memory, which makes totalram_pages() < 3GB but
it is possible to allocate a pfn >= 4GB.

I will discuss this internally.

Miles
> .
> 
> but
> ... I don't know if I understood what "enable_4GB" needs/implies
> ... I don't know if this is applicable to VMs
>  at all (on real HW such
> memory hotplug setups should not exist)
> ... I don't know how this code would react to memory hotplug, so if the
> condition changes after the driver loaded and enable_4GB would
> suddenly apply. Again, most probably not relevant on real HW, only
> for VMs.
> 

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu


Re: [PATCH] iommu/mediatek: Use totalram_pages to setup enable_4GB

2020-06-04 Thread David Hildenbrand
On 04.06.20 10:01, Miles Chen wrote:
> To build this driver as a kernel module, we cannot use
> the unexported symbol "max_pfn" to setup enable_4GB.
> 
> Use totalram_pages() instead to setup enable_4GB.
> 
> Suggested-by: Mike Rapoport 
> Signed-off-by: Miles Chen 
> Cc: David Hildenbrand 
> Cc: Yong Wu 
> Cc: Chao Hao 
> ---
>  drivers/iommu/mtk_iommu.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> index 5f4d6df59cf6..c2798a6e0e38 100644
> --- a/drivers/iommu/mtk_iommu.c
> +++ b/drivers/iommu/mtk_iommu.c
> @@ -3,7 +3,6 @@
>   * Copyright (c) 2015-2016 MediaTek Inc.
>   * Author: Yong Wu 
>   */
> -#include 
>  #include 
>  #include 
>  #include 
> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
>   return -ENOMEM;
>   data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
>  
> - /* Whether the current dram is over 4GB */
> - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> + /* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
> + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> 
> PAGE_SHIFT));

A similar thing seems to be done by
drivers/media/platform/mtk-vpu/mtk_vpu.c:
vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));

I do wonder if some weird memory hotplug setups might give you false
negatives.

E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on
x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should
return something < 2GB.

Same can happen when you have a VM and use ballooning to fake-unplug
memory, making totalram_pages() return something < 4GB, but leaving
usable pfns >= 4GB.

but
... I don't know if I understood what "enable_4GB" needs/implies
... I don't know if this is applicable to VMs at all (on real HW such
memory hotplug setups should not exist)
... I don't know how this code would react to memory hotplug, so if the
condition changes after the driver loaded and enable_4GB would
suddenly apply. Again, most probably not relevant on real HW, only
for VMs.

-- 
Thanks,

David / dhildenb

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu