"Huang, Ying" <ying.hu...@intel.com> writes:
> Alistair Popple <apop...@nvidia.com> writes:
>
>> "Huang, Ying" <ying.hu...@intel.com> writes:
>>
>>> Alistair Popple <apop...@nvidia.com> writes:
>>>
>>>> "Huang, Ying" <ying.hu...@intel.com> writes:
>>>>
>>>>> Alistair Popple <apop...@nvidia.com> writes:
>>>>>
>>>>>> Huang Ying <ying.hu...@intel.com> writes:
>>>>>>
>>>>>>> Previously, a fixed abstract distance MEMTIER_DEFAULT_DAX_ADISTANCE is
>>>>>>> used for slow memory type in kmem driver. This limits the usage of
>>>>>>> kmem driver, for example, it cannot be used for HBM (high bandwidth
>>>>>>> memory).
>>>>>>>
>>>>>>> So, we use the general abstract distance calculation mechanism in kmem
>>>>>>> drivers to get more accurate abstract distance on systems with proper
>>>>>>> support. The original MEMTIER_DEFAULT_DAX_ADISTANCE is used as
>>>>>>> fallback only.
>>>>>>>
>>>>>>> Now, multiple memory types may be managed by kmem. These memory types
>>>>>>> are put into the "kmem_memory_types" list and protected by
>>>>>>> kmem_memory_type_lock.
>>>>>>
>>>>>> See below but I wonder if kmem_memory_types could be a common helper
>>>>>> rather than kdax specific?
>>>>>>
>>>>>>> Signed-off-by: "Huang, Ying" <ying.hu...@intel.com>
>>>>>>> Cc: Aneesh Kumar K.V <aneesh.ku...@linux.ibm.com>
>>>>>>> Cc: Wei Xu <weix...@google.com>
>>>>>>> Cc: Alistair Popple <apop...@nvidia.com>
>>>>>>> Cc: Dan Williams <dan.j.willi...@intel.com>
>>>>>>> Cc: Dave Hansen <dave.han...@intel.com>
>>>>>>> Cc: Davidlohr Bueso <d...@stgolabs.net>
>>>>>>> Cc: Johannes Weiner <han...@cmpxchg.org>
>>>>>>> Cc: Jonathan Cameron <jonathan.came...@huawei.com>
>>>>>>> Cc: Michal Hocko <mho...@kernel.org>
>>>>>>> Cc: Yang Shi <shy828...@gmail.com>
>>>>>>> Cc: Rafael J Wysocki <rafael.j.wyso...@intel.com>
>>>>>>> ---
>>>>>>> drivers/dax/kmem.c | 54 +++++++++++++++++++++++++++---------
>>>>>>> include/linux/memory-tiers.h | 2 ++
>>>>>>> mm/memory-tiers.c | 2 +-
>>>>>>> 3 files changed, 44 insertions(+), 14 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
>>>>>>> index 898ca9505754..837165037231 100644
>>>>>>> --- a/drivers/dax/kmem.c
>>>>>>> +++ b/drivers/dax/kmem.c
>>>>>>> @@ -49,14 +49,40 @@ struct dax_kmem_data {
>>>>>>> struct resource *res[];
>>>>>>> };
>>>>>>>
>>>>>>> -static struct memory_dev_type *dax_slowmem_type;
>>>>>>> +static DEFINE_MUTEX(kmem_memory_type_lock);
>>>>>>> +static LIST_HEAD(kmem_memory_types);
>>>>>>> +
>>>>>>> +static struct memory_dev_type *kmem_find_alloc_memorty_type(int adist)
>>>>>>> +{
>>>>>>> + bool found = false;
>>>>>>> + struct memory_dev_type *mtype;
>>>>>>> +
>>>>>>> + mutex_lock(&kmem_memory_type_lock);
>>>>>>> + list_for_each_entry(mtype, &kmem_memory_types, list) {
>>>>>>> + if (mtype->adistance == adist) {
>>>>>>> + found = true;
>>>>>>> + break;
>>>>>>> + }
>>>>>>> + }
>>>>>>> + if (!found) {
>>>>>>> + mtype = alloc_memory_type(adist);
>>>>>>> + if (!IS_ERR(mtype))
>>>>>>> + list_add(&mtype->list, &kmem_memory_types);
>>>>>>> + }
>>>>>>> + mutex_unlock(&kmem_memory_type_lock);
>>>>>>> +
>>>>>>> + return mtype;
>>>>>>> +}
>>>>>>> +
>>>>>>> static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>>>>>>> {
>>>>>>> struct device *dev = &dev_dax->dev;
>>>>>>> unsigned long total_len = 0;
>>>>>>> struct dax_kmem_data *data;
>>>>>>> + struct memory_dev_type *mtype;
>>>>>>> int i, rc, mapped = 0;
>>>>>>> int numa_node;
>>>>>>> + int adist = MEMTIER_DEFAULT_DAX_ADISTANCE;
>>>>>>>
>>>>>>> /*
>>>>>>> * Ensure good NUMA information for the persistent memory.
>>>>>>> @@ -71,6 +97,11 @@ static int dev_dax_kmem_probe(struct dev_dax
>>>>>>> *dev_dax)
>>>>>>> return -EINVAL;
>>>>>>> }
>>>>>>>
>>>>>>> + mt_calc_adistance(numa_node, &adist);
>>>>>>> + mtype = kmem_find_alloc_memorty_type(adist);
>>>>>>> + if (IS_ERR(mtype))
>>>>>>> + return PTR_ERR(mtype);
>>>>>>> +
>>>>>>
>>>>>> I wrote my own quick and dirty module to test this and wrote basically
>>>>>> the same code sequence.
>>>>>>
>>>>>> I notice your using a list of memory types here though. I think it would
>>>>>> be nice to have a common helper that other users could call to do the
>>>>>> mt_calc_adistance() / kmem_find_alloc_memory_type() /
>>>>>> init_node_memory_type() sequence and cleanup as my naive approach would
>>>>>> result in a new memory_dev_type per device even though adist might be
>>>>>> the same. A common helper would make it easy to de-dup those.
>>>>>
>>>>> If it's useful, we can move kmem_find_alloc_memory_type() to
>>>>> memory-tier.c after some revision. But I tend to move it after we have
>>>>> the second user. What do you think about that?
>>>>
>>>> Usually I would agree, but this series already introduces a general
>>>> interface for calculating adist even though there's only one user and
>>>> implementation. So if we're going to add a general interface I think it
>>>> would be better to make it more usable now rather than after variations
>>>> of it have been cut and pasted into other drivers.
>>>
>>> In general, I would like to introduce complexity when necessary. So, we
>>> can discuss the necessity of the general interface firstly. We can do
>>> that in [1/4] of the series.
>>
>> Do we need one memory_dev_type per adistance or per adistance+device?
>>
>> If IUC correctly I think it's the former. Logically that means
>> memory_dev_types should be managed by the memory-tiering subsystem
>> because they are system wide rather than driver specific resources. That
>> we need to add the list field to struct memory_dev_type specifically for
>> use by dax/kmem supports that idea.
>
> In the original design (page 9/10/11 of [1]), memory_dev_type (Memory
> Type) is driver specific.
Oh fair enough. I was making these comments based on the incorrect
understanding that these were a global rather than driver specific
resource. Thanks for correcting that!
>> Also I'm not sure why you consider moving the
>> kmem_memory_types/kmem_find_alloc_memory_type()/etc. functions into
>> mm/memory-tiers.c to add complexity. Isn't it just moving code around or
>> am I missing some other subtlety that makes this hard? I really think
>> logically memory-tiering.c is where management of the various
>> memory_dev_types belongs.
>
> IMHO, it depends on whether these functions are shared by at least 2
> drivers. If so, we can put them in mm/memory-tiers.c. Otherwise, we
> should keep them in the driver.
Ok. Not sure I entirely agree because I suspect it would still make the
code clearer even for a single user. But generally you're correct and as
these memory_dev_type's are *supposed* to be driver specific (rather
than one per adistance) I don't think it's such a big issue.
