Re: [PATCH v4 08/13] mm/mempolicy: Create a page allocator for policy
On Wed, Apr 14, 2021 at 03:08:19PM +0200, Michal Hocko wrote:
> On Wed 17-03-21 11:40:05, Feng Tang wrote:
> > From: Ben Widawsky
> >
> > Add a helper function which takes care of handling multiple preferred
> > nodes. It will be called by future patches that need to handle this,
> > specifically VMA based page allocation, and task based page allocation.
> > Huge pages don't quite fit the same pattern because they use different
> > underlying page allocation functions. This consumes the previous
> > interleave policy specific allocation function to make a one stop shop
> > for policy based allocation.
> >
> > With this, MPOL_PREFERRED_MANY's semantic is more like MPOL_PREFERRED
> > that it will first try the preferred node/nodes, and fallback to all
> > other nodes when first try fails. Thanks to Michal Hocko for suggestions
> > on this.
> >
> > For now, only interleaved policy will be used so there should be no
> > functional change yet. However, if bisection points to issues in the
> > next few commits, it was likely the fault of this patch.
>
> I am not sure this is helping much. Let's see in later patches but I
> would keep them separate and rather create a dedicated function for the
> new policy allocation mode.
Thanks for the suggestion, we will rethink the implementations.
- Feng
> > Similar functionality is offered via policy_node() and
> > policy_nodemask(). By themselves however, neither can achieve this
> > fallback style of sets of nodes.
> >
> > [ Feng: for the first try, add NOWARN flag, and skip the direct reclaim
> > to speedup allocation in some case ]
> >
> > Link:
> > https://lore.kernel.org/r/20200630212517.308045-9-ben.widaw...@intel.com
> > Signed-off-by: Ben Widawsky
> > Signed-off-by: Feng Tang
> > ---
> > mm/mempolicy.c | 65
> > ++
> > 1 file changed, 52 insertions(+), 13 deletions(-)
> >
> > diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> > index d945f29..d21105b 100644
> > --- a/mm/mempolicy.c
> > +++ b/mm/mempolicy.c
> > @@ -2187,22 +2187,60 @@ bool mempolicy_nodemask_intersects(struct
> > task_struct *tsk,
> > return ret;
> > }
> >
> > -/* Allocate a page in interleaved policy.
> > - Own path because it needs to do special accounting. */
> > -static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
> > - unsigned nid)
> > +/* Handle page allocation for all but interleaved policies */
> > +static struct page *alloc_pages_policy(struct mempolicy *pol, gfp_t gfp,
> > + unsigned int order, int preferred_nid)
> > {
> > struct page *page;
> > + gfp_t gfp_mask = gfp;
> >
> > - page = __alloc_pages(gfp, order, nid);
> > - /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
> > - if (!static_branch_likely(_numa_stat_key))
> > + if (pol->mode == MPOL_INTERLEAVE) {
> > + page = __alloc_pages(gfp, order, preferred_nid);
> > + /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is
> > disabled */
> > + if (!static_branch_likely(_numa_stat_key))
> > + return page;
> > + if (page && page_to_nid(page) == preferred_nid) {
> > + preempt_disable();
> > + __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> > + preempt_enable();
> > + }
> > return page;
> > - if (page && page_to_nid(page) == nid) {
> > - preempt_disable();
> > - __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> > - preempt_enable();
> > }
> > +
> > + VM_BUG_ON(preferred_nid != NUMA_NO_NODE);
> > +
> > + preferred_nid = numa_node_id();
> > +
> > + /*
> > +* There is a two pass approach implemented here for
> > +* MPOL_PREFERRED_MANY. In the first pass we try the preferred nodes
> > +* but allow the allocation to fail. The below table explains how
> > +* this is achieved.
> > +*
> > +* | Policy| preferred nid | nodemask |
> > +* |---|---||
> > +* | MPOL_DEFAULT | local | NULL |
> > +* | MPOL_PREFERRED| best | NULL |
> > +* | MPOL_INTERLEAVE | ERR | ERR|
> > +* | MPOL_BIND | local | pol->nodes |
> > +* | MPOL_PREFERRED_MANY | best | pol->nodes |
> > +* | MPOL_PREFERRED_MANY (round 2) | local | NULL |
> > +* +---+---++
> > +*/
> > + if (pol->mode == MPOL_PREFERRED_MANY) {
> > + gfp_mask |= __GFP_NOWARN;
> > +
> > + /* Skip direct reclaim, as there will be a second try */
> > + gfp_mask &= ~__GFP_DIRECT_RECLAIM;
> > + }
> > +
> > + page = __alloc_pages_nodemask(gfp_mask,
Re: [PATCH v4 08/13] mm/mempolicy: Create a page allocator for policy
On Wed 17-03-21 11:40:05, Feng Tang wrote:
> From: Ben Widawsky
>
> Add a helper function which takes care of handling multiple preferred
> nodes. It will be called by future patches that need to handle this,
> specifically VMA based page allocation, and task based page allocation.
> Huge pages don't quite fit the same pattern because they use different
> underlying page allocation functions. This consumes the previous
> interleave policy specific allocation function to make a one stop shop
> for policy based allocation.
>
> With this, MPOL_PREFERRED_MANY's semantic is more like MPOL_PREFERRED
> that it will first try the preferred node/nodes, and fallback to all
> other nodes when first try fails. Thanks to Michal Hocko for suggestions
> on this.
>
> For now, only interleaved policy will be used so there should be no
> functional change yet. However, if bisection points to issues in the
> next few commits, it was likely the fault of this patch.
I am not sure this is helping much. Let's see in later patches but I
would keep them separate and rather create a dedicated function for the
new policy allocation mode.
> Similar functionality is offered via policy_node() and
> policy_nodemask(). By themselves however, neither can achieve this
> fallback style of sets of nodes.
>
> [ Feng: for the first try, add NOWARN flag, and skip the direct reclaim
> to speedup allocation in some case ]
>
> Link: https://lore.kernel.org/r/20200630212517.308045-9-ben.widaw...@intel.com
> Signed-off-by: Ben Widawsky
> Signed-off-by: Feng Tang
> ---
> mm/mempolicy.c | 65
> ++
> 1 file changed, 52 insertions(+), 13 deletions(-)
>
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index d945f29..d21105b 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -2187,22 +2187,60 @@ bool mempolicy_nodemask_intersects(struct task_struct
> *tsk,
> return ret;
> }
>
> -/* Allocate a page in interleaved policy.
> - Own path because it needs to do special accounting. */
> -static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
> - unsigned nid)
> +/* Handle page allocation for all but interleaved policies */
> +static struct page *alloc_pages_policy(struct mempolicy *pol, gfp_t gfp,
> +unsigned int order, int preferred_nid)
> {
> struct page *page;
> + gfp_t gfp_mask = gfp;
>
> - page = __alloc_pages(gfp, order, nid);
> - /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
> - if (!static_branch_likely(_numa_stat_key))
> + if (pol->mode == MPOL_INTERLEAVE) {
> + page = __alloc_pages(gfp, order, preferred_nid);
> + /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is
> disabled */
> + if (!static_branch_likely(_numa_stat_key))
> + return page;
> + if (page && page_to_nid(page) == preferred_nid) {
> + preempt_disable();
> + __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> + preempt_enable();
> + }
> return page;
> - if (page && page_to_nid(page) == nid) {
> - preempt_disable();
> - __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> - preempt_enable();
> }
> +
> + VM_BUG_ON(preferred_nid != NUMA_NO_NODE);
> +
> + preferred_nid = numa_node_id();
> +
> + /*
> + * There is a two pass approach implemented here for
> + * MPOL_PREFERRED_MANY. In the first pass we try the preferred nodes
> + * but allow the allocation to fail. The below table explains how
> + * this is achieved.
> + *
> + * | Policy| preferred nid | nodemask |
> + * |---|---||
> + * | MPOL_DEFAULT | local | NULL |
> + * | MPOL_PREFERRED| best | NULL |
> + * | MPOL_INTERLEAVE | ERR | ERR|
> + * | MPOL_BIND | local | pol->nodes |
> + * | MPOL_PREFERRED_MANY | best | pol->nodes |
> + * | MPOL_PREFERRED_MANY (round 2) | local | NULL |
> + * +---+---++
> + */
> + if (pol->mode == MPOL_PREFERRED_MANY) {
> + gfp_mask |= __GFP_NOWARN;
> +
> + /* Skip direct reclaim, as there will be a second try */
> + gfp_mask &= ~__GFP_DIRECT_RECLAIM;
> + }
> +
> + page = __alloc_pages_nodemask(gfp_mask, order,
> + policy_node(gfp, pol, preferred_nid),
> + policy_nodemask(gfp, pol));
> +
> + if (unlikely(!page && pol->mode == MPOL_PREFERRED_MANY))
> + page =
