Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Huang, Ying]

Johannes Weiner  writes:

> On Thu, Mar 30, 2017 at 12:15:13PM +0800, Huang, Ying wrote:
>> Johannes Weiner  writes:
>> > On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
>> >> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
>> >> *list)
>> >>   VM_BUG_ON_PAGE(!PageLocked(page), page);
>> >>   VM_BUG_ON_PAGE(!PageUptodate(page), page);
>> >>  
>> >> + if (unlikely(PageTransHuge(page))) {
>> >> + err = add_to_swap_trans_huge(page, list);
>> >> + switch (err) {
>> >> + case 1:
>> >> + return 1;
>> >> + case 0:
>> >> + /* fallback to split firstly if return 0 */
>> >> + break;
>> >> + default:
>> >> + return 0;
>> >> + }
>> >> + }
>> >>   entry = get_swap_page();
>> >>   if (!entry.val)
>> >>   return 0;
>> >
>> > add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
>> > makes the code error prone for future modifications to the swap slot
>> > allocation protocol.
>> >
>> > This should read:
>> >
>> > retry:
>> >entry = get_swap_page(page);
>> >if (!entry.val) {
>> >if (PageTransHuge(page)) {
>> >split_huge_page_to_list(page, list);
>> >goto retry;
>> >}
>> >return 0;
>> >}
>> 
>> If the swap space is used up, that is, get_swap_page() cannot allocate
>> even 1 swap entry for a normal page.  We will split THP unnecessarily
>> with the change, but in the original code, we just skip the THP.  There
>> may be a performance regression here.  Similar problem exists for
>> mem_cgroup_try_charge_swap() too.  If the mem cgroup exceeds the swap
>> limit, the THP will be split unnecessary with the change too.
>
> If we skip the page, we're swapping out another page hotter than this
> one. Giving THP preservation priority over LRU order is an issue best
> kept for a separate patch set;

In my original patch, if we failed to allocate the swap space for a THP,
and we can allocate the swap space for a normal page, we will split the
THP.  We skip the page only if we cannot allocate the swap space for a
normal page, that is, nr_swap_pages is 0.  So we will not give THP
preservation priority over LRU order in the patch.

> this one is supposed to be a mechanical
> implementation of THP swapping. Let's nail down the basics first.

Yes.  So I tried to keep the original behavior to deal with THP if we
cannot allocate the swap space (a swap cluster) for a whole THP.

Per my understanding, the difference between what you suggested and the
original behavior is that, when nr_swap_pages is 0, whether to split the
THP.

> Such a decision would need proof that splitting THPs on full swap
> devices is a concern for real applications. I would assume that we're
> pretty close to OOM anyway; it's much more likely that a single slot
> frees up than a full cluster, at which point we'll be splitting THPs
> anyway; etc. I have my doubts that this would be measurable.
>
> But even if so, I don't think we'd have to duplicate the main code
> flow to handle this corner case. You can extend get_swap_page() to
> return an error code that tells add_to_swap() whether to split and
> retry, or to fail and move on. So this way should be future proof.

Yes.  I will try to merge add_to_swap_trans_huge() into add_to_swap() in
the next version.  But if we want to keep the original behavior, we will
need an extra "nr_entries" parameter for mem_cgroup_try_charge_swap().

Best Regards,
Huang, Ying

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Huang, Ying]

Johannes Weiner  writes:

> On Thu, Mar 30, 2017 at 12:15:13PM +0800, Huang, Ying wrote:
>> Johannes Weiner  writes:
>> > On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
>> >> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
>> >> *list)
>> >>   VM_BUG_ON_PAGE(!PageLocked(page), page);
>> >>   VM_BUG_ON_PAGE(!PageUptodate(page), page);
>> >>  
>> >> + if (unlikely(PageTransHuge(page))) {
>> >> + err = add_to_swap_trans_huge(page, list);
>> >> + switch (err) {
>> >> + case 1:
>> >> + return 1;
>> >> + case 0:
>> >> + /* fallback to split firstly if return 0 */
>> >> + break;
>> >> + default:
>> >> + return 0;
>> >> + }
>> >> + }
>> >>   entry = get_swap_page();
>> >>   if (!entry.val)
>> >>   return 0;
>> >
>> > add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
>> > makes the code error prone for future modifications to the swap slot
>> > allocation protocol.
>> >
>> > This should read:
>> >
>> > retry:
>> >entry = get_swap_page(page);
>> >if (!entry.val) {
>> >if (PageTransHuge(page)) {
>> >split_huge_page_to_list(page, list);
>> >goto retry;
>> >}
>> >return 0;
>> >}
>> 
>> If the swap space is used up, that is, get_swap_page() cannot allocate
>> even 1 swap entry for a normal page.  We will split THP unnecessarily
>> with the change, but in the original code, we just skip the THP.  There
>> may be a performance regression here.  Similar problem exists for
>> mem_cgroup_try_charge_swap() too.  If the mem cgroup exceeds the swap
>> limit, the THP will be split unnecessary with the change too.
>
> If we skip the page, we're swapping out another page hotter than this
> one. Giving THP preservation priority over LRU order is an issue best
> kept for a separate patch set;

In my original patch, if we failed to allocate the swap space for a THP,
and we can allocate the swap space for a normal page, we will split the
THP.  We skip the page only if we cannot allocate the swap space for a
normal page, that is, nr_swap_pages is 0.  So we will not give THP
preservation priority over LRU order in the patch.

> this one is supposed to be a mechanical
> implementation of THP swapping. Let's nail down the basics first.

Yes.  So I tried to keep the original behavior to deal with THP if we
cannot allocate the swap space (a swap cluster) for a whole THP.

Per my understanding, the difference between what you suggested and the
original behavior is that, when nr_swap_pages is 0, whether to split the
THP.

> Such a decision would need proof that splitting THPs on full swap
> devices is a concern for real applications. I would assume that we're
> pretty close to OOM anyway; it's much more likely that a single slot
> frees up than a full cluster, at which point we'll be splitting THPs
> anyway; etc. I have my doubts that this would be measurable.
>
> But even if so, I don't think we'd have to duplicate the main code
> flow to handle this corner case. You can extend get_swap_page() to
> return an error code that tells add_to_swap() whether to split and
> retry, or to fail and move on. So this way should be future proof.

Yes.  I will try to merge add_to_swap_trans_huge() into add_to_swap() in
the next version.  But if we want to keep the original behavior, we will
need an extra "nr_entries" parameter for mem_cgroup_try_charge_swap().

Best Regards,
Huang, Ying

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Johannes Weiner]

On Thu, Mar 30, 2017 at 12:15:13PM +0800, Huang, Ying wrote:
> Johannes Weiner  writes:
> > On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
> >> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
> >> *list)
> >>VM_BUG_ON_PAGE(!PageLocked(page), page);
> >>VM_BUG_ON_PAGE(!PageUptodate(page), page);
> >>  
> >> +  if (unlikely(PageTransHuge(page))) {
> >> +  err = add_to_swap_trans_huge(page, list);
> >> +  switch (err) {
> >> +  case 1:
> >> +  return 1;
> >> +  case 0:
> >> +  /* fallback to split firstly if return 0 */
> >> +  break;
> >> +  default:
> >> +  return 0;
> >> +  }
> >> +  }
> >>entry = get_swap_page();
> >>if (!entry.val)
> >>return 0;
> >
> > add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
> > makes the code error prone for future modifications to the swap slot
> > allocation protocol.
> >
> > This should read:
> >
> > retry:
> > entry = get_swap_page(page);
> > if (!entry.val) {
> > if (PageTransHuge(page)) {
> > split_huge_page_to_list(page, list);
> > goto retry;
> > }
> > return 0;
> > }
> 
> If the swap space is used up, that is, get_swap_page() cannot allocate
> even 1 swap entry for a normal page.  We will split THP unnecessarily
> with the change, but in the original code, we just skip the THP.  There
> may be a performance regression here.  Similar problem exists for
> mem_cgroup_try_charge_swap() too.  If the mem cgroup exceeds the swap
> limit, the THP will be split unnecessary with the change too.

If we skip the page, we're swapping out another page hotter than this
one. Giving THP preservation priority over LRU order is an issue best
kept for a separate patch set; this one is supposed to be a mechanical
implementation of THP swapping. Let's nail down the basics first.

Such a decision would need proof that splitting THPs on full swap
devices is a concern for real applications. I would assume that we're
pretty close to OOM anyway; it's much more likely that a single slot
frees up than a full cluster, at which point we'll be splitting THPs
anyway; etc. I have my doubts that this would be measurable.

But even if so, I don't think we'd have to duplicate the main code
flow to handle this corner case. You can extend get_swap_page() to
return an error code that tells add_to_swap() whether to split and
retry, or to fail and move on. So this way should be future proof.

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Johannes Weiner]

On Thu, Mar 30, 2017 at 12:15:13PM +0800, Huang, Ying wrote:
> Johannes Weiner  writes:
> > On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
> >> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
> >> *list)
> >>VM_BUG_ON_PAGE(!PageLocked(page), page);
> >>VM_BUG_ON_PAGE(!PageUptodate(page), page);
> >>  
> >> +  if (unlikely(PageTransHuge(page))) {
> >> +  err = add_to_swap_trans_huge(page, list);
> >> +  switch (err) {
> >> +  case 1:
> >> +  return 1;
> >> +  case 0:
> >> +  /* fallback to split firstly if return 0 */
> >> +  break;
> >> +  default:
> >> +  return 0;
> >> +  }
> >> +  }
> >>entry = get_swap_page();
> >>if (!entry.val)
> >>return 0;
> >
> > add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
> > makes the code error prone for future modifications to the swap slot
> > allocation protocol.
> >
> > This should read:
> >
> > retry:
> > entry = get_swap_page(page);
> > if (!entry.val) {
> > if (PageTransHuge(page)) {
> > split_huge_page_to_list(page, list);
> > goto retry;
> > }
> > return 0;
> > }
> 
> If the swap space is used up, that is, get_swap_page() cannot allocate
> even 1 swap entry for a normal page.  We will split THP unnecessarily
> with the change, but in the original code, we just skip the THP.  There
> may be a performance regression here.  Similar problem exists for
> mem_cgroup_try_charge_swap() too.  If the mem cgroup exceeds the swap
> limit, the THP will be split unnecessary with the change too.

If we skip the page, we're swapping out another page hotter than this
one. Giving THP preservation priority over LRU order is an issue best
kept for a separate patch set; this one is supposed to be a mechanical
implementation of THP swapping. Let's nail down the basics first.

Such a decision would need proof that splitting THPs on full swap
devices is a concern for real applications. I would assume that we're
pretty close to OOM anyway; it's much more likely that a single slot
frees up than a full cluster, at which point we'll be splitting THPs
anyway; etc. I have my doubts that this would be measurable.

But even if so, I don't think we'd have to duplicate the main code
flow to handle this corner case. You can extend get_swap_page() to
return an error code that tells add_to_swap() whether to split and
retry, or to fail and move on. So this way should be future proof.

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Huang, Ying]

Johannes Weiner  writes:

> On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
>> @@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
>>  ADD_CACHE_INFO(del_total, nr);
>>  }
>>  
>> +#ifdef CONFIG_THP_SWAP_CLUSTER
>> +int add_to_swap_trans_huge(struct page *page, struct list_head *list)
>> +{
>> +swp_entry_t entry;
>> +int ret = 0;
>> +
>> +/* cannot split, which may be needed during swap in, skip it */
>> +if (!can_split_huge_page(page, NULL))
>> +return -EBUSY;
>> +/* fallback to split huge page firstly if no PMD map */
>> +if (!compound_mapcount(page))
>> +return 0;
>> +entry = get_huge_swap_page();
>> +if (!entry.val)
>> +return 0;
>> +if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
>> +__swapcache_free(entry, true);
>> +return -EOVERFLOW;
>> +}
>> +ret = add_to_swap_cache(page, entry,
>> +__GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
>> +/* -ENOMEM radix-tree allocation failure */
>> +if (ret) {
>> +__swapcache_free(entry, true);
>> +return 0;
>> +}
>> +ret = split_huge_page_to_list(page, list);
>> +if (ret) {
>> +delete_from_swap_cache(page);
>> +return -EBUSY;
>> +}
>> +return 1;
>> +}
>> +#else
>> +static inline int add_to_swap_trans_huge(struct page *page,
>> + struct list_head *list)
>> +{
>> +return 0;
>> +}
>> +#endif
>> +
>>  /**
>>   * add_to_swap - allocate swap space for a page
>>   * @page: page we want to move to swap
>>   *
>>   * Allocate swap space for the page and add the page to the
>> - * swap cache.  Caller needs to hold the page lock. 
>> + * swap cache.  Caller needs to hold the page lock.
>>   */
>>  int add_to_swap(struct page *page, struct list_head *list)
>>  {
>> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
>> *list)
>>  VM_BUG_ON_PAGE(!PageLocked(page), page);
>>  VM_BUG_ON_PAGE(!PageUptodate(page), page);
>>  
>> +if (unlikely(PageTransHuge(page))) {
>> +err = add_to_swap_trans_huge(page, list);
>> +switch (err) {
>> +case 1:
>> +return 1;
>> +case 0:
>> +/* fallback to split firstly if return 0 */
>> +break;
>> +default:
>> +return 0;
>> +}
>> +}
>>  entry = get_swap_page();
>>  if (!entry.val)
>>  return 0;
>
> add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
> makes the code error prone for future modifications to the swap slot
> allocation protocol.
>
> This should read:
>
> retry:
>   entry = get_swap_page(page);
>   if (!entry.val) {
>   if (PageTransHuge(page)) {
>   split_huge_page_to_list(page, list);
>   goto retry;
>   }
>   return 0;
>   }

If the swap space is used up, that is, get_swap_page() cannot allocate
even 1 swap entry for a normal page.  We will split THP unnecessarily
with the change, but in the original code, we just skip the THP.  There
may be a performance regression here.  Similar problem exists for
mem_cgroup_try_charge_swap() too.  If the mem cgroup exceeds the swap
limit, the THP will be split unnecessary with the change too.

> And get_swap_page(), mem_cgroup_try_charge_swap() etc. should all
> check PageTransHuge() instead of having extra parameters or separate
> code paths for the huge page case.
>
> In general, don't try to tack this feature onto the side of the
> VM. Because right now, this looks a bit like the hugetlb code, with
> one big branch in the beginning that opens up an alternate
> reality. Instead, these functions should handle THP all the way down
> the stack, and without passing down redundant information.

Yes.  We should share the code as much as possible.  I just have some
questions as above.  Could you help me on that?

Best Regards,
Huang, Ying

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Huang, Ying]

Johannes Weiner  writes:

> On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
>> @@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
>>  ADD_CACHE_INFO(del_total, nr);
>>  }
>>  
>> +#ifdef CONFIG_THP_SWAP_CLUSTER
>> +int add_to_swap_trans_huge(struct page *page, struct list_head *list)
>> +{
>> +swp_entry_t entry;
>> +int ret = 0;
>> +
>> +/* cannot split, which may be needed during swap in, skip it */
>> +if (!can_split_huge_page(page, NULL))
>> +return -EBUSY;
>> +/* fallback to split huge page firstly if no PMD map */
>> +if (!compound_mapcount(page))
>> +return 0;
>> +entry = get_huge_swap_page();
>> +if (!entry.val)
>> +return 0;
>> +if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
>> +__swapcache_free(entry, true);
>> +return -EOVERFLOW;
>> +}
>> +ret = add_to_swap_cache(page, entry,
>> +__GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
>> +/* -ENOMEM radix-tree allocation failure */
>> +if (ret) {
>> +__swapcache_free(entry, true);
>> +return 0;
>> +}
>> +ret = split_huge_page_to_list(page, list);
>> +if (ret) {
>> +delete_from_swap_cache(page);
>> +return -EBUSY;
>> +}
>> +return 1;
>> +}
>> +#else
>> +static inline int add_to_swap_trans_huge(struct page *page,
>> + struct list_head *list)
>> +{
>> +return 0;
>> +}
>> +#endif
>> +
>>  /**
>>   * add_to_swap - allocate swap space for a page
>>   * @page: page we want to move to swap
>>   *
>>   * Allocate swap space for the page and add the page to the
>> - * swap cache.  Caller needs to hold the page lock. 
>> + * swap cache.  Caller needs to hold the page lock.
>>   */
>>  int add_to_swap(struct page *page, struct list_head *list)
>>  {
>> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
>> *list)
>>  VM_BUG_ON_PAGE(!PageLocked(page), page);
>>  VM_BUG_ON_PAGE(!PageUptodate(page), page);
>>  
>> +if (unlikely(PageTransHuge(page))) {
>> +err = add_to_swap_trans_huge(page, list);
>> +switch (err) {
>> +case 1:
>> +return 1;
>> +case 0:
>> +/* fallback to split firstly if return 0 */
>> +break;
>> +default:
>> +return 0;
>> +}
>> +}
>>  entry = get_swap_page();
>>  if (!entry.val)
>>  return 0;
>
> add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
> makes the code error prone for future modifications to the swap slot
> allocation protocol.
>
> This should read:
>
> retry:
>   entry = get_swap_page(page);
>   if (!entry.val) {
>   if (PageTransHuge(page)) {
>   split_huge_page_to_list(page, list);
>   goto retry;
>   }
>   return 0;
>   }

If the swap space is used up, that is, get_swap_page() cannot allocate
even 1 swap entry for a normal page.  We will split THP unnecessarily
with the change, but in the original code, we just skip the THP.  There
may be a performance regression here.  Similar problem exists for
mem_cgroup_try_charge_swap() too.  If the mem cgroup exceeds the swap
limit, the THP will be split unnecessary with the change too.

> And get_swap_page(), mem_cgroup_try_charge_swap() etc. should all
> check PageTransHuge() instead of having extra parameters or separate
> code paths for the huge page case.
>
> In general, don't try to tack this feature onto the side of the
> VM. Because right now, this looks a bit like the hugetlb code, with
> one big branch in the beginning that opens up an alternate
> reality. Instead, these functions should handle THP all the way down
> the stack, and without passing down redundant information.

Yes.  We should share the code as much as possible.  I just have some
questions as above.  Could you help me on that?

Best Regards,
Huang, Ying

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Johannes Weiner]

On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
> @@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
>   ADD_CACHE_INFO(del_total, nr);
>  }
>  
> +#ifdef CONFIG_THP_SWAP_CLUSTER
> +int add_to_swap_trans_huge(struct page *page, struct list_head *list)
> +{
> + swp_entry_t entry;
> + int ret = 0;
> +
> + /* cannot split, which may be needed during swap in, skip it */
> + if (!can_split_huge_page(page, NULL))
> + return -EBUSY;
> + /* fallback to split huge page firstly if no PMD map */
> + if (!compound_mapcount(page))
> + return 0;
> + entry = get_huge_swap_page();
> + if (!entry.val)
> + return 0;
> + if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
> + __swapcache_free(entry, true);
> + return -EOVERFLOW;
> + }
> + ret = add_to_swap_cache(page, entry,
> + __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
> + /* -ENOMEM radix-tree allocation failure */
> + if (ret) {
> + __swapcache_free(entry, true);
> + return 0;
> + }
> + ret = split_huge_page_to_list(page, list);
> + if (ret) {
> + delete_from_swap_cache(page);
> + return -EBUSY;
> + }
> + return 1;
> +}
> +#else
> +static inline int add_to_swap_trans_huge(struct page *page,
> +  struct list_head *list)
> +{
> + return 0;
> +}
> +#endif
> +
>  /**
>   * add_to_swap - allocate swap space for a page
>   * @page: page we want to move to swap
>   *
>   * Allocate swap space for the page and add the page to the
> - * swap cache.  Caller needs to hold the page lock. 
> + * swap cache.  Caller needs to hold the page lock.
>   */
>  int add_to_swap(struct page *page, struct list_head *list)
>  {
> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
> *list)
>   VM_BUG_ON_PAGE(!PageLocked(page), page);
>   VM_BUG_ON_PAGE(!PageUptodate(page), page);
>  
> + if (unlikely(PageTransHuge(page))) {
> + err = add_to_swap_trans_huge(page, list);
> + switch (err) {
> + case 1:
> + return 1;
> + case 0:
> + /* fallback to split firstly if return 0 */
> + break;
> + default:
> + return 0;
> + }
> + }
>   entry = get_swap_page();
>   if (!entry.val)
>   return 0;

add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
makes the code error prone for future modifications to the swap slot
allocation protocol.

This should read:

retry:
entry = get_swap_page(page);
if (!entry.val) {
if (PageTransHuge(page)) {
split_huge_page_to_list(page, list);
goto retry;
}
return 0;
}

And get_swap_page(), mem_cgroup_try_charge_swap() etc. should all
check PageTransHuge() instead of having extra parameters or separate
code paths for the huge page case.

In general, don't try to tack this feature onto the side of the
VM. Because right now, this looks a bit like the hugetlb code, with
one big branch in the beginning that opens up an alternate
reality. Instead, these functions should handle THP all the way down
the stack, and without passing down redundant information.

Re: [PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Johannes Weiner]

On Tue, Mar 28, 2017 at 01:32:09PM +0800, Huang, Ying wrote:
> @@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
>   ADD_CACHE_INFO(del_total, nr);
>  }
>  
> +#ifdef CONFIG_THP_SWAP_CLUSTER
> +int add_to_swap_trans_huge(struct page *page, struct list_head *list)
> +{
> + swp_entry_t entry;
> + int ret = 0;
> +
> + /* cannot split, which may be needed during swap in, skip it */
> + if (!can_split_huge_page(page, NULL))
> + return -EBUSY;
> + /* fallback to split huge page firstly if no PMD map */
> + if (!compound_mapcount(page))
> + return 0;
> + entry = get_huge_swap_page();
> + if (!entry.val)
> + return 0;
> + if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
> + __swapcache_free(entry, true);
> + return -EOVERFLOW;
> + }
> + ret = add_to_swap_cache(page, entry,
> + __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
> + /* -ENOMEM radix-tree allocation failure */
> + if (ret) {
> + __swapcache_free(entry, true);
> + return 0;
> + }
> + ret = split_huge_page_to_list(page, list);
> + if (ret) {
> + delete_from_swap_cache(page);
> + return -EBUSY;
> + }
> + return 1;
> +}
> +#else
> +static inline int add_to_swap_trans_huge(struct page *page,
> +  struct list_head *list)
> +{
> + return 0;
> +}
> +#endif
> +
>  /**
>   * add_to_swap - allocate swap space for a page
>   * @page: page we want to move to swap
>   *
>   * Allocate swap space for the page and add the page to the
> - * swap cache.  Caller needs to hold the page lock. 
> + * swap cache.  Caller needs to hold the page lock.
>   */
>  int add_to_swap(struct page *page, struct list_head *list)
>  {
> @@ -198,6 +240,18 @@ int add_to_swap(struct page *page, struct list_head 
> *list)
>   VM_BUG_ON_PAGE(!PageLocked(page), page);
>   VM_BUG_ON_PAGE(!PageUptodate(page), page);
>  
> + if (unlikely(PageTransHuge(page))) {
> + err = add_to_swap_trans_huge(page, list);
> + switch (err) {
> + case 1:
> + return 1;
> + case 0:
> + /* fallback to split firstly if return 0 */
> + break;
> + default:
> + return 0;
> + }
> + }
>   entry = get_swap_page();
>   if (!entry.val)
>   return 0;

add_to_swap_trans_huge() is too close a copy of add_to_swap(), which
makes the code error prone for future modifications to the swap slot
allocation protocol.

This should read:

retry:
entry = get_swap_page(page);
if (!entry.val) {
if (PageTransHuge(page)) {
split_huge_page_to_list(page, list);
goto retry;
}
return 0;
}

And get_swap_page(), mem_cgroup_try_charge_swap() etc. should all
check PageTransHuge() instead of having extra parameters or separate
code paths for the huge page case.

In general, don't try to tack this feature onto the side of the
VM. Because right now, this looks a bit like the hugetlb code, with
one big branch in the beginning that opens up an alternate
reality. Instead, these functions should handle THP all the way down
the stack, and without passing down redundant information.

[PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Huang, Ying]

From: Huang Ying 

In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the
THP (Transparent Huge Page) and adding the THP into the swap cache.
This will reduce lock acquiring/releasing for the locks used for the
swap cache management.

This is the first step for the THP swap support.  The plan is to delay
splitting the THP step by step and avoid splitting the THP finally.

The advantages of the THP swap support include:

- Batch the swap operations for the THP to reduce lock
  acquiring/releasing, including allocating/freeing the swap space,
  adding/deleting to/from the swap cache, and writing/reading the swap
  space, etc.  This will help to improve the THP swap performance.

- The THP swap space read/write will be 2M sequential IO.  It is
  particularly helpful for the swap read, which usually are 4k random
  IO.  This will help to improve the THP swap performance too.

- It will help the memory fragmentation, especially when the THP is
  heavily used by the applications.  The 2M continuous pages will be
  free up after the THP swapping out.

- It will improve the THP utilization on the system with the swap
  turned on.  Because the speed for khugepaged to collapse the normal
  pages into the THP is quite slow.  After the THP is split during the
  swapping out, it will take quite long time for the normal pages to
  collapse back into the THP after being swapped in.  The high THP
  utilization helps the efficiency of the page based memory management
  too.

There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device.  To deal with that, the THP swap in should be
turned on only when necessary.  For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.

With the patchset, the swap out throughput improves 14.9% (from about
3.77GB/s to about 4.34GB/s) in the vm-scalability swap-w-seq test case
with 8 processes.  The test is done on a Xeon E5 v3 system.  The swap
device used is a RAM simulated PMEM (persistent memory) device.  To
test the sequential swapping out, the test case creates 8 processes,
which sequentially allocate and write to the anonymous pages until the
RAM and part of the swap device is used up.

The detailed comparison result is as follow,

base base+patchset
 --
 %stddev %change %stddev
 \  |\
   7043990 ±  0% +21.2%8536807 ±  0%  vm-scalability.throughput
109.94 ±  1% -16.2%  92.09 ±  0%  vm-scalability.time.elapsed_time
   3957091 ±  0% +14.9%4547173 ±  0%  vmstat.swap.so
 31.46 ±  1% -38.3%  19.42 ±  0%  perf-stat.cache-miss-rate%
  1.04 ±  1% +22.2%   1.27 ±  0%  perf-stat.ipc
  9.33 ±  2% -60.7%   3.67 ±  1%  
perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node

Signed-off-by: "Huang, Ying" 
---
 mm/swap_state.c | 60 ++---
 1 file changed, 57 insertions(+), 3 deletions(-)

diff --git a/mm/swap_state.c b/mm/swap_state.c
index 504f67d73f67..6d63ff703d39 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -19,6 +19,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
ADD_CACHE_INFO(del_total, nr);
 }
 
+#ifdef CONFIG_THP_SWAP_CLUSTER
+int add_to_swap_trans_huge(struct page *page, struct list_head *list)
+{
+   swp_entry_t entry;
+   int ret = 0;
+
+   /* cannot split, which may be needed during swap in, skip it */
+   if (!can_split_huge_page(page, NULL))
+   return -EBUSY;
+   /* fallback to split huge page firstly if no PMD map */
+   if (!compound_mapcount(page))
+   return 0;
+   entry = get_huge_swap_page();
+   if (!entry.val)
+   return 0;
+   if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
+   __swapcache_free(entry, true);
+   return -EOVERFLOW;
+   }
+   ret = add_to_swap_cache(page, entry,
+   __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
+   /* -ENOMEM radix-tree allocation failure */
+   if (ret) {
+   __swapcache_free(entry, true);
+   return 0;
+   }
+   ret = split_huge_page_to_list(page, list);
+   if (ret) {
+   delete_from_swap_cache(page);
+   return -EBUSY;
+   }
+   return 1;
+}
+#else
+static inline int add_to_swap_trans_huge(struct page *page,
+struct list_head *list)
+{
+   return 0;
+}
+#endif
+
 /**
  * 

[PATCH -mm -v7 9/9] mm, THP, swap: Delay splitting THP during swap out

[Huang, Ying]

From: Huang Ying 

In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the
THP (Transparent Huge Page) and adding the THP into the swap cache.
This will reduce lock acquiring/releasing for the locks used for the
swap cache management.

This is the first step for the THP swap support.  The plan is to delay
splitting the THP step by step and avoid splitting the THP finally.

The advantages of the THP swap support include:

- Batch the swap operations for the THP to reduce lock
  acquiring/releasing, including allocating/freeing the swap space,
  adding/deleting to/from the swap cache, and writing/reading the swap
  space, etc.  This will help to improve the THP swap performance.

- The THP swap space read/write will be 2M sequential IO.  It is
  particularly helpful for the swap read, which usually are 4k random
  IO.  This will help to improve the THP swap performance too.

- It will help the memory fragmentation, especially when the THP is
  heavily used by the applications.  The 2M continuous pages will be
  free up after the THP swapping out.

- It will improve the THP utilization on the system with the swap
  turned on.  Because the speed for khugepaged to collapse the normal
  pages into the THP is quite slow.  After the THP is split during the
  swapping out, it will take quite long time for the normal pages to
  collapse back into the THP after being swapped in.  The high THP
  utilization helps the efficiency of the page based memory management
  too.

There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device.  To deal with that, the THP swap in should be
turned on only when necessary.  For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.

With the patchset, the swap out throughput improves 14.9% (from about
3.77GB/s to about 4.34GB/s) in the vm-scalability swap-w-seq test case
with 8 processes.  The test is done on a Xeon E5 v3 system.  The swap
device used is a RAM simulated PMEM (persistent memory) device.  To
test the sequential swapping out, the test case creates 8 processes,
which sequentially allocate and write to the anonymous pages until the
RAM and part of the swap device is used up.

The detailed comparison result is as follow,

base base+patchset
 --
 %stddev %change %stddev
 \  |\
   7043990 ±  0% +21.2%8536807 ±  0%  vm-scalability.throughput
109.94 ±  1% -16.2%  92.09 ±  0%  vm-scalability.time.elapsed_time
   3957091 ±  0% +14.9%4547173 ±  0%  vmstat.swap.so
 31.46 ±  1% -38.3%  19.42 ±  0%  perf-stat.cache-miss-rate%
  1.04 ±  1% +22.2%   1.27 ±  0%  perf-stat.ipc
  9.33 ±  2% -60.7%   3.67 ±  1%  
perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node

Signed-off-by: "Huang, Ying" 
---
 mm/swap_state.c | 60 ++---
 1 file changed, 57 insertions(+), 3 deletions(-)

diff --git a/mm/swap_state.c b/mm/swap_state.c
index 504f67d73f67..6d63ff703d39 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -19,6 +19,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -183,12 +184,53 @@ void __delete_from_swap_cache(struct page *page)
ADD_CACHE_INFO(del_total, nr);
 }
 
+#ifdef CONFIG_THP_SWAP_CLUSTER
+int add_to_swap_trans_huge(struct page *page, struct list_head *list)
+{
+   swp_entry_t entry;
+   int ret = 0;
+
+   /* cannot split, which may be needed during swap in, skip it */
+   if (!can_split_huge_page(page, NULL))
+   return -EBUSY;
+   /* fallback to split huge page firstly if no PMD map */
+   if (!compound_mapcount(page))
+   return 0;
+   entry = get_huge_swap_page();
+   if (!entry.val)
+   return 0;
+   if (mem_cgroup_try_charge_swap(page, entry, HPAGE_PMD_NR)) {
+   __swapcache_free(entry, true);
+   return -EOVERFLOW;
+   }
+   ret = add_to_swap_cache(page, entry,
+   __GFP_HIGH | __GFP_NOMEMALLOC|__GFP_NOWARN);
+   /* -ENOMEM radix-tree allocation failure */
+   if (ret) {
+   __swapcache_free(entry, true);
+   return 0;
+   }
+   ret = split_huge_page_to_list(page, list);
+   if (ret) {
+   delete_from_swap_cache(page);
+   return -EBUSY;
+   }
+   return 1;
+}
+#else
+static inline int add_to_swap_trans_huge(struct page *page,
+struct list_head *list)
+{
+   return 0;
+}
+#endif
+
 /**
  * add_to_swap - allocate swap space for a page
  *