Re: [RFC] mm: activate access-more-than-once page via NUMA balancing
Yu Zhao writes: > On Fri, Mar 26, 2021 at 12:21 AM Huang, Ying wrote: >> >> Mel Gorman writes: >> >> > On Thu, Mar 25, 2021 at 12:33:45PM +0800, Huang, Ying wrote: >> >> > I caution against this patch. >> >> > >> >> > It's non-deterministic for a number of reasons. As it requires NUMA >> >> > balancing to be enabled, the pageout behaviour of a system changes when >> >> > NUMA balancing is active. If this led to pages being artificially and >> >> > inappropriately preserved, NUMA balancing could be disabled for the >> >> > wrong reasons. It only applies to pages that have no target node so >> >> > memory policies affect which pages are activated differently. Similarly, >> >> > NUMA balancing does not scan all VMAs and some pages may never trap a >> >> > NUMA fault as a result. The timing of when an address space gets scanned >> >> > is driven by the locality of pages and so the timing of page activation >> >> > potentially becomes linked to whether pages are local or need to migrate >> >> > (although not right now for this patch as it only affects pages with a >> >> > target nid of NUMA_NO_NODE). In other words, changes in NUMA balancing >> >> > that affect migration potentially affect the aging rate. Similarly, >> >> > the activate rate of a process with a single thread and multiple threads >> >> > potentially have different activation rates. >> >> > >> >> > Finally, the NUMA balancing scan algorithm is sub-optimal. It >> >> > potentially >> >> > scans the entire address space even though only a small number of pages >> >> > are scanned. This is particularly problematic when a process has a lot >> >> > of threads because threads are redundantly scanning the same regions. If >> >> > NUMA balancing ever introduced range tracking of faulted pages to limit >> >> > how much scanning it has to do, it would inadvertently cause a change in >> >> > page activation rate. >> >> > >> >> > NUMA balancing is about page locality, it should not get conflated with >> >> > page aging. >> >> >> >> I understand your concerns about binding the NUMA balancing and page >> >> reclaiming. The requirement of the page locality and page aging is >> >> different, so the policies need to be different. This is the wrong part >> >> of the patch. >> >> >> >> From another point of view, it's still possible to share some underlying >> >> mechanisms (and code) between them. That is, scanning the page tables >> >> to make pages unaccessible and capture the page accesses via the page >> >> fault. >> > >> > Potentially yes but not necessarily recommended for page aging. NUMA >> > balancing has to be careful about the rate it scans pages to avoid >> > excessive overhead so it's driven by locality. The scanning happens >> > within a tasks context so during that time, the task is not executing >> > its normal work and it incurs the overhead for faults. Generally, this >> > is not too much overhead because pages get migrated locally, the scan >> > rate drops and so does the overhead. >> > >> > However, if you want to drive page aging, that is constant so the rate >> > could not be easily adapted in a way that would be deterministic. >> > >> >> Now these page accessing information is used for the page >> >> locality. Do you think it's a good idea to use these information for >> >> the page aging too (but with a different policy as you pointed out)? >> >> >> > >> > I'm not completely opposed to it but I think the overhead it would >> > introduce could be severe. Worse, if a workload fits in memory and there >> > is limited to no memory pressure, it's all overhead for no gain. Early >> > generations of NUMA balancing had to find a balance to sure the gains >> > from locality exceeded the cost of measuring locality and doing the same >> > for page aging in some ways is even more challenging. >> >> Yes. I will think more about it from the overhead vs. gain point of >> view. Thanks a lot for your sharing on that. >> >> >> From yet another point of view :-), in current NUMA balancing >> >> implementation, it's assumed that the node private pages can fit in the >> >> accessing node. But this may be not always true. Is it a valid >> >> optimization to migrate the hot private pages first? >> >> >> > >> > I'm not sure how the hotness of pages could be ranked. At the time of a >> > hinting fault, the page is by definition active now because it was been >> > accessed. Prioritising what pages to migrate based on the number of faults >> > that have been trapped would have to be stored somewhere. >> >> Yes. We need to store some information about that. In an old version >> of the patchset which uses NUMA balancing to promote hot pages from the >> PMEM to DRAM, we have designed a method to measure the hotness of the >> pages. The basic idea is as follows, >> >> - When the page table of a process is scanned, the latest N scanning >> address ranges and scan times are recorded in a ring buffer of >> mm_struct. >> >> - In hint page fault handler,
Re: [RFC] mm: activate access-more-than-once page via NUMA balancing
On Fri, Mar 26, 2021 at 12:21 AM Huang, Ying wrote: > > Mel Gorman writes: > > > On Thu, Mar 25, 2021 at 12:33:45PM +0800, Huang, Ying wrote: > >> > I caution against this patch. > >> > > >> > It's non-deterministic for a number of reasons. As it requires NUMA > >> > balancing to be enabled, the pageout behaviour of a system changes when > >> > NUMA balancing is active. If this led to pages being artificially and > >> > inappropriately preserved, NUMA balancing could be disabled for the > >> > wrong reasons. It only applies to pages that have no target node so > >> > memory policies affect which pages are activated differently. Similarly, > >> > NUMA balancing does not scan all VMAs and some pages may never trap a > >> > NUMA fault as a result. The timing of when an address space gets scanned > >> > is driven by the locality of pages and so the timing of page activation > >> > potentially becomes linked to whether pages are local or need to migrate > >> > (although not right now for this patch as it only affects pages with a > >> > target nid of NUMA_NO_NODE). In other words, changes in NUMA balancing > >> > that affect migration potentially affect the aging rate. Similarly, > >> > the activate rate of a process with a single thread and multiple threads > >> > potentially have different activation rates. > >> > > >> > Finally, the NUMA balancing scan algorithm is sub-optimal. It potentially > >> > scans the entire address space even though only a small number of pages > >> > are scanned. This is particularly problematic when a process has a lot > >> > of threads because threads are redundantly scanning the same regions. If > >> > NUMA balancing ever introduced range tracking of faulted pages to limit > >> > how much scanning it has to do, it would inadvertently cause a change in > >> > page activation rate. > >> > > >> > NUMA balancing is about page locality, it should not get conflated with > >> > page aging. > >> > >> I understand your concerns about binding the NUMA balancing and page > >> reclaiming. The requirement of the page locality and page aging is > >> different, so the policies need to be different. This is the wrong part > >> of the patch. > >> > >> From another point of view, it's still possible to share some underlying > >> mechanisms (and code) between them. That is, scanning the page tables > >> to make pages unaccessible and capture the page accesses via the page > >> fault. > > > > Potentially yes but not necessarily recommended for page aging. NUMA > > balancing has to be careful about the rate it scans pages to avoid > > excessive overhead so it's driven by locality. The scanning happens > > within a tasks context so during that time, the task is not executing > > its normal work and it incurs the overhead for faults. Generally, this > > is not too much overhead because pages get migrated locally, the scan > > rate drops and so does the overhead. > > > > However, if you want to drive page aging, that is constant so the rate > > could not be easily adapted in a way that would be deterministic. > > > >> Now these page accessing information is used for the page > >> locality. Do you think it's a good idea to use these information for > >> the page aging too (but with a different policy as you pointed out)? > >> > > > > I'm not completely opposed to it but I think the overhead it would > > introduce could be severe. Worse, if a workload fits in memory and there > > is limited to no memory pressure, it's all overhead for no gain. Early > > generations of NUMA balancing had to find a balance to sure the gains > > from locality exceeded the cost of measuring locality and doing the same > > for page aging in some ways is even more challenging. > > Yes. I will think more about it from the overhead vs. gain point of > view. Thanks a lot for your sharing on that. > > >> From yet another point of view :-), in current NUMA balancing > >> implementation, it's assumed that the node private pages can fit in the > >> accessing node. But this may be not always true. Is it a valid > >> optimization to migrate the hot private pages first? > >> > > > > I'm not sure how the hotness of pages could be ranked. At the time of a > > hinting fault, the page is by definition active now because it was been > > accessed. Prioritising what pages to migrate based on the number of faults > > that have been trapped would have to be stored somewhere. > > Yes. We need to store some information about that. In an old version > of the patchset which uses NUMA balancing to promote hot pages from the > PMEM to DRAM, we have designed a method to measure the hotness of the > pages. The basic idea is as follows, > > - When the page table of a process is scanned, the latest N scanning > address ranges and scan times are recorded in a ring buffer of > mm_struct. > > - In hint page fault handler, the ring buffer is search with the fault > address, to get the scan time. > > Then the hint page fault latency of
Re: [RFC] mm: activate access-more-than-once page via NUMA balancing
Mel Gorman writes: > On Thu, Mar 25, 2021 at 12:33:45PM +0800, Huang, Ying wrote: >> > I caution against this patch. >> > >> > It's non-deterministic for a number of reasons. As it requires NUMA >> > balancing to be enabled, the pageout behaviour of a system changes when >> > NUMA balancing is active. If this led to pages being artificially and >> > inappropriately preserved, NUMA balancing could be disabled for the >> > wrong reasons. It only applies to pages that have no target node so >> > memory policies affect which pages are activated differently. Similarly, >> > NUMA balancing does not scan all VMAs and some pages may never trap a >> > NUMA fault as a result. The timing of when an address space gets scanned >> > is driven by the locality of pages and so the timing of page activation >> > potentially becomes linked to whether pages are local or need to migrate >> > (although not right now for this patch as it only affects pages with a >> > target nid of NUMA_NO_NODE). In other words, changes in NUMA balancing >> > that affect migration potentially affect the aging rate. Similarly, >> > the activate rate of a process with a single thread and multiple threads >> > potentially have different activation rates. >> > >> > Finally, the NUMA balancing scan algorithm is sub-optimal. It potentially >> > scans the entire address space even though only a small number of pages >> > are scanned. This is particularly problematic when a process has a lot >> > of threads because threads are redundantly scanning the same regions. If >> > NUMA balancing ever introduced range tracking of faulted pages to limit >> > how much scanning it has to do, it would inadvertently cause a change in >> > page activation rate. >> > >> > NUMA balancing is about page locality, it should not get conflated with >> > page aging. >> >> I understand your concerns about binding the NUMA balancing and page >> reclaiming. The requirement of the page locality and page aging is >> different, so the policies need to be different. This is the wrong part >> of the patch. >> >> From another point of view, it's still possible to share some underlying >> mechanisms (and code) between them. That is, scanning the page tables >> to make pages unaccessible and capture the page accesses via the page >> fault. > > Potentially yes but not necessarily recommended for page aging. NUMA > balancing has to be careful about the rate it scans pages to avoid > excessive overhead so it's driven by locality. The scanning happens > within a tasks context so during that time, the task is not executing > its normal work and it incurs the overhead for faults. Generally, this > is not too much overhead because pages get migrated locally, the scan > rate drops and so does the overhead. > > However, if you want to drive page aging, that is constant so the rate > could not be easily adapted in a way that would be deterministic. > >> Now these page accessing information is used for the page >> locality. Do you think it's a good idea to use these information for >> the page aging too (but with a different policy as you pointed out)? >> > > I'm not completely opposed to it but I think the overhead it would > introduce could be severe. Worse, if a workload fits in memory and there > is limited to no memory pressure, it's all overhead for no gain. Early > generations of NUMA balancing had to find a balance to sure the gains > from locality exceeded the cost of measuring locality and doing the same > for page aging in some ways is even more challenging. Yes. I will think more about it from the overhead vs. gain point of view. Thanks a lot for your sharing on that. >> From yet another point of view :-), in current NUMA balancing >> implementation, it's assumed that the node private pages can fit in the >> accessing node. But this may be not always true. Is it a valid >> optimization to migrate the hot private pages first? >> > > I'm not sure how the hotness of pages could be ranked. At the time of a > hinting fault, the page is by definition active now because it was been > accessed. Prioritising what pages to migrate based on the number of faults > that have been trapped would have to be stored somewhere. Yes. We need to store some information about that. In an old version of the patchset which uses NUMA balancing to promote hot pages from the PMEM to DRAM, we have designed a method to measure the hotness of the pages. The basic idea is as follows, - When the page table of a process is scanned, the latest N scanning address ranges and scan times are recorded in a ring buffer of mm_struct. - In hint page fault handler, the ring buffer is search with the fault address, to get the scan time. Then the hint page fault latency of the page is defined as, hint page fault latency = fault time - scan time The shorter the hint page fault latency, the hotter the page. Then we need a way to determine the hot/cold threshold. We used a rate limit based
Re: [RFC] mm: activate access-more-than-once page via NUMA balancing
On Thu, Mar 25, 2021 at 12:33:45PM +0800, Huang, Ying wrote: > > I caution against this patch. > > > > It's non-deterministic for a number of reasons. As it requires NUMA > > balancing to be enabled, the pageout behaviour of a system changes when > > NUMA balancing is active. If this led to pages being artificially and > > inappropriately preserved, NUMA balancing could be disabled for the > > wrong reasons. It only applies to pages that have no target node so > > memory policies affect which pages are activated differently. Similarly, > > NUMA balancing does not scan all VMAs and some pages may never trap a > > NUMA fault as a result. The timing of when an address space gets scanned > > is driven by the locality of pages and so the timing of page activation > > potentially becomes linked to whether pages are local or need to migrate > > (although not right now for this patch as it only affects pages with a > > target nid of NUMA_NO_NODE). In other words, changes in NUMA balancing > > that affect migration potentially affect the aging rate. Similarly, > > the activate rate of a process with a single thread and multiple threads > > potentially have different activation rates. > > > > Finally, the NUMA balancing scan algorithm is sub-optimal. It potentially > > scans the entire address space even though only a small number of pages > > are scanned. This is particularly problematic when a process has a lot > > of threads because threads are redundantly scanning the same regions. If > > NUMA balancing ever introduced range tracking of faulted pages to limit > > how much scanning it has to do, it would inadvertently cause a change in > > page activation rate. > > > > NUMA balancing is about page locality, it should not get conflated with > > page aging. > > I understand your concerns about binding the NUMA balancing and page > reclaiming. The requirement of the page locality and page aging is > different, so the policies need to be different. This is the wrong part > of the patch. > > From another point of view, it's still possible to share some underlying > mechanisms (and code) between them. That is, scanning the page tables > to make pages unaccessible and capture the page accesses via the page > fault. Potentially yes but not necessarily recommended for page aging. NUMA balancing has to be careful about the rate it scans pages to avoid excessive overhead so it's driven by locality. The scanning happens within a tasks context so during that time, the task is not executing its normal work and it incurs the overhead for faults. Generally, this is not too much overhead because pages get migrated locally, the scan rate drops and so does the overhead. However, if you want to drive page aging, that is constant so the rate could not be easily adapted in a way that would be deterministic. > Now these page accessing information is used for the page > locality. Do you think it's a good idea to use these information for > the page aging too (but with a different policy as you pointed out)? > I'm not completely opposed to it but I think the overhead it would introduce could be severe. Worse, if a workload fits in memory and there is limited to no memory pressure, it's all overhead for no gain. Early generations of NUMA balancing had to find a balance to sure the gains from locality exceeded the cost of measuring locality and doing the same for page aging in some ways is even more challenging. > From yet another point of view :-), in current NUMA balancing > implementation, it's assumed that the node private pages can fit in the > accessing node. But this may be not always true. Is it a valid > optimization to migrate the hot private pages first? > I'm not sure how the hotness of pages could be ranked. At the time of a hinting fault, the page is by definition active now because it was been accessed. Prioritising what pages to migrate based on the number of faults that have been trapped would have to be stored somewhere. -- Mel Gorman SUSE Labs
Re: [RFC] mm: activate access-more-than-once page via NUMA balancing
Hi, Mel, Thanks for comment! Mel Gorman writes: > On Wed, Mar 24, 2021 at 04:32:09PM +0800, Huang Ying wrote: >> One idea behind the LRU page reclaiming algorithm is to put the >> access-once pages in the inactive list and access-more-than-once pages >> in the active list. This is true for the file pages that are accessed >> via syscall (read()/write(), etc.), but not for the pages accessed via >> the page tables. We can only activate them via page reclaim scanning >> now. This may cause some problems. For example, even if there are >> only hot file pages accessed via the page tables in the inactive list, >> we will enable the cache trim mode incorrectly to scan only the hot >> file pages instead of cold anon pages. >> > > I caution against this patch. > > It's non-deterministic for a number of reasons. As it requires NUMA > balancing to be enabled, the pageout behaviour of a system changes when > NUMA balancing is active. If this led to pages being artificially and > inappropriately preserved, NUMA balancing could be disabled for the > wrong reasons. It only applies to pages that have no target node so > memory policies affect which pages are activated differently. Similarly, > NUMA balancing does not scan all VMAs and some pages may never trap a > NUMA fault as a result. The timing of when an address space gets scanned > is driven by the locality of pages and so the timing of page activation > potentially becomes linked to whether pages are local or need to migrate > (although not right now for this patch as it only affects pages with a > target nid of NUMA_NO_NODE). In other words, changes in NUMA balancing > that affect migration potentially affect the aging rate. Similarly, > the activate rate of a process with a single thread and multiple threads > potentially have different activation rates. > > Finally, the NUMA balancing scan algorithm is sub-optimal. It potentially > scans the entire address space even though only a small number of pages > are scanned. This is particularly problematic when a process has a lot > of threads because threads are redundantly scanning the same regions. If > NUMA balancing ever introduced range tracking of faulted pages to limit > how much scanning it has to do, it would inadvertently cause a change in > page activation rate. > > NUMA balancing is about page locality, it should not get conflated with > page aging. I understand your concerns about binding the NUMA balancing and page reclaiming. The requirement of the page locality and page aging is different, so the policies need to be different. This is the wrong part of the patch. >From another point of view, it's still possible to share some underlying mechanisms (and code) between them. That is, scanning the page tables to make pages unaccessible and capture the page accesses via the page fault. Now these page accessing information is used for the page locality. Do you think it's a good idea to use these information for the page aging too (but with a different policy as you pointed out)? >From yet another point of view :-), in current NUMA balancing implementation, it's assumed that the node private pages can fit in the accessing node. But this may be not always true. Is it a valid optimization to migrate the hot private pages first? Best Regards, Huang, Ying
Re: [RFC] mm: activate access-more-than-once page via NUMA balancing
On Wed, Mar 24, 2021 at 04:32:09PM +0800, Huang Ying wrote: > One idea behind the LRU page reclaiming algorithm is to put the > access-once pages in the inactive list and access-more-than-once pages > in the active list. This is true for the file pages that are accessed > via syscall (read()/write(), etc.), but not for the pages accessed via > the page tables. We can only activate them via page reclaim scanning > now. This may cause some problems. For example, even if there are > only hot file pages accessed via the page tables in the inactive list, > we will enable the cache trim mode incorrectly to scan only the hot > file pages instead of cold anon pages. > I caution against this patch. It's non-deterministic for a number of reasons. As it requires NUMA balancing to be enabled, the pageout behaviour of a system changes when NUMA balancing is active. If this led to pages being artificially and inappropriately preserved, NUMA balancing could be disabled for the wrong reasons. It only applies to pages that have no target node so memory policies affect which pages are activated differently. Similarly, NUMA balancing does not scan all VMAs and some pages may never trap a NUMA fault as a result. The timing of when an address space gets scanned is driven by the locality of pages and so the timing of page activation potentially becomes linked to whether pages are local or need to migrate (although not right now for this patch as it only affects pages with a target nid of NUMA_NO_NODE). In other words, changes in NUMA balancing that affect migration potentially affect the aging rate. Similarly, the activate rate of a process with a single thread and multiple threads potentially have different activation rates. Finally, the NUMA balancing scan algorithm is sub-optimal. It potentially scans the entire address space even though only a small number of pages are scanned. This is particularly problematic when a process has a lot of threads because threads are redundantly scanning the same regions. If NUMA balancing ever introduced range tracking of faulted pages to limit how much scanning it has to do, it would inadvertently cause a change in page activation rate. NUMA balancing is about page locality, it should not get conflated with page aging. -- Mel Gorman SUSE Labs
