Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: > OK. We will try to improve on this side. But it is not an easy task > for us to provided easy to use simple binaries. Do you think something > like Docker image is easy to use? Docker muck would just make me turn around and run.
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: > OK. We will try to improve on this side. But it is not an easy task > for us to provided easy to use simple binaries. Do you think something > like Docker image is easy to use? Docker muck would just make me turn around and run.
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Darren Hartwrites: > On Tue, Mar 29, 2016 at 09:12:56AM +0800, Huang, Ying wrote: >> Darren Hart writes: >> >> > On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: >> >> Thomas Gleixner writes: >> >> >> >> > On Mon, 21 Mar 2016, Huang, Ying wrote: >> >> >> > FYI, we noticed 25.6% performance improvement due to commit >> >> >> > >> >> >> >65d8fc777f6d "futex: Remove requirement for lock_page() in >> >> >> > get_futex_key()" >> >> >> > >> >> >> > in the will-it-scale.per_process_ops test. >> >> >> > >> >> >> > will-it-scale.per_process_ops tests the futex operations for >> >> >> > process shared >> >> >> > futexes (Or whatever that test really does). >> >> >> >> >> >> There is a futex sub test case for will-it-scale test suite. But I >> >> >> got your >> >> >> point, we need some description for the test case. If email is not too >> >> >> limited for the full description, we will put it in some web site and >> >> >> include short description and link to the full description in email. >> >> > >> >> > Ok. Just make sure the short description gives enough information for >> >> > the >> >> > casual reader. >> >> > >> >> >> > The commit has no significant impact on any other test in the test >> >> >> > suite. >> >> >> >> >> >> Sorry, we have no enough machine power to test all test cases for each >> >> >> bisect result. So we will have no such information until we find a way >> >> >> to do that. >> >> > >> >> > Well, then I really have to ask how I should interpret the data here: >> >> > >> >> >5076304 . 0% +25.6%6374220 . 0% >> >> > will-it-scale.per_process_ops >> >> > >> >> >^^^ That's the reason why you sent the mail in the first place >> >> > >> >> >1194117 . 0% +14.4%1366153 . 1% >> >> > will-it-scale.per_thread_ops >> >> > 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability >> >> > 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages >> >> > 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs >> >> > 2848 . 32%+141.2% 6870 . 65% >> >> > numa-meminfo.node1.Active(anon) >> >> > 2832 . 31% +57.6% 4465 . 27% >> >> > numa-meminfo.node1.AnonPages >> >> > 15018 . 12% -23.3% 11515 . 15% >> >> > numa-meminfo.node2.AnonPages >> >> > 1214 . 14% -22.8% 936.75 . 20% >> >> > numa-meminfo.node3.PageTables >> >> > 712.25 . 32%+141.2% 1718 . 65% >> >> > numa-vmstat.node1.nr_active_anon >> >> > 708.25 . 31% +57.7% 1116 . 27% >> >> > numa-vmstat.node1.nr_anon_pages >> >> > >> >> > How is this related and what should I do about this information? >> >> >> >> For each will-it-scale sub test case, it will be run in both process >> >> mode and thread mode, and task number will change from 1 to CPU number. >> >> will-it-scale.per_thread_ops shows thread mode main result. >> >> will-it-scale.scalability is calculated to measure how per_process_ops >> >> and per_thread_ops scaled along with the task number. These are default >> >> behavior of will-it-scale test suite. >> >> >> >> Others are monitors output. That is, other information collected during >> >> test. For example, meminfo is a monitor to sampling /proc/meminfo >> >> contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for >> >> the average value of AnonHugePages line of /proc/meminfo. Similarly >> >> vmstat.system.cs is the average value of cs column of system column >> >> group of /usr/bin/vmstat. >> >> >> >> We hope these information are helpful for root causing the regression. >> >> >> >> > If it's important then I have to admit, that I fail to understand why. >> >> > >> >> > If it's not important then I have to ask why is this included. >> >> > >> >> >> > So that allows me to reproduce that test more or less with no >> >> >> > effort. And >> >> >> > that's the really important part. >> >> >> >> >> >> For reproducing, now we use lkp-tests tool, which includes scripts to >> >> >> build the test case, run the test, collect various information, compare >> >> >> the test result, with the job file attached with the report email. >> >> >> That >> >> >> is not the easiest way, we will continuously improve it. >> >> > >> >> > I know and lkp-tests is a pain to work with. So please look into a way >> >> > to >> >> > extract the relevant binaries, so it's simple for developers to >> >> > reproduce. >> >> >> >> OK. We will try to improve on this side. But it is not an easy task >> >> for us to provided easy to use simple binaries. Do you think something >> >> like Docker image is easy to use? >> > >> > Thomas, I presume you are interested in binaries to be positive we're >> > reproducing with exactly the same bits? I agree that's good to have. I'd >> > also >> > want to have or be pointed to the sources with a straight forward way to >> > rebuild and to inspect
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Darren Hart writes: > On Tue, Mar 29, 2016 at 09:12:56AM +0800, Huang, Ying wrote: >> Darren Hart writes: >> >> > On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: >> >> Thomas Gleixner writes: >> >> >> >> > On Mon, 21 Mar 2016, Huang, Ying wrote: >> >> >> > FYI, we noticed 25.6% performance improvement due to commit >> >> >> > >> >> >> >65d8fc777f6d "futex: Remove requirement for lock_page() in >> >> >> > get_futex_key()" >> >> >> > >> >> >> > in the will-it-scale.per_process_ops test. >> >> >> > >> >> >> > will-it-scale.per_process_ops tests the futex operations for >> >> >> > process shared >> >> >> > futexes (Or whatever that test really does). >> >> >> >> >> >> There is a futex sub test case for will-it-scale test suite. But I >> >> >> got your >> >> >> point, we need some description for the test case. If email is not too >> >> >> limited for the full description, we will put it in some web site and >> >> >> include short description and link to the full description in email. >> >> > >> >> > Ok. Just make sure the short description gives enough information for >> >> > the >> >> > casual reader. >> >> > >> >> >> > The commit has no significant impact on any other test in the test >> >> >> > suite. >> >> >> >> >> >> Sorry, we have no enough machine power to test all test cases for each >> >> >> bisect result. So we will have no such information until we find a way >> >> >> to do that. >> >> > >> >> > Well, then I really have to ask how I should interpret the data here: >> >> > >> >> >5076304 . 0% +25.6%6374220 . 0% >> >> > will-it-scale.per_process_ops >> >> > >> >> >^^^ That's the reason why you sent the mail in the first place >> >> > >> >> >1194117 . 0% +14.4%1366153 . 1% >> >> > will-it-scale.per_thread_ops >> >> > 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability >> >> > 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages >> >> > 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs >> >> > 2848 . 32%+141.2% 6870 . 65% >> >> > numa-meminfo.node1.Active(anon) >> >> > 2832 . 31% +57.6% 4465 . 27% >> >> > numa-meminfo.node1.AnonPages >> >> > 15018 . 12% -23.3% 11515 . 15% >> >> > numa-meminfo.node2.AnonPages >> >> > 1214 . 14% -22.8% 936.75 . 20% >> >> > numa-meminfo.node3.PageTables >> >> > 712.25 . 32%+141.2% 1718 . 65% >> >> > numa-vmstat.node1.nr_active_anon >> >> > 708.25 . 31% +57.7% 1116 . 27% >> >> > numa-vmstat.node1.nr_anon_pages >> >> > >> >> > How is this related and what should I do about this information? >> >> >> >> For each will-it-scale sub test case, it will be run in both process >> >> mode and thread mode, and task number will change from 1 to CPU number. >> >> will-it-scale.per_thread_ops shows thread mode main result. >> >> will-it-scale.scalability is calculated to measure how per_process_ops >> >> and per_thread_ops scaled along with the task number. These are default >> >> behavior of will-it-scale test suite. >> >> >> >> Others are monitors output. That is, other information collected during >> >> test. For example, meminfo is a monitor to sampling /proc/meminfo >> >> contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for >> >> the average value of AnonHugePages line of /proc/meminfo. Similarly >> >> vmstat.system.cs is the average value of cs column of system column >> >> group of /usr/bin/vmstat. >> >> >> >> We hope these information are helpful for root causing the regression. >> >> >> >> > If it's important then I have to admit, that I fail to understand why. >> >> > >> >> > If it's not important then I have to ask why is this included. >> >> > >> >> >> > So that allows me to reproduce that test more or less with no >> >> >> > effort. And >> >> >> > that's the really important part. >> >> >> >> >> >> For reproducing, now we use lkp-tests tool, which includes scripts to >> >> >> build the test case, run the test, collect various information, compare >> >> >> the test result, with the job file attached with the report email. >> >> >> That >> >> >> is not the easiest way, we will continuously improve it. >> >> > >> >> > I know and lkp-tests is a pain to work with. So please look into a way >> >> > to >> >> > extract the relevant binaries, so it's simple for developers to >> >> > reproduce. >> >> >> >> OK. We will try to improve on this side. But it is not an easy task >> >> for us to provided easy to use simple binaries. Do you think something >> >> like Docker image is easy to use? >> > >> > Thomas, I presume you are interested in binaries to be positive we're >> > reproducing with exactly the same bits? I agree that's good to have. I'd >> > also >> > want to have or be pointed to the sources with a straight forward way to >> > rebuild and to inspect what exactly the test is doing. (I assume this is >> > implied, but
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Tue, Mar 29, 2016 at 09:12:56AM +0800, Huang, Ying wrote: > Darren Hartwrites: > > > On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: > >> Thomas Gleixner writes: > >> > >> > On Mon, 21 Mar 2016, Huang, Ying wrote: > >> >> > FYI, we noticed 25.6% performance improvement due to commit > >> >> > > >> >> >65d8fc777f6d "futex: Remove requirement for lock_page() in > >> >> > get_futex_key()" > >> >> > > >> >> > in the will-it-scale.per_process_ops test. > >> >> > > >> >> > will-it-scale.per_process_ops tests the futex operations for process > >> >> > shared > >> >> > futexes (Or whatever that test really does). > >> >> > >> >> There is a futex sub test case for will-it-scale test suite. But I got > >> >> your > >> >> point, we need some description for the test case. If email is not too > >> >> limited for the full description, we will put it in some web site and > >> >> include short description and link to the full description in email. > >> > > >> > Ok. Just make sure the short description gives enough information for the > >> > casual reader. > >> > > >> >> > The commit has no significant impact on any other test in the test > >> >> > suite. > >> >> > >> >> Sorry, we have no enough machine power to test all test cases for each > >> >> bisect result. So we will have no such information until we find a way > >> >> to do that. > >> > > >> > Well, then I really have to ask how I should interpret the data here: > >> > > >> >5076304 . 0% +25.6%6374220 . 0% > >> > will-it-scale.per_process_ops > >> > > >> >^^^ That's the reason why you sent the mail in the first place > >> > > >> >1194117 . 0% +14.4%1366153 . 1% > >> > will-it-scale.per_thread_ops > >> > 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability > >> > 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages > >> > 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs > >> > 2848 . 32%+141.2% 6870 . 65% > >> > numa-meminfo.node1.Active(anon) > >> > 2832 . 31% +57.6% 4465 . 27% > >> > numa-meminfo.node1.AnonPages > >> > 15018 . 12% -23.3% 11515 . 15% > >> > numa-meminfo.node2.AnonPages > >> > 1214 . 14% -22.8% 936.75 . 20% > >> > numa-meminfo.node3.PageTables > >> > 712.25 . 32%+141.2% 1718 . 65% > >> > numa-vmstat.node1.nr_active_anon > >> > 708.25 . 31% +57.7% 1116 . 27% > >> > numa-vmstat.node1.nr_anon_pages > >> > > >> > How is this related and what should I do about this information? > >> > >> For each will-it-scale sub test case, it will be run in both process > >> mode and thread mode, and task number will change from 1 to CPU number. > >> will-it-scale.per_thread_ops shows thread mode main result. > >> will-it-scale.scalability is calculated to measure how per_process_ops > >> and per_thread_ops scaled along with the task number. These are default > >> behavior of will-it-scale test suite. > >> > >> Others are monitors output. That is, other information collected during > >> test. For example, meminfo is a monitor to sampling /proc/meminfo > >> contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for > >> the average value of AnonHugePages line of /proc/meminfo. Similarly > >> vmstat.system.cs is the average value of cs column of system column > >> group of /usr/bin/vmstat. > >> > >> We hope these information are helpful for root causing the regression. > >> > >> > If it's important then I have to admit, that I fail to understand why. > >> > > >> > If it's not important then I have to ask why is this included. > >> > > >> >> > So that allows me to reproduce that test more or less with no effort. > >> >> > And > >> >> > that's the really important part. > >> >> > >> >> For reproducing, now we use lkp-tests tool, which includes scripts to > >> >> build the test case, run the test, collect various information, compare > >> >> the test result, with the job file attached with the report email. That > >> >> is not the easiest way, we will continuously improve it. > >> > > >> > I know and lkp-tests is a pain to work with. So please look into a way to > >> > extract the relevant binaries, so it's simple for developers to > >> > reproduce. > >> > >> OK. We will try to improve on this side. But it is not an easy task > >> for us to provided easy to use simple binaries. Do you think something > >> like Docker image is easy to use? > > > > Thomas, I presume you are interested in binaries to be positive we're > > reproducing with exactly the same bits? I agree that's good to have. I'd > > also > > want to have or be pointed to the sources with a straight forward way to > > rebuild and to inspect what exactly the test is doing. (I assume this is > > implied, but just to make sure it's stated). > > lkp-tests has the scripts to download the source, apply some patch, and >
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Tue, Mar 29, 2016 at 09:12:56AM +0800, Huang, Ying wrote: > Darren Hart writes: > > > On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: > >> Thomas Gleixner writes: > >> > >> > On Mon, 21 Mar 2016, Huang, Ying wrote: > >> >> > FYI, we noticed 25.6% performance improvement due to commit > >> >> > > >> >> >65d8fc777f6d "futex: Remove requirement for lock_page() in > >> >> > get_futex_key()" > >> >> > > >> >> > in the will-it-scale.per_process_ops test. > >> >> > > >> >> > will-it-scale.per_process_ops tests the futex operations for process > >> >> > shared > >> >> > futexes (Or whatever that test really does). > >> >> > >> >> There is a futex sub test case for will-it-scale test suite. But I got > >> >> your > >> >> point, we need some description for the test case. If email is not too > >> >> limited for the full description, we will put it in some web site and > >> >> include short description and link to the full description in email. > >> > > >> > Ok. Just make sure the short description gives enough information for the > >> > casual reader. > >> > > >> >> > The commit has no significant impact on any other test in the test > >> >> > suite. > >> >> > >> >> Sorry, we have no enough machine power to test all test cases for each > >> >> bisect result. So we will have no such information until we find a way > >> >> to do that. > >> > > >> > Well, then I really have to ask how I should interpret the data here: > >> > > >> >5076304 . 0% +25.6%6374220 . 0% > >> > will-it-scale.per_process_ops > >> > > >> >^^^ That's the reason why you sent the mail in the first place > >> > > >> >1194117 . 0% +14.4%1366153 . 1% > >> > will-it-scale.per_thread_ops > >> > 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability > >> > 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages > >> > 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs > >> > 2848 . 32%+141.2% 6870 . 65% > >> > numa-meminfo.node1.Active(anon) > >> > 2832 . 31% +57.6% 4465 . 27% > >> > numa-meminfo.node1.AnonPages > >> > 15018 . 12% -23.3% 11515 . 15% > >> > numa-meminfo.node2.AnonPages > >> > 1214 . 14% -22.8% 936.75 . 20% > >> > numa-meminfo.node3.PageTables > >> > 712.25 . 32%+141.2% 1718 . 65% > >> > numa-vmstat.node1.nr_active_anon > >> > 708.25 . 31% +57.7% 1116 . 27% > >> > numa-vmstat.node1.nr_anon_pages > >> > > >> > How is this related and what should I do about this information? > >> > >> For each will-it-scale sub test case, it will be run in both process > >> mode and thread mode, and task number will change from 1 to CPU number. > >> will-it-scale.per_thread_ops shows thread mode main result. > >> will-it-scale.scalability is calculated to measure how per_process_ops > >> and per_thread_ops scaled along with the task number. These are default > >> behavior of will-it-scale test suite. > >> > >> Others are monitors output. That is, other information collected during > >> test. For example, meminfo is a monitor to sampling /proc/meminfo > >> contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for > >> the average value of AnonHugePages line of /proc/meminfo. Similarly > >> vmstat.system.cs is the average value of cs column of system column > >> group of /usr/bin/vmstat. > >> > >> We hope these information are helpful for root causing the regression. > >> > >> > If it's important then I have to admit, that I fail to understand why. > >> > > >> > If it's not important then I have to ask why is this included. > >> > > >> >> > So that allows me to reproduce that test more or less with no effort. > >> >> > And > >> >> > that's the really important part. > >> >> > >> >> For reproducing, now we use lkp-tests tool, which includes scripts to > >> >> build the test case, run the test, collect various information, compare > >> >> the test result, with the job file attached with the report email. That > >> >> is not the easiest way, we will continuously improve it. > >> > > >> > I know and lkp-tests is a pain to work with. So please look into a way to > >> > extract the relevant binaries, so it's simple for developers to > >> > reproduce. > >> > >> OK. We will try to improve on this side. But it is not an easy task > >> for us to provided easy to use simple binaries. Do you think something > >> like Docker image is easy to use? > > > > Thomas, I presume you are interested in binaries to be positive we're > > reproducing with exactly the same bits? I agree that's good to have. I'd > > also > > want to have or be pointed to the sources with a straight forward way to > > rebuild and to inspect what exactly the test is doing. (I assume this is > > implied, but just to make sure it's stated). > > lkp-tests has the scripts to download the source, apply some patch, and > build the binary. It is not a very
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Darren Hartwrites: > On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: >> Thomas Gleixner writes: >> >> > On Mon, 21 Mar 2016, Huang, Ying wrote: >> >> > FYI, we noticed 25.6% performance improvement due to commit >> >> > >> >> >65d8fc777f6d "futex: Remove requirement for lock_page() in >> >> > get_futex_key()" >> >> > >> >> > in the will-it-scale.per_process_ops test. >> >> > >> >> > will-it-scale.per_process_ops tests the futex operations for process >> >> > shared >> >> > futexes (Or whatever that test really does). >> >> >> >> There is a futex sub test case for will-it-scale test suite. But I got >> >> your >> >> point, we need some description for the test case. If email is not too >> >> limited for the full description, we will put it in some web site and >> >> include short description and link to the full description in email. >> > >> > Ok. Just make sure the short description gives enough information for the >> > casual reader. >> > >> >> > The commit has no significant impact on any other test in the test >> >> > suite. >> >> >> >> Sorry, we have no enough machine power to test all test cases for each >> >> bisect result. So we will have no such information until we find a way >> >> to do that. >> > >> > Well, then I really have to ask how I should interpret the data here: >> > >> >5076304 . 0% +25.6%6374220 . 0% will-it-scale.per_process_ops >> > >> >^^^ That's the reason why you sent the mail in the first place >> > >> >1194117 . 0% +14.4%1366153 . 1% will-it-scale.per_thread_ops >> > 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability >> > 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages >> > 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs >> > 2848 . 32%+141.2% 6870 . 65% >> > numa-meminfo.node1.Active(anon) >> > 2832 . 31% +57.6% 4465 . 27% numa-meminfo.node1.AnonPages >> > 15018 . 12% -23.3% 11515 . 15% numa-meminfo.node2.AnonPages >> > 1214 . 14% -22.8% 936.75 . 20% numa-meminfo.node3.PageTables >> > 712.25 . 32%+141.2% 1718 . 65% >> > numa-vmstat.node1.nr_active_anon >> > 708.25 . 31% +57.7% 1116 . 27% >> > numa-vmstat.node1.nr_anon_pages >> > >> > How is this related and what should I do about this information? >> >> For each will-it-scale sub test case, it will be run in both process >> mode and thread mode, and task number will change from 1 to CPU number. >> will-it-scale.per_thread_ops shows thread mode main result. >> will-it-scale.scalability is calculated to measure how per_process_ops >> and per_thread_ops scaled along with the task number. These are default >> behavior of will-it-scale test suite. >> >> Others are monitors output. That is, other information collected during >> test. For example, meminfo is a monitor to sampling /proc/meminfo >> contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for >> the average value of AnonHugePages line of /proc/meminfo. Similarly >> vmstat.system.cs is the average value of cs column of system column >> group of /usr/bin/vmstat. >> >> We hope these information are helpful for root causing the regression. >> >> > If it's important then I have to admit, that I fail to understand why. >> > >> > If it's not important then I have to ask why is this included. >> > >> >> > So that allows me to reproduce that test more or less with no effort. >> >> > And >> >> > that's the really important part. >> >> >> >> For reproducing, now we use lkp-tests tool, which includes scripts to >> >> build the test case, run the test, collect various information, compare >> >> the test result, with the job file attached with the report email. That >> >> is not the easiest way, we will continuously improve it. >> > >> > I know and lkp-tests is a pain to work with. So please look into a way to >> > extract the relevant binaries, so it's simple for developers to reproduce. >> >> OK. We will try to improve on this side. But it is not an easy task >> for us to provided easy to use simple binaries. Do you think something >> like Docker image is easy to use? > > Thomas, I presume you are interested in binaries to be positive we're > reproducing with exactly the same bits? I agree that's good to have. I'd also > want to have or be pointed to the sources with a straight forward way to > rebuild and to inspect what exactly the test is doing. (I assume this is > implied, but just to make sure it's stated). lkp-tests has the scripts to download the source, apply some patch, and build the binary. It is not a very straight forward way, but the script is quite simple. > Huang, what makes the binaries difficult to package? And how would docker make > that any simpler? The binaries is not difficult to package. But the test is not only benchmark binary. We may do some setup for the
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Darren Hart writes: > On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: >> Thomas Gleixner writes: >> >> > On Mon, 21 Mar 2016, Huang, Ying wrote: >> >> > FYI, we noticed 25.6% performance improvement due to commit >> >> > >> >> >65d8fc777f6d "futex: Remove requirement for lock_page() in >> >> > get_futex_key()" >> >> > >> >> > in the will-it-scale.per_process_ops test. >> >> > >> >> > will-it-scale.per_process_ops tests the futex operations for process >> >> > shared >> >> > futexes (Or whatever that test really does). >> >> >> >> There is a futex sub test case for will-it-scale test suite. But I got >> >> your >> >> point, we need some description for the test case. If email is not too >> >> limited for the full description, we will put it in some web site and >> >> include short description and link to the full description in email. >> > >> > Ok. Just make sure the short description gives enough information for the >> > casual reader. >> > >> >> > The commit has no significant impact on any other test in the test >> >> > suite. >> >> >> >> Sorry, we have no enough machine power to test all test cases for each >> >> bisect result. So we will have no such information until we find a way >> >> to do that. >> > >> > Well, then I really have to ask how I should interpret the data here: >> > >> >5076304 . 0% +25.6%6374220 . 0% will-it-scale.per_process_ops >> > >> >^^^ That's the reason why you sent the mail in the first place >> > >> >1194117 . 0% +14.4%1366153 . 1% will-it-scale.per_thread_ops >> > 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability >> > 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages >> > 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs >> > 2848 . 32%+141.2% 6870 . 65% >> > numa-meminfo.node1.Active(anon) >> > 2832 . 31% +57.6% 4465 . 27% numa-meminfo.node1.AnonPages >> > 15018 . 12% -23.3% 11515 . 15% numa-meminfo.node2.AnonPages >> > 1214 . 14% -22.8% 936.75 . 20% numa-meminfo.node3.PageTables >> > 712.25 . 32%+141.2% 1718 . 65% >> > numa-vmstat.node1.nr_active_anon >> > 708.25 . 31% +57.7% 1116 . 27% >> > numa-vmstat.node1.nr_anon_pages >> > >> > How is this related and what should I do about this information? >> >> For each will-it-scale sub test case, it will be run in both process >> mode and thread mode, and task number will change from 1 to CPU number. >> will-it-scale.per_thread_ops shows thread mode main result. >> will-it-scale.scalability is calculated to measure how per_process_ops >> and per_thread_ops scaled along with the task number. These are default >> behavior of will-it-scale test suite. >> >> Others are monitors output. That is, other information collected during >> test. For example, meminfo is a monitor to sampling /proc/meminfo >> contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for >> the average value of AnonHugePages line of /proc/meminfo. Similarly >> vmstat.system.cs is the average value of cs column of system column >> group of /usr/bin/vmstat. >> >> We hope these information are helpful for root causing the regression. >> >> > If it's important then I have to admit, that I fail to understand why. >> > >> > If it's not important then I have to ask why is this included. >> > >> >> > So that allows me to reproduce that test more or less with no effort. >> >> > And >> >> > that's the really important part. >> >> >> >> For reproducing, now we use lkp-tests tool, which includes scripts to >> >> build the test case, run the test, collect various information, compare >> >> the test result, with the job file attached with the report email. That >> >> is not the easiest way, we will continuously improve it. >> > >> > I know and lkp-tests is a pain to work with. So please look into a way to >> > extract the relevant binaries, so it's simple for developers to reproduce. >> >> OK. We will try to improve on this side. But it is not an easy task >> for us to provided easy to use simple binaries. Do you think something >> like Docker image is easy to use? > > Thomas, I presume you are interested in binaries to be positive we're > reproducing with exactly the same bits? I agree that's good to have. I'd also > want to have or be pointed to the sources with a straight forward way to > rebuild and to inspect what exactly the test is doing. (I assume this is > implied, but just to make sure it's stated). lkp-tests has the scripts to download the source, apply some patch, and build the binary. It is not a very straight forward way, but the script is quite simple. > Huang, what makes the binaries difficult to package? And how would docker make > that any simpler? The binaries is not difficult to package. But the test is not only benchmark binary. We may do some setup for the specific test, for example, change the cpu
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: > Thomas Gleixnerwrites: > > > On Mon, 21 Mar 2016, Huang, Ying wrote: > >> > FYI, we noticed 25.6% performance improvement due to commit > >> > > >> >65d8fc777f6d "futex: Remove requirement for lock_page() in > >> > get_futex_key()" > >> > > >> > in the will-it-scale.per_process_ops test. > >> > > >> > will-it-scale.per_process_ops tests the futex operations for process > >> > shared > >> > futexes (Or whatever that test really does). > >> > >> There is a futex sub test case for will-it-scale test suite. But I got > >> your > >> point, we need some description for the test case. If email is not too > >> limited for the full description, we will put it in some web site and > >> include short description and link to the full description in email. > > > > Ok. Just make sure the short description gives enough information for the > > casual reader. > > > >> > The commit has no significant impact on any other test in the test > >> > suite. > >> > >> Sorry, we have no enough machine power to test all test cases for each > >> bisect result. So we will have no such information until we find a way > >> to do that. > > > > Well, then I really have to ask how I should interpret the data here: > > > >5076304 ± 0% +25.6%6374220 ± 0% will-it-scale.per_process_ops > > > >^^^ That's the reason why you sent the mail in the first place > > > >1194117 ± 0% +14.4%1366153 ± 1% will-it-scale.per_thread_ops > > 0.58 ± 0% -2.0% 0.57 ± 0% will-it-scale.scalability > > 6820 ± 0% -19.6% 5483 ± 15% meminfo.AnonHugePages > > 2652 ± 5% -10.4% 2375 ± 2% vmstat.system.cs > > 2848 ± 32%+141.2% 6870 ± 65% > > numa-meminfo.node1.Active(anon) > > 2832 ± 31% +57.6% 4465 ± 27% numa-meminfo.node1.AnonPages > > 15018 ± 12% -23.3% 11515 ± 15% numa-meminfo.node2.AnonPages > > 1214 ± 14% -22.8% 936.75 ± 20% numa-meminfo.node3.PageTables > > 712.25 ± 32%+141.2% 1718 ± 65% > > numa-vmstat.node1.nr_active_anon > > 708.25 ± 31% +57.7% 1116 ± 27% > > numa-vmstat.node1.nr_anon_pages > > > > How is this related and what should I do about this information? > > For each will-it-scale sub test case, it will be run in both process > mode and thread mode, and task number will change from 1 to CPU number. > will-it-scale.per_thread_ops shows thread mode main result. > will-it-scale.scalability is calculated to measure how per_process_ops > and per_thread_ops scaled along with the task number. These are default > behavior of will-it-scale test suite. > > Others are monitors output. That is, other information collected during > test. For example, meminfo is a monitor to sampling /proc/meminfo > contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for > the average value of AnonHugePages line of /proc/meminfo. Similarly > vmstat.system.cs is the average value of cs column of system column > group of /usr/bin/vmstat. > > We hope these information are helpful for root causing the regression. > > > If it's important then I have to admit, that I fail to understand why. > > > > If it's not important then I have to ask why is this included. > > > >> > So that allows me to reproduce that test more or less with no effort. And > >> > that's the really important part. > >> > >> For reproducing, now we use lkp-tests tool, which includes scripts to > >> build the test case, run the test, collect various information, compare > >> the test result, with the job file attached with the report email. That > >> is not the easiest way, we will continuously improve it. > > > > I know and lkp-tests is a pain to work with. So please look into a way to > > extract the relevant binaries, so it's simple for developers to reproduce. > > OK. We will try to improve on this side. But it is not an easy task > for us to provided easy to use simple binaries. Do you think something > like Docker image is easy to use? Thomas, I presume you are interested in binaries to be positive we're reproducing with exactly the same bits? I agree that's good to have. I'd also want to have or be pointed to the sources with a straight forward way to rebuild and to inspect what exactly the test is doing. (I assume this is implied, but just to make sure it's stated). Huang, what makes the binaries difficult to package? And how would docker make that any simpler? > > >> > You can provide nice charts and full comparison tables for all tests on > >> > a web > >> > site for those who are interested in large stats and pretty charts. > >> > > >> > Full results: http://wherever.you.store/your/results/test-nr/results > >> > >> Before we have a website for detailed information, we will still put > >> some details into report email. > > > > Ok, but please make them understandable for mere mortals.
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Mon, Mar 21, 2016 at 04:42:47PM +0800, Huang, Ying wrote: > Thomas Gleixner writes: > > > On Mon, 21 Mar 2016, Huang, Ying wrote: > >> > FYI, we noticed 25.6% performance improvement due to commit > >> > > >> >65d8fc777f6d "futex: Remove requirement for lock_page() in > >> > get_futex_key()" > >> > > >> > in the will-it-scale.per_process_ops test. > >> > > >> > will-it-scale.per_process_ops tests the futex operations for process > >> > shared > >> > futexes (Or whatever that test really does). > >> > >> There is a futex sub test case for will-it-scale test suite. But I got > >> your > >> point, we need some description for the test case. If email is not too > >> limited for the full description, we will put it in some web site and > >> include short description and link to the full description in email. > > > > Ok. Just make sure the short description gives enough information for the > > casual reader. > > > >> > The commit has no significant impact on any other test in the test > >> > suite. > >> > >> Sorry, we have no enough machine power to test all test cases for each > >> bisect result. So we will have no such information until we find a way > >> to do that. > > > > Well, then I really have to ask how I should interpret the data here: > > > >5076304 ± 0% +25.6%6374220 ± 0% will-it-scale.per_process_ops > > > >^^^ That's the reason why you sent the mail in the first place > > > >1194117 ± 0% +14.4%1366153 ± 1% will-it-scale.per_thread_ops > > 0.58 ± 0% -2.0% 0.57 ± 0% will-it-scale.scalability > > 6820 ± 0% -19.6% 5483 ± 15% meminfo.AnonHugePages > > 2652 ± 5% -10.4% 2375 ± 2% vmstat.system.cs > > 2848 ± 32%+141.2% 6870 ± 65% > > numa-meminfo.node1.Active(anon) > > 2832 ± 31% +57.6% 4465 ± 27% numa-meminfo.node1.AnonPages > > 15018 ± 12% -23.3% 11515 ± 15% numa-meminfo.node2.AnonPages > > 1214 ± 14% -22.8% 936.75 ± 20% numa-meminfo.node3.PageTables > > 712.25 ± 32%+141.2% 1718 ± 65% > > numa-vmstat.node1.nr_active_anon > > 708.25 ± 31% +57.7% 1116 ± 27% > > numa-vmstat.node1.nr_anon_pages > > > > How is this related and what should I do about this information? > > For each will-it-scale sub test case, it will be run in both process > mode and thread mode, and task number will change from 1 to CPU number. > will-it-scale.per_thread_ops shows thread mode main result. > will-it-scale.scalability is calculated to measure how per_process_ops > and per_thread_ops scaled along with the task number. These are default > behavior of will-it-scale test suite. > > Others are monitors output. That is, other information collected during > test. For example, meminfo is a monitor to sampling /proc/meminfo > contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for > the average value of AnonHugePages line of /proc/meminfo. Similarly > vmstat.system.cs is the average value of cs column of system column > group of /usr/bin/vmstat. > > We hope these information are helpful for root causing the regression. > > > If it's important then I have to admit, that I fail to understand why. > > > > If it's not important then I have to ask why is this included. > > > >> > So that allows me to reproduce that test more or less with no effort. And > >> > that's the really important part. > >> > >> For reproducing, now we use lkp-tests tool, which includes scripts to > >> build the test case, run the test, collect various information, compare > >> the test result, with the job file attached with the report email. That > >> is not the easiest way, we will continuously improve it. > > > > I know and lkp-tests is a pain to work with. So please look into a way to > > extract the relevant binaries, so it's simple for developers to reproduce. > > OK. We will try to improve on this side. But it is not an easy task > for us to provided easy to use simple binaries. Do you think something > like Docker image is easy to use? Thomas, I presume you are interested in binaries to be positive we're reproducing with exactly the same bits? I agree that's good to have. I'd also want to have or be pointed to the sources with a straight forward way to rebuild and to inspect what exactly the test is doing. (I assume this is implied, but just to make sure it's stated). Huang, what makes the binaries difficult to package? And how would docker make that any simpler? > > >> > You can provide nice charts and full comparison tables for all tests on > >> > a web > >> > site for those who are interested in large stats and pretty charts. > >> > > >> > Full results: http://wherever.you.store/your/results/test-nr/results > >> > >> Before we have a website for detailed information, we will still put > >> some details into report email. > > > > Ok, but please make them understandable for mere mortals. Thanks for speaking
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Thomas Gleixnerwrites: > On Mon, 21 Mar 2016, Huang, Ying wrote: >> > FYI, we noticed 25.6% performance improvement due to commit >> > >> >65d8fc777f6d "futex: Remove requirement for lock_page() in >> > get_futex_key()" >> > >> > in the will-it-scale.per_process_ops test. >> > >> > will-it-scale.per_process_ops tests the futex operations for process >> > shared >> > futexes (Or whatever that test really does). >> >> There is a futex sub test case for will-it-scale test suite. But I got your >> point, we need some description for the test case. If email is not too >> limited for the full description, we will put it in some web site and >> include short description and link to the full description in email. > > Ok. Just make sure the short description gives enough information for the > casual reader. > >> > The commit has no significant impact on any other test in the test suite. >> >> Sorry, we have no enough machine power to test all test cases for each >> bisect result. So we will have no such information until we find a way >> to do that. > > Well, then I really have to ask how I should interpret the data here: > >5076304 ± 0% +25.6%6374220 ± 0% will-it-scale.per_process_ops > >^^^ That's the reason why you sent the mail in the first place > >1194117 ± 0% +14.4%1366153 ± 1% will-it-scale.per_thread_ops > 0.58 ± 0% -2.0% 0.57 ± 0% will-it-scale.scalability > 6820 ± 0% -19.6% 5483 ± 15% meminfo.AnonHugePages > 2652 ± 5% -10.4% 2375 ± 2% vmstat.system.cs > 2848 ± 32%+141.2% 6870 ± 65% numa-meminfo.node1.Active(anon) > 2832 ± 31% +57.6% 4465 ± 27% numa-meminfo.node1.AnonPages > 15018 ± 12% -23.3% 11515 ± 15% numa-meminfo.node2.AnonPages > 1214 ± 14% -22.8% 936.75 ± 20% numa-meminfo.node3.PageTables > 712.25 ± 32%+141.2% 1718 ± 65% numa-vmstat.node1.nr_active_anon > 708.25 ± 31% +57.7% 1116 ± 27% numa-vmstat.node1.nr_anon_pages > > How is this related and what should I do about this information? For each will-it-scale sub test case, it will be run in both process mode and thread mode, and task number will change from 1 to CPU number. will-it-scale.per_thread_ops shows thread mode main result. will-it-scale.scalability is calculated to measure how per_process_ops and per_thread_ops scaled along with the task number. These are default behavior of will-it-scale test suite. Others are monitors output. That is, other information collected during test. For example, meminfo is a monitor to sampling /proc/meminfo contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for the average value of AnonHugePages line of /proc/meminfo. Similarly vmstat.system.cs is the average value of cs column of system column group of /usr/bin/vmstat. We hope these information are helpful for root causing the regression. > If it's important then I have to admit, that I fail to understand why. > > If it's not important then I have to ask why is this included. > >> > So that allows me to reproduce that test more or less with no effort. And >> > that's the really important part. >> >> For reproducing, now we use lkp-tests tool, which includes scripts to >> build the test case, run the test, collect various information, compare >> the test result, with the job file attached with the report email. That >> is not the easiest way, we will continuously improve it. > > I know and lkp-tests is a pain to work with. So please look into a way to > extract the relevant binaries, so it's simple for developers to reproduce. OK. We will try to improve on this side. But it is not an easy task for us to provided easy to use simple binaries. Do you think something like Docker image is easy to use? >> > You can provide nice charts and full comparison tables for all tests on a >> > web >> > site for those who are interested in large stats and pretty charts. >> > >> > Full results: http://wherever.you.store/your/results/test-nr/results >> >> Before we have a website for detailed information, we will still put >> some details into report email. > > Ok, but please make them understandable for mere mortals. Sure. Best Regards, Huang, Ying > Thanks, > > tglx
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Thomas Gleixner writes: > On Mon, 21 Mar 2016, Huang, Ying wrote: >> > FYI, we noticed 25.6% performance improvement due to commit >> > >> >65d8fc777f6d "futex: Remove requirement for lock_page() in >> > get_futex_key()" >> > >> > in the will-it-scale.per_process_ops test. >> > >> > will-it-scale.per_process_ops tests the futex operations for process >> > shared >> > futexes (Or whatever that test really does). >> >> There is a futex sub test case for will-it-scale test suite. But I got your >> point, we need some description for the test case. If email is not too >> limited for the full description, we will put it in some web site and >> include short description and link to the full description in email. > > Ok. Just make sure the short description gives enough information for the > casual reader. > >> > The commit has no significant impact on any other test in the test suite. >> >> Sorry, we have no enough machine power to test all test cases for each >> bisect result. So we will have no such information until we find a way >> to do that. > > Well, then I really have to ask how I should interpret the data here: > >5076304 ± 0% +25.6%6374220 ± 0% will-it-scale.per_process_ops > >^^^ That's the reason why you sent the mail in the first place > >1194117 ± 0% +14.4%1366153 ± 1% will-it-scale.per_thread_ops > 0.58 ± 0% -2.0% 0.57 ± 0% will-it-scale.scalability > 6820 ± 0% -19.6% 5483 ± 15% meminfo.AnonHugePages > 2652 ± 5% -10.4% 2375 ± 2% vmstat.system.cs > 2848 ± 32%+141.2% 6870 ± 65% numa-meminfo.node1.Active(anon) > 2832 ± 31% +57.6% 4465 ± 27% numa-meminfo.node1.AnonPages > 15018 ± 12% -23.3% 11515 ± 15% numa-meminfo.node2.AnonPages > 1214 ± 14% -22.8% 936.75 ± 20% numa-meminfo.node3.PageTables > 712.25 ± 32%+141.2% 1718 ± 65% numa-vmstat.node1.nr_active_anon > 708.25 ± 31% +57.7% 1116 ± 27% numa-vmstat.node1.nr_anon_pages > > How is this related and what should I do about this information? For each will-it-scale sub test case, it will be run in both process mode and thread mode, and task number will change from 1 to CPU number. will-it-scale.per_thread_ops shows thread mode main result. will-it-scale.scalability is calculated to measure how per_process_ops and per_thread_ops scaled along with the task number. These are default behavior of will-it-scale test suite. Others are monitors output. That is, other information collected during test. For example, meminfo is a monitor to sampling /proc/meminfo contents, AnonHugePages is a line in it. meminfo.AnonHugePages is for the average value of AnonHugePages line of /proc/meminfo. Similarly vmstat.system.cs is the average value of cs column of system column group of /usr/bin/vmstat. We hope these information are helpful for root causing the regression. > If it's important then I have to admit, that I fail to understand why. > > If it's not important then I have to ask why is this included. > >> > So that allows me to reproduce that test more or less with no effort. And >> > that's the really important part. >> >> For reproducing, now we use lkp-tests tool, which includes scripts to >> build the test case, run the test, collect various information, compare >> the test result, with the job file attached with the report email. That >> is not the easiest way, we will continuously improve it. > > I know and lkp-tests is a pain to work with. So please look into a way to > extract the relevant binaries, so it's simple for developers to reproduce. OK. We will try to improve on this side. But it is not an easy task for us to provided easy to use simple binaries. Do you think something like Docker image is easy to use? >> > You can provide nice charts and full comparison tables for all tests on a >> > web >> > site for those who are interested in large stats and pretty charts. >> > >> > Full results: http://wherever.you.store/your/results/test-nr/results >> >> Before we have a website for detailed information, we will still put >> some details into report email. > > Ok, but please make them understandable for mere mortals. Sure. Best Regards, Huang, Ying > Thanks, > > tglx
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Mon, 21 Mar 2016, Huang, Ying wrote: > > FYI, we noticed 25.6% performance improvement due to commit > > > >65d8fc777f6d "futex: Remove requirement for lock_page() in > > get_futex_key()" > > > > in the will-it-scale.per_process_ops test. > > > > will-it-scale.per_process_ops tests the futex operations for process shared > > futexes (Or whatever that test really does). > > There is a futex sub test case for will-it-scale test suite. But I got your > point, we need some description for the test case. If email is not too > limited for the full description, we will put it in some web site and > include short description and link to the full description in email. Ok. Just make sure the short description gives enough information for the casual reader. > > The commit has no significant impact on any other test in the test suite. > > Sorry, we have no enough machine power to test all test cases for each > bisect result. So we will have no such information until we find a way > to do that. Well, then I really have to ask how I should interpret the data here: 5076304 ± 0% +25.6%6374220 ± 0% will-it-scale.per_process_ops ^^^ That's the reason why you sent the mail in the first place 1194117 ± 0% +14.4%1366153 ± 1% will-it-scale.per_thread_ops 0.58 ± 0% -2.0% 0.57 ± 0% will-it-scale.scalability 6820 ± 0% -19.6% 5483 ± 15% meminfo.AnonHugePages 2652 ± 5% -10.4% 2375 ± 2% vmstat.system.cs 2848 ± 32%+141.2% 6870 ± 65% numa-meminfo.node1.Active(anon) 2832 ± 31% +57.6% 4465 ± 27% numa-meminfo.node1.AnonPages 15018 ± 12% -23.3% 11515 ± 15% numa-meminfo.node2.AnonPages 1214 ± 14% -22.8% 936.75 ± 20% numa-meminfo.node3.PageTables 712.25 ± 32%+141.2% 1718 ± 65% numa-vmstat.node1.nr_active_anon 708.25 ± 31% +57.7% 1116 ± 27% numa-vmstat.node1.nr_anon_pages How is this related and what should I do about this information? If it's important then I have to admit, that I fail to understand why. If it's not important then I have to ask why is this included. > > So that allows me to reproduce that test more or less with no effort. And > > that's the really important part. > > For reproducing, now we use lkp-tests tool, which includes scripts to > build the test case, run the test, collect various information, compare > the test result, with the job file attached with the report email. That > is not the easiest way, we will continuously improve it. I know and lkp-tests is a pain to work with. So please look into a way to extract the relevant binaries, so it's simple for developers to reproduce. > > You can provide nice charts and full comparison tables for all tests on a > > web > > site for those who are interested in large stats and pretty charts. > > > > Full results: http://wherever.you.store/your/results/test-nr/results > > Before we have a website for detailed information, we will still put > some details into report email. Ok, but please make them understandable for mere mortals. Thanks, tglx
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Mon, 21 Mar 2016, Huang, Ying wrote: > > FYI, we noticed 25.6% performance improvement due to commit > > > >65d8fc777f6d "futex: Remove requirement for lock_page() in > > get_futex_key()" > > > > in the will-it-scale.per_process_ops test. > > > > will-it-scale.per_process_ops tests the futex operations for process shared > > futexes (Or whatever that test really does). > > There is a futex sub test case for will-it-scale test suite. But I got your > point, we need some description for the test case. If email is not too > limited for the full description, we will put it in some web site and > include short description and link to the full description in email. Ok. Just make sure the short description gives enough information for the casual reader. > > The commit has no significant impact on any other test in the test suite. > > Sorry, we have no enough machine power to test all test cases for each > bisect result. So we will have no such information until we find a way > to do that. Well, then I really have to ask how I should interpret the data here: 5076304 ± 0% +25.6%6374220 ± 0% will-it-scale.per_process_ops ^^^ That's the reason why you sent the mail in the first place 1194117 ± 0% +14.4%1366153 ± 1% will-it-scale.per_thread_ops 0.58 ± 0% -2.0% 0.57 ± 0% will-it-scale.scalability 6820 ± 0% -19.6% 5483 ± 15% meminfo.AnonHugePages 2652 ± 5% -10.4% 2375 ± 2% vmstat.system.cs 2848 ± 32%+141.2% 6870 ± 65% numa-meminfo.node1.Active(anon) 2832 ± 31% +57.6% 4465 ± 27% numa-meminfo.node1.AnonPages 15018 ± 12% -23.3% 11515 ± 15% numa-meminfo.node2.AnonPages 1214 ± 14% -22.8% 936.75 ± 20% numa-meminfo.node3.PageTables 712.25 ± 32%+141.2% 1718 ± 65% numa-vmstat.node1.nr_active_anon 708.25 ± 31% +57.7% 1116 ± 27% numa-vmstat.node1.nr_anon_pages How is this related and what should I do about this information? If it's important then I have to admit, that I fail to understand why. If it's not important then I have to ask why is this included. > > So that allows me to reproduce that test more or less with no effort. And > > that's the really important part. > > For reproducing, now we use lkp-tests tool, which includes scripts to > build the test case, run the test, collect various information, compare > the test result, with the job file attached with the report email. That > is not the easiest way, we will continuously improve it. I know and lkp-tests is a pain to work with. So please look into a way to extract the relevant binaries, so it's simple for developers to reproduce. > > You can provide nice charts and full comparison tables for all tests on a > > web > > site for those who are interested in large stats and pretty charts. > > > > Full results: http://wherever.you.store/your/results/test-nr/results > > Before we have a website for detailed information, we will still put > some details into report email. Ok, but please make them understandable for mere mortals. Thanks, tglx
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Hi, Thomas, Thanks a lot for your valuable input! Thomas Gleixnerwrites: > On Fri, 18 Mar 2016, Huang, Ying wrote: >> Usually we will put most important change we think in the subject of the >> mail, for this email, it is, >> >> +25.6% will-it-scale.per_process_ops > > That is confusing on it's own, because the reader does not know at all whether > this is an improvement or a regression. > > So something like this might be useful: > > Subject: subsystem: 12digitsha1: 25% performance improvement > > or in some other case > > Subject: subsystem: 12digitsha1: 25% performance regression > > So in the latter case I will look into that mail immediately. The improvement > one can wait until I have cared about urgent stuff. > > In the subject line it is pretty much irrelevant which foo-bla-ops test has > produced that result. It really does not matter. If it's a regression, it's > urgent. If it's an improvement it's informal and it can wait to be read. > > So in that case it would be: > > futex: 65d8fc777f6d: 25% performance improvement > > You can grab the subsystem prefix from the commit. We will include regression/improvement information in subject at least. >> and, we try to put most important changes at the top of the comparison >> result below. That is the will-it-scale.xxx below. >> >> We are thinking about how to improve this. You input is valuable for >> us. We are thinking change the "below changes" line to something like >> below. >> >> FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on >> ... >> >> Does this looks better? > > A bit, but it still does not tell me much. It's completely non obvious what > 'will-it-scale.per_process_ops' means. will-it-scale is a test suite, per_process_ops is one of its results. That is the convention used in original report. > Let me give you an example how a useful > and easy to understand summary of the change could look like: > > > FYI, we noticed 25.6% performance improvement due to commit > >65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" > > in the will-it-scale.per_process_ops test. > > will-it-scale.per_process_ops tests the futex operations for process shared > futexes (Or whatever that test really does). There is a futex sub test case for will-it-scale test suite. But I got your point, we need some description for the test case. If email is not too limited for the full description, we will put it in some web site and include short description and link to the full description in email. > The commit has no significant impact on any other test in the test suite. Sorry, we have no enough machine power to test all test cases for each bisect result. So we will have no such information until we find a way to do that. > So those few lines tell precisely what this is about. It's something I already > expected, so I really can skip the rest of the mail unless I'm interested in > reproducing the result. We will put important information at first of the email. And details later. Better to have clear mark. So people can get important information and ignore the details if they don't want. > Now lets look at a performance regression. > > Subject: futex: 65d8fc777f6d: 25% performance regression > > FYI, we noticed a 25.2% performance regression due to commit > > 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" > > in the will-it-scale.per_process_ops test. > > will-it-scale.per_process_ops tests the futex operations for process shared > futexes (Or whatever that test really does). > > The commit has no significant impact on any other test in the test suite. > > In that case I will certainly be interested how to reproduce that test. So I > need the following information: > > Machine description: Intel IvyBridge 2 sockets, 32 cores, 64G RAM > Config file: http://wherever.you.store/your/results/test-nr/config We have some information about this before. But not organized good enough, will improve it. > Test: > http://wherever.you.store/your/tests/will-it-scale.per_process_ops.tar.bz2 > > That tarball should contain: > > README > test_script.sh > test_binary > > README should tell: > >will-it-scale.per_process_ops > >Short explanation of the test > >Preliminaries: > - perf > - whatever > > So that allows me to reproduce that test more or less with no effort. And > that's the really important part. For reproducing, now we use lkp-tests tool, which includes scripts to build the test case, run the test, collect various information, compare the test result, with the job file attached with the report email. That is not the easiest way, we will continuously improve it. > You can provide nice charts and full comparison tables for all tests on a web > site for those who are interested in large stats and pretty charts. > > Full results: http://wherever.you.store/your/results/test-nr/results Before we have
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Hi, Thomas, Thanks a lot for your valuable input! Thomas Gleixner writes: > On Fri, 18 Mar 2016, Huang, Ying wrote: >> Usually we will put most important change we think in the subject of the >> mail, for this email, it is, >> >> +25.6% will-it-scale.per_process_ops > > That is confusing on it's own, because the reader does not know at all whether > this is an improvement or a regression. > > So something like this might be useful: > > Subject: subsystem: 12digitsha1: 25% performance improvement > > or in some other case > > Subject: subsystem: 12digitsha1: 25% performance regression > > So in the latter case I will look into that mail immediately. The improvement > one can wait until I have cared about urgent stuff. > > In the subject line it is pretty much irrelevant which foo-bla-ops test has > produced that result. It really does not matter. If it's a regression, it's > urgent. If it's an improvement it's informal and it can wait to be read. > > So in that case it would be: > > futex: 65d8fc777f6d: 25% performance improvement > > You can grab the subsystem prefix from the commit. We will include regression/improvement information in subject at least. >> and, we try to put most important changes at the top of the comparison >> result below. That is the will-it-scale.xxx below. >> >> We are thinking about how to improve this. You input is valuable for >> us. We are thinking change the "below changes" line to something like >> below. >> >> FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on >> ... >> >> Does this looks better? > > A bit, but it still does not tell me much. It's completely non obvious what > 'will-it-scale.per_process_ops' means. will-it-scale is a test suite, per_process_ops is one of its results. That is the convention used in original report. > Let me give you an example how a useful > and easy to understand summary of the change could look like: > > > FYI, we noticed 25.6% performance improvement due to commit > >65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" > > in the will-it-scale.per_process_ops test. > > will-it-scale.per_process_ops tests the futex operations for process shared > futexes (Or whatever that test really does). There is a futex sub test case for will-it-scale test suite. But I got your point, we need some description for the test case. If email is not too limited for the full description, we will put it in some web site and include short description and link to the full description in email. > The commit has no significant impact on any other test in the test suite. Sorry, we have no enough machine power to test all test cases for each bisect result. So we will have no such information until we find a way to do that. > So those few lines tell precisely what this is about. It's something I already > expected, so I really can skip the rest of the mail unless I'm interested in > reproducing the result. We will put important information at first of the email. And details later. Better to have clear mark. So people can get important information and ignore the details if they don't want. > Now lets look at a performance regression. > > Subject: futex: 65d8fc777f6d: 25% performance regression > > FYI, we noticed a 25.2% performance regression due to commit > > 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" > > in the will-it-scale.per_process_ops test. > > will-it-scale.per_process_ops tests the futex operations for process shared > futexes (Or whatever that test really does). > > The commit has no significant impact on any other test in the test suite. > > In that case I will certainly be interested how to reproduce that test. So I > need the following information: > > Machine description: Intel IvyBridge 2 sockets, 32 cores, 64G RAM > Config file: http://wherever.you.store/your/results/test-nr/config We have some information about this before. But not organized good enough, will improve it. > Test: > http://wherever.you.store/your/tests/will-it-scale.per_process_ops.tar.bz2 > > That tarball should contain: > > README > test_script.sh > test_binary > > README should tell: > >will-it-scale.per_process_ops > >Short explanation of the test > >Preliminaries: > - perf > - whatever > > So that allows me to reproduce that test more or less with no effort. And > that's the really important part. For reproducing, now we use lkp-tests tool, which includes scripts to build the test case, run the test, collect various information, compare the test result, with the job file attached with the report email. That is not the easiest way, we will continuously improve it. > You can provide nice charts and full comparison tables for all tests on a web > site for those who are interested in large stats and pretty charts. > > Full results: http://wherever.you.store/your/results/test-nr/results Before we have a website for
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Sorry, for late. Ingo Molnarwrites: > * kernel test robot wrote: > >> FYI, we noticed the below changes on >> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master >> commit 65d8fc777f6dcfee12785c057a6b57f679641c90 ("futex: Remove requirement >> for >> lock_page() in get_futex_key()") > > I have asked for this before, but let me try again: could you _PLEASE_ make > these > emails more readable? > > For example what are the 'below changes'? Changes in the profile output? > Profiles > always change from run to run, so that alone is not informative. > > Also, there are a lot of changes - which ones prompted the email to be > generated? Usually we will put most important change we think in the subject of the mail, for this email, it is, +25.6% will-it-scale.per_process_ops and, we try to put most important changes at the top of the comparison result below. That is the will-it-scale.xxx below. We are thinking about how to improve this. You input is valuable for us. We are thinking change the "below changes" line to something like below. FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on ... Does this looks better? > All in one, this email is hard to parse, because it just dumps a lot of > information with very little explanatory structure for someone not versed in > their > format. Please try to create an easy to parse 'story' that leads the reader > towards what you want these emails to tell - not just a raw dump of seemingly > unconnected pieces of data ... Which kind of story? Something like, we tested some benchmark on some machine, which triggered some regression, we do bisect, and find the first bad commit is xxx. In addition to benchmark result, we collected some other information, hope they are helpful for you. We just try to help. Sorry for confusion. We try to provide all information needed to describe the change and help for root causing the changes. But we know, we are still far from there, so your input is important for us to improve our test report. Which part of the report do you think should be changed firstly? The overall structure? Or the data format? Best Regards, Huang, Ying > Thanks, > > Ingo > >> >> >> = >> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase: >> >> gcc-4.9/performance/x86_64-rhel/debian-x86_64-2015-02-07.cgz/lkp-sbx04/futex1/will-it-scale >> >> commit: >> 8ad7b378d0d016309014cae0f640434bca7b5e11 >> 65d8fc777f6dcfee12785c057a6b57f679641c90 >> >> 8ad7b378d0d01630 65d8fc777f6dcfee12785c057a >> -- >> %stddev %change %stddev >> \ |\ >>5076304 . 0% +25.6%6374220 . 0% will-it-scale.per_process_ops >>1194117 . 0% +14.4%1366153 . 1% will-it-scale.per_thread_ops >> 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability >> 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages >> 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs >> 2848 . 32%+141.2% 6870 . 65% numa-meminfo.node1.Active(anon) >> 2832 . 31% +57.6% 4465 . 27% numa-meminfo.node1.AnonPages >> 15018 . 12% -23.3% 11515 . 15% numa-meminfo.node2.AnonPages >> 1214 . 14% -22.8% 936.75 . 20% numa-meminfo.node3.PageTables >> 712.25 . 32%+141.2% 1718 . 65% >> numa-vmstat.node1.nr_active_anon >> 708.25 . 31% +57.7% 1116 . 27% numa-vmstat.node1.nr_anon_pages >> 3754 . 12% -23.3% 2879 . 15% numa-vmstat.node2.nr_anon_pages >> 304.75 . 14% -23.1% 234.50 . 20% >> numa-vmstat.node3.nr_page_table_pages >> 3.53 . 1% -100.0% 0.00 . -1% > perf-profile.cycles.___might_sleep.__might_sleep.get_futex_key.futex_wake.do_futex >> 4.34 . 1% -100.0% 0.00 . -1% > perf-profile.cycles.__might_sleep.get_futex_key.futex_wake.do_futex.sys_futex >> 1.27 . 3% -100.0% 0.00 . -1% > perf-profile.cycles.__wake_up_bit.unlock_page.get_futex_key.futex_wake.do_futex >> 4.36 . 1% +29.6% 5.65 . 1% > perf-profile.cycles.drop_futex_key_refs.isra.12.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 6.69 . 1% +28.1% 8.57 . 0% >> perf-profile.cycles.entry_SYSCALL_64 >> 6.73 . 0% +30.6% 8.79 . 0% >> perf-profile.cycles.entry_SYSCALL_64_after_swapgs >> 74.21 . 0% -11.0% 66.06 . 0% > perf-profile.cycles.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 59.05 . 0% -21.4% 46.40 . 0% > perf-profile.cycles.get_futex_key.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 4.12 . 0% +78.5% 7.36 . 1% > perf-profile.cycles.get_futex_key_refs.isra.11.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 2.27 . 3% +24.1% 2.82 . 4% >
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Sorry, for late. Ingo Molnar writes: > * kernel test robot wrote: > >> FYI, we noticed the below changes on >> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master >> commit 65d8fc777f6dcfee12785c057a6b57f679641c90 ("futex: Remove requirement >> for >> lock_page() in get_futex_key()") > > I have asked for this before, but let me try again: could you _PLEASE_ make > these > emails more readable? > > For example what are the 'below changes'? Changes in the profile output? > Profiles > always change from run to run, so that alone is not informative. > > Also, there are a lot of changes - which ones prompted the email to be > generated? Usually we will put most important change we think in the subject of the mail, for this email, it is, +25.6% will-it-scale.per_process_ops and, we try to put most important changes at the top of the comparison result below. That is the will-it-scale.xxx below. We are thinking about how to improve this. You input is valuable for us. We are thinking change the "below changes" line to something like below. FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on ... Does this looks better? > All in one, this email is hard to parse, because it just dumps a lot of > information with very little explanatory structure for someone not versed in > their > format. Please try to create an easy to parse 'story' that leads the reader > towards what you want these emails to tell - not just a raw dump of seemingly > unconnected pieces of data ... Which kind of story? Something like, we tested some benchmark on some machine, which triggered some regression, we do bisect, and find the first bad commit is xxx. In addition to benchmark result, we collected some other information, hope they are helpful for you. We just try to help. Sorry for confusion. We try to provide all information needed to describe the change and help for root causing the changes. But we know, we are still far from there, so your input is important for us to improve our test report. Which part of the report do you think should be changed firstly? The overall structure? Or the data format? Best Regards, Huang, Ying > Thanks, > > Ingo > >> >> >> = >> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase: >> >> gcc-4.9/performance/x86_64-rhel/debian-x86_64-2015-02-07.cgz/lkp-sbx04/futex1/will-it-scale >> >> commit: >> 8ad7b378d0d016309014cae0f640434bca7b5e11 >> 65d8fc777f6dcfee12785c057a6b57f679641c90 >> >> 8ad7b378d0d01630 65d8fc777f6dcfee12785c057a >> -- >> %stddev %change %stddev >> \ |\ >>5076304 . 0% +25.6%6374220 . 0% will-it-scale.per_process_ops >>1194117 . 0% +14.4%1366153 . 1% will-it-scale.per_thread_ops >> 0.58 . 0% -2.0% 0.57 . 0% will-it-scale.scalability >> 6820 . 0% -19.6% 5483 . 15% meminfo.AnonHugePages >> 2652 . 5% -10.4% 2375 . 2% vmstat.system.cs >> 2848 . 32%+141.2% 6870 . 65% numa-meminfo.node1.Active(anon) >> 2832 . 31% +57.6% 4465 . 27% numa-meminfo.node1.AnonPages >> 15018 . 12% -23.3% 11515 . 15% numa-meminfo.node2.AnonPages >> 1214 . 14% -22.8% 936.75 . 20% numa-meminfo.node3.PageTables >> 712.25 . 32%+141.2% 1718 . 65% >> numa-vmstat.node1.nr_active_anon >> 708.25 . 31% +57.7% 1116 . 27% numa-vmstat.node1.nr_anon_pages >> 3754 . 12% -23.3% 2879 . 15% numa-vmstat.node2.nr_anon_pages >> 304.75 . 14% -23.1% 234.50 . 20% >> numa-vmstat.node3.nr_page_table_pages >> 3.53 . 1% -100.0% 0.00 . -1% > perf-profile.cycles.___might_sleep.__might_sleep.get_futex_key.futex_wake.do_futex >> 4.34 . 1% -100.0% 0.00 . -1% > perf-profile.cycles.__might_sleep.get_futex_key.futex_wake.do_futex.sys_futex >> 1.27 . 3% -100.0% 0.00 . -1% > perf-profile.cycles.__wake_up_bit.unlock_page.get_futex_key.futex_wake.do_futex >> 4.36 . 1% +29.6% 5.65 . 1% > perf-profile.cycles.drop_futex_key_refs.isra.12.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 6.69 . 1% +28.1% 8.57 . 0% >> perf-profile.cycles.entry_SYSCALL_64 >> 6.73 . 0% +30.6% 8.79 . 0% >> perf-profile.cycles.entry_SYSCALL_64_after_swapgs >> 74.21 . 0% -11.0% 66.06 . 0% > perf-profile.cycles.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 59.05 . 0% -21.4% 46.40 . 0% > perf-profile.cycles.get_futex_key.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 4.12 . 0% +78.5% 7.36 . 1% > perf-profile.cycles.get_futex_key_refs.isra.11.futex_wake.do_futex.sys_futex.entry_SYSCALL_64_fastpath >> 2.27 . 3% +24.1% 2.82 . 4% >
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Fri, 18 Mar 2016, Huang, Ying wrote: > Usually we will put most important change we think in the subject of the > mail, for this email, it is, > > +25.6% will-it-scale.per_process_ops That is confusing on it's own, because the reader does not know at all whether this is an improvement or a regression. So something like this might be useful: Subject: subsystem: 12digitsha1: 25% performance improvement or in some other case Subject: subsystem: 12digitsha1: 25% performance regression So in the latter case I will look into that mail immediately. The improvement one can wait until I have cared about urgent stuff. In the subject line it is pretty much irrelevant which foo-bla-ops test has produced that result. It really does not matter. If it's a regression, it's urgent. If it's an improvement it's informal and it can wait to be read. So in that case it would be: futex: 65d8fc777f6d: 25% performance improvement You can grab the subsystem prefix from the commit. > and, we try to put most important changes at the top of the comparison > result below. That is the will-it-scale.xxx below. > > We are thinking about how to improve this. You input is valuable for > us. We are thinking change the "below changes" line to something like > below. > > FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on > ... > > Does this looks better? A bit, but it still does not tell me much. It's completely non obvious what 'will-it-scale.per_process_ops' means. Let me give you an example how a useful and easy to understand summary of the change could look like: FYI, we noticed 25.6% performance improvement due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_process_ops test. will-it-scale.per_process_ops tests the futex operations for process shared futexes (Or whatever that test really does). The commit has no significant impact on any other test in the test suite. So those few lines tell precisely what this is about. It's something I already expected, so I really can skip the rest of the mail unless I'm interested in reproducing the result. Now lets look at a performance regression. Subject: futex: 65d8fc777f6d: 25% performance regression FYI, we noticed a 25.2% performance regression due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_process_ops test. will-it-scale.per_process_ops tests the futex operations for process shared futexes (Or whatever that test really does). The commit has no significant impact on any other test in the test suite. In that case I will certainly be interested how to reproduce that test. So I need the following information: Machine description: Intel IvyBridge 2 sockets, 32 cores, 64G RAM Config file: http://wherever.you.store/your/results/test-nr/config Test: http://wherever.you.store/your/tests/will-it-scale.per_process_ops.tar.bz2 That tarball should contain: README test_script.sh test_binary README should tell: will-it-scale.per_process_ops Short explanation of the test Preliminaries: - perf - whatever So that allows me to reproduce that test more or less with no effort. And that's the really important part. You can provide nice charts and full comparison tables for all tests on a web site for those who are interested in large stats and pretty charts. Full results: http://wherever.you.store/your/results/test-nr/results So now lets look at a scenario where that commit results in a performance improvement on will-it-scale.per_process_ops and a regression on will-it-scale.per_task. Subject: futex: 65d8fc777f6d: 25% performance regression FYI, we noticed a 25.2% performance regression due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_task test. will-it-scale.per_tasks tests the futex operations for process private futexes (Or whatever that test really does). The commit has significant impact on the following tests in the test suite: will-it-scale.per_process_ops: 25% improvement The 25% improvement is really not interesting. What's interesting is the regression. That's what people need to look at. You still can provide the information about all the tests including the 25% improvement data on Full results: http://wherever.you.store/your/results/test-nr/results Now, if you have commits which have mixed impact on various tests, then it's important to point out the most relevant issue clearly in the summary. I.e. you need to filter the tests for the maximum regression/improvement and use that as the main information. You still can provide the information about the impact on other tests in a very condensed form. The commit has significant impact on the following tests in the test suite: test1: 0.5% Regression test2: 2.0% Improvement testN:
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
On Fri, 18 Mar 2016, Huang, Ying wrote: > Usually we will put most important change we think in the subject of the > mail, for this email, it is, > > +25.6% will-it-scale.per_process_ops That is confusing on it's own, because the reader does not know at all whether this is an improvement or a regression. So something like this might be useful: Subject: subsystem: 12digitsha1: 25% performance improvement or in some other case Subject: subsystem: 12digitsha1: 25% performance regression So in the latter case I will look into that mail immediately. The improvement one can wait until I have cared about urgent stuff. In the subject line it is pretty much irrelevant which foo-bla-ops test has produced that result. It really does not matter. If it's a regression, it's urgent. If it's an improvement it's informal and it can wait to be read. So in that case it would be: futex: 65d8fc777f6d: 25% performance improvement You can grab the subsystem prefix from the commit. > and, we try to put most important changes at the top of the comparison > result below. That is the will-it-scale.xxx below. > > We are thinking about how to improve this. You input is valuable for > us. We are thinking change the "below changes" line to something like > below. > > FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on > ... > > Does this looks better? A bit, but it still does not tell me much. It's completely non obvious what 'will-it-scale.per_process_ops' means. Let me give you an example how a useful and easy to understand summary of the change could look like: FYI, we noticed 25.6% performance improvement due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_process_ops test. will-it-scale.per_process_ops tests the futex operations for process shared futexes (Or whatever that test really does). The commit has no significant impact on any other test in the test suite. So those few lines tell precisely what this is about. It's something I already expected, so I really can skip the rest of the mail unless I'm interested in reproducing the result. Now lets look at a performance regression. Subject: futex: 65d8fc777f6d: 25% performance regression FYI, we noticed a 25.2% performance regression due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_process_ops test. will-it-scale.per_process_ops tests the futex operations for process shared futexes (Or whatever that test really does). The commit has no significant impact on any other test in the test suite. In that case I will certainly be interested how to reproduce that test. So I need the following information: Machine description: Intel IvyBridge 2 sockets, 32 cores, 64G RAM Config file: http://wherever.you.store/your/results/test-nr/config Test: http://wherever.you.store/your/tests/will-it-scale.per_process_ops.tar.bz2 That tarball should contain: README test_script.sh test_binary README should tell: will-it-scale.per_process_ops Short explanation of the test Preliminaries: - perf - whatever So that allows me to reproduce that test more or less with no effort. And that's the really important part. You can provide nice charts and full comparison tables for all tests on a web site for those who are interested in large stats and pretty charts. Full results: http://wherever.you.store/your/results/test-nr/results So now lets look at a scenario where that commit results in a performance improvement on will-it-scale.per_process_ops and a regression on will-it-scale.per_task. Subject: futex: 65d8fc777f6d: 25% performance regression FYI, we noticed a 25.2% performance regression due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_task test. will-it-scale.per_tasks tests the futex operations for process private futexes (Or whatever that test really does). The commit has significant impact on the following tests in the test suite: will-it-scale.per_process_ops: 25% improvement The 25% improvement is really not interesting. What's interesting is the regression. That's what people need to look at. You still can provide the information about all the tests including the 25% improvement data on Full results: http://wherever.you.store/your/results/test-nr/results Now, if you have commits which have mixed impact on various tests, then it's important to point out the most relevant issue clearly in the summary. I.e. you need to filter the tests for the maximum regression/improvement and use that as the main information. You still can provide the information about the impact on other tests in a very condensed form. The commit has significant impact on the following tests in the test suite: test1: 0.5% Regression test2: 2.0% Improvement testN:
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Hi, Ingo, Ingo Molnarwrites: > * kernel test robot wrote: > >> FYI, we noticed the below changes on >> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master >> commit 65d8fc777f6dcfee12785c057a6b57f679641c90 ("futex: Remove requirement >> for >> lock_page() in get_futex_key()") > > I have asked for this before, but let me try again: could you _PLEASE_ make > these > emails more readable? > > For example what are the 'below changes'? Changes in the profile output? > Profiles > always change from run to run, so that alone is not informative. > > Also, there are a lot of changes - which ones prompted the email to be > generated? > > All in one, this email is hard to parse, because it just dumps a lot of > information with very little explanatory structure for someone not versed in > their > format. Please try to create an easy to parse 'story' that leads the reader > towards what you want these emails to tell - not just a raw dump of seemingly > unconnected pieces of data ... Your input are valuable for us. We are discussing how to improve our reporting to be helpful for kernel developers. Will go back to you soon on this. Best Regards, Huang, Ying > Thanks, > > Ingo >
Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops
Hi, Ingo, Ingo Molnar writes: > * kernel test robot wrote: > >> FYI, we noticed the below changes on >> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master >> commit 65d8fc777f6dcfee12785c057a6b57f679641c90 ("futex: Remove requirement >> for >> lock_page() in get_futex_key()") > > I have asked for this before, but let me try again: could you _PLEASE_ make > these > emails more readable? > > For example what are the 'below changes'? Changes in the profile output? > Profiles > always change from run to run, so that alone is not informative. > > Also, there are a lot of changes - which ones prompted the email to be > generated? > > All in one, this email is hard to parse, because it just dumps a lot of > information with very little explanatory structure for someone not versed in > their > format. Please try to create an easy to parse 'story' that leads the reader > towards what you want these emails to tell - not just a raw dump of seemingly > unconnected pieces of data ... Your input are valuable for us. We are discussing how to improve our reporting to be helpful for kernel developers. Will go back to you soon on this. Best Regards, Huang, Ying > Thanks, > > Ingo >