Re: Request for Testing: TCP RACK
(...) Backup server is https://www.rsync.net/ (free 500GB for FreeBSD developers). Nuno Teixeira escreveu (quarta, 10/04/2024 à(s) 13:39): > With base stack I can complete restic check successfully > downloading/reading/checking all files from a "big" remote compressed > backup. > Changing it to RACK stack, it fails. > > I run this command often because in the past, compression corruption > occured and this is the equivalent of restoring backup to check its > integrity. > > Maybe someone could do a restic test to check if this is reproducible. > > Thanks, > > > > escreveu (quarta, 10/04/2024 à(s) 13:12): > >> >> >> > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: >> > >> > Hello all, >> > >> > @ current 1500018 and fetching torrents with net-p2p/qbittorrent >> finished ~2GB download and connection UP until the end: >> > >> > --- >> > Apr 10 11:26:46 leg kernel: re0: watchdog timeout >> > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN >> > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 >> > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): >> 255.255.255.0 >> > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): >> 192.168.1.255 >> > Apr 10 11:26:49 leg kernel: re0: link state changed to UP >> > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 >> > --- >> > >> > In the past tests, I've got more watchdog timeouts, connection goes >> down and a reboot needed to put it back (`service netif restart` didn't >> work). >> > >> > Other way to reproduce this is using sysutils/restic (backup program) >> to read/check all files from a remote server via sftp: >> > >> > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB >> compressed backup. >> > >> > --- >> > watchdog timeout x3 as above >> > --- >> > >> > restic check fail log @ 15% progress: >> > --- >> > >> > Load(, 17310001, 0) returned error, retrying after >> 1.7670599s: connection lost >> > Load(, 17456892, 0) returned error, retrying after >> 4.619104908s: connection lost >> > Load(, 17310001, 0) returned error, retrying after >> 5.477648517s: connection lost >> > List(lock) returned error, retrying after 293.057766ms: connection lost >> > List(lock) returned error, retrying after 385.206693ms: connection lost >> > List(lock) returned error, retrying after 1.577594281s: connection lost >> > >> > >> > Connection continues UP. >> Hi, >> >> I'm not sure what the issue is you are reporting. Could you state >> what behavior you are experiencing with the base stack and with >> the RACK stack. In particular, what the difference is? >> >> Best regards >> Michael >> > >> > Cheers, >> > >> > escreveu (quinta, 28/03/2024 à(s) 15:53): >> >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: >> >> >> >> Hello all! >> >> >> >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop >> (amd64)! >> >> >> >> Thanks all! >> > Thanks for the feedback! >> > >> > Best regards >> > Michael >> >> >> >> Drew Gallatin escreveu (quinta, 21/03/2024 >> à(s) 12:58): >> >> The entire point is to *NOT* go through the overhead of scheduling >> something asynchronously, but to take advantage of the fact that a >> user/kernel transition is going to trash the cache anyway. >> >> >> >> In the common case of a system which has less than the threshold >> number of connections , we access the tcp_hpts_softclock function pointer, >> make one function call, and access hpts_that_need_softclock, and then >> return. So that's 2 variables and a function call. >> >> >> >> I think it would be preferable to avoid that call, and to move the >> declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they >> are in the same cacheline. Then we'd be hitting just a single line in the >> common case. (I've made comments on the review to that effect). >> >> >> >> Also, I wonder if the threshold could get higher by default, so that >> hpts is never called in this context unless we're to the point where we're >> scheduling thousands of runs of the hpts thread (and taking all those clock >> interrupts). >> >> >> >> Drew >> >> >> >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >> >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: >> Ok I have created >> >> https://reviews.freebsd.org/D44420 >> >> >> To address the issue. I also attach a short version of the patch >> that Nuno >> can try and validate >> >> it works. Drew you may want to try this and validate the >> optimization does >> kick in since I can >> >> only now test that it does not on my local box :) >> >>> The patch still causes access to all cpu's cachelines on each userret. >> >>> It would be much better to inc/check the threshold and only schedule >> the >> >>> call when exceeded. Then the call can occur in some dedicated >> context, >> >>> like per-CPU thread, instead of userret. >> >>> >> >> >> R >> >> >>
Re: Request for Testing: TCP RACK
With base stack I can complete restic check successfully downloading/reading/checking all files from a "big" remote compressed backup. Changing it to RACK stack, it fails. I run this command often because in the past, compression corruption occured and this is the equivalent of restoring backup to check its integrity. Maybe someone could do a restic test to check if this is reproducible. Thanks, escreveu (quarta, 10/04/2024 à(s) 13:12): > > > > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: > > > > Hello all, > > > > @ current 1500018 and fetching torrents with net-p2p/qbittorrent > finished ~2GB download and connection UP until the end: > > > > --- > > Apr 10 11:26:46 leg kernel: re0: watchdog timeout > > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN > > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 > > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.2550 > > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): > 192.168.1.255 > > Apr 10 11:26:49 leg kernel: re0: link state changed to UP > > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > > --- > > > > In the past tests, I've got more watchdog timeouts, connection goes down > and a reboot needed to put it back (`service netif restart` didn't work). > > > > Other way to reproduce this is using sysutils/restic (backup program) to > read/check all files from a remote server via sftp: > > > > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB > compressed backup. > > > > --- > > watchdog timeout x3 as above > > --- > > > > restic check fail log @ 15% progress: > > --- > > > > Load(, 17310001, 0) returned error, retrying after > 1.7670599s: connection lost > > Load(, 17456892, 0) returned error, retrying after > 4.619104908s: connection lost > > Load(, 17310001, 0) returned error, retrying after > 5.477648517s: connection lost > > List(lock) returned error, retrying after 293.057766ms: connection lost > > List(lock) returned error, retrying after 385.206693ms: connection lost > > List(lock) returned error, retrying after 1.577594281s: connection lost > > > > > > Connection continues UP. > Hi, > > I'm not sure what the issue is you are reporting. Could you state > what behavior you are experiencing with the base stack and with > the RACK stack. In particular, what the difference is? > > Best regards > Michael > > > > Cheers, > > > > escreveu (quinta, 28/03/2024 à(s) 15:53): > >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > >> > >> Hello all! > >> > >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop > (amd64)! > >> > >> Thanks all! > > Thanks for the feedback! > > > > Best regards > > Michael > >> > >> Drew Gallatin escreveu (quinta, 21/03/2024 à(s) > 12:58): > >> The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > >> > >> In the common case of a system which has less than the threshold > number of connections , we access the tcp_hpts_softclock function pointer, > make one function call, and access hpts_that_need_softclock, and then > return. So that's 2 variables and a function call. > >> > >> I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they > are in the same cacheline. Then we'd be hitting just a single line in the > common case. (I've made comments on the review to that effect). > >> > >> Also, I wonder if the threshold could get higher by default, so that > hpts is never called in this context unless we're to the point where we're > scheduling thousands of runs of the hpts thread (and taking all those clock > interrupts). > >> > >> Drew > >> > >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > Ok I have created > > https://reviews.freebsd.org/D44420 > > > To address the issue. I also attach a short version of the patch that > Nuno > can try and validate > > it works. Drew you may want to try this and validate the optimization > does > kick in since I can > > only now test that it does not on my local box :) > >>> The patch still causes access to all cpu's cachelines on each userret. > >>> It would be much better to inc/check the threshold and only schedule > the > >>> call when exceeded. Then the call can occur in some dedicated context, > >>> like per-CPU thread, instead of userret. > >>> > > > R > > > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > No. The goal is to run on every return to userspace for every > thread. > > > > Drew > > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > >> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >
Re: Request for Testing: TCP RACK
> On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: > > Hello all, > > @ current 1500018 and fetching torrents with net-p2p/qbittorrent finished > ~2GB download and connection UP until the end: > > --- > Apr 10 11:26:46 leg kernel: re0: watchdog timeout > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.255.0 > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): > 192.168.1.255 > Apr 10 11:26:49 leg kernel: re0: link state changed to UP > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > --- > > In the past tests, I've got more watchdog timeouts, connection goes down and > a reboot needed to put it back (`service netif restart` didn't work). > > Other way to reproduce this is using sysutils/restic (backup program) to > read/check all files from a remote server via sftp: > > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB > compressed backup. > > --- > watchdog timeout x3 as above > --- > > restic check fail log @ 15% progress: > --- > > Load(, 17310001, 0) returned error, retrying after > 1.7670599s: connection lost > Load(, 17456892, 0) returned error, retrying after > 4.619104908s: connection lost > Load(, 17310001, 0) returned error, retrying after > 5.477648517s: connection lost > List(lock) returned error, retrying after 293.057766ms: connection lost > List(lock) returned error, retrying after 385.206693ms: connection lost > List(lock) returned error, retrying after 1.577594281s: connection lost > > > Connection continues UP. Hi, I'm not sure what the issue is you are reporting. Could you state what behavior you are experiencing with the base stack and with the RACK stack. In particular, what the difference is? Best regards Michael > > Cheers, > > escreveu (quinta, 28/03/2024 à(s) 15:53): >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: >> >> Hello all! >> >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop (amd64)! >> >> Thanks all! > Thanks for the feedback! > > Best regards > Michael >> >> Drew Gallatin escreveu (quinta, 21/03/2024 à(s) >> 12:58): >> The entire point is to *NOT* go through the overhead of scheduling something >> asynchronously, but to take advantage of the fact that a user/kernel >> transition is going to trash the cache anyway. >> >> In the common case of a system which has less than the threshold number of >> connections , we access the tcp_hpts_softclock function pointer, make one >> function call, and access hpts_that_need_softclock, and then return. So >> that's 2 variables and a function call. >> >> I think it would be preferable to avoid that call, and to move the >> declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they >> are in the same cacheline. Then we'd be hitting just a single line in the >> common case. (I've made comments on the review to that effect). >> >> Also, I wonder if the threshold could get higher by default, so that hpts is >> never called in this context unless we're to the point where we're >> scheduling thousands of runs of the hpts thread (and taking all those clock >> interrupts). >> >> Drew >> >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: Ok I have created https://reviews.freebsd.org/D44420 To address the issue. I also attach a short version of the patch that Nuno can try and validate it works. Drew you may want to try this and validate the optimization does kick in since I can only now test that it does not on my local box :) >>> The patch still causes access to all cpu's cachelines on each userret. >>> It would be much better to inc/check the threshold and only schedule the >>> call when exceeded. Then the call can occur in some dedicated context, >>> like per-CPU thread, instead of userret. >>> R On 3/18/24 3:42 PM, Drew Gallatin wrote: > No. The goal is to run on every return to userspace for every thread. > > Drew > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: >> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >>> I got the idea from >>> https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >>> The gist is that the TCP pacing stuff needs to run frequently, and >>> rather than run it out of a clock interrupt, its more efficient to run >>> it out of a system call context at just the point where we return to >>> userspace and the cache is trashed anyway. The current implementation >>> is fine for our workload, but probably not idea for a generic system. >>> Especially one where something is banging on system calls. >>> >>> Ast's could be the right
Re: Request for Testing: TCP RACK
Hello all, @ current 1500018 and fetching torrents with net-p2p/qbittorrent finished ~2GB download and connection UP until the end: --- Apr 10 11:26:46 leg kernel: re0: watchdog timeout Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.255.0 Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): 192.168.1.255 Apr 10 11:26:49 leg kernel: re0: link state changed to UP Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 --- In the past tests, I've got more watchdog timeouts, connection goes down and a reboot needed to put it back (`service netif restart` didn't work). Other way to reproduce this is using sysutils/restic (backup program) to read/check all files from a remote server via sftp: `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB compressed backup. --- watchdog timeout x3 as above --- restic check fail log @ 15% progress: --- Load(, 17310001, 0) returned error, retrying after 1.7670599s: connection lost Load(, 17456892, 0) returned error, retrying after 4.619104908s: connection lost Load(, 17310001, 0) returned error, retrying after 5.477648517s: connection lost List(lock) returned error, retrying after 293.057766ms: connection lost List(lock) returned error, retrying after 385.206693ms: connection lost List(lock) returned error, retrying after 1.577594281s: connection lost Connection continues UP. Cheers, escreveu (quinta, 28/03/2024 à(s) 15:53): > > On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > > > > Hello all! > > > > Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop > (amd64)! > > > > Thanks all! > Thanks for the feedback! > > Best regards > Michael > > > > Drew Gallatin escreveu (quinta, 21/03/2024 à(s) > 12:58): > > The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > > > > In the common case of a system which has less than the threshold number > of connections , we access the tcp_hpts_softclock function pointer, make > one function call, and access hpts_that_need_softclock, and then return. > So that's 2 variables and a function call. > > > > I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they > are in the same cacheline. Then we'd be hitting just a single line in the > common case. (I've made comments on the review to that effect). > > > > Also, I wonder if the threshold could get higher by default, so that > hpts is never called in this context unless we're to the point where we're > scheduling thousands of runs of the hpts thread (and taking all those clock > interrupts). > > > > Drew > > > > On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > >> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > >>> Ok I have created > >>> > >>> https://reviews.freebsd.org/D44420 > >>> > >>> > >>> To address the issue. I also attach a short version of the patch that > Nuno > >>> can try and validate > >>> > >>> it works. Drew you may want to try this and validate the optimization > does > >>> kick in since I can > >>> > >>> only now test that it does not on my local box :) > >> The patch still causes access to all cpu's cachelines on each userret. > >> It would be much better to inc/check the threshold and only schedule the > >> call when exceeded. Then the call can occur in some dedicated context, > >> like per-CPU thread, instead of userret. > >> > >>> > >>> > >>> R > >>> > >>> > >>> > >>> On 3/18/24 3:42 PM, Drew Gallatin wrote: > No. The goal is to run on every return to userspace for every thread. > > Drew > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > >> I got the idea from > >> > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > >> The gist is that the TCP pacing stuff needs to run frequently, and > >> rather than run it out of a clock interrupt, its more efficient to > run > >> it out of a system call context at just the point where we return to > >> userspace and the cache is trashed anyway. The current > implementation > >> is fine for our workload, but probably not idea for a generic > system. > >> Especially one where something is banging on system calls. > >> > >> Ast's could be the right tool for this, but I'm super unfamiliar > with > >> them, and I can't find any docs on them. > >> > >> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > >> what's happening here? > > This call would need some AST number added, and then it registers the > > ast to run on next return to userspace, for
Re: Request for Testing: TCP RACK
> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > > Hello all! > > Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop (amd64)! > > Thanks all! Thanks for the feedback! Best regards Michael > > Drew Gallatin escreveu (quinta, 21/03/2024 à(s) 12:58): > The entire point is to *NOT* go through the overhead of scheduling something > asynchronously, but to take advantage of the fact that a user/kernel > transition is going to trash the cache anyway. > > In the common case of a system which has less than the threshold number of > connections , we access the tcp_hpts_softclock function pointer, make one > function call, and access hpts_that_need_softclock, and then return. So > that's 2 variables and a function call. > > I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they > are in the same cacheline. Then we'd be hitting just a single line in the > common case. (I've made comments on the review to that effect). > > Also, I wonder if the threshold could get higher by default, so that hpts is > never called in this context unless we're to the point where we're scheduling > thousands of runs of the hpts thread (and taking all those clock interrupts). > > Drew > > On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: >>> Ok I have created >>> >>> https://reviews.freebsd.org/D44420 >>> >>> >>> To address the issue. I also attach a short version of the patch that Nuno >>> can try and validate >>> >>> it works. Drew you may want to try this and validate the optimization does >>> kick in since I can >>> >>> only now test that it does not on my local box :) >> The patch still causes access to all cpu's cachelines on each userret. >> It would be much better to inc/check the threshold and only schedule the >> call when exceeded. Then the call can occur in some dedicated context, >> like per-CPU thread, instead of userret. >> >>> >>> >>> R >>> >>> >>> >>> On 3/18/24 3:42 PM, Drew Gallatin wrote: No. The goal is to run on every return to userspace for every thread. Drew On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >> I got the idea from >> https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >> The gist is that the TCP pacing stuff needs to run frequently, and >> rather than run it out of a clock interrupt, its more efficient to run >> it out of a system call context at just the point where we return to >> userspace and the cache is trashed anyway. The current implementation >> is fine for our workload, but probably not idea for a generic system. >> Especially one where something is banging on system calls. >> >> Ast's could be the right tool for this, but I'm super unfamiliar with >> them, and I can't find any docs on them. >> >> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to >> what's happening here? > This call would need some AST number added, and then it registers the > ast to run on next return to userspace, for the current thread. > > Is it enough? >> >> Drew > >> >> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: >>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: >> On 18. Mar 2024, at 12:42, Nuno Teixeira > wrote: >> >> Hello all! >> >> It works just fine! >> System performance is OK. >> Using patch on main-n268841-b0aaf8beb126(-dirty). >> >> --- >> net.inet.tcp.functions_available: >> Stack D > AliasPCB count >> freebsd freebsd 0 >> rack* > rack 38 >> --- >> >> It would be so nice that we can have a sysctl tunnable for > this patch >> so we could do more tests without recompiling kernel. > Thanks for testing! > > @gallatin: can you come up with a patch that is acceptable > for Netflix > and allows to mitigate the performance regression. Ideally, tcphpts could enable this automatically when it > starts to be used (enough?), but a sysctl could select auto/on/off. >>> There is already a well-known mechanism to request execution of the >>> specific function on return to userspace, namely AST. The difference >>> with the current hack is that the execution is requested for one > callback >>> in the context of the specific thread. >>> >>> Still, it might be worth a try to use it; what
Re: Request for Testing: TCP RACK
Hello all! Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop (amd64)! Thanks all! Drew Gallatin escreveu (quinta, 21/03/2024 à(s) 12:58): > The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > > In the common case of a system which has less than the threshold number > of connections , we access the tcp_hpts_softclock function pointer, make > one function call, and access hpts_that_need_softclock, and then return. > So that's 2 variables and a function call. > > I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they > are in the same cacheline. Then we'd be hitting just a single line in the > common case. (I've made comments on the review to that effect). > > Also, I wonder if the threshold could get higher by default, so that hpts > is never called in this context unless we're to the point where we're > scheduling thousands of runs of the hpts thread (and taking all those clock > interrupts). > > Drew > > On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > > On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > > Ok I have created > > > > https://reviews.freebsd.org/D44420 > > > > > > To address the issue. I also attach a short version of the patch that > Nuno > > can try and validate > > > > it works. Drew you may want to try this and validate the optimization > does > > kick in since I can > > > > only now test that it does not on my local box :) > The patch still causes access to all cpu's cachelines on each userret. > It would be much better to inc/check the threshold and only schedule the > call when exceeded. Then the call can occur in some dedicated context, > like per-CPU thread, instead of userret. > > > > > > > R > > > > > > > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > > No. The goal is to run on every return to userspace for every thread. > > > > > > Drew > > > > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > > > > I got the idea from > > > > > > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > > > > > The gist is that the TCP pacing stuff needs to run frequently, and > > > > > rather than run it out of a clock interrupt, its more efficient to > run > > > > > it out of a system call context at just the point where we return > to > > > > > userspace and the cache is trashed anyway. The current > implementation > > > > > is fine for our workload, but probably not idea for a generic > system. > > > > > Especially one where something is banging on system calls. > > > > > > > > > > Ast's could be the right tool for this, but I'm super unfamiliar > with > > > > > them, and I can't find any docs on them. > > > > > > > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent > to > > > > > what's happening here? > > > > This call would need some AST number added, and then it registers the > > > > ast to run on next return to userspace, for the current thread. > > > > > > > > Is it enough? > > > > > > > > > > Drew > > > > > > > > > > > > > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > > > > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > > > > > > > > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira > > > > wrote: > > > > > > > >> > > > > > > > >> Hello all! > > > > > > > >> > > > > > > > >> It works just fine! > > > > > > > >> System performance is OK. > > > > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > > > > >> > > > > > > > >> --- > > > > > > > >> net.inet.tcp.functions_available: > > > > > > > >> Stack D > > > > AliasPCB count > > > > > > > >> freebsd freebsd 0 > > > > > > > >> rack* > > > > rack 38 > > > > > > > >> --- > > > > > > > >> > > > > > > > >> It would be so nice that we can have a sysctl tunnable for > > > > this patch > > > > > > > >> so we could do more tests without recompiling kernel. > > > > > > > > Thanks for testing! > > > > > > > > > > > > > > > > @gallatin: can you come up with a patch that is acceptable > > > > for Netflix > > > > > > > > and allows to mitigate the performance regression. > > > > > > > > > > > > > > Ideally, tcphpts could enable this automatically when it > > > > starts to be > > > > > > > used (enough?), but a sysctl could select auto/on/off. > > > > > > There is already a well-known mechanism to request execution of > the > > > > > > specific function on return to userspace, namely AST. The > difference > > > > > > with the current hack is that the execution is requested for one > > > >
Re: Request for Testing: TCP RACK
The entire point is to *NOT* go through the overhead of scheduling something asynchronously, but to take advantage of the fact that a user/kernel transition is going to trash the cache anyway. In the common case of a system which has less than the threshold number of connections , we access the tcp_hpts_softclock function pointer, make one function call, and access hpts_that_need_softclock, and then return. So that's 2 variables and a function call. I think it would be preferable to avoid that call, and to move the declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they are in the same cacheline. Then we'd be hitting just a single line in the common case. (I've made comments on the review to that effect). Also, I wonder if the threshold could get higher by default, so that hpts is never called in this context unless we're to the point where we're scheduling thousands of runs of the hpts thread (and taking all those clock interrupts). Drew On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > > Ok I have created > > > > https://reviews.freebsd.org/D44420 > > > > > > To address the issue. I also attach a short version of the patch that Nuno > > can try and validate > > > > it works. Drew you may want to try this and validate the optimization does > > kick in since I can > > > > only now test that it does not on my local box :) > The patch still causes access to all cpu's cachelines on each userret. > It would be much better to inc/check the threshold and only schedule the > call when exceeded. Then the call can occur in some dedicated context, > like per-CPU thread, instead of userret. > > > > > > > R > > > > > > > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > > No. The goal is to run on every return to userspace for every thread. > > > > > > Drew > > > > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > > > > I got the idea from > > > > > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > > > > > The gist is that the TCP pacing stuff needs to run frequently, and > > > > > rather than run it out of a clock interrupt, its more efficient to run > > > > > it out of a system call context at just the point where we return to > > > > > userspace and the cache is trashed anyway. The current implementation > > > > > is fine for our workload, but probably not idea for a generic system. > > > > > Especially one where something is banging on system calls. > > > > > > > > > > Ast's could be the right tool for this, but I'm super unfamiliar with > > > > > them, and I can't find any docs on them. > > > > > > > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > > > > > what's happening here? > > > > This call would need some AST number added, and then it registers the > > > > ast to run on next return to userspace, for the current thread. > > > > > > > > Is it enough? > > > > > > > > > > Drew > > > > > > > > > > > > > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > > > > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > > > > > > > > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira > > > > wrote: > > > > > > > >> > > > > > > > >> Hello all! > > > > > > > >> > > > > > > > >> It works just fine! > > > > > > > >> System performance is OK. > > > > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > > > > >> > > > > > > > >> --- > > > > > > > >> net.inet.tcp.functions_available: > > > > > > > >> Stack D > > > > AliasPCB count > > > > > > > >> freebsd freebsd 0 > > > > > > > >> rack* > > > > rack 38 > > > > > > > >> --- > > > > > > > >> > > > > > > > >> It would be so nice that we can have a sysctl tunnable for > > > > this patch > > > > > > > >> so we could do more tests without recompiling kernel. > > > > > > > > Thanks for testing! > > > > > > > > > > > > > > > > @gallatin: can you come up with a patch that is acceptable > > > > for Netflix > > > > > > > > and allows to mitigate the performance regression. > > > > > > > > > > > > > > Ideally, tcphpts could enable this automatically when it > > > > starts to be > > > > > > > used (enough?), but a sysctl could select auto/on/off. > > > > > > There is already a well-known mechanism to request execution of the > > > > > > specific function on return to userspace, namely AST. The > > > > > > difference > > > > > > with the current hack is that the execution is requested for one > > > > callback > > > > > > in the context of the specific thread. > > > > > > > > > > > > Still, it might be worth a try to use it; what is the reason to > > > > hit a thread > > > > > > that
Re: Request for Testing: TCP RACK
On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > Ok I have created > > https://reviews.freebsd.org/D44420 > > > To address the issue. I also attach a short version of the patch that Nuno > can try and validate > > it works. Drew you may want to try this and validate the optimization does > kick in since I can > > only now test that it does not on my local box :) The patch still causes access to all cpu's cachelines on each userret. It would be much better to inc/check the threshold and only schedule the call when exceeded. Then the call can occur in some dedicated context, like per-CPU thread, instead of userret. > > > R > > > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > No. The goal is to run on every return to userspace for every thread. > > > > Drew > > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > > > I got the idea from > > > > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > > > > The gist is that the TCP pacing stuff needs to run frequently, and > > > > rather than run it out of a clock interrupt, its more efficient to run > > > > it out of a system call context at just the point where we return to > > > > userspace and the cache is trashed anyway. The current implementation > > > > is fine for our workload, but probably not idea for a generic system. > > > > Especially one where something is banging on system calls. > > > > > > > > Ast's could be the right tool for this, but I'm super unfamiliar with > > > > them, and I can't find any docs on them. > > > > > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > > > > what's happening here? > > > This call would need some AST number added, and then it registers the > > > ast to run on next return to userspace, for the current thread. > > > > > > Is it enough? > > > > > > > > Drew > > > > > > > > > > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > > > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > > > > > > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira > > > wrote: > > > > > > >> > > > > > > >> Hello all! > > > > > > >> > > > > > > >> It works just fine! > > > > > > >> System performance is OK. > > > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > > > >> > > > > > > >> --- > > > > > > >> net.inet.tcp.functions_available: > > > > > > >> Stack D > > > Alias PCB count > > > > > > >> freebsd freebsd 0 > > > > > > >> rack * > > > rack 38 > > > > > > >> --- > > > > > > >> > > > > > > >> It would be so nice that we can have a sysctl tunnable for > > > this patch > > > > > > >> so we could do more tests without recompiling kernel. > > > > > > > Thanks for testing! > > > > > > > > > > > > > > @gallatin: can you come up with a patch that is acceptable > > > for Netflix > > > > > > > and allows to mitigate the performance regression. > > > > > > > > > > > > Ideally, tcphpts could enable this automatically when it > > > starts to be > > > > > > used (enough?), but a sysctl could select auto/on/off. > > > > > There is already a well-known mechanism to request execution of the > > > > > specific function on return to userspace, namely AST. The difference > > > > > with the current hack is that the execution is requested for one > > > callback > > > > > in the context of the specific thread. > > > > > > > > > > Still, it might be worth a try to use it; what is the reason to > > > hit a thread > > > > > that does not do networking, with TCP processing? > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > Best regards > > > > > > > Michael > > > > > > >> > > > > > > >> Thanks all! > > > > > > >> Really happy here :) > > > > > > >> > > > > > > >> Cheers, > > > > > > >> > > > > > > >> Nuno Teixeira escreveu (domingo, > > > 17/03/2024 à(s) 20:26): > > > > > > >>> > > > > > > >>> Hello, > > > > > > >>> > > > > > > I don't have the full context, but it seems like the > > > complaint is a performance regression in bonnie++ and perhaps other > > > things when tcp_hpts is loaded, even when it is not used. Is that > > > correct? > > > > > > > > > > > > If so, I suspect its because we drive the > > > tcp_hpts_softclock() routine from userret(), in order to avoid tons > > > of timer interrupts and context switches. To test this theory, you > > > could apply a patch like: > > > > > > >>> > > > > > > >>> It's affecting overall system performance, bonnie was just > > > a way to > > > > > > >>> get some numbers to compare. > > > > > > >>> > > > > > > >>> Tomorrow I will test patch. > > > > > > >>> > > > > > > >>> Thanks! > > > > > > >>> > > > > > > >>> -- > > > > > > >>> Nuno Teixeira > > > > > > >>> FreeBSD Committer (ports) > >
Re: Request for Testing: TCP RACK
No. The goal is to run on every return to userspace for every thread. Drew On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > I got the idea from > > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > > The gist is that the TCP pacing stuff needs to run frequently, and > > rather than run it out of a clock interrupt, its more efficient to run > > it out of a system call context at just the point where we return to > > userspace and the cache is trashed anyway. The current implementation > > is fine for our workload, but probably not idea for a generic system. > > Especially one where something is banging on system calls. > > > > Ast's could be the right tool for this, but I'm super unfamiliar with > > them, and I can't find any docs on them. > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > > what's happening here? > This call would need some AST number added, and then it registers the > ast to run on next return to userspace, for the current thread. > > Is it enough? > > > > Drew > > > > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > > > >> > > > > >> Hello all! > > > > >> > > > > >> It works just fine! > > > > >> System performance is OK. > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > >> > > > > >> --- > > > > >> net.inet.tcp.functions_available: > > > > >> Stack D Alias > > > > >> PCB count > > > > >> freebsd freebsd 0 > > > > >> rack* rack 38 > > > > >> --- > > > > >> > > > > >> It would be so nice that we can have a sysctl tunnable for this patch > > > > >> so we could do more tests without recompiling kernel. > > > > > Thanks for testing! > > > > > > > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > > > > and allows to mitigate the performance regression. > > > > > > > > Ideally, tcphpts could enable this automatically when it starts to be > > > > used (enough?), but a sysctl could select auto/on/off. > > > There is already a well-known mechanism to request execution of the > > > specific function on return to userspace, namely AST. The difference > > > with the current hack is that the execution is requested for one callback > > > in the context of the specific thread. > > > > > > Still, it might be worth a try to use it; what is the reason to hit a > > > thread > > > that does not do networking, with TCP processing? > > > > > > > > > > > Mike > > > > > > > > > Best regards > > > > > Michael > > > > >> > > > > >> Thanks all! > > > > >> Really happy here :) > > > > >> > > > > >> Cheers, > > > > >> > > > > >> Nuno Teixeira escreveu (domingo, 17/03/2024 > > > > >> à(s) 20:26): > > > > >>> > > > > >>> Hello, > > > > >>> > > > > I don't have the full context, but it seems like the complaint is > > > > a performance regression in bonnie++ and perhaps other things when > > > > tcp_hpts is loaded, even when it is not used. Is that correct? > > > > > > > > If so, I suspect its because we drive the tcp_hpts_softclock() > > > > routine from userret(), in order to avoid tons of timer interrupts > > > > and context switches. To test this theory, you could apply a > > > > patch like: > > > > >>> > > > > >>> It's affecting overall system performance, bonnie was just a way to > > > > >>> get some numbers to compare. > > > > >>> > > > > >>> Tomorrow I will test patch. > > > > >>> > > > > >>> Thanks! > > > > >>> > > > > >>> -- > > > > >>> Nuno Teixeira > > > > >>> FreeBSD Committer (ports) > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> Nuno Teixeira > > > > >> FreeBSD Committer (ports) > > > > > > > >
Re: Request for Testing: TCP RACK
On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > I got the idea from > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > The gist is that the TCP pacing stuff needs to run frequently, and > rather than run it out of a clock interrupt, its more efficient to run > it out of a system call context at just the point where we return to > userspace and the cache is trashed anyway. The current implementation > is fine for our workload, but probably not idea for a generic system. > Especially one where something is banging on system calls. > > Ast's could be the right tool for this, but I'm super unfamiliar with > them, and I can't find any docs on them. > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > what's happening here? This call would need some AST number added, and then it registers the ast to run on next return to userspace, for the current thread. Is it enough? > > Drew > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > > >> > > > >> Hello all! > > > >> > > > >> It works just fine! > > > >> System performance is OK. > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > >> > > > >> --- > > > >> net.inet.tcp.functions_available: > > > >> Stack D AliasPCB > > > >> count > > > >> freebsd freebsd 0 > > > >> rack* rack 38 > > > >> --- > > > >> > > > >> It would be so nice that we can have a sysctl tunnable for this patch > > > >> so we could do more tests without recompiling kernel. > > > > Thanks for testing! > > > > > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > > > and allows to mitigate the performance regression. > > > > > > Ideally, tcphpts could enable this automatically when it starts to be > > > used (enough?), but a sysctl could select auto/on/off. > > There is already a well-known mechanism to request execution of the > > specific function on return to userspace, namely AST. The difference > > with the current hack is that the execution is requested for one callback > > in the context of the specific thread. > > > > Still, it might be worth a try to use it; what is the reason to hit a thread > > that does not do networking, with TCP processing? > > > > > > > > Mike > > > > > > > Best regards > > > > Michael > > > >> > > > >> Thanks all! > > > >> Really happy here :) > > > >> > > > >> Cheers, > > > >> > > > >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) > > > >> 20:26): > > > >>> > > > >>> Hello, > > > >>> > > > I don't have the full context, but it seems like the complaint is a > > > performance regression in bonnie++ and perhaps other things when > > > tcp_hpts is loaded, even when it is not used. Is that correct? > > > > > > If so, I suspect its because we drive the tcp_hpts_softclock() > > > routine from userret(), in order to avoid tons of timer interrupts > > > and context switches. To test this theory, you could apply a patch > > > like: > > > >>> > > > >>> It's affecting overall system performance, bonnie was just a way to > > > >>> get some numbers to compare. > > > >>> > > > >>> Tomorrow I will test patch. > > > >>> > > > >>> Thanks! > > > >>> > > > >>> -- > > > >>> Nuno Teixeira > > > >>> FreeBSD Committer (ports) > > > >> > > > >> > > > >> > > > >> -- > > > >> Nuno Teixeira > > > >> FreeBSD Committer (ports) > > > > >
Re: Request for Testing: TCP RACK
I got the idea from https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf The gist is that the TCP pacing stuff needs to run frequently, and rather than run it out of a clock interrupt, its more efficient to run it out of a system call context at just the point where we return to userspace and the cache is trashed anyway. The current implementation is fine for our workload, but probably not idea for a generic system. Especially one where something is banging on system calls. Ast's could be the right tool for this, but I'm super unfamiliar with them, and I can't find any docs on them. Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to what's happening here? Drew On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > >> > > >> Hello all! > > >> > > >> It works just fine! > > >> System performance is OK. > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > >> > > >> --- > > >> net.inet.tcp.functions_available: > > >> Stack D AliasPCB > > >> count > > >> freebsd freebsd 0 > > >> rack* rack 38 > > >> --- > > >> > > >> It would be so nice that we can have a sysctl tunnable for this patch > > >> so we could do more tests without recompiling kernel. > > > Thanks for testing! > > > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > > and allows to mitigate the performance regression. > > > > Ideally, tcphpts could enable this automatically when it starts to be > > used (enough?), but a sysctl could select auto/on/off. > There is already a well-known mechanism to request execution of the > specific function on return to userspace, namely AST. The difference > with the current hack is that the execution is requested for one callback > in the context of the specific thread. > > Still, it might be worth a try to use it; what is the reason to hit a thread > that does not do networking, with TCP processing? > > > > > Mike > > > > > Best regards > > > Michael > > >> > > >> Thanks all! > > >> Really happy here :) > > >> > > >> Cheers, > > >> > > >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) > > >> 20:26): > > >>> > > >>> Hello, > > >>> > > I don't have the full context, but it seems like the complaint is a > > performance regression in bonnie++ and perhaps other things when > > tcp_hpts is loaded, even when it is not used. Is that correct? > > > > If so, I suspect its because we drive the tcp_hpts_softclock() routine > > from userret(), in order to avoid tons of timer interrupts and context > > switches. To test this theory, you could apply a patch like: > > >>> > > >>> It's affecting overall system performance, bonnie was just a way to > > >>> get some numbers to compare. > > >>> > > >>> Tomorrow I will test patch. > > >>> > > >>> Thanks! > > >>> > > >>> -- > > >>> Nuno Teixeira > > >>> FreeBSD Committer (ports) > > >> > > >> > > >> > > >> -- > > >> Nuno Teixeira > > >> FreeBSD Committer (ports) > > >
Re: Request for Testing: TCP RACK
On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > >> > >> Hello all! > >> > >> It works just fine! > >> System performance is OK. > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > >> > >> --- > >> net.inet.tcp.functions_available: > >> Stack D AliasPCB > >> count > >> freebsd freebsd 0 > >> rack* rack 38 > >> --- > >> > >> It would be so nice that we can have a sysctl tunnable for this patch > >> so we could do more tests without recompiling kernel. > > Thanks for testing! > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > and allows to mitigate the performance regression. > > Ideally, tcphpts could enable this automatically when it starts to be > used (enough?), but a sysctl could select auto/on/off. There is already a well-known mechanism to request execution of the specific function on return to userspace, namely AST. The difference with the current hack is that the execution is requested for one callback in the context of the specific thread. Still, it might be worth a try to use it; what is the reason to hit a thread that does not do networking, with TCP processing? > > Mike > > > Best regards > > Michael > >> > >> Thanks all! > >> Really happy here :) > >> > >> Cheers, > >> > >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) > >> 20:26): > >>> > >>> Hello, > >>> > I don't have the full context, but it seems like the complaint is a > performance regression in bonnie++ and perhaps other things when > tcp_hpts is loaded, even when it is not used. Is that correct? > > If so, I suspect its because we drive the tcp_hpts_softclock() routine > from userret(), in order to avoid tons of timer interrupts and context > switches. To test this theory, you could apply a patch like: > >>> > >>> It's affecting overall system performance, bonnie was just a way to > >>> get some numbers to compare. > >>> > >>> Tomorrow I will test patch. > >>> > >>> Thanks! > >>> > >>> -- > >>> Nuno Teixeira > >>> FreeBSD Committer (ports) > >> > >> > >> > >> -- > >> Nuno Teixeira > >> FreeBSD Committer (ports) >
Re: Request for Testing: TCP RACK
On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote:
> ...
> >> It would be so nice that we can have a sysctl tunnable for this patch
> >> so we could do more tests without recompiling kernel.
> > Thanks for testing!
> >
> > @gallatin: can you come up with a patch that is acceptable for Netflix
> > and allows to mitigate the performance regression.
>
> Ideally, tcphpts could enable this automatically when it starts to be
> used (enough?), but a sysctl could select auto/on/off.
>
> Mike
>
Or (at some risk of over-{complicat,engineer}ing things), perhaps
sysctl/tunable for high- and low-water marks?
Peace,
david (who is quite unlikely to be writing the code, so "grain of salt")
--
David H. Wolfskill da...@catwhisker.org
Alexey Navalny was a courageous man; Putin has made him a martyr.
See https://www.catwhisker.org/~david/publickey.gpg for my public key.
signature.asc
Description: PGP signature
Re: Request for Testing: TCP RACK
On 18 Mar 2024, at 7:04, tue...@freebsd.org wrote: >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: >> >> Hello all! >> >> It works just fine! >> System performance is OK. >> Using patch on main-n268841-b0aaf8beb126(-dirty). >> >> --- >> net.inet.tcp.functions_available: >> Stack D AliasPCB count >> freebsd freebsd 0 >> rack* rack 38 >> --- >> >> It would be so nice that we can have a sysctl tunnable for this patch >> so we could do more tests without recompiling kernel. > Thanks for testing! > > @gallatin: can you come up with a patch that is acceptable for Netflix > and allows to mitigate the performance regression. Ideally, tcphpts could enable this automatically when it starts to be used (enough?), but a sysctl could select auto/on/off. Mike > Best regards > Michael >> >> Thanks all! >> Really happy here :) >> >> Cheers, >> >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) >> 20:26): >>> >>> Hello, >>> I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct? If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like: >>> >>> It's affecting overall system performance, bonnie was just a way to >>> get some numbers to compare. >>> >>> Tomorrow I will test patch. >>> >>> Thanks! >>> >>> -- >>> Nuno Teixeira >>> FreeBSD Committer (ports) >> >> >> >> -- >> Nuno Teixeira >> FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > Hello all! > > It works just fine! > System performance is OK. > Using patch on main-n268841-b0aaf8beb126(-dirty). > > --- > net.inet.tcp.functions_available: > Stack D AliasPCB count > freebsd freebsd 0 > rack* rack 38 > --- > > It would be so nice that we can have a sysctl tunnable for this patch > so we could do more tests without recompiling kernel. Thanks for testing! @gallatin: can you come up with a patch that is acceptable for Netflix and allows to mitigate the performance regression. Best regards Michael > > Thanks all! > Really happy here :) > > Cheers, > > Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) 20:26): >> >> Hello, >> >>> I don't have the full context, but it seems like the complaint is a >>> performance regression in bonnie++ and perhaps other things when tcp_hpts >>> is loaded, even when it is not used. Is that correct? >>> >>> If so, I suspect its because we drive the tcp_hpts_softclock() routine from >>> userret(), in order to avoid tons of timer interrupts and context switches. >>> To test this theory, you could apply a patch like: >> >> It's affecting overall system performance, bonnie was just a way to >> get some numbers to compare. >> >> Tomorrow I will test patch. >> >> Thanks! >> >> -- >> Nuno Teixeira >> FreeBSD Committer (ports) > > > > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
Hello all! It works just fine! System performance is OK. Using patch on main-n268841-b0aaf8beb126(-dirty). --- net.inet.tcp.functions_available: Stack D AliasPCB count freebsd freebsd 0 rack* rack 38 --- It would be so nice that we can have a sysctl tunnable for this patch so we could do more tests without recompiling kernel. Thanks all! Really happy here :) Cheers, Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) 20:26): > > Hello, > > > I don't have the full context, but it seems like the complaint is a > > performance regression in bonnie++ and perhaps other things when tcp_hpts > > is loaded, even when it is not used. Is that correct? > > > > If so, I suspect its because we drive the tcp_hpts_softclock() routine from > > userret(), in order to avoid tons of timer interrupts and context switches. > > To test this theory, you could apply a patch like: > > It's affecting overall system performance, bonnie was just a way to > get some numbers to compare. > > Tomorrow I will test patch. > > Thanks! > > -- > Nuno Teixeira > FreeBSD Committer (ports) -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
Hello, > I don't have the full context, but it seems like the complaint is a > performance regression in bonnie++ and perhaps other things when tcp_hpts is > loaded, even when it is not used. Is that correct? > > If so, I suspect its because we drive the tcp_hpts_softclock() routine from > userret(), in order to avoid tons of timer interrupts and context switches. > To test this theory, you could apply a patch like: It's affecting overall system performance, bonnie was just a way to get some numbers to compare. Tomorrow I will test patch. Thanks! -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 17. Mar 2024, at 16:39, Drew Gallatin wrote: > > I don't have the full context, but it seems like the complaint is a > performance regression in bonnie++ and perhaps other things when tcp_hpts is > loaded, even when it is not used. Is that correct? Correct. > > If so, I suspect its because we drive the tcp_hpts_softclock() routine from > userret(), in order to avoid tons of timer interrupts and context switches. > To test this theory, you could apply a patch like: > > diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c > index e9a16cd0b36e..54b540c97123 100644 > --- a/sys/kern/subr_trap.c > +++ b/sys/kern/subr_trap.c > @@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame) > * Software Timer Support for Network Processing" > * by Mohit Aron and Peter Druschel. > */ > - tcp_hpts_softclock(); > + /*tcp_hpts_softclock();*/ >/* > * Let the scheduler adjust our priority etc. > */ > > > If that fixes it, I suspect we should either make this hook optional for > casual users of tcp_hpts(), or add some kind of "last called" timestamp to > prevent it being called over and over and over on workloads which are syscall > heavy. Nuno, can you test that? > > Note that for non-casual users of hpts (like Netflix, with hundreds of > thousands of TCP connections managed by hpts), this call is a huge win, so I > think we'd prefer that it remain in some form. Sure. Best regards Michael > > Drew
Re: Request for Testing: TCP RACK
On Sun, 17 Mar 2024 11:40:54 -0400 "Drew Gallatin" wrote: > Resending with the patch as an attachment. > > Drew > > On Sun, Mar 17, 2024, at 11:39 AM, Drew Gallatin wrote: > > I don't have the full context, but it seems like the complaint is a > > performance regression in bonnie++ and perhaps other things when tcp_hpts > > is loaded, even when it is not used. Is that correct? > > > > If so, I suspect its because we drive the tcp_hpts_softclock() routine from > > userret(), in order to avoid tons of timer interrupts and context switches. > > To test this theory, you could apply a patch like: > > > > diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c > > index e9a16cd0b36e..54b540c97123 100644 > > --- a/sys/kern/subr_trap.c > > +++ b/sys/kern/subr_trap.c > > @@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame) > > * Software Timer Support for Network Processing" > > * by Mohit Aron and Peter Druschel. > > */ > > - tcp_hpts_softclock(); > > + /*tcp_hpts_softclock();*/ > > /* > > * Let the scheduler adjust our priority etc. > > */ > > > > > > If that fixes it, I suspect we should either make this hook optional for > > casual users of tcp_hpts(), or add some kind of "last called" timestamp to > > prevent it being called over and over and over on workloads which are > > syscall heavy. > > > > Note that for non-casual users of hpts (like Netflix, with hundreds of > > thousands of TCP connections managed by hpts), this call is a huge win, so > > I think we'd prefer that it remain in some form. > > > > Drew Controlled via RW or RWTUN sysctl/tunable? -- Tomoaki AOKI
Re: Request for Testing: TCP RACK
Resending with the patch as an attachment. Drew On Sun, Mar 17, 2024, at 11:39 AM, Drew Gallatin wrote: > I don't have the full context, but it seems like the complaint is a > performance regression in bonnie++ and perhaps other things when tcp_hpts is > loaded, even when it is not used. Is that correct? > > If so, I suspect its because we drive the tcp_hpts_softclock() routine from > userret(), in order to avoid tons of timer interrupts and context switches. > To test this theory, you could apply a patch like: > > diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c > index e9a16cd0b36e..54b540c97123 100644 > --- a/sys/kern/subr_trap.c > +++ b/sys/kern/subr_trap.c > @@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame) > * Software Timer Support for Network Processing" > * by Mohit Aron and Peter Druschel. > */ > - tcp_hpts_softclock(); > + /*tcp_hpts_softclock();*/ > /* > * Let the scheduler adjust our priority etc. > */ > > > If that fixes it, I suspect we should either make this hook optional for > casual users of tcp_hpts(), or add some kind of "last called" timestamp to > prevent it being called over and over and over on workloads which are syscall > heavy. > > Note that for non-casual users of hpts (like Netflix, with hundreds of > thousands of TCP connections managed by hpts), this call is a huge win, so I > think we'd prefer that it remain in some form. > > Drew > nerf_tcp_hpts_softclock.diff Description: Binary data
Re: Request for Testing: TCP RACK
I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct? If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like: diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c index e9a16cd0b36e..54b540c97123 100644 --- a/sys/kern/subr_trap.c +++ b/sys/kern/subr_trap.c @@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame) * Software Timer Support for Network Processing" * by Mohit Aron and Peter Druschel. */ - tcp_hpts_softclock(); + /*tcp_hpts_softclock();*/ /* * Let the scheduler adjust our priority etc. */ If that fixes it, I suspect we should either make this hook optional for casual users of tcp_hpts(), or add some kind of "last called" timestamp to prevent it being called over and over and over on workloads which are syscall heavy. Note that for non-casual users of hpts (like Netflix, with hundreds of thousands of TCP connections managed by hpts), this call is a huge win, so I think we'd prefer that it remain in some form. Drew
Re: Request for Testing: TCP RACK
Hello, > > - I can't remember better tests to do as I can feel the entire OS is > > being slow, without errors, just slow. > This is interesting. It seems a consequence on loading TCPHPTS, not actually > using it. Exactly, just loading module and not using it by setting sysctl. > I have CCed Drew and Randall, who know much more about HPTS and might have > follow up questions. I'll bring the issue up in the FreeBSD transport call > next Thursday. > > What hardware are you using? Laptop: Legion 5-15IMH05 (Lenovo) - Type 82AU Thanks! -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 16. Mar 2024, at 21:29, Nuno Teixeira wrote: > >> Just to double check: you only load the tcp_rack. You don't run >> sysctl net.inet.tcp.functions_default=rack > > I'm not using sysctl, just loading module. And you also don't have net.inet.tcp.functions_default=rack in /etc/sysctl.conf This means that no TCP connection will actually use the RACK stack. > >> What does "poudriere testport net/gitup" do? Only build stuff or does is >> also download something? >> >> What does bonnie++ do? > > poudriere is for testing ports and it uses jails to build stuff. > It have restrict access to net to fetch distfiles (not the case as > distfile is present on disk) > > bonnie++ is a disk benchmark Thanks for clarifying. > >> Could you reboot the system, run the test, do kldload tcphpts, run >> the test again, do kldload tcp_rack, and run it again. > > The previous test [2] was obtained by loading tcp_rack > > Now I will post results for test [3] by loading (only) tcphpts module. Great. This makes sure that the degradation is related to TCPHPTS and not the RACK stack. > > [3] kldload tcphpts: > > ==> poudriere testport net/gitup: > 55.26s real 5.23s user 1m19.91s sys So this test goes up from 11 seconds to 55 seconds... > > ==> bonnie++ > Version 1.98 --Sequential Output-- --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > leg.home32G 12k 99 73.0m 99 51.1m 99 27k 99 128m 99 8038 > 2194 > Latency 1763ms 194ms 23979us 431ms1267us2776us > Version 1.98 --Sequential Create-- Random > Create > leg.home-Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >16 7017.934752 98 21243.823180 100 7780.918458 99 > 9281.48 98 21368.647657 100 7457.828733 99 > Latency 3015us 220us2398us1106us 386us2473us > > Summary: > > - I can't remember better tests to do as I can feel the entire OS is > being slow, without errors, just slow. This is interesting. It seems a consequence on loading TCPHPTS, not actually using it. I have CCed Drew and Randall, who know much more about HPTS and might have follow up questions. I'll bring the issue up in the FreeBSD transport call next Thursday. What hardware are you using? Best regards Michael > - Approx. results as test [2] > > > > > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
On Sun, 19 Nov 2023 04:01:01 +0900 Tomoaki AOKI wrote: > On Sat, 18 Nov 2023 09:50:43 +0100 > tue...@freebsd.org wrote: > > > > On Nov 18, 2023, at 00:37, Tomoaki AOKI wrote: > > > > > > On Fri, 17 Nov 2023 18:51:05 +0100 > > > tue...@freebsd.org wrote: > > > > > >>> On Nov 17, 2023, at 17:06, Johan Hendriks > > >>> wrote: > > >>> > > >>> I am running the rack stack for quiet some time now on a baremetal > > >>> machiene and never had problems. > > >>> Also use pf. This is a test machine so not a lot happening on it. > > >>> > > >>> Are there any thing we can test? Do we have some test scripts we can > > >>> run? > > >> We are actually interested in feedback about using the stack in whatever > > >> use case you use TCP for. The stack has been tested with the Netflix use > > >> case, but not so much with others. That is why we ask for broader > > >> testing. > > >> > > >> Best regards > > >> Michael > > > > > > Are there any difference regarding with performance between main and > > > stable/14? If so, please ignore below. > > > > > > I have stable/14 environment which is configured to be able to switch > > > to TCP RACK and experienced huge performance loss when writing > > > a large file to smbfs share on commercial NAS. CC is cubic. > > > Testing large archive on the smbfs share doesn't seem to be > > > affected. > > > > > > Comparison between default (freebsd) and rack TCP stack using > > > sysutils/clone on stable/14 at commit 7d1321288ad9, amd64. > > > Last 3 lines of outputs from clone (upload to NAS) are shown. > > Thank you very much for testing. This is what we are looking > > for. Would it be possible to repeat the test using NewReno as > > the CC? > > > > Best regards > > Michael > > Sure. Here we go! > > sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: freebsd > sysctl net.inet.tcp.cc.algorithm > net.inet.tcp.cc.algorithm: newreno > Umounted and remounted smbfs share. > > 1 item copied, 2343.1 MB in 37.65 s -- 62.2 MB/s > Leaked memory: 0 bytes > No errors occured. > > > sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: rack > Umounted and remounted smbfs share. > > 1 item copied, 2343.1 MB in 905.17 s -- 2.6 MB/s > Leaked memory: 0 bytes > No errors occured. > > > sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: freebsd > Without umount and remount to reproduce previous oddness, maybe caused > by keep-alive. > > 1 item copied, 2343.1 MB in 897.67 s -- 2.6 MB/s > Leaked memory: 0 bytes > No errors occured. > > > Umounted and remounted, without change for CC and TCP stack. > > 1 item copied, 2343.1 MB in 37.43 s -- 62.6 MB/s > Leaked memory: 0 bytes > No errors occured. > > > All test are proceeded simultaneously. So the last one is with > CC=newreno and TCP stack=freebsd. > > Not exactly recorded, but testing transferred file by > diff -u -p -N was roughly 30MB/sec, while roughly 25MB/sec with > CC=cubic. > > > > > > > > Before switching to rack: > > > 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s > > > Leaked memory: 0 bytes > > > No errors occured. > > > > > > Unmount the smbfs share, switch to rack, and after remount: > > > 1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s > > > Leaked memory: 0 bytes > > > No errors occured. > > > > > > Switch back to freebsd (default) without unmounting: > > > 1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s > > > Leaked memory: 0 bytes > > > No errors occured. > > > > > > Unmount and remount the smbfs share: > > > 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s > > > Leaked memory: 0 bytes > > > No errors occured. > > > > > > > > > Regards. > > > > > > -- > > > Tomoaki AOKI > > > -- > Tomoaki AOKI Just a follow up. This situation does not changed on stable/14, amd64 until commit a15b8e32942b2cbf70c7fc71e9c82d2b292e82c3. (So I've never reported again until now. Not tested on every updates.) tcphpts.ko is loaded. Note that options RATELIMIT and options TCPHPTS are dropped when changes to RACK is MFC'ed and added tcphpts_load="YES" in /boot/loader.conf instaad. Another note: Differences between CC=newreno and CC=cubic on testing transferred file seems to be just a fudge factor. both vaies mostly between the two values which I've previously reported. -- Tomoaki AOKI
Re: Request for Testing: TCP RACK
> Just to double check: you only load the tcp_rack. You don't run > sysctl net.inet.tcp.functions_default=rack I'm not using sysctl, just loading module. > What does "poudriere testport net/gitup" do? Only build stuff or does is > also download something? > > What does bonnie++ do? poudriere is for testing ports and it uses jails to build stuff. It have restrict access to net to fetch distfiles (not the case as distfile is present on disk) bonnie++ is a disk benchmark > Could you reboot the system, run the test, do kldload tcphpts, run > the test again, do kldload tcp_rack, and run it again. The previous test [2] was obtained by loading tcp_rack Now I will post results for test [3] by loading (only) tcphpts module. [3] kldload tcphpts: ==> poudriere testport net/gitup: 55.26s real 5.23s user 1m19.91s sys ==> bonnie++ Version 1.98 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP leg.home32G 12k 99 73.0m 99 51.1m 99 27k 99 128m 99 8038 2194 Latency 1763ms 194ms 23979us 431ms1267us2776us Version 1.98 --Sequential Create-- Random Create leg.home-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7017.934752 98 21243.823180 100 7780.918458 99 9281.48 98 21368.647657 100 7457.828733 99 Latency 3015us 220us2398us1106us 386us2473us Summary: - I can't remember better tests to do as I can feel the entire OS is being slow, without errors, just slow. - Approx. results as test [2] Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 16. Mar 2024, at 20:06, Nuno Teixeira wrote: > >>> Will update amd64 laptop to main-n268827-75464941dc17 (Mar 16) and test it. >> Please do so... > > @main-n268827-75464941dc17 GENERIC-NODEBUG amd64 > > Ok, I think I have here some numbers related to disk performance being > affected by tcp_rack that somehow is messing with something. > > NOTES: > - test [1] was done after a boot without tcp_rack in loader.conf and > [2] tcp_rack was loaded manually with kldload (without rebooting) > - After unloading tcp_rack, same results as [2] > - Cannot unload tcptpts: device busy tcphpts does not support unloading. So that is fine. Just to double check: you only load the tcp_rack. You don't run sysctl net.inet.tcp.functions_default=rack This means you load RACK, but you don't use it. What does "poudriere testport net/gitup" do? Only build stuff or does is also download something? What does bonnie++ do? Could you reboot the system, run the test, do kldload tcphpts, run the test again, do kldload tcp_rack, and run it again. I'm wondering if tcphpts is affecting the measurements... Thanks for testing. Best regards Michael > > [1] without tcp_rack loaded: > > ==> poudriere testport net/gitup: > 11.16s real 5.35s user 6.35s sys > > ==> bonnie++: > Version 1.98 --Sequential Output-- --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > leg.home32G 25k 99 105m 99 88.7m 99 77k 99 198m 99 12716 > 1784 > Latency 351ms1793us2340us 241ms 638us2514us > Version 1.98 --Sequential Create-- Random > Create > leg.home-Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >16 14908.273230 97 + +++ 6878.516657 99 > 15194.808063 97 + +++ 8019.731670 99 > Latency 1108us 182us2228us1013us 152us2424us > > [2] kldload tcp_rack: > > ==> poudriere testport net/gitup: > 1m0.54s real4.98s user 1m31.38s sys > > ==> bonnie++: > Version 1.98 --Sequential Output-- --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > leg.home32G 14k 99 78.0m 99 46.6m 99 25k 99 120m 99 6688 > 2161 > Latency 676ms 18309us 76171us 385ms 924us2274us > Version 1.98 --Sequential Create-- Random > Create > leg.home-Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >16 8139.513260 96 19437.792294 99 5494.638508 99 > 8099.275425 96 19723.528878 99 6363.123671 99 > Latency 2982us 338us3072us1135us 591us3236us > > Cheers, > > > > > > > > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> > Will update amd64 laptop to main-n268827-75464941dc17 (Mar 16) and test it. > Please do so... @main-n268827-75464941dc17 GENERIC-NODEBUG amd64 Ok, I think I have here some numbers related to disk performance being affected by tcp_rack that somehow is messing with something. NOTES: - test [1] was done after a boot without tcp_rack in loader.conf and [2] tcp_rack was loaded manually with kldload (without rebooting) - After unloading tcp_rack, same results as [2] - Cannot unload tcptpts: device busy [1] without tcp_rack loaded: ==> poudriere testport net/gitup: 11.16s real 5.35s user 6.35s sys ==> bonnie++: Version 1.98 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP leg.home32G 25k 99 105m 99 88.7m 99 77k 99 198m 99 12716 1784 Latency 351ms1793us2340us 241ms 638us2514us Version 1.98 --Sequential Create-- Random Create leg.home-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14908.273230 97 + +++ 6878.516657 99 15194.808063 97 + +++ 8019.731670 99 Latency 1108us 182us2228us1013us 152us2424us [2] kldload tcp_rack: ==> poudriere testport net/gitup: 1m0.54s real4.98s user 1m31.38s sys ==> bonnie++: Version 1.98 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP leg.home32G 14k 99 78.0m 99 46.6m 99 25k 99 120m 99 6688 2161 Latency 676ms 18309us 76171us 385ms 924us2274us Version 1.98 --Sequential Create-- Random Create leg.home-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8139.513260 96 19437.792294 99 5494.638508 99 8099.275425 96 19723.528878 99 6363.123671 99 Latency 2982us 338us3072us1135us 591us3236us Cheers, -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 16. Mar 2024, at 15:00, Nuno Teixeira wrote: > >> If you load tcp_rack via kldload, tcphtps get loaded automatically. >> If you load if via /boot/loader.conf, you need to have >> tcphpts_load="YES" >> in addition to >> tcp_rack_load="YES" > > As of my tests, loader.conf: tcp_rack_load="YES" loads tcphtps.ko auto: > > 31 0x81f53000545e0 tcp_rack.ko > 42 0x81fa800014588 tcphpts.ko Interesting. Works on my arm64 machine, too. I think when I tested things for writing the article, I figured out that just doing tcp_rack_load="YES" in /boot/loader.conf was not working on a kernel configured without TCPHPTS. > > On aarch64 (rpi4) I didn't get any performance issues in > > main-n268730-96ad640178ea (Mar 8) > and > main-n268827-75464941dc17 (Mar 16) > > So it seems not related to rack commit e18b97bd63a8: Update to bring > the rack stack with all its fixes in. > > Will update amd64 laptop to main-n268827-75464941dc17 (Mar 16) and test it. Please do so... Best regards Michael > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> If you load tcp_rack via kldload, tcphtps get loaded automatically. > If you load if via /boot/loader.conf, you need to have > tcphpts_load="YES" > in addition to > tcp_rack_load="YES" As of my tests, loader.conf: tcp_rack_load="YES" loads tcphtps.ko auto: 31 0x81f53000545e0 tcp_rack.ko 42 0x81fa800014588 tcphpts.ko On aarch64 (rpi4) I didn't get any performance issues in main-n268730-96ad640178ea (Mar 8) and main-n268827-75464941dc17 (Mar 16) So it seems not related to rack commit e18b97bd63a8: Update to bring the rack stack with all its fixes in. Will update amd64 laptop to main-n268827-75464941dc17 (Mar 16) and test it. -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 16. Mar 2024, at 11:59, Nuno Teixeira wrote: > >>> Resuming I only need to add: >>> >>> options TCPHPTS >>> >>> Is this correct? >>> >> >> Yeah, that will probably fix it. According to a comment in >> /usr/src/sys/netinet/tcp_hpts.c it adds a high precision timer >> system for tcp, used by RACK and BBR. > > As tuexen said, on main, loader.conf: tcp_rack_load="YES" will load > tcphpts.ko as I am seing in my rpi4 right now. If you load tcp_rack via kldload, tcphtps get loaded automatically. If you load if via /boot/loader.conf, you need to have tcphpts_load="YES" in addition to tcp_rack_load="YES" Best regards Michael > I'm testing it and check its performance. > > I will test again on my amd64 laptop and run more tests too. > > > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> > Resuming I only need to add: > > > > options TCPHPTS > > > > Is this correct? > > > > Yeah, that will probably fix it. According to a comment in > /usr/src/sys/netinet/tcp_hpts.c it adds a high precision timer > system for tcp, used by RACK and BBR. As tuexen said, on main, loader.conf: tcp_rack_load="YES" will load tcphpts.ko as I am seing in my rpi4 right now. I'm testing it and check its performance. I will test again on my amd64 laptop and run more tests too. -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
On Sat, 16 Mar 2024 10:11:22 + Nuno Teixeira wrote: > (...) > > > These entries are missing in GENERIC: > > > > makeoptions WITH_EXTRA_TCP_STACKS=1 > > From > https://cgit.freebsd.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc > WITH_EXTRA_TCP_STACKS was dropped. > > > options TCPHPTS > > That's the missing option in GENERIC that could lead to my slow > opearations problem > > > options TCP_RACK > > Don't think I need this one as I will use kernel module instead of > building it in kernel. > > Resuming I only need to add: > > options TCPHPTS > > Is this correct? > Yeah, that will probably fix it. According to a comment in /usr/src/sys/netinet/tcp_hpts.c it adds a high precision timer system for tcp, used by RACK and BBR. -- Gary Jennejohn
Re: Request for Testing: TCP RACK
> On 16. Mar 2024, at 11:11, Nuno Teixeira wrote: > > (...) > >> These entries are missing in GENERIC: >> >> makeoptions WITH_EXTRA_TCP_STACKS=1 > > From > https://cgit.freebsd.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc > WITH_EXTRA_TCP_STACKS was dropped. > >> options TCPHPTS > > That's the missing option in GENERIC that could lead to my slow > opearations problem On main, this will be automatically loaded as a module when you load tcp_rack. > >> options TCP_RACK > > Don't think I need this one as I will use kernel module instead of > building it in kernel. Correct. > > Resuming I only need to add: > > options TCPHPTS > > Is this correct? No. Not on main. Best regards Michael > > Thanks, > > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
(...) > These entries are missing in GENERIC: > > makeoptions WITH_EXTRA_TCP_STACKS=1 >From >https://cgit.freebsd.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc WITH_EXTRA_TCP_STACKS was dropped. > options TCPHPTS That's the missing option in GENERIC that could lead to my slow opearations problem > options TCP_RACK Don't think I need this one as I will use kernel module instead of building it in kernel. Resuming I only need to add: options TCPHPTS Is this correct? Thanks, -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
On Sat, 16 Mar 2024 09:49:24 + Nuno Teixeira wrote: > Hello Gary, > > Nice that you found this. > > tcp_tack manual doesn't mention that we need extra config in kernel > but it loads module and it is shown in sysctl > net.inet.tcp.functions_available without errors. > > I will add missing config to GENERIC and see how it goes. > All the required information is here: https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/ -- Gary Jennejohn
Re: Request for Testing: TCP RACK
Hello Gary, Nice that you found this. tcp_tack manual doesn't mention that we need extra config in kernel but it loads module and it is shown in sysctl net.inet.tcp.functions_available without errors. I will add missing config to GENERIC and see how it goes. Cheers, Gary Jennejohn escreveu (sábado, 16/03/2024 à(s) 09:40): > > On Sat, 16 Mar 2024 09:41:13 +0100 > tue...@freebsd.org wrote: > > > > On 16. Mar 2024, at 08:57, Nuno Teixeira wrote: > > > > > > Hello all, > > > > > > On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've > > > noticed that all operations on OS was increased by 3 to 5 times > > > longer. > > > examples: > > > - firefox took way long time to open > > > - poudriere testport tooked up to 3 times longer to finished > > > > make.conf: > > KERNCONF=GENERIC-NODEBUG > > src.conf: > > WITH_MALLOC_PRODUCTION=yes > > > > tested on main-n268800-6a6ec90681cf > > > > > How did you enable the RACK stack? Does the poudriere involve > > network interaction? > > > > Interesting. RACK works for me: > > net.inet.tcp.functions_available: > Stack D AliasPCB count > freebsd freebsd 0 > rack* rack 23 > > I don't see any lags when starting/using FireFox or any other browser. > > Mail delivery (in/out) is also not affected. > > But GENERIC, which is loaded by GENERIC-NODEBUG, doesn't support RACK. > > These entries are missing in GENERIC: > > makeoptions WITH_EXTRA_TCP_STACKS=1 > options TCPHPTS > options TCP_RACK > > -- > Gary Jennejohn -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
Followed man tcp_rack: loader.conf: tcp_rack_load="YES" sysctl.conf: net.inet.tcp.functions_default=rack poudriere have restricted access to network, usually for fetch distfiles. escreveu (sábado, 16/03/2024 à(s) 08:41): > > > On 16. Mar 2024, at 08:57, Nuno Teixeira wrote: > > > > Hello all, > > > > On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've > > noticed that all operations on OS was increased by 3 to 5 times > > longer. > > examples: > > - firefox took way long time to open > > - poudriere testport tooked up to 3 times longer to finished > How did you enable the RACK stack? Does the poudriere involve > network interaction? > > Best regards > Michael > > > > make.conf: > > KERNCONF=GENERIC-NODEBUG > > src.conf: > > WITH_MALLOC_PRODUCTION=yes > > > > tested on main-n268800-6a6ec90681cf > > > > Thanks, > > > > escreveu (quinta, 14/03/2024 à(s) 10:51): > >> > >>> On 14. Mar 2024, at 11:04, Dag-Erling Smørgrav wrote: > >>> > >>> tue...@freebsd.org writes: > Gary Jennejohn writes: > > In the article we have option TCPHPTS and option TCP_RACK. But in > > /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and > > not option. > Thanks for reporting, that is a typo in the article. It should > always read options instead of option. > >>> > >>> It's not a typo, both spellings work, cf. config(5). > >> Thank you very much for the hint. I did not know this. I wrote > >> option in the article (for whatever reason) and tested the > >> configs using options... > >> > >> Best regards > >> Michael > >>> > >>> DES > >>> -- > >>> Dag-Erling Smørgrav - d...@freebsd.org > >> > > > > > > -- > > Nuno Teixeira > > FreeBSD Committer (ports) > -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
On Sat, 16 Mar 2024 09:41:13 +0100 tue...@freebsd.org wrote: > > On 16. Mar 2024, at 08:57, Nuno Teixeira wrote: > > > > Hello all, > > > > On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've > > noticed that all operations on OS was increased by 3 to 5 times > > longer. > > examples: > > - firefox took way long time to open > > - poudriere testport tooked up to 3 times longer to finished > > make.conf: > KERNCONF=GENERIC-NODEBUG > src.conf: > WITH_MALLOC_PRODUCTION=yes > > tested on main-n268800-6a6ec90681cf > > How did you enable the RACK stack? Does the poudriere involve > network interaction? > Interesting. RACK works for me: net.inet.tcp.functions_available: Stack D AliasPCB count freebsd freebsd 0 rack* rack 23 I don't see any lags when starting/using FireFox or any other browser. Mail delivery (in/out) is also not affected. But GENERIC, which is loaded by GENERIC-NODEBUG, doesn't support RACK. These entries are missing in GENERIC: makeoptions WITH_EXTRA_TCP_STACKS=1 options TCPHPTS options TCP_RACK -- Gary Jennejohn
Re: Request for Testing: TCP RACK
> On 16. Mar 2024, at 08:57, Nuno Teixeira wrote: > > Hello all, > > On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've > noticed that all operations on OS was increased by 3 to 5 times > longer. > examples: > - firefox took way long time to open > - poudriere testport tooked up to 3 times longer to finished How did you enable the RACK stack? Does the poudriere involve network interaction? Best regards Michael > > make.conf: > KERNCONF=GENERIC-NODEBUG > src.conf: > WITH_MALLOC_PRODUCTION=yes > > tested on main-n268800-6a6ec90681cf > > Thanks, > > escreveu (quinta, 14/03/2024 à(s) 10:51): >> >>> On 14. Mar 2024, at 11:04, Dag-Erling Smørgrav wrote: >>> >>> tue...@freebsd.org writes: Gary Jennejohn writes: > In the article we have option TCPHPTS and option TCP_RACK. But in > /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and > not option. Thanks for reporting, that is a typo in the article. It should always read options instead of option. >>> >>> It's not a typo, both spellings work, cf. config(5). >> Thank you very much for the hint. I did not know this. I wrote >> option in the article (for whatever reason) and tested the >> configs using options... >> >> Best regards >> Michael >>> >>> DES >>> -- >>> Dag-Erling Smørgrav - d...@freebsd.org >> > > > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
Hello all, On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've noticed that all operations on OS was increased by 3 to 5 times longer. examples: - firefox took way long time to open - poudriere testport tooked up to 3 times longer to finished make.conf: KERNCONF=GENERIC-NODEBUG src.conf: WITH_MALLOC_PRODUCTION=yes tested on main-n268800-6a6ec90681cf Thanks, escreveu (quinta, 14/03/2024 à(s) 10:51): > > > On 14. Mar 2024, at 11:04, Dag-Erling Smørgrav wrote: > > > > tue...@freebsd.org writes: > >> Gary Jennejohn writes: > >>> In the article we have option TCPHPTS and option TCP_RACK. But in > >>> /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and > >>> not option. > >> Thanks for reporting, that is a typo in the article. It should > >> always read options instead of option. > > > > It's not a typo, both spellings work, cf. config(5). > Thank you very much for the hint. I did not know this. I wrote > option in the article (for whatever reason) and tested the > configs using options... > > Best regards > Michael > > > > DES > > -- > > Dag-Erling Smørgrav - d...@freebsd.org > -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
> On 14. Mar 2024, at 11:04, Dag-Erling Smørgrav wrote: > > tue...@freebsd.org writes: >> Gary Jennejohn writes: >>> In the article we have option TCPHPTS and option TCP_RACK. But in >>> /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and >>> not option. >> Thanks for reporting, that is a typo in the article. It should >> always read options instead of option. > > It's not a typo, both spellings work, cf. config(5). Thank you very much for the hint. I did not know this. I wrote option in the article (for whatever reason) and tested the configs using options... Best regards Michael > > DES > -- > Dag-Erling Smørgrav - d...@freebsd.org
Re: Request for Testing: TCP RACK
tue...@freebsd.org writes: > Gary Jennejohn writes: > > In the article we have option TCPHPTS and option TCP_RACK. But in > > /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and > > not option. > Thanks for reporting, that is a typo in the article. It should > always read options instead of option. It's not a typo, both spellings work, cf. config(5). DES -- Dag-Erling Smørgrav - d...@freebsd.org
Re: Request for Testing: TCP RACK
> On 13. Mar 2024, at 08:06, Gary Jennejohn wrote: > > On Tue, 12 Mar 2024 20:14:17 +0100 > tue...@freebsd.org wrote: > >>> On 12. Mar 2024, at 14:39, Nuno Teixeira wrote: >>> >>> Hello, >>> >>> I'm curious about tcp RACK. >>> >>> As I do not run on a server background, only a laptop and a rpi4 for >>> poudriere, git, browsing, some torrent and ssh/sftp connections, will >>> I see any difference using RACK? >>> What tests should I do for comparison? >> You might want to read the following article in the FreeBSD Journal: >> https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/ >> >> There is no specific area for testing. Just test the stack on >> the systems you use with the workload you use and report back >> any issues... >> > > In the article we have option TCPHPTS and option TCP_RACK. But in > /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and > not option. Thanks for reporting, that is a typo in the article. It should always read options instead of option. Best regards Michael > > -- > Gary Jennejohn
Re: Request for Testing: TCP RACK
On Tue, 12 Mar 2024 20:14:17 +0100 tue...@freebsd.org wrote: > > On 12. Mar 2024, at 14:39, Nuno Teixeira wrote: > > > > Hello, > > > > I'm curious about tcp RACK. > > > > As I do not run on a server background, only a laptop and a rpi4 for > > poudriere, git, browsing, some torrent and ssh/sftp connections, will > > I see any difference using RACK? > > What tests should I do for comparison? > You might want to read the following article in the FreeBSD Journal: > https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/ > > There is no specific area for testing. Just test the stack on > the systems you use with the workload you use and report back > any issues... > In the article we have option TCPHPTS and option TCP_RACK. But in /sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and not option. -- Gary Jennejohn
Re: Request for Testing: TCP RACK
> On 12. Mar 2024, at 14:39, Nuno Teixeira wrote: > > Hello, > > I'm curious about tcp RACK. > > As I do not run on a server background, only a laptop and a rpi4 for > poudriere, git, browsing, some torrent and ssh/sftp connections, will > I see any difference using RACK? > What tests should I do for comparison? You might want to read the following article in the FreeBSD Journal: https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/ There is no specific area for testing. Just test the stack on the systems you use with the workload you use and report back any issues... Best regards Michael > > Thanks, > > escreveu (quinta, 16/11/2023 à(s) 15:10): >> >> Dear all, >> >> recently the main branch was changed to build the TCP RACK stack >> which is a loadable kernel module, by default: >> https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc >> >> As discussed on the bi-weekly transport call, it would be great if people >> could test the RACK stack for their workload. Please report any problems to >> the >> net@ mailing list or open an issue in the bug tracker and drop me a note via >> email. >> This includes regressions in CPU usage, regressions in performance or any >> other >> unexpected change you observe. >> >> You can load the kernel module using >> kldload tcp_rack >> >> You can make the RACK stack the default stack using >> sysctl net.inet.tcp.functions_default=rack >> >> Based on the feedback we get, the default stack might be switched to the >> RACK stack. >> >> Please let me know if you have any questions. >> >> Best regards >> Michael >> >> >> > > > -- > Nuno Teixeira > FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
On 3/12/24 6:39 AM, Nuno Teixeira wrote: Hello, I'm curious about tcp RACK. As I do not run on a server background, only a laptop and a rpi4 for poudriere, git, browsing, some torrent and ssh/sftp connections, will I see any difference using RACK? What tests should I do for comparison? I found this blog post from Klara was a good backgrounder to get my comfortable with testing out RACK: https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/ I've been using it on a busy'ish server and a workstation without any issues, but I'll defer to others as to what areas of focus for testing are needed. -pete -- Pete Wright p...@nomadlogic.org
Re: Request for Testing: TCP RACK
Hello, I'm curious about tcp RACK. As I do not run on a server background, only a laptop and a rpi4 for poudriere, git, browsing, some torrent and ssh/sftp connections, will I see any difference using RACK? What tests should I do for comparison? Thanks, escreveu (quinta, 16/11/2023 à(s) 15:10): > > Dear all, > > recently the main branch was changed to build the TCP RACK stack > which is a loadable kernel module, by default: > https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc > > As discussed on the bi-weekly transport call, it would be great if people > could test the RACK stack for their workload. Please report any problems to > the > net@ mailing list or open an issue in the bug tracker and drop me a note via > email. > This includes regressions in CPU usage, regressions in performance or any > other > unexpected change you observe. > > You can load the kernel module using > kldload tcp_rack > > You can make the RACK stack the default stack using > sysctl net.inet.tcp.functions_default=rack > > Based on the feedback we get, the default stack might be switched to the > RACK stack. > > Please let me know if you have any questions. > > Best regards > Michael > > > -- Nuno Teixeira FreeBSD Committer (ports)
Re: Request for Testing: TCP RACK
On Tue, 06 Feb 2024 23:05:27 +0100, tue...@freebsd.org wrote: > > > On Jan 5, 2024, at 08:48, tue...@freebsd.org wrote: > > > >> On Jan 4, 2024, at 21:39, Herbert J. Skuhra wrote: > >> > >> On Thu, 04 Jan 2024 21:22:22 +0100, tue...@freebsd.org wrote: > >>> > On Jan 4, 2024, at 18:52, Herbert J. Skuhra wrote: > > On Thu, 04 Jan 2024 11:40:35 +0100, "Herbert J. Skuhra" wrote: > > > > On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: > >> > >> Hi, > >> > >> On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > >>> > On Nov 16, 2023, at 20:06, Herbert J. Skuhra > wrote: > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > > > On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > > wrote: > > > >> > >> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > >> > >> After setting "sysctl net.inet.tcp.functions_default=rack" git no > >> longer works: > >> > >> > > Are you using a fresh 15 head or a specific network setup ? > > > > Because I'm not able to reproduce your problem on my system: > > > > $ uname -a > > FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > > main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > > root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > > amd64 > > $ cat /usr/src/sys/amd64/conf/TCPHPTS > > include GENERIC-NODEBUG > > ident TCPHPTS > > options TCPHPTS > > $ sysctl net.inet.tcp.functions_default > > net.inet.tcp.functions_default: rack > > $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo > > working > > working > > $ > > OK, (g)it works if I disable pf. Do you use pf? > >>> Can you share your pf config such that I can reproduce the problem > >>> locally? > >> > >> 1. It even fails with a simple pf.conf: > >> pass in all > >> pass out all > >> > >> 2. Fetching port distfiles also failed. > >> > >> 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > > > Disabling lro also resolves the issue. > > If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to > disable rxcsum/tcxsum or lro on igb0. > >>> Does the problem also goes away if you disable pf completely, but keep > >>> compressed acks enabled? > >> > >> Yes, it works with pf disabled and compressed acks enabled. > > Thanks for the information! > A patch is available at: > https://reviews.freebsd.org/D43769 Hi Michael, the patch resolves the issue. Thanks a lot. Best regards, Herbert
Re: Request for Testing: TCP RACK
> On Jan 5, 2024, at 08:48, tue...@freebsd.org wrote: > >> On Jan 4, 2024, at 21:39, Herbert J. Skuhra wrote: >> >> On Thu, 04 Jan 2024 21:22:22 +0100, tue...@freebsd.org wrote: >>> On Jan 4, 2024, at 18:52, Herbert J. Skuhra wrote: On Thu, 04 Jan 2024 11:40:35 +0100, "Herbert J. Skuhra" wrote: > > On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: >> >> Hi, >> >> On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: >>> On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > wrote: > >> >> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". >> >> After setting "sysctl net.inet.tcp.functions_default=rack" git no >> longer works: >> >> > Are you using a fresh 15 head or a specific network setup ? > > Because I'm not able to reproduce your problem on my system: > > $ uname -a > FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > amd64 > $ cat /usr/src/sys/amd64/conf/TCPHPTS > include GENERIC-NODEBUG > ident TCPHPTS > options TCPHPTS > $ sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: rack > $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > working > $ OK, (g)it works if I disable pf. Do you use pf? >>> Can you share your pf config such that I can reproduce the problem >>> locally? >> >> 1. It even fails with a simple pf.conf: >> pass in all >> pass out all >> >> 2. Fetching port distfiles also failed. >> >> 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > Disabling lro also resolves the issue. If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to disable rxcsum/tcxsum or lro on igb0. >>> Does the problem also goes away if you disable pf completely, but keep >>> compressed acks enabled? >> >> Yes, it works with pf disabled and compressed acks enabled. > Thanks for the information! A patch is available at: https://reviews.freebsd.org/D43769 Best regards Michael > > Best regards > Michael >> >> -- >> Herbert > >
Re: Request for Testing: TCP RACK
W dniu 17.11.2023 o 00:13, tue...@fh-muenster.de pisze: On Nov 16, 2023, at 17:50, Marek Zarychta wrote: W dniu 16.11.2023 o 10:13, tue...@freebsd.org pisze: Dear all, recently the main branch was changed to build the TCP RACK stack which is a loadable kernel module, by default: https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc That's really good news and long-awaited change. Thank you. As discussed on the bi-weekly transport call, it would be great if people could test the RACK stack for their workload. Please report any problems to the net@ mailing list or open an issue in the bug tracker and drop me a note via email. This includes regressions in CPU usage, regressions in performance or any other unexpected change you observe. You can load the kernel module using kldload tcp_rack You can make the RACK stack the default stack using sysctl net.inet.tcp.functions_default=rack Based on the feedback we get, the default stack might be switched to the RACK stack. Yes, please do it, at least for CURRENT. Last problem I have spotted with RACK was fixed in June: it was missing support TCP-MD5: https://cgit.freebsd.org/src/commit/?id=02b885b09d1e90574162a1442b9ede06cef2b13a We switched to RACK since upgrading to early stable/13 and genuinely appreciate this gift from Netflix. The performance improvement is invaluable, both in a lossy LAN and on a long-haul overseas TCP connection. Please let me know if you have any questions. Are any MFCs planned, especially to stable/14 ? Now, when stable/14 is almost in sync with main aka CURRENT it's an optimal time for such a MFC. When the change hits stable/14, it would be possible to test it extensively and have it fully functional in 14.1-RELEASE. Let me bring that up on the next Transport call. Will report back. Best regards Michael Thank you ! From now on stable/14 and in the future on 14.1-RELEASE one will be able to load tcp_rack and tcp_bbr out of the box running GENERIC kernel.[1] 1. https://cgit.freebsd.org/src/commit/?id=123fd2a -- Marek Zarychta
Re: Request for Testing: TCP RACK
> On Jan 4, 2024, at 21:39, Herbert J. Skuhra wrote: > > On Thu, 04 Jan 2024 21:22:22 +0100, tue...@freebsd.org wrote: >> >>> On Jan 4, 2024, at 18:52, Herbert J. Skuhra wrote: >>> >>> On Thu, 04 Jan 2024 11:40:35 +0100, "Herbert J. Skuhra" wrote: On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: > > Hi, > > On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: >> >>> On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: >>> >>> On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra wrote: > > OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > > After setting "sysctl net.inet.tcp.functions_default=rack" git no > longer works: > > Are you using a fresh 15 head or a specific network setup ? Because I'm not able to reproduce your problem on my system: $ uname -a FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS amd64 $ cat /usr/src/sys/amd64/conf/TCPHPTS include GENERIC-NODEBUG ident TCPHPTS options TCPHPTS $ sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: rack $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working working $ >>> >>> OK, (g)it works if I disable pf. Do you use pf? >> Can you share your pf config such that I can reproduce the problem >> locally? > > 1. It even fails with a simple pf.conf: > pass in all > pass out all > > 2. Fetching port distfiles also failed. > > 3. If I disable rxcsum on the ethernet adapter (igb0) it works. Disabling lro also resolves the issue. >>> >>> If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to >>> disable rxcsum/tcxsum or lro on igb0. >> Does the problem also goes away if you disable pf completely, but keep >> compressed acks enabled? > > Yes, it works with pf disabled and compressed acks enabled. Thanks for the information! Best regards Michael > > -- > Herbert
Re: Request for Testing: TCP RACK
On Thu, 04 Jan 2024 21:22:22 +0100, tue...@freebsd.org wrote: > > > On Jan 4, 2024, at 18:52, Herbert J. Skuhra wrote: > > > > On Thu, 04 Jan 2024 11:40:35 +0100, "Herbert J. Skuhra" wrote: > >> > >> On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: > >>> > >>> Hi, > >>> > >>> On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > > > On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > >> > >> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > >> wrote: > >> > >>> > >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > >>> > >>> After setting "sysctl net.inet.tcp.functions_default=rack" git no > >>> longer works: > >>> > >>> > >> Are you using a fresh 15 head or a specific network setup ? > >> > >> Because I'm not able to reproduce your problem on my system: > >> > >> $ uname -a > >> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > >> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > >> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > >> amd64 > >> $ cat /usr/src/sys/amd64/conf/TCPHPTS > >> include GENERIC-NODEBUG > >> ident TCPHPTS > >> options TCPHPTS > >> $ sysctl net.inet.tcp.functions_default > >> net.inet.tcp.functions_default: rack > >> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > >> working > >> $ > > > > OK, (g)it works if I disable pf. Do you use pf? > Can you share your pf config such that I can reproduce the problem > locally? > >>> > >>> 1. It even fails with a simple pf.conf: > >>> pass in all > >>> pass out all > >>> > >>> 2. Fetching port distfiles also failed. > >>> > >>> 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > >> > >> Disabling lro also resolves the issue. > > > > If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to > > disable rxcsum/tcxsum or lro on igb0. > Does the problem also goes away if you disable pf completely, but keep > compressed acks enabled? Yes, it works with pf disabled and compressed acks enabled. -- Herbert
Re: Request for Testing: TCP RACK
> On Jan 4, 2024, at 18:52, Herbert J. Skuhra wrote: > > On Thu, 04 Jan 2024 11:40:35 +0100, "Herbert J. Skuhra" wrote: >> >> On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: >>> >>> Hi, >>> >>> On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: >> >> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra >> wrote: >> >>> >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". >>> >>> After setting "sysctl net.inet.tcp.functions_default=rack" git no >>> longer works: >>> >>> >> Are you using a fresh 15 head or a specific network setup ? >> >> Because I'm not able to reproduce your problem on my system: >> >> $ uname -a >> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 >> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 >> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS >> amd64 >> $ cat /usr/src/sys/amd64/conf/TCPHPTS >> include GENERIC-NODEBUG >> ident TCPHPTS >> options TCPHPTS >> $ sysctl net.inet.tcp.functions_default >> net.inet.tcp.functions_default: rack >> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working >> working >> $ > > OK, (g)it works if I disable pf. Do you use pf? Can you share your pf config such that I can reproduce the problem locally? >>> >>> 1. It even fails with a simple pf.conf: >>> pass in all >>> pass out all >>> >>> 2. Fetching port distfiles also failed. >>> >>> 3. If I disable rxcsum on the ethernet adapter (igb0) it works. >> >> Disabling lro also resolves the issue. > > If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to > disable rxcsum/tcxsum or lro on igb0. Does the problem also goes away if you disable pf completely, but keep compressed acks enabled? Best regards Michael > > -- > Herbert >
Re: Request for Testing: TCP RACK
On Thu, 04 Jan 2024 11:40:35 +0100, "Herbert J. Skuhra" wrote: > > On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: > > > > Hi, > > > > On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > > > > > > > On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > > > > > > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > > >> > > > >> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > > > >> wrote: > > > >> > > > >>> > > > >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > > > >>> > > > >>> After setting "sysctl net.inet.tcp.functions_default=rack" git no > > > >>> longer works: > > > >>> > > > >>> > > > >> Are you using a fresh 15 head or a specific network setup ? > > > >> > > > >> Because I'm not able to reproduce your problem on my system: > > > >> > > > >> $ uname -a > > > >> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > > > >> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > > > >> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > > > >> amd64 > > > >> $ cat /usr/src/sys/amd64/conf/TCPHPTS > > > >> include GENERIC-NODEBUG > > > >> ident TCPHPTS > > > >> options TCPHPTS > > > >> $ sysctl net.inet.tcp.functions_default > > > >> net.inet.tcp.functions_default: rack > > > >> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > > > >> working > > > >> $ > > > > > > > > OK, (g)it works if I disable pf. Do you use pf? > > > Can you share your pf config such that I can reproduce the problem > > > locally? > > > > 1. It even fails with a simple pf.conf: > >pass in all > >pass out all > > > > 2. Fetching port distfiles also failed. > > > > 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > Disabling lro also resolves the issue. If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to disable rxcsum/tcxsum or lro on igb0. -- Herbert
Re: Request for Testing: TCP RACK
> On Jan 4, 2024, at 15:22, Herbert J. Skuhra wrote: > > On Thu, 04 Jan 2024 14:57:59 +0100, tue...@fh-muenster.de wrote: >> >>> On Jan 4, 2024, at 11:40, Herbert J. Skuhra wrote: >>> >>> On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: Hi, On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > >> On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: >> >> On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: >>> >>> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra >>> wrote: >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". After setting "sysctl net.inet.tcp.functions_default=rack" git no longer works: >>> Are you using a fresh 15 head or a specific network setup ? >>> >>> Because I'm not able to reproduce your problem on my system: >>> >>> $ uname -a >>> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 >>> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 >>> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS >>> amd64 >>> $ cat /usr/src/sys/amd64/conf/TCPHPTS >>> include GENERIC-NODEBUG >>> ident TCPHPTS >>> options TCPHPTS >>> $ sysctl net.inet.tcp.functions_default >>> net.inet.tcp.functions_default: rack >>> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working >>> working >>> $ >> >> OK, (g)it works if I disable pf. Do you use pf? > Can you share your pf config such that I can reproduce the problem > locally? 1. It even fails with a simple pf.conf: pass in all pass out all 2. Fetching port distfiles also failed. 3. If I disable rxcsum on the ethernet adapter (igb0) it works. >>> >>> Disabling lro also resolves the issue. >>> >>> Not OK: >>> >>> igb0: flags=1008843 metric >>> 0 mtu 1500 >>> >>> options=4e527bb >>> >>> OK: >>> >>> igb0: flags=1008843 metric >>> 0 mtu 1500 >>> >>> options=4e523bb >> What kind of NIC do you have? Can you post the output of >> dmesg | grep igb0 > > igb0: port 0xf000-0xf01f mem > 0xfc20-0xfc27,0xfc28-0xfc283fff irq 28 at device 0.0 on pci3 > igb0: EEPROM V3.16-0 eTrack 0x84d6 > igb0: Using 1024 TX descriptors and 1024 RX descriptors > igb0: Using 4 RX queues 4 TX queues > igb0: Using MSI-X interrupts with 5 vectors > igb0: Ethernet address: aa:bb:cc:dd:ee:ff > igb0: netmap queues/slots: TX 4/1024, RX 4/1024 > igb0: link state changed to UP > igb0: link state changed to DOWN > igb0: link state changed to UP Hi Herbert, thank you very much. I'll see if I have such a NIC in one of my test systems and will report back. Best regards Michael > > -- > Herbert >
Re: Request for Testing: TCP RACK
> On Jan 4, 2024, at 11:40, Herbert J. Skuhra wrote: > > On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: >> >> Hi, >> >> On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: >>> On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > wrote: > >> >> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". >> >> After setting "sysctl net.inet.tcp.functions_default=rack" git no >> longer works: >> >> > Are you using a fresh 15 head or a specific network setup ? > > Because I'm not able to reproduce your problem on my system: > > $ uname -a > FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > amd64 > $ cat /usr/src/sys/amd64/conf/TCPHPTS > include GENERIC-NODEBUG > ident TCPHPTS > options TCPHPTS > $ sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: rack > $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > working > $ OK, (g)it works if I disable pf. Do you use pf? >>> Can you share your pf config such that I can reproduce the problem locally? >> >> 1. It even fails with a simple pf.conf: >> pass in all >> pass out all >> >> 2. Fetching port distfiles also failed. >> >> 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > Disabling lro also resolves the issue. > > Not OK: > > igb0: flags=1008843 metric 0 > mtu 1500 > > options=4e527bb > > OK: > > igb0: flags=1008843 metric 0 > mtu 1500 > > options=4e523bb What kind of NIC do you have? Can you post the output of dmesg | grep igb0 Best regards Michael > > -- > Herbert >
Re: Request for Testing: TCP RACK
On Thu, 04 Jan 2024 14:57:59 +0100, tue...@fh-muenster.de wrote: > > > On Jan 4, 2024, at 11:40, Herbert J. Skuhra wrote: > > > > On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: > >> > >> Hi, > >> > >> On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > >>> > On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > > > On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > > wrote: > > > >> > >> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > >> > >> After setting "sysctl net.inet.tcp.functions_default=rack" git no > >> longer works: > >> > >> > > Are you using a fresh 15 head or a specific network setup ? > > > > Because I'm not able to reproduce your problem on my system: > > > > $ uname -a > > FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > > main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > > root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > > amd64 > > $ cat /usr/src/sys/amd64/conf/TCPHPTS > > include GENERIC-NODEBUG > > ident TCPHPTS > > options TCPHPTS > > $ sysctl net.inet.tcp.functions_default > > net.inet.tcp.functions_default: rack > > $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > > working > > $ > > OK, (g)it works if I disable pf. Do you use pf? > >>> Can you share your pf config such that I can reproduce the problem > >>> locally? > >> > >> 1. It even fails with a simple pf.conf: > >> pass in all > >> pass out all > >> > >> 2. Fetching port distfiles also failed. > >> > >> 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > > > Disabling lro also resolves the issue. > > > > Not OK: > > > > igb0: flags=1008843 metric > > 0 mtu 1500 > > > > options=4e527bb > > > > OK: > > > > igb0: flags=1008843 metric > > 0 mtu 1500 > > > > options=4e523bb > What kind of NIC do you have? Can you post the output of > dmesg | grep igb0 igb0: port 0xf000-0xf01f mem 0xfc20-0xfc27,0xfc28-0xfc283fff irq 28 at device 0.0 on pci3 igb0: EEPROM V3.16-0 eTrack 0x84d6 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 4 RX queues 4 TX queues igb0: Using MSI-X interrupts with 5 vectors igb0: Ethernet address: aa:bb:cc:dd:ee:ff igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb0: link state changed to UP igb0: link state changed to DOWN igb0: link state changed to UP -- Herbert
Re: Request for Testing: TCP RACK
On Fri, 17 Nov 2023 14:31:02 +0100, "Herbert J. Skuhra" wrote: > > Hi, > > On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > > > > > On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > > > > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > >> > > >> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > > >> wrote: > > >> > > >>> > > >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > > >>> > > >>> After setting "sysctl net.inet.tcp.functions_default=rack" git no > > >>> longer works: > > >>> > > >>> > > >> Are you using a fresh 15 head or a specific network setup ? > > >> > > >> Because I'm not able to reproduce your problem on my system: > > >> > > >> $ uname -a > > >> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > > >> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > > >> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > > >> amd64 > > >> $ cat /usr/src/sys/amd64/conf/TCPHPTS > > >> include GENERIC-NODEBUG > > >> ident TCPHPTS > > >> options TCPHPTS > > >> $ sysctl net.inet.tcp.functions_default > > >> net.inet.tcp.functions_default: rack > > >> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > > >> working > > >> $ > > > > > > OK, (g)it works if I disable pf. Do you use pf? > > Can you share your pf config such that I can reproduce the problem locally? > > 1. It even fails with a simple pf.conf: >pass in all >pass out all > > 2. Fetching port distfiles also failed. > > 3. If I disable rxcsum on the ethernet adapter (igb0) it works. Disabling lro also resolves the issue. Not OK: igb0: flags=1008843 metric 0 mtu 1500 options=4e527bb OK: igb0: flags=1008843 metric 0 mtu 1500 options=4e523bb -- Herbert
Re: Request for Testing: TCP RACK
> On Nov 19, 2023, at 2:35 AM, Zhenlei Huang wrote: > > > >> On Nov 16, 2023, at 5:13 PM, tue...@freebsd.org wrote: >> >> Dear all, >> >> recently the main branch was changed to build the TCP RACK stack >> which is a loadable kernel module, by default: >> https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc >> >> As discussed on the bi-weekly transport call, it would be great if people >> could test the RACK stack for their workload. Please report any problems to >> the >> net@ mailing list or open an issue in the bug tracker and drop me a note via >> email. >> This includes regressions in CPU usage, regressions in performance or any >> other >> unexpected change you observe. > > I see some performance regression with the rack stack. > > This is iperf3 test on local host (bare metal, an old i5 2 cores 4 threads > MBP). > > freebsd: 16.1 Gbits/sec > rack: 12.3 Gbits/sec > > The congestion control algorithm is cubic. > > Note this is a quick test with DEBUG options enabled. > > I'll try with no debug options and report later. ** UPDATE ** With no debug options: freebsd:37.2 Gbits/sec rack: 27.9 Gbits/sec > >> >> You can load the kernel module using >> kldload tcp_rack >> >> You can make the RACK stack the default stack using >> sysctl net.inet.tcp.functions_default=rack >> >> Based on the feedback we get, the default stack might be switched to the >> RACK stack. >> >> Please let me know if you have any questions. >> >> Best regards >> Michael >> >> >> > > Best regards, > Zhenlei
Re: Request for Testing: TCP RACK
On Sat, 18 Nov 2023 09:50:43 +0100 tue...@freebsd.org wrote: > > On Nov 18, 2023, at 00:37, Tomoaki AOKI wrote: > > > > On Fri, 17 Nov 2023 18:51:05 +0100 > > tue...@freebsd.org wrote: > > > >>> On Nov 17, 2023, at 17:06, Johan Hendriks wrote: > >>> > >>> I am running the rack stack for quiet some time now on a baremetal > >>> machiene and never had problems. > >>> Also use pf. This is a test machine so not a lot happening on it. > >>> > >>> Are there any thing we can test? Do we have some test scripts we can run? > >> We are actually interested in feedback about using the stack in whatever > >> use case you use TCP for. The stack has been tested with the Netflix use > >> case, but not so much with others. That is why we ask for broader testing. > >> > >> Best regards > >> Michael > > > > Are there any difference regarding with performance between main and > > stable/14? If so, please ignore below. > > > > I have stable/14 environment which is configured to be able to switch > > to TCP RACK and experienced huge performance loss when writing > > a large file to smbfs share on commercial NAS. CC is cubic. > > Testing large archive on the smbfs share doesn't seem to be > > affected. > > > > Comparison between default (freebsd) and rack TCP stack using > > sysutils/clone on stable/14 at commit 7d1321288ad9, amd64. > > Last 3 lines of outputs from clone (upload to NAS) are shown. > Thank you very much for testing. This is what we are looking > for. Would it be possible to repeat the test using NewReno as > the CC? > > Best regards > Michael Sure. Here we go! sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: freebsd sysctl net.inet.tcp.cc.algorithm net.inet.tcp.cc.algorithm: newreno Umounted and remounted smbfs share. 1 item copied, 2343.1 MB in 37.65 s -- 62.2 MB/s Leaked memory: 0 bytes No errors occured. sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: rack Umounted and remounted smbfs share. 1 item copied, 2343.1 MB in 905.17 s -- 2.6 MB/s Leaked memory: 0 bytes No errors occured. sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: freebsd Without umount and remount to reproduce previous oddness, maybe caused by keep-alive. 1 item copied, 2343.1 MB in 897.67 s -- 2.6 MB/s Leaked memory: 0 bytes No errors occured. Umounted and remounted, without change for CC and TCP stack. 1 item copied, 2343.1 MB in 37.43 s -- 62.6 MB/s Leaked memory: 0 bytes No errors occured. All test are proceeded simultaneously. So the last one is with CC=newreno and TCP stack=freebsd. Not exactly recorded, but testing transferred file by diff -u -p -N was roughly 30MB/sec, while roughly 25MB/sec with CC=cubic. > > > > Before switching to rack: > > 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s > > Leaked memory: 0 bytes > > No errors occured. > > > > Unmount the smbfs share, switch to rack, and after remount: > > 1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s > > Leaked memory: 0 bytes > > No errors occured. > > > > Switch back to freebsd (default) without unmounting: > > 1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s > > Leaked memory: 0 bytes > > No errors occured. > > > > Unmount and remount the smbfs share: > > 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s > > Leaked memory: 0 bytes > > No errors occured. > > > > > > Regards. > > > > -- > > Tomoaki AOKI -- Tomoaki AOKI
Re: Request for Testing: TCP RACK
> On Nov 16, 2023, at 5:13 PM, tue...@freebsd.org wrote: > > Dear all, > > recently the main branch was changed to build the TCP RACK stack > which is a loadable kernel module, by default: > https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc > > As discussed on the bi-weekly transport call, it would be great if people > could test the RACK stack for their workload. Please report any problems to > the > net@ mailing list or open an issue in the bug tracker and drop me a note via > email. > This includes regressions in CPU usage, regressions in performance or any > other > unexpected change you observe. I see some performance regression with the rack stack. This is iperf3 test on local host (bare metal, an old i5 2 cores 4 threads MBP). freebsd:16.1 Gbits/sec rack: 12.3 Gbits/sec The congestion control algorithm is cubic. Note this is a quick test with DEBUG options enabled. I'll try with no debug options and report later. > > You can load the kernel module using > kldload tcp_rack > > You can make the RACK stack the default stack using > sysctl net.inet.tcp.functions_default=rack > > Based on the feedback we get, the default stack might be switched to the > RACK stack. > > Please let me know if you have any questions. > > Best regards > Michael > > > Best regards, Zhenlei
Re: Request for Testing: TCP RACK
> On Nov 18, 2023, at 00:37, Tomoaki AOKI wrote: > > On Fri, 17 Nov 2023 18:51:05 +0100 > tue...@freebsd.org wrote: > >>> On Nov 17, 2023, at 17:06, Johan Hendriks wrote: >>> >>> I am running the rack stack for quiet some time now on a baremetal machiene >>> and never had problems. >>> Also use pf. This is a test machine so not a lot happening on it. >>> >>> Are there any thing we can test? Do we have some test scripts we can run? >> We are actually interested in feedback about using the stack in whatever >> use case you use TCP for. The stack has been tested with the Netflix use >> case, but not so much with others. That is why we ask for broader testing. >> >> Best regards >> Michael > > Are there any difference regarding with performance between main and > stable/14? If so, please ignore below. > > I have stable/14 environment which is configured to be able to switch > to TCP RACK and experienced huge performance loss when writing > a large file to smbfs share on commercial NAS. CC is cubic. > Testing large archive on the smbfs share doesn't seem to be > affected. > > Comparison between default (freebsd) and rack TCP stack using > sysutils/clone on stable/14 at commit 7d1321288ad9, amd64. > Last 3 lines of outputs from clone (upload to NAS) are shown. Thank you very much for testing. This is what we are looking for. Would it be possible to repeat the test using NewReno as the CC? Best regards Michael > > Before switching to rack: > 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s > Leaked memory: 0 bytes > No errors occured. > > Unmount the smbfs share, switch to rack, and after remount: > 1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s > Leaked memory: 0 bytes > No errors occured. > > Switch back to freebsd (default) without unmounting: > 1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s > Leaked memory: 0 bytes > No errors occured. > > Unmount and remount the smbfs share: > 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s > Leaked memory: 0 bytes > No errors occured. > > > Regards. > > -- > Tomoaki AOKI
Re: Request for Testing: TCP RACK
On Fri, 17 Nov 2023 18:51:05 +0100 tue...@freebsd.org wrote: > > On Nov 17, 2023, at 17:06, Johan Hendriks wrote: > > > > I am running the rack stack for quiet some time now on a baremetal machiene > > and never had problems. > > Also use pf. This is a test machine so not a lot happening on it. > > > > Are there any thing we can test? Do we have some test scripts we can run? > We are actually interested in feedback about using the stack in whatever > use case you use TCP for. The stack has been tested with the Netflix use > case, but not so much with others. That is why we ask for broader testing. > > Best regards > Michael Are there any difference regarding with performance between main and stable/14? If so, please ignore below. I have stable/14 environment which is configured to be able to switch to TCP RACK and experienced huge performance loss when writing a large file to smbfs share on commercial NAS. CC is cubic. Testing large archive on the smbfs share doesn't seem to be affected. Comparison between default (freebsd) and rack TCP stack using sysutils/clone on stable/14 at commit 7d1321288ad9, amd64. Last 3 lines of outputs from clone (upload to NAS) are shown. Before switching to rack: 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s Leaked memory: 0 bytes No errors occured. Unmount the smbfs share, switch to rack, and after remount: 1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s Leaked memory: 0 bytes No errors occured. Switch back to freebsd (default) without unmounting: 1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s Leaked memory: 0 bytes No errors occured. Unmount and remount the smbfs share: 1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s Leaked memory: 0 bytes No errors occured. Regards. -- Tomoaki AOKI
Re: Request for Testing: TCP RACK
> On Nov 17, 2023, at 17:06, Johan Hendriks wrote: > > I am running the rack stack for quiet some time now on a baremetal machiene > and never had problems. > Also use pf. This is a test machine so not a lot happening on it. > > Are there any thing we can test? Do we have some test scripts we can run? We are actually interested in feedback about using the stack in whatever use case you use TCP for. The stack has been tested with the Netflix use case, but not so much with others. That is why we ask for broader testing. Best regards Michael > > > >
Re: Request for Testing: TCP RACK
I am running the rack stack for quiet some time now on a baremetal machiene and never had problems. Also use pf. This is a test machine so not a lot happening on it. Are there any thing we can test? Do we have some test scripts we can run?
Re: Request for Testing: TCP RACK
Am 2023-11-17 14:29, schrieb void: On Thu, Nov 16, 2023 at 10:13:05AM +0100, tue...@freebsd.org wrote: You can load the kernel module using kldload tcp_rack You can make the RACK stack the default stack using sysctl net.inet.tcp.functions_default=rack Hi, thank you for this. https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/ mentions this needs to be set in /etc/src.conf : WITH_EXTRA_TCP_STACKS=1 Is this still the case? Context here is -current both in a vm and bare metal, on various machines, on various connections, from DSL to 10Gb. On a recent -current: this is not needed anymore, it is part of the defaults now. But you may still compile the kernel with "option TCPHPTS" (until it's added to the defaults too). Is there a method (yet) for enabling this functionality in various -RELENG maybe where one can compile in a vm built for that purpose, then transferring to the production vm? Copy the kernel which was build according to the acticle from klara systems to your target VM. Would it be expected to work on arm64? Yes (I use it on an ampere VM in the cloud). Bye, Alexander. -- http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.orgnetch...@freebsd.org : PGP 0x8F31830F9F2772BF signature.asc Description: OpenPGP digital signature
Re: Request for Testing: TCP RACK
Hi, On Fri, Nov 17, 2023 at 02:46:52PM +0100, Olivier Cochard-Labbé wrote: > On Fri, Nov 17, 2023 at 2:31 PM Herbert J. Skuhra wrote: > > > > > 1. It even fails with a simple pf.conf: > >pass in all > >pass out all > > > > 2. Fetching port distfiles also failed. > > > > 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > > > > I can't reproduce it with pfctl too (same igb drivers with default RXCSUM > enabled). > > $ cat /etc/pf.conf > pass in all > pass out all > $ service pf onestart > Enabling pf > . > $ pfctl -sr > pass in all flags S/SA keep state > pass out all flags S/SA keep state > $ sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: rack > $ ifconfig igb0 | grep option > > options=4e523bb > nd6 options=21 > > $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > working > > What is your igb chipset exactly ? (pciconf -lv | grep -B 3 -F "network") igb0@pci0:41:0:0: class=0x02 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x1849 subdevice=0x1533 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network > What is your netstat -ss output ? tcp: 946 packets sent 933 data packets (41681 bytes) 7946 ack-only packets (2 delayed) 1 control packet 999 packets received 860 acks (for 41681 bytes) 910 packets (118790 bytes) received in-sequence 12 completely duplicate packets (17136 bytes) 1 connection request 1 connection established (including accepts) 1 time used RTT from hostcache 1 time used RTT variance from hostcache 1 connection closed (including 1 drop) 1 connection updated cached RTT on close 1 connection updated cached RTT variance on close 862 segments updated rtt (of 847 attempts) 71 correct ACK header predictions 124 correct data packet header predictions 7910 SACK options (SACK blocks) sent TCP connection count by state: 7 connections in LISTEN state 1 connection in ESTABLISHED state udp: 39 datagrams received 39 delivered 39 datagrams output ip: 80 total packets received 23 packets for this host 23 packets sent from this host icmp: ICMP address mask responses are disabled igmp: arp: ip6: 933 total packets received 931 packets for this host 8891 packets sent from this host Input histogram: TCP: 915 UDP: 16 ICMP6: 2 Mbuf statistics: 870 one mbuf 63 one ext mbuf 0 two or more ext mbuf source addresses on an outgoing I/F 2 link-locals 3 globals source addresses of same scope 2 link-locals 3 globals Source addresses selection rule applied: 5 first candidate 3 appropriate scope icmp6: Output histogram: neighbor solicitation: 2 Input histogram: neighbor advertisement: 2 Histogram of error messages to be generated: rip6: Thanks. -- Herbert
Re: Request for Testing: TCP RACK
On Fri, Nov 17, 2023 at 2:31 PM Herbert J. Skuhra wrote: > > 1. It even fails with a simple pf.conf: >pass in all >pass out all > > 2. Fetching port distfiles also failed. > > 3. If I disable rxcsum on the ethernet adapter (igb0) it works. > > > I can't reproduce it with pfctl too (same igb drivers with default RXCSUM enabled). $ cat /etc/pf.conf pass in all pass out all $ service pf onestart Enabling pf . $ pfctl -sr pass in all flags S/SA keep state pass out all flags S/SA keep state $ sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: rack $ ifconfig igb0 | grep option options=4e523bb nd6 options=21 $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working working What is your igb chipset exactly ? (pciconf -lv | grep -B 3 -F "network") What is your netstat -ss output ?
Re: Request for Testing: TCP RACK
Hi, On Fri, 17 Nov 2023 00:15:13 +0100, tue...@freebsd.org wrote: > > > On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > >> > >> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra > >> wrote: > >> > >>> > >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > >>> > >>> After setting "sysctl net.inet.tcp.functions_default=rack" git no > >>> longer works: > >>> > >>> > >> Are you using a fresh 15 head or a specific network setup ? > >> > >> Because I'm not able to reproduce your problem on my system: > >> > >> $ uname -a > >> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > >> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > >> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > >> amd64 > >> $ cat /usr/src/sys/amd64/conf/TCPHPTS > >> include GENERIC-NODEBUG > >> ident TCPHPTS > >> options TCPHPTS > >> $ sysctl net.inet.tcp.functions_default > >> net.inet.tcp.functions_default: rack > >> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > >> working > >> $ > > > > OK, (g)it works if I disable pf. Do you use pf? > Can you share your pf config such that I can reproduce the problem locally? 1. It even fails with a simple pf.conf: pass in all pass out all 2. Fetching port distfiles also failed. 3. If I disable rxcsum on the ethernet adapter (igb0) it works. Thanks. -- Herbert
Re: Request for Testing: TCP RACK
On Thu, Nov 16, 2023 at 10:13:05AM +0100, tue...@freebsd.org wrote: You can load the kernel module using kldload tcp_rack You can make the RACK stack the default stack using sysctl net.inet.tcp.functions_default=rack Hi, thank you for this. https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/ mentions this needs to be set in /etc/src.conf : WITH_EXTRA_TCP_STACKS=1 Is this still the case? Context here is -current both in a vm and bare metal, on various machines, on various connections, from DSL to 10Gb. Is there a method (yet) for enabling this functionality in various -RELENG maybe where one can compile in a vm built for that purpose, then transferring to the production vm? Would it be expected to work on arm64? What testing would you like doing? --
Re: Request for Testing: TCP RACK
> On Nov 16, 2023, at 20:06, Herbert J. Skuhra wrote: > > On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: >> >> On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra wrote: >> >>> >>> OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". >>> >>> After setting "sysctl net.inet.tcp.functions_default=rack" git no >>> longer works: >>> >>> >> Are you using a fresh 15 head or a specific network setup ? >> >> Because I'm not able to reproduce your problem on my system: >> >> $ uname -a >> FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 >> main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 >> root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS >> amd64 >> $ cat /usr/src/sys/amd64/conf/TCPHPTS >> include GENERIC-NODEBUG >> ident TCPHPTS >> options TCPHPTS >> $ sysctl net.inet.tcp.functions_default >> net.inet.tcp.functions_default: rack >> $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working >> working >> $ > > OK, (g)it works if I disable pf. Do you use pf? Can you share your pf config such that I can reproduce the problem locally? Best regards Michael > > -- > Herbert
Re: Request for Testing: TCP RACK
> On Nov 16, 2023, at 14:05, Herbert J. Skuhra wrote: > > On Thu, 16 Nov 2023 13:06:03 +0100, "Herbert J. Skuhra" wrote: >> >> Hi, >> >> On Thu, 16 Nov 2023 10:13:05 +0100, tue...@freebsd.org wrote: >>> >>> Dear all, >>> >>> recently the main branch was changed to build the TCP RACK stack >>> which is a loadable kernel module, by default: >>> https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc >>> >>> As discussed on the bi-weekly transport call, it would be great if people >>> could test the RACK stack for their workload. Please report any problems to >>> the >>> net@ mailing list or open an issue in the bug tracker and drop me a note >>> via email. >>> This includes regressions in CPU usage, regressions in performance or any >>> other >>> unexpected change you observe. >>> >>> You can load the kernel module using >>> kldload tcp_rack >>> >>> You can make the RACK stack the default stack using >>> sysctl net.inet.tcp.functions_default=rack >>> >>> Based on the feedback we get, the default stack might be switched to the >>> RACK stack. >>> >>> Please let me know if you have any questions. >> >> I am running main-n266450-a592812327de with a GENERIC-NODEBUG kernel. >> >> # kldload tcp_rack >> kldload: an error occurred while loading module tcp_rack. Please check >> dmesg(8) for more details. >> >> In dmesg: >> KLD tcp_rack.ko: depends on tcphpts - not available or version mismatch >> linker_load_file: /boot/kernel/tcp_rack.ko - unsupported file type >> >> So you have to build a kernel with "options TCPHPTS" first? > > OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > > After setting "sysctl net.inet.tcp.functions_default=rack" git no > longer works: > > # sysctl net.inet.tcp.functions_default=rack > $ git pull > client_loop: send disconnect: Broken pipe > fatal: the remote end hung up upon initial contact > # sysctl net.inet.tcp.functions_default=freebsd > $ git pull > Already up to date. Interesting. It works on my development system: tuexen@head:~/freebsd-src % sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: rack tuexen@head:~/freebsd-src % git pull Already up to date. tuexen@head:~/freebsd-src % Could you provide a .pcapng file covering the `git pull` command? Best regards Michael > > -- > Herbert
Re: Request for Testing: TCP RACK
> On Nov 16, 2023, at 13:06, Herbert J. Skuhra wrote: > > Hi, > > On Thu, 16 Nov 2023 10:13:05 +0100, tue...@freebsd.org wrote: >> >> Dear all, >> >> recently the main branch was changed to build the TCP RACK stack >> which is a loadable kernel module, by default: >> https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc >> >> As discussed on the bi-weekly transport call, it would be great if people >> could test the RACK stack for their workload. Please report any problems to >> the >> net@ mailing list or open an issue in the bug tracker and drop me a note via >> email. >> This includes regressions in CPU usage, regressions in performance or any >> other >> unexpected change you observe. >> >> You can load the kernel module using >> kldload tcp_rack >> >> You can make the RACK stack the default stack using >> sysctl net.inet.tcp.functions_default=rack >> >> Based on the feedback we get, the default stack might be switched to the >> RACK stack. >> >> Please let me know if you have any questions. > > I am running main-n266450-a592812327de with a GENERIC-NODEBUG kernel. > > # kldload tcp_rack > kldload: an error occurred while loading module tcp_rack. Please check > dmesg(8) for more details. > > In dmesg: > KLD tcp_rack.ko: depends on tcphpts - not available or version mismatch > linker_load_file: /boot/kernel/tcp_rack.ko - unsupported file type > > So you have to build a kernel with "options TCPHPTS" first? Hi Herbert, yes this is correct. For whatever reason I was assuming the TCPHPTS is already enabled in all configs, but this is not correct. Will put up an review to do this tomorrow. Thanks for reporting. Best regards Michael > > -- > Herbert
Re: Request for Testing: TCP RACK
On Thu, 16 Nov 2023 19:07:29 +0100, Olivier Cochard-Labbé wrote: > > On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra wrote: > > > > > OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > > > > After setting "sysctl net.inet.tcp.functions_default=rack" git no > > longer works: > > > > > Are you using a fresh 15 head or a specific network setup ? > > Because I'm not able to reproduce your problem on my system: > > $ uname -a > FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 > main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 > root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS > amd64 > $ cat /usr/src/sys/amd64/conf/TCPHPTS > include GENERIC-NODEBUG > ident TCPHPTS > options TCPHPTS > $ sysctl net.inet.tcp.functions_default > net.inet.tcp.functions_default: rack > $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working > working > $ OK, (g)it works if I disable pf. Do you use pf? -- Herbert
Re: Request for Testing: TCP RACK
On Thu, Nov 16, 2023 at 5:10 PM Herbert J. Skuhra wrote: > > OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". > > After setting "sysctl net.inet.tcp.functions_default=rack" git no > longer works: > > Are you using a fresh 15 head or a specific network setup ? Because I'm not able to reproduce your problem on my system: $ uname -a FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023 root@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS amd64 $ cat /usr/src/sys/amd64/conf/TCPHPTS include GENERIC-NODEBUG ident TCPHPTS options TCPHPTS $ sysctl net.inet.tcp.functions_default net.inet.tcp.functions_default: rack $ git clone -q g...@github.com:freebsd/freebsd-src.git && echo working working $
Re: Request for Testing: TCP RACK
W dniu 16.11.2023 o 10:13, tue...@freebsd.org pisze: Dear all, recently the main branch was changed to build the TCP RACK stack which is a loadable kernel module, by default: https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc That's really good news and long-awaited change. Thank you. As discussed on the bi-weekly transport call, it would be great if people could test the RACK stack for their workload. Please report any problems to the net@ mailing list or open an issue in the bug tracker and drop me a note via email. This includes regressions in CPU usage, regressions in performance or any other unexpected change you observe. You can load the kernel module using kldload tcp_rack You can make the RACK stack the default stack using sysctl net.inet.tcp.functions_default=rack Based on the feedback we get, the default stack might be switched to the RACK stack. Yes, please do it, at least for CURRENT. Last problem I have spotted with RACK was fixed in June: it was missing support TCP-MD5: https://cgit.freebsd.org/src/commit/?id=02b885b09d1e90574162a1442b9ede06cef2b13a We switched to RACK since upgrading to early stable/13 and genuinely appreciate this gift from Netflix. The performance improvement is invaluable, both in a lossy LAN and on a long-haul overseas TCP connection. Please let me know if you have any questions. Are any MFCs planned, especially to stable/14 ? Now, when stable/14 is almost in sync with main aka CURRENT it's an optimal time for such a MFC. When the change hits stable/14, it would be possible to test it extensively and have it fully functional in 14.1-RELEASE. Best regards -- Marek Zarychta
Re: Request for Testing: TCP RACK
On Thu, 16 Nov 2023 13:06:03 +0100, "Herbert J. Skuhra" wrote: > > Hi, > > On Thu, 16 Nov 2023 10:13:05 +0100, tue...@freebsd.org wrote: > > > > Dear all, > > > > recently the main branch was changed to build the TCP RACK stack > > which is a loadable kernel module, by default: > > https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc > > > > As discussed on the bi-weekly transport call, it would be great if people > > could test the RACK stack for their workload. Please report any problems to > > the > > net@ mailing list or open an issue in the bug tracker and drop me a note > > via email. > > This includes regressions in CPU usage, regressions in performance or any > > other > > unexpected change you observe. > > > > You can load the kernel module using > > kldload tcp_rack > > > > You can make the RACK stack the default stack using > > sysctl net.inet.tcp.functions_default=rack > > > > Based on the feedback we get, the default stack might be switched to the > > RACK stack. > > > > Please let me know if you have any questions. > > I am running main-n266450-a592812327de with a GENERIC-NODEBUG kernel. > > # kldload tcp_rack > kldload: an error occurred while loading module tcp_rack. Please check > dmesg(8) for more details. > > In dmesg: > KLD tcp_rack.ko: depends on tcphpts - not available or version mismatch > linker_load_file: /boot/kernel/tcp_rack.ko - unsupported file type > > So you have to build a kernel with "options TCPHPTS" first? OK, I am now running GENERIC-NODEBUG + "options TCPHPTS". After setting "sysctl net.inet.tcp.functions_default=rack" git no longer works: # sysctl net.inet.tcp.functions_default=rack $ git pull client_loop: send disconnect: Broken pipe fatal: the remote end hung up upon initial contact # sysctl net.inet.tcp.functions_default=freebsd $ git pull Already up to date. -- Herbert
Re: Request for Testing: TCP RACK
Hi, On Thu, 16 Nov 2023 10:13:05 +0100, tue...@freebsd.org wrote: > > Dear all, > > recently the main branch was changed to build the TCP RACK stack > which is a loadable kernel module, by default: > https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc > > As discussed on the bi-weekly transport call, it would be great if people > could test the RACK stack for their workload. Please report any problems to > the > net@ mailing list or open an issue in the bug tracker and drop me a note via > email. > This includes regressions in CPU usage, regressions in performance or any > other > unexpected change you observe. > > You can load the kernel module using > kldload tcp_rack > > You can make the RACK stack the default stack using > sysctl net.inet.tcp.functions_default=rack > > Based on the feedback we get, the default stack might be switched to the > RACK stack. > > Please let me know if you have any questions. I am running main-n266450-a592812327de with a GENERIC-NODEBUG kernel. # kldload tcp_rack kldload: an error occurred while loading module tcp_rack. Please check dmesg(8) for more details. In dmesg: KLD tcp_rack.ko: depends on tcphpts - not available or version mismatch linker_load_file: /boot/kernel/tcp_rack.ko - unsupported file type So you have to build a kernel with "options TCPHPTS" first? -- Herbert
Request for Testing: TCP RACK
Dear all, recently the main branch was changed to build the TCP RACK stack which is a loadable kernel module, by default: https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc As discussed on the bi-weekly transport call, it would be great if people could test the RACK stack for their workload. Please report any problems to the net@ mailing list or open an issue in the bug tracker and drop me a note via email. This includes regressions in CPU usage, regressions in performance or any other unexpected change you observe. You can load the kernel module using kldload tcp_rack You can make the RACK stack the default stack using sysctl net.inet.tcp.functions_default=rack Based on the feedback we get, the default stack might be switched to the RACK stack. Please let me know if you have any questions. Best regards Michael
