Re: Request for Testing: TCP RACK
Hello all, @ current 1500018 and fetching torrents with net-p2p/qbittorrent finished ~2GB download and connection UP until the end: --- Apr 10 11:26:46 leg kernel: re0: watchdog timeout Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.255.0 Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): 192.168.1.255 Apr 10 11:26:49 leg kernel: re0: link state changed to UP Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 --- In the past tests, I've got more watchdog timeouts, connection goes down and a reboot needed to put it back (`service netif restart` didn't work). Other way to reproduce this is using sysutils/restic (backup program) to read/check all files from a remote server via sftp: `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB compressed backup. --- watchdog timeout x3 as above --- restic check fail log @ 15% progress: --- Load(, 17310001, 0) returned error, retrying after 1.7670599s: connection lost Load(, 17456892, 0) returned error, retrying after 4.619104908s: connection lost Load(, 17310001, 0) returned error, retrying after 5.477648517s: connection lost List(lock) returned error, retrying after 293.057766ms: connection lost List(lock) returned error, retrying after 385.206693ms: connection lost List(lock) returned error, retrying after 1.577594281s: connection lost Connection continues UP. Cheers, escreveu (quinta, 28/03/2024 à(s) 15:53): > > On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > > > > Hello all! > > > > Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop > (amd64)! > > > > Thanks all! > Thanks for the feedback! > > Best regards > Michael > > > > Drew Gallatin escreveu (quinta, 21/03/2024 à(s) > 12:58): > > The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > > > > In the common case of a system which has less than the threshold number > of connections , we access the tcp_hpts_softclock function pointer, make > one function call, and access hpts_that_need_softclock, and then return. > So that's 2 variables and a function call. > > > > I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they > are in the same cacheline. Then we'd be hitting just a single line in the > common case. (I've made comments on the review to that effect). > > > > Also, I wonder if the threshold could get higher by default, so that > hpts is never called in this context unless we're to the point where we're > scheduling thousands of runs of the hpts thread (and taking all those clock > interrupts). > > > > Drew > > > > On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > >> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > >>> Ok I have created > >>> > >>> https://reviews.freebsd.org/D44420 > >>> > >>> > >>> To address the issue. I also attach a short version of the patch that > Nuno > >>> can try and validate > >>> > >>> it works. Drew you may want to try this and validate the optimization > does > >>> kick in since I can > >>> > >>> only now test that it does not on my local box :) > >> The patch still causes access to all cpu's cachelines on each userret. > >> It would be much better to inc/check the threshold and only schedule the > >> call when exceeded. Then the call can occur in some dedicated context, > >> like per-CPU thread, instead of userret. > >> > >>> > >>> > >>> R > >>> > >>> > >>> > >>> On 3/18/24 3:42 PM, Drew Gallatin wrote: > No. The goal is to run on every return to userspace for every thread. > > Drew > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > >> I got the idea from > >> > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > >> The gist is that the TCP pacing stuff needs to run frequently, and > >> rather than run it out of a clock interrupt, its more efficient to > run > >> it out of a system call context at just the point where we return to > >> userspace and the cache is trashed anyway. The current > implementation > >> is fine for our workload, but probably not idea for a generic > system. > >> Especially one where something is banging on system calls. > >> > >> Ast's could be the right tool for this, but I'm super unfamiliar > with > >> them, and I can't find any docs on them. > >> > >> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > >> what's happening here? > > This call would need some AST number added, and then it registers the > > ast to run on next return to userspace, for
Re: Request for Testing: TCP RACK
> On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: > > Hello all, > > @ current 1500018 and fetching torrents with net-p2p/qbittorrent finished > ~2GB download and connection UP until the end: > > --- > Apr 10 11:26:46 leg kernel: re0: watchdog timeout > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.255.0 > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): > 192.168.1.255 > Apr 10 11:26:49 leg kernel: re0: link state changed to UP > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > --- > > In the past tests, I've got more watchdog timeouts, connection goes down and > a reboot needed to put it back (`service netif restart` didn't work). > > Other way to reproduce this is using sysutils/restic (backup program) to > read/check all files from a remote server via sftp: > > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB > compressed backup. > > --- > watchdog timeout x3 as above > --- > > restic check fail log @ 15% progress: > --- > > Load(, 17310001, 0) returned error, retrying after > 1.7670599s: connection lost > Load(, 17456892, 0) returned error, retrying after > 4.619104908s: connection lost > Load(, 17310001, 0) returned error, retrying after > 5.477648517s: connection lost > List(lock) returned error, retrying after 293.057766ms: connection lost > List(lock) returned error, retrying after 385.206693ms: connection lost > List(lock) returned error, retrying after 1.577594281s: connection lost > > > Connection continues UP. Hi, I'm not sure what the issue is you are reporting. Could you state what behavior you are experiencing with the base stack and with the RACK stack. In particular, what the difference is? Best regards Michael > > Cheers, > > escreveu (quinta, 28/03/2024 à(s) 15:53): >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: >> >> Hello all! >> >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop (amd64)! >> >> Thanks all! > Thanks for the feedback! > > Best regards > Michael >> >> Drew Gallatin escreveu (quinta, 21/03/2024 à(s) >> 12:58): >> The entire point is to *NOT* go through the overhead of scheduling something >> asynchronously, but to take advantage of the fact that a user/kernel >> transition is going to trash the cache anyway. >> >> In the common case of a system which has less than the threshold number of >> connections , we access the tcp_hpts_softclock function pointer, make one >> function call, and access hpts_that_need_softclock, and then return. So >> that's 2 variables and a function call. >> >> I think it would be preferable to avoid that call, and to move the >> declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they >> are in the same cacheline. Then we'd be hitting just a single line in the >> common case. (I've made comments on the review to that effect). >> >> Also, I wonder if the threshold could get higher by default, so that hpts is >> never called in this context unless we're to the point where we're >> scheduling thousands of runs of the hpts thread (and taking all those clock >> interrupts). >> >> Drew >> >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: Ok I have created https://reviews.freebsd.org/D44420 To address the issue. I also attach a short version of the patch that Nuno can try and validate it works. Drew you may want to try this and validate the optimization does kick in since I can only now test that it does not on my local box :) >>> The patch still causes access to all cpu's cachelines on each userret. >>> It would be much better to inc/check the threshold and only schedule the >>> call when exceeded. Then the call can occur in some dedicated context, >>> like per-CPU thread, instead of userret. >>> R On 3/18/24 3:42 PM, Drew Gallatin wrote: > No. The goal is to run on every return to userspace for every thread. > > Drew > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: >> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >>> I got the idea from >>> https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >>> The gist is that the TCP pacing stuff needs to run frequently, and >>> rather than run it out of a clock interrupt, its more efficient to run >>> it out of a system call context at just the point where we return to >>> userspace and the cache is trashed anyway. The current implementation >>> is fine for our workload, but probably not idea for a generic system. >>> Especially one where something is banging on system calls. >>> >>> Ast's could be the right
Resolved: Panic after update main-n269202-4e7aa03b7076 -> n269230-f6f67f58c19d
After the update to main-n269261-1e6db7be6921, head built & booted OK. FreeBSD 15.0-CURRENT #45 main-n269261-1e6db7be6921: Wed Apr 10 11:11:50 UTC 2024 r...@freebeast.catwhisker.org:/common/S4/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1500018 1500018 Peace, david -- David H. Wolfskill da...@catwhisker.org Alexey Navalny was a courageous man; Putin has made him a martyr. See https://www.catwhisker.org/~david/publickey.gpg for my public key. signature.asc Description: PGP signature
Re: Request for Testing: TCP RACK
With base stack I can complete restic check successfully downloading/reading/checking all files from a "big" remote compressed backup. Changing it to RACK stack, it fails. I run this command often because in the past, compression corruption occured and this is the equivalent of restoring backup to check its integrity. Maybe someone could do a restic test to check if this is reproducible. Thanks, escreveu (quarta, 10/04/2024 à(s) 13:12): > > > > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: > > > > Hello all, > > > > @ current 1500018 and fetching torrents with net-p2p/qbittorrent > finished ~2GB download and connection UP until the end: > > > > --- > > Apr 10 11:26:46 leg kernel: re0: watchdog timeout > > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN > > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 > > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.2550 > > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): > 192.168.1.255 > > Apr 10 11:26:49 leg kernel: re0: link state changed to UP > > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > > --- > > > > In the past tests, I've got more watchdog timeouts, connection goes down > and a reboot needed to put it back (`service netif restart` didn't work). > > > > Other way to reproduce this is using sysutils/restic (backup program) to > read/check all files from a remote server via sftp: > > > > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB > compressed backup. > > > > --- > > watchdog timeout x3 as above > > --- > > > > restic check fail log @ 15% progress: > > --- > > > > Load(, 17310001, 0) returned error, retrying after > 1.7670599s: connection lost > > Load(, 17456892, 0) returned error, retrying after > 4.619104908s: connection lost > > Load(, 17310001, 0) returned error, retrying after > 5.477648517s: connection lost > > List(lock) returned error, retrying after 293.057766ms: connection lost > > List(lock) returned error, retrying after 385.206693ms: connection lost > > List(lock) returned error, retrying after 1.577594281s: connection lost > > > > > > Connection continues UP. > Hi, > > I'm not sure what the issue is you are reporting. Could you state > what behavior you are experiencing with the base stack and with > the RACK stack. In particular, what the difference is? > > Best regards > Michael > > > > Cheers, > > > > escreveu (quinta, 28/03/2024 à(s) 15:53): > >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > >> > >> Hello all! > >> > >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop > (amd64)! > >> > >> Thanks all! > > Thanks for the feedback! > > > > Best regards > > Michael > >> > >> Drew Gallatin escreveu (quinta, 21/03/2024 à(s) > 12:58): > >> The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > >> > >> In the common case of a system which has less than the threshold > number of connections , we access the tcp_hpts_softclock function pointer, > make one function call, and access hpts_that_need_softclock, and then > return. So that's 2 variables and a function call. > >> > >> I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they > are in the same cacheline. Then we'd be hitting just a single line in the > common case. (I've made comments on the review to that effect). > >> > >> Also, I wonder if the threshold could get higher by default, so that > hpts is never called in this context unless we're to the point where we're > scheduling thousands of runs of the hpts thread (and taking all those clock > interrupts). > >> > >> Drew > >> > >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > Ok I have created > > https://reviews.freebsd.org/D44420 > > > To address the issue. I also attach a short version of the patch that > Nuno > can try and validate > > it works. Drew you may want to try this and validate the optimization > does > kick in since I can > > only now test that it does not on my local box :) > >>> The patch still causes access to all cpu's cachelines on each userret. > >>> It would be much better to inc/check the threshold and only schedule > the > >>> call when exceeded. Then the call can occur in some dedicated context, > >>> like per-CPU thread, instead of userret. > >>> > > > R > > > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > No. The goal is to run on every return to userspace for every > thread. > > > > Drew > > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > >> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >
Re: Request for Testing: TCP RACK
(...) Backup server is https://www.rsync.net/ (free 500GB for FreeBSD developers). Nuno Teixeira escreveu (quarta, 10/04/2024 à(s) 13:39): > With base stack I can complete restic check successfully > downloading/reading/checking all files from a "big" remote compressed > backup. > Changing it to RACK stack, it fails. > > I run this command often because in the past, compression corruption > occured and this is the equivalent of restoring backup to check its > integrity. > > Maybe someone could do a restic test to check if this is reproducible. > > Thanks, > > > > escreveu (quarta, 10/04/2024 à(s) 13:12): > >> >> >> > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: >> > >> > Hello all, >> > >> > @ current 1500018 and fetching torrents with net-p2p/qbittorrent >> finished ~2GB download and connection UP until the end: >> > >> > --- >> > Apr 10 11:26:46 leg kernel: re0: watchdog timeout >> > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN >> > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 >> > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): >> 255.255.255.0 >> > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): >> 192.168.1.255 >> > Apr 10 11:26:49 leg kernel: re0: link state changed to UP >> > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 >> > --- >> > >> > In the past tests, I've got more watchdog timeouts, connection goes >> down and a reboot needed to put it back (`service netif restart` didn't >> work). >> > >> > Other way to reproduce this is using sysutils/restic (backup program) >> to read/check all files from a remote server via sftp: >> > >> > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB >> compressed backup. >> > >> > --- >> > watchdog timeout x3 as above >> > --- >> > >> > restic check fail log @ 15% progress: >> > --- >> > >> > Load(, 17310001, 0) returned error, retrying after >> 1.7670599s: connection lost >> > Load(, 17456892, 0) returned error, retrying after >> 4.619104908s: connection lost >> > Load(, 17310001, 0) returned error, retrying after >> 5.477648517s: connection lost >> > List(lock) returned error, retrying after 293.057766ms: connection lost >> > List(lock) returned error, retrying after 385.206693ms: connection lost >> > List(lock) returned error, retrying after 1.577594281s: connection lost >> > >> > >> > Connection continues UP. >> Hi, >> >> I'm not sure what the issue is you are reporting. Could you state >> what behavior you are experiencing with the base stack and with >> the RACK stack. In particular, what the difference is? >> >> Best regards >> Michael >> > >> > Cheers, >> > >> > escreveu (quinta, 28/03/2024 à(s) 15:53): >> >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: >> >> >> >> Hello all! >> >> >> >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop >> (amd64)! >> >> >> >> Thanks all! >> > Thanks for the feedback! >> > >> > Best regards >> > Michael >> >> >> >> Drew Gallatin escreveu (quinta, 21/03/2024 >> à(s) 12:58): >> >> The entire point is to *NOT* go through the overhead of scheduling >> something asynchronously, but to take advantage of the fact that a >> user/kernel transition is going to trash the cache anyway. >> >> >> >> In the common case of a system which has less than the threshold >> number of connections , we access the tcp_hpts_softclock function pointer, >> make one function call, and access hpts_that_need_softclock, and then >> return. So that's 2 variables and a function call. >> >> >> >> I think it would be preferable to avoid that call, and to move the >> declaration of tcp_hpts_softclock and hpts_that_need_softclock so that they >> are in the same cacheline. Then we'd be hitting just a single line in the >> common case. (I've made comments on the review to that effect). >> >> >> >> Also, I wonder if the threshold could get higher by default, so that >> hpts is never called in this context unless we're to the point where we're >> scheduling thousands of runs of the hpts thread (and taking all those clock >> interrupts). >> >> >> >> Drew >> >> >> >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >> >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: >> Ok I have created >> >> https://reviews.freebsd.org/D44420 >> >> >> To address the issue. I also attach a short version of the patch >> that Nuno >> can try and validate >> >> it works. Drew you may want to try this and validate the >> optimization does >> kick in since I can >> >> only now test that it does not on my local box :) >> >>> The patch still causes access to all cpu's cachelines on each userret. >> >>> It would be much better to inc/check the threshold and only schedule >> the >> >>> call when exceeded. Then the call can occur in some dedicated >> context, >> >>> like per-CPU thread, instead of userret. >> >>> >> >> >> R >> >> >>
