[Citadel Development] Re: Removed background and restart from citserver.
While i am super rusty, if we need a non-prod FBSD install to beat on, i can always prop-up a VM. I installed it on that 'box of crap' not long ago out of desperation, and enough memories did come back so i was able to get it to install ok, and some add-ons. Tho i wouldn't be able to use it for its purpose in life due to some missing python stuff :(. But then it promptly crashed under load. .Just like Linux/windows does .
[Citadel Development] Re: Removed background and restart from citserver.
I am not. What is the secret incantation that makes it build on FreeBSD? Fri Sep 08 2023 22:41:47 EDT from IGnatius T Foobar Subject: Re: Removed background and restart from citserver. Kitty -- I hope you're having good results on FreeBSD -- please let me know if it works for you.
[Citadel Development] Re: Removed background and restart from citserver.
Geez. I can't believe how much more stable Citadel Server is with this new improved backend. it was worth spending most of the summer working on this. Kitty -- I hope you're having good results on FreeBSD -- please let me know if it works for you. I've got a test running right now with the load tester running in an auto-restarting loop in one window, and in the other window Citadel Server is running under timeout(1) so it automatically dies from signal 9 after 20-30 seconds. It's been running for close to an hour now, and no matter how catastrophically I make the server die, the database recovers cleanly on the next startup every time. My faith in BDB's recoverability has been restored. I'm just wondering whether we've been using it wrong for the last 20+ years (doubtful, considering LS's substantial talent at this sort of thing) or if some change came along later and silently made it wrong. Anyway, the purpose of this exercise was originally to prepare for moving away from BDB, and we are in a position to add new backends now ... but then I unexpectedly made the BDB backend more or less bulletproof during a late-night coding and testing binge. And so, here is the NEW plan. 1. I am not going to begin working on the LMDB backend at this time. I want to spend my time on WebCit-NG. 2. If anyone wants to work on a backend of any type (LMDB, SQL, whatever) I'll be happy to provide whatever support you need, including modifications to the interface if needed. 3. Naturally, if any stability/durability issues are reported, fixing them will take first priority. 4. I'll do my best to keep everything buildable on FreeBSD. It'll still require gcc and gmake for the time being, but as of right now it does work. I'm trying to line up new hardware to run my stuff on. Hopefully soon. That'll be when I finally move Uncensored to a 64-bit system. I'll proably run the container version.
[Citadel Development] Re: Removed background and restart from citserver.
>The rc system fully detaches it from the entire call stack, in the >end it just looks like init ran the daemon. The rc system is really >just a series of shell scripts, nothing too fancy. That's why PID >files are so important, the rc system uses those to figure out which >PID to send signals to. Cool. Give the new version a try and let me know how it works for you. We don't have the ability to auto-background anymore, but we can still write a PID file. SIGTERM and SIGINT will make the server shut itself down cleanly, and it does so much faster now. SIGKILL and SIGQUIT will make the server exit abruptly, and as previously noted, it doesn't seem to corrupt the db anymore when you do this.
[Citadel Development] Re: Removed background and restart from citserver.
The rc system fully detaches it from the entire call stack, in the end it just looks like init ran the daemon. The rc system is really just a series of shell scripts, nothing too fancy. That's why PID files are so important, the rc system uses those to figure out which PID to send signals to. Fri Sep 01 2023 16:40:26 EDT from IGnatius T Foobar Subject: Re: Removed background and restart from citserver. I have to go examine other BSD rc scripts, but I'm guessing they simply do "&" to put a process in the background, and then they keep track of the pid so they can SIGTERM it later.
[Citadel Development] Re: Removed background and restart from citserver.
FreeBSD's in-base (default) compiler is clang. Moving all of base from gcc to clang was a monumental effort. Fri Sep 01 2023 23:30:53 EDT from IGnatius T Foobar Subject: Re: Removed background and restart from citserver. All righty then. The current code in git master is running on FreeBSD. Here's the deal: 1. You need ldap client library (`pkg install openldap26-client`) even if you aren't using LDAP 2. `gmake` and `gcc` are required. It won't build with whatever FreeBSD is using natively. 3. The configure script now detects whether it needs `-lresolv` and/or `-lintl` automatically 4. We now omit the `chkpw` and `chkpwd` programs on systems where they won't work. I'd actually like to remove system auth completely but I don't know if anyone is using it. On my FreeBSD system I was able to get the server running and I ran the new `loadtest` utility with a few dozen threads beating the crap out of it. I abruptly killed the server with SIGKILL and SIGQUIT numerous times, during periods of high write activity, and even under the worst conditions it did not corrupt the database. Looking really good here.
[Citadel Development] Re: Removed background and restart from citserver.
All righty then. The current code in git master is running on FreeBSD. Here's the deal: 1. You need ldap client library (`pkg install openldap26-client`) even if you aren't using LDAP 2. `gmake` and `gcc` are required. It won't build with whatever FreeBSD is using natively. 3. The configure script now detects whether it needs `-lresolv` and/or `-lintl` automatically 4. We now omit the `chkpw` and `chkpwd` programs on systems where they won't work. I'd actually like to remove system auth completely but I don't know if anyone is using it. On my FreeBSD system I was able to get the server running and I ran the new `loadtest` utility with a few dozen threads beating the crap out of it. I abruptly killed the server with SIGKILL and SIGQUIT numerous times, during periods of high write activity, and even under the worst conditions it did not corrupt the database. Looking really good here.
[Citadel Development] Re: Removed background and restart from citserver.
UUU--- Drive 0 on my desktop is fux0red. This was my main workstation and also where I had my FreeBSD vm installed. Time to rebuild.
[Citadel Development] Re: Removed background and restart from citserver.
nd just like that, my main machine at home bit the dust. *grumble*
[Citadel Development] Re: Removed background and restart from citserver.
>So FreeBSD's rc system sends a SIGTERM when you do "doas service >citserver stop". This should run the same code that cleanly shuts >down citserver when you do a ".ATN y" from the text client. All the Perfect. That's exactly the desired behavior, and it will be "even more that way" now. I have to go examine other BSD rc scripts, but I'm guessing they simply do "&" to put a process in the background, and then they keep track of the pid so they can SIGTERM it later. SIGINT and SIGTERM are handled by running the very same shutdown code that the server's "DOWN" command runs (which is what the text client's "<.A>dmin erminate-server ow" command executes). Unmount the databases cleanly, then exit(0) to GTFO. What's new, however, is that the database "environment" is now on mmap+disk instead of just in-memory. I don't know how we went for so many years without knowing that we had that wrong. This accomplishes two things: 1. The server no longer goes SIGSEGV when we try to close the environment 2. The database never, ever shows up as corrupted when we restart it In other words, the damn journaling system works correctly now. The funny thing is, it used to work fine even in its old form, so I don't know what's changed in the last few versions of Berkeley DB. And here we are, with a BDB implementation that's now rock solid, as a result of a two month long development effort intended to get us *off* BDB. But with that done, I'm gonna focus on FreeBSD for a while.
[Citadel Development] Re: Removed background and restart from citserver.
So FreeBSD's rc system sends a SIGTERM when you do "doas service citserver stop". This should run the same code that cleanly shuts down citserver when you do a ".ATN y" from the text client. All the BSDs opurrate this way, and since it is a standard way of doing things, SMF (Solaris/OpenSolaris/Illumos) also supports this method of daemon shutdown. Probably also a good idea to handle SIGQUIT and SIGKILL this way (for the odd platform that says these aren't untrappable). If FreeBSD's rc system can't get a process to exit cleanly after a short period of time, it will SIGKILL, and that's most likely lead to database corruption on my installation. Thu Aug 31 2023 23:39:48 EDT from IGnatius T Foobar Subject: Re: Removed background and restart from citserver. citserver should exit cleanly when it receives a SIGINT, SIGQUIT, or SIGKILL. This may be why using FreeBSD's rc system caused badness on my system. And it so happens that FreeBSD is the reason I pulled that section of code out. This is an opportunity to remove all of the "clever" stuff and really focus on portable, standard code. I do feel that there's no longer any reason why a server program should have to do the work to put itself into the background. The operating system's supervisor program should be doing that. I'm sure I'll be learning more about how FreeBSD's rc system works. Seems like it's held pretty closely to Unix System III. That's fine, because I've always hated the System V init scripts. I did release what we've got so far as Citadel 991, with some of the FreeBSD changes in place, but I'm going to keep poking at it until it builds cleanly without needing help. Also I have an ACloudGuru account paid for by my employer and that lets me stand up cloud sandboxes for testing things. I'm going to start using that for an occasional build test, I think. The only problem is that the sandboxes automatically self-destruct after four hours, so it's not useful for much more than testing. Right now I've got the latest Easy Install build running on a Scamazon instance, with a 30-thread load test pounding away at it. I figure it's better to abuse their SSDs than mine.
[Citadel Development] Re: Removed background and restart from citserver.
>citserver should exit cleanly when it receives a SIGINT, SIGQUIT, or >SIGKILL. > >This may be why using FreeBSD's rc system caused badness on my >system. And it so happens that FreeBSD is the reason I pulled that section of code out. This is an opportunity to remove all of the "clever" stuff and really focus on portable, standard code. I do feel that there's no longer any reason why a server program should have to do the work to put itself into the background. The operating system's supervisor program should be doing that. I'm sure I'll be learning more about how FreeBSD's rc system works. Seems like it's held pretty closely to Unix System III. That's fine, because I've always hated the System V init scripts. I did release what we've got so far as Citadel 991, with some of the FreeBSD changes in place, but I'm going to keep poking at it until it builds cleanly without needing help. Also I have an ACloudGuru account paid for by my employer and that lets me stand up cloud sandboxes for testing things. I'm going to start using that for an occasional build test, I think. The only problem is that the sandboxes automatically self-destruct after four hours, so it's not useful for much more than testing. Right now I've got the latest Easy Install build running on a Scamazon instance, with a 30-thread load test pounding away at it. I figure it's better to abuse their SSDs than mine.
[Citadel Development] Re: Removed background and restart from citserver.
citserver should exit cleanly when it receives a SIGINT, SIGQUIT, or SIGKILL. This may be why using FreeBSD's rc system caused badness on my system. Thu Aug 31 2023 11:10:32 EDT from rss <> Subject: Removed background and restart from citserver. These functions should be managed by an external supervisor program.
