[Citadel Development] Re: y creashRe: Database Recovery Tools
I was mostly teasing on that. "get off my lawn" sort of thing :) Tho as i scanned thru it seemed like one of the no-SQL sorts of things. Tue Aug 08 2023 09:23:09 AM EDT from IGnatius T Foobar Subject: Re: y creashRe: Database Recovery Tools The only thing that's "all the rage with the kids" is *calling* it "NoSQL". Berkeley DB has been a simple key/value data store for 30 years. And until recently it's worked fine for us.
[Citadel Development] Re: y creashRe: Database Recovery Tools
Ok, here's what I'm starting to do. You can totally follow the commit logs to stay up to date. database.c has been renamed to database_bdb.c for the obvious reason. (We've been here before, but it was about 25 years ago when we switched from GDBM to Berkeley DB.) The database interface was built to be swapped out. It's going to be even more modular soon. If you look in database.h you're going to see a set of functions that is basically the "driver API" for database backends. My work over the next couple of days will be to completely decouple Berkeley DB from the main server code and turn it into a server module that registers as a database backend. At that point we will be able to build in any database driver we want -- just like the OpenLDAP folks did. I'll probably build the LMDB driver, and configure it to only build on 64-bit systems. With the modular interface, you guys can build SQLite or MariaDB or whatever tickles your fancy.
[Citadel Development] Re: y creashRe: Database Recovery Tools
The only thing that's "all the rage with the kids" is *calling* it "NoSQL". Berkeley DB has been a simple key/value data store for 30 years. And until recently it's worked fine for us.
[Citadel Development] Re: y creashRe: Database Recovery Tools
One of those new fangled no-sql things that is all the rage with the kids these days? Mon Aug 07 2023 10:44:40 AM EDT from IGnatius T Foobar Subject: Re: y creashRe: Database Recovery Tools To be honest I'm leaning towards LMDB,
[Citadel Development] Re: y creashRe: Database Recovery Tools
To be honest I'm leaning towards LMDB, and for the people running on Raspberry Pi we will have to say "look, you're just going to have to install the 64-bit OS before you upgrade" Read through this presentation on LMDB. It's practically designed for our use case: [ http://schd.ws/hosted_files/buildstuff14/96/20141120-BuildStuff-Lightning.pdf ]
[Citadel Development] Re: y creashRe: Database Recovery Tools
The options currently under consideration are SQLite, MariaDB, and LMDB. SQLite: well known, well respected, embeddable. To do it right, however, would require adding a schema layer. The idea of just throwing away the SQL semantics and storing all records as blobs kind of bothers me from a design point of view. MariaDB: requires a separate database server process, plus the schema issue. To be honest I'm not a big fan of this idea. LMDB: also very well regarded, and considered by most in the open source community to be "the" replacement for Berkeley DB. It uses memory mapped file semantics as its storage layer, so it's fast and stable. The downside is that because of memory mapped files, the maximum database size is 4 GB on a 32-bit system. This is not enough for Citadel, so we would have to end support for 32-bit hardware. On x86 this is no problem, but on ARM it could be an issue. More on this soon. We have to do something.
[Citadel Development] Re: y creashRe: Database Recovery Tools
With as long as it was gone, i figured it was not intentional. :( I know i cant contribute code from lack of time and i'm so out of practice, but SQLite might be an easy and safe replacement? ( unless you go with the total abstraction model then its academic ) Mon Aug 07 2023 09:56:00 AM EDT from IGnatius T Foobar Subject: Re: y creashRe: Database Recovery Tools Aaaand Uncensored crashed last night and took the database with it. I don't know what the deal is with this; we've been running Berkeley DB for more than 20 years and it's never been this unstable before. But now it's a liability and I am afraid that some people will begin shying away from Citadel because of it -- if they're not already. It has to go.
[Citadel Development] Re: y creashRe: Database Recovery Tools
Ok, I think I found it. [ https://docs.oracle.com/cd/E17276_01/html/installation/build_unix_small.html ] Easy Install now uses the "--enable-smallbuild" flag, which disables verify. I am going to change it to --with-cryptography=no --disable-hash --disable-heap --disable-queue \ --disable-replication --disable-statistics \ That'll give us back the "verify" code and more verbose error messages too.
[Citadel Development] Re: y creashRe: Database Recovery Tools
I changed your project role from Developer to Maintainer, so hopefully you can do merges now. I'm also going to try to get GitLab to send more "news" to this room.
