So this is happening ...
[ https://code.citadel.org/?p=ig-dkim.git ]
I found a nice tidy little DKIM-signing library, just a few hundred lines
of code, one source module, abandoned for about twelve years after the project
that built it ceased to exist. Big thanks to them for
I'm really pleased with how libical does the heavy lifting for me here. It
has improved greatly since the old days when we were its caretakers. Comparing
two timestamps that are in different time zones actually works. I don't know
how it works, but it works. Just to make sure it wasn't just
Well that's a fine howdy-do, I tagged the release without also mentioning why I pushed it at this time. This version has the config option to advertise STARTTLS on the public SMTP server.
All righty then. Pulling in the RSS feed from cgit, since OinkLab is no longer
in use. There will be some dupes. Here it comes.
For those of you who are pulling updates from the Citadel git repository,
please take note: the address of the repository has changed (again).
It is now: https://code.citadel.org/git/citadel.git
There is also a GitWeb instance at that address, for the benefit of anyone
who wants to
Me too, which is why it took me so long to get out of that headspace. The
truth is, however, Citadel is extremely popular with the operators of small
autonomous sites, not so much the "Exchange killer" I once thought it could
become. That crap all moved to the cloud anyway.
Even now as I am
>That is a hard one. But keeping them separate might help with
>scalability?Â
That was true for WebCit-classic which contains a server-side user interface.
And until a few years ago they could run on separate machines, so you could
have multiple WebCit instances running with a load
I've been thinking lately that it might begin to make sense to put the web
server (really, the web API) directly into Citadel Server. Since it's now
a web API and no longer a user interface, it could be an improvement.
*shrug*
I'm starting to be quite happy with this. WebCit-NG is starting to take
the shape of a very competent webmail client. Very clean, too, since it doesn't
need to push a bunch of other garbage in your face the way commercial offerings
do: "Here's a feature we really want you to use and we won't
Whenever I have some time to kill (boring meeting, waiting for something
to finish, etc) sometimes I will start up a test environment in the clown.
I have a subscription to a training site that allows free time-limited
sandboxes
in the three big hyperscale clowns. And I'll run a Citadel
>git push from an airplane? showoff
Yup. :) Unfortunately I was en route to World Of Crap (AWS re:Invent) while
on the plane.
As a t-mobile customer I get a lot of free wifi benefits. I think they provide
the ground links or something.
Ok, so it has to stay, then. Does sendmail not support LMTP?
*sigh*
Another day, another person who won't listen.
Quick poll for fellow Citerati out there. Does anyone think we still need
the `citmail` utility? As a refresher, this program is an MDA, analagous
to `procmail` or `maildrop`, that files incoming mail into Citadel mailboxes
after
So here's what is on my mind regarding this. Suggestions are welcome.
`/var/run` is a good place to put the PID file, which as previously mentioned
is also the lock file. Mounting the same database from two different instances
of `citserver` would be A Bad Thing. (Even though I'm still
Whoa, what font is that ?!!
I am aware that `/var/run` is the conventional location, and may yet move
it there. Doing so would, however, prevent multiple instances of Citadel
Server from running on the same host. There were definitely people doing
that in the past, but I wonder if
Quick update regarding GitLab and git itself:
I've removed SSH access to the repository. All access should be through
HTTPS.
This is much more secure. And yes, you can store your creds. The recommended
procedure if you plan to commit is to grab yourself a Personal Access Token
using
your
Uncensored has been updated from git master as of right now (Saturday morning).
This includes a fix for a security vulnerability that some weirdo expected
me to be able to read in Japanese. :(
Here's what worked for me:
Going through my history... here's what I've got. (This is with `bash`, alter
accordingly for your favorite shell.)
pkg install gmake gcc autoconf automake binutils gsed libtool intltool openssl
curl expat libical libiconv icu bash git
curl
Geez. I can't believe how much more stable Citadel Server is with this new improved backend. it was worth spending most of the summer working on this.
Kitty -- I hope you're having good results on FreeBSD -- please let me know if it works for you.
I've got a test running right now with the
The above commit has to do with the experience a few of us had with rooms
suddenly containing all new messages.
There was no database corruption. THE DREADED AUTO-PURGER did this, and
it appears to have believed that it was supposed to do that. When we purge
`visit` records, we first
Phooey. This was a bit of a brown-paper-bug. I have to do a full release
of the software to correct it.
Dumping and loading a test system was fine, but when I tried to dump a clone
of Uncensored it showed the memory leak.
No pressure at all! Just looking out to see if you are ok and still with
us. But after a break you'll notice that there is likely some tweaking to
do. Specifically, if we are to have multiple versions of the visit record
(and I do think that's a good way to go) we'll need to tweak `ctdldump` to
>The rc system fully detaches it from the entire call stack, in the
>end it just looks like init ran the daemon. The rc system is really
>just a series of shell scripts, nothing too fancy. That's why PID
>files are so important, the rc system uses those to figure out which
>PID to
All righty then. The current code in git master is running on FreeBSD.
Here's the deal:
1. You need ldap client library (`pkg install openldap26-client`) even if
you aren't using LDAP
2. `gmake` and `gcc` are required. It won't build with whatever FreeBSD
is using natively.
3. The
UUU---
Drive 0 on my desktop is fux0red.
This was my main workstation and also where I had my FreeBSD vm installed.
Time to rebuild.
nd just like that, my main machine at home bit the dust. *grumble*
>So FreeBSD's rc system sends a SIGTERM when you do "doas service
>citserver stop". This should run the same code that cleanly shuts
>down citserver when you do a ".ATN y" from the text client. All the
Perfect. That's exactly the desired behavior, and it will be "even more
that
>citserver should exit cleanly when it receives a SIGINT, SIGQUIT, or
>SIGKILL.
>
>This may be why using FreeBSD's rc system caused badness on my
>system.
And it so happens that FreeBSD is the reason I pulled that section of code
out.
This is an opportunity to remove all
>When I build from master now, when I shutdown citserver, it core dumps right
>after closing the 0d database. Since I had an issue where webcit was core
>dumping and was not re-creatable by anyone else, I am wondering if this
is
>just me.
Remember that discussion, from early
Yet another status update!
With some sleep under my belt and a thankfully slow day at work I've been
able to poke around a little more.
Uncensored crashed a couple of times last night but I think it was because
of a null pointer bug I somehow introduced. Turning off the indexer
Yeesh. It's nearly 2:00am and I still haven't figured out the deadlock
problem.
But I can make it appear pretty reliably by doing these things together:
1. loadtest, with 3 threads
2. while true ; do ./sendcommand tdap ; sleep 1 ; done
I also have a room on the test system that
Yecch. The new code has a deadlock problem. The DB is rock solid though.
On it.
Oh, and in case I didn't make it abundantly clear: the remaining backend
decoupling work is complete, and work on more modern backends can begin soon.
LMDB? MariaDB? SeattleCloud Hosted DB? You name it, we'll be able to do
it.
Ok, I've got my FreeBSD system installed and have begun making changes to
fix any problems that came up.
I am also now running Uncensored on 'git master' (commit
6960933711b8353ad5cddcc17994db670911a1dc)
as of right now. When it comes to dealing with database corruption, in the
words of
Ok, I've got my FreeBSD VM set up and will give the Citadel build a shot today.
\
I'm having *really* good results with this latest commit. I haven't gotten
it to corrupt the db no matter how badly I abuse it. This is the one.
The only issue so far is that under multiple very heavy load tests, I have
gotten it to deadlock. But I don't know whether that's a new
All right everyone, this is the one to test. I think I've fixed all of the
conditions that would make it corrupt the database. PLEASE test this version
if you have the ability to. (The version currently in git master, not the
release version on the web site.)
The funny thing is, in
Geez. The more I look into the Berkeley DB back end the more fucked up I
find it is. Disregard the previous comment about the locking subsystem.
I changed dbenv->open() flags again, this time
DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN |
DB_RECOVER
| DB_THREAD
Some interesting developments here.
I disabled Berkeley DB's locking subsystem, because Citadel handles locking
and concurrency on its own. This seems to have fixed the segfault when it
closes the database environment. I also have not managed to corrupt the
database
no matter how badly
Clarification of the above commit -- because it's important.
When traversing an entire table (as opposed to just fetching one row for
which you already know the key) the backend API now returns both a key and
a value. This is done by returning a "struct cdbkeyval" which simply contains
a
HarlowSolutions: I noticed that the IMAP Flags branch is no longer in the
repo. Are you refactoring it? Do you need anything from me to move forward?
I've been heads down on this stupid database thing lately.
Putting HTML directly into the editor does work if you push the "HTML" button
:)
(...and someday soon, when this whole war on berkeley db thing is finished,
we'll get back to webcit-ng...)
UFarx: the formatter messed up that patch, I think.
just remove the br tags
I ended up having to apply your changes manually. Thank you for the patch, but in the future please either post it as an attachment (good) or upload it to our gitlab as a merge request (better).
And thank you
UFarx: the formatter messed up that patch, I think. Can you put it in as
a merge request? I'll pm you a login.
>if that's welcome, i'm happy to beef things up a little without
>sacrificing any compatibility.
I'd be pleased to see that. Maybe just do one component first so we can
see what it will look like.
We try to make the build work cleanly on FreeBSD and Linux. There is no
interest in
All right, that worked much better. The database backend interface has been
reverted to what it was before. The backend for Berkeley DB still needs
thread-specific
data, but now it maintains that data locally to the module instead of tainting
the server with its own needs.
Sensibly, when
> Therefore I am changing the API so that cdb_rewind() returns a cursor,
>and cdb_next_item() must be passed that cursor with every call. As
Well that was a disaster, and with more people involved now I ought to test
this stuff before announcing an API change.
I think I'm going to
>as for creating a patch, i would but if you wanna switch to cmake
>anyway it might not be necessary anymore. a word of caution though
>regarding cmake:
CMake is interesting and I'm happy to learn about it, but we've got too much
going on right now to think about switching to a new
>I already did the account creation thingy.
Ok, I approved the account. I guess I need to set up more alerts on that
sort of thing.
Absolutely! Just go to https://code.citadel.org and create an account. You
will need to use your @uncensored.citadel.org address, at least initially.
I'll approve it in short order.
>If it were up to me, the entire build system would be using CMake.Â
That does look nice and clean.
My objective wasn't to stop using Make, but to stop using the GNU Autotools
stuff. Those tools are built to handle all sorts of obscure edge cases that
don't exist anymore. There just
More notes on the database abstraction layer:
I did point out that the API isn't totally frozen :)
For database cursors, we were keeping cursor variables around
per-thread-per-table,
which requires putting Berkeley DB specific stuff in server/threads.h , and
that's got to go.
UFarx: your suggestions for the build system are most welcome.
I don't suppose we could ask you to submit a patch? We have a GitLab instance
at [ https://code.citadel.org ] which accepts merge requests to the repository.
(You'll need to sign up for it using uf...@uncensored.citadel.org
Work on the abstraction layer for the storage backend continues.
There is a new directory ./server/backends/ that is similar in design to
./server/modules/ for the rest of the server modules. I chose to open a
different
hierarchy because, as previously mentioned, the storage backend
All righty, folks ... anyone looking to build a database backend of their
own, please check out the latest batch of changes in the git repository.
It basically works like this:
* database.c has been renamed to database_bdb.c
* database.h now basically declares an API for storage
Ok, here's what I'm starting to do. You can totally follow the commit logs
to stay up to date.
database.c has been renamed to database_bdb.c for the obvious reason. (We've
been here before, but it was about 25 years ago when we switched from GDBM
to Berkeley DB.) The database interface
The only thing that's "all the rage with the kids" is *calling* it "NoSQL".
Berkeley DB has been a simple key/value data store for 30 years. And until
recently it's worked fine for us.
To be honest I'm leaning towards LMDB, and for the people running on Raspberry
Pi we will have to say "look, you're just going to have to install the 64-bit
OS before you upgrade"
Read through this presentation on LMDB. It's practically designed for our
use case:
[
The options currently under consideration are SQLite, MariaDB, and LMDB.
SQLite: well known, well respected, embeddable. To do it right, however,
would require adding a schema layer. The idea of just throwing away the SQL
semantics and storing all records as blobs kind of bothers me
Ok, I think I found it.
[ https://docs.oracle.com/cd/E17276_01/html/installation/build_unix_small.html
]
Easy Install now uses the "--enable-smallbuild" flag, which disables verify.
I am going to change it to
--with-cryptography=no --disable-hash
I changed your project role from Developer to Maintainer, so hopefully you
can do merges now. I'm also going to try to get GitLab to send more "news"
to this room.
Ok, here is the DB installation:
$MAKE $MAKEOPTS || die
$MAKE install_lib || die
$MAKE install_include || die
$MAKE install_utilities || die
So they *should* be there. Are your tools not on the system, or did it throw
an error message when you tried
Did your db get corrupted because you were playing around with a development
system, or did it happen on its own on a production system?
I've just about had it with Berkeley DB, which is why I've spent a large
number of hours on ctdldump/ctdlload. This is our escape hatch once we have
an
Crap. Ok, I'll do the merge, but I really need to figure out what the heck
is happening.
Sorry about that, it just got away from me. I approved it. Looks good.
I agree. The developer who introduced StrBuf and ConstBuf was helpful in
giving us an elastic string class, but I wish he hadn't gone back through
old code and converted stuff without a really deep inspection of how it was
used.
>Not sure why the parameter was empty, but a lot of code accessed ConstStr
>pointers without checking for zero length and pointer was null and could
>coredump, so went through and added checks.
*sigh*
The IMAP server was written completely with straight C pointers and string
>I am changing to updating the records on they fly rather than upgrading,
so
>the version number is really not that important. New records use a version
>signature and don't compare to the citserver version so no conflict.
Sounds good. We are officially changed over from
That is more than sufficient, thanks!
I just pushed an update out that's going to do something ... that will either
win big or lose big.
It's possible that the occasional database corruption has something to do
with aborting so quickly when an occasional segfault happens (for whatever
reason) and it just stops without closing
I would certainly welcome having a chance to test the build on that target
and see how it goes.
Do you have a live system you can use as a test of source data? If so, the
most excellent test would be (after I get ctdldump and ctdlload into the build)
to dump your system, load the dump on another system, and go through it
carefully
to see if the source system was reproduced with 100%
Oh, and I pushed the release button a few times, so when the visit merge is
done we'll have to adjust the version numbers being applied.
> If not then I'll merge and then troubleshoot. I think when we tell it
>not to squash the commits, it brings in your entire commit history
>followed by a merge commit.
Ok, I went ahead and tried it. It did exactly that. The commits that make
up the merge are written to the log
F*g hell. I had to restore from a backup this afternoon because my
database
bit the dust.
Looks like it's time to accelerate the replacement of Berkeley DB.
ctdldump/ctdlload is functionally complete, and we can start testing it.
I'm also thinking that maybe we want to split
I had the same problem, and then I realized I was not logged in. Any chance
that's happening to you?
If not then I'll merge and then troubleshoot. I think when we tell it not
to squash the commits, it brings in your entire commit history followed by
a merge commit.
Geez. Nice find. Approved, of course.
All good, take as much time as you need on the IMAP flags stuff. We'll make
sure we do it right.
As you've undoubtedly noticed if you follow the commit logs, over the last
couple of weeks I've written "ctdldump" and "ctdlload" utilities that can
be used to dump a Citadel database into a flat
To get things started, I have begun work on `ctdldump` and `ctdlload`
frameworks.
I've also improved the Makefile for Citadel Server so that it doesn't require
compiling the entire server every time even one file is touched. I got lazy
when I removed the GNU Autotools and just had it
Potentially, yes. I'm thinking more in terms of how to refactor the code.
The database access stuff is all in `database.c` because there *was* a somewhat
pluggable architecture in the past -- from flat files to gdbm and later to
Berkeley DB.
Since we've been using Berkeley DB exclusively
Grrr. Unrecoverable error on my database this morning. I think maybe it
is time to put Berkeley DB out to pasture after all.
Let's think about what we want the next to look like.
>Pretty consistent for me. I put a check in the Berkley code to work around,
>but not sure of the root cause either.
Something must be getting double freed, but I can't figure out what.
>I just realized that a Seen message set of *:2099348686 means all messages
>have been seen. Were all your messages unseen before the upgrade? I
do
A seen message set of *:2099348686 means all messages have been seen up to
and including 2099348686.
We are using what RFC 3501
>I was trying to figure out how to reject the request myself. Does not
look
Do you have the button "Close merge request" on the bottom of the screen?
>It looks like the Berkley code is passing a NULL to a memory free routine
>that does not check for NULL. Anyone else having a problem? If not,
I
>will look into how I build the server.
It happens to me too, some of the time, and I have been busting my butt trying
to figure it
Ok then, I think the proper workflow is to cancel the merge request and start
a new one? I could reject it but that seems rude :)
It isn't just "a seen message set" -- it was ALL of them, across all rooms,
at least for me. But if you want an example, my current seen-set for this
room
You mean like this? ;)
[ https://code.citadel.org/citadel/citadel-docker ]
I'd like to get things trending towards more deployment of the Docker image
(or technically an OCI compatible container, since you don't have to run it
under Docker, blah blah blah)
The current distribution is
Ok, I tried out the proposed merge. Here are the results.
1. I made a copy of the Uncensored database and ran it on the main line code
to verify that the copy is functioning identically to the production database.
2. I switched over to the merged branch and built everything from
>If you want to stay with index/data for the database, I think I might go
back
>and see if I can finish up a database.c replacement with MySql. What I
Hey, I'm totally cool with moving to a SQL database, if someone else is doing
to be doing the bulk of the work. Seriously. I am just
>Just to chime in, the database.c code is very clean. The only problem
I
>have had with Berkeley is that if the code or system crashes, most of the
>time, the database is corrupted and recover does not help sometimes and
I
>have to revert to a backup. The code has become a lot more
I'm away on vacation right now and casually browsing the changes. I'll give
it a good thorough testing when I'm back at my desk after the weekend, but
so far I'm quite impressed. It looks like the changes really follow not just
the coding style, but the *design* style of the existing Citadel
>Not sure if regression, but if you try to sent a message with a blank body,
>webcit coredumps. Just checked for empty message text and skipped trying
to
>output.
I wasn't able to reproduce the problem but the patch looks like it will add
some safety so I pulled it in anyway.
Right, and we do table-level locking in the application before we even attempt
to touch the database, so that has worked pretty well.
I did end up using an alternate --prefix just to keep the build a little
more solid, but that seems to just change the symbol names internally without
actually
Heh. Most of the database layer is still pretty much the same code you wrote
22 years ago. You did a great job on that.
I've thought about switching to a SQL model. It would confer a lot of
benefits.
It would also open up a huge effort in getting all of the code converted
over to
I see the merge request and will check it out. Thanks!
A bit of thinking out loud with regard to long term plans for Citadel Server.
Berkeley DB seems to have gained a bad reputation in the open source community.
Much of it began in 2013 when Oracle switched it from the Sleepycat license
to the AGPL. Debian discontinued updating it
Oh! Ok, that's why I didn't see a merge request. Cool, that's exactly how
it should be. :)
In other news, I finally figured out why the server kept crashing (on this
site). Someone put a match string in their inbox rules that doesn't compile
to a regular expression. regcomp() fails, and
I also see that the `Support_IMAP_Flagged_Deleted_Draft_Recent_flags` branch
is still there. I thought it was going to get deleted in the merge. Mind
if I delete it?
I also have to make sure I'm not "squashing" multiple commits so they all
get their own commit log messages.
Thanks for
I'm assuming you're familiar with the industry-wide standard for how to write
a commit message (summary line of 50 characters or less, then if you need
to type more, skip a line and then make the longer description lines of 72
characters or less).
If you're doing that in the commit message,
>From my experience using other gitlab installations, yes you have to create
a branch and then commit that branch, and then you can create a merge request.
When the merge is approved (and I haven't set any approval rules so we might
have to experiment there) there is a "delete source branch"
1 - 100 of 1678 matches
Mail list logo
