[Citadel Development] (no subject)

[dothebart]

yes, groupdav iseems more than dead.
btw, openssl?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=828267

[Citadel Development] (no subject)

[dothebart]

while you're at it, maybe also add the initialisation code for ED?
https://github.com/arangodb/arangodb/commit/4a7e54e215e13b12792f1a01d25795455e708071
the chatter about it and what else could be done: https://github.com/arangodb/arangodb/issues/1771
(though the ed just needs some additional initialisation of ssl contexts - see above commit)

[Citadel Development] (no subject)

[dothebart]

I don't think that swaggerized API description is a playbed.
If you want to do REST, you have to design a URL patern, so you can reliably parse them and split "floor / room \ subroom / item" when parsing the URL.
I think the code to work with mail inside of webcit is sophisticated enough to void the aproach of rewriting that in _javascript_.
But if you want to render mails via _javascript_, we have a json template, which you could use to easily get mails in a way the browser js can learn about it.

[Citadel Development] (no subject)

[dothebart]

see some of the docs here:
https://docs.arangodb.com/HttpAqlQueryCursor/QueryResults.html
https://docs.arangodb.com/HttpAqlQuery/index.html
 
The cute thing is, that once you've got documentation like this, generating swagger.io yaml files can be used to automate questioning of the URLs.

[Citadel Development] (no subject)

[dothebart]

ArangoDB does it like that:
 
/_db/databasename/_someothercall/parameter
 
This would prehibit creating of rooms with a leading underscore, but could be an easy way to achieve this:
/main_floor/some/sub/room/_msg/idhash/_attchID/filename.png

[Citadel Development] (no subject)

[dothebart]

 

Tue Apr 19 2016 02:36:05 EDT from the_mgt @ Uncensored 


Mon Apr 18 2016 10:30:57 EDTfrom IGnatius T Foobar @ Uncensored 
By the way, I've made a decision to get rid of the decimal points in our version numbers.  The next version after Citadel 9.01 will be Citadel 902, followed by 903 and 904 and so on.

Why waste the opportunity to call it CitadelX? Why not call the version with the improvements under the hood and the new webinterface CTDLX and then drop the decimal point and simply start counting upwards? It was once called Citadel U/X, iirc. I liked that.


I'm still not convinced this is a good idea.
 
Regarding Rest, we should discuss / find a good solution how to start identifiers for mails in paths, so you can easily tell appart message from A URL like: floor/roomtl/roomsub/message

[Citadel Development] (no subject)

[dothebart]

whoa, this is cool:
https://stackedit.io/editor
 
and, its apache licensed.

[Citadel Development] (no subject)

[dothebart]

no, it seems its outputting them anyways. The macros turn off everything with 'LOG_DEBUG'  by default.
Some of the changes lower precedence from INFO to DEBUG.
the macros also have another advantage the way they're implemented now: you can turn on and off single components at runtime.So you want to follow pop3 sessions? one sendcommand or webcit setting a checkmark - done.
And, without the macros adding the context information they're next to useless, in production - you could #if 0 them for non debug builds, else its like sitting with your but onto a buzz-saw and trying to figure out which of the teeth was the first to bite you.
I experienced several situations, were I wanted to follow exactly one session in the log. Thanks to the macro context information, its easily possible:
citadel/techdoc/logging.txt
explains what to grep in which way.

[Citadel Development] (no subject)

[dothebart]

if you want your log flooded with everything, use
citserver -x all
which will remove all output.
Parts of these macros also add information about the current context to the log string. So, if you have more than one of these rss aggregation jobs running, and want to be able to differenciate between them (which one sent an error message) you need these macros. Else you can simply remove all of these syslog lines, since their output without is meaningless.

[Citadel Development] (no subject)

[dothebart]

Iggy,
if you changed parts of the base64 sh* once more, please remember to check:
  - base64 de/encoding for email attachments
  - authentication in pop3/smtp/imap
  - the xml export/importer...
directix brought us lots of pain with that patch :-(

[Citadel Development] (no subject)

[dothebart]

since # 03cd7985fad9d51530f613304869f3b7960b9f81 removes the ability to mute rss syslog output I will revert it.
even if its debug, having steady noise on syslog is a bad thing to have - as you can read by more than one post in the support room.

[Citadel Development] (no subject)

[dothebart]

well, if you're suffering from the NIH syndrome, we already have something that can serialize json, you only need to add the parser. Probably doable in some hundret lines of code.

[Citadel Development] (no subject)

[dothebart]

 

Wed Jan 06 2016 21:56:12 EST from IGnatius T Foobar @ Uncensored 

I doubt it would look anything like that if I were building it today.  I would probably serialize it to XML.


so at least the parser doesn't have any chance to become fast? XML is s 90'ies.
in the modern world we use json to persist complex structures.

[Citadel Development] (no subject)

[dothebart]

as long as you don't parse stuff by hand in each file again, i'm fine with that. ;-)

[Citadel Development] (no subject)

[dothebart]

Considering uncensored: 300K    /usr/local/citadel/netconfigs/
and outgesourced: 216K    /etc/citadel/netconfigswhich both could be considered to have many netconfigs, and the available memory on a $35 pi, I don't think keeping both is much of a problem?

[Citadel Development] (no subject)

[dothebart]

next missing is the tags handling.
another thing I plan is to have a separate html tree to simply view a blog. Should be easier to brew new layouts with such a reduced set of pages, which may eventually become used in regular webcit later on.
 

[Citadel Development] (no subject)

[dothebart]

ok, I've added a form that uses some js magic to create such a tandem of a blog and a html edit room, plus sets up flags so they share files.
I've also put up a start page into that edit wiki that will demonstrate html syntax and adding the articles the user wants to compose in there.
 
Now next to go - a 'publish' button in the wiki view with some more js magic, and some error handling ;-)

[Citadel Development] (no subject)

[dothebart]

fancy. stackoverflow can even do a visualized diff of markdown edits:
http://stackoverflow.com/posts/34196368/revisions

[Citadel Development] Re: implementing tags for blogs...

[dothebart]

 

Mon Dec 14 2015 07:07:12 EST from IGnatius T Foobar @ Uncensored  Subject: Re: implementing tags for blogs...

You could use a new header byte ... or an RFC822 header 


yep, thats what I thought by eenVelopeTo is useless enough in this usecase.
So I'll go with that.
I'm also thinking about having a fixed structure like that:
 
blogroom -> filedirectory 'blogroom'
blogroom\edit->firedirectory 'blogroom' too.
so, files uploaded in the edit room should be visible in the final blog room too.
Now blogroom\edit will be a wiki room where one can compose the articles. Once its finished, a special *publish* button will copy the article into the blogroom.
 
Only question, the way wikis currently work, and the way comments are attached to blogposts, will editing the wiki entry afterpublishing it work? if one would have to re-publish it, will the comments remain attached to the old article?

[Citadel Development] implementing tags for blogs...

[dothebart]

I'm thinking about a way to implement tags for blogs.
A list of all tags used in that blog should be in the netconfig of that respective room, and I think about abusing one of the existing headers for the tags inside of the toplevel messages:
V    eenVelopeTo       enVelope-to    eenVelopeTo                  The recipient specified in incoming SMTP messages.
feasible since blog posts never are sent in via SMTP?

[Citadel Development] (no subject)

[dothebart]

hm, have a look after that article:
http://www.linuxjournal.com/content/take-control-your-pc-uefi-secure-boot
they embedd a js irc client to freenode.. neat.

[Citadel Development] (no subject)

[dothebart]

commit 8ca6b69ea93167ada31f0f157b7db0fd03bf019fAuthor: Wilfried Goesgens Date:   Tue Nov 17 23:09:21 2015 +0100    Make the swap buffer content function publically available.diff --git a/libcitadel/lib/libcitadel.h b/libcitadel/lib/libcitadel.h+void SwapBuffers(StrBuf *A, StrBuf *B);...
-static inline void SwapBuffers(StrBuf *A, StrBuf *B)+static inline void iSwapBuffers(StrBuf *A, StrBuf *B)...
+void SwapBuffers(StrBuf *A, StrBuf *B)+{+   iSwapBuffers(A, B);+}
 
I guess you have the header, but not the lib. 

[Citadel Development] (no subject)

[dothebart]

thats fresh in libcitadel. you need to have the latest.
It wasn't exported before

[Citadel Development] (no subject)

[dothebart]

are you shure that the chatting, screen sharing and all that stuff is done via SIP?
otoh, thats wbxml via udp, right?

[Citadel Development] (no subject)

[dothebart]

vcard templating done - at least feature wise its similar to what was there before. 
now maybe some goodies and - tada - startup configurable vcard fields, maybe finally images.

[Citadel Development] (no subject)

[dothebart]

 

Wed Nov 18 2015 18:34:16 EST from IGnatius T Foobar @ Uncensored 

While it's obvious that the global footprint of XMPP is now going to be a lot smaller, it's doubtful that the existing XMPP clients are going away, and our implementation of it is pretty much finished anyway. We certainly won't be investing any more time into it. 


Except for implementing Profile picture sending, binary file transfer, audio telephony, video conferencing, ...
So from the perception of the users its far from complete.
(afair Microsoft Lync is based on XMPP and does all that)

[Citadel Development] (no subject)

[dothebart]

http://www.heise.de/newsticker/meldung/FastMail-Google-und-Facebook-graben-XMPP-das-Wasser-ab-2922658.html
(well, the h-open offering is gone, so this is german only...)
XMPP on the way out. FB and Google discontinued their XMPP offerings last big big supporter Fastmail.fm stopped their offerings; whatsapp is winning ground for private users.
Professional users cling to... IRC. yes - me too.
 

[Citadel Development] (no subject)

[dothebart]

http://fedoramagazine.org/make-github-pages-blog-with-pelican/
nice thing. using markdown, as we can.

[Citadel Development] (no subject)

[dothebart]

ok, pop3 client seems to work now.
when the migration also does its job (- transmitts network configs, doesn't loose messages...) i'd be in for a next release.

[Citadel Development] Re: merge

[dothebart]

meld is probably a good tool to use.
usually you start by git pull master and resolve the conflicts.
I wonder what these conflicts may be? I didn't do anything about the configs yet... 

[Citadel Development] Re: merge

[dothebart]

(I wanted to change them so they are transmitted inline in the migrate process instead of r'synced - because of that may fail badly...)

[Citadel Development] Re: Networking issues in 9.01

[dothebart]

implemented with 7b5b6987e1077892e01f9ebdb1f27374011af6db
if an async context didn't do any IO in 10 minutes, its declared dead and will be killed.
So if you want to try it out ;-)

[Citadel Development] Re: Networking issues in 9.01

[dothebart]

Hm, I think I'll write some timeout killer for the event queue jobs; they already cary a timestamp of their last activity so that should work straight ahead.
We will then see whether the kill facility works properly ;-)

[Citadel Development] Re: Networking issues in 9.01

[dothebart]

if you run sendcommand RWHO do you see a citadel networker (believing to be...) talking to uncensored?

[Citadel Development] (no subject)

[dothebart]

Some love to the 'i can't get email page':
http://www.citadelorg/doku.php/faq:troubleshooting:tracking_citadel

[Citadel Development] Re: Config System

[dothebart]

yes - and since thats a very special case, make it a commandline option. Citadel will work for most who do easy install, .deb, .rpm or ebuild.
And if it doesn't, they need to edit their init scripts.

[Citadel Development] Re: Config System

[dothebart]

or - other idea, stat some db file, and pick the uid from that, and just have it specify once at start?

[Citadel Development] Re: Config System

[dothebart]

the -u should be fetched from /etc/defaults/citadel from the startup scripts.
The startup script itself should be static.

[Citadel Development] Re: Config System

[dothebart]

hm, so you have to create the db files as root and chown them?
Regarding the uid, if not commandline,  doing getpwnam of the user citadel to get the uid sounds clever to me - since that should be the default, and like -h the others are very special cases...

[Citadel Development] (no subject)

[dothebart]

@harryc, there is a branch, check_messages; these tests may help you with your pdfs.

[Citadel Development] (no subject)

[dothebart]

yes, Y not.
Having worked for a company that offered an email/imap/calendar/addressbook/syncml server based on c / iiop (corba) and Oracle SQL backend I can tell you a bit about that one isn't going to scale on that.
Working for ArangoDB now, I can tell you clustering isn't going to improve performance either.
The multithreaded aproach citadel has probably offers a good scalability vector in our current situation: cores, more more more cores. Disk space cheap cheaper cheaper

[Citadel Development] (no subject)

[dothebart]

To scale current citadel systems - probably nginx imap reverse proxies, postfix / postscreen.
Heartbeat / HA has STONIS to make shure just one item is running on top of a drdb share.
XMPP / IMAP / whatever clients will reconnect.
You can run webcits on several hosts.
Once you're operating two or more sets of hardware with virtual cloud and blablabla you will find something to give the standby to do while its waiting for the other one to die.
I realy don't think its a usecase.
The advantage of the embedded db engine is a very clear one: you don't need to copy stuff back and forth via a network socket..
If you change that you will add a penalty to each of those db calls, which right now are lightning fast. Plus you have to create the schema and such...
You also have to know that some of the calls work by filtering messages _after_ loading their bodies from the DB, which will then become a performance nightmare.
I relaly don't think its a good idea.

[Citadel Development] Re: CODE FREEZE - RELEASE IMMINENT

[dothebart]

 

Fri Apr 03 2015 10:58:37 EDTfrom IGnatius T Foobar @ Uncensored Subject: Re: CODE FREEZE - RELEASE IMMINENT
From the very beginning, the idea of a pluggable data store has been part of the architecture. We've done it once -- originally we were using "gdbm" but because that was so unstable we switched to Berkeley DB. dothebart would undoubtedly want to use ArangoDB. 
 

well, no. using memory mapped files as database backend for citadel probably wouldn't perform better.

I'm ok with it as long as it does NOT destabilize the code base. We've been on Berkeley DB for a long time now, so the code is probably a bit ossified, and we'll need to do some work to get back that loose coupling we need. Yes, everyone loves to hate Oracle, but the Berkeley DB code is ABSOLUTELY ROCK SOLID and we don't want to gain a reputation for unstable performance -- that honor belongs to Exchange alone. :) But yes you are correct: I want to get *everything* into the database. No flat files anywhere. This includes things like the netconfigs and bio and userpics etc etc etc. This will make the XML export format a lot easier to work with too, because it will *include* all of those pieces without having to rsync them over from system to system. A single dump file really will cover the whole Citadel installation. Since the pluggable DB would be something configured at compile time -- switching databases on an existing installation would be a matter of exporting to XML, recompiling Citadel Server with a new database back end, and loading the dump file. 
 

Yes, getting rid of the rest of the config  text files would be a good thing to do.
However, I don't think images / binary data should be put into the DB - filesystem is perfect for that.
From what I know from Arangodb Clustering doesn't improve performance but rather the opposite plus it makes the code branch in lots of places.
I really don't think we should try to fill that gap - since nobody had scalability issues with citadel so far. Just for failover drdb is perfect.
I also don't think we should opt for a SQL backend. i.e. dbmail is already doing that. It also takes away a significant advantage of being easily configurable, plus the scalability on very small systems is void afterwards.
 
In the things missing i'd  rather add subjects  senders to the fulltext index, and maybe look around for something similar to solr.
Regarding js hacking - fixing that the autocompletion only works for the first recipient would be a good thing to add.

[Citadel Development] Re: CODE FREEZE - RELEASE IMMINENT

[dothebart]

 

Wed Apr 01 2015 12:45:07 EDT from IGnatius T Foobar @ Uncensored  Subject: Re: CODE FREEZE - RELEASE IMMINENT


Whats the release version gonna be? 8.30? 
9.01 I'm updating it now. 


did you try out the discount configure etc. code? I wasn't really able to test it, as I would have liked with queasy install.
 
Btw, why are our env-getters in citadel setup all #if 0'ed?

[Citadel Development] Re: webcit: kernel: [..] traps: webcit[24889] general protection w/patch

[dothebart]

Many thanks for the good work.
to ease the deployment, can you re-send your mails with the diffs as an  attachment?

[Citadel Development] Re: hc changes to date, git diff format attached

[dothebart]

Thanks a lot.
 
I guess if you run CREU http://citadel.org/doku.php/documentation:appproto:users#creu.create.new.user.account with a script, you may get that.
I don't know which admin interface you use to your ldap tree?
 

Thu Mar 19 2015 11:58:19 EDT from harryc @ Uncensored  Subject: hc changes to date, git diff format attached

Please find attached a 'git diff' showing all my suggestions noted recently above, except I didn't implement the variable ldap search string.   I'm still not certain what the best way is there.
I thought to edit out all the changes to automatically generated files, but I decided to leave them in there so you can see what Ubuntu Trusty LTS 14.04 does differently.
There's a further implementation puzzle I'd like to support without having to call it 'an ugly hack'.  When a new person shows up to work, they get a shiny new entry in the LDAP database.  What I need is for that same account to show up in the Citadel database's 'global address book' --- within an hour or so of the ldap record being created--- and here's the puzzle:  before their first log in, or even without requiring the person with the new LDAP record to log in.
That is to say: there's value in having the global address book updated for new ldap entries independently of when or if that person decides to log in or get their first email message.  Any notions?  I have a little script that picks out ldap users that aren't yet listed in PF with a nice filled out vcard.  I'd like to kick Citadel into loading their particulars without having to know their password.  Preferably some 'sendcommand' magic.
Thanks for keeping this solid performer going!
Harry
 

[Citadel Development] Re: hc changes to date, git diff format attached

[dothebart]

hm, the patch only contains the autogenerated files? Can you send the other files as diff attachments?

[Citadel Development] Re: hc changes to date, git diff format attached

[dothebart]

 

Thu Mar 19 2015 13:54:08 EDT from harryc @ Uncensored  Subject: Re: hc changes to date, git diff format attached

Thanks for the thought.  I noticed the citadel server has a line of code in there to block CREU from working when there is LDAP auth. 
 
# sendcommand CREU ItSeemedLikeAGoodIdeasendcommand: started (pid=29593) connecting to Citadel server at /var/run/citadel/citadel-admin.socket200 email1 Citadel server ADMIN CONNECTION ready.CREU ItSeemedLikeAGoodIdea570 User accounts are not created within Citadel in host authentication mode.sendcommand: processing ended.
My intuition was that line of code blocking CREU was added to let LDAP using folks know LDAP was the place to create accounts and auth.
Maybe the CREU code could change when using LDAP so that, only if in LDAP mode, and the account lookup fails, give the LDAP engine a chance to find it.  And if found, then load the details into the vcard, then report 'the account already exists'.  Only if the account doesn't exist does citserver emit the above error message.
What say?? 



Ah, yes didn't know that, but yes, thats probably the best place to alter.
CREU - poll LDAP, if exist, continue doing all the stuff, if not return error user not found on ldap, creu not able to run without in ldap mode.
 

[Citadel Development] Re: CODE FREEZE - RELEASE IMMINENT

[dothebart]

Ok, fixed some tiny leaks, and install stuff.
easy install git will now drag discount - wasn't able to test it with the download yet, can we get queasy install working once more?
http://www.pell.portland.or.us/~orc/Code/discount/discount-2.1.8.tar.gz
should be copied to the download directory.
@oates, the_mgt
discount is the markdown parser library, it should be readily available so you just need to add the build dependency - in debian its libmarkdown.
we have http://oscargodson.github.io/EpicEditor/ as markdown support for the users; it will be part of the webcit tarball.
the .debs currently configure to release it too; the install make target will copy it.
Optionally one can  configure webcit to load it from else where by 
--with-markdowneditordir=/usr/share/citadel-webcit/epic/
(or wherevel else; it will expect to find js/epiceditor.js there)
Whats the release version gonna be? 8.30? 

[Citadel Development] Re: CODE FREEZE - RELEASE IMMINENT

[dothebart]

hm, I've fixed the init script for the .debs, wanted to continue with easy install, but can't seem to find it?
wouldn't setup contain them as here document?

[Citadel Development] (no subject)

[dothebart]

i've added a tiny section for using jshint to revalidate wclib.js  friends:
http://www.citadel.org/doku.php/documentation:customize_webcit#usingjshintforjavascriptsyntaxchecking
could help saving precious time in searching errors.

[Citadel Development] (no subject)

[dothebart]

ok, I like the shape of the stream transcoding now.
The interface seems sufficient so it doesn't need to be changed again after a release giving us new library version for no value.
we can now compress chunks, so we can READ one chunk, compress it, send it as http chunk to the browser, and continue with the next.again.
That way we should be faster sending it, and not have much in ram, so we can handle bigger files more easily.
We also can handle as much of range requests, that http pseudo streaming is possible.
The only thing missing is, that zlib sugests using 64k windows, and we do  4/16k right now - which we should fix. 

[Citadel Development] (no subject)

[dothebart]

the only thing its able to - render markdown to html.
Its used to create markdown wiki rooms - for those who don't like html wiki (like me)

[Citadel Development] (no subject)

[dothebart]

ok, hopefully after the next update uncensored rss should work fine again.
should we start out on another queasy install round?
we've got a new dependency:
http://www.pell.portland.or.us/~orc/Code/discount/
is the q(easy) install script somewhere in git?

[Citadel Development] (no subject)

[dothebart]

hm, theoatmeal is so kind to tell us is time...
   dc:date2015-01-11T11:36:55+01:00/dc:date
which doen't go well with checksumming a feed to see whether one already saw it...

[Citadel Development] (no subject)

[dothebart]

the whole feed
I don't get it why its not identifying the single items properly... it doesn't provide a guid, so this should be a checksum over subject and link, which also don't seem to change...
i've enabled seen logging on uncensored, so maybe this tells us more...
 

[Citadel Development] (no subject)

[dothebart]

hm, yes. I think that log output gives good pointers.
lets fix it.

[Citadel Development] (no subject)

[dothebart]

ok, found  fixed.
For your compfort, the room menu has a relative new point 'zap duplicates' 

[Citadel Development] (no subject)

[dothebart]

finally. been working on that for a while.
zlib compression with chunked encoding is working.
So if you download a file, webcit will only load one chunk from citserver, compress it, send it as http chunk, redo until finished.
 
oh, and in other news, another thing anoying me is gone, gotonext now brings you to the blog entry with new comments instead of the overview.

[Citadel Development] (no subject)

[dothebart]

hm, pretty uncool.
we afe c_baseroom on the config struct; its loaded from the config file, and put in there by the defaults.
However, its not exposed to the CONF interface, so if it gets broken, there is no other way than hexeditor to repair this?
if its not there, the _BASEROOM_ macro is broken, and logging into webcit doesn't work anymore...
whats the best way to fix this? remove c_baseroom at all? or set it on each serverstart?

[Citadel Development] (no subject)

[dothebart]

oops, one more bug in the digest recipients fixed.
one more uncensored update please.

[Citadel Development] (no subject)

[dothebart]

WHOA!
 
something which anoyed me for years fixed.
 
now the folder counters follow the current numbers.
Read a message, new decreases.
Delete/move a message new/total decreases.
Refresh message folder - numbers refreshed.

[Citadel Development] (no subject)

[dothebart]

si.

[Citadel Development] (no subject)

[dothebart]

so.. uncensored has to be updated asap.

[Citadel Development] (no subject)

[dothebart]

ok, another nasty bug caused by our new base64 encoder.
it adds linebreaks, which some places don't expect to.
this breaks...
 - smtp auth
  - imap auth
 - list subscription
 - webcit sieve editing
well, these cases don't gain as much from the improved speed (it was 10 times afair) but now they work with the new behaviour.

[Citadel Development] (no subject)

[dothebart]

hm, maybe finally a successor to tinymce?
http://coolwanglu.github.io/vim.js/experimental/vim.html
;-)

[Citadel Development] (no subject)

[dothebart]

hm, fail. some xmpp was committed which shouldn't, and smtp job finalisation doesn't work - be shure to delete all entries in the smtp queue room before shutting down.

[Citadel Development] (no subject)

[dothebart]

hm, funny #b11aa21e7968de3591f30d7af1ba561fab7bc53f - I had valgrind crash on me when locking the user... after shuffeling stuff a little bit around it wouldn't crash no more...

[Citadel Development] Re: Little patch to serv_migrate.c

[dothebart]

thanks - committed.

[Citadel Development] (no subject)

[dothebart]

Wed Oct 29 2014 07:28:22 EDT from IGnatius T Foobar @ Uncensored 

That's actually a bug that's plagued us for many years, isn't it? 


well, its probably been around since I added the flood protection for aide messages.
in all other cases the code should be slow enough ;-)
However, it seems there is still something similar left in the very same corner :-(

[Citadel Development] (no subject)

[dothebart]

 

Sun Sep 07 2014 05:04:42 EDT from Freakdog @ Dog Pound BBS II 

 

Wed Sep 03 2014 09:23:26 AM EDT from dothebart @ Uncensored 

 

Wed Sep 03 2014 07:51:50 EDT from Freakdog @ Dog Pound BBS II 

 

Tue Sep 02 2014 05:20:32 PM EDT from dothebart @ Uncensored 

I don't have any troubles with spool files, probably due to the single coredness of my server.
If you have, you will have to provide more information so that we can fix thisWFM YMMV ;-)


Hmm...I'm running my Citadel in a VirtualBox VM which has been allocated a single core. The underlying server is running a quad core CPU.I find that if I remove the offending spool file, the issue goes away...I can always identify the offending spool file, easily, as it's the largest spool file in the spoolout directory...continuing to grow but never sent/picked up.
I haven't encountered it in quite a while, though.


so if you re-sync a room this may happen to you?
where does it break? in collectding? or sending?
or does it break while collecting the spoolfiles with large attachments to the spoolfile to transmit?


I haven't resync'd a room in quite a long time...and I was not resyncing any rooms at the times this happened.
It appears to break while sending...if I was reading the logs, correctly, while spooling out, not while actually transmitting the spoolfile. In fact, the spoolfile was growing, I believe, because my node could not connect to the remote, nor did the remote seem to be connecting to me.
I'm not aware of any attachments that may have been in the spoolfiles at the time.


Ok, we have some fixes for this now. I found one missing critical section, and some conditions which would introduce race conditions with the queueing mechanism.
I guess it would be worth while retrying it.

[Citadel Development] (no subject)

[dothebart]

ok, it seems as if I found two possible problems for the networker.

[Citadel Development] (no subject)

[dothebart]

its possible with authentication - however you probably won't get a modified environment without any further exploit - As we don't add i.e. the useragent to the environment ;-)

[Citadel Development] (no subject)

[dothebart]

hm, so the wiki module and the serv_migrate is vulnerable to shell shock? (as they use system() / popen())
I wouldn't count the migrate as that much of a risk, but the wiki for shure.

[Citadel Development] (no subject)

[dothebart]

ok, I had something here which was different to the smtp stack which I changed. It seemed as if it would continue to run after an abort state should have terminated it.

[Citadel Development] (no subject)

[dothebart]

 

Wed Sep 03 2014 07:51:50 EDT from Freakdog @ Dog Pound BBS II 

 

Tue Sep 02 2014 05:20:32 PM EDT from dothebart @ Uncensored 

I don't have any troubles with spool files, probably due to the single coredness of my server.
If you have, you will have to provide more information so that we can fix this.WFM YMMV ;-)


Hmm...I'm running my Citadel in a VirtualBox VM which has been allocated a single core. The underlying server is running a quad core CPU.I find that if I remove the offending spool file, the issue goes away...I can always identify the offending spool file, easily, as it's the largest spool file in the spoolout directory..continuing to grow but never sent/picked up.
I haven't encountered it in quite a while, though.


so if you re-sync a room this may happen to you?
where does it break? in collectding? or sending?
or does it break while collecting the spoolfiles with large attachments to the spoolfile to transmit?

[Citadel Development] (no subject)

[dothebart]

 

Tue Sep 02 2014 09:33:37 EDT from Freakdog @ Dog Pound BBS II 

 

Mon Sep 01 2014 03:20:13 AM EDT from dothebart @ Uncensored 

hm,the disk is full on startup case isn't caught at all in the db-layer...
some pointers become NULL and we crash.


Is this in response to my question about spool files?
If so, the disk isn't full when the spool file issue crops up.
If not, please ignore this message and carry on.


no, neither was.
I don't have any troubles with spool files, probably due to the single coredness of my server.
If you have, you will have to provide more information so that we can fix this.WFM YMMV ;-)

[Citadel Development] (no subject)

[dothebart]

hm,the disk is full on startup case isn't caught at all in the db-layer...
some pointers become NULL and we crash.

[Citadel Development] re: 8.25

[dothebart]

 

Thu Jul 24 2014 11:13:05 EDTfrom harryc @ Uncensored Subject: re: 8.25


Thu Jul 24 2014 03:47:28 EDTfrom dothebart @ Uncensored Subject: re: 8.25

"ok - now that you say it, this makes a lot more sense.
however, if you are to set this string, isn't that rather going to happen during setup?"
You've spotted a little quirk in citadel that I struggle with as well.   Notice that using the command line interface or using sendcommand, all the possible ldap items are there available to be set:  basedn, binddn, and with my little patch also the search string.  So I thought adding the search string setup to the exact same spot as the other ldap items was in keeping with the sentiment of the main authors.  But in webcit one can only choose 'authentication mode' and if that happens to be one of the ldap choices there is no way to set that up other than using the command line or sendcommand.  I wondered perhaps it was the plan of the authors to remove the authentication matter entirely from webcit and that's why they didn't add the ldap setup details? Or was it the ldap setup details were on the list to be added to webcit?   Anyhow it was beyond my available minutes to puzzle that one through so I just continued in the direction I thought I saw there.






 switching authentication modes isn't supported - since you will loose access to the accounts you've created in the other authentication means.
However, loading the contact data probably isn't as permissive.
One idea - sendcommand IGAB should probably do the ldap polling too if used with an ldap enabled setup.. Spidering the ldap for all accounts (and creating them if not there) then would probably also be a good thing to do.






"so at the place where you choose LDAP/AD you would need another mask, defaulted with the two strings?
And, what about editing it? should webcit expose this to the user?"
I suppose if it is the intention of the authors to make webcit able to do everything authentication related the command line / sendcommand is capable of doing, you'd want the current "access" tab on webcit to allow gui entry of the basedn, binddn, and a drop-down box offering the current choices with an added one "LDAP (custom search)".  When the "custom search" is chosen then a text box that is not user editable beneath the access item would become user editable.  For security I'd advise forcing the response to be A-Za-z0-9 and having webcit paste it in (=%s).  Somehow allowing even power-users access to the whole % thing strikes me as an invitation to trouble.    I suppose in a perfect world webcit would create a dropdown on the fly composed of all the items in the ldap schema it finds with format compatible with being a user id and not allow the user to have to know in advance but just choose from a list."






yes, you're shurely right. theres no fun to be made with %s.






 
"On other topics, git master has the draft code for ldaps binding - maybe you could give it a try?"
What a welcome change that would have been for me a year ago.   So I needed security as well, and was trying to decide whether to patch citadel to do ldaps or try some other approach.  In the end, with storage being so cheap and the content of the ldap database changing so slowly (hence the reason for ldap to exist separately from databases) I decided to set up the citadel system as an ldap replication slave using secure communications between the ldap replication slave and the master ldap systems.  In that fashion, a hacker would not be able to tell based on ldap traffic between the mail systems and the ldap servers what's going on on the email system or be able to interfere.  Only when there's a change on the ldap server would there be ldap traffic between the ldap systems and the email systems, and that happens so infrequently it's much easier to detect security shenanigans than lookups every email transit.  Meanwhile all the ldap lookups for email purposes would be local socket calls, never hitting the net.
In direct answer to your question, perhaps a bit shame-facedly,  so... err...  I really have never pulled a development system fresh from the source, compiled and ran it.   I've always used the 'apt-get source' and dpkg-build... tools to keep the systems tidy.   Using debian/ubuntu what's the best practice to keep systems from colliding to get and compile and 'psuedo-real-world install' these sandbox ideas?   Set up a VM I suppose then... well,  I need a clue.
Harry




You can ./bootstrap and then use dpkg-buildpackage from git too, see http://www.citadel.org/dokuphp/installation:debian for details.
LDAPS/Certificates - yes.
A feature complete ssl client would be able to do like that; a first start would however be to have a working encryption at all.
 

[Citadel Development] re: 8.25

[dothebart]

ok - now that you say it, this makes a lot more sense.
however, if you are to set this string, isn't that rather going to happen during setup?
so at the place where you choose LDAP/AD you would need another mask, defaulted with the two strings?
And, what about editing it? should webcit expose this to the user?
 
On other topics, git master has the draft code for ldaps binding - maybe you could give it a try?

[Citadel Development] Re: A certain message format segfaults Citserver 8.24. Patch included.

[dothebart]

Thanks for working this out. If only all troubles were presented in such a clear form :]
The development branch has a special function to check for whether a message field is empty or not - so this fix is only important for releases up to 8.2x; for that reason I'll go for a !=NULL check in that specific case.

[Citadel Development] (no subject)

[dothebart]

8.25? Now?

[Citadel Development] re: 8.25

[dothebart]

please attach the patch to the mail.

[Citadel Development] (no subject)

[dothebart]

btw, currently the xkcd room shows what happenes if any link is translated into a room name - the session is redirected to the lobby, messages are never marked read.

[Citadel Development] Re: Crazy idea

[dothebart]

 

Tue Mar 18 2014 12:00:32 EDT from IGnatius T Foobar @ Uncensored  Subject: Re: Crazy idea

It's an interesting idea, but how would it be superior to simply implementing a clusterable set of Citadel servers running in an active/standby configuration? Right now there are people running Citadel sites in active/standby by sticking the entire thing on top of DRDB, but if we moved *everything* into Berkeley DB that currently sits in flat files, we could use Berkeley DB replication to build that functionality right into the Citadel server. 


you will like that I moved all network config reading/writing into one place.
you only need to change directory iterating and file open/close/read/write by the bdb versions.
One thing which was painfull in the past was that we would loose config information in case of disk full for the netconfig directory - a DB version should make shure under all circumstances that re-creating a config set is successfull before removing the original dataset.
Another file which is meanwhile handled via that mechanism is the mailaliases - also a good thing to have, but creating it initially isn't as easy anymore then.

[Citadel Development] (no subject)

[dothebart]

  Wed Feb 05 2014 09:52:52 EST from IGnatius T Foobar @ Uncensored 


Ok folks, is there anything left in git master that would prevent us from
moving towards a release on that track? 

I'm going to need to start working on some data model changes and would
really like to get a new stable going. 

  

  

  

I didn't have time yet to finalize work on xmpp (currently broken), and zlib
streaming may be desired.  

however, thats not a big change which I don't give a huge chance to conflict
with your changes, so we could cherrypick that over once its done.

[Citadel Development] (no subject)

[dothebart]

in master - yes. at least incomplete.

[Citadel Development] (no subject)

[dothebart]

more broken than in stable.

[Citadel Development] (no subject)

[dothebart]

 

Wed Jan 22 2014 02:50:32 EST from dothebart @ Uncensored 

found fineuploader.com - GPLv3 if you build it yourself.
that way I found out about nodejs, npm, grunt and all its tiny plugins they include minification and other stuff.
Maybe we should add the ability for minification during building the dist-tarballs for webcit too.
 
first files uploaded using that, seems nice.


OK,
Fineuploader fixes my upload issues and works nice.
I hope to resolve one tiny issue, then we can cherrypick that over  do a 8.24 release.

[Citadel Development] (no subject)

[dothebart]

found fineuploader.com - GPLv3 if you build it yourself.
that way I found out about nodejs, npm, grunt and all its tiny plugins they include minification and other stuff.
Maybe we should add the ability for minification during building the dist-tarballs for webcit too.
 
first files uploaded using that, seems nice.

[Citadel Development] (no subject)

[dothebart]

so we have some troubles with our attachment uploading facility.
if one has an nginx proxy, the upload will get interrupted after roughly 8k; regardless of https or not.
This behaviour seems to be portable amongst chromium  iceweasel
Its even broken if one tries to upload files to uncensored, files no bigger 160k fail silently.
i've tried
http://paste.debian.net/76064/
as static.local/test.html
do_template?template=test
and tada, upload working.
any ideas?

[Citadel Development] (no subject)

[dothebart]

its still some work left to do - but not that much.
I guess its forward-cherry-pickeable.
9.0 - yes that was also my Idea.
or... start an 8.xx branch, where we will fork 8.3x from, and if needed maybe an 8.4x; depending on how fast you progress on the nntp hacking.

[Citadel Development] (no subject)

[dothebart]

IG, I sugest you create 8.3x branch now if you start working on nntp - I need some stabilization work, then this would be due to a release.

[Citadel Development] (no subject)

[dothebart]

So took care of the migrate import scenario.
Performance of old (buggy) version with my test data (some but not many big messages): 
time /usr/sbin/sendcommand -h /test migr import  /var/tmp/*xmlsendcommand: started (pid=24356) connecting to Citadel server at /test/var/run/citadel/citadel-admin.socket200 potzblitz Citadel server ADMIN CONNECTION ready.migr import400 sock it to mesendcommand: processing ended.real    3m2.986suser    0m1.886ssys 0m15.951s
 
Whether one uses Splice or not doesn't seem to be so important
time ./citadel/sendcommand -w 5000 -h /test migr import  /var/tmp/*xmlsendcommand: started (pid=7682) connecting to Citadel server at /test/var/run/citadel/citadel-admin.socket200 potzblitz Citadel server ADMIN CONNECTION ready.migr import400 sock it to mesendcommand: processing ended.real    2m14.315suser    0m0.010ssys 0m0.350s
 
So we roughly save 25% real time.
the sendcommand enhancement accounts from 17.8s - 0.36s
it seems not to be so important whether one uses zerocopy (splice) or blockbuffered read/writing in our scenario.

[Citadel Development] (no subject)

[dothebart]

hm, it seems as if after re-importing sieve scripts are not detected anymore - while still present
so, somehow
    CtdlForEachMessage(MSGS_LAST, 1, NULL, SIEVECONFIG, NULL,        get_sieve_config_backend, (void *)u );doesn't grab it anymore.
currently sendcommand import is broken (by me) need more hacking on that.

[Citadel Development] (no subject)

[dothebart]

 

Mon Dec 30 2013 10:17:07 EST from IGnatius T Foobar @ Uncensored 

 

Sun Dec 29 2013 06:14:53 AM EST from dothebart @ Uncensored 

To allow admins to fuzz with webcits renderers (i.e. for hardcore mailq maintenance, or inspecting user settings like sieve scripts) i've created this faq to hand out to those in citadel support
http://www.citadel.org/doku.php/faq:troubleshooting:viewhack


Our UI is starting to look dated again.  I have some ideas, but I won't want to do anything that will conflict with the_mgt's rewrite.
What's the status of that and how can we work around conflicts?


maybe start fixing the_mgts firefox quirks?
 
 

[Citadel Development] (no subject)

[dothebart]

To allow admins to fuzz with webcits renderers (i.e. for hardcore mailq maintenance, or inspecting user settings like sieve scripts) i've created this faq to hand out to those in citadel support
http://www.citadel.org/doku.php/faq:troubleshooting:viewhack

[Citadel Development] (no subject)

[dothebart]

 

Tue Dec 24 2013 07:03:27 EST from dothebart @ Uncensored 

hm, it seems as somewhere in the migrate chain is a 4k limit which can be hit by the seen-records.
my hot guess right now is sendcommand.


Before:
time /usr/sbin/sendcommand migr export  /tmp/blarg.dmp sendcommand: started (pid=29199) connecting to Citadel server at /var/run/citadel/citadel-admin.socket200 potzblitz Citadel server ADMIN CONNECTION ready.migr export100 Exporting all Citadel databases.sendcommand: processing ended.real    2m35.401suser    0m21.811ssys 2m5.756s
After:
time ./sendcommand MIGR export  /tmp/out.dmpsendcommand: started (pid=29014) connecting to Citadel server at /var/run/citadel/citadel-admin.socket200 potzblitz Citadel server ADMIN CONNECTION ready.MIGR export100 Exporting all Citadel databases.sendcommand: processing ended.real    0m10.288suser    0m2.417ssys 0m1.823s
Plus it now doesn't cut overlong lines anymore.
The second place cutting overlong lines:
/* * Import begins here */void migr_do_import(void) {    StrBuf *Buf;    XML_Parser xp;    int linelen;        unbuffer_output();    Buf = NewStrBufPlain(NULL, SIZ);    xp = XML_ParserCreate(NULL);    if (!xp) {        cprintf("%d Failed to create XML parser instance\n", ERROR+INTERNAL_ERROR);        return;    }    XML_SetElementHandler(xp, migr_xml_start, migr_xml_end);    XML_SetCharacterDataHandler(xp, migr_xml_chardata);    CC-dont_term = 1;    cprintf("%d sock it to me\n", SEND_LISTING);    unbuffer_output();    while (CtdlClientGetLine(Buf) = 0  strcmp(ChrPtr(Buf), "000")) {        linelen = StrLength(Buf);        StrBufAppendBufPlain(Buf, HKEY("\n"), 0);        if (server_shutting_down)            break;    // Should we break or return?                if (linelen == 0)            continue;        XML_Parse(xp, ChrPtr(Buf), linelen, 0);    }    XML_Parse(xp, "", 0, 1);    XML_ParserFree(xp);    FreeStrBuf(Buf);    rebuild_euid_index();    rebuild_usersbynumber();    CC-dont_term = 0;}
probably also gained speed next to being non-4k limited anymore.

[Citadel Development] (no subject)

[dothebart]

hm, it seems as somewhere in the migrate chain is a 4k limit which can be hit by the seen-records.
my hot guess right now is sendcommand.

[Citadel Development] (no subject)

[dothebart]

there are those of us, who believe that a 'byzantine syntax' is core concept of a wiki.
while fancy html may offer the plethora of format options, this is advantage and disadvantage at once:
 - documents can wysiwyg look like the user wants it to
 - each document ends up looking individual, you end up with something looking like a turkish bazar.
'byzantine syntax' solves this; all documents have the same style.
so, from now on, citadel will offer the choice; you may choose the roomtype 'wiki' as before to get the old behaviour
Next to this, there is a new roomtype, which offers the very same wiki engine with a different document type:
Markdown: http://daringfireball.net/projects/markdown/
while this is not a classic wiki syntax as i.e. known under wiki-creole, its a syntax gaining more and more ground due to the use on Github, Stack Overflow and various blog softwares.
It also doesn't need any downmix for the textclient, since its pretty well plaintext readable.
We display this by introducing a new mimetype renderer in msg_renderers.c for all documents of the type 'text/x-markdown'.
While this would classicaly mean iso-8859-1 text, we assume this to be UTF-8. To make it clear, we add the proper encoding on saving. 
 
It however introduces two new dependencies:
  - Epic Editor http://epiceditor.com/ to aid the users during creation.( an alternative may be: http://code.google.com/p/pagedown/ )
  - https://github.com/Orc/discount offers libmarkdown2, which seems to be pretty marture since its already available in debian oldstable.
 
More about Markdown: 
http://en.wikipedia.org/wiki/Markdown