[imp] Horde IMAP features (or why our webmail client is better than your webmail client)
Been meaning to do this for awhile, and this is only sort of a half-hearted first attempt/draft, but in the past there has been interest into what has actually gone on internally in Horde 4/IMP 5 to improve performance and features. Answer: a bunch. I can proudly state that Horde_Imap_Client, the Horde library that powers IMP, is by far the most complete, fastest, and most feature rich PHP library out there. As a webmail developer, I often monitor the other freely available options there to gauge our process. And my honest assessment is that we are unarguably the best solution when it comes to features, performance, and stability. So it might be good to list what I think some of these features are so that others can compare and see for themselves what we do better than anyone else. To further this goal, I've started to add some content to the Horde_Imap_Client page: http://wiki.horde.org/Project/HordeImapLib If there is interest, I will try to expand on some of the existing entries (and add additional information). I've thought about providing side-by-side outputs of things like IMAP transaction logs with other clients (webmail or desktop) to show our efficiency, but I'm wondering if that might be too esoteric for most people - IMAP is an incredibly complex protocol so the benefits may not be immediately or easily recognizable. But if there is enough interest, I would consider providing these kind of demonstrations. For those that don't care about any of the above, and want a distilled version, I'll present the three things I am most proud about the Horde_Imap_Client library: 1. PHP Memory stream support. Unlike every other PHP library I am aware of, our library does not keep the full data of body parts in memory. Since attachments can be huge (10-20 MB), this could cause memory issues on the server. Those running IMP 4 might be aware of this - memory limits for PHP had to be set high, partially because the PHP imap extension commands we used required all data to live in memory. However, Horde_Imap_Client instead stores body part data in temporary streams. These streams only store a limited amount of data in memory (2 MB), with the rest being spooled to a temporary file somewhere on your local system. This results in vastly reduced memory loads when viewing/downloading message body parts. As an example - I have had my memory limit set to 64 MB for over a year now. I have not once run out of memory (as a developer, I often play around with large messages just to test for things like these). 2. CONDSTORE/QRESYNC support. These are two recent IMAP extensions specifically created to add disconnected clients (like IMP) keep their local caches synced and to speed viewing when reloading a mailbox. IMP is the *ONLY* open-source webmail client that I am aware of that supports these extensions. Discussing the exact details is beyond the scope of this message, but absence of CONDSTORE/QRESYNC support means two things: * You can not properly cache mailbox data. Namely, flags are not guaranteed to be synced unless they are downloaded EVERY time you access a mailbox. Which is very slow. So, in other words, all other webmail clients are essentially broken - only IMP does caching correctly. * You can not properly keep an AJAX display synchronized. A well-written AJAX client contains yet another cache of mailbox data - it needs to be synchronized with the PHP server (which in turn needs to be synchronized with the IMAP server. Confused yet?) Without CONDSTORE/QRESYNC, this can not be done correctly absent the AJAX interface *constantly* reloading data from the server. But this is just a waste of bandwidth, processing power, and the user's time. Only IMP does AJAX display correctly because we only download what is needed. 3. Fast. The PHP imap (c-client) extension is notoriously bad at its IMAP conversations. It would send duplicate requests, couldn't cache entries, and would request data for messages that you didn't even care about (the infamous lookahead feature). Additionally, there were many things that couldn't be done at all - like obtaining namespace information on the server - because the extension did not support it so an entirely separate library needed to be written, and and an entirely separate connection to the IMAP server needed to be made, to fill in the gaps. Horde_Imap_Client has replaced all this was a single, coherent library that has been relentlessly tuned to ensure that we are doing things the most efficient way possible. This is probably a good place to end this message. I try not to be braggy very often about the projects I am involved in, but now that IMP 5 has begun to stabilize I figure the next step is to try to inform people why it should be their choice. michael ___ Michael Slusarz
Re: [imp] db migration and sorting
Quoting Janis Eisaks j...@ktf.rtu.lv: Quoting Jan Schneider j...@horde.org: 2. what can be done to display folder list in alphabetical order for selected language - for example letter č in Latvian follows c, not z Horde 1.0.8 places national letters before A, 1.2.9 and Horde4/IMP5 - after Z We use strnatcasecmp and strcasecmp for sorting mailboxes. Those have the advantage of correctly sorting mailboxes with numbers and being binary-safe. For better locale-specific sorting, we would have to use strcoll, but this function doesn't support either of the former features. so - resumé - it is not possible to get folders in correct alphabetical order rathen than for ASCII-named, isn't it? It looks like we might be able to use Collator::compare() (http://www.php.net/manual/en/collator.compare.php) to do the sorting. But this is only available in PHP 5.3.0+. michael ___ Michael Slusarz [slus...@horde.org] -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] db migration and sorting
Zitat von Michael M Slusarz slus...@horde.org: Quoting Janis Eisaks j...@ktf.rtu.lv: Quoting Jan Schneider j...@horde.org: 2. what can be done to display folder list in alphabetical order for selected language - for example letter č in Latvian follows c, not z Horde 1.0.8 places national letters before A, 1.2.9 and Horde4/IMP5 - after Z We use strnatcasecmp and strcasecmp for sorting mailboxes. Those have the advantage of correctly sorting mailboxes with numbers and being binary-safe. For better locale-specific sorting, we would have to use strcoll, but this function doesn't support either of the former features. so - resumé - it is not possible to get folders in correct alphabetical order rathen than for ASCII-named, isn't it? It looks like we might be able to use Collator::compare() (http://www.php.net/manual/en/collator.compare.php) to do the sorting. But this is only available in PHP 5.3.0+. We could use it conditionally, it's also available through intl from pecl for PHP 5.2. But I'm not sure if that would get us anything that strcoll doesn't. The documentation is sparse, but to me it sounds like it doesn't get us natural numeric sort or binary safety (probably not necessary for mailbox sorting anyway) either. Jan. -- Do you need professional PHP or Horde consulting? http://horde.org/consulting/ -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] pdf
Zitat von J?nis j...@ktf.rtu.lv: Hi! I noticed that during install the Horde PDF library is installed along others - where (and how) it is used? Not in IMP. You can export notes to PDF in Mnemo. Jan. -- Do you need professional PHP or Horde consulting? http://horde.org/consulting/ -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
[imp] Spam Problem ... close to a solution ... may be you could help?
Hi, I did not find the compromised account yet, but I see a lot off messages like the following one in our logs: /var/log/httpd/ssl_request_log.1:[21/May/2011:01:10:54 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=721hskg326yc HTTP/1.1 92 /var/log/httpd/ssl_request_log.1:[21/May/2011:01:14:38 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=6khanz8ousab HTTP/1.1 92 /var/log/httpd/ssl_request_log.1:[21/May/2011:01:24:41 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=2bcbqsb503hi HTTP/1.1 92 May be anyone has an idea how to protect against such direct postings... if it is possible anyway? Any suggestion is welcome -- Götz Reinicke IT-Koordinator Tel. +49 7141 969 420 Fax +49 7141 969 55 420 E-Mail goetz.reini...@filmakademie.de Filmakademie Baden-Württemberg GmbH Akademiehof 10 71638 Ludwigsburg www.filmakademie.de Eintragung Amtsgericht Stuttgart HRB 205016 Vorsitzende des Aufsichtsrats: Prof. Dr. Claudia Hübner Geschäftsführer: Prof. Thomas Schadt smime.p7s Description: S/MIME Cryptographic Signature -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] Spam Problem ... close to a solution ... may be you could help?
On 05/24/2011 07:53 AM, � wrote: Hi, I did not find the compromised account yet, but I see a lot off messages like the following one in our logs: /var/log/httpd/ssl_request_log.1:[21/May/2011:01:10:54 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=721hskg326yc HTTP/1.1 92 /var/log/httpd/ssl_request_log.1:[21/May/2011:01:14:38 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=6khanz8ousab HTTP/1.1 92 /var/log/httpd/ssl_request_log.1:[21/May/2011:01:24:41 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=2bcbqsb503hi HTTP/1.1 92 May be anyone has an idea how to protect against such direct postings... if it is possible anyway? Any suggestion is welcome OK, you said any suggestion... We use two techniques to stop outgoing spam. The first is pretty complex and not for everyone. You must use version control like Git or Bazaar to keep your local code changes safe when you do upstream updates (and using PEAR is out). The second technique is simple but not free. 1. We run memcache on the horde servers. We then added local code to horde/imp/lib/Compose.php to save and update a 24 hour count of recipients in memcache for a sender. Then when a sender hits the 24 hr limit or a limit for the number of addresses in a single email (spammers love to send to 40 or 50 BCC addresses), we deactivate the sender (so they can not send any more until an admin has taken action) and send a note to an admin so someone can follow up and decide if this is a spammer OR a valid user whose account was stolen. I am sure someone like Chuck or Jan could write a patch for you on a consulting basis. I have been trying to get our code organized and clean enough that we could submit it as something just about anyone could use. But right now it is tied pretty closely to our LDAP user store which is pretty complicated. 2. This is not free...For the horde mailer config item we use smtp and point it to antespam.com. AnteSpam checks the outgoing email for spam and viruses. When it finds either, it quarantines the email and sends a note to the user and the domain admin. The user can manually free the email from quarantine. But since a spammer needs to send a large quantity of spam, needing to release emails from quarantine quickly kills his/her profits. Also, the domain admin can also look at the email and if it is really spam, they can quickly shut down the spammer. I wish I had a better idea to suggest. But we have found that spammers are clever and persistent and hard to stop. Good luck, -- Andy Dorman FanMail.com -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] db migration and sorting
Quoting Jan Schneider j...@horde.org: Zitat von Michael M Slusarz slus...@horde.org: Quoting Janis Eisaks j...@ktf.rtu.lv: Quoting Jan Schneider j...@horde.org: 2. what can be done to display folder list in alphabetical order for selected language - for example letter č in Latvian follows c, not z Horde 1.0.8 places national letters before A, 1.2.9 and Horde4/IMP5 - after Z We use strnatcasecmp and strcasecmp for sorting mailboxes. Those have the advantage of correctly sorting mailboxes with numbers and being binary-safe. For better locale-specific sorting, we would have to use strcoll, but this function doesn't support either of the former features. so - resumé - it is not possible to get folders in correct alphabetical order rathen than for ASCII-named, isn't it? It looks like we might be able to use Collator::compare() (http://www.php.net/manual/en/collator.compare.php) to do the sorting. But this is only available in PHP 5.3.0+. We could use it conditionally, it's also available through intl from pecl for PHP 5.2. But I'm not sure if that would get us anything that strcoll doesn't. The documentation is sparse, but to me it sounds like it doesn't get us natural numeric sort or binary safety (probably not necessary for mailbox sorting anyway) either. Well... I'm not sure that our current sorting method is all that bad. It's at least consistent. And if PHP does not provide the proper tools to easily do this kind of sorting, I would rather not reinvent the wheel. michael ___ Michael Slusarz [slus...@horde.org] -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] Spam Problem ... close to a solution ... may be you could help?
Quoting Andy Dorman ador...@ironicdesign.com: Also, the domain admin can also look at the email and if it is really spam, they can quickly shut down the spammer. Off-topic - I like to know how much spam they would have sent, so when I verify it's spam I redirect their outgoing mail to /dev/null but continue to count the recipients. It blows your mind sometimes... I also apply the same 'counting' to my entire outgoing queue. I've found that while the per-user limits are helpful, those smarter spammers will just create more accounts. By monitoring the entire mail flow for traffic spikes, I can shut down the entire outgoing queue and remove the abuser(s). I use qmail, but any SMTP server should work. Essentially: 1. Route all 'non-verified' users mail from the incoming SMTP server to 192.168.1.1. 2. Route all mail from 192.168.1.1 to 192.168.1.2. 3. 192.168.1.2 runs smtp-delay to 'pause' traffic on 192.168.1.1 4. Run a cronjob that counts the number of emails in queue on 192.168.1.1 every minute. You will need to tweak both the counts and delay times for your environment (and as you environment scales up). So for example, if you consistently have 20 emails in queue, and spike to 60 under normal operations, set your program to shut the queue down at 70 and alert the admin. You will obviously need staff to manage the alert (if you're swamped with spammers) and/or understanding from your users that this will occur. Rick -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] db migration and sorting
Zitat von Michael M Slusarz slus...@horde.org: Quoting Jan Schneider j...@horde.org: Zitat von Michael M Slusarz slus...@horde.org: Quoting Janis Eisaks j...@ktf.rtu.lv: Quoting Jan Schneider j...@horde.org: 2. what can be done to display folder list in alphabetical order for selected language - for example letter č in Latvian follows c, not z Horde 1.0.8 places national letters before A, 1.2.9 and Horde4/IMP5 - after Z We use strnatcasecmp and strcasecmp for sorting mailboxes. Those have the advantage of correctly sorting mailboxes with numbers and being binary-safe. For better locale-specific sorting, we would have to use strcoll, but this function doesn't support either of the former features. so - resumé - it is not possible to get folders in correct alphabetical order rathen than for ASCII-named, isn't it? It looks like we might be able to use Collator::compare() (http://www.php.net/manual/en/collator.compare.php) to do the sorting. But this is only available in PHP 5.3.0+. We could use it conditionally, it's also available through intl from pecl for PHP 5.2. But I'm not sure if that would get us anything that strcoll doesn't. The documentation is sparse, but to me it sounds like it doesn't get us natural numeric sort or binary safety (probably not necessary for mailbox sorting anyway) either. Well... I'm not sure that our current sorting method is all that bad. It's at least consistent. And if PHP does not provide the proper tools to easily do this kind of sorting, I would rather not reinvent the wheel. Yes, I absolutely agree. I could only imagine that we make it configurable whether we use strnatcasecmp or strcoll for sorting. Implementing sorting ourselves is not an option. Jan. -- Do you need professional PHP or Horde consulting? http://horde.org/consulting/ -- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
Re: [imp] Spam Problem ... close to a solution ... may be you could help?
On Tue, 24 May 2011, Götz Reinicke - IT-Koordinator wrote: Hi, I did not find the compromised account yet, but I see a lot off messages like the following one in our logs: /var/log/httpd/ssl_request_log.1:[21/May/2011:01:10:54 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=721hskg326yc HTTP/1.1 92 /var/log/httpd/ssl_request_log.1:[21/May/2011:01:14:38 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=6khanz8ousab HTTP/1.1 92 /var/log/httpd/ssl_request_log.1:[21/May/2011:01:24:41 +0200] 74.82.171.30 TLSv1 RC4-MD5 POST /horde/imp/compose.php?uniq=2bcbqsb503hi HTTP/1.1 92 May be anyone has an idea how to protect against such direct postings... if it is possible anyway? I'm not sure what you mean by direct postings. There is nothing inherently evil about calling compose.php multiple times. One thing I forgot to mention about identifying compromised accounts - the spammers like to put the content of their message (the spam) into the user's signature block. That simplifies the creation and sending of the spam because IMP will automatically include the signature block in any message. You could search your preferences backend (MySQL or whatever) for the signature preference, possibly qualifying your search by looking for strings longer/larger than a certain amount. You'll also see the reply-to and identity preferences are frequently changed by spammers. Once you see the preferences of a compromised account, you'll know what to look for in the future. It's very obvious. Andy-- IMP mailing list Frequently Asked Questions: http://horde.org/faq/ To unsubscribe, mail: imp-unsubscr...@lists.horde.org
