Re: Vacation
This does bring up an important point... we had to disable duplicate suppression on our server because it was just causing too much contention. When every single mail message being delivered has to check against the database, it is a problem... especially in a cluster where multiple servers are running in ACTIVE mode (there *are* disadvantages to the whole cluster idea as well, even though I love running that way!). It most certainly broke vacation support, and I had to hack the code so that we could actually run with duplicate suppression disabled but still allow vacation messages to dip into the database normally... Why can't vacation messages, or better yet, sieve in general (since there might be a use for something mroe than vacation messages down the road?), get its own database and not tie it in so much to duplicate suppression? Scott --On Saturday, November 10, 2007 10:10 PM -0500 Tyler Spires [EMAIL PROTECTED] wrote: Vacation does work with duplicatesuppression disabled, I have duplicate suppression disabled on my server, and I recently (Friday) setup a vacation responder for one of my users. Hope this helps. Side note: I disabled duplicate suppression when I initially setup my cyrus mail server because I had to copy mail from the old server a couple of times, and cyrus would 'suppress' the second copies, I just haven't gotten around to testing and turning it back on. -Tyler Simon Matter wrote: I'm not sure vacation works with duplicatesuppression disabled because it want's to trck which person it already has sent a vacation message. And when you test make sure you understand that Cyrus will send a vacation message to a recipient only once in a period of time. That usually confuses peoples testing vacation. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkins Work (740)593-9478 Fax (740)593-1944 UNIX Systems Engineer mailto:[EMAIL PROTECTED] +---+ PGP Public Key http://edirectory.ohio.edu/?$search?uid=adkinss pgpw5MRUiXMZe.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: UC Davis Cyrus Incident September 2007
--On Thursday, October 18, 2007 9:58 AM +0200 Pascal Gienger [EMAIL PROTECTED] wrote: Scott Adkins [EMAIL PROTECTED] wrote: Meanwhile, we hacked around this in a very cool way. We copied the imapd process 60 times (assuming average of 12,000 processes, shooting for 200 processes per executable, that is 60 individual executables). These were named /usr/cyrus/bin/imapd_001 through /usr/cyrus/bin/imapd_060. We then symlinked the imapd binary to imapd_001. We then wrote a cron job that ran once a minute and relinked the imapd symlink to the next numbered executable, And how did you solve the problem of the deadlock resulting in deleteing the symbolic link, setting the new one? Between these events an exec of imapd would break resulting in an error to the customer. Funny hack though. Actually, I don't see a deadlock situation at all... I am guessing that theorettically, it is possible... but the ln -sf option makes the overwriting of the symlink an atomic action (as much as it can), which is why it seems to work. I stress tested this as much as I could prior to putting it into production, simulating tons of users logging in and out of the server while rotating the symlink as fast as I could. There was not one case of a connection failure at any point during the test. There are other factors that are involved too... Process re-use keeps a lot of problems to a minimum... We don't use preforking, but I imagine that preforking would further reduce any possibilities of deadlocking. Users coming into the system would be funneled to a socket associated with an IMAP process that has already been running for a bit, so it is clean from their perspective. The only issue is actually between master and imapd, and I don't know what master does if it tries to spawn an imapd and it fails... it may just try again... I dounbt it just out-right refused the incoming connection. I need to look at the source code to understand it better. Scott -- +---+ Scott W. Adkins Work (740)593-9478 Fax (740)593-1944 UNIX Systems Engineer mailto:[EMAIL PROTECTED] +---+ PGP Public Key http://edirectory.ohio.edu/?$search?uid=adkinss pgpPAbPHC07cU.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: UC Davis Cyrus Incident September 2007
--On Tuesday, October 16, 2007 3:39 PM -0700 Vincent Fox [EMAIL PROTECTED] wrote: Omen Wild (University of California Davis) The root problem seems to be an interaction between Solaris' concept of global memory consistency and the fact that Cyrus spawns many processes that all memory map (mmap) the same file. Whenever any process updates any part of a memory mapped file, Solaris freezes all of the processes that have that file mmaped, updates their memory tables, and then re-schedules the processes to run. When we have problems we see the load average go extremely high and no useful work gets done by Cyrus. Logins get processed by saslauthd, but listing an inbox either takes a long time or completely times out. Apparently AIX also runs into this issue. I talked to one email administrator that had this exact issue under AIX. That admin talked to the kernel engineers at IBM who explained that this is a feature, not a bug. They eventually switched to Linux which solved their issues, although they did move to more Linux boxes with fewer users per box. Oh man... Horrible memories just flood right back... Wow. I was reading your e-mail and thinking to myself that this sounded like the same problem we had. Then I got to the above section and *bam*, there it was... We had significant problems with our e-mail last year (this year was a perfect start!) a week before students came back. We didn't resolve the problems until the end of September and we were dismayed at our final solution. We run Tru64 5.1b on a 4 member cluster. Tru64's kernel suffers from the same exact issue as described above. We have regularly 12,000 cyrus procs running at any one time during the day, and that cluster also receives on average 300k-500k e-mails each day (that is after spam/virus work). What was finally identified was that the number of processes that were mapped to that single physical executable (/usr/cyrus/imapd) was causing a lot of lock contention in the kernel. The executable would have a link list of all the processes running off of it in kernel memory. When one of the processes would go away, the kernel would start at the beginning of the list and search for the process in order to clean up its resources. During that time, the kernel would lock everything and execution would essentially stop for everything (basically, the whole system appeared to simply freeze on us). The kernel would reach a time threshold and stop in order to let other things happen (unfreeze). This time was very short, but if we had a lot of processes going away in a very short period of time, we would noticeably see the freeze, since the kernel was going into this lock-down mode a lot in a very short period of time. That is a simplified view of what really happened. HP recommends that we keep the linked list down to only a few hundred processes at most. They were working on a kernel patch to make it a hash instead of a linked list in the kernel, but as they got deeper into the making of this patch, they found that it impacts a lot more than they initially realized. The last I heard, this might make it into the PK7 patch release, which is likely sometime next year. Meanwhile, we hacked around this in a very cool way. We copied the imapd process 60 times (assuming average of 12,000 processes, shooting for 200 processes per executable, that is 60 individual executables). These were named /usr/cyrus/bin/imapd_001 through /usr/cyrus/bin/imapd_060. We then symlinked the imapd binary to imapd_001. We then wrote a cron job that ran once a minute and relinked the imapd symlink to the next numbered executable, rotating around to imapd_001 when the end was reached. This worked like a charm and *all* of our problems went away... In fact, our system has continued to get busier and we are still running pretty good. I don't think the hack is ideal, but man, does it work! Scott -- +---+ Scott W. Adkins Work (740)593-9478 Fax (740)593-1944 UNIX Systems Engineer mailto:[EMAIL PROTECTED] +---+ PGP Public Key http://edirectory.ohio.edu/?$search?uid=adkinss pgpBBWFTSoKEK.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: see who is online
Have you verified that the proc files that are that old are really processes
that no longer exist? On our system, we do have lots of old proc files, but
the processes are still around as well (and connections are ESTABLISHED).
We are looking at turning on KEEPALIVE in the kernel, which likely will get
rid of a number of these if for some reason there is something wrong with
the connections, even if they do say they are ESTABLISHED. In any case, we
have seen processes stick around for months as a result.
On the other hand, if your server crashes or you have processes that go away
in a bad way (which isn't good for Cyrus in general), you will have proc files
that stick around. What we have done is have our Cyrus startup script wipe
the directory prior to startup. The easiest way to do this is something like
the following:
mv /var/imap/proc /var/imap/proc-delete
mkdir /var/imap/proc
chmod 750 /var/imap/proc may vary in your environment
chown cyrus /var/imap/procmay vary in your environment
startup cyrus
rm -rf /var/imap/proc-delete
The above procedure prevents the cleanup of the directory from interfering
with the speed of trying to get the server back up and running.
Scott
--On Tuesday, November 14, 2006 8:21 AM -0300 Marcelo Maraboli [EMAIL
PROTECTED] wrote:
I wouldn´t recommend that because I have lots of files
in that directory that are 3 months old...
doesn´t the master process clean this up on shutdown ??
regards,
Chaskiel M Grundman wrote:
--On Monday, November 13, 2006 08:39:39 PM +0100 Wolfgang Hennerbichler
[EMAIL PROTECTED] wrote:
I am trying to find a way to see which users are currently running an
IMAP session. Does anybody have a clue if (or how) that is possible? My
setup runs through saslauthd and ldap right now, all the processess are
owned by user 'cyrus'.
${configdirectory}/proc contains one file per active imapd, which lists
the hostname, ip address, username, and selected mailbox of that
process's connection
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
+---+
Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/
UNIX Systems Engineer mailto:[EMAIL PROTECTED]
ICQ 7626282 Work (740)593-9478 Fax (740)593-1944
+---+
PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/
pgpkw69Xx0zJD.pgp
Description: PGP signature
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, clusters, GFS - HA yet again
--On Monday, October 30, 2006 10:38 AM +0200 Janne Peltonen [EMAIL PROTECTED] wrote: IIRC there are people running Cyrus servers that way on other systems like Tru64 or Veritas cluster. And that's precisely what I'm trying to find out: since there are people that have succeeded in running Cyrus in an enviroment that's essentially the same as mine, I'd like to know how they have found their way around the DB error and other obstacles mentioned on this list in earlier posts. Since my name has already come up on this thread, and I have weight in a number of times on the clustering, I will throw my hat into the ring right now and hopefully get some more time tonight or tomorrow to look at all the messages and offer some opinions. I can say that I solved the above problem by dumping BerkeleyDB. :) Even though it is not recommended to use (at least the last time I remember the recommendation to be) Skiplist for mailboxes.db, that is exactly what I use. The performance hit for using Skiplist is negligible to the amount of problems I saw when using various versions of BerleleyDB on Tru64. I don't necessarily believe it is BerkeleyDB that is fully the problem, but maybe BerkeleyDB's interaction with Tru64 that is the problem. In any case, I have been MUCH happier dumping BerkeleyDB across the board. Here is my configuration in my /etc/imapd.conf file: # Duplicate Suppression duplicatesuppression: no # Database Formats annotation_db: skiplist duplicate_db: skiplist mboxlist_db: skiplist ptscache_db: skiplist quota_db: quotalegacy seenstate_db: skiplist subscriptions_db: flat tlscache_db: skiplist We receive a lot of e-mail in our environment, and taking the time to suppress duplicate messages wasn't worth it considering the problem it invited, and particularly the time it took to manage the write locks on that database given the high rate or incoming e-mail being delivered. However, turning off duplicate suppression breaks sieve, so I had to modify the code to allow sieve to continue using the duplicate_db, even when duplicate suppression has been turned off. So, we still use the database a little bit, but that is it... Which, makes skiplist just fine for that. Anyways, if there are any questions you have about our environment, feel free to e-mail off-list and ask me. I will do my best to help. As I eluded to above, we are a Tru64 show and their CFS has worked wonders for us. We have an e-mail project under way that will migrate Cyrus off Tru64 into Linux (RedHat) with Polyserve as the cluster filesystem. We have Polyserve purchased but not deployed. I will be happy to share our deployment document once we have completed the project (which is likely at least in the next summer timeframe). Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpPwqiy9h7bR.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus, clusters, GFS - HA yet again
I have to agree here, the reason we like the cluster filesystem method is its shear simplicity. It is nothing to add extra nodes to split the load across more servers. We don't have to partition users, we don't have to worry about losing part of the user community when a server goes down, only one place to hit for backups (any one server will do), one address for the users to hit (without complicated means of achieving it), etc etc. In essence, if you have a good CFS system running, then it is as simple as if you can run Cyrus on one server, you can run it on all of them. As somebody (Sebastion maybe?) pointed out, you do have to take care that certain directories are *NOT* shared between the cluster nodes... the /var/imap/socket directory is one, and I would highly recommend not sharing the /var/imap/proc directory. On our system, we also do not share the /var/imap/log directory, though I do not recall why... it is empty, and probably is empty because we abandoned BerkeleyDB... For SASL, we run SASL on all the nodes using UNIX domain sockets, and thus the socket directory SASL uses also needs not be shared... After that, it all just works... Scott --On Monday, October 30, 2006 11:09 AM +0200 Janne Peltonen [EMAIL PROTECTED] wrote: Hi, and thanks for the answer. On Sun, Oct 29, 2006 at 09:11:53PM -0500, Wesley Craig wrote: On 27 Oct 2006, at 03:35, Janne Peltonen wrote: Or should I just give up and start considering Murder? Before you decide on whether to give up on clustering, you should thoroughly consider Murder. At a minimum, it provides the location independence that you'd like. With replication, you have a realistic, immediate recovery path for machine failure. Both of those are much better than what you have, while not as good as a real HA system. But murder + replication will be a lot less work than developing an HA architecture from scratch. It is very true that building an HA architecture from scratch is a lot of work. But I'm not really trying to build it from scratch: I'm trying to recreate with Linux and RedHat GFS an architecture that's been in production use with Tru64 and its cluster FS (in Ohio), or with some other *nix and Veritas (in Pittsburgh). There were a couple of reasons I wanted to try this solution before Murder, replication and friends. The main reason was, if this could be get to work, it would have a lot less moving parts than Murder: no need for differently configured front and back end cyrus servers; no need for a MUPDATE master server. Each server could be more or less identical, and if one of them went down, there would be no loss of service (ok, the load-balancing box would have to notice that one of the ip addresses it shares went away, and the active imap connections to the dead server would die, but that's it, really). And adding a new server to the pool could be almost as easy as putting a new HP Blade on the rack, configuring its MAC in our DHCP server, installing an HD image on its local HD, booting it, and configuring its IP on the IP-sharing box. Of course, if it won't work, it won't work, and Murder is the supported solution. When my predecessor made his decision on what solution to pursue, he considered Murder much too complicated, and wasn't sure whether 2.3 was mature enough (that is, replication and friends), but this all might have changed - and he might have been wrong even then. ;) But then, other people seem to have got this to work - or something very similar. The simplicity of the setup is so appealing that I'd really like to give it a try. Regards, --Janne Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpjkr6zzBp0X.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Probe needed for Cyrus IMAP
Of course, contacting SSL ports are a bit more difficult. If you need to
probe an IMAPS or POP3S port, telnet just doesn't work. In that case, you
can use openssl itself to make the connection and still feed it into an
expect script:
LD_LIBRARY_PATH /usr/local/ssl/lib
export LD_LIBRARY_PATH
/usr/local/ssl/bin/openssl s_client -connect localhost:imaps
If you know Perl, then maybe you can take it to another level with something
like the following:
#!/usr/bin/perl -w
use strict;
use Net::SSLeay;
$Net::SSLeay::slowly = 1;
($p) = Net::SSLeay::sslcat(localhost, 993, .);
print $p;
The banner should be printed as a result of the above, but you can make the
script do anything you want.
For non-SSL services, maybe something like the following would work:
#!/usr/bin/perl -w
use strict;
use IO::Socket::INET;
my $socket = new IO::Socket::INET(localhost:143);
if (!defined $socket) {
print Unable to connect: [EMAIL PROTECTED];
exit(1);
}
print $socket-getline();
print $socket . logout\n;
$socket-close();
The above also prints the banner. Keep in mind that I spit these out
pretty quickly, but I did test them and the scripts work.
Hope that helps.
Scott
--On Monday, October 30, 2006 6:10 PM +0100 Phil Pennock [EMAIL PROTECTED]
wrote:
On 2006-10-30 at 08:57 -0600, Gary Mills wrote:
`telnet' might work, but the script would need to analyze the output
and terminate the connection. A timeout would also be helpful. Is
`imtest' better for this?
Do you have expect installed? Something like the below might help.
8 cut here 8--
# !/usr/local/bin/expect -f
spawn -noecho telnet localhost imap
expect {
timeout {exit 1}
-re Escape character is '(.*)'.\r*\n {}
}
expect {
timeout {exit 1}
eof {exit 2}
-re ^. OK .* server ready\r*\n {}
-re . {exit 2}
}
send . LOGOUT\r\n
exit 0
8 cut here 8--
Regards,
-Phil
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
--
+---+
Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/
UNIX Systems Engineer mailto:[EMAIL PROTECTED]
ICQ 7626282 Work (740)593-9478 Fax (740)593-1944
+---+
PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/
pgptdUtHzrAu2.pgp
Description: PGP signature
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Moving the Cyrus Mailstore
Another option is to get the physical partition created and setup and then create a cyrus partition on that and migrate the users to it via the server itself, as opposed to doing it physically. The moves are transparent and does not require you to down the server. Here are a couple of suggestions: 1) Let's say you create a /home/cyrus physical partition (filesystem). I suggest that you then create additional cyrus partitions within that, since dealing with a smaller set of users is better, and it allows you to possibly moves some of these cyrus partitions to a physical filesystem or disk at some point down the road if you find that is necessary. It may be better for backups too, since you can seperate the backups into smaller jobs (one failure doesn't mean a failure for all of cyrus, but just a smaller subset of cyrus). 2) So, let's say you choose 26 cyrus partitions. They could be located as /home/cyrus/a through /home/cyrus/b. You need to reflect these partitions in the /etc/imapd.conf file: # default directory defaultpartition: default partition-default: /var/spool/imap # the new partitions partition-a: /home/cyrus/a partition-b: /home/cyrus/b partition-c: /home/cyrus/c partition-z: /home/cyrus/z Note: The partition-a through partition-z is what actually gets stored in your mailboxes.db file. You can change the physical location of the partition at any point by modifying your imapd.conf file. As long as you don't change the name of the partition itself, you don't need to worry about the user account in mailboxes.db 3) You need to make sure your imapd.conf file also allows you to rename user accounts... This is done with the following: allowusermoves: yes Now, the nice thing is that you can essentially rename a user from one partition (defaultpartition) to the new partition (partition-X). The Cyrus server will take care of the rest automatically. 4) You can use the cyradm command to rename the mailbox, as follows: renamemailbox user.adkinss user.adkinss partition-a It is recommended that the users you are moving are not logged in at the time their accounts are being moved. The easiest way to determine that is to look at the /var/imap/proc directory to see if their userid appears anywhere in the list. This would catch shared folders as well. Doing the moves in small subsets, such as by letters of the alphabet, allows you to move things only in the middle of the night (when there is less usage), advertise to the community that maintenance for that subet of users will occur, etc etc. This is the method we used when we migrated from 4 partitions (default, staff, faculty and students) to the 27 we have now (default, a-z). Anyways, just throwing out another suggestion for you. Oh, by the way, I believe you can add or make certain kinds of changes to the imapd.conf file and then send a signal to the master server to reread the config file. This might be an option for getting the new partitions into the configuration without having to down your IMAP server. The UNIX side of the work (creating a new filesystem) shoulnd't require any outages either, unless you have to down the server to add a new physical disk or something. Good luck! Scott --On Friday, October 20, 2006 11:30 AM -0600 Wickham, Larry [EMAIL PROTECTED] wrote: Hi, Our server is suffering from impending doom, while servicing 11,500 users. Our mail store is in the default location under /var/spool/imap. The partition that / (/var is under here in our setup) is mounted on is 25 GB and I would like to move it to the partition where /home is mounted which has 1+ TB available. We have considered several solutions. The chosen solution would have minimal downtime and minimal chance of failure. 1) Repartitioning: Down the server and boot from a rescue platform. Backup and Resize /home. Create a new partition in the free space formerly occupied by /home. Copy /var to the new partition. Rename the existing /var. Edit fstab and boot. 2) Swaping mount points: Down the server and boot from a rescue platform. Move the contents of /home to a folder on the smaller drive. Copy /var to the larger drive. Modify fstab. 3) Migrating between mail stores Create a new mail store under /home and migrate the mail boxes? Not sure how or if we can do this. Please let me know your opinion on these solutions and any new suggestions are welcome and appreciated. Larry Wickham [EMAIL PROTECTED] Systems Operations Specialist Eastern New Mexico University 1500 S. Ave K Portales, NM 88130 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkins
Re: idled vs poll
--On Wednesday, October 18, 2006 2:00 PM +0200 Sebastian Hagedorn [EMAIL PROTECTED] wrote: --On 18. Oktober 2006 11:25:09 +0100 Adam Stephens [EMAIL PROTECTED] wrote: We deployed idled here over the summer on our main staff IMAP server, a Sunfire v480 running Solaris 8 and cyrus-imap 2.2.12. It typically has about 13,000 concurrent IMAP processes Wow, that's a lot of sessions! I'm surprised about that. We have about 35,000 users, yet we rarely have more than 700 concurrent IMAP processes. We regularly have between 10,000 and 13,000 concurrent processes each day, depending on the day of the week (heavier on Mon-Wed). cannot connect to saslauthd server: Connection refused Failed to connect to socket /var/cyrus/imap/socket/lmtp for local_cyrus_deliver transport: Connection refused error sending to idled: 0 The first thing I would do here is to use LMTP as a TCP socket instead of a UNIX domain socket. We use Tru64 in our environment, and when I first deployed Cyrus under that architecture, the first thing I noticed what that as load increased, the more connection refused messages I saw with LMTP. Pointing my MTA to a TCP LMTP socket completely eliminated that problem for us. However, you should still keep the UNIX socket, since the deliver program still uses that instead of the TCP socket (the last time I checked). Additionally, when setting up the TCP socket, it would be good to set it to listen only on localhost or a private address (to prevent Internet users from connecting to your LMTP server and bypassing your MTA and spam/virus filtering controls). It is either that, or you configure authentication for the LMTP server (which, admittedly, I have never done). For SASL, I don't know if there can be any changes there. We use UNIX sockets for it as well, and I haven't investigated to see if there is a TCP socket option. That might help if there is one. On our system, when I see load increase, I definitely see SASL authentication take longer as well. We have worked most of our load problems out (Tru64 related), so that has improved considerably. We use the poll method, not idled. Good luck. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpTpqBffmAzI.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance issue (imap spool on san)
--On Monday, September 11, 2006 2:14 PM +0200 Daniel Eckl [EMAIL PROTECTED] wrote: Everyone is entitled to their opinion and yours is certainly welcome. However, beauty is in the eye of the beholder... I have heard comments the cover pretty much the whole spectrum with regards to its usability and looks, and really, I don't see any particular opinion winning over the other. For the most part, opinions are based on how well that particular user loved a favorite e-mail client they previously used. And it misses a lot of features I use every day. Virtual folders, inline attachments (jpegs for example), forwarding emails attached, view attached emails, Drag and Drop support and so on and so forth. I must admit, I would love to see Virtual Folder support... however, is this implemented in the majority of IMAP clients out there? I am not sure that it is... so, to single out Mulberry for that is unreasonable. Is it a wish-list item? Certainly. Inline attachments are a long heated topic of debate here... of course there are features left to implement... however, Cyrus is one man and he has to prioritize what order of features should get implemented in. When Mulberry was not a free product, the features that were requested most and/or payed for were the features to get implemented first. Now that Cyrus is a one-man show (currently), it may be awhile before we see any new features get added... Forwarding as attachments is a function that already exists. Viewing attached e-mails exists, but maybe not the way you would like... If it can be viewed inline, Mulberry allows it... otherwise, you right-click and click view and it opens an external viewer... Drag and drop support is also implemented... though, without context in your complaint, I am not sure what you are expecting... I certainly can drag and drop an attachment from my desktop to the attachment section of my draft and it works as expected. Admittedly, I haven't dragged and dropped things anywhere else... Your arguments that things aren't as intuitive and easy to find is a good argument. Some people don't have problems (like me), but other do. This may be why some of the features you mention above aren't known to you. There was a project from CMU (by students?) that had taken on the job of analyzing the Mulberry interface to make recommendations on how to improve it. They did surveys and usability testing, and had entered an agreement (contract?) with Cyrusosft (Isamet) to implement most if not all of the recommended changes. This was a great step forward and a great idea. However, the project seem to go slowly and then disbanded when Isamet fell apart. And a thunderbird with cached headers is multiple times faster in resorting and scrolling, not only over 3 MBit DSL Line, but even over LAN. It's fine that mulberry doesn't need to cache headers, but why isn't it able to do so? Loading on demand and then caching it would be the best of both worlds. The same could be said for IMSP preferences. It isn't as noticeable over high speed connections, but over dialup... *whoosh*. The first thing I see is Mulberry download all of my preferences (and I have a lot). Then I see it turn around and write all the preferences back (at least from what I could see). Only then can I start to use Mulberry. Shutting down is also just as hard... all of my preferences get written back. I am not sure if there is a good answer... especially since I don't REALLY know what is going on... Another popular problem people complained about is the multi-threaded nature of Mulberry. Some Mulberry actions freezes the whole application and prevents you from doing anything until those actions are done. The point it, there is lots of improvements that could still be made and were in the works. Cyrus acknowledged the problems it had and was very clear where he stood and where he was going. It is a tough job being the author of one of the most popular IMAP clients on the marget, as a lot more demand and expectation gets added to that... So just implementing every IMAP feature available might be the best thing for the server and the protocol, but not for the user. You need a intuitive interface and nowadays it really has to look nice, too if a non-geek should use it. And to be honest, mulberry simply looks horrible... I would not agree with the horrible philosophy. I personally like the look and feel of Mulberry. I think the CMU approach was a good one and it would be nice for something like that to be resurrected. There is something to be said for surveys and usability testing... it answers some of the issues you have brought up. But it's nice, that everyone who doesn't care about looking and usability now has a suitable free IMAP client availiable. Uh... okay :) Of course, we really don't know how big the everyone who doesn't care about looking at usability really is now, do we? As I said before, there are a lot of people on BOTH sides of the
Re: LMTPD sockets
In a cluster (multi-machine) environment, such as Tru64 or Polyserve (on Linux), that problem is solved by the use of CDSL (a context-dependent symbolic link). You basically turn the file or directory into a CDSL and suddenly each machine in the cluster gets its own personal copy of it that is not shared. We use TCP for LMTP, which I personally feel is a better way of delivering messages to LMTP. If you have to use the deliver program though, then yeah, you still need the socket file. That file needs to be a CDSL. The same goes with SASL and its socket file. Scott --On Thursday, August 17, 2006 4:50 PM -0400 David Korpiewski [EMAIL PROTECTED] wrote: This is building on my original discussion about simultaneous access: We are having a problem where two servers accessing the same SAN now give us a LMTPD error because they are stomping on each other's sockets (they share the same database on the SAN with the same socket directory). Is there any way to prevent this from happening or easily configuring them not to not to use the same sockets? I've gone into cyrus.conf and changed the socket directory listed for lmtpd but for some reason the deliver still fails on accessing the original socket directory. So somewhere else in the system it is trying to access configdirectory/socket/lmtpd. Thanks for any help David Michael Menge wrote: This subject has also been on this list a few weeks ago. Look for the subject High availability email server Quoting Wesley Craig [EMAIL PROTECTED]: On 16 Aug 2006, at 14:52, David Korpiewski wrote: My director had us go back to using our Macs to try to build a failover cluster. We were able to set up two mail servers sharing the same backend XSAN. We put our database for cyrus on the XSAN and had both mail servers use _the same_ database and the same mail store locations. So far in limited testing it works flawlessly. This is very strange to us. Is cyrus set up to allow multiple simultaneous accesses to its database without corruption or are we just lucky to not have corrupted the database yet? This is cyrus 2.2.12 running on two OSX 10.4.7 servers. The main process which updates mailboxes.db etc on a backend server is imapd. Since multiple imapd's are allowed, as long as your XSAN provides appropriate locking, you are not just lucky. See the compile documentation for a discussion of how to select the locking method: --with-lock=METHOD Specifies the locking method to use. Currently implemented locking methods are: flock flock() locking fcntl fcntl() locking By default, configure uses fcntl if the fcntl() function exists, flock otherwise. AFAIK, fcntl( ..., F_SETLK*, ...) always locks the whole file, a la flock(). Also, I think I see a bug in master/service.c, where instead of using the locking method specified during compile-time, fcntl() is always used. :wes Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html M.Menge Tel.: (49) 7071/29-70316 Universitaet Tuebingen Fax.: (49) 7071/29-5912 Zentrum fuer Datenverarbeitung mail: [EMAIL PROTECTED] Waechterstrasse 76 72074 Tuebingen Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpjv9w9BInBX.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: What exactly *IS* the status of 2GB quotas? [WAS Re: ****Re: Odd quota problem]
--On Wednesday, August 09, 2006 7:10 PM +0200 Daniel Eckl [EMAIL PROTECTED] wrote: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=1212 https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2690 Thanks! --On Wednesday, August 09, 2006 8:36 PM +0200 Simon Matter [EMAIL PROTECTED] wrote: With 2.2 it was 2GB, with 2.3 it may be 2TB or something, just usually large enough. Great! 64-bit quota has been made in 2.3 and backported to 2.2. They work the same way. I don't know what your problem is but maybe you have to update quota with quota -f after switching your mailspool to the 64bit quota thing. Heh... Did I say what my problem is? :) No problem... no problem... I just wanted to know the definitive answer, which you guys gave me! Does the large quota patch require a 64-bit machine/os in order to permit quotas larger than 2GB? No, it works fine on 32bit cpu/os. Thank you both for your quick answers. FYI, we are running on a version older than 2.2.12. However, we will be likely upgrading to Linux and the latest version of Cyrus maybe in December. It is unlikely we will patch our software right now to accomodate quotas larger than 2GB, but maybe it won't be up to me on that... Only 1 user is requesting a larger quota than that right now. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpnKiI8k9BKe.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus and Polyserve
We are looking at possibly running our Cyrus environment behind a pair of F5 load balancers on a bunch of RedHat AS3 IBM BladeCenter blades. We are migrating from a Tru64 TruCluster Alpha environment which provides us with a very nice Cluster File System. We have not needed the use of the Murder setup as a result... we would like to avoid it when migrating to Linux if at all possible. We have looked at a number of Cluster File Systems and keep coming back to Polyserve and the one product that could solve a lot of our needs, even the problems not related to e-mail. So, my questions are these: 1) Is anyone out there running Polyserve in a Cyrus environment? 2) If so, can you describe the environment (hardware, software and usage)? We are very interested in knowing if anyone else is doing this... We are aware of the Casper people and their report, but have been unable to reach anyone over their at the moment. Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0WYshlFucZ.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: from 2.2.12 to 2.3.1, mailbox has invalid format
--On Monday, February 13, 2006 2:19 PM -0500 Ken Murchison [EMAIL PROTECTED] wrote: Nicolas KOWALSKI wrote: Hello, I am testing the migration from a 2.2.12 installation, compiled with the default options (./configure without any option), to a 2.3.1 installation, also compiled with the default options. The databases used are the default for both. The system is Debian 3.1, using db3 for the berkeley databases (but this should not matter because we do not use tls caching or duplicate suppression). When I start the server using the new binaries, accessing some mailboxes is not possible, the server sending a mailbox has invalid format error. If I run reconstruct on these mailboxes, everything runs fine afterwards. Did I forget something ? Should I run reconstruct unconditionnaly on all mailboxes to prevent any inconsistencies during upgrades ? You shouldn't have to do a reconstruct unless there truely is something wrong with the mailbox. The various services are supposed to do an on-the-fly upgrade of the mailboxes. What system are you running this on? Last year, we upgraded from 2.2.1 to 2.2.10. Our system is a Tru64 Alpha Cluster and we ran into exactly the same problem with the same response... Cyrus should automatically upgrade the mailboxes when the user logs in or LMTP does a delivery of a message. What I found was that in all my testing, this was exactly the case. I even did stress testing and could not get Cyrus to *not* upgrade the mailboxes. However, when we moved to production, our production system seemed to upgraded some user's mailboxes, but produced the above error other user's mailboxes. As more and more users logged on, the errors increased in frequency and the problem got worse. The users that got the errors had to have their mailboxes reconstructed before they could log in and check their mail. I have no idea why this happened, but it caused a lot of pain for us. We backed out of the upgrade the first time and I spent quite a bit of time trying to reproduce the problem without success. We tried the upgrade again and the problem came back. This time, we decided to plow on. I wrote a script that watched the syslogs for any user getting the above error and immediately launched a reconstruct of their account. We ran another parallel script that worked its way through all the accounts, reconstructing everything (which took a long time to complete). Once all the accounts were reconstructed, we have run fine ever since. Anyways, I don't think that helps you much, but I thought I would share my experience. I don't relish having to do this again on the next upgrade. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpHLxSgvCuSN.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Max Quota Size?
I already have a hunch, and my research on the archived mailing list kinda suggests the same thing as well, but I would feel better if I had the answer clearly stated by a more official power :) Anyways, I am currently running Cyrus IMAP 2.2.10. What is the maximum size of a quota I can set for a user? I suspect it is 2GB. Is this true? Is this size increased in a later version of Cyrus (like in the 2.3 stream)? Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpjYSgR3VF5t.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: High-Availability IMAP server
--On Monday, September 26, 2005 6:45 PM +0200 David [EMAIL PROTECTED] wrote: Hello, I have a 'pseudo' High Availability SMTP system consisting in two servers running cyrus 2.2.5. The main problem I have is that only one of the two nodes can access to the mailboxes in order to keep the integrity of the cyrus databases despite the filesystem (GFS) has support to allow to two different servers access in R/W mode. I am curious about this statement... What kind of locking is being used on GFS that prevents two nodes from accessing mailboxes without destroying the integrity of the cyrus database? In our environment, we have a cluster of 4 Alpha machines, two of them are ES40's and two of them are ES80's. They run Tru64 5.1 (TruCluster) and are attached to an HA San using AdvFS. All the members of the cluster can see all the filesystems and can access all the files and directories. We are currently only running Cyrus on the two ES80 machines, but we could easily run them on all four cluster members if we wanted too... we don't because we do run other things (i.e. Sendmail) and it is better not to mix Cyrus and Sendmail on the same machines in our environment. That being said... the mailboxes are all available from the Cyrus servers running on any cluster member. We don't see any integrity issues and it seems to run pretty good. Since Tru64 and Alpha's are on their way out the door, we are looking for a future solution that would give us the as much of the same capabilities our current environment has. This is most likely going to include Linux, but that then means we need to find a suitable cluster-like file system to replace AdvFS, which could be GFS. Anyways, I am interested in the shortcomings that you guys have encountered with reliability and integrity when trying to run an HA Cyrus server... Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpktJjzewxXD.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: imsp 1.8 virtdomain patch
--On Wednesday, January 26, 2005 12:37 PM +0100 Martin Ziegler [EMAIL PROTECTED] wrote: Hi again, i compiled imsp 1.8 and applied the patch from William K. Hardeman which enables support for virtdomain logins. Everything works fine with the exception of PLAIN logins. Is there any patch ot there which also works with PLAIN authentication ? Thanks, Martin IMSP 1.8??? I didn't see an announcement on that... what changes are in this version? Is there integrated SSL/STARTTLS support? I see that the ftp site shows it was put there on Jan 10th... Just surprised I didn't see anything about it... Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpEmcnBDma7W.pgp Description: PGP signature
Re: Malbox has an invalid format after 2.1.14 to 2.2.9 upgrade
We just recently upgraded our system from 2.2.1b to 2.2.10 and had very
similar problems. In fact, I spent quite a bit of time debugging the code
and testing it on the system we were upgrading on (we did the upgrade once
before and had to back out because of these errors) and the testing showed
everything was going to work just fine. However, when the upgrade occurred,
the errors once again came back...
The problem occurs with the cyrus.* files in the user's mailbox directory.
The new version of Cyrus upgrades those files to the current version of that
file format. In our case, we were two versions out of date, but that should
not matter with how the upgrade process works.
The disturbing thing was that it was unpredictable as to what mailboxes it
was going to hit. About 2/3rds of the mailboxes were upgraded to the new
version, but the other third failed. There were no error messages relating
to the failure, only errors relating to the access of these mailboxes that
had failed the upgrade... For instance, we saw the errors when LMTP tried
to deliver messages to the mailboxes. Sometimes we would see errors when
the user connected via IMAP, but most of the time, we didn't.
At this point in time, we are leaning to something funny in the OS, maybe
with regards to the lock files, but that is inconclusive, and may not be
the case at all. If you are experiencing the problem, it may very well be
a Cyrus issue yet. What OS are you using?
We fixed the issue by reconstructing every mailbox on the system. This took
a couple days to complete, since we have nearly 100,000 accounts, and many
of those accounts have a lot of mailboxes on them. Reconstructing does not
lose flag information, so it is safe to do.
Scott
--On Thursday, December 23, 2004 9:17 AM -0800 Curses!
[EMAIL PROTECTED] wrote:
Greetings:
I am trying to upgrade our Cyrus IMAP server from verison 2.1.14 to 2.2.9
to address the security issues announced early this month.
After upgrading, the IMAP server issues the error NO Mailbox has an
invalid format when selecting any mailbox. Here's a snippet from the
beginning of the server log for the session showing the error:
-- michael_moore Mon Dec 6 23:23:44 2004
1102404224A2 SELECT INBOX
1102404224A2 NO Mailbox has an invalid format
The entire log output for the invocation of the server can be seen at
http://followingedge.com/images/log.html where I have highlighted the
boring information relative to the session above (process 18883).
I have not been able to spot the source of the issue in examining the
included documentation and mailing list archive. Thanks to anyone who can
point me in the correct direction. Here's some additional information that
may be relevant to my issue:
Since I saw there were changes in the default database formats and the
addition of run-time database format directives, I added those directives
to indicate the same skiplist/flat configurations I had used previously.
I also added the allowusermoves and lmtp_downcase_rcpt directives
since I wanted to try them in the new version. Here's the complete
configuration:
seenstate_db: flat
duplicate_db: skiplist
quota_db: quotalegacy
tlscache_db: skiplist
mboxlist_db: skiplist
subscription_db: flat
annotation_db: skiplist
ptscache_db: skiplist
allowusermoves: 1
lmtp_downcase_rcpt: 1
configdirectory: /var/imap
partition-default: /var/imap/mail
admins: admin
sendmail: /usr/sbin/sendmail
postuser: shared
popminpoll: 2
defaultacl: anyone lrs
loginuseacl: yes
allowanonymouslogin: no
allowplaintext: no
tls_cert_file: /var/imap/cert/server.pem
tls_key_file: /var/imap/cert/server.pem
tls_ca_file: /var/imap/cert/server.pem
tls_ca_path: /var/imap/cert
I had configured the source with the following command (although I guess
the database directives are no longer useful):
./configure --with-cyrus-group=mail --with-cyrus-user=mail \
--with-dbdir=/usr/local/BerkeleyDB.3.3 --with-duplicate-db=skiplist
\ --with-tls-db=skiplist --disable-sieve --without-snmp
Based on the notes in the changelog and other documentation, I expected
that the server would upgrade the mailbox format on the fly. I am now
wondering if I need to reconstruct all the mailboxes using the new
utility prior to running the new server. If so, will I lose message flags?
In a possibly related issue, after briefly invoking the new version of the
server, I also observed that the quotes for several mailboxes were
mis-calculated (as seen here from quota output):
251675 4188018 shared.csnw.slm.corp_accts
I found that quota -f {mailbox} did not fix the problem and the
mailboxes had to be reconstructed as here:
$ /usr/cyrus/bin/reconstruct -r shared.csnw.slm.corp_accts
shared.csnw.slm.corp_accts
$ /usr/cyrus/bin/quota -f shared.csnw.slm.corp_accts
shared.csnw.slm.corp_accts:
Re: allow duplicate message
I don't believe this is the case... We set it to no on our site because we see a lot of lock contention on the duplicate delivery database. The problem is that setting it to no doesn't help if you have SIEVE enabled in your server. From what I could tell in the source code, SIEVE makes very active use of the duplicate delivery database, for vacationing, forwarding, redirections, etc. The end result is that we patched our server (both in 2.2.1 and in 2.2.8) to honor the no for even Sieve operations. This works, but the downside is that this will break vacation stuff, as it is needed to prevent the auto-response message going back to the same user every single time a message is received from that user. Fortunately, nobody is using vacation via SIEVE in our environment yet, so it works for now... The long term solution is to determine what in SIEVE actually truly needs to deal with the duplicate delivery suppression and code based on that. From what I can tell, the forwarding, rediction, etc deals with the DB only for the purposes of determine duplicate messages. So, if it is set to no, then it shouldn't check. With regard to vacation messages, SIEVE is doing something different here and just uses the DB as a convenient way of storing the last time it sent a particular user an auto-response message. I think that can be ignored and still use the DB, even when the setting is no. Is there anything else in SIEVE that should ignore the option? Anyways, that is my 2 cents. Incidentally, if you disable sieve when compiling the server, the option works exactly as expected. Sieve is enabled by default. Scott --On Friday, November 19, 2004 11:14 AM +0100 Fredrik Jönsson [EMAIL PROTECTED] wrote: Hi, From the man-page imapd.conf(5) duplicatesuppression: yes If enabled, lmtpd will suppress delivery of a message to a mailbox if a message with the same message-id (or resent-message-id) is recorded as having already been delivered to the mailbox. Records the mailbox and message-id/resent-message-idofallsuccessful deliveries. Setting it to no (and restarting the server I think) will allow duplicate delivery. It has worked for us previously, though we quickly turned it back on again. Regards, /Fredrik Jönsson On Fri, 2004-11-19 at 08:08, Antoine Jacoutot wrote: On Thursday 18 November 2004 22:52, torgan wrote: cyrus.conf # this is only necessary if using duplicate delivery suppression, # Sieve or NNTP # delprune cmd=cyr_expire -E 3 at=0400 hope that would help Well, thanks but not really. I already saw that but I don't see how it would allow me to receive 2 instances of the same message. It is strange, I never had this problem with any other imap server. Antoine --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpdIqAzYVhb7.pgp Description: PGP signature
Re: allow duplicate message
--On Friday, November 19, 2004 10:17 AM -0800 Wil Cooley [EMAIL PROTECTED] wrote: IOn Fri, 2004-11-19 at 10:58 -0500, Scott Adkins wrote: I don't believe this is the case... We set it to no on our site because we see a lot of lock contention on the duplicate delivery database. The problem is that setting it to no doesn't help if you have SIEVE enabled in your server. From what I could tell in the source code, SIEVE makes very active use of the duplicate delivery database, for vacationing, forwarding, redirections, etc. Are you saying that using Sieve makes it impossible to disable duplicate suppression or that Sieve just uses the duplicate delivery database regardless of the 'duplicatesuppression' setting? The former should be reported as a bug (http://bugzilla.andrew.cmu.edu/); the latter a point for discussion. I think it kinda works both ways... the way the code was written (I am working from memory now) was that if the option was set to no, then it simply did not open the delivery database. The code would still call the duplicate suppression routines, but those routines would bail out first thing if it found the database not to be open. With SIEVE, the code more reads like this: if SIEVE not defined and the setting is 'no' then do not open the database. Consequently, the database *is* opened (because of the presense of SIEVE), and all of the duplicate suppression routines then work as normal. So, it turns into an all or nothing event. The right solution is to not base the decision on whether or not the DupDB is open, but on whether or not a portion of code should check the DB based on the current setting... I don't think there were *that* many places in the code that checked for duplicate messages... Scott Wil -- Wil Cooley [EMAIL PROTECTED] Naked Ape Consulting, Ltd. http://naked-ape.com --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpcutZnBU5MD.pgp Description: PGP signature
Quota inconsistency...
We have a user that was reporting that she was way *way* over her quota. She was alotted only 100MB, and for some reason, she was showing that she was using over 400MB of space on her IMAP account! Furthermore, looking at actual disk space consumption, she waas actually only using 261KB, which is barely anything at all. I have yet to figure out a good way to fix this discrepancy. I thought that maybe doing a recursive reconstruct on her account would maybe sync up her quota entry to what was actually in use. Nope. I thought that the quota command would be able to do something with it, again nope, unless I am just missing an option somwehere. I even used cyradm to do a setquota on the user to see if it would fix it, and again it did not. I editted the file (we are using legacy file format, which is plain text) manually and removed a couple digits. This brought her down to about 41MB of disk space used, and thus 41% of her quota. This gets her working, but I still can't find a way to have the server re-figure out how much she is actually using and write that to the file. I finally just deleted the quota file altogether and used cyradm to create it again using setquota. Well, it created a new quota file for her, but now it shows her sitting at 0KB used out of 100MB, so it is still not right. Other than doing a du on her disk space and manually typing in the value in the quota file, what should I be doing? Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpwVjbOXHGXy.pgp Description: PGP signature
Upgrade failure from 2.2.1 to 2.2.8
Okay, so I have just attempted for a second time to upgrade our Cyrus server on the production box from version 2.2.1 to version 2.2.8. The upgrade was a failure and I had to back out and go through a not-quite-painless recovery process on the user accounts affected by the upgrade. So, the first time I did this we ran into two obvious bugs on the Tru64 5.x platform. The first bug was that after a bit, the processes leaked all their file descriptors and eventually starting crapping out. In a previous e-mail, I described this as a problem with getnameinfo() on Tru64 and we will likely have to work this problem through HP's tech support. The second bug was tied to some user's sieve scripts failing. In another previous e-mail, I described that bug and supplied a fix. What was most problematic about that upgrade was that quite a number of users ended up with corrupted cyrus.header and cyrus.index files and had to be reconstructed. The users reported various problems, such as opening their mailboxes and finding no messages in them, even though they knew there were... or finding messages, but when opening a message, it would be blank when they knew it wasn't. Reconstructing the accounts fixed the problems. So, after fixing the above bugs, we attempted another upgrade tonight. I started seeing a lot of log messages from LMTP to the following effect: verify_user(user.cyrus) failed: Mailbox has an invalid format The server ran for nearly 20 minutes. in that time, there were 1717 of these errors affecting 136 different users. I believe that this is related in some way to the mailbox corruption during the previous upgrade. The archives suggest reconstructing accounts that show this, and maybe we would be okay by continuing to run on the new version and reconstructing all existing accounts, but we have over 93,000 accounts and over 376,000 mailboxes... so, reconstructing everything is not the way to go. I read the upgrade instructions and the only things mentioned when going from 2.2.1 to 2.2.8 is the bytecode format change (i.e. recompiling all the user sieve scripts) and the backend database stuff is now specified in the imapd.conf file (which I did and double-checked to make sure that what is supplied matches what I had compiled in the 2.2.1 version of the server). However, the cyrus.header and cyrus.index files are not related to any of the backend database stuff, so I don't understand what is going on here. Does anyone have any ideas on how to troubleshoot this? I guess the first thing I need to do is to duplicate it in our test environment... All the testing I have done never once triggered this problem. Anyways, I hope somebody can help me on this. Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpxXxWWNPgwW.pgp Description: PGP signature
sievec core dumps when hitting 'address' or 'envelope'
We ran into a problem on Tru64 where the sieve compiler was core dumping
if it tried to compile a script that used 'address' or 'envelope' in it.
For example, the following script would core dump sievec:
if address :contains To [EMAIL PROTECTED] {
stop;
}
Changing 'address' to 'header' would work just fine.
I tracked it down to a header file, tree.h. Basically, there is a union
defined inside the 'Test' structure that itself defines a few structures
that would share the same memory space (so to speak... if you understand
C programming, then you should understand what unions are). The problem
is that two of the structures defined many of the same variables as each
other, but in different orders...
For example, the first structure in the union declares A, B, then C, but
the second structure in the union declares A, C, then B.
This shouldn't normally be a problem, but on Tru64 5.x, it certainly is.
I am not sure if it is a weirdness with 'cc', or if it is something deeper.
The code compiles and without warnings, but it core dumps at runtime.
Simply changing the order in one structure or the other to match solves
the problem nicely.
This could very well be related to the following bug:
https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2476
I submitted a patch, which I will include here, that should fix it. It
is a *very* small patch :)
Scott
--
+---+
Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/
UNIX Systems Engineer mailto:[EMAIL PROTECTED]
ICQ 7626282 Work (740)593-9478 Fax (740)593-1944
+---+
PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/
SIEVE_UNION.patch
Description: Binary data
pgpjsabmK0EnL.pgp
Description: PGP signature
Tru64 and rpath
We are about to upgrade our production system to 2.2.8. One of the biggest problems that I am fighting is the rpath issue. Basically, it is important for the binaries (imapd, pop3d, etc) to be able to find both the SSL libs and the SASL2 libs. When I configure the server, I set LDFLAGS to set the appropriate rpath options so that when I do compile the server, it will include the library paths in the binaries. Basically, it looks like this: export CC=cc export LDFLAGS=-Wl,-rpath,$SSLDIR/lib:$SNMPDIR/lib:$SASLDIR/lib ./configure --with-sasl=SASLDIR --with-openssl=$SSLDIR .. What is the problem? The problem is that it doesn't work. The compiler on Tru64 (cc), does not handle multiple rpath options on the command line in the same way that gcc does. In fact, Tru64 expects only one rpath option on the command line, and if there are multiple rpath's specified, only the last one on the command line will be accepted. The proper way to handle this in Tru64 is to make the path colon delimited (as in the above LDFLAGS setting) and make sure that it appears as the only one on the command line, or that it appears last on the command line. This is now how it works with Cyrus... the configure process builds the rpath stuff as it is configuring it. Since I set the LDFLAGS stuff before configuration, the configure script puts it first on the command line. As it builds the libraries that it must include (using the -L and -l stuff), it adds to the command line and adds additional rpath options as well, which override the previous ones. The end result is that my binaries are dumb with how to find all but one of the shared libraries. The only solution that I have right now is to hack all the Makefile's after the configuration process has completed to remove the extra rpaths. That is quite painful. The other thing I could do is hack the startup scripts to set LD_LIBRARY_PATH, and hack the perl scripts to do the same and make sure that anything we write that interfaces with Cyrus does the same too. Personally, I would rather see it solved at configuration time, but I have yet to figure out a way to do so. Any ideas? There doesn't seem to be a way to force configure to either create an appended form of a single rpath, or to not build rpath information at all. Either method would work for me. Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpbSt2SwLX46.pgp Description: PGP signature
[PATCH: sendmail] LMTP Plus User Support
As requested, I am posting my patch to sendmail that allows plus addressing to work properly between Sendmail and LMTP. We connect to our LMTP via TCP in the sendmail configuration and I have never been able to get +addressing to work at all. I usually get a user unknown error, rejecting the e-mail. Note: If you use the old cyrus mailer that calls deliver to deliver the e-mail, plus addressing works just fine there. You need to include the FEATURE(`preserve_local_plus_detail') line in the MC file, though. Anyways, the patch code is only enabled if you include the '' flag in the cyrus mailer definition. Once enabled, when sendmail determines if the user is valid, it will stop at the '+' character in the username portion of the e-mail address. Also, sendmail typically lowercases the username, so again, it will stop at the '+' when doing the lowercasing. This allows the folder names to have their casing preserved. For testing purposes, I created test1, test box1, Test2, and Test Box2 folders and added anyone: post ACLs on all of them. I then used the following e-mail addresses to test: [EMAIL PROTECTED] user+test\ [EMAIL PROTECTED] [EMAIL PROTECTED] user+Test\ [EMAIL PROTECTED] The only reason I said anything about how I tested is that it isn't common knowledge about how to test with mailboxes containing spaces in them. So, there you have it. I have submitted the patch to sendmail, so maybe it will get into the next version... who knows... The patch applies to sendmail-8.13.0. I had to change the flag I was using to an ''. My previous version was using 'W', which now appears to be in use already in 8.13.0. Ah well. Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ lmtp-8.13.0.patch Description: Binary data pgprVunpJ7fhB.pgp Description: PGP signature
Re: Imap mailbox aliases (two names for the same box)
The solution we opted for was to patch Sendmail to make it work :) I have an lmtp patch that doesn't drop the stuff after the + in the address, and it also does not mess with the case with anything after the +. I implemented as a mailer flag, so including a 'W' in the cyrus mailer flags triggers the code to work. I can post it if you are interested... I have tried working with the m4 code in the sendmail.cf file to make that same functionality work without having to patch sendmail. I have yet been able to succeed. Does anyone have a sample config file (other than CMU :-) that works? CMU's code was so complex I couldn't figure it out. Basically, I got all of their config files, and they do a lot in their environment, so it was like looking for a needle in a hay stack :) Scott --On Tuesday, July 20, 2004 6:39 PM +0200 Ghislain ADNET [EMAIL PROTECTED] wrote: Hello, I know some of you had the problem of the mailbox with an upper case that are not recognized when goign through sendmail. For exemple writing to [EMAIL PROTECTED] is translated [EMAIL PROTECTED] My problem is Thunderbird use Junk, Trash Sent name boxes etc... and have special icons to show those folders that permit easy identification of the folder by mundane users. But i cannot forward to uppercase adress folders so i have to choose between : 1/ use sent instead of Sent and forget about the special icon in thunderbird and mozilla 2/ find a way to create an alias at the server level between Sent and sent (so basicaly only one folder for two names) Do any of you have a solution for the second solution ? best regards, Ghislain. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpcH0DYR34IY.pgp Description: PGP signature
Re: IMAP/LMTP/Quota locking problem
--On Tuesday, May 18, 2004 9:13 AM +0200 Sebastian Hagedorn [EMAIL PROTECTED] wrote: thanks for the info. The patch was definitely added *after* 2.2.1 ... hmm, now that I think about it: seems like it was even added after 2.2.3! Is bug 1270 really fixed in CVS only? Good question! :) I hope it is fixed :) If I remember right, it has something to do with SSL. Are you using SSL? Basically, the quota file gets locked and stays locked because the IMAP process (which is usually the one that has the quota file open for write) is trying to do a read on an SSL socket (this is from memory and from what I have seen in our traces). The quota lock problem is just a side effect of that issue... Currently, we manage the problem by watching the sendmail syslog file for System I/O errors on the Cyrus mailer (grep for 'mailer=cyrus.*System.I.O') We don't get those. I wonder if we actually have separate problems. Or maybe not. However, in our case the sendmail processes get stuck as well, maybe due to our setting of DeliveryMode to interactive!? I find that interesting... after an IMAP process locks the quota file, the LMTP processes start to stack up. After awhile, anything trying to be delivered to that user generates a System I/O error. It makes it easy to catch, since as soon as we see a few errors for a particular user, we run fuser on that user's quota file to get a list of processes, run lsof on that list of processes to find the process that has the quota file open for write and we kill it. The next entry in the syslog should show that a message was sent instead of deferred, which means we fixed the problem. The problem occurs regularly, with some users more reliably triggering the problem than others. What does regularly mean? We've been running the server in prdouction since March, 1, but it happened for the first time a little more than a week ago ... On average, I would say at least once every day or two. Sometimes it gets worse and we get a few in a day, and sometimes, a single user may do it a couple times in a day... It does seem like some users trigger it a lot more than the majority. This suggests some kind of interaction between the IMAP process and the e-mail client, possibly back to that SSL read issue again. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgpPDk4Ziu4dw.pgp Description: PGP signature
Re: IMAP/LMTP/Quota locking problem
I do know that as of 2.2.1, the problem still exists. I find it impossible to trigger the bug on my own in our test environment, and we won't be doing an upgrade to 2.2.3 (or whatever happens to be out) until the summer time frame. Currently, we manage the problem by watching the sendmail syslog file for System I/O errors on the Cyrus mailer (grep for 'mailer=cyrus.*System.I.O') and when we see some, we run another script that tells us which cyrus proc has the write lock on their quota file and we kill that process. The problem occurs regularly, with some users more reliably triggering the problem than others. For those who are curious, our Cyrus server runs on a cluster of 3 machines running with Alpha Tru64 5.2. Scott --On Monday, May 17, 2004 2:47 PM +0200 Sebastian Hagedorn [EMAIL PROTECTED] wrote: Hi, sorry, this is quite long ... I'm looking for the updated status of the problem discussed in the threads stuck lmtpd processes: http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrusms g=24927 and followup: stuck lmtpd processes: http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrusse archterm=stuck%20lmtpdmsg=24934 AFAICT Cyrus IMAP 2.1.x hasn't been updated since. Could anyone comment on whether this is fixed in 2.2.3 and if we *have* to update in order to get that fix? In the last two weeks we've had this (or a very similar) problem twice with 2.1.16 und Red Hat AS 2.1. For some reason in both cases tons of sendmail processes were building up. We have configured sendmail to deliver in interactive mode, so each instance tries to connect to lmtpd. I would've expected those attempts to time out relatively quickly. I'm not sure why they don't. Anyway, that's somewhat off-topic ... The first time it happened I was around and collected some info while it was happening. I'd noticed that mail was no longer being delivered. We have maxprocs set to 30 for lmtpd: [EMAIL PROTECTED] root]# ps -aef|grep lmtp cyrus 1383 2095 0 10:10 ?00:00:18 lmtpd cyrus 1974 2095 0 10:11 ?00:00:00 lmtpd cyrus10631 2095 0 10:36 ?00:00:00 lmtpd cyrus17162 2095 0 10:52 ?00:00:00 lmtpd cyrus17262 2095 0 10:52 ?00:00:06 lmtpd cyrus25687 2095 0 11:17 ?00:00:00 lmtpd cyrus28377 2095 0 11:23 ?00:00:09 lmtpd cyrus31251 2095 0 11:32 ?00:00:00 lmtpd cyrus 3824 2095 0 11:46 ?00:00:07 lmtpd cyrus 7534 2095 0 11:56 ?00:00:00 lmtpd cyrus 8248 2095 0 11:58 ?00:00:08 lmtpd cyrus22103 2095 0 12:35 ?00:00:00 lmtpd cyrus23725 2095 0 12:39 ?00:00:01 lmtpd cyrus25366 2095 0 12:45 ?00:00:00 lmtpd cyrus 6108 2095 0 13:23 ?00:00:00 lmtpd cyrus10832 2095 0 13:37 ?00:00:00 lmtpd cyrus12972 2095 0 13:42 ?00:00:00 lmtpd cyrus16114 2095 0 13:53 ?00:00:00 lmtpd cyrus21377 2095 0 14:07 ?00:00:09 lmtpd cyrus24581 2095 0 14:16 ?00:00:00 lmtpd cyrus26200 2095 0 14:21 ?00:00:14 lmtpd cyrus26577 2095 0 14:22 ?00:00:00 lmtpd cyrus31893 2095 0 14:35 ?00:00:03 lmtpd cyrus32575 2095 0 14:36 ?00:00:07 lmtpd cyrus32659 2095 0 14:37 ?00:00:01 lmtpd cyrus14146 2095 0 15:16 ?00:00:00 lmtpd cyrus21049 2095 0 15:36 ?00:00:00 lmtpd cyrus26969 2095 0 15:56 ?00:00:11 lmtpd cyrus13275 2095 0 15:15 ?00:00:04 lmtpd cyrus17679 2095 0 13:57 ?00:00:00 lmtpd Obviously something was wrong. [EMAIL PROTECTED] root]# lsof -p 26969 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME lmtpd 26969 cyrus cwdDIR9,0 4096393296 /root snip lmtpd 26969 cyrus memREG 8,73 384157630 /var/lib/imap/mailboxes.db lmtpd 26969 cyrus memREG 8,49 194 145326099 /var/spool/imap/M/user/therbst/cyrus.header snip lmtpd 26969 cyrus9u REG 8,49 3130 56013510 /var/spool/imap/stage./26969-1083938870 lmtpd 26969 cyrus 10u unix 0xef1ce040 614472274 socket lmtpd 26969 cyrus 11u REG 8,49 194 145326099 /var/spool/imap/M/user/therbst/cyrus.header I gathered that /var/spool/imap/M/user/therbst/cyrus.header was causing the problems.Its node number (is that the inode??) is 145326099: [EMAIL PROTECTED] root]# grep 145326099 /proc/locks 1162: POSIX ADVISORY WRITE 1974 08:31:145326099 0 EOF e9b58d54 e9b587f4 cd5a7964 d1c324c0 1162: - POSIX ADVISORY WRITE 1383 08:31:145326099 0 EOF d1c324b4 c7fd745c f3bd0964 e9b58d54 c7fd7464 1162: - POSIX ADVISORY WRITE 10631 08:31:145326099 0 EOF c7fd7458 d1c3245c d1c324b8 e9b58d54 d1c32464 1162: - POSIX ADVISORY WRITE 17162 08:31:145326099 0 EOF d1c32458 d6989400 c7fd745c e9b58d54 d6989408 1162: - POSIX ADVISORY WRITE 17262 08:31:145326099 0 EOF d69893fc dfb2eca0 d1c3245c e9b58d54 dfb2eca8 1162: - POSIX
sendmail hooks for pre-checking of over-quota imap users?
I was curious if anyone has implemented a mechanism in sendmail to determine if a user is over their Cyrus quota before attempting LMTP delivery of the message. If so, how was it impemented? In our environment, I would have to say that easily, 3/4ths of all our e-mail hitting the LMTP server is over-quota'd e-mail. What I have done is to create a new hash file called /etc/mail/overquota.db that gets updated periodically (once an hour or maybe even less) with the list of users currently over their quota's. I then modified the sendmail queuegroup rules to check for the existence of a user in that hash file and move then to the overquota queue if so. I find this better than dumping all e-mail destined for local delivery to the cyrus queue, and then using a queue mover to find all the messages that had attempted LMTP delivery, but failed with an Over quota message. That turns out to be very expensive with regards to disk I/O and CPU utilization. Also, if anyone else has other interesting ideas on how they handle lots of over quota e-mail (besides shortening the time that e-mail is kept on the server or rejecting that e-mail outright), I would be interested in hearing about it. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Error: mailbox is reserved
What causes this error? We have a case with a particular user (twice in the last week or so now) was trying to create a subfolder and could not, getting the error mailbox is reserved. The folder indeed did not exist, and the user just couldn't get it created. What I have found in the archives was just one thread. The user was trying to do the reverse, delete and already existing folder. Rob responded that the database needed to be recovered by running 'ctl_cyrusdb -r' on it. So, my question is, what causes existing folders to get reserved (so they can't get removed) and what causes a non-existing folder to get reserved (so they can't get created). The latter boggles my mind, as it means that a non-existant folder somehow gets into the database... Is it possible that the user tries to create the folder, it fails, gets reserved, and then all future attempts at creating the folder gets the above error? Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
postmaster mail
Okay, I thought I would pull from the wisdom from the group, but I hope it isn't too far off topic. We have been using an alias for the root account to get all root destined mail to the /var/spool/mail/root mailbox, bypassing the Cyrus system. However, our system has grown significantly over the years and now we get on average 25,000 messages a day to the root account alone. Basically, we just fire up mailx and delete everything. If there is anything useful in that mailbox, it is doubtful we would have ever found it anyways. So, today, I created the user.root mailbox and changed the alias so that all the mail can go to it. I also created a postmaster subfolder and directory the postmaster alias to send its mail to root+postmaster. It all works... really works... :) It is clear that 99.9% of all our mail is postmaster mail. It looks like it is mostly unknown user mails, over quota mails, connection refused or deferred mails, etc. I know that there may be some important postmaster mail that we should keep... obviously, we have been deleting stuff for a long time, but it would be nice to not delete blindly and actually keep the important messages we should be dealing with. Incidentally, in the last 20 minutes, 300 messages came into the postmaster folder. So, the question is, do we just turn off postmaster mail altogether (is that actually doable? maybe by setting PostmasterCopy to nothing?), or do I somehow build an intelligent sieve script that throws away most the stuff that we think are junk? What have you guys done? I am more interested in the larger email system, ones receiving on the order of 500k+ messages a day. Is it abnormal to see such a large volume of postmaster mail like this? Have any of you developed a sieve script that would be useful to the Cyrus community as a whole? Enquiring minds want to know! Thanks for any thoughts on the subject... Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
RE: postmaster mail
--On Wednesday, January 21, 2004 12:52 PM -0800 Pat Lashley [EMAIL PROTECTED] wrote: --On Wednesday, January 21, 2004 14:22:33 -0600 Robert Covell [EMAIL PROTECTED] wrote: I too would be interested in the question about abnormal amount of emails to postmaster. We get about 15K a day of these and just recently started to pipe them into /dev/null. It is more of a burden to delete them manually. Anyone else have high postmaster email counts? I'll bet you're both using Sendmail and have Mailer-Daemon aliased to postmaster. If so, change the Mailer-Daemon alias to /dev/null That should eliminate the various bounce messages without stopping anything that is actually addressed to postmaster. (One of the things I like about Exim is that it doesn't use Mailer- Daemon; and the default handling of undeliverable bounce messages is much more sane.) -Pat Actually, the aliases for postmaster and mailer-daemon are as follows: postmaster: root+postmaster mailer-daemon: root+mailer-daemon The problem is that the mail isn't going *to* Mailer-Daemon, it is coming *from* Mailer-Daemon *to* postmaster. So, setting the alias to mailer-daemon isn't going to matter here. The relevant sendmail.cf configuration is as follows: # who (if anyone) should get extra copies of error messages #O PostmasterCopy=Postmaster # where do errors that occur when sending errors get sent? #O DoubleBounceAddress=postmaster I find it a bit confusing, as PostmasterCopy is commented out, which according to the docs means that it is undefined and thus does not send copies of error messages to the postmaster. The DoubleBounceAddress option is also commented out, which means that errors generated from already bounced e-mails will be sent to postmaster by default. This may be the culprit for our large amount of mail, and we can uncomment it and set it to nothing (i.e. remove 'postmaster') in order to not have e-mail sent to postmaster. The big question I am asking is what other people are doing. Are they turning it off (via configuration options or forwarding to /dev/null) or are they employing some filtering to throw away the bad in order to see the good? From what I am seeing in the messages, it isn't the easist thing in the world to come up with a good sieve filter to do it for me. By the way, wouldn't it be better to disable the sending of those e-mails instead of forwarding to /dev/null. I am guessing that this would save on both CPU and IO if these e-mails weren't being generated to begin with. We receive easily over 500K messages a day, and it pains me to think that 25K of these are messages that we are throwing away, whether it be dumping them to /dev/null or manually deleting them. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: Cyrus IMSPd 1.6a4 and 1.7a Released
--On Friday, December 12, 2003 4:53 PM -0500 Cyrus Daboo [EMAIL PROTECTED] wrote: Hi Scott, --On Friday, December 12, 2003 3:10 PM -0500 Scott Adkins [EMAIL PROTECTED] wrote: | Are there changelogs for these releases? I am particularly interested in | whether or not the SSL patches previously posted for IMSP has been rolled | in or not. What is the difference between the 1.6 and 1.7 releases, | since they were both released at the same time? SSL (STARTTLS) is currently not in the v1.7a code-base, but I do plan on integrating our implementation in in the next few weeks. I suspect that will initially need to go out as a beta to get some thorough testing under load. -- Cyrus Daboo Hi Cyrus, I look forward to trying out your Beta version when it becomes available. Will it have similar support as the Cyrus IMAP server does? What I mean is that just running imspd would allow the server to listen only on port 406 with STARTTLS support. Running imspd -s would allow the server to listen only on port 906 and would negotiate an SSL session as soon as the connection is made. Currently, we are running imspd with plaintext support and running stunnel to get the imspd -s support. It would be nice to eliminate the stunnel altogether and just running the IMSP server. Another thing I would like to see is something that can be included in the global options file to turn off plaintext logins on the port 406 and force the use of STARTTLS if they are going to connect to that port. This is how Cyrus IMAP works with the imapd.conf file (i.e. allowplaintext: no). Finally, I have a patch that allows case insensitive address books. Would you be interested in it? Before we switched to Cyrus IMSP, we were using some other IMSP server (can't even remember now what it was, since it has been so long ago). If you would like to know more about that, let me know. Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: Cyrus IMSPd 1.6a4 and 1.7a Released
Are there changelogs for these releases? I am particularly interested in whether or not the SSL patches previously posted for IMSP has been rolled in or not. What is the difference between the 1.6 and 1.7 releases, since they were both released at the same time? Scott --On Friday, December 12, 2003 2:25 PM -0500 Rob Siemborski [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 This message is to announce the release of Cyrus IMSPd 1.6a4 and 1.7a on ftp.andrew.cmu.edu These releases correct a recently discovered buffer overflow vulnerability, as well as clean up a significant amount of buffer handling throughout the code. I'd like to thank Cyrus Daboo for the time he spent cleaning up a great deal of the code, and Felix Lindner of n.runs for alerting us to the vulnerability. All sites are urged to upgrade as soon as possible. The distribution is available at: ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imspd-v1.6a4.tar.gz ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imspd-v1.7a.tar.gz and http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imspd-v1.6a4.tar.gz http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imspd-v1.7a.tar.gz - -Rob - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper -BEGIN PGP SIGNATURE- Version: PGP 6.5.8 Comment: Made with pgp4pine 1.76 iQA/AwUBP9oWLmes8cJc4y/MEQLINgCeJKZ8lEI9I2eGI26/bwFS8TbTSpEAoL5N guQ7DY7iuHeEZo8rWWAH7HrO =fZrU -END PGP SIGNATURE- -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: Cyrus IMAPd 2.2.2-BETA Released
--On Thursday, December 04, 2003 10:46 AM -0500 Rob Siemborski [EMAIL PROTECTED] wrote: On Thu, 4 Dec 2003, Luca Olivetti wrote: Is there any way to make it into a shared library? Sure -- but theres not a compelling reason to, and it would massively increase the complexity of the build process (as different systems have different ways of boilding shared libraries -- hense libtool). Wouldn't the memory savings for doing that be significant though? Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: STARTTLS Question
--On Tuesday, October 07, 2003 6:25 PM -0400 Daniel Whelan [EMAIL PROTECTED] wrote: I'll go ahead and answer my own question, as I evidently haven't been paying as much attention to the mailing list as I should have lately and found my solution buried back a couple months. On 30 July 2003 Matt Bernstein started a thread entitled requiring encryption but not from localhost?, where Scott Adkins proposed a solution. I implemented something more or less like he proposed, and it worked. Specifically, I created a second imapd.conf (imapd-local.conf) and configured it with allowplaintext: yes. Then, I edited my cyrus.conf to look like the following: imaplocal cmd=imapd -U 30 -C /etc/imapd-local.conf listen=localhost:ima plocal prefork=0 maxchild=100 imapcmd=imapd -U 30 listen=imap prefork=0 maxchild=100 imaps cmd=imapd -s -U 30 listen=imaps prefork=0 maxchild=100 I couldn't get imaplocal to listen localhost on the imap port, so I defined an imaplocal port in /etc/services as port 144, and pointed webmail at that. All is well now...webmail from localhost gets plaintext, and everyone else gets IMAPS or IMAP/STARTTLS. If I had to guess, the reason you couldn't get imaplocal to listen to localhost:imap is probably because the imap and imaps services were already listening on the imap/imaps ports, specifically *:imap and *:imaps. I don't know what order Cyrus internally starts to configure the ports for listening on, but that is likely the issue here (port binding). The solution that should work is the following (trimmed for clarity): imaplocal cmd=imapd -C /etc/imapd-local.conf listen=localhost:imap imap cmd=imapd listen=hostname:imap imaps cmd=imapd -s listen=hostname:imaps Whatever the hostname of your machine is should replace hostname in the above configuration. If you want to be able to telnet to the imaps port on localhost for who knows what reason, then take hostname out of the imaps line and just leave it as imaps. Hope that helps... Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: followup: stuck lmtpd processes
I just wanted to add something to this discussion... First of all, we see the problem in Tru64 as well. When we upgraded to the 2.2 series, we put in the locking patch that John described below. This has helped us, but the locking problem has *not* gone away... in fact, it does a better job of *hiding* the problem than fixing it... besides, doing some kind of backoff timeout mechansim doesn't solve the problem that some other process has permanently placed a lock on a file, all it does is prevent lots of LMTP processes from stacking up *because* some other process permanently locked a file. As a result, we still experience the problem, and it manifests itself as being a particular user is unable to receive any email... and until that user actually notices that this is happening, we may not notice. The log files (the syslog mail log particularly) does give some clues, however, such as a lot of System I/O errors when talking to LMTP for a particular user). When looking at what file the processes are all waiting to get a lock on, it usually turns out to be the cyrus.header file and not the quota file. Is this still the same bug described by Rob on bugzilla? Does it have to be the quota file? Also, when we find the specific imaps process that happens to have the cyrus.header lock file opened for writing and has it locked, if we kill it off, we find that the write lock goes to another imaps process or to one of the LMTP processes and gets stuck there... we kill that one off and it goes to the next one and gets stuck. We never saw a case where all the other processes became unstuck and the problem went away. As a consquence, the only solution we have when we see the problem is to restart the Cyrus server (we usually wait until after work hours at least). I am not convinced the patch described below has helped us much, as when we saw the LMTP processes stacking up, it was right in our face and we could deal with it sooner than later. Anyways, those are my thoughts on the subject. Scott --On Tuesday, September 23, 2003 10:35 PM -0500 John Wade [EMAIL PROTECTED] wrote: Hi Andrew, I was the one who wrote the message you found. I finally came to the conclusion that the flat file locking mechanism is somewhat broken in Cyrus, but I was never a good enough C programmer to pin down what was happening. (The mmap stuff makes it really tricky to debug.)I wanted to blame it on the Linux kernel, but I know that others have experienced the same problems in Solaris. I finally gave up and wrote a locking timeout patch for 2.0.16. see http://www.oakton.edu/~jwade/cyrus/ for the patch and full details A number of other folks have tried this patch successfully on 2.0.16 and 2.1.x, and I know it has resolved our problem. If you can solve the particular bug that causes this, more power to you, if not, my work around resolves a number of possible deadlock issues. Enjoy, John Andrew Morgan wrote: Following up on my previous post about stuck lmtpd processes. I found this incredibly detailed faq at: http://www.faqchest.com/prgm/cyrus-l/cyrus-01/cyrus-0111/cyrus-011102/cy rus0023_33254.html This isn't exactly the same problem, but the steps on that page helped me figure out that they are all stuck trying to get a lock on: /private/cyrus/mail/k/user/krolickp/cyrus.header Looking at /proc/locks shows: 7: POSIX ADVISORY WRITE 21903 08:11:42107658 0 EOF d23895e0 c3217f44 c510e4c4 ccbf076c 7: - POSIX ADVISORY WRITE 32485 08:11:42107658 0 EOF ccbf0760 ee36ac44 f3bb26a4 d23895e0 ee36ac4c 7: - POSIX ADVISORY WRITE 1802 08:11:42107658 0 EOF ee36ac40 c050ea04 ccbf0764 d23895e0 c050ea0c 7: - POSIX ADVISORY WRITE 1217 08:11:42107658 0 EOF c050ea00 ee36a344 ee36ac44 d23895e0 ee36a34c ... I don't see how this deadlock occurred, but I'm willing to help debug it. Andy -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: imapd mem exhausted
What version of Cyrus are you using? We are using 2.2b1 here, and are experiencing something similar with memory issues for IMAP. I brought up the discussion a few weeks ago about it and it was characterized as some kind of mmap() weirdness on our Tru64 platform, which I don't really think is the case. I will look in our logs to see if we are seeing a similar issue with a lot of accepted connections as you are. My experiences are as follows with the problem: 1) It seems that the RSS footprint of the process increases when a user makes a connection (from observation). 2) Most of the time, we may see a process jump up to 26 or 27MB in size (and lately, we are now seeing them jump up to 30-32MB in size), but typically go back down to several hundred KB shortly thereafter. If the system starts to slow down for whatever reason, the time for a process to shrink increases, and we have more processes at the bigger RSS size and run the risk of exhausting memory and swap. 3) The problem was characterized as an mmap() problem on Tru64 because our mailboxes.db file is about 27MB in size. However, we are seeing the sizes jump to 30-32MB in size, and our mailboxes.db file is still only about 27MB in size. *shrugs* 4) Restarting the Cyrus server has a dramatic effect on memory usage by putting all the processes back at the base several hundred KB size. It takes quite a while before we see the issue #2 become a problem again (even on a busy server). Anyways, I have been looking at this problem over that last several days and my gut feeling has been leaning towards the on-connection route... which certainly jives with your message below... Scott --On Wednesday, September 24, 2003 12:02 PM +0100 Patrick Welche [EMAIL PROTECTED] wrote: Sep 24 09:50:30 imp imap[2030]: executed Sep 24 09:50:30 imp imapd[2030]: accepted connection Sep 24 09:50:47 imp imapd[2030]: accepted connection Sep 24 09:51:15 imp imapd[2030]: accepted connection Sep 24 10:26:23 imp imapd[2030]: Fatal error: Virtual memory exhausted That was then a reasonably long lasting connection that was just doing CREATE followed by stacks of APPEND non-stop. There was only me as a user (only 1 imapd). I suppose that is rather different to the usual usage pattern of lots of connections doing a few things. After a few minutes I see: PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 19526 cyrus 2035M 37M select 0:10 0.93% 0.93% imapd Is this a memory leak, or are the messages being stored in memory ie., it's meant to happen? Is there a simple way of telling master to give imapd some more memory when it forks? Cheers, Patrick -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: followup: stuck lmtpd processes
Well, that could definitely be a problem... Next time we see a lock problem occur, I will look based on the information below to see if it is really a lock problem on the quota file. Thanks, Scott --On Wednesday, September 24, 2003 12:32 PM -0700 Andrew Morgan [EMAIL PROTECTED] wrote: On Wed, 24 Sep 2003, Scott Adkins wrote: When looking at what file the processes are all waiting to get a lock on, it usually turns out to be the cyrus.header file and not the quota file. Is this still the same bug described by Rob on bugzilla? Does it have to be the quota file? Also, when we find the specific imaps process that happens to have the cyrus.header lock file opened for writing and has it locked, if we kill it off, we find that the write lock goes to another imaps process or to one of the LMTP processes and gets stuck there... we kill that one off and it goes to the next one and gets stuck. We never saw a case where all the other processes became unstuck and the problem went away. Are you sure that the processes are hung on the cyrus.header lock? That's what I originally thought when I was only looking at the output of lsof and /proc/locks (linux). When I actually ran a gdb backtrace on one of the stuck processes, it became obvious that the lock was on the quota file instead: (gdb) bt # 0 0x402ae5fb in fcntl () from /lib/libc.so.6 # 1 0x08077504 in lock_reopen (fd=16, filename=0xbfffa098 # /var/spool/cyrus/config/quota/k/user.krolickp, sbuf=0xbfffa040, failaction=0xbfffa03c) at lock_fcntl.c:87 # 2 0x080570b6 in mailbox_lock_quota (quota=0xbfffc3c4) at mailbox.c:1016 # 3 0x08053f73 in append_setup (as=0xbfffc118, name=0xbfffb114 # user.krolickp, format=0, userid=0x0, auth_state=0x0, aclcheck=0, quotacheck=0) at append.c:209 I also saw exactly the behavior you describe when killing processes. I originally tried killing all the lmtpd process that were stuck because I believed that one of the lmtpd process was stuck holding the lock on cyrus.header. When I killed the one holding the lock on cyrus.header, another lmtpd process would grab the lock but still be stuck. When I finally killed the process holding the quota file lock (an imaps process), all the lmtpd processes got unstuck and delivered the waiting mail. It sounds to me like you're not actually killing the process that has the lock that all the other processes are waiting for. Andy -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: database recovery...
Well, I am not sure that it is something bizarre going on with the mmap() method that configure chose at compile time. I still have to do some testing, but I am not really convinced that the 27MB/28MB sizes are tied in to mailboxes.db being nearly the same size. So, what kind of things would cause the process to grow? If a user makes a connection to IMAP, it starts out small. Does the memory footprint grow as they open and close folders, reading in the various cyrus files listed in that particular folder? I imagine that if somebody had a really big folder (like the many around here who never delete their mail), could that drive the footprint up a bit? With process resuse, especially with 250 connections per process, I can imagine that the older process will be the ones that are much bigger. Anyways, it is really hard to pinpoint on the system. Scott --On Wednesday, September 10, 2003 9:28 AM -0400 Rob Siemborski [EMAIL PROTECTED] wrote: On Wed, 10 Sep 2003, Scott Adkins wrote: So, with 3000+ cyrus process averaging about 20MB each, it consumed pretty much all our real RAM (we have 8GB on each cluster member). I would say about 6GB of memory was consumed in just Cyrus processes. This sounds like something bizarre is going on with what cyrus chose for its mmap() method. (Or the Tru64 mmap is doing something silly in terms of memory allocation). Since this didn't really change between 2.0 and 2.1, I don't offhand know what to blame (though perhaps a change of database formats could do this also--since skiplist will grow much larger than flat for the same data). -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: database recovery...
So, you guys seeing around a 20MB virtual size and a 1MB resident size when looking at the process table, right? Scott --On Wednesday, September 10, 2003 11:09 AM -0400 Rob Siemborski [EMAIL PROTECTED] wrote: On Wed, 10 Sep 2003, Igor Brezac wrote: My installation of cyrus 2.2-CVS on Solaris 9 shows the same memory footprint per process. 20M is taken by mmap(), the rest is shared physical memory and about 1M of 'private' physical memory per process (imapd, pop3d, lmtpd, etc). This is consistant with our 2.1 installation as well. That's why I'm lead to suspect either mmap() wasn't detected properly by cyrus (And you're therefore using map_nommap), or the mmap implementation itself is doing something silly. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
database recovery...
We are running a Tru64 TruCluster system. We have 2 members in the cluster and run Cyrus IMAP 2.2.1b. We typically ran the system with Cyrus being CAA'd and only running on one member at a time. The stuff would relocate to the other cluster member if for some reason it could not run on the first one or we had to take it down for maintenance or whatever. Well, it appears that this new version uses a lot more memory than the 2.0.16 version did, with a lot of the processes settling on 27MB or 28MB of resident memory in use (not virtual memory, which the processes always indicate has more in use, but real memory in use). On Tru64, there is no way to determine exactly where that memory is going, unlike Solaris where you can run any of the proc tools, like pmap, to get a break down of what memory is shared, what is stored in the heap and what is consumed by the stack. Running lsof doesn't help, as they all show the same thing... interestingly enough, our mailboxes.db file is about 27MB in size, but I can find a lot of processes that are only a couple megabytes in size and that file is opened with them as well, so I think it is just a coincidence. Has anyone else noticed the larger memory footprint? So, with 3000+ cyrus process averaging about 20MB each, it consumed pretty much all our real RAM (we have 8GB on each cluster member). I would say about 6GB of memory was consumed in just Cyrus processes. We decided to run Cyrus on both cluster members at the same time. Since we are using a cluster file system which uses flock() to keep things working properly, it shouldn't be a problem. For those not familiar with Tru64's cluster file system, this is not NFS. It is basically a local file system as far as each member is concerned, but it is shared like NFS on all the members. Anyways, as Cyrus starts up on each member, it runs the ctl_cyrusdb -r command. The problem with that is that it runs it on each member at the same time (if I start them at the same time), so mailboxes.db has two of these processes hitting it at the same time. Worse, one member may finish faster than the other and start accepting connections before the other member has completed the recovery process. This doesn't appear to cause any side effects, but I would like to know if there would be any from this... especially if users are hitting the file while a recovery is in progress. Also, it takes a really *really* long time for the recovery process to run, which means even a simple restart is felt by all, as it takes several minutes for it to complete. In 2.0.16 with a flat file database, there was no wait at all for the restart to occur, and most people may not even notice it, since their email clients would silently reopen IMAP connections that were closed on them. Is there any way to shorten the duration of the recovery process? For instance, increasing the frequency of checkpoints considerably is one idea I have... would that help? Is there a point that I could do the recovery process on a schedule (like once a night) instead of running it at startup time to cut down on the overhead? Anyways, I am looking for some insight into this process... Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
cvt_cyrusdb vs ctl_mboxlist
Okay, so we are using skiplist for our database now. If we wanted to do some simple searches on the database (we used to just grep for usernames on the flat file version of mailboxes.db), we have to first convert the database to a flat file. Really, the best way to do it is by running ctl_mboxlist and saving the output to a file. But before I discovered the command, I was trying to use cvt_cyrusdb to convert from skiplist to flat file, which basically took forever (in fact, I never got it to really finish, as I would Ctrl-C out of it before it was done). The problem with cvt_cyrusdb is that it seems to take a record from the skiplist database and writes it to a file to a new flat file database. If we call the new database flat.db, it would create a new file called flat.db.NEW with that one line added to the end of it, then move it to flat.db right after that. This process would be done for every record in the mailboxes.db file (ours is huge!). Wouldn't it be better to to enhance the conversion program to simply open the flat file database for append and just dump the contents of the old db right to it? It seems like the logic for writing to a flat file db is prohibitively expensive. Really, you want the conversion process from any database format to flat file format to dumb itself down to be what ctl_mboxlist does. The reason ctl_mboxlist works so well is that it just dumps the output to stdout and not to a file, so there are not .NEW files or renames or anything like that. Just a thought. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: append_check() failed
I had this problem initially when moving to 2.2.1b. The problem was that the users were actually over quota and that was it. Rob committed some code to CVS for the next version to make the append_check() errors display a more verbose message about why it failed... When I applied the patch, it became very clear that quotas were the real issue here. I will attach the patch to this email. It applies directly to imap/lmtpd.c and only 2 lines get changed. See if that helps describe the failure msgs a bit better. Scott --On Thursday, September 04, 2003 2:40 PM +0400 Gleb Smirnoff [EMAIL PROTECTED] wrote: Hi. For last two days I was trying to make Cyrus Imapd 2.2.1.b to work on FreeBSD. I have fresh ports collection, and Cyrus Imapd 2.2.1.b installed from ports with db3 support. Here are contents of imapd.conf (everything else is left default): configdirectory: /var/imap defaultpartition: default partition-default: /var/spool/imap admins: cyrus unixhierarchysep: yes servername: relay.bestcom.ru singleinstancestore: yes duplicatesuppression: yes reject8bit: no sieveusehomedir: false sievedir: /var/imap/sieve Here is sendmail.mc (important parts of it): OSTYPE(freebsd4) DOMAIN(generic) FEATURE(mailertable, `hash -o /etc/mail/mailertable') FEATURE(local_lmtp) FEATURE(`no_default_msa') FEATURE(`preserve_local_plus_detail') define(`confLOCAL_MAILER', `cyrusv2') define(`CYRUS_MAILER_PATH', /usr/local/cyrus/bin/deliver) define(`confDONT_BLAME_SENDMAIL', `GroupReadableSASLDBFile') MAILER(`local') MAILER(`smtp') MAILER(`cyrusv2') After installation I do: # echo /usr/local/cyrus/bin/mkimap | su -m cyrus all directories were created successfully. Then master process is started... cyrus admin account created: # saslpasswd2 cyrus Then test mailbox is created: # cyradm --user cyrus --auth login localhost localhost cm user.test Then I try to deliver mail to user test's mailbox: # mail test Subejct: test test .. EOF And it fails. Here are the log entries: Sep 4 14:15:07 local6.debug relay master[75947]: about to exec /usr/local/cyrus/bin/lmtpd Sep 4 14:15:07 local6.debug relay lmtpunix[75947]: executed Sep 4 14:15:07 local6.debug relay lmtpunix[75947]: accepted connection Sep 4 14:15:07 local6.debug relay lmtpunix[75947]: lmtp connection preauth'd as postman Sep 4 14:15:07 local6.debug relay lmtpunix[75947]: append_check() of 'user.test' failed Sep 4 14:15:07 local6.debug relay lmtpunix[75947]: append_check() of 'user.root' failed Sep 4 14:15:07 local6.debug relay lmtpunix[75947]: append_check() of 'user.postmaster' failed Sep 4 14:16:07 local6.debug relay master[75885]: process 75947 exited, status 0 I have already tried db4 and db41 support, and configure with different --with-mboxlist-db, nothing helps. I have search maillists, but it happens that people get similar problems only in virtualdomain confgiuration. Does anyone have an idea? -- Totus tuus, Glebius. GLEBIUS-RIPN GLEB-RIPE -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ append_check.patch Description: Binary data MSL31026.sig Description: PGP signature
Re: append_check() failed
Does the directory for that user physically exist? If it is in the default partition, then check /var/spool/imap/user/test and see if it is there. Maybe something happened that the entries got populated in the mailboxes.db file, but the directory didn't get created or something... Scott --On Thursday, September 04, 2003 9:42 PM +0400 Gleb Smirnoff [EMAIL PROTECTED] wrote: On Thu, Sep 04, 2003 at 11:44:34AM -0400, Scott Adkins wrote: S I will attach the patch to this email. It applies directly to imap/lmtpd.c S and only 2 lines get changed. See if that helps describe the failure msgs S a bit better. Thanks for the patch. However, it doesn't made situation clear. Here are logs with patch: Sep 4 21:38:24 local6.debug relay lmtpunix[82168]: append_check() of 'user.test' failed (Mailbox does not exist) At the same time mailbox seems to exist and quota is OK: # cyradm --user cyrus --auth login localhost localhost lm user.test (\HasNoChildren) user.test2 (\HasNoChildren) localhost lam user.test anyone lrs localhost lq user.test STORAGE 0/1 (0%) localhost Does anyone has any ideas? -- Totus tuus, Glebius. GLEBIUS-RIPN GLEB-RIPE -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: imap
--On Wednesday, August 27, 2003 3:21 PM -0400 J. S. Diaz [EMAIL PROTECTED] wrote: Hi Scott, last month you posted a question about a cyrus imap install getting the db4: unable to join error and I was wondering if you resolved this or got any feedback on it? I am getting the same error on a similar setup as you and I dont really see anything obvious. thanks, sebastian Sure... I should post something about that... I talked to SleepyCat about it and they suggested making sure that the threads library was being linked into the IMAP server. It used to be that when compiling db3 that the IMAP server would automatically link in the threads library, probably because of an rpath listed in the db3 shared library. They must have changed something with db4, since the threads library is no longer linked into applications if they link against the db4 library. As it turns out, linking with threads (I believe I simply included the -pthread CC command line option to have that happen) caused the join errors to go away. I was still having some flakiness with db4 though... In the end, we decided not to run with db4 at all. This is how we currently got things setup: ./configure --with-sasl=/usr/local/sasl-2.1.15 \ --with-ucdsnmp=/usr/local/net-snmp-4.2.1 \ --with-openssl=/usr/local/ssl \ --with-egd-socket=/var/run/egd-pool \ --enable-listext --with-lock=flock \ --with-duplicate-db=skiplist \ --with-tls-db=skiplist \ --with-pts-db=skiplist One thing to note is that we forced flock to be used, as that is the method of locking that should be used in a Tru64 TruCluster system. We decided to run without duplicate suppression for email delivery, but keep the database so that sieve can still read and write to it. Since only sieve is using it, there should be a lot more reads than writes, as I think only vacation support in sieve will be writing to the database. Skiplist is ideal for high reads/low writes. Also, we ran just fine in the 2.0.x series without any kind of TLS caching, so we decided to disable that as well. We still specify skiplist above, but imapd.conf disables its use. With regards to pts-db, we don't have support for it, so the configuration process disables it. The imapd.conf file has the following lines in it: tls_session_timeout: 0 duplicatesuppression: no The first disables support for tls-db, and the latter disables support for dup-db with respect to LMTP delivery. If you have any questions, let me know. The only issue we have to work out yet is that the byte compiler for sieve is core dumping when uploading a new script to the server via timsieved. Oddly, it works on our test cluster system, but in production it core dumps. *shrugs* Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Sieve errors?
I am seeing a lot of these type errors in the log file: IOERROR: fstating sieve script /usr/sieve/a/ap934499/defaultbc: No such file or directory append_check() of 'user.mh202200' failed append_check() of 'user.am109499' failed IOERROR: fstating sieve script /usr/sieve/a/ak132301/defaultbc: No such file or directory append_check() of 'user.ag178900' failed append_check() of 'user.lm237599' failed What do they mean exactly? Debugging is set pretty high right now... Also, there aren't any sieve scripts on the system right now, as we just turned it on for the first time. This means that all of the a-z directories in /usr/sieve are empty. Should I have created a directory for every user prior to enabling sieve? What about the defaultbc files? Should I have a 'default' file of some kind copied to all the user directories? Or should I assume that these errors are normal and that it is just extra debugging information that won't be shown if I wasn't logging so high? Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: requiring encryption but not from localhost?
--On Wednesday, July 30, 2003 1:13 PM +0100 Matt Bernstein mb/[EMAIL PROTECTED] wrote: At 13:47 +0200 Sebastian Hagedorn wrote: --On Mittwoch, 30. Juli 2003 12:21 Uhr +0100 Matt Bernstein mb/[EMAIL PROTECTED] wrote: I forgot to say that at present we still need the use of the PLAIN mechanism. Is it possible to only accept PLAIN (and LOGIN, for that matter) after TLS or on the imaps port? Sure: allowplaintext: no Wrong. allowplaintext: yes Allow the use of the SASL PLAIN mechanism. Sorry. Matt Actually, I believe you were right the first time around, with no being the correct answer. I believe setting it to no means that you can't connect to the standard IMAP port and issue a plain text login without first issuing a STARTTLS command. Going to the IMAPS port is no issue. This is how we have it configured on our sysetems and it works as desired. Part of the questions I have seen related to the topic (I haven't followed all that close to the discussion) is two-fold: 1) Only allow plain text logins from localhost (meaning, you can login on the IMAP port without using STARTTLS): SOLUTION: In /etc/imapd.conf (the default file), have allowplaintext:no in it. In another config file, maybe /etc/imapd-local.conf, have yes as the value of that paramter. Then in your cyrus.conf file, you can call the services like the following: imap cmd=imapd listen=hostname:imap imapp cmd=imapd -C /etc/imapd-local.conf listen=localhost:imap imaps cmd=imapd -s listen=imaps This is off the top of my head, so you might want to check to the man pages to make sure I have it right. You have to specify your machine's hostname in the listen parameter of imap, since the default is to listen on all interfaces (including localhost), thus causing the next line to likely fail with a bind error. 2) How to accept plain text logins only after SSL/TLS has been initiated. SOLUTION is described above with allowplaintext:no in the config file. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: requiring encryption but not from localhost?
--On Wednesday, July 30, 2003 2:35 PM +0100 Matt Bernstein mb/[EMAIL PROTECTED] wrote: At 08:59 -0400 Scott Adkins wrote: SOLUTION: In /etc/imapd.conf (the default file), have allowplaintext:no in it. In another config file, maybe /etc/imapd-local.conf, have yes as the value of that paramter. Then in your cyrus.conf file, you can call the services like the following: imap cmd=imapd listen=hostname:imap imapp cmd=imapd -C /etc/imapd-local.conf listen=localhost:imap imaps cmd=imapd -s listen=imaps Neat trick! :) I'm not sure what imapp is, but I've got sieve only listening on localhost for now. A necessary evil till we either find PHP 4.3 in sid, or move to Red Hat.. The imapp tag is just that, a tag.. it means nothing, except that it has to be different form the other lines... and I believe that is what should show up in SNMP queries if you compiled that support in. I just chose the 'p' to mean plaintext port. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: flock vs fnctl
--On Monday, July 28, 2003 10:26 AM -0400 Ken Murchison [EMAIL PROTECTED] wrote: Rob Siemborski wrote: However, in Scott's case, he's not renaming the user, he's just moving it between partitions. I don't think we should be iterating across the list to fix the username in this case (so we should fix this ;)... Scott, Try this (untested) patch which I just applied to CVS: http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/imapd.c. diff?r1=1.398.2.88r2=1.398.2.89f=u I can confirm that this patch works as expected! The rename of a user from one partition to another without changing the name of the user now takes 16 seconds (on the user we tried) to complete. This is the same amount of time it took to rename the same user under the 2.0.16 code. Thanks! Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: I just don't understand...
By the way, I through ladebug at the ctl_cyrudb program. The stack trace looks like the following: 0 0x304f624 in __db_errcall(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so #1 0x304f4b0 in __db_err(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so #2 0x3071f00 in __db_e_attach(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so #3 0x3072170 in __db_e_remove(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so #4 0x306d770 in UnknownProcedure3FromFile59(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so #5 0x306d064 in __dbenv_open(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so #6 0x1200312a0 in init(dbdir=0x11fffaee8=/var/imap/db, myflags=1) cyrusdb_berkeley.c:177 #7 0x120030a38 in cyrusdb_init() cyrusdb.c:80 #8 0x1200281c8 in libcyrus_init() libcyr_cfg.c:149 #9 0x120008b88 in cyrus_init(alt_config=0x0, ident=0x14170 =ctl_cyrusdb) global.c:168 #10 0x120007fe4 in main(argc=2, argv=0x11fffc018) ctl_cyrusdb.c:218 #11 0x120007b38 in __start(0x3ffc008a2b0, 0x0, 0x57, 0x0, 0x0, 0x0) in ctl_cyrusdb The error produced is DBERROR db4: unable to join the environment. It looks like it tries to initialize the database by opening /var/imap/db and never comes back... the process is aborted somewhere in the call. Scott --On Friday, July 25, 2003 1:28 AM -0400 Scott Adkins [EMAIL PROTECTED] wrote: I have tried everything to get DB stuff working... I just don't know what is going on. Once again, I am on a Tru64 5.1a cluster, was using BerkDB 4.0.14, then switched to 4.1.25. This is with Cyrus IMAP 2.2.1-beta. Basically, I get the following errors (just the ctl_cyrusdb lines): DBERROR db4: unable to join the environment DBERROR: dbenv-open '/var/imap/db' failed: Operation would block DBERROR: init() on berkeley DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such file or directory checkpointing cyrus databases archiving database file: /var/imap/annotations.db archiving database file: /var/imap/mailboxes.db done checkpointing cyrus databases If I run the ctl_cyrusdb command by hand (basically, logged in as cyrus and run 'ctl_cyrusdb -r'), I eventually get the following error: 1434184:./ctl_cyrusdb: /sbin/loader: Fatal Error: call to unresolved symbol from /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so (pc=0x3016418) Running ldd against ctl_cyrusdb gives me the following: Main = ctl_cyrusdb libsasl2.so = /usr/local/sasl-2.1.15/lib/libsasl2.so libdb-4.1.so = /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so libssl.so = /usr/local/ssl/lib/libssl.so libcrypto.so = /usr/local/ssl/lib/libcrypto.so libc.so = /usr/shlib/libc.so As I mentioned above, I was using BerkDB 4.0.14, and decided to try out 4.1.25 to see if anything changed... it didn't. S... I recompiled Cyrus to with the following configuration: export CFLAGS=-g2 ./configure --with-sasl=/usr/local/sasl-2.1.15 \ --with-dbdir=/usr/local/BerkeleyDB.4.1 \ --with-ucdsnmp=/usr/local/net-snmp-4.2.1 \ --with-openssl=/usr/local/ssl \ --with-egd-socket=/var/run/egd-pool \ --with-perl=/usr/local/bin/perl \ --enable-listext --with-lock=flock \ --with-duplicate-db=skiplist \ --with-tls-db=skiplist \ --with-pts-db=skiplist Basically, nothing uses berkeley or berkely_nosync anymore, choosing skiplist as the replacement (I am just trying to get something to work right now, not worry about performance). I thought that maybe I could compile without linking in BerkDB at all, but configure won't let me do it... if I comment out the lines in configure, then I get errors at compile time because it tries to compile cyrusdb_berkeley.c, which requires the libs in order to get compiled... even if it supposedly isn't being used. Okay, so I got it compiled and freshly installed. Running ldd against the binaries shows the same output as above. When I start the Cyrus server up, I get the same errors!!! I thought I disabled BerkeleyDB by specifying skiplist for everything? So, why am I still getting the db4: unable to join the environment errors and the follow-up errors regarding /var/imap/db? Has anyone else seen this problem? I know that the last person who posted something about this problem (back in March) was also running on Tru64 and never did get a response. The next thing I can try is to use BerkelyDB 3.x and see what happens. It obviously works with Cyrus IMAP 2.0.16, so... After that, I will be hacking the BerkDB stuff *out* of the code and solve the problem that way... Scott
Re: Tru64 and 2.2 problems
Even though your suggestions were good, they weren't applicable to this
particular issue. In fact, running the find command to find all the
source files with indented directives only showed up a couple lines as
follows (using a slightly different command than you supplied):
% find . -name '*.[ch]' | xargs egrep '^[: :]+#'
./sieve/addr-lex.c: #pragma warn -rch
./sieve/addr-lex.c: #pragma warn -use
./sieve/sieve-lex.c: #pragma warn -rch
./sieve/sieve-lex.c: #pragma warn -use
Even though it has no bearing on the problem I was having, it is worth
fixing those few lines (by removing the single space before the #pragma)
just to increase the portability of the source code.
Scott
--On Wednesday, July 23, 2003 7:24 AM +0200 Nikola Milutinovic
[EMAIL PROTECTED] wrote:
First thing first, to quote your message. Have you cleaned up the
source from indented CPP directives?
GCC (and possibly other C compilers) support indentation of CPP
directives (#define, #include, etc.). CC doesn't deal with this and
ignores such lines. There is no public standard that allows it - the most
that is allowed is to indent the directive, but to keep # in column 1.
So, this is what GNU allows:
# ifdef HAS_GETADDRINF
#define OK
#include sys/socket.h
# else
#include sys/sock5.h
# endif
DEC CC would go no further than this:
# ifdef HAS_GETADDRINF
#define OK
#include sys/socket.h
# else
#include sys/sock5.h
# endif
So, the first thing to do is clean up all *.c, *.h and configure*
files that you can find. I use a script:
find . \( -name configure* -o -name *.c -o -name *.h \) -exec grep
-E ^( |\t)( |\t)*#(i|d|e|p|u) {} \; -print
(I have placed \t for TABs, in my script they are really TABs - cant
display TABs in MS Outlook Express).
Then look at what's reported and edit all those files. Or write another
script to do the change - sed comes to mind. Or Perl.
When you have cleaned up, then you can start tracking errors and bugs.
Nix.
--
+---+
Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/
UNIX Systems Engineer mailto:[EMAIL PROTECTED]
ICQ 7626282 Work (740)593-9478 Fax (740)593-1944
+---+
PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/
pgp0.pgp
Description: PGP signature
Re: Tru64 and 2.2 problems - cast-to-union
--On Wednesday, July 23, 2003 11:23 AM +0100 Christos Soulios [EMAIL PROTECTED] wrote: I also compiled cyrus-imapd-2.2.1-BETA on a Solaris 9 box with Forte C compiler. I had some of the same problems, which come from the fact that cyrus is mostly gcc compiler oriented. [text deleted about (union cyrus_config_value) type cast stuff] This is because there is a cast to a union. Union casts are not ansi C compliant, but a gcc extension. What I did is to remove the cast to a union type. This makes the syntax ansi c compliant. Right, this is what I did as well... basically, I was left with the int and const char * type casts, which naturally generated warnings. However, I had some warnings for the assignments of integers to a char *. In my case (Sun with Solaris 9) this creates no further problems, because the char *s and the integer members of the union share the same memory. But I am not sure what happens with different architectures that although the union members share the same memory, the byte alignment between the string and the integer may differ. Yes, this is a concern for me... I am not exactly sure how this will be stored in Tru64. We are talking about a char * and an int stored in the same location. I have always known them to consume 4 bytes of memory, but my manager isn't so sure, thinking that maybe it is 8 bytes of memory these days for more efficient storage. The little-endian and big-endian issue may definitely be a problem, and I don't know if that will affect us. The only thing I can really do is maybe write a little test program that does what the Cyrus code was trying to do and see how it is stored. I looked at the command line options of CC to see if I could loosen up the ANSI compliant restrictions a bit and make it work... there is an option called -noansi_alias that does that, but it didn't help. I then found some documentation where it is explicitely stated that Tru64 CC does not support the union-to-cast extension and has no intension of doing so in the future. So, there you have it. Isn't there a more portable way of doing this? Requiring GCC for the compilation of Cyrus is a bit too strict, in my opinion. Rob sent another message out asking me why I just don't use GCC anyways, because it would be simpler than them rewriting the code to make it more portable. I just don't agree with that... in fact, the Tru64 CC compile is highly optimized for the platform, takes advantage of all the pipelining and special ways that the architecture is designed to execute code. Sure, Compaq supplied a bunch of code to GCC, but it is nowhere near what is already in the CC compiler, and on top of that, the engineers there have stated that they are constantly comparing CC to GCC to see how it compares, and CC always blows away GCC with regards to performance. Why switch to GCC and lose any performance gains we would have had just to support a GCC specific extension which is a bit controversial in nature to begin with? I would be in favor of simply modifying the imapopts structure so that we can make it work without portability problems... something I may look at down the road... whether CMU is interested that is another question. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: Tru64 and 2.2 problems - __attribute__ in prot.h
--On Wednesday, July 23, 2003 11:23 AM +0100 Christos Soulios [EMAIL PROTECTED] wrote: 3) After that, it compiles for awhile and then stops on imap/protocol.c with the following error: cc: Error: ./../lib/prot.h, line 209: Missing ;. (nosemi) __attribute__ ((format (printf, 2, 3))); Looking in lib/prot.h, I see the following: extern int prot_printf(struct protstream *, const char *, ...) __attribute__ ((format (printf, 2, 3))); What is that supposed to do? I simply deleted the line that has the __attribute__ on it and put a semicolon on the previous line and it compiles. What problems will I see by doing that? __atribute__ keyword handling is done at the configure script and for compilers that do not support this extension there is a #define statement. However, a '#include config.h' statement is missing from the lib/prot.h file and for this reason it does not build with other than gcc compilers. If you add it, then it is ok. You are absolutely correct, and this indeed fixed the problem elegantly. Rob, can you throw a #include config.h at the top of prot.h in CVS, or do you need a patch given to you? Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: flock vs fnctl
Thanks for the info... I switched over to using flock() and I can confirm that it is now being used instead of fnctl(). The problem is that I still see the same problem as before with regards to over 16500 instances of the following: stat(/var/imap/mailboxes.db, 0x00011FFF9C98) = 0 flock(6, LOCK_UN) = 0 flock(6, LOCK_SH) = 0 fstat(6, 0x00011FFF9D38)= 0 At least it uses flock() now :-) It is interesting to see that this only occurs after all the mkdir/copy/unlink operations have been completed. I don't know what it is trying to do, but it is quite painful... It adds at least another minute to the operation of the IMAP RENAME command after everything has been renamed! Anwyays, Rob requested a backtrace. I am trying to get ladebug working so that I can get a decent backtrace... I am very interested in seeing what part of the code this is occurring in. Scott --On Wednesday, July 23, 2003 4:25 PM -0400 Rob Siemborski [EMAIL PROTECTED] wrote: On Wed, 23 Jul 2003, Lawrence Greenfield wrote: Date: Tue, 22 Jul 2003 17:11:39 -0400 (EDT) From: Rob Siemborski [EMAIL PROTECTED] [...] No, fcntl is not the default. I'll have to look at the documentation. --with-lock=flock should fix this for you though if it makes a difference. Except that it has been since 2.1.7: Change default locking method to fcntl from flock Despite what I said in the message, I apparently changed the documentation to be correct. Hmm, I wonder what I was thinking when I wrote the message. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: Tru64 and 2.2 problems - getaddrinfo
--On Tuesday, July 22, 2003 5:08 PM -0400 Rob Siemborski [EMAIL PROTECTED] wrote: On Tue, 22 Jul 2003, Scott Adkins wrote: With respect to the compile errors, this is what I have found: 1) The configure process appears to pick up on the fact that Tru64 does have getnameinfo(), but not getaddrinfo(). The problem is that if one is found and the other is not, the definitions for both are included in lib/gai.h. Well, getnameinfo() on Tru64 has a slightly different definition and produces a conflict that the compiler can't resolve. The solution was to modify gai.h and wrap the getaddrinfo() related routines with #ifndef HAVE_GETADDRINFO/#endif macros, and wrap the getnameinfo() with similar HAVE_GETNAMEINFO macros. Please bugzilla this and hopefully we can come up with a better configure system for it. As you are already aware, this has been opened in buzilla and a suggested patch has been provided. I can confirm that the patch works great and I recommend it to be used... getaddrinfo() is now being properly detected in Tru64 and gets used. I still run into a problem with gai.h where getnameinfo() for Tru64 is different than what the prototype in that header file describes... as a result, the compilation process halts with a fatal error. I wrapped the getnameinfo() prototypes with #ifndef HAVE_GETNAMEINFO/#endif, and wrapped the getaddrinfo() and freeaddrinfo() prototypes with a HAVE_GETADDRINFO like block. I will provide that in a patch in the next mailing. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Re: Tru64 and 2.2 problems
--On Thursday, July 24, 2003 2:56 PM -0400 Ken Murchison [EMAIL PROTECTED] wrote: Scott Adkins wrote: Even though your suggestions were good, they weren't applicable to this particular issue. In fact, running the find command to find all the source files with indented directives only showed up a couple lines as follows (using a slightly different command than you supplied): % find . -name '*.[ch]' | xargs egrep '^[: :]+#' ./sieve/addr-lex.c: #pragma warn -rch ./sieve/addr-lex.c: #pragma warn -use ./sieve/sieve-lex.c: #pragma warn -rch ./sieve/sieve-lex.c: #pragma warn -use Even though it has no bearing on the problem I was having, it is worth fixing those few lines (by removing the single space before the #pragma) just to increase the portability of the source code. These files are generated by flex, and I don't think we want to get into massaging its output. Are you saying that the leading space is illegal, or that your compiler just doesn't like it? Well, I looked at those files and thought they would be generated by flex, but when I untarred the distribution, they were already there. Are they recreated on the fly, or are the ones provided in the distribution used? As far as the leading space goes, it works on the Tru64 compiler, but I *have* used compilers where the leading space was not permitted. It is actually a CPP problem, not a compiler problem. With CPP's that don't like it, the compile will fail and not just issue warnings. Keep in mind that the only directives in the whole source of Cyrus that has leading spaces are the 4 lines in the above 2 files. The fixes are purely for portability reasons. If you want indentation, then you put the spaces between the # and the pragma, as long as the # is the first character of the line. Anyways, obviously minor. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
DBERRORs: unable to join the environment
Okay, on to the next set of problems. I am getting a whole slew of DBERRORS when the server starts up. The following is a snippet from the syslog when the Cyrus server is first started: master[1334637]: process started master[1334878]: about to exec /usr/cyrus/bin/ctl_cyrusdb ctl_cyrusdb[1334878]: DBERROR db4: unable to join the environment master[1334637]: process 1334878 exited, status 1 master[1334637]: ready for work master[1334358]: about to exec /usr/cyrus/bin/ctl_cyrusdb ctl_cyrusdb[1334358]: DBERROR db4: unable to join the environment ctl_cyrusdb[1334358]: DBERROR: dbenv-open '/var/imap/db' failed: Operation would block ctl_cyrusdb[1334358]: DBERROR: init() on berkeley ctl_cyrusdb[1334358]: DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such file or directory ctl_cyrusdb[1334358]: checkpointing cyrus databases ctl_cyrusdb[1334358]: DBERROR db4: txn_checkpoint interface requires an environment configured for the transaction subsystem ctl_cyrusdb[1334358]: DBERROR: couldn't checkpoint: Invalid argument ctl_cyrusdb[1334358]: DBERROR: sync /var/imap/db: cyrusdb error ctl_cyrusdb[1334358]: DBERROR db4: DB_ENV-log_archive interface requires an environment configured for the logging subsystem ctl_cyrusdb[1334358]: DBERROR: error listing log files: Invalid argument ctl_cyrusdb[1334358]: DBERROR: archive /var/imap/db: cyrusdb error ctl_cyrusdb[1334358]: DBERROR db4: txn_checkpoint interface requires an environment configured for the transaction subsystem ctl_cyrusdb[1334358]: DBERROR: couldn't checkpoint: Invalid argument ctl_cyrusdb[1334358]: DBERROR: sync /var/imap/db: cyrusdb error ctl_cyrusdb[1334358]: DBERROR db4: DB_ENV-log_archive interface requires an environment configured for the logging subsystem ctl_cyrusdb[1334358]: DBERROR: error listing log files: Invalid argument ctl_cyrusdb[1334358]: DBERROR: archive /var/imap/db: cyrusdb error ctl_cyrusdb[1334358]: archiving database file: /var/imap/annotations.db ctl_cyrusdb[1334358]: archiving database file: /var/imap/mailboxes.db ctl_cyrusdb[1334358]: done checkpointing cyrus databases The directory structure looks pretty much like the following: drwxr-x--- cyrus mail /var/imap drwxr-xr-x cyrus mail /var/imap/proc drwxr-xr-x cyrus mail /var/imap/db -rw--- cyrus mail /var/imap/db/__db.001 drwxr-xr-x cyrus mail /var/imap/log -rw--- cyrus mail /var/imap/mailboxes.db drwxr-xr-x cyrus mail /var/imap/msg -rw-r--r-- cyrus mail /var/imap/msg/save.shutdown drwxr-xr-x cyrus mail /var/imap/socket srwxrwxrwx root mail /var/imap/socket/lmtp -rw--- cyrus mail /var/imap/socket/imap-0.lock drwxr-xr-x cyrus mail /var/imap/user drwxr-xr-x cyrus mail /var/imap/user/a drwxr-xr-x cyrus mail /var/imap/user/... drwxr-xr-x cyrus mail /var/imap/user/z drwxr-xr-x cyrus mail /var/imap/quota drwxr-xr-x cyrus mail /var/imap/quota/a drwxr-xr-x cyrus mail /var/imap/quota/... drwxr-xr-x cyrus mail /var/imap/quota/z drwxr-xr-x cyrus mail /var/imap/ptclient -rw--- cyrus mail /var/imap/annotations.db Any ideas? I have been scouring the archives, and so far, I see that two people have posted messages about the above problem (one last November and one this past March), and there were no responses. I am using Cyrus 2.2.1-beta with BerkeleyDB 4.0.14 on a Tru64 5.1a cluster. Any help would be appreciated! Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
I just don't understand...
I have tried everything to get DB stuff working... I just don't know what is going on. Once again, I am on a Tru64 5.1a cluster, was using BerkDB 4.0.14, then switched to 4.1.25. This is with Cyrus IMAP 2.2.1-beta. Basically, I get the following errors (just the ctl_cyrusdb lines): DBERROR db4: unable to join the environment DBERROR: dbenv-open '/var/imap/db' failed: Operation would block DBERROR: init() on berkeley DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such file or directory checkpointing cyrus databases archiving database file: /var/imap/annotations.db archiving database file: /var/imap/mailboxes.db done checkpointing cyrus databases If I run the ctl_cyrusdb command by hand (basically, logged in as cyrus and run 'ctl_cyrusdb -r'), I eventually get the following error: 1434184:./ctl_cyrusdb: /sbin/loader: Fatal Error: call to unresolved symbol from /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so (pc=0x3016418) Running ldd against ctl_cyrusdb gives me the following: Main = ctl_cyrusdb libsasl2.so = /usr/local/sasl-2.1.15/lib/libsasl2.so libdb-4.1.so = /usr/local/BerkeleyDB.4.1/lib/libdb-4.1.so libssl.so = /usr/local/ssl/lib/libssl.so libcrypto.so = /usr/local/ssl/lib/libcrypto.so libc.so = /usr/shlib/libc.so As I mentioned above, I was using BerkDB 4.0.14, and decided to try out 4.1.25 to see if anything changed... it didn't. S... I recompiled Cyrus to with the following configuration: export CFLAGS=-g2 ./configure --with-sasl=/usr/local/sasl-2.1.15 \ --with-dbdir=/usr/local/BerkeleyDB.4.1 \ --with-ucdsnmp=/usr/local/net-snmp-4.2.1 \ --with-openssl=/usr/local/ssl \ --with-egd-socket=/var/run/egd-pool \ --with-perl=/usr/local/bin/perl \ --enable-listext --with-lock=flock \ --with-duplicate-db=skiplist \ --with-tls-db=skiplist \ --with-pts-db=skiplist Basically, nothing uses berkeley or berkely_nosync anymore, choosing skiplist as the replacement (I am just trying to get something to work right now, not worry about performance). I thought that maybe I could compile without linking in BerkDB at all, but configure won't let me do it... if I comment out the lines in configure, then I get errors at compile time because it tries to compile cyrusdb_berkeley.c, which requires the libs in order to get compiled... even if it supposedly isn't being used. Okay, so I got it compiled and freshly installed. Running ldd against the binaries shows the same output as above. When I start the Cyrus server up, I get the same errors!!! I thought I disabled BerkeleyDB by specifying skiplist for everything? So, why am I still getting the db4: unable to join the environment errors and the follow-up errors regarding /var/imap/db? Has anyone else seen this problem? I know that the last person who posted something about this problem (back in March) was also running on Tru64 and never did get a response. The next thing I can try is to use BerkelyDB 3.x and see what happens. It obviously works with Cyrus IMAP 2.0.16, so... After that, I will be hacking the BerkDB stuff *out* of the code and solve the problem that way... Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Tru64 and 2.2 problems
Okay, I have enough problems now to post something.
We are running on a Tru64 5.1a cluster (2 machines). Compiling 2.2.1 has
not been a pleasant prospect. We are using Tru64's cc compiler, and are
not permitted to use the gcc compiler.
First thing is first, we are using SASL-2.15.1 and BerkeleyDB 4.0.14. The
configure process is ran with the following code in a script:
SSL=/usr/local/ssl
SASL=/usr/local/sasl-2.1.15
BERK=/usr/local/BerkeleyDB.4.0
SNMP=/usr/local/net-snmp-4.2.1
CFLAGS=-I$SSL/include -I$SASL/include -I$BERK/include -I$SNMP/include
-gall
RPATH=-Wl,-rpath,$SSL/lib:$SASL/lib:$BERK/lib:$SNMP/lib
LDFLAGS=-L$SSL/lib -L$SASL/lib -L$BERK/lib -L$SNMP/lib $RPATH
export CFLAGS LDFLAGS
./configure --with-sasl=/usr/local/sasl-2.1.15 \
--with-dbdir=/usr/local/BerkeleyDB.4.0 \
--with-ucdsnmp=/usr/local/net-snmp-4.2.1 \
--with-openssl=/usr/local/ssl \
--with-egd-socket=/var/run/egd-pool \
--with-pidfile --enable-listext \
21 | tee configure.output
With respect to the compile errors, this is what I have found:
1) The configure process appears to pick up on the fact that Tru64 does
have getnameinfo(), but not getaddrinfo(). The problem is that if one
is found and the other is not, the definitions for both are included
in lib/gai.h. Well, getnameinfo() on Tru64 has a slightly different
definition and produces a conflict that the compiler can't resolve.
The solution was to modify gai.h and wrap the getaddrinfo() related
routines with #ifndef HAVE_GETADDRINFO/#endif macros, and wrap the
getnameinfo() with similar HAVE_GETNAMEINFO macros.
2) So, I get past the above, and run right smack into another problem,
this time with lib/imapopts.c. I get over a 100 lines of errors along
the lines of the following:
{IMAPOPT_ADMINS,admins,0,(union config_value)((const char
*)),OPT_STRING},
---^
{IMAPOPT_ALLOWALLSUBSCRIBE,allowallsubscribe,0,(union
config_value)((int)0),OPT_SWITCH},
-^
{IMAPOPT_ALLOWNEWNEWS,allownewnews,0,(union
config_value)((int)0),OPT_SWITCH},
---^
Basically, ever line in that table generates an error on the union
construct. I don't understand the error message, been even more so, I
don't understand why the union is even there at all. It appears that
this file is automatically generated by the following command:
../tools/config2header imapopts.c imapopts.h imapoptions
The config2header script talks about playing an interesting game to get
the union to initialize itself in a syntacticly valid manner, namely to
initialize the union itself and not the members of the union, as well as
to ensure that the union is initialized to something of a type that is
in that union. What do I say about that? Very bizarre, and it doesn't
work in Tru64.
Doing a mass delete of all the '(union config_value)' stuff in the file
gets the code to compile, but I don't know what kind of problems that
is
going to cause me if it has to do with initializing things.
Any thoughts on how to handle this problem?
I get similar errors when compiling libcyr_cfg.c as well, but in a
slightly different context, and doing the same deletion gets it to at
least compile.
3) After that, it compiles for awhile and then stops on imap/protocol.c
with the following error:
cc: Error: ./../lib/prot.h, line 209: Missing ;. (nosemi)
__attribute__ ((format (printf, 2, 3)));
Looking in lib/prot.h, I see the following:
extern int prot_printf(struct protstream *, const char *, ...)
__attribute__ ((format (printf, 2, 3)));
What is that supposed to do? I simply deleted the line that has the
__attribute__ on it and put a semicolon on the previous line and it
compiles. What problems will I see by doing that?
4) Okay, everything compiled. However, I saw a lot of these errors when
linking the final executables:
Unrecognized argument in LIBS ignored:
'-Wl,-rpath,/usr/local/BerkeleyDB.4.0/lib'
Unrecognized argument in LIBS ignored:
'-Wl,-rpath,/usr/local/sasl-2.1.15/lib'
Unrecognized argument in LIBS ignored: '-Wl,-rpath,/usr/local/ssl/lib'
In fact, if I do an 'ldd imap/imapd', I get the following error:
imap/imapd: ldd: Fatal Error: Cannot generate dynamic dependencies for
library libsasl2.so
Of course, the executables won't even run at this point. I can set the
LD_LIBRARY_PATH info by hand and get them to work, but that isn't ideal
and I shouldn't have to with the current technology of using rpath. I
don't know how it works in Linux or other systems, but in Tru64, the use
of multiple rpaths are *not* additive... the last one provided on the
command line wins... I specified my own in the LDFLAGS variable before
compiling, but apparently, the configure process added a whole
flock vs fnctl
In version 2.0.16, flock() was being used for file locking. However, in 2.2.1, I am wondering if this is still the case. I looked in the configure output of 2.0.16 and it detects the flock() function call, but in the output of 2.2.1, it doesn't even look like it checks for flock(). Has the default changed or the behavior for locking changed between those versions? The docs seem to indicate that flock() is still the default, if it is detected. We noticed the change in behavior when we went from flat file format in 2.0.16 to skiplist format in 2.2.1 for the mailboxes.db file. We were playing around with renaming a user account with a lot of files and see that it takes more than twice as long to rename the user account (from one partition to another partition, same account name) using the skiplist database. Trussing the process, we see that it does about the same time period for the actual traversing of the user directory trees, copying files from one partition to another, deleting the old files off, etc etc. However, after the imap server finishes with all the file/directory related stuff for that user rename, it then procedes to work on the mailboxes.db file. It goes into this nearly endless loop where most of the time is spent doing the following: stat(/var/imap/mailboxes.db, 0x00011FFF9C88) = 0 fcntl(7, F_SETLKW, 0x00011FFF9D98) = 0 fcntl(7, F_SETLKW, 0x00011FFF9C40) = 0 (7 is the file descriptor associated with mailboxes.db currently opened for read/write operations) The user had 6 directories (5 folders directly under the INBOX folder) and 704 files total (686 messages and 18 cyrus header/index/cache files). Looking through the truss output, there were 1486 open operations that involved the user account itself, generally, one for the file in the old partition, and one for the file in the new partition. There were 39 mkdir operations (6 of which actually succeeded... the other 33 were on directories that already existed). That all seems reasonable. The problem is when it gets to mailboxes.db with the above 3 lines, in which case it does those 3 calls consecutively 16,521 times!!! I have no idea what it is doing, and that is what I will be figuring out tomorrow. I don't know if this is related to the flock() (or the lack of flock() being called) or not. If anyone has any ideas, I would appreciate it. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Multiple master processes?
With the lock problems we had with LMTP in 2.0.16, we are thinking about maybe running two master processes, one configured just for LMTP and the other for all the other stuff. We would gain the added benefit that we could actually run the LMTP master process on one cluster member of our Tru64 5.1a cluster and the rest of the stuff on the other member. I am curious about the experiences of others out there that might have run more than one cyrus master on the same server for whatever reason... Are there any issues with interacting with the databases? My guess is that as locking is working properly, there should be any issues at all. I just needed to confirm it with others first before we look at putting this philosophy into production. By the way, with regards to the locking issues: 1) Yes, I have the lock patch that has been posted applied, so we are less likely to see it. 2) No, we aren't going to limit the LMTP processes to some ridiculously low number so that there is a 1:1 relationship of LMTP processes to IMAP partitions. I/O performance is a bit more complex than that... Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
RENAMES while users is logged in...
We are going to restruture our IMAP stores across more partitions (meaning, more disks) to improve our disk I/O performance and to improve our ability to get these things backed up. The two ways that I am aware of doing this is as follows: 1) issue RENAME commands at the IMAP protocol level, renaming a user from one partition to another 2) moving the IMAP directories at the UNIX level and modifying the mailboxes.db file manually to reflect the changed location We are looking at doing the first one. Currently, we are doing this on the 2.0.16 version with the mailboxes.db file in the flat file format. So, what happens if the user is logged in? I definitely saw the warning in the docs about avoiding logged in users for rename commands. We aren't able to shutdown the server to do the work, so I am not exactly sure how we determine if a user is already logged in or how to disable a user from logging in while the process works. Would doing a reconstruct on the user account as the very last thing help solve some of the inconsistencies that might develop with the server and client getting out of sync? Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
RESEND: Upgrade from 2.0.16 to 2.2a
[NOTE: I sent this yesterday, but the message appears to have not made it to the list for some reason... I hope this doesn't result in copies of the message being posted. Anyways, right after sending this, that was when version 2.2.1 was released into Beta... the question applies to any 2.2 version of Cyrus. ] I was just curious if anyone out there has upgraded from a 2.0.x version to the 2.2a version and what issues I should be looking out for when I do that. The 2.0.16 version we are running with has no duplicate delivery and uses flat files for mailboxes.db, subscribe and seen files. When we move to 2.2a, we will be using the recommended setup, which means skiplist for mailboxes.db and seen files, db3_nosync for dupdb and tlsdb, and flat for subscription files. We will be using BerkeleyDB 4.0.14 and SASL2 2.1.15 (just released). So, any issues I should be watching out for? Also, when we upgraded from the version 1 of Cyrus to the 2.0 series of Cyrus, the upgrade path auto-upgraded the format of some of the files. I am concerned about the seen files, as we have a lot of them. Do I have to convert these files myself to skiplist format? Finally, anybody out there running 2.2a in production? Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ pgp0.pgp Description: PGP signature
Cyrus file locking issues
We are running into issues on our system where IMAP and LMTP processes become stuck and start to stack up and never go away. The IMAP processes aren't so bad, but LMTP definitely is... Investigating the problem with truss and lsof shows that all of the *stuck* processes (the ones that seem to be old and not doing anything but sleeping) are all blocked on a single user waiting for a lock to become free. Usually, it is the cyrus.header file that the processes are waiting on. If I move the file out of the way and reconstruct the user's imap folder, some new LMTP processes pop up to deliver mail, but get blocked on a file lock for cyrus.index. I managed to get one user working by moving it out of the way as well and doing another reconstruct. All the other times, that doesn't work either... I still can't get mail delivered to that user's INBOX. I have tried various things. Using lsof, I can find out which process currently has the write lock on the file. Killing it (with -TERM) gets rid of the process, but the write lock moves to another process and gets stuck there... Killing all the IMAP processes dealing with that file lock doesn't help, as the write lock finally moves to one of the LMTP processes and gets stuck. Ultimately, I have to shutdown the Cyrus server, and when I do that, the locks are freed and things are okay. When I bring the Cyrus server back up again, everything is happy and that user starts receiving email again. Each time I have seen this problem pop up, it has been a different user. We also have 5 different partitions defined, all to different file domains (basically, different disks) and I have seen this problem occur on several different partitions at this point, so I don't believe a particular disk is going bad or anything like that. This is a Tru64 5.1 Cluster running on Compaq Alpha hardware. The Cyrus server is 2.0.16. The duplicate delivery database has not been compiled in (we had all kinds of locking problems with BerkeleyDB), so this means that LMTP is not linked with BerkeleyDB or Pthreads (yay!). We also use flat file format for the mailboxes.db, which also means that the rest of the cyrus processes are not linked with BerkeleyDB or Pthreads as well. The reason I mention that is that when trussing one of the stuck procs, it looks like it is stuck within a pthread type lock. I know the process isn't threaded, so this indicates that the lock is within the kernel, which makes sence with regards to flock() and it works (I believe). Finally, before anyone says anything, yes, I know we should be running 2.0.17 or higher, but we can't make that kind of upgrade with all the changes made to 2.0.17 and the custom changes we made to 2.0.16 during the school year without careful testing and careful planning. We have a very *very* busy email system, and it is the main email system for the whole university, so the less we do to it, the better. We will upgrade to the latest greatest version when summer gets here, which seems to get here pretty quick anyways... So, has anyone else seen this problem? If this was a recognized problem and has since been fixed, which version of the Cyrus server was this finally fixed in? Thanks! Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg11190/pgp0.pgp Description: PGP signature
Re: imapd's hang when maxchild count is reached
I solved this problem a long time ago by passing an environment variable from the master process to the child process when the child process is spawned indicating that the server is full. I used CYRUS_MAXCHILD, and the child process already checks for the CYRUS_VERBOSE variable when it starts in order to properly set the debugging level. If that variable was set, then the child would output an error message indicating that the server was full and to try again later. It would then close the client connection and then exit. A couple things to note: 1) I prefer the client to be notified when the server is full and not simply get connection refused messages or have the email client just sit there, appearing to hang, while the server waits for a connection to become free. 2) My method worked, but it didn't take advantage of the process reuse feature. Basically, the master process only gets one chance to pass an environment variable off to the child process. So, once that variable is set in the child, the child will always believe that the max has been reached. That is why I had the child process go away, as it is basically useless after handling that one connection. 3) If there was a good way for the master to notify the child process on each connection pass (either when passing the connection to an already available child, or when passing the connectioin to a newly spawn child) what the current status of maxchild is, then it would be quite efficient to send the server full messages, close the connection and wait for the master process to hand it another. I don't know enough about how the master and child process communicate to know how to make this work. Scott --On Wednesday, February 05, 2003 12:57 PM -0500 Lawrence Greenfield [EMAIL PROTECTED] wrote: Date: Wed, 05 Feb 2003 18:51:35 +0100 From: Sebastian Hagedorn [EMAIL PROTECTED] [...] Wouldn't it be possible (and better) to refuse further connections instead of having to wait for them to time out? Maybe I haven't thought this through properly, but it seems to me as if that were cleaner. Yes, that would be desirable. The easiest way of doing this would be to close the socket used to accept() new connections. However, it's open in all of the children, so closing it is infeasible. The next option would be to have master accept() and then immediately close() the connection. This raises interesting blocking concerns and is thus somewhat harder to implement. Thus the current solution seems acceptable. Larry -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg10910/pgp0.pgp Description: PGP signature
Sendmail + LMTP AUTH
Okay, I must be clueless here... I have looked through the docs, looked through the archives, done google searches, etc. I just plain don't understand how to configure sendmail to do LMTP authentication correctly. I am in need to open up the TCP port of LMTP to more than just [localhost] and believe that LMTP AUTH is the way to do it. We do not have tcpwrapper support compiled in, and really need to find a way to do it without having to recompile lmtpd. In the cyrus config file, I have the following config lines: # LMTP is required for delivery #lmtp cmd=lmtpd -a listen=localhost:lmtp lmtp cmd=lmtpd listen=lmtp lmtpunix cmd=lmtpd listen=/var/imap/socket/lmtp We were running with the -a option forced to localhost. I removed it and removed the localhost reference. The lmtpunix line is there in case anyone uses the deliver program to deliver a message. As far as the sendmail configuration goes, I have the following: dnl server configuration for SMTP AUTH define(`confAUTH_MECHANISMS', `PLAIN')dnl TRUST_AUTH_MECH(`PLAIN') define(`confAUTH_OPTIONS', `p,y')dnl dnl server configuration for SMTPS DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s') dnl cyrus configuration define(`confLOCAL_MAILER', `cyrus') MAILER(`local') MAILER(`smtp') MAILER_DEFINITIONS Mcyrus, P=[IPC], F=lsDFMnqAwW@/:|SmXz, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, U=cyrus:mail, A=TCP [localhost] lmtp Note: This is obviously not the full file, but hopefully just the parts relevant to the discussion. Now, what do I need to change in order to get LMTP AUTH working? I do have some questions regarding how LMTP AUTH actually occurs... First, what username/password gets used for the authentication? Is it going to be postman, like it is for when lmtpd is ran with the -a option? I can only guess that this might be the case, since delivery of email is coming from an anonymous source and not a physical user on the system. If this is the case, then what about the password? Is it stored in the config file, or cached or what? Secondly, I assume this occurs in the plain-text format, so, should I be setting up LMTP to be done over SSL? Will sendmail use SSL to LMTP correctly? If this is the case, how do I change the above setup to make SSL LMTP work? By the way, please note that I have allowplaintext: 1 in the imapd.conf file, so authenticating over a plain-text connection for IMAP and POP is not allowed... they have to authenticate over an SSL/TLS connection first. Since this is set, won't this affect LMTP authentication as well? Okay, lots of questions... I hope I can get some answers... I feel that I am pouring more time and energy into this problem than I need, and really need to turn my attention elsewhere... Thanks! Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg10668/pgp0.pgp Description: PGP signature
Re: SNMP cyrus monitoring.
--On Thursday, January 23, 2003 5:58 PM +0600 Dmitry Novosjolov [EMAIL PROTECTED] wrote: Hi All, has anybody succeded in using SNMP statistics of cyrus IMAP server ? If so, can you please point me in right direction of how to monitor the activity of Cyrus-imapd-2.1.11 server? I've heard about togowar, but cyrus documents are empty in this chapter ... -- Best regards, Novosjolov Dmitry I actually wrote a couple scripts that monitor the server. One script is just meant to be called from the command line (snmp_query) and displays the results in a clean easy to understand format. The sample output is as follows: Cyrus IMAP Server v2.0.16 Thu Jan 23 09:10:55 EST 2003 Up 5 days, 0:36:29 Services Forks Running Maximum --- -- -- -- imap 204631 48 pop3 11582 11 imaps14181 16722127 pop3s392223 72 imaps_silky 355222 62 lmtp 135032 113 lmtpunix 150 0 1 === == == == Total26359 17822434 NOTES --- Forks = Total number of forks since server was started. Running = Total number of processes currently running. Maximum = Maximum number of processes running concurrently. The other script (check_cyrus) is for logging and I use a cron job that runs it periodically and appends the output to a log file. The output of that script looks like the following (all on one line, though): 01/23 09:11 5 days, 0:37:08 imap=31/48 pop3=2/11 imaps=1672/2127 pop3s=23/72 imaps_silky=22/62 lmtp=32/113 lmtpunix=0/1 The above data should be easy enough to parse so that you can push it through a grapher, such as gnuplot or maybe even excel. I will attach both scripts. Maybe if the CMU folks like it, they can put it in the contrib directory? Oh, I actually just called the snmpwalk program directly and didn't use the SNMP perl module. Maybe somebody else can retrofit the script to do that. Anyawys, it should require just minor tweaking to get it to work on your system. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ snmp_query.pl Description: Binary data check_cyrus.pl Description: Binary data MSL10918.sig Description: PGP signature
Re: SNMP cyrus monitoring.
Taking a stab in the dark, do you have an SNMP server running on the machine? If you have the snmp tools installed, then check chkconfig with the following command to see if you got the service configured to start at boot time. Before you turn on the snmp service via the /etc/init.d scripts, make sure that you have it configured properly (I am thinking in terms of access control). Anyways, that is my stab in the dark guess ;) Oh, you will have to restart your cyrus server after you start the snmp server, that way, cyrus will initialize properly with the snmp server and start sending it stats. Scott --On Thursday, January 23, 2003 12:36 PM -0500 Jared Watkins [EMAIL PROTECTED] wrote: I don't know a lot about the innards of snmp... but here is my problem. I configured cyrus with the --with-ucdsnmp flag.. but when I query with snmpwalk it gives no result.. as if that part of the tree is not known to snmp. I ran the query using the base OID from your scripts. I'm doing this on my test RH 7.3 box with the rpm packages of snmp. What else has to be done to get this working for cyrus? Jared Scott Adkins wrote: --On Thursday, January 23, 2003 5:58 PM +0600 Dmitry Novosjolov [EMAIL PROTECTED] wrote: Hi All, has anybody succeded in using SNMP statistics of cyrus IMAP server ? If so, can you please point me in right direction of how to monitor the activity of Cyrus-imapd-2.1.11 server? I've heard about togowar, but cyrus documents are empty in this chapter ... -- Best regards, Novosjolov Dmitry I actually wrote a couple scripts that monitor the server. One script is just meant to be called from the command line (snmp_query) and displays the results in a clean easy to understand format. The sample output is as follows: Cyrus IMAP Server v2.0.16 Thu Jan 23 09:10:55 EST 2003 Up 5 days, 0:36:29 Services Forks Running Maximum --- -- -- -- imap 204631 48 pop3 11582 11 imaps14181 16722127 pop3s392223 72 imaps_silky 355222 62 lmtp 135032 113 lmtpunix 150 0 1 === == == == Total26359 17822434 NOTES --- Forks = Total number of forks since server was started. Running = Total number of processes currently running. Maximum = Maximum number of processes running concurrently. The other script (check_cyrus) is for logging and I use a cron job that runs it periodically and appends the output to a log file. The output of that script looks like the following (all on one line, though): 01/23 09:11 5 days, 0:37:08 imap=31/48 pop3=2/11 imaps=1672/2127 pop3s=23/72 imaps_silky=22/62 lmtp=32/113 lmtpunix=0/1 The above data should be easy enough to parse so that you can push it through a grapher, such as gnuplot or maybe even excel. I will attach both scripts. Maybe if the CMU folks like it, they can put it in the contrib directory? Oh, I actually just called the snmpwalk program directly and didn't use the SNMP perl module. Maybe somebody else can retrofit the script to do that. Anyawys, it should require just minor tweaking to get it to work on your system. Scott -- -BEGIN GEEK CODE BLOCK- Version: 3.12 GIT/S/B d- s-:+ a- C$ UL$ P--- L+++$ E--- W+++ N++ o+ K- w O- M-- !V PS+ PE Y++ PGP++ t+ 5- X+ R* tv+ b++ DI+ D G e+ h+ r+++ z* --END GEEK CODE BLOCK-- -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg10614/pgp0.pgp Description: PGP signature
Received: headers after LMTP delivery
I find it useful to look at the Received headers to track the path the email might have taken to get from the desktop to the mailbox when we are having problems. Particularly, I look at the delays between hops to find out if one of our machines is holding onto mail a lot longer than it should be. I know that the first hop (desktop to the server) may not reflect an accurate time, since it looks like to me that the time on the PC is what is often reflected in the first Received header. The problem I am having is determining when a message physically gets delivered by LMTP to the mailbox. I can see the last Received line in the path, which is one mail server handing the message off to the last mail server which contains the IMAP server (LMTP is running on localhost only). Then after that, I see the Return-Path line in the header, which I know gets added by LMTP when it delivers the message. What is missing is the time that this occurred, as I have no idea how long it sat in the sendmial queue on the server before it was handed to LMTP. Would it be possible to have LMTP add a similar Received line right before it adds the Return-Path line that indicates the time of delivery? I haven't read the RFC's to see if there are any specific rules for how the Received lines would look, but it would be nice to see it indicate that the message was received by LMTP for delivery at a specific time. Thoughts? Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg10486/pgp0.pgp Description: PGP signature
Re: [PATCH][CVS IMAPd 2.1] lmtp_downcase_rcpt implementation (Re:Case Sensitivity)
--On Tuesday, December 24, 2002 12:01 AM -0200 Henrique de Moraes Holschuh [EMAIL PROTECTED] wrote: Here's the cleaned up patch, against 2.1 CVS. It could be enhanced not to touch the +fooobar part of the recipient, I suppose. Yes, and I wanted to add a comment on this... Put the work in now to make it not touch the +foobar part of the recipient, before it is given to the CMU folk for integration into CVS. We have all kinds of +plussed folders here at our site, and many of them contain mixed case characters (usually, the first character gets up-cased). So, I imagine that it would be a common issue elsewhere as well. Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg10058/pgp0.pgp Description: PGP signature
Sendmail/Plussed Folders --WAS-- Re: [PATCH][CVS IMAPd 2.1]lmtp_downcase_rcpt implementation (2)
--On Wednesday, December 25, 2002 10:59 AM -0200 Henrique de Moraes Holschuh [EMAIL PROTECTED] wrote: Here's the improved patch. It skips over the +Mailbox part of the recipient, in the most straightforward way possible ;-) Sorry, didn't see this message until too late... ignore my last post on the subject... you just fixed it :) On another note, it seems that sendmail has issues with talking directly to LMTP (meaning, not using the deliver program) and being able to deliver messages to plussed folders of a user account. At least as of sendmail 8.12.5, messages sent to plussed folders are simply deposited to the user's INBOX. Maybe that has been fixed by mucking around with the M4 code for the Cyrus configuration, but I doubt it. There is a PRESERVE_LOCAL_PLUS_DETAIL option in proto.m4 that can be used, but as I have seen from my testing, it only works when sendmail calls the deliver program directly (deliver -e -m $h -- $u was what we used), and the reasoning for that is that the hostname and the username were broken out into two seperate macros and passed individually on the deliver command line. Using that option with LMTP directly ends up losing the $h portion of the macro, and only $u is sent via the LMTP protocol. S I ended up patching sendmail (I can post the patch, which is current as of 8.12.5, but should work for 8.12.6) that allows plussed folder details to be passed along untouched to the LMTP server, and even implemented a flag that would allow the username to be lowercased and leave the plussed folder portion untouched before passing it to LMTP. So, my question to the list is this: Has anyone else had this problem (plussed folder emails sent to sendmail that is configured to talk to LMTP directly, either my UNIX domain socket or my TCP socket)? If so, how was it resolved? Is there a better way to do this using the M4 macros as opposed to how I solved the problem by patching the code? Thanks, Scott -- +---+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +---+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg10062/pgp0.pgp Description: PGP signature
Re: Cyrus IMAPd 2.1.10 Released
--On Friday, November 15, 2002 9:36 AM -0500 Rob Siemborski [EMAIL PROTECTED] wrote: On 15 Nov 2002, Erik Enge wrote: Then again, LaTex (or SGML) would probably give you the most possibilities (and problems ;). Right, as does HTML (atleast, for our intents and purposes), without the overhead of having to learn a new toolset and language (or the complexity of LaTeX). I guess to an extent we lose the ability to do PDF and PS documentation. Actually, I develop all my documentation in HTML as well. I still produce PDF and PS documents as well. How I acommplish this is that I use IE, go to the documented HTML web page, select a printer and print, select the little checkbox to SAVE TO FILE, and specify the location and name of where to save it. The resulting file is a PS file, and then I use ps2pdf, which I believe is part of the ghostview collection, to convert it to PDF format. It works great for my purposes... Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:adkinss;ohio.edu ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg09250/pgp0.pgp Description: PGP signature
Re: [PATCH] Updated master.c process counting patch
For what its worth, we run into this problem a lot in our environment as well... I had come to the same conclusion as Jeremy and Henrique did before they even posted their message about it to the list. The environment we are running is as follows: Cyrus IMAPD 2.0.16 Compaq Tru64 5.1 on Alpha Platform Cyrus SASL 1.5.27 Sleepycat BerkeleyDB 3.2.9 OpenSSL 0.9.6 UCD-SNMP 4.2.1 lock = flock auth = unix mboxlist.db = flat subs.db = flat seen.db = flat duplicate.db = db3-nosync tls.db = none The problem existed before we enabled SSL, so it isn't SSL related, and the problem existed before we added SNMP to the mix. The only place we use BerkeleyDB is for the duplicate delivery database and only lmtpd is linked against the libdb.so file (I modified the Makefile to not link pthreads or BerkeleyDB into imapd, pop3d, etc, since they are using a flat file database anyways). As for when the problem would arise... well, obviously, when we ran into resource issues, master would lose track of the count of children as they segfaulted, etc. This problem was just recently described. However, we rarely had resource issues like that. The other times that we could catch the problem occuring is when the system is booting and the Cyrus server is first started. For some reason, we would lose a couple of the services within a short period of time after Cyrus was started. We would have to shut down Cyrus and restart it again. Rarely, the problem would occur again within the first few minutes of the restart, and we would have to do it again. Usually, though, the restart would work fine. Finally, we would periodically lose services for no apparent reason, meaning that the machine had not been recently rebooted, and we were not suffering from any apparent resource shortages (either at a system level or at the cyrus user level). In any the case, the problem exists in our Tru64 environment. Scott --On Thursday, May 16, 2002 9:25 AM -0400 Ken Murchison [EMAIL PROTECTED] wrote: I'm running a config almost the same as you and have never seen this problem. AFAIK, the CMU guys have never seen this either. Do you have a core that you can run a backtrace on, or can you force a core by setting MALLOC_CHECK_=2 before starting master (see malloc(3) for details)? What's your DB config look like? Are you using skiplist for everything by any chance? name : Cyrus IMAPD version: v2.1.4 2002/05/14 16:51:51 vendor : Project Cyrus support-url: http://asg.web.cmu.edu/cyrus os : Linux os-version : 2.4.18-SGI_XFS_1.1smp command: imapd arguments : environment: Cyrus SASL 2.1.3 Sleepycat Software: Berkeley DB 3.3.11: (July 12, 2001) OpenSSL 0.9.6b [engine] 9 Jul 2001 CMU Sieve 2.2 TCP Wrappers UCD-SNMP 4.2.3 lock = flock auth = unix idle = idled mboxlist.db = skiplist subs.db = flat seen.db = skiplist duplicate.db = db3-nosync tls.db = db3-nosync -- Kenneth Murchison Oceana Matrix Ltd. Software Engineer 21 Princeton Place 716-662-8973 x26 Orchard Park, NY 14127 --PGP Public Key--http://www.oceana.com/~ken/ksm.pgp -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg07820/pgp0.pgp Description: PGP signature
Re: [PATCH] Updated master.c process counting patch
I am still waiting to hear from Ken and Lawrence on what they think about these patches? Will any or all of them be implented in the next release? Scott --On Wednesday, May 15, 2002 12:41 PM +1000 Jeremy Howard [EMAIL PROTECTED] wrote: Thanks to Jaska Kivela, some patch formatting problems that caused the master.c process counting patch to not apply cleanly have been resolved. The patch set has been updated, and now also incorporates the master.c race condition patch: http://jhoward.fastmail.fm/patches/cyrus/imap-diff.tgz The only file changed in the patch set is master-avail.diff . master-avail.diff solves the problem that master fails to correctly keep count of child processes if processes do not exit cleanly. This manifests itself as Cyrus failing to accept new connections on one or more ports after a while, when using preforking in imapd.conf. -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg07755/pgp0.pgp Description: PGP signature
Re: Master prefork problems; CLOSE_WAIT problems
--On Thursday, May 02, 2002 9:52 AM +1000 Jeremy Howard [EMAIL PROTECTED] wrote: I've seen a couple of problems over the last few weeks with master apparently failing to correctly maintain the prefork pool. We particularly see this problem with pop3d, which has more connects/disconnects than IMAP because of the nature of the protocol. The first issue is that in shut_down() sockets are not closed. It seems that this can leave sockets in CLOSE_WAIT state in certain error situations where popd_reset() is not called. The second issue is that we sometimes see sockets remain in a CLOSE_WAIT state because there is still data to be read. It appears that prot_fill() should be called in popd_reset() and shut_down(). The third issue is that when a process fails to shutdown correctly, such as if it segfaults, master does not seem to correctly keep track of the child process count. As a result, eventually the pool runs out and no more connections are accepted. Do the resolutions to the first two issues sound correct (we have made these changes and it seems to have fixed things for us)? Does anyone have a fix for the third issue? YES! I believe you have hit the nail on the head on all 3 of the issues above! Good job! We have been particularly bitten by the third issue with the master process losing track of the number of child processes in each service maintained. There has to be a better way for the master process to manage its children. I was thinking that it would be nice if the cyrus server used shared memory to keep track of the children, which ones were active or idle, which ones haven't checked in with the master in awhile (possible problem), etc. If the master had a incoming client queue, the children could pick up the next client and run with it. Furthermore, all the client information, such as number of clients each one has handled, etc would be available to the master, which is the only cyrus process that currently has SNMP support. This would make SNMP stats far more useful. Anyways, I wish I had the time right now to dive into the master/child communicatin problem, but I am glad somebody else has seen the problem too! Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg07532/pgp0.pgp Description: PGP signature
Re: Problem with imapd 2.1.3 not closing TCP session
I will post a problem report from what we have experienced on the Compaq Alpha Tru64 system a little later this morning (I am pressed for time at this very second). In any the case, we do experience the lossage of ports from time to time, but I believe I understand why (which is what I will explain). As for the CLOSE_WAIT and FIN stuff, I believe this is an unrelated problem to the above. In fact, I don't believe Cyrus has anything to do with TCP states... the socket has either been closed or it has not been closed. I can't seem to recall exactly, but I do not believe that even doing raw sockets can you affect the state of TCP connections. This is an operating system issue (and thus Linux's issue), and the problem is in the kernel and not the application. Unfortunately, I don't have any answers to this. Of course, I could be wrong, in which case I will probably see a lot of corrections in my mailbox... Scott --On Friday, March 29, 2002 9:44 PM +1100 Jeremy Howard [EMAIL PROTECTED] wrote: Mike Brady wrote: I am observing a problem where imapd occasionally does not close the TCP session properly. This only seems to occur with 2.1.3. ...original details at end... We are also seeing an odd problem with 2.1.3. It may or may not be related to Mike's issue. We are using the skiplist backend for mailboxes and seen state. We have disabled the TLS session cache. We are using Linux 2.4.18 with Ext3. 2 times in the last week our IMAP server has suddenly stopped accepting IMAP connections on port 143 on its ethernet interface. However, it continues to accept IMAP connections on port 993 (SSL) and the localhost interface. Our cyrus.conf lines are: imap cmd=imapd listen=www.fastmail.fm:imap prefork=20 imaplocal cmd=imapd listen=[127.0.0.1]:imap prefork=25 imaps cmd=imapd -s listen=www.fastmail.fm:imaps prefork=2 Restarting Cyrus clears the problem. There are no unusual messages in imapd.conf or log/messages. We have plenty of spare system file descriptors. Both times this has happened has been at the busiest time of the day. However we had plenty of IO and CPU capacity spare at the time. It looks to me like somehow that particular port on that interface got 'filled up'. I'm not a TCP guru so I don't know exactly what that might look like--is there some limit on concurrent connections or a queue of waiting connections? When the lockup occured, telneting to the port simply resulted in nothing at all--it just sat there waiting for 5 minutes... Could Mike's problem report of TCP connections not being closed correctly lead to this kind of lockup? The rest of Mike's message... The details are as follows. My system is RedHat 7.2 with all appropriate errata except the kernel ones. The kernel is 2.4.18 compiled from tarball, but this issue also occurs with the Redhat 2.4.7-10 kernel. I am currently using Outlook 2000 on W2K Pro as my mail client (sorry :-). In addition I have compiled Postfix 1.1.4, Cyrus SASL 2.1.1 and Cyrus IMAP 2.1.3 from their respective tarballs. The server is lightly loaded. This is my home server, I am the only user and it is only doing my e-mail at the moment. For the most part mail works. That is, I can send and receive e-mail with no problems. The symptom of the problem is that Outlook gets stuck while retrieving messages (I can't remember the exact message on the status bar). By stuck I mean that the Outlook UI is still accepting input, but it doesn't do anything. The Outlook UI can be closed, but the Outlook processes does not die. The only way to get rid of it is to use Task Manager to kill it. On the server side netstat shows that the tcp session state is CLOSE_WAIT. I have a tcpdump capture which shows that Outlook (well the tcp stack underneath it) has sent a FIN packet and imapd has acked it, which is why it is in the CLOSE_WAIT state. So far so good. However, to complete the tcp session close imapd should next send a FIN which Outlook should then ack. imapd never sends the FIN. I have waited for hours. The only way to get rid of the imapd process is to kill it, at which point it does send a RST packet. However, by this stage W2K/Outlook is so out to lunch the RST does not do anything. The only way to recover is to kill the Outlook.exe process. I have not been able to determine a way to make this happen. It has occurred within seconds of opening Outlook, but sometimes it takes hours. My initial figuring out Cyrus install/testing was done with 2.1.2 and I did not see this behaviour. 2.1.3 came out while I was doing through my learning phase, so when I did my real install I use that. I went back to 2.1.2 on my live server about 24 hours ago and this problem has not occurred at all. It has been many years since I did any C coding so I haven't tried to look at the source to figure this out. If there is something specific that I can look
Re: cyrus-imapd-2.1.3 w/ BDB4: small typo in log output ofimapd,pop3d?
--On Tuesday, March 12, 2002 8:10 AM +0100 Birger Toedtmann [EMAIL PROTECTED] wrote: I just set up cyrus-imapd-2.1.3 on a RH7.2 box that has BDB4 installed (re- compiled src.rpm). Any ldd's of the imapd/pop3d binaries show correct bindings to /lib/libdb-4.0.so, so everything seems ok - service works nice - despite some log messagess: Mar 12 07:57:50 testbox pop3d[19141]: DBERROR db3: 9 lockers ^^^ Mmmh. As far as I could see, this may not directly lie within cyrus-imapd's code but BDB4s, but I could not find the string db3 in any of them (imapd, db-4.0.so). Ideas? Regards, Birger If it bothers you too much, take a look at lib/cyrusdb_db3.c and search for errpfx. You should see the following line: dbenv-set_errpfx(dbenv, db3); As you can see, if is a BerkeleyDB call that sets the a prefix string to be prepended to the error messages that BerkeleyDB produces. Maybe it should simply be db? At least until other things are implemented, such as dbm, mysql, etc (which I doubt ever will be) :) Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ PGP Public Key available at http://www.cns.ohiou.edu/~sadkins/pgp/ msg06545/pgp0.pgp Description: PGP signature
Re: deliver fails on nonunix lmtp-socket
The problem is that deliver *only* works with unix lmtpd sockets. So, how we dealt with the issue in our environment is run with both lines in the cyrus.conf file, allowing either TCP connections or unix socket connections. Then we configured sendmail to use the TCP connection when delivering the mail to LMTP. If some external application needs to use deliver for any reason, then that will work too, since deliver will use the unix socket. It is too bad that deliver doesn't read the cyrus.conf file to determine what sockets are available and to use what it finds (giving preference to the TCP socket if they are both turned on). Scott --On Thursday, December 13, 2001 2:40 PM +0100 Peter Pilsl [EMAIL PROTECTED] wrote: I try to replace the unix-lmtpd-socket with a 'real' socket, cause I want use lmtpd for other applications too. however, if I do so, deliver wont work anymore: # cat /tmp/m | /usr/cyrus/bin/deliver -a cyrus peter couldn't connect to lmtpd: Invalid argument 421 4.3.0 deliver: couldn't connect to lmtpd # cat /tmp/m | /usr/cyrus/bin/deliver -l -a cyrus peter connect failed: Connection refused 421 4.3.0 deliver: connect failed and imapd.log reveals in both cases: Dec 13 14:29:50 server deliver[24093]: connect(/data/imap/config/socket/lmtp) failed: Connection refused but my cyrus.conf: # at least one LMTP is required for delivery lmtp cmd=lmtpd listen=lmtp prefork=1 # lmtpunix cmd=lmtpd listen=/data/imap/config/socket/lmtp # prefork=0 lmtpd is listening on 2003 and - yes - I've killed all questionable processes. dont know why deliver still tries to access the unix-socket. cyrus.conf should be the only place where this socket is defined ... !!?? thnx, peter -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Eudora, STARTTLS and the alternate port
Okay, we just got bitten by the Eudora 5.x STARTTLS problem that was
discussed last month. We have the same problem where only those clients
cannot negotiate a TLS connection properly, and thus fails to login at
all. So...
Ken suggested removing or commenting out the following lines:
if (tlsonly) {
off |= SSL_OP_NO_SSLv2;
off |= SSL_OP_NO_SSLv3;
}
I am wondering exactly what effect this will have on us... how does this
affect clients that *do* TLS just fine, such as Mulberry, for instance?
Would the other clients still use TLS and Eudora use SSLv3?
For my next question, I am curious if there is a way to turn off the
STARTTLS capability on the main imap port, but still allow the use of
the alternate IMAP SSL port. I don't see this capability in the server,
appearing to be an all or nothing type thing based onthe tls options
listed in the imapd.conf file. Using stunnel to wrap imap on an imaps
port is not really an option here, but I know that is one way to do it.
Thanks,
Scott
--
+-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+
Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/
UNIX Systems Engineer mailto:[EMAIL PROTECTED]
ICQ 7626282 Work (740)593-9478 Fax (740)593-1944
+-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+
CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: Eudora, STARTTLS and the alternate port
--On Friday, October 19, 2001 5:18 PM +0300 Leena Heino [EMAIL PROTECTED]
wrote:
On Fri, 19 Oct 2001, Scott Adkins wrote:
Okay, we just got bitten by the Eudora 5.x STARTTLS problem that was
discussed last month. We have the same problem where only those clients
cannot negotiate a TLS connection properly, and thus fails to login at
all. So...
Ken suggested removing or commenting out the following lines:
if (tlsonly) {
off |= SSL_OP_NO_SSLv2;
off |= SSL_OP_NO_SSLv3;
}
I am wondering exactly what effect this will have on us... how does this
affect clients that *do* TLS just fine, such as Mulberry, for instance?
Would the other clients still use TLS and Eudora use SSLv3?
I've commented out the above lines as Ken suggested and then tried the
STARTTLS with pine4.40 and it seems to work nicely.
Right... but that doesn't exactly answer my question. I know I have a
deep lack of understanding of SSL and how it works. In a nutshell, I
want to know if by commenting those lines out above that the email
clients will end up using a weaker algorithm for encryption that what
would normally be used.
Is TLS a method of choosing an SSL protocol for encryption? Or is it
another method of encrypting like SSLv2 and SSLv3. I guess I need to dig
up all the docs on this stuf and read it... but time is always a problem.
Scott
--
+-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+
Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/
UNIX Systems Engineer mailto:[EMAIL PROTECTED]
ICQ 7626282 Work (740)593-9478 Fax (740)593-1944
+-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+
CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: Memory usage of imapd
--On Friday, October 19, 2001 4:08 PM +0200 Jan Moravec [EMAIL PROTECTED] wrote: On my FreeBSD 4.4 machine running Cyrus imapd 2.0.16 (installed from the ports collection), I see that each imapd process eats up around 2500K of memory (RES - resident portion of the process) and its total size is around 20M. In the FAQ, it says each process should have around 512K or so. I wonder, is the FAQ out-of-date, is this normal or have I misconfigured something? Thanks for your opinions! -- Jan Moravec Actually, we have made the same obvservation here after upgrading to 2.0.16 from 1.5.14. Many of our imap processes climb up to 14MB in size (though, I have never seen one bigger than that) and stay there. One of my guesses as to why that is the case is based on the fact that the imapd processes now stick around for multiple connections. I wonder if the process is not properly freeing up memory after one user disconnects and before the next user disconnects. I also don't know how much of this is due to the way the OS (Tru64) manages process memory. This is on my list of things to do to determine why the resident memory footprint is so large. It is definitely larger than the older version, and with several thousand imap servers running, this is a lot of memory. Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: lmtp sockets and sieve / duplicate suppression
--On Friday, September 07, 2001 11:01 AM -0400 Scott Russell [EMAIL PROTECTED] wrote: On Fri, Sep 07, 2001 at 10:24:59AM -0400, Scott Adkins wrote: --On Friday, September 07, 2001 9:11 AM +0200 Carsten Hoeger [EMAIL PROTECTED] wrote: On Thu, Sep 06, Scott Russell wrote: Question about using lmtp sockets vs using deliver. Does using lmtp sockets on cyrus at all take away my ability to use sieve scripts or duplicate suppression? No, of course not. Maybe just a *little* more description would help :- Anyways, deliver is nothing more than a wrapper that connects to the LMTP socket itself. The deliver program doesn't actually do the delivery anymore. So, the effect is the same... This is what I've read here on the lists myself. What prompted me to ask was the switches that deliver takes. For example deliver takes a switch to turn off duplicate email suppression. Obviously I cannot pass switches to my lmtp socket... So if deliver is nothing but a lmtp wrapper and it's no different than doing direct lmtp delivery why does deliver take switches for delivery control? Also, will the deliver wrapper be dropped from future releases of cyrus imapd? Ah, okay. Actually, doesn't deliver take a switch to turn *on* duplicate email suppression (the -e switch)? Anyways, the -e option is depracated and now actually does nothing. Duplicate delivery suppression is on by default. As far as I can tell, there is no way to turn off the use of duplicate delivery suppresion, unless you modify the source code. As for the removal of the deliver wrapper in a future version, I am not sure, but I hope not. Deliver is useful if you need some means of deliverying a message to a user's mailbox from a shell script or similar program. For example, we had one of our imap partitions corrupt last week. We were able to restore from tape, but there was the time period from when the backup was made and when the system went down that were not backed up. This makes sense when you think of backups being done only at nights. We were able to salvage our disk with a utility that restored most of the files onto some backup partition we had. Using deliver, we were able to re-deliver all the messages between the backup time and crash time, so the users actually didn't lose any mail... it was just severely delayed. In any the case, I think there will always be a good use for deliver. As far as duplicate delivery suppression goes, I think we need a protocol expansion or something that allows it to be done, much like how the ignore quota feature works. I wonder if that would be possible? Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: IMSP address book switch state not preserved from Simeon serverto 1.6a3
--On Thursday, September 06, 2001 10:26 AM -0400 Steve Heist [EMAIL PROTECTED] wrote: Hello all, We are currently testing a move from the Simeon IMSP server to CMU's 1.6a3 and we seem to have a problem where the address book switch values of Use for Nick-names, Open on Startup and Use for Searching are not preserved in the Mulberry 1.4.5 client. Did you say 1.4.5? *whistles* That is a *really* old client. Try upgrading to a newer version. Version 2.1 was just released, and prior to that, 2.0.8 was available. The problem you described was fixed in one of the earlier 2.0 releases (I think at least by 2.0.5). On a different note... You might not have noticed yet, or maybe it isn't a problem on your system, but the Simeon IMSP server allows addresses books with mixed case characters in them. The CMU IMSP server does not, which means that any address book with mixed case characters will still exist, but not be available for use under the new server. I patched our server to allow mixed case characters. If this is something you are interested in, let me know. Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: Duplicate Delivery Database
--On Tuesday, September 04, 2001 6:01 PM -0400 Lawrence Greenfield [EMAIL PROTECTED] wrote: Date: Mon, 27 Aug 2001 09:38:26 -0400 From: Scott Adkins [EMAIL PROTECTED] [ ... many duplicate deliverdb errors deleted ... ] [ And so on, and so forth... hitting all the delivery databases ] [ there is a mixture of duplicate_check and duplicate-mark entries also ] You definitely need to run recovery on your duplicate delivery database. I might just delete the database entirely; since the duplicate delivery database isn't transactional, it can't guarantee consistency, and there's no critical database in it. We stopped the server and nuked the delivery files altogether. This only worked for 2 days, however, and now we are back to where we were. We see a constant stream of duplicate delivery database errors... To run recover, you must kill at lmtpd's (and any other processes that might have the database open) and run ctl_deliver -r. Just stopping (and waiting) and starting the master process should do this. I was curious how we could do this without stopping our production server. So, basically, I need to turn off our cron jobs that process the mail queues (thus, talking to lmtp via TCP), kill off any sendmail's currently doing queue processing, then kill off any lmtp daemons. At that point, I can run the ctl_deliver process. So, does ctl_deliver actually clear out all the locks in the database as part of the recovery operation? Possibly a system crash or a process crash/being killed at just the wrong time, due to the lack of transactions. I don't know, but this strikes me as being an extremeley fragile system. We seem to have database errors more than we don't. We also have caught on a number of occasions some lmtp processes getting stuck, spinning at 99.9% CPU in the process table. Throwing a debugger at them shows they are busy waiting for a lock to become available. We usually have to kill them off. So, what kind of information should I provide you to help track down the problem? It sounds like there is either a bug, or something has to be added to increase the robustness and recovery of the db3 locking mechanism. Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
SUMMARY: LMTP Unix Sockets
Okay, we were originally having problems with the deliver program connecting to LMTP via UNIX file sockets. We thought about going to TCP sockets and use deliver, but from what I saw in the source code, it may be that deliver only support UNIX file sockets. This meant that we needed to dump deliver altogether and get sendmail to talk to LMTP directly via TCP. The problem we had here was that we needed to support plussed users *and* have .forward file support (via the 'w' flag in the sendmail.cf file). If the plussed user feature was added to the sendmail config file, sendmail would strip off the plussed portion of the email address and stuff it into a temporary variable. When the passwd file check came, the plussed stuff was already stripped off and the check would be successful, thus allowing .foward files to work. Sendmail would then exec deliver, passing the plussed portion of the address (i.e. mailbox) on the command line so the message would get delivered in the correct place. However, with LMTP, you have no command line. Communication occurs over a UNIX file socket or a TCP socket. So, if you include the plussed user feature, it would store the plussed portion of the address in the temp variable, but nothing would be done with the variable later on... So, all the mail would end up in the users INBOX (but at least .forward files still worked). Our solution was to introduce a new mailer flag. We wanted to call it '+', but apparently, sendmail reserved that one for some other use. So, we call our new flag 'W'. What it means is that when the passwd file check occurs, only use the portion of the username up to the plus sign if it is present. This means that adkinss and adkinss+test would only have adkinss checked against the passwd file. Since the address isn't rewritten, it would remain as is when passed onto LMTP and delivery would occur as you would expect with plussed user support. If anyone is interested in the patch, let me know. That patch is only 8 lines of real code, so it is really small :) Now, for the rest of our configuration. We needed to configure the Cyrus server to use LMTP via TCP, and we needed to configure Sendmail to do the same. Since we aren't running SMTP AUTH just yet, and we are concerned about access control, and we don't have TCP Wrapper support either, the best way to run LTMP at this point is with the following line: lmtp cmd=lmtpd -a listen=localhost:lmtp As for our sendmail config, the MC file would contain the following for the LMTP mailer definition: MAILER_DEFINITIONS Mcyrus, P=[IPC], F=lsDFMnqAw@/:|SmXzW, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, U=cyrus:mail, A=TCP [localhost] lmtp LOCAL_RULE_0 Rbb + $+ @ $=w . $#cyrus $: + $1 Note: If you don't have 'listen=localhost:lmtp' in the cyrus.conf file, that is, you just have 'listen=lmtp', then the A=TCP section of the MC file should read A=TCP $h lmtp instead. Anyways, hope that helps others out there with similar delimnas. I did notice that there was far less information about running LMTP in TCP mode in the mail archives than the default unix file socket method. Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: Maxchild control
This was reported back in June and seems to still be a problem in 2.0.16. If I have the following in my /etc/cyrus.conf file, I find that I can still connect to port 9143 as many times as I want and never did find a maximum number: imap cmd=imapd listen=imap imap_testcmd=imapd listen=9143 maxchild=2 I have tried various values for maxchild, and even placed it in quotes to see if that helped, but it just seems to be ignored, as far as I can tell. This would be something very useful to us if it works... Scott --On Thursday, June 28, 2001 12:09 PM +0100 Julien Marchal [EMAIL PROTECTED] wrote: I try cyrus 2.0.14 on solaris 2.6 I have put the following line in /etc/cyrus.conf imap cmd=imapd listen=imap prefork=3 maxchild=50 But When I try cyrus make more than 50 symultaneous connexion (I try 80) the master fork 80 imapd process. The master don't follow the maxchild option ? I don't know if I make a mistake in my configuration. thank for help. -- Julien Marchal -- Email : [EMAIL PROTECTED] Université NANCY 2 - C.R.I. 13, Rue Michel Ney CO 75 - 54037 Nancy Cédex -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
LMTP unix sockets
We are seeing a lot of messages getting deferred in our sendmail logs when deliver is executed. Basically, deliver is exitting with EX_TEMPFAIL. Looking at the cyrus logs, I can correlate most of those failures to the following message: deliver: connect(/var/imap/socket/lmtp) failed: Permission denied Does anyone know the nature of this problem? I am not as familiar with UNIX file sockets as I am with TCP sockets. I assume, however, that the listen queue (backlog queue) still works the same for either type of socket, and since the master process is the one that is in charge and listening on the various sockets, that maybe I am hitting a wall with the backlog queue being too small if lots of connections are being made to imap/pop/lmtp all at the same time. Am I off base here? Anybody have any other ideas? We are going to move from a UNIX file socket to a TCP socket for lmtp and see if that helps. I will probably bump up the listen queue to see if that helps as well. Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: Help us! :)
Yikes... this isn't good news. The funny thing is that we had this working just fine with the old IMAP server (1.5.24) using deliver with Sendmail 8.12.0. Here is the cyrus section that we were using: Mcyrus, P=/usr/cyrus/bin/deliver, F=lsDFMnPqwA5@:|/, S=EnvFromL, R=EnvToL/HdrToL, U=cyrus:mail, T=DNS/RFC822/X-Unix, A=deliver -e -m $h -- $u Mcyrusbb, P=/usr/cyrus/bin/deliver, F=lsDFMnPu, S=EnvFromL, R=EnvToL/HdrToL, U=cyrus:mail, T=DNS/RFC822/X-Unix, A=deliver -e -m $u Notice the w flag listed in the first definition. Not only did plussed folders work for delivery (something I know was working when we went to Sendmail 8.12.0 the previous day, since over half my mail gets delivered to plussed folders), but forward files worked properly as well. So, there must be some way to fix this... I am going to forward this to the sendmail folks to see if they can help us as well. Scott --On Thursday, August 23, 2001 11:07 PM -0400 Lawrence Greenfield [EMAIL PROTECTED] wrote: The 'w' flag for the mailer says to make sure that all users it accepts mail for are in /etc/passwd. You'll probably need to remove it. Fortunately or unfortunately, this will also disable ~/.forward file processing. Since we don't use .forward file processing, it will be hard for me to immediately tell you how to make both work. Larry Date: Thu, 23 Aug 2001 20:05:19 -0400 From: Scott Adkins [EMAIL PROTECTED] cc: Cyrus-Bugs [EMAIL PROTECTED] --==1923943158== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline --On Thursday, August 23, 2001 7:25 PM -0400 Lawrence Greenfield [EMAIL PROTECTED] wrote: Date: Thu, 23 Aug 2001 19:20:47 -0400 From: Scott Adkins [EMAIL PROTECTED] cc: Lawrence Greenfield [EMAIL PROTECTED], Cyrus-Bugs [EMAIL PROTECTED] Okay, we upgraded to IMAP 2.0.16 with Sendmail 8.12.0. Everything seemsto be working except for delivery of email to subfolders. What ishappening instead is the emails are getting dumped to the user's INBOX.The ACLs are correct, and everything was working previously before theupgrade (using sendmail 8.12.0 and deliver). Anyone have ideas on whatis happening here? You have to make sure that Sendmail isn't stripping off the plus sign from the delivery. Try adding FEATURE(`preserve_local_plus_detail') to your Sendmail 8.12 mc file. Larry Okay, we just did that and we are still having problems. Mail can be delivered to the INBOX just fine, but a user+mbox@address causes a user unknown error and bounces the message back. Can you tell us exactly what your sendmail configuration is? This is what our cyrus section is in sendmail.cf: Mcyrus, P=[IPC], F=lsDFMnqAw@/:|SmXz, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, U=cyrus:mail, A=FILE /var/imap/socket/lmtp Also, attached is the actual MC file we are using to create our sendmail.cf file. Any help is greatly appreciated! We are in a lurch with a partial upgrade and a slightly broken email delivery system :-( Scott --==1923943158== Content-Type: text/plain; charset=us-ascii; name=cyrusv2_ou2.txt Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=cyrusv2_ou2.txt; size=1387 # divert(-1) # # (C) Copyright 2000 by Carnegie Mellon University # # This sample mc file is for a site that uses the Cyrus IMAP server # exclusively for local mail. This requires Sendmail 8.10 or later. # # $Id: cyrusv2.mc,v 1.3 2000/09/09 04:02:11 leg Exp $ divert(0)dnl VERSIONID(`cyrus v2 sample configuartion') OSTYPE(osf1) define(`confBIND_OPTS',`-DNSRCH -DEFNAMES') define(`confTO_IDENT',`0') define(`confQUEUE_LA',`25')dnl define(`confREFUSE_LA',`30')dnl define(`confCONNECTION_RATE_THROTTLE',`10')dnl define(`confMAX_DAEMON_CHILDREN',`200')dnl define(`confMAX_MESSAGE_SIZE',`1700')dnl define(`confTO_QUEUERETURN',`2d')dnl dnl setting cyrus as the trusted user will make it easier to pass dnl Sendmail's safefile checks. however, it means that someone with dnl the cyrus password could easily become root. dnl define(`confTRUSTED_USER', `cyrus') define(`confLOCAL_MAILER', `cyrus') FEATURE(`nocanonify') FEATURE(`always_add_domain') FEATURE(smrsh)dnl FEATURE(use_cw_file)dnl FEATURE(access_db)dnl FEATURE(blacklist_recipients)dnl FEATURE(`preserve_local_plus_detail')dnl MAILER(`local') MAILER(`smtp') MAILER_DEFINITIONS Mcyrus,P=[IPC], F=lsDFMnqAw@/:|SmXz, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS
PARTIAL SOLUTION: plussed users
Okay, the problem I was having was when we upgraded to Cyrus IMAPD 2.0.16 and Sendmail 8.12.0 using LMTP, plussed address (user+folder@address) no longer worked. If the 'w' flag was specified for the cyrus mailer in the sendmail.cf file, messages to plussed addresses would bounce with a user unknown message. Removing the 'w' flag would make that work again, but now .forward files no longer worked. What we couldn't understand was why it was working before when the same version of sendmail (8.12.0) was configured to use deliver for email delivery. The 'w' flag was there for it as well, but email messages were still being delivered successfully to user's subfolders. So, we went back to our old sendmail config file that uses deliver and now everything works. It works because the deliver program that comes with IMAPD 2.0.16 just turns around and sends it to the LMTP server anyways. Testing shows that email does indeed go to plussed address successfully and .forward files still work. Now, why does it work with the old sendmail configuration? It turns out that the old MC config files for sendmail (the ones you use to build the sendmail.cf file) did some magic with addresses before the 'w' flag got processed. Basically, the stuff after the plus was saved away to the $h macro, and the address was rewritten to not have the plus stuff in it. When it was time to pass the message off to deliver for email delivery, it would sort of reinsert the $h contents back into the address, which would make things work in the end. The final solution for us will have to be rewriting the MC file to use LMTP (since we want the performance gain of sendmail talking to LMTP directly) with the old style plussed address processing. Anyways, that is where we are at. Thanks to those who helped us with our problem, and especially thanks to Lawrence who was has been a great help to us from the very beginning when started down this upgrade path! Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: Help us! :)
--On Thursday, August 23, 2001 7:25 PM -0400 Lawrence Greenfield [EMAIL PROTECTED] wrote: Date: Thu, 23 Aug 2001 19:20:47 -0400 From: Scott Adkins [EMAIL PROTECTED] cc: Lawrence Greenfield [EMAIL PROTECTED], Cyrus-Bugs [EMAIL PROTECTED] Okay, we upgraded to IMAP 2.0.16 with Sendmail 8.12.0. Everything seemsto be working except for delivery of email to subfolders. What ishappening instead is the emails are getting dumped to the user's INBOX.The ACLs are correct, and everything was working previously before theupgrade (using sendmail 8.12.0 and deliver). Anyone have ideas on whatis happening here? You have to make sure that Sendmail isn't stripping off the plus sign from the delivery. Try adding FEATURE(`preserve_local_plus_detail') to your Sendmail 8.12 mc file. Larry Okay, we just did that and we are still having problems. Mail can be delivered to the INBOX just fine, but a user+mbox@address causes a user unknown error and bounces the message back. Can you tell us exactly what your sendmail configuration is? This is what our cyrus section is in sendmail.cf: Mcyrus, P=[IPC], F=lsDFMnqAw@/:|SmXz, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, U=cyrus:mail, A=FILE /var/imap/socket/lmtp Also, attached is the actual MC file we are using to create our sendmail.cf file. Any help is greatly appreciated! We are in a lurch with a partial upgrade and a slightly broken email delivery system :-( Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979 # divert(-1) # # (C) Copyright 2000 by Carnegie Mellon University # # This sample mc file is for a site that uses the Cyrus IMAP server # exclusively for local mail. This requires Sendmail 8.10 or later. # # $Id: cyrusv2.mc,v 1.3 2000/09/09 04:02:11 leg Exp $ divert(0)dnl VERSIONID(`cyrus v2 sample configuartion') OSTYPE(osf1) define(`confBIND_OPTS',`-DNSRCH -DEFNAMES') define(`confTO_IDENT',`0') define(`confQUEUE_LA',`25')dnl define(`confREFUSE_LA',`30')dnl define(`confCONNECTION_RATE_THROTTLE',`10')dnl define(`confMAX_DAEMON_CHILDREN',`200')dnl define(`confMAX_MESSAGE_SIZE',`1700')dnl define(`confTO_QUEUERETURN',`2d')dnl dnl setting cyrus as the trusted user will make it easier to pass dnl Sendmail's safefile checks. however, it means that someone with the dnl cyrus password could easily become root. dnl define(`confTRUSTED_USER', `cyrus') define(`confLOCAL_MAILER', `cyrus') FEATURE(`nocanonify') FEATURE(`always_add_domain') FEATURE(smrsh)dnl FEATURE(use_cw_file)dnl FEATURE(access_db)dnl FEATURE(blacklist_recipients)dnl FEATURE(`preserve_local_plus_detail')dnl MAILER(`local') MAILER(`smtp') MAILER_DEFINITIONS Mcyrus, P=[IPC], F=lsDFMnqAw@/:|SmXz, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, U=cyrus:mail, A=FILE /var/imap/socket/lmtp LOCAL_RULE_0 Rbb + $+ @ $=w . $#cyrus $: + $1
Re: Imap Administration via IMAP port
In addition to that, it should be stated that the difference between administrative commands and non-administrative commands is purely one of access rights. You must be logged into an adminstrative account to do anything useful to user accounts. In any the case, the command set is the same... just how much you can do with those commands is the question. Scott --On Monday, July 16, 2001 4:59 PM -0400 Michael T. Bacon [EMAIL PROTECTED] wrote: Cyradm is simply a tool for issuing commands to the IMAP port. As such, all of the commands in cyradm are just cleaned up IMAP commands, and could be issued by some other program or a human with a telnet connection to port 143. Michael Adi Linden wrote: Hi, What commands are available to administer cyrus-imap via connection on the imap port instead of using cyradm? TTYL, Adi -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: configure can't find sasl_getprop (v. 2.0.13)
--On Tuesday, May 01, 2001 2:44 PM +0200 Werner Reisberger [EMAIL PROTECTED] wrote: I want to upgrade from 1.6.24 to 2.0.13 but configure isn't able to find libsasl although I am using the library since many month with 1.6.24: checking for sasl_getprop in -lsasl... no My sasl libraries are in /usr/local/sasl/lib and I called configure with --with-sasl=/usr/local/sasl If I check libsasl.so.7.1.8 with nm I see the sasl_getprop symbol. Any hints? Werner A new SASL library was released as beta at the end of last March. It was released as 1.5.27. In any the case, it has the sasl_getprop() function that you described. My question is this... Even though configure doesn't see sasl_getprop() in your SASL library, it shouldn't be disabling SASL as a whole... Are you still able to compile the IMAP server and get it to work? I would expect that to be the case (but may not be at the moment), especially since the latest SASL release was considered beta. Anyways, if you need the library, you can get it at the following location: ftp://ftp.andrew.cmu.edu/pub/cyrus-mail/BETA/cyrus-sasl-1.5.27.tar.gz Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
IMAP Benchmark Tools
Does anyone have any tools that they use to test the performance of the IMAP server under load? Thanks, Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: 2.0.12 and DB
--On Tuesday, March 06, 2001 11:32 AM +1000 David Richards [EMAIL PROTECTED] wrote: Further to my previous message, the exact output is: See previous message for fix... :-) ### Making all in /messaging/richard2/builds/new-imap/cyrus-imapd-2.0.12/perl/imap cc -c -I../../lib -I/usr/local/include -fprm d -ieee -std -DLANGUAGE_C -O4 -DVERSION=\"1.00\" -DXS_VERSION=\"1.00\" -I/usr/lib/perl-5.005/lib/5.00503/alpha-dec_osf/CORE -DPERL_POLLUTE IMAP.c cc: Warning: IMAP.xs, line 492: In this statement, the referenced type of the pointer value "(pcb==PL_sv_undef?imclient_xs_fcmdcb:imclient_xs_cb)" is "function (pointer to struct imclient, pointer to struct xsccb, pointer to struct imclient_reply) returning void", which is not compatible with "function (pointer to struct imclient, pointer to void, pointer to struct imclient_reply) returning void". (ptrmismatch) (pcb == sv_undef ? --^ By the way, I do not know what this is about, and I believe I still get it after I fixed the problem below... but I haven't seen any side affects yet. I firmly believe it has to do with a Perl version issue. I know on 5.0a, Compaq distributed Perl 5.004_04. I don't know what it is on 5.1. This error doesn't seem to come up on Solaris where I have at least 5.005 or 5.6 installed. Running Mkbootstrap for Cyrus::IMAP () chmod 644 IMAP.bs ar cr tmp.a IMAP.o ../../lib/libcyrus.a : tmp.a LD_RUN_PATH="/usr/local/lib" ld -o blib/arch/auto/Cyrus/IMAP/IMAP.so -shared -expect_unresolved "*" -O4 -msym -s -all tmp.a -none -L/usr/local/lib -lsasl ld: Object file format error in: tmp.a(libcyrus.a): read_cur_obj_info: bad This was what I was talking about in the previous email. Look at the above ld line. You will see "-all tmp.a -none". The "tmp.a" library needs to be after "-none", not after "-all". Basically, it needs to read "-all -none tmp.a" instead. Given the nature of what "-all" and "-none" mean, they probably both need to go and just "tmp.a" would suffice. Anyways, this is exactly what the problem is, which happens to be defined in one of the perl modules associated with MakeMaker. Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: 2.0.12 and DB
--On Tuesday, March 06, 2001 10:48 AM +1000 David Richards [EMAIL PROTECTED] wrote: Yep, that seems to do better. Now I get a problem in 'perl/imap' something about Bad object file tmp.a(licyrus.a) Bad file magic number . Any ideas? Dave. Yup :-) I actually believe this to be a problem with the Perl installation that comes with Compaq, but I am not sure. I know where to fix it on Compaq, but I also believe that I should *not* do this, and that installing a newer version of Perl in /usr/local would be the better solution (but not always permitted in every environment). So, change to the perl/imap directory and edit Makefile.PL. Go to line 57. Change the 2 following lines as follows: OLD: 'OBJECT'= 'IMAP.o ../../lib/libcyrus.a', 'LIBS'= ["$SASL_LIB -lssl -lcrypto"], NEW: 'OBJECT'= 'IMAP.o', 'LIBS'= ["../lib/libcyrus.a $SASL_LIB -lssl -lcrypto"], The problem on Compaq is that it tries to create a library called tmp.a that contains 2 files, IMAP.o and libcyrus.a. However, when linking, the linker doesn't know what to do with the library within a library issue. With proper reorganization of the linker line in the Makefile, it can be made to work with this issue. However, it would be simpler to just move libcyrus.a to the LIBS section where it belongs. There is no reason for it to get dumped into tmp.a. In addition to the above fix, I tested this on the Solaris platform as well and the above changes work just fine as well... so I will be sending this in as a patch as well. In any the case, I am sure this is the last issue you will be running into. I was hoping it wouldn't be a problem for you, but apparently it was :-) Let us know how it goes! Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
Re: 2.0.12 and DB
--On Monday, March 05, 2001 10:44 AM +1000 David Richards [EMAIL PROTECTED] wrote: I am having problems getting v2.0.12 to compile on Compaq Tru64 v5.1. I have gone to quite extensive means to try and make it compile. Yes, I have had similar problems, but given time, I was able to work most of these out. I will try to help you as much as I can. I searched the mail list archives and found a posting from Ron Roskens on 3 Jun, 2000 which suggested something like: [CONFIGURATION STUFF DELETED] First, I created a RUNME script that contained all the necessary stuff. It even saves the output to a file for later reference. The most important aspect of the script is that it does set the environment properly to detect the BerkeleyDB library, and it sets the necessary compile time flags to make sure the executable can find the libraries at run-time. Without further ado, here is the script: #!/bin/ksh LOCAL="/usr/local" BERKELEY="/usr/local/BerkeleyDB.3.2" rm -f config.status config.log rm -f configure.output make.output export CC=cc export CFLAGS="-g -I$LOCAL/include -I$BERKELEY/include" export LDFLAGS="-lpthread -L$LOCAL/lib -Wl,-rpath,$LOCAL/lib \ -L$BERKELEY/lib -Wl,-rpath,$BERKELEY/lib" ./configure --with-dbdir=$BERKELEY \ --with-openssl=/usr/local/ssl \ --with-sasl=/usr/local \ --disable-sieve \ --without-agentx \ --without-zephyr \ | tee configure.output Then I edited the 'imap/Makefile' and added to the DEPLIBS variable the value '/usr/local/BerkeleyDB.3.0/lib/libdb.a' then the thing actually compiled. But I got the error: With the above configuration, you shouldn't have to do this... in fact, I would start over with freshly unpackaged sources. By the way, I noticed you were using "--with-sasldir" instead of "--with-sasl". The sasldir one does not exist in 2.0.12 from what I can see with configuration help. From the help, the following is listed: --with-sasl=DIRCompile with libsasl in DIR Don't know how to make 'BUILTSOURCES' This one is easy. Edit imap/Makefile.in and go to line 114, which is the PROGS section. You will see the following for the PROGS section, as well as the following BUILTSOURCES section: PROGS = imapd lmtpd pop3d \ fud reconstruct quota \ mbpath ipurge deliver \ ctl_mboxlist ctl_deliver \ @IMAP_PROGS@ BUILTSOURCES = imap_err.h pushstats.h lmtpstats.h The problem is that @IMAP_PROGS@ is empty after configuration, which means the line turns into a blank line. The Make program on Compaq simply sees something like the following: ctl_mboxlist ctl_deliver \ BUILTSOURCES = imap_err.h pushstats.h lmtpstats.h This makes the BUILTSOURCES section look like it is part of the PROGS section, which is not even in a correct Makefile format. In any the case, it causes an error. The solution is to combine @IMAP_PROGS@ with the previous line and remove the backslash: PROGS = imapd lmtpd pop3d \ fud reconstruct quota \ mbpath ipurge deliver \ ctl_mboxlist ctl_deliver @IMAP_PROGS@ BUILTSOURCES = imap_err.h pushstats.h lmtpstats.h Since this is done in Makefile.in, it will persist through all your configuration runs (since Makefile will be created properly). I will be sending in a patch to fix this issue. Some "makes" handle this better than others... but it doesn't hurt to do the above to make it work with them all. So, I gave up. To me it seems like a problem with the Compaq linker 'ld'. I am off to install gcc and see how that goes, I will respond to this posting (for archives sake) to say what happens. In the meantime, if anyone has any pointers for me could you please let me know. My suggestion is to try the above recipe using CC and not GCC. CC will produce much better code on Compaq than what GCC will. Also, I recommend moving to BerkeleyDB 3.2 or something higher than 3.0. Of course, if you are simply upgrading your IMAP server, you may be commited to 3.0 and may not want to upgrade your databases, but still, I recommend upgrading. Incidentally, the above recipe works fine on 5.0a. I haven't had the opportunity to try it on 5.1 yet, but I know I will be in the near future. Good luck! Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
[SUMMARY] Re: IMAP 2.0.9/SASL 1.5.24 Auth Issues
I have a solution to the following problem now. The problem is that in order to do SIA authentication under Compaq Tru64, you must be root. The reason the IMSP server didn't have any problems was because the server *does* run as root. However, the IMAP server immediately changes to the cyrus user, and is no longer root when doing authentication. The end result is the "authentication failed. generic failure" messages, which tell me very little about the true nature of the problem. The real solution was to use the pwcheck daemon, adding the SIA routines to it to make it work. Since the daemon runs as root and SASL will connect to it via a unix named socket, it all works good (though, not to my liking, but hey, it still works). I will send a patch into the SASL group so that the SIA routine stuff gets into the next version. Anwyays, I feel significantly better for getting past this hurdle :-) Scott --On Thursday, February 08, 2001 9:47 AM -0500 Scott Adkins [EMAIL PROTECTED] wrote: I have compiled up the Cyrus IMAP server (2.0.9) on Tru64 5.0a. It is using the SASL libraries (1.5.24) for authentication. The IMAP capability command produces the following: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT MULTIAPPEND SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES IDLE AUTH=ANONYMOUS AUTH=DIGEST-MD5 AUTH=CRAM-MD5 The only authentication method that has worked so far is ANONYMOUS. All of the other methods give me "NO authentication failure" followed by a "Authentication failed. generic failure". I have a /etc/sasldb file, owned by cyrus, and populated using saslpasswd (with "root" and "sadkins" accounts). The sasldblistusers commands produce the following output: user: root realm: cats.ohiou.edu mech: DIGEST-MD5 user: sadkins realm: cats.ohiou.edu mech: PLAIN user: sadkins realm: cats.ohiou.edu mech: CRAM-MD5 user: root realm: cats.ohiou.edu mech: PLAIN user: root realm: cats.ohiou.edu mech: CRAM-MD5 user: sadkins realm: cats.ohiou.edu mech: DIGEST-MD5 The /usr/lib/sasl/Cyrus.conf file lists the following: pwcheck_method: sia The /etc/imapd.conf also lists a line "sasl_pwcheck_method: sia". However, using imtest or telnetting to the port directly, I choose other auth mechs as well, and they all fail (except anonymous). I suspected the SASL library was at fault, but here is the clincher. I also compiled up the latest IMSP server, which uses the same libraries. It uses the "sia" mech as well (which is the password file lookup mechanism for Tru64). I can authenticate just fine on the IMSP server. I even had suspected at one time or another that maybe SASL authentication was failing on the IMSP server and that it was defaulting to a local authentication scheme, but after debugging the server a little bit, I proved that IMSP was indeed using SASL for its authentication, and that it was authenticating successfully (where-as, the IMAP server is failing). So, does anyone have any ideas what could be wrong here? I am at a total loss here... Thanks, Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
IMAP 2.0.9/SASL 1.5.24 Auth Issues
I am about at wits end with this, so I need to seek help. I have been trying to get on this list for some time, so I hope this message gets there. I have compiled up the Cyrus IMAP server (2.0.9) on Tru64 5.0a. It is using the SASL libraries (1.5.24) for authentication. The IMAP capability command produces the following: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT MULTIAPPEND SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES IDLE AUTH=ANONYMOUS AUTH=DIGEST-MD5 AUTH=CRAM-MD5 The only authentication method that has worked so far is ANONYMOUS. All of the other methods give me "NO authentication failure" followed by a "Authentication failed. generic failure". I have a /etc/sasldb file, owned by cyrus, and populated using saslpasswd (with "root" and "sadkins" accounts). The sasldblistusers commands produce the following output: user: root realm: cats.ohiou.edu mech: DIGEST-MD5 user: sadkins realm: cats.ohiou.edu mech: PLAIN user: sadkins realm: cats.ohiou.edu mech: CRAM-MD5 user: root realm: cats.ohiou.edu mech: PLAIN user: root realm: cats.ohiou.edu mech: CRAM-MD5 user: sadkins realm: cats.ohiou.edu mech: DIGEST-MD5 The /usr/lib/sasl/Cyrus.conf file lists the following: pwcheck_method: sia The /etc/imapd.conf also lists a line "sasl_pwcheck_method: sia". However, using imtest or telnetting to the port directly, I choose other auth mechs as well, and they all fail (except anonymous). I suspected the SASL library was at fault, but here is the clincher. I also compiled up the latest IMSP server, which uses the same libraries. It uses the "sia" mech as well (which is the password file lookup mechanism for Tru64). I can authenticate just fine on the IMSP server. I even had suspected at one time or another that maybe SASL authentication was failing on the IMSP server and that it was defaulting to a local authentication scheme, but after debugging the server a little bit, I proved that IMSP was indeed using SASL for its authentication, and that it was authenticating successfully (where-as, the IMAP server is failing). So, does anyone have any ideas what could be wrong here? I am at a total loss here... Thanks, Scott -- +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ Scott W. Adkinshttp://www.cns.ohiou.edu/~sadkins/ UNIX Systems Engineer mailto:[EMAIL PROTECTED] ICQ 7626282 Work (740)593-9478 Fax (740)593-1944 +-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=+=-=-=-=-=-=-=-=-+ CNS, HDL Center, Suite 301, Ohio University, Athens, OH 45701-2979
