Hi Paul, I think what I'm seeing is really a locking conflict between imapd and smtpd (or possibly lmtp too).
imapd will try to do inserts as it puts messages into the Sent folder, or updates msg flags (SEEN, etc). In my test cases, I'm using a imap client to send a message to multiple local users. I'm unsure on the timings imapd uses, but here is what I think is happening... The imap client causes imapd to open the db, which will update flag (SEEN, etc) which puts a RESERVED lock on the sqlite db. At what point that gets released, I'm not sure, but it should be following the first (auto?)commit. When I send a message, imapd wants to stuff it into the SENT folder, which again does a RESERVED lock. Most, if not all, of the time the imap client is open, there could be a shared lock on the db (I guess I need to dig into the imapd code too). smtp tries to insert the mail passed from the MTA while all this is going on. It also needs a RESERVED lock. If I'm reading the sqlite docs right, smtp can't get a RESERVED lock if any SHARED locks already exist. It will get a SHARED lock while waiting for the reserved lock, which might in turn block imapd from committing any write (not sure how long that SHARED lock would be held). So, if imapd is holding SHARED locks, then all the retries in dbsqlite.c/db_query will not help resolve the SQLITE_BUSY status. imapd would need to release any SHARED locks, and then smtp would have to try the INSERT (RESERVED lock) again. Am I making any sense? --Keith [EMAIL PROTECTED] wrote: > A NOTE has been added to this issue. > ====================================================================== > http://dbmail.org/mantis/view.php?id=619 > ====================================================================== > Reported By: asianux-keith > Assigned To: paul > ====================================================================== > Project: DBMail > Issue ID: 619 > Category: Database layer > Reproducibility: always > Severity: minor > Priority: normal > Status: acknowledged > target: > ====================================================================== > Date Submitted: 04-Jul-07 03:39 CEST > Last Modified: 05-Jul-07 06:57 CEST > ====================================================================== > Summary: sqlite locking: "sqlite3_get_table failed" results > in msgs not being inserted into db, msgs bounced. > Description: > When sending a message from thunderbird or evolution to a server with > postfix and dbmail 2.2.5 and a sqlite(3) backend, the messages are often > not delivered because dbmail-smtp/lmtp cannot lock/write to the sqlite db. > Non-delivered msgs bounce. > > In each test, a msg is crafted with three local TO: address (including > self). > > 1st test: (logfile sqlite-non-delivery-log.txt) > dbmail-lmtp, dbmail-imapd and dbmail-pop3d are all running. > The client sending the msg uses imap. One receiving client is imap, and > the other is pop. postfix uses dbmail-smtp to deliver. > > 2nd test: (logfile sqlite-lmtp-non-delivery-log.txt) > dbmail-lmtp, dbmail-imapd and dbmail-pop3d are all running. > The client sending the msg uses imap. One receiving client is imap, and > the other is pop. postfix uses dbmail-lmtp to deliver. > > 3rd test: (logfile sqlite-pop_only-non-delivery-log.txt) > only dbmail-pop3d is running. > The client sending the msg uses pop. Two receiving clients are imap. > postfix uses dbmail-smtp to deliver. > > > ====================================================================== > > ---------------------------------------------------------------------- > paul - 04-Jul-07 10:32 > ---------------------------------------------------------------------- > I've added a retry loop to the sqlite code to help deal with these > situation. Please try the attached patch. > > Also: try to lower the concurrency settings in postfix's master.cf. Set > the width of the lmtp and pipe transports to 1. > > ---------------------------------------------------------------------- > asianux-keith - 05-Jul-07 06:57 > ---------------------------------------------------------------------- > The patch by itself does not resolve the problem, even with just three > processes battling for control. > > The process which I think had the initial lock took less than two seconds > to complete, and the two other processes fought and it seems both lost to > each other. > > I'd suggest building in some incremental, or random, delay with each loop > of the retry code. Something between 1/10 and 1 sec? > > On the other hand, limiting the concurrency in postfix had a dramatic > effect, and allowed the delivery of all messages (even without the patch) > in a simple test. They only issue was that copying the message to the > Sent folder with the client using imap would 'sometimes' hang. > > Combining the two did even better, until I stressed it by sending five > different messages (all with three local TO: addresses). Not all five > were delivered. > user1 (imap client) sent 2 msgs. user2 (imap client) sent 1 msg. root > (pop3) sent the last two msgs. > Oddly enough, user1 received only three msgs, and returned 1 bounce msg; > user 2 received four msgs, and root received all five. > There was at least one hang copying to Sent. > > To main.cf, I added > dbmail-lmtp_destination_concurrency_limit = 1 > dbmail-smtp_destination_concurrency_limit = 1 > > and to master.cf, I changed > dbmail-lmtp unix - - n - 1 lmtp -o > disable_dns_lookups=yes > dbmail-smtp unix - n n - 1 pipe > > Issue History > Date Modified Username Field Change > ====================================================================== > 04-Jul-07 03:39 asianux-keith New Issue > 04-Jul-07 10:32 paul Note Added: 0002282 > 04-Jul-07 10:33 paul File Added: > 0001-add-retry-loop-to-sqlite-db_query.txt > 04-Jul-07 10:34 paul Assigned To => paul > 04-Jul-07 10:34 paul Severity major => minor > 04-Jul-07 10:34 paul Status new => acknowledged > 05-Jul-07 06:57 asianux-keith Note Added: 0002284 > ====================================================================== > > _______________________________________________ > Dbmail-dev mailing list > [email protected] > http://twister.fastxs.net/mailman/listinfo/dbmail-dev > > > _______________________________________________ Dbmail-dev mailing list [email protected] http://twister.fastxs.net/mailman/listinfo/dbmail-dev
