You're complicating things... even while bearerbox _process_ the DLR's as they arrive, they're "awaited for" on someplace else than the store, either a memory queue or a DB table, that's my point. Your example of a disconnected box doesn't apply to a real-case scenario at all and it's just a distraction from the original topic.
I was explaining that you can't use a "validity period" approach to expire unfinished dlrs because they're on the DB (or the non-persisted "internal" storage, but we were focusing on the DB-persisted case) and you don't really have a list of all of them at hand. I don't see the point of bringing all the bearerbox complexities into the problem, I think I know pretty well how the whole thing work, I just didn't want to complicate things for no reason. Now, back to the problem, another approach would be to keep an in-memory list of all pending DLR's in memory and use the DB backend only for persistance (just like the message store does). You'd need to load it at boot and keep it synchronized at all times (add new DLR's as they arrive and delete them as they are finished). With that approach, a "validity period" system could be easily developed on top of it. Regards, Alex 2010/8/8 Nikos Balkanas <[email protected]> > Sorry Alex, are you talking about DLRs or dlr_entries? What are lost > between restarts are dlr_entries, not DLRs. DLRs are still received, but > cannot find a matching dlr_entry and are discarded. Otherwise, if they are > matched, they are send to smsbox. If smsbox is not available, or doesn't > send an ACK, then they are kept in store, like any other MO, waiting for > smsbox. How else can you find DLRs in store? Try sending a few SMS with > dlr-mask, db storage for dlr_entries, and disconnect smsbox. Examine the > store from HTTP admin after a while, when access.log logs in an incoming > DLR. You will find that DLR in store. When you connect smsbox (or sqlbox, or > smppbox) again, this DLR will be processed and deleted for storage. > > > BR, > Nikos > ----- Original Message ----- From: Alejandro Guerrieri > To: Nikos Balkanas > Cc: Konstantin Vayner ; Users > Sent: Sunday, August 08, 2010 10:50 PM > > Subject: Re: Large and growing number of queued DLR > > > Why would we need the DB storage if that's the case? The DLR _matching_ is > performed against the DB, not the store. > > > If you use "in-memory" storage, DLR's are lost when restarting the service. > That wouldn't happen if the store would hold them isn't it? > > > Regards, > > > Alex > > > > 2010/8/8 Nikos Balkanas <[email protected]> > > Alex, > > Are you sure about that? AFAIK DLRs are saved to store, like any other MO, > until bb receives an ACK from upstream (smsbox, sqlbox, smppbox). > > BR, > Nikos > ----- Original Message ----- From: Alejandro Guerrieri > To: Konstantin Vayner > Cc: Users > Sent: Sunday, August 08, 2010 12:56 PM > Subject: Re: Large and growing number of queued DLR > > > > Well, that poses some problems. > > > DLR's are a different animal than regular messages, they're not hold on the > message store. DLR's are by default held in memory or, if you wish to > persist them across restarts, you use DB storage. When you use a DB storage > DLR's are no longer held in memory. > > > Now, in order to reuse validity period, you'd need to have all DLR's in > memory at all times and that's only true for in-memory DLR's. For DB-stored > DLR's, since they're no longer in memory, all the operations are done over > SQL queries so you don't longer have the ability of traversing them in > memory. > > > The approach with DB-based DLR's is to run a query like this: > > > DELETE FROM dlr WHERE stamp < (NOW()-INTERVAL 5 DAY) > > > Now, if you use MySQL's MyISAM storage, that query locks the whole table, > so you need to be very careful when running it (we do it at low traffic > hours and we run it repeatedly with a LIMIT clause to make it return faster > and allow for the rest of the queries to unlock. > > > In other words, I wouldn't put that kind of query on a thread to execute > repeatedly unattended. > > > A possible approach would be to load all DLR's in memory when the service > starts, the same way it's done with the regular messages and the file store. > Once that's done, we'd be able to traverse the old DLR's without locking the > system. > > > Now, from a higher level perspective, this poses the question of the reason > why DLR's are treated different than the rest of the messages that pass thru > Kannel, but that's for another thread I guess ;) > > > Regards, > > > Alex > > > On Sun, Aug 8, 2010 at 7:56 AM, Konstantin Vayner <[email protected]> > wrote: > > Alejandro, > > > Actually, kannel can reuse validity period and automatically issue > (perhapse configurable?) expiration dlr + wipe a message when a message is > past that period by some good value (lets say + 1 day - this should be more > than enough). > In fact we could issue another kind of dlr even - maybe "unknown" or > smth... > this would both solve a problem with bb building up queue of dlrs that will > never be matched and aid external application logic imho > > > Regards, > Konstantin > > > > On Fri, Aug 6, 2010 at 6:36 PM, Alejandro Guerrieri < > [email protected]> wrote: > > Dlr's are far from perfect. Many carriers don't return it on all cases, so > it's logical that it'll eventually grow over time. > > > Check with your carriers what's the expected lifetime of SMS on their > network and run a query to clean records older than that period (if they > hold it for 3 days, you shouldn't be expecting any DLR's after that period). > > > If you don't know/can't answer, 5-7 days should be a safe ballpark figure. > > > Regards, > > > Alex > > > > On Fri, Aug 6, 2010 at 4:09 PM, brett skinner <[email protected]> > wrote: > > Hi > > > Sorry forgot to add that I am using mysql storage > > > > ---------- Forwarded message ---------- > From: brett skinner <[email protected]> > Date: Fri, Aug 6, 2010 at 4:06 PM > Subject: Large and growing number of queued DLR > To: Users <[email protected]> > > > > Hi > > > We have been sending sms successfully and receiving the DLRs. As far as I > can tell we have been processing them correctly and all the logs in the web > application point to no failures. I had a look at the logs for smsbox and > there were a couple of URLs that it could not fetch but those have already > been fixed and read in from the log file. However the DLR storage number is > still growing and there have been no further errors in the logs. > > > Is there a rate at which Kannel tries to post the DLR to the specified URL? > Is it perhaps because Kannel may have been restarted and now it has no > knowledge of those DLR because the information that it needs to do the look > up is now gone? > > > Regards >
