Hi Stipe, Sounds great - as you say being able to use disk based replication techniques would be good but also providing persistence as a default option is also very valuable. I'll be keen to do some tests on IO performance - my biggest concern for larger installs (as often disk choice is not necessarily under our control).
If anyone cooks up any test scripts to analyse please feel free to share (as I will). I'm just trying to think of an easy way to generate several thousand in the DLR store and have them all expire (identical validity period or similar). Thanks again Stipe - nice work! Cheers, Alan On Mon, 2012-08-06 at 14:16 +0200, Stipe Tolj wrote: > Hi all, > > I have commuted a new DLR storage module, implementing a DLR storage as > spool directory on the local file system. > > It can be configured as usually in the 'group = core' context, i.e. > > group = core > ... > dlr-storage = spool > dlr-spool = /var/spool/kannel/dlr > ... > > This adds to the 'dlr-storage = internal' a second natively (no extra > libs required) storage type, that grantees persistence! > > Implementation notes: > > The tricky part in storing a DLR temp msg, as we do it in-memory or via > the external DB calls is that we can't use our own UUID as the message > identifier, but have to rely on the foreign ID we get back from the > SMSC. For some SMSC protocols we can assume they are unique, i.e. SMPP, > but for others we need to handle the destination address as part of the > identifier heuristic. We do this already for all the DLR storage types > that have been implemented. > > The DLR spool type does it the "same way", by using a hashed prefix and > a surrogated suffix in the filename of the message we store into the > DLR spool. (BTW, we pack/unpack the message the same way as for the > spool directory itself, containing the MT/MO payload messages). > > The hashed prefix is SHA1_digest(smsc_id+foreign_id), and the surrogated > suffix HEX_digest(destination). I.e. if we need to store a > temporary DLR message for smsc_id=foobar, foreign_id=1234567890 with > destination=49111111 this would result in: > > prefix: cc8ffce63741d3a8a388cbd71f335ad7670c6123 > suffix: 3439313131313131 > > So, we store i.e. to file: > /var/spool/kannel/dlr/15/cc8ffce63741d3a8a388cbd71f335ad7670c61233439313131313131 > > Ok, how do we resolve this on the DLR receive time you may ask? > > At DLR time, we have the smsc_id and the foreign_id from the SMSC, so we > can compute the same hash digest. Due that we know how long the HASH > digest is, we can isolate the "surrogate", which is the destination > address. We'll simply scan the directory for all files having the hash > digest as name prefix, and test all against the destination surrogate. > The comparison of the destination part follows the same heuristic as we > do in the DB storage implementations or the in-memory module. Meaning, > we assume that the destination we "may" receive at the DLR event is > "smaller" then the destination address we had available at sent time, > which was also the one we stored. > > The performance is reasonable good, especially if you have fast IO > disks. The drawback is the directory scan per DLR resolution, which we > could "overcome" by maintaining a in-memory hash of the files within > the spool directory, and create it at start-up time. This may be next > phase of the implementation. > > A benefit of using the file system "natively" as DLR storage is that you > can utilize commonly known disk block replications techniques, i.e. DRBD > and for HA you could use even shared file systems (ocfs2, gfs) on top of > that file system replication. > > Comments are as always welcome. > > Stipe >
