Hi Pamod, On Mon, Jun 8, 2015 at 10:43 AM, Pamod Sylvester <[email protected]> wrote:
> Hi Sasikala, > > As discussed offline, let's try to figure out the reason for the ruminant. > If the reason is due to the load (i.e db latency to delete the content) > another option we could try out is updating TTL to delete the content > (Cassandra and most of the relation databases support this AFAIK). So that > the db will gradually delete it. > For messages with expiry details we can set a TTL at insertion. But for messages without expiration time we have to add that value when we want to delete the message. Will this work? For Cassandra if we want to update a TTL for an entry that doesn't have a TTL we need to re-insert the content [1]. For this we need to read the message and then re insert with a TTL value. (because when we do the delete call we only have the queue name and the message id) If you want to change the TTL of expiring data, you have to re-insert the > data with a new TTL. In Cassandra, the insertion of data is actually an > insertion or update operation, depending on whether or not a previous > version of the data exists. IMO this will not work since content remains in DB because we couldn't send a delete call for content before server gets killed. Therefore we won't have time to update TTL either. May be I'm wrong here. [1] http://docs.datastax.com/en/cql/3.1/cql/cql_using/use_expire_c.html Thanks, Asitha Thanks, > Pamod > > On Mon, Jun 8, 2015 at 9:56 AM, Sasikala Kottegoda <[email protected]> > wrote: > >> Hi all, >> >> WSO2MB currently stores messages in two separate tables named metadata >> and content. >> Metadata table stores all the important information about the messages >> whereas the content table only stores the message content chunks. >> Therefore, content with no associated metadata is useless, is a waste of >> space and reduces performance. >> >> The current implementation of message content deletion includes a >> scheduled task for periodical deletion of message content. This leads to an >> undesirable situation in the case of a node failure where message content >> could be left without being deleted(1000s of content chunks) in the db with >> no associated metadata. >> >> Message instant deletion was implemented as a solution for this (and is >> under test for the impact on the performance), but still it results in a >> small remnant (around 50). The current suggestions are, >> >> - To impose referential integrity on the two tables. But it could >> only be done in relational databases. >> - To develop a separate cleanup tool so that we could run it whenever >> desired. >> - To add a feature to the cluster coordinator to do the cleanup >> whenever a node fails. If the coordinator itself fails, the new >> coordinator >> could take over the duty. >> >> Any suggestions or recommendations are highly appreciated. >> >> Thank you >> -- >> Sasikala Kottegoda >> *Software Engineer* >> WSO2 Inc., http://wso2.com/ >> lean. enterprise. middleware >> Mobile: +94 774835928/712792401 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Pamod Sylvester * > > *WSO2 Inc.; http://wso2.com <http://wso2.com>* > cell: +94 77 7779495 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
