Hi all, The problem was solved by changing the database engine type to innoDB. Previously it was set to MyISAM which does not provide transactional support.
Thank you On Mon, Jun 8, 2015 at 12:36 PM, Pamod Sylvester <[email protected]> wrote: > Hi Asitha, > > Guess my explanation was too brief earlier. What i meant was not related > to setting an expiry for the messages during an insertion. What i meant was > updating the message content column with TTL, based on [1] it has > illustrated some examples of UPDATE queries setting TTL. I agree 100% if it > requires retrieval and insertion this will not work for this use case. > > The suggestion was to update a particular content column with a TTL > instead of deleting it instantly. (the assumption here is that the root > cause for remnant of the messages is due to the latency when it comes to > deletion of the content ). The content could be large 1MB and if there're > 100s of messages to be deleted instantly it might take a overhead from the > db side. Event if TTL is being used we need to update the columns so that > the deletion would happen gradually. i.e if there're 100 messages first 10 > messages we set x as the TTL and the next ten we increment it by a specific > constant value so that at a glance there will not be a overhead for the > database. > > However, we still need to verify that the message remnant was due to the > load in returning to db queries when deleting message content. If that's > not the case we cannot go with the option of using referential integrity > either, we will have to either go with the 2nd and 3rd option that was > initially suggested, to introduce separate clean up logic. > > [1] http://docs.datastax.com/en/cql/3.0/cql/cql_reference/update_r.html > > Thanks, > Pamod > > > On Mon, Jun 8, 2015 at 11:43 AM, Asitha Nanayakkara <[email protected]> > wrote: > >> Hi Pamod, >> >> On Mon, Jun 8, 2015 at 10:43 AM, Pamod Sylvester <[email protected]> wrote: >> >>> Hi Sasikala, >>> >>> As discussed offline, let's try to figure out the reason for the >>> ruminant. If the reason is due to the load (i.e db latency to delete the >>> content) another option we could try out is updating TTL to delete the >>> content (Cassandra and most of the relation databases support this AFAIK). >>> So that the db will gradually delete it. >>> >> >> For messages with expiry details we can set a TTL at insertion. But for >> messages without expiration time we have to add that value when we want to >> delete the message. Will this work? For Cassandra if we want to update a >> TTL for an entry that doesn't have a TTL we need to re-insert the content >> [1]. For this we need to read the message and then re insert with a TTL >> value. (because when we do the delete call we only have the queue name and >> the message id) >> >> If you want to change the TTL of expiring data, you have to re-insert the >>> data with a new TTL. In Cassandra, the insertion of data is actually an >>> insertion or update operation, depending on whether or not a previous >>> version of the data exists. >> >> >> IMO this will not work since content remains in DB because we couldn't >> send a delete call for content before server gets killed. Therefore we >> won't have time to update TTL either. May be I'm wrong here. >> >> [1] http://docs.datastax.com/en/cql/3.1/cql/cql_using/use_expire_c.html >> >> Thanks, >> Asitha >> >> Thanks, >>> Pamod >>> >>> On Mon, Jun 8, 2015 at 9:56 AM, Sasikala Kottegoda <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> WSO2MB currently stores messages in two separate tables named metadata >>>> and content. >>>> Metadata table stores all the important information about the messages >>>> whereas the content table only stores the message content chunks. >>>> Therefore, content with no associated metadata is useless, is a waste of >>>> space and reduces performance. >>>> >>>> The current implementation of message content deletion includes a >>>> scheduled task for periodical deletion of message content. This leads to an >>>> undesirable situation in the case of a node failure where message content >>>> could be left without being deleted(1000s of content chunks) in the db with >>>> no associated metadata. >>>> >>>> Message instant deletion was implemented as a solution for this (and is >>>> under test for the impact on the performance), but still it results in a >>>> small remnant (around 50). The current suggestions are, >>>> >>>> - To impose referential integrity on the two tables. But it could >>>> only be done in relational databases. >>>> - To develop a separate cleanup tool so that we could run it >>>> whenever desired. >>>> - To add a feature to the cluster coordinator to do the cleanup >>>> whenever a node fails. If the coordinator itself fails, the new >>>> coordinator >>>> could take over the duty. >>>> >>>> Any suggestions or recommendations are highly appreciated. >>>> >>>> Thank you >>>> -- >>>> Sasikala Kottegoda >>>> *Software Engineer* >>>> WSO2 Inc., http://wso2.com/ >>>> lean. enterprise. middleware >>>> Mobile: +94 774835928/712792401 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Pamod Sylvester * >>> >>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* >>> cell: +94 77 7779495 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Asitha Nanayakkara* >> Software Engineer >> WSO2, Inc. http://wso2.com/ >> Mob: + 94 77 85 30 682 >> >> > > > -- > *Pamod Sylvester * > > *WSO2 Inc.; http://wso2.com <http://wso2.com>* > cell: +94 77 7779495 > -- Sasikala Kottegoda *Software Engineer* WSO2 Inc., http://wso2.com/ lean. enterprise. middleware Mobile: +94 774835928/712792401
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
