+1 Agreed, also as you and Ramith mentioned deleting the content directly will not be possible with the current architecture, since matadata is partitioned based on the queue/topic but the message content is not and ranges could overlap between multiple topics/queues.
Therefore, we could invoke the content deletion task for the moment. Also another question i have is as to why we cannot partition the message content the same way we partition the meta information ? Also, from the UI when purging we could still refer the unassigned slots i believe. That's my observation. Do correct me if i am wrong. Thanks, Pamod On Thu, Mar 12, 2015 at 1:29 PM, Hasitha Amal De Silva <[email protected]> wrote: > Hi Pamod, > > +1 for the solution. as discussed, we can use a single service call to > fetch all slots from the coordinator for the given queue, and trigger > single message deletions (referring to OnflightMessageTracker and > SlotDeliveryWorker) . This will ensure that tombstones are not read when > purging in a subscription closed scenario. > > However, we also need to think about triggering a queue purge from the UI, > at which point we may not have active subscriptions or slot references. > > Thanks > > On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester <[email protected]> wrote: > >> Hi All, >> >> During the subscription disconnection/deletion purge operation is being >> called on the the relevant non durable topic the subscription/s was bound >> to. During purge operation all existing data (if any) relevant to that >> topic will be removed. >> >> When using hector the removal is done through a range query following LOC, >> >> List<Long> currentPage = >> HectorDataAccessHelper.getNumericColumnKeysOfRow >> (keyspace, >> CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, >> lastProcessedID); >> >> Initially the relevant message ids are retrieved from MetaData CF and >> then removed from the MessageContent CF. Once more than 200,000 messages >> are sent since the range query queries the entire row of data belonging to >> a relevant topic, it results in the following, >> >> >> >> *Caused by: >> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat >> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) >> ~[apache-cassandra-2.1.2.jar:2.1.2]at >> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) >> ~[apache-cassandra-2.1.2.jar:2.1.2]* >> >> And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: >> TimedOutException() *in the broker end, >> >> As solution, >> >> Since the idea of purging during disconnection is to clear out remaining >> data which have not being given out to its subscribers, we could query the >> slots which are still remaining and directly delete them from the >> MessageContentCF instead of querying them from meta data CF. Would there be >> any implication to this ? WDYT ? >> >> Thanks, >> Pamod >> >> -- >> *Pamod Sylvester * >> *Senior Software Engineer * >> Integration Technologies Team, WSO2 Inc.; http://wso2.com >> email: [email protected] cell: +94 77 7779495 >> > > > > -- > Cheers, > > Hasitha Amal De Silva > Software Engineer > Mobile : 0772037426 > Blog : http://devnutshell.tumblr.com/ > WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) > -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: [email protected] cell: +94 77 7779495
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
