Re: [Dev] [MB] TombstoneOverwhelmingException When Purge Operation is Triggered When Subscription Disconnection/Deletion
+1 that's a good point. We could define a function to normalize the slot range through consolidating of all the remaining slots together. On Thu, Mar 12, 2015 at 1:22 PM, Asitha Nanayakkara asi...@wso2.com wrote: Hi Pamod Querying the remaining slots and removing content directly is OK. As an improvement you can add a separate method in slot coordinator to get that message range of all the slots, to reduce cluster communication, and then do the necessary processing in the local node it self. Thanks, Asitha On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, During the subscription disconnection/deletion purge operation is being called on the the relevant non durable topic the subscription/s was bound to. During purge operation all existing data (if any) relevant to that topic will be removed. When using hector the removal is done through a range query following LOC, ListLong currentPage = HectorDataAccessHelper.getNumericColumnKeysOfRow (keyspace, CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, lastProcessedID); Initially the relevant message ids are retrieved from MetaData CF and then removed from the MessageContent CF. Once more than 200,000 messages are sent since the range query queries the entire row of data belonging to a relevant topic, it results in the following, *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]* And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() *in the broker end, As solution, Since the idea of purging during disconnection is to clear out remaining data which have not being given out to its subscribers, we could query the slots which are still remaining and directly delete them from the MessageContentCF instead of querying them from meta data CF. Would there be any implication to this ? WDYT ? Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682 -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] TombstoneOverwhelmingException When Purge Operation is Triggered When Subscription Disconnection/Deletion
Hi Pamod, +1 for the solution. as discussed, we can use a single service call to fetch all slots from the coordinator for the given queue, and trigger single message deletions (referring to OnflightMessageTracker and SlotDeliveryWorker) . This will ensure that tombstones are not read when purging in a subscription closed scenario. However, we also need to think about triggering a queue purge from the UI, at which point we may not have active subscriptions or slot references. Thanks On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, During the subscription disconnection/deletion purge operation is being called on the the relevant non durable topic the subscription/s was bound to. During purge operation all existing data (if any) relevant to that topic will be removed. When using hector the removal is done through a range query following LOC, ListLong currentPage = HectorDataAccessHelper.getNumericColumnKeysOfRow (keyspace, CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, lastProcessedID); Initially the relevant message ids are retrieved from MetaData CF and then removed from the MessageContent CF. Once more than 200,000 messages are sent since the range query queries the entire row of data belonging to a relevant topic, it results in the following, *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]* And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() *in the broker end, As solution, Since the idea of purging during disconnection is to clear out remaining data which have not being given out to its subscribers, we could query the slots which are still remaining and directly delete them from the MessageContentCF instead of querying them from meta data CF. Would there be any implication to this ? WDYT ? Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- Cheers, Hasitha Amal De Silva Software Engineer Mobile : 0772037426 Blog: http://devnutshell.tumblr.com/ WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] TombstoneOverwhelmingException When Purge Operation is Triggered When Subscription Disconnection/Deletion
Hi Pamod Querying the remaining slots and removing content directly is OK. As an improvement you can add a separate method in slot coordinator to get that message range of all the slots, to reduce cluster communication, and then do the necessary processing in the local node it self. Thanks, Asitha On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, During the subscription disconnection/deletion purge operation is being called on the the relevant non durable topic the subscription/s was bound to. During purge operation all existing data (if any) relevant to that topic will be removed. When using hector the removal is done through a range query following LOC, ListLong currentPage = HectorDataAccessHelper.getNumericColumnKeysOfRow (keyspace, CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, lastProcessedID); Initially the relevant message ids are retrieved from MetaData CF and then removed from the MessageContent CF. Once more than 200,000 messages are sent since the range query queries the entire row of data belonging to a relevant topic, it results in the following, *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]* And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() *in the broker end, As solution, Since the idea of purging during disconnection is to clear out remaining data which have not being given out to its subscribers, we could query the slots which are still remaining and directly delete them from the MessageContentCF instead of querying them from meta data CF. Would there be any implication to this ? WDYT ? Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] TombstoneOverwhelmingException When Purge Operation is Triggered When Subscription Disconnection/Deletion
Hi, Idea was when last subscriber goes purge the messages without getting slots involved because it need to happen lightweight and fast. Thanks On Thu, Mar 12, 2015 at 3:16 AM, Pamod Sylvester pa...@wso2.com wrote: +1 Agreed, also as you and Ramith mentioned deleting the content directly will not be possible with the current architecture, since matadata is partitioned based on the queue/topic but the message content is not and ranges could overlap between multiple topics/queues. Therefore, we could invoke the content deletion task for the moment. Also another question i have is as to why we cannot partition the message content the same way we partition the meta information ? Also, from the UI when purging we could still refer the unassigned slots i believe. That's my observation. Do correct me if i am wrong. Thanks, Pamod On Thu, Mar 12, 2015 at 1:29 PM, Hasitha Amal De Silva hasit...@wso2.com wrote: Hi Pamod, +1 for the solution. as discussed, we can use a single service call to fetch all slots from the coordinator for the given queue, and trigger single message deletions (referring to OnflightMessageTracker and SlotDeliveryWorker) . This will ensure that tombstones are not read when purging in a subscription closed scenario. However, we also need to think about triggering a queue purge from the UI, at which point we may not have active subscriptions or slot references. Thanks On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, During the subscription disconnection/deletion purge operation is being called on the the relevant non durable topic the subscription/s was bound to. During purge operation all existing data (if any) relevant to that topic will be removed. When using hector the removal is done through a range query following LOC, ListLong currentPage = HectorDataAccessHelper.getNumericColumnKeysOfRow (keyspace, CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, lastProcessedID); Initially the relevant message ids are retrieved from MetaData CF and then removed from the MessageContent CF. Once more than 200,000 messages are sent since the range query queries the entire row of data belonging to a relevant topic, it results in the following, *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]* And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() *in the broker end, As solution, Since the idea of purging during disconnection is to clear out remaining data which have not being given out to its subscribers, we could query the slots which are still remaining and directly delete them from the MessageContentCF instead of querying them from meta data CF. Would there be any implication to this ? WDYT ? Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- Cheers, Hasitha Amal De Silva Software Engineer Mobile : 0772037426 Blog: http://devnutshell.tumblr.com/ WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* http://abeykoon.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB] TombstoneOverwhelmingException When Purge Operation is Triggered When Subscription Disconnection/Deletion
Tombstones became the killer :( The plan is to consolidate the unassigned slot ranges together and schedule the content to be removed within that range only so that the query will not grab tombstones. Thanks, Pamod On Thu, Mar 12, 2015 at 2:11 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi, Idea was when last subscriber goes purge the messages without getting slots involved because it need to happen lightweight and fast. Thanks On Thu, Mar 12, 2015 at 3:16 AM, Pamod Sylvester pa...@wso2.com wrote: +1 Agreed, also as you and Ramith mentioned deleting the content directly will not be possible with the current architecture, since matadata is partitioned based on the queue/topic but the message content is not and ranges could overlap between multiple topics/queues. Therefore, we could invoke the content deletion task for the moment. Also another question i have is as to why we cannot partition the message content the same way we partition the meta information ? Also, from the UI when purging we could still refer the unassigned slots i believe. That's my observation. Do correct me if i am wrong. Thanks, Pamod On Thu, Mar 12, 2015 at 1:29 PM, Hasitha Amal De Silva hasit...@wso2.com wrote: Hi Pamod, +1 for the solution. as discussed, we can use a single service call to fetch all slots from the coordinator for the given queue, and trigger single message deletions (referring to OnflightMessageTracker and SlotDeliveryWorker) . This will ensure that tombstones are not read when purging in a subscription closed scenario. However, we also need to think about triggering a queue purge from the UI, at which point we may not have active subscriptions or slot references. Thanks On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester pa...@wso2.com wrote: Hi All, During the subscription disconnection/deletion purge operation is being called on the the relevant non durable topic the subscription/s was bound to. During purge operation all existing data (if any) relevant to that topic will be removed. When using hector the removal is done through a range query following LOC, ListLong currentPage = HectorDataAccessHelper.getNumericColumnKeysOfRow (keyspace, CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, lastProcessedID); Initially the relevant message ids are retrieved from MetaData CF and then removed from the MessageContent CF. Once more than 200,000 messages are sent since the range query queries the entire row of data belonging to a relevant topic, it results in the following, *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]* And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() *in the broker end, As solution, Since the idea of purging during disconnection is to clear out remaining data which have not being given out to its subscribers, we could query the slots which are still remaining and directly delete them from the MessageContentCF instead of querying them from meta data CF. Would there be any implication to this ? WDYT ? Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- Cheers, Hasitha Amal De Silva Software Engineer Mobile : 0772037426 Blog: http://devnutshell.tumblr.com/ WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* http://abeykoon.blogspot.com -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [MB] TombstoneOverwhelmingException When Purge Operation is Triggered When Subscription Disconnection/Deletion
Hi All, During the subscription disconnection/deletion purge operation is being called on the the relevant non durable topic the subscription/s was bound to. During purge operation all existing data (if any) relevant to that topic will be removed. When using hector the removal is done through a range query following LOC, ListLong currentPage = HectorDataAccessHelper.getNumericColumnKeysOfRow (keyspace, CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, lastProcessedID); Initially the relevant message ids are retrieved from MetaData CF and then removed from the MessageContent CF. Once more than 200,000 messages are sent since the range query queries the entire row of data belonging to a relevant topic, it results in the following, *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2]at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2]* And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() *in the broker end, As solution, Since the idea of purging during disconnection is to clear out remaining data which have not being given out to its subscribers, we could query the slots which are still remaining and directly delete them from the MessageContentCF instead of querying them from meta data CF. Would there be any implication to this ? WDYT ? Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
