Re: Question about how to remove data
Thanks guys for the answers! Saludos / Regards. Analía Lorenzatto. Hapiness is not something really made. It comes from your own actions by Dalai Lama On 21 Aug 2015 2:31 pm, Sebastian Estevez sebastian.este...@datastax.com wrote: To clarify, you do not need a ttl for deletes to be compacted away in Cassandra. When you delete, we create a tombstone which will remain in the system __at least__ gc grace seconds. We wait this long to give the tombstone a chance to make it to all replica nodes, the best practice is to run repairs as often as gc grace seconds in order to ensure edge cases where data comes back to life (i.e. the tombstone was never sent to one of your replicas and when the tombstones and data are removed from the other two replicas, all that is left is the old value. __at least__ are the key words in the previous paragraph, there are more conditions that need to be met in order for a tombstone to actually get cleaned up. As most things in Cassandra, these conditions are configurable (via the following compaction sub-properties): http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_configure_compaction_t.html All the best, [image: datastax_logo.png] http://www.datastax.com/ Sebastián Estévez Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com [image: linkedin.png] https://www.linkedin.com/company/datastax [image: facebook.png] https://www.facebook.com/datastax [image: twitter.png] https://twitter.com/datastax [image: g+.png] https://plus.google.com/+Datastax/about http://feeds.feedburner.com/datastax http://cassandrasummit-datastax.com/?utm_campaign=summit15utm_medium=summiticonutm_source=emailsignature DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world’s most innovative enterprises. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. On Thu, Aug 20, 2015 at 4:13 PM, Daniel Chia danc...@coursera.org wrote: The TTL shouldn't matter if you deleted the data, since to my understanding the delete should shadow the data signaling to C* that the data is a candidate for removal on compaction. Others might know better, but it could very well be the fact that gc_grace_seconds is 0 that is causing your problems. Others might have other suggestions, but you could potentially use sstable2json to see the raw contents of the sstable on disk and see why data is still there. Thanks, Daniel On Thu, Aug 20, 2015 at 12:55 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello, Daniel, I am using Size Tiered compaction. My concern is that as I do not have a TTL defined on the Column family, and I do not have the possibility to create it. Perhaps, the deleted data is never actually going to be removed? Thanks a lot! On Thu, Aug 20, 2015 at 4:24 AM, Daniel Chia danc...@coursera.org wrote: Is this a LCS family, or Size Tiered? Manually running compaction on LCS doesn't do anything until C* 2.2 ( https://issues.apache.org/jira/browse/CASSANDRA-7272) Thanks, Daniel On Wed, Aug 19, 2015 at 6:56 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello Michael, Thanks for responding! I do not have snapshots on any node of the cluster. Saludos / Regards. Analía Lorenzatto. Hapiness is not something really made. It comes from your own actions by Dalai Lama On 19 Aug 2015 6:19 pm, Laing, Michael michael.la...@nytimes.com wrote: Possibly you have snapshots? If so, use nodetool to clear them. On Wed, Aug 19, 2015 at 4:54 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster 2.1 comprised of 4 nodes. I removed a lot of data in a Column Family, then I ran manually a compaction on this Column family on every node. After doing that, If I query that data, cassandra correctly says this data is not there. But the space on disk is exactly the same before removing that data. Also, I realized that gc_grace_seconds = 0. Some people on the internet say that it could produce zombie data, what do you think? I do not have a TTL defined on the Column family, and I do not have the possibility to create it. So my questions is, given that I do not have a TTL defined is data going to be removed? or the deleted data is never actually going to be deleted due to I do not have a TTL? Thanks in advance! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Re: Question about how to remove data
Hello, Daniel, I am using Size Tiered compaction. My concern is that as I do not have a TTL defined on the Column family, and I do not have the possibility to create it. Perhaps, the deleted data is never actually going to be removed? Thanks a lot! On Thu, Aug 20, 2015 at 4:24 AM, Daniel Chia danc...@coursera.org wrote: Is this a LCS family, or Size Tiered? Manually running compaction on LCS doesn't do anything until C* 2.2 ( https://issues.apache.org/jira/browse/CASSANDRA-7272) Thanks, Daniel On Wed, Aug 19, 2015 at 6:56 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello Michael, Thanks for responding! I do not have snapshots on any node of the cluster. Saludos / Regards. Analía Lorenzatto. Hapiness is not something really made. It comes from your own actions by Dalai Lama On 19 Aug 2015 6:19 pm, Laing, Michael michael.la...@nytimes.com wrote: Possibly you have snapshots? If so, use nodetool to clear them. On Wed, Aug 19, 2015 at 4:54 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster 2.1 comprised of 4 nodes. I removed a lot of data in a Column Family, then I ran manually a compaction on this Column family on every node. After doing that, If I query that data, cassandra correctly says this data is not there. But the space on disk is exactly the same before removing that data. Also, I realized that gc_grace_seconds = 0. Some people on the internet say that it could produce zombie data, what do you think? I do not have a TTL defined on the Column family, and I do not have the possibility to create it. So my questions is, given that I do not have a TTL defined is data going to be removed? or the deleted data is never actually going to be deleted due to I do not have a TTL? Thanks in advance! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Question about how to remove data
Hello guys, I have a cassandra cluster 2.1 comprised of 4 nodes. I removed a lot of data in a Column Family, then I ran manually a compaction on this Column family on every node. After doing that, If I query that data, cassandra correctly says this data is not there. But the space on disk is exactly the same before removing that data. Also, I realized that gc_grace_seconds = 0. Some people on the internet say that it could produce zombie data, what do you think? I do not have a TTL defined on the Column family, and I do not have the possibility to create it. So my questions is, given that I do not have a TTL defined is data going to be removed? or the deleted data is never actually going to be deleted due to I do not have a TTL? Thanks in advance! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Re: Question about how to remove data
Hello Michael, Thanks for responding! I do not have snapshots on any node of the cluster. Saludos / Regards. Analía Lorenzatto. Hapiness is not something really made. It comes from your own actions by Dalai Lama On 19 Aug 2015 6:19 pm, Laing, Michael michael.la...@nytimes.com wrote: Possibly you have snapshots? If so, use nodetool to clear them. On Wed, Aug 19, 2015 at 4:54 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster 2.1 comprised of 4 nodes. I removed a lot of data in a Column Family, then I ran manually a compaction on this Column family on every node. After doing that, If I query that data, cassandra correctly says this data is not there. But the space on disk is exactly the same before removing that data. Also, I realized that gc_grace_seconds = 0. Some people on the internet say that it could produce zombie data, what do you think? I do not have a TTL defined on the Column family, and I do not have the possibility to create it. So my questions is, given that I do not have a TTL defined is data going to be removed? or the deleted data is never actually going to be deleted due to I do not have a TTL? Thanks in advance! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Re: After running nodetool clean up, the used disk space was increased
Yes Robert, I already cleared the snapshots. After that, the used disk space is: 10.x.x.b@$ df -h /mnt/cassandra Filesystem Size Used Avail Use% Mounted on /dev/xvdb1 745G 174G 572G 24% /mnt/cassandra But, the cluster shows me a different thing: $ nodetool status Datacenter: us-east === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.x.x.a 420.74 GB 256 66.7% eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c UN 10.x.x.b 416.42 GB 256 68.3% 19492c26-4458-4a0b-af04-72e0aab6598e 1c UN 10.x.x.c 165.15 GB 256 64.9% b8da952c-24b3-444a-a34e-7a1804eee6e6 1c I do not understand why the cluster still sees even more data before adding the third node. Thanks a lot!! On Fri, May 15, 2015 at 6:27 PM, Robert Wille rwi...@fold3.com wrote: Have you cleared snapshots? On May 15, 2015, at 2:24 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: The Replication Factor = 2. The RP is the default, but not sure how to check it. I am attaching the output of: nodetool ring Thanks a lot! On Fri, May 15, 2015 at 4:17 PM, Kiran mk coolkiran2...@gmail.com wrote: run cleanup on all the nodes and wait till it completes. On May 15, 2015 10:47 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes. I successfully added the third node last week. After that, I ran nodetool cleanup on one of the other two nodes, and it finished well but it increased the used disk space. Before running the clean up the node was 197 GB of used space, and after that it is 329GB used. It is my understanding that the clean up frees up some space, but in this case it was highly increased. I am running out of space, that's why I added a third node. Do you have any clue on how to proceed with that situation? Thanks in advance!! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. list -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Re: After running nodetool clean up, the used disk space was increased
The Replication Factor = 2. The RP is the default, but not sure how to check it. I am attaching the output of: nodetool ring Thanks a lot! On Fri, May 15, 2015 at 4:17 PM, Kiran mk coolkiran2...@gmail.com wrote: run cleanup on all the nodes and wait till it completes. On May 15, 2015 10:47 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes. I successfully added the third node last week. After that, I ran nodetool cleanup on one of the other two nodes, and it finished well but it increased the used disk space. Before running the clean up the node was 197 GB of used space, and after that it is 329GB used. It is my understanding that the clean up frees up some space, but in this case it was highly increased. I am running out of space, that's why I added a third node. Do you have any clue on how to proceed with that situation? Thanks in advance!! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. list Description: Binary data
Re: After running nodetool clean up, the used disk space was increased
Thanks Kiran for answering! I already ran cleanup on just one node. At this moment, I am running on the second one, but it did not finish there. On Fri, May 15, 2015 at 3:37 PM, Kiran mk coolkiran2...@gmail.com wrote: Did you try running nodetool cleanup on all the nodes ? On May 15, 2015 10:47 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes. I successfully added the third node last week. After that, I ran nodetool cleanup on one of the other two nodes, and it finished well but it increased the used disk space. Before running the clean up the node was 197 GB of used space, and after that it is 329GB used. It is my understanding that the clean up frees up some space, but in this case it was highly increased. I am running out of space, that's why I added a third node. Do you have any clue on how to proceed with that situation? Thanks in advance!! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Re: After running nodetool clean up, the used disk space was increased
The cluster is comprised of 3 nodes, with a RP=2: # nodetool status Datacenter: us-east === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.x.x.a 420.05 GB 256 66.7% eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c UN 10.x.x.b 415.68 GB 256 68.3% 19492c26-4458-4a0b-af04-72e0aab6598e 1c UN 10.x.x.c 164.45 GB 256 64.9% b8da952c-24b3-444a-a34e-7a1804eee6e6 1c 10.x.x.c is the last node added. Thanks! On Fri, May 15, 2015 at 4:02 PM, Kiran mk coolkiran2...@gmail.com wrote: What is the data distribution status across nodes ? What is the RP ? On May 16, 2015 12:30 AM, Analia Lorenzatto analialorenza...@gmail.com wrote: Thanks Kiran for answering! I already ran cleanup on just one node. At this moment, I am running on the second one, but it did not finish there. On Fri, May 15, 2015 at 3:37 PM, Kiran mk coolkiran2...@gmail.com wrote: Did you try running nodetool cleanup on all the nodes ? On May 15, 2015 10:47 PM, Analia Lorenzatto analialorenza...@gmail.com wrote: Hello guys, I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes. I successfully added the third node last week. After that, I ran nodetool cleanup on one of the other two nodes, and it finished well but it increased the used disk space. Before running the clean up the node was 197 GB of used space, and after that it is 329GB used. It is my understanding that the clean up frees up some space, but in this case it was highly increased. I am running out of space, that's why I added a third node. Do you have any clue on how to proceed with that situation? Thanks in advance!! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard. -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
After running nodetool clean up, the used disk space was increased
Hello guys, I have a cassandra cluster = 2.1.0-2 comprised of 3 nodes. I successfully added the third node last week. After that, I ran nodetool cleanup on one of the other two nodes, and it finished well but it increased the used disk space. Before running the clean up the node was 197 GB of used space, and after that it is 329GB used. It is my understanding that the clean up frees up some space, but in this case it was highly increased. I am running out of space, that's why I added a third node. Do you have any clue on how to proceed with that situation? Thanks in advance!! -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Re: Unexpected behavior after adding successffully a new node
Just in case I want to clarify that after bootstrapping the third node, it got data and seemed to be working fine. But it was the last night when the cluster started behaving in a weird way. The last node (successfully added last week) were being reported up and down all the time. After restarting cassandra service, the node got empty but reported as Up and Normal. Also, the amount of data reported by nodetool on the other nodes is much more than the amount already used in the device, is that normal? root@10.0.0.a:~# df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/xvdb1 745G 163G 583G 22% /mnt/cassandra root@:10.0.0.b~$ df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/xvdb1 745G 196G 550G 27% /mnt/cassandra # nodetool status Datacenter: us-east === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.0.0.a 391.34 GB 256 66.7% eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c UN 10.0.0.b 383.38 GB 256 68.3% 19492c26-4458-4a0b-af04-72e0aab6598e 1c Thanks in advance! On Tue, May 12, 2015 at 2:35 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, May 12, 2015 at 9:59 AM, arun sirimalla arunsi...@gmail.com wrote: Try running repair on node 3. Mostly disagree. If a node is empty after a bootstrap, remove it and re-bootstrap it. =Rob -- Saludos / Regards. Analía Lorenzatto. “It's possible to commit no errors and still lose. That is not weakness. That is life. By Captain Jean-Luc Picard.
Unexpected behavior after adding successffully a new node
Hello guys,
I have a cluster 2.1.0-2 comprised of 3 nodes. The replication factor=2.
We successfully added the third node last week. After that, We ran clean
ups on one node at that time. Then we ran repairs on all the nodes, and
finally compactions on all the CFs.
Last night, I noticed the cluster started behaving in a weird way. The
last node (successfully added last week) were being reported up and down
all the time. I could see a lot of messages like this on logs:
WARN [SharedPool-Worker-33] 2015-05-11 21:31:45,125
AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
Thread[SharedPool-Worker-33,5,main]: {}
java.lang.RuntimeException: java.io.FileNotFoundException:
/mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
(No such file or directory)
At the same time the consumption of heap used was on the top, up to the
point the rest of the cluster saw this node as down. After that, I just
restarted the cassandra service with no problems on that node.
Now, I can see the three nodes on the cluster Up and Normal, but this last
node (which was rebooted) does not have data. But it has all the structure
of cassandra data.
I can query against the new node and I get the same result as if do the
query against the others nodes. But, on this new node I do not have any
SStables:
root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
Datacenter: us-east
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.0.0.a 390.28 GB 256 66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c
UN 10.0.0.b 382.36 GB 256 68.3%
19492c26-4458-4a0b-af04-72e0aab6598e 1c
UN 10.0.0.c 40.61 MB 256 64.9%
b8da952c-24b3-444a-a34e-7a1804eee6e6 1c
What do you recommend to do? Leave this as if, remove it and try to join
this or a new one?
Thanks in advance!!
--
Saludos / Regards.
Analía Lorenzatto.
“It's possible to commit no errors and still lose. That is not weakness.
That is life. By Captain Jean-Luc Picard.
Re: New node got stuck joining the cluster after a while
Thanks very much for answering!
Do you think that after failing the joining task of a node to the cluster
should I run some repairs and cleanups?
Thanks!
On Tue, Apr 28, 2015 at 5:13 AM, Carlos Rolo r...@pythian.com wrote:
Hi,
The 2.1.x series is not recommeded for use, especially the first versions.
I would downgrade to 2.0.14 or if must stay on 2.1 upgrade your cluster to
2.1.4 or the imminent release of 2.1.5.
This mailing list as a few tips how to deal with the 2.1.x releases, but
the best way is indeed a downgrade or wait for the 2.1.5.
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
http://linkedin.com/in/carlosjuzarterolo*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Tue, Apr 28, 2015 at 3:30 AM, Analia Lorenzatto
analialorenza...@gmail.com wrote:
Hello guys,
I have a cluster comprised of 2 nodes, configured with vnodes. Using
2.1.0-2 version of cassandra.
And I am facing an issue when I want to joing a new node to the cluster.
At first starting joining but then it got stuck:
UN 1x.x.x.x 348.11 GB 256 100.0% 1c
UN 1x.x.x.x 342.74 GB 256 100.0% 1c
UJ 1x.x.x.x 26.86 GB 256 ? 1c
I can see some errors on the already working nodes:
*WARN [SharedPool-Worker-7] 2015-04-27 17:41:16,060
SliceQueryFilter.java:236 - Read 5001 live and 66548 tombstoned cells in
usmc.userpixel (see tombstone_warn_threshol*
*d). 5000 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647
2147483647}*
*WARN [SharedPool-Worker-32] 2015-04-27 17:41:16,668
SliceQueryFilter.java:236 - Read 2012 live and 30440 tombstoned cells in
usmc.userpixel (see tombstone_warn_thresho*
*ld). 5001 columns was requested,
slices=[b6d051df-0a8f-4c13-b93c-1b4ff0d82b8d:date-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}*
*ERROR [CompactionExecutor:35638] 2015-04-27 19:06:07,613
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:35638,1,main]*
*java.lang.AssertionError: Memory was freed*
*at
org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:281)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at org.apache.cassandra.io.util.Memory.getInt(Memory.java:233)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:118)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]*
* at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:92)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1209)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1165)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
~[apache-cassandra-2.1.0.jar:2.1.0*
*]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:365)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:127)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:112)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:229)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_51]*
*at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]*
*at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]*
But I do not see any warning or error message in logs of the joining
nodes. I just see an exception there when I run: nodetool info:
root@:~# nodetool info
ID : f5e49647-59fa-474f-b6af-9f65abc43581
Gossip active: true
Thrift active: false
Native Transport active: false
Load : 26.86 GB
Generation No: 1430163258
Uptime (seconds) : 18799
Heap Memory (MB) : 4185.15 / 7566.00
error: null
-- StackTrace --
java.lang.AssertionError
New node got stuck joining the cluster after a while
Hello guys,
I have a cluster comprised of 2 nodes, configured with vnodes. Using
2.1.0-2 version of cassandra.
And I am facing an issue when I want to joing a new node to the cluster.
At first starting joining but then it got stuck:
UN 1x.x.x.x 348.11 GB 256 100.0% 1c
UN 1x.x.x.x 342.74 GB 256 100.0% 1c
UJ 1x.x.x.x 26.86 GB 256 ? 1c
I can see some errors on the already working nodes:
*WARN [SharedPool-Worker-7] 2015-04-27 17:41:16,060
SliceQueryFilter.java:236 - Read 5001 live and 66548 tombstoned cells in
usmc.userpixel (see tombstone_warn_threshol*
*d). 5000 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647
2147483647}*
*WARN [SharedPool-Worker-32] 2015-04-27 17:41:16,668
SliceQueryFilter.java:236 - Read 2012 live and 30440 tombstoned cells in
usmc.userpixel (see tombstone_warn_thresho*
*ld). 5001 columns was requested,
slices=[b6d051df-0a8f-4c13-b93c-1b4ff0d82b8d:date-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}*
*ERROR [CompactionExecutor:35638] 2015-04-27 19:06:07,613
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:35638,1,main]*
*java.lang.AssertionError: Memory was freed*
*at
org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:281)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at org.apache.cassandra.io.util.Memory.getInt(Memory.java:233)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:118)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:123)
~[apache-cassandra-2.1.0.jar:2.1.0]*
* at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:92)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1209)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1165)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
~[apache-cassandra-2.1.0.jar:2.1.0*
*]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:365)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:127)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:112)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:229)
~[apache-cassandra-2.1.0.jar:2.1.0]*
*at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_51]*
*at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_51]*
*at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]*
But I do not see any warning or error message in logs of the joining
nodes. I just see an exception there when I run: nodetool info:
root@:~# nodetool info
ID : f5e49647-59fa-474f-b6af-9f65abc43581
Gossip active: true
Thrift active: false
Native Transport active: false
Load : 26.86 GB
Generation No: 1430163258
Uptime (seconds) : 18799
Heap Memory (MB) : 4185.15 / 7566.00
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:440)
at
org.apache.cassandra.service.StorageService.getTokens(StorageService.java:2079)
at
org.apache.cassandra.service.StorageService.getTokens(StorageService.java:2068)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
