Re: cannot login cassandra
Caused by: java.io.IOException: Channel not open for writing - cannot extend file to required size at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:851) at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192) You should investigate there. No more memory for mmapped file ? On Mon, Aug 18, 2014 at 2:40 AM, 鄢来琼 laiqiong@gtafe.com wrote: Dear All, After creating table “mykeysapce. cffex_l23 “, some data are insert into the table, then the connection is closed and quit; And then reconnection Cassandra, error is thrown. The following errors are recorded in the system.log file. ERROR [SSTableBatchOpen:1] 2014-08-15 17:08:22,912 CassandraDaemon.java (line 199) Exception in thread Thread[SSTableBatchOpen:1,5,main] FSReadError in /opt/apache-cassandra-2.0.9/cassandraData/data/mykeyspace/cffex_l23/mykeyspace-cffex_l23-jb-17-Index.db at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200) at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168) at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:457) at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:264) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Channel not open for writing - cannot extend file to required size at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:851) at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192) ... 11 more ERROR [SSTableBatchOpen:1] 2014-08-15 17:08:22,913 StorageService.java (line 366) Stopping gossiper WARN [SSTableBatchOpen:1] 2014-08-15 17:08:22,913 StorageService.java (line 280) Stopping gossip by operator request INFO [SSTableBatchOpen:1] 2014-08-15 17:08:22,913 Gossiper.java (line 1279) Announcing shutdown ERROR [SSTableBatchOpen:1] 2014-08-15 17:08:24,916 StorageService.java (line 371) Stopping RPC server INFO [SSTableBatchOpen:1] 2014-08-15 17:08:24,916 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [SSTableBatchOpen:1] 2014-08-15 17:08:24,920 StorageService.java (line 376) Stopping native transport INFO [SSTableBatchOpen:1] 2014-08-15 17:08:24,937 Server.java (line 182) Stop listening for CQL clients WARN [Native-Transport-Requests:55] 2014-08-15 17:08:24,973 Slf4JLogger.java (line 76) An exception was thrown by an exception handler. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:115) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:73) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:57) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46) at org.jboss.netty.channel.Channels.write(Channels.java:725) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60) at org.jboss.netty.channel.Channels.write(Channels.java:725) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.jboss.netty.handler.execution.ExecutionHandler.handleDownstream(ExecutionHandler.java:186) at org.jboss.netty.channel.Channels.write(Channels.java:704) at
disk space and tombstones
In our Twitter-like application users have their own timelines with news from subscriptions. To populate timelines we're using fanout on write. But we forced to trim it to keep free disk space under control. We use wide rows pattern and trim them with DELETE by primary key USING TIMESTAMP. But it seems our efforts have no effect and disk free space decreases rapidly, even after compaction. It is clear for us that it's not the best use case for Cassandra, but maybe there is a way to decrease disk utilisation for this pattern? Our cluster consists of 15 c3.4xlarge nodes with 300 GB storage. Timeline's files take up 170 GB on each node. gc_grace is 300, rf=3 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: disk space and tombstones
Is that GC_grace 300 days? Rahul Neelakantan On Aug 18, 2014, at 5:51 AM, Dimetrio dimet...@flysoft.ru wrote: In our Twitter-like application users have their own timelines with news from subscriptions. To populate timelines we're using fanout on write. But we forced to trim it to keep free disk space under control. We use wide rows pattern and trim them with DELETE by primary key USING TIMESTAMP. But it seems our efforts have no effect and disk free space decreases rapidly, even after compaction. It is clear for us that it's not the best use case for Cassandra, but maybe there is a way to decrease disk utilisation for this pattern? Our cluster consists of 15 c3.4xlarge nodes with 300 GB storage. Timeline's files take up 170 GB on each node. gc_grace is 300, rf=3 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: disk space and tombstones
I was exactly in your same situation, I could only reclaim disk space for trimmed data this way: very low gc_grace + size tiered compaction + slice timestamp deletes + major compaction 2014-08-18 12:06 GMT+02:00 Rahul Neelakantan ra...@rahul.be: Is that GC_grace 300 days? Rahul Neelakantan On Aug 18, 2014, at 5:51 AM, Dimetrio dimet...@flysoft.ru wrote: In our Twitter-like application users have their own timelines with news from subscriptions. To populate timelines we're using fanout on write. But we forced to trim it to keep free disk space under control. We use wide rows pattern and trim them with DELETE by primary key USING TIMESTAMP. But it seems our efforts have no effect and disk free space decreases rapidly, even after compaction. It is clear for us that it's not the best use case for Cassandra, but maybe there is a way to decrease disk utilisation for this pattern? Our cluster consists of 15 c3.4xlarge nodes with 300 GB storage. Timeline's files take up 170 GB on each node. gc_grace is 300, rf=3 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Cassandra Wiki Immutable?
Hi, What is the state of Cassandra Wiki -- http://wiki.apache.org/cassandra ? I tried to update a few pages, but it looks like pages are immutable. Do I need to have my Wiki username (OtisGospodnetic) added to some ACL? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/
Re: Cassandra Wiki Immutable?
Ah, I missed that. Thanks. Email sent. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Mon, Aug 18, 2014 at 12:19 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Otis, On the front page, https://wiki.apache.org/cassandra/FrontPage there are instructions on how to get edit permissions: If you would like to contribute to this wiki, please send an email to the mailing list dev.at.cassandra.apache-dot-org with your wiki username and we will be happy to add you. Contributions welcome! Mark On Mon, Aug 18, 2014 at 11:15 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, What is the state of Cassandra Wiki -- http://wiki.apache.org/cassandra ? I tried to update a few pages, but it looks like pages are immutable. Do I need to have my Wiki username (OtisGospodnetic) added to some ACL? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/
Re: disk space and tombstones
In our case major compaction (using sstableresetlevel) will take 15 days for 15 nodes plus trimming time. So it turns into never ending maintenance mode. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356p7596361.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: disk space and tombstones
No, it is in seconds -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356p7596363.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: disk space and tombstones
what about you timeline versioning? every time a timeline has more than x columns, you bump its version (which should be part of its row key) and start writing on that one (though this will make your app substantially more complex). AFAIK reclaiming disk space for delete rows is far easier than reclaiming it for deleted columns. 2014-08-18 12:30 GMT+02:00 Dimetrio dimet...@flysoft.ru: No, it is in seconds -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356p7596363.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Compression during bootstrap
On Wednesday, August 13, 2014, Robert Coli wrote: On Wed, Aug 13, 2014 at 5:53 AM, Ruchir Jha ruchir@gmail.com javascript:_e(%7B%7D,'cvml','ruchir@gmail.com'); wrote: We are adding nodes currently and it seems like compression is falling behind. I judge that by the fact that the new node which has a 4.5T disk fills up to 100% while its bootstrapping. Can we avoid this problem with the LZ4 compressor because of better compression or do we just need a bigger disk? 2TB per node is a lot of data. 4.5 would be a huge amount of data. Sure. Do you mean we should have started adding nodes before we got here? Do you mean compaction is falling behind? Do you setcompactionthroughput 0 while bootstrapping new nodes? I did a nodetool getcompactionthroughput and I got 0 MB/ s. It seems like that just disables compaction throttling which seems like a good thingin my scenario. Is that correct? I don't think compression is involved here? Why do you think it does? This is why I thought compression is involved: http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_about_config_compress_c.html Side note : we also increased number of concurrent compactors from 3 to 10. Because we had a lot of idle CPU lying around but that's not helping everytime we start bootstrapping we are still hitting 4.5 tb and them we run out of disk space. =Rob
Re: disk space and tombstones
That scheme assumes we have to read counter value before write something to the timeline. This is what we try to avoid as an anti-pattern. By the way, is there any difference between slice trimming of one row and sharding pattern in terms of compaction? AFAIK, delete with timestamp by primary key also creates single row tombstone. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356p7596367.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
LeveledCompaction, streaming bulkload, and lot's of small sstables
Hi! I'm bulkloading via streaming from Hadoop to my Cassandra cluster. This results in a rather large set of relatively small (~1MiB) sstables as the number of mappers that generate sstables on the hadoop cluster is high. With SizeTieredCompactionStrategy, the cassandra cluster would quickly compact all these small sstables into decently sized sstables. With LeveledCompactionStrategy however, it takes a much longer time. I have multithreaded_compaction: true, but it is only taking on 32 sstables at a time in one single compaction task, so when it starts with ~1500 sstables, it takes quite some time. I'm not running out of I/O. Is there some configuration knob I can tune to make this happen faster? I'm getting a bit confused by the description for min_sstable_size, bucket_high, bucket_low etc - and I'm not sure if they apply in this case. I'm pondering options for decreasing the number of sstables being streamed from the hadoop side, but if that is possible remains to be seen. Thanks! \EF
Re: sstableloader and ttls
sstableloader just loads given SSTables as they are. TTLed columns are sent and will be compacted at the destination node eventually. On Sat, Aug 16, 2014 at 4:28 AM, Erik Forsberg forsb...@opera.com wrote: Hi! If I use sstableloader to load data to a cluster, and the source sstables contain some columns where the TTL has expired, i.e. the sstable has not yet been compacted - will those entries be properly removed on the destination side? Thanks, \EF -- Yuki Morishita t:yukim (http://twitter.com/yukim)
Re: LeveledCompaction, streaming bulkload, and lot's of small sstables
On Mon, Aug 18, 2014 at 6:21 AM, Erik Forsberg forsb...@opera.com wrote: Is there some configuration knob I can tune to make this happen faster? I'm getting a bit confused by the description for min_sstable_size, bucket_high, bucket_low etc - and I'm not sure if they apply in this case. You probably don't want to use multi-threaded compaction, it is removed upstream. nodetool setcompactionthroughput 0 Assuming you have enough IO headroom etc. =Rob
Re: Compaction before Decommission and Bootstrapping
On Sun, Aug 17, 2014 at 6:46 AM, Maxime maxim...@gmail.com wrote: I've been spending the last few days trying to move a cluster on DigitalOcean 2GB machines to 4GB machines (same provider). To do so I wanted to create the new nodes, bootstrap them, then decommission the old ones (one by one seems to be the only available option). Don't do this, use this process instead. https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/ It will be significantly faster. You do have to repair the node after restarting it, because hints for the old node IP are discarded. =Rob
Re: Running sstableloader from live Cassandra server
On Sat, Aug 16, 2014 at 2:40 AM, Erik Forsberg forsb...@opera.com wrote: I will want to run the sstableloader on the source cluster, but at the same time, that source cluster needs to keep running to serve data to clients. Use the on-node bulkLoad interface, which is designed for this, instead of sstableloader, which isn't? http://palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra A notable difference between bulkLoad and sstableloader is that bulkLoad does not have sstableloader's --ignores option, which means you can't tell it to ignore replica targets on failure. bulkLoad there is the JMX call. =Rob
Re: disk space and tombstones
2014-08-18 13:25 GMT+02:00 clslrns vitaly.chir...@flysoft.ru: That scheme assumes we have to read counter value before write something to the timeline. This is what we try to avoid as an anti-pattern. You can work around the read counter before read, but I agree that it would be much better if disk space was reclaimed after compaction in a more human understandable way. By the way, is there any difference between slice trimming of one row and sharding pattern in terms of compaction? AFAIK, delete with timestamp by primary key also creates single row tombstone. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356p7596367.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Reasonable range for the max number of tables?
We use Cassandra for multi-tenant application. Each tenant has own set of tables and we have 1592 tables in total in our production Cassandra cluster. It's running well and doesn't have any memory consumption issue, but the challenge confronting us is the schema change problem.We have such a large amount of tables, moreover a significant amount of them have around as many as 50 columns, which totals 33099 rows in schema_columns table in system keyspace. Every time we do schema change for all of our tenants, the whole cluster will get very busy and applications running against it need to be shut down for several hours to accommodate this change. The way we solve this is using a new feature we developed called template. There is a detailed description in the JIRA issue we opened: https://issues.apache.org/jira/browse/CASSANDRA-7643 We have some performance results in our 15-node test cluster. Normally creating 400 tables takes more than hours for all the migration stage tasks to complete , but if we create 400 tables with templates, *it just takes 1 to 2 seconds*. It also works great for alter table. [image: Inline image 1] [image: Inline image 1] table # in the graph means the number of existing tables in user keyspaces. We created 400 more tables and measure the time all tasks in migration stage take to complete. Besides, we also measure the migration task completion time for adding one column for a template, which will also add the column for all the column families with that template. We believe what we proposed here can be very useful for other people in the Cassandra community as well. We have attached the patch in the JIRA. You can also read the community feedbacks there. Thanks, Cheng On Tue, Aug 5, 2014 at 5:43 AM, Michal Michalski michal.michal...@boxever.com wrote: - Use a keyspace per customer These effectively amount to the same thing and they both fall foul to the limit in the number of column families so do not scale. But then you can scale by moving some of the customers to a new cluster easily. If you keep everything in a single keyspace or - worse - if you do your multitenancy by prefixing row keys with customer ids of some kind, it won't be that easy, as you wrote later in your e-mail. M. Kind regards, Michał Michalski, michal.michal...@boxever.com On 5 August 2014 12:36, Phil Luckhurst phil.luckhu...@powerassure.com wrote: Hi Mark, Mark Reddy wrote To segregate customer data, you could: - Use customer specific column families under a single keyspace - Use a keyspace per customer These effectively amount to the same thing and they both fall foul to the limit in the number of column families so do not scale. Mark Reddy wrote - Use the same column families and have a column that identifies the customer. On the application layer ensure that there are sufficient checks so one customer can't read another customers data And while this gets around the column family limit it does not allow the same level of data segregation. For example with a separate keyspace or column families it is trivial to remove a single customer's data or move that data to another system. With one set of column families for all customers these types of actions become much more difficult as any change impacts all customers but perhaps that's the price we have to pay to scale. And I still think this needs to be made more prominent in the documentation. Thanks Phil -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Reasonable-range-for-the-max-number-of-tables-tp7596094p7596119.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: disk space and tombstones
That scheme assumes we have to read counter value before write something to the timeline. This is what we try to avoid as an anti-pattern. Hummm, it looks like there is a need for a tool to take care of the bucketing switch. I've seen a lot of use cases where people need to do wide row bucketing but bumps into concurrency issues for keeping the state of the current partition. Managing it client-side at application level can be a nightmare. On Mon, Aug 18, 2014 at 1:25 PM, clslrns vitaly.chir...@flysoft.ru wrote: That scheme assumes we have to read counter value before write something to the timeline. This is what we try to avoid as an anti-pattern. By the way, is there any difference between slice trimming of one row and sharding pattern in terms of compaction? AFAIK, delete with timestamp by primary key also creates single row tombstone. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/disk-space-and-tombstones-tp7596356p7596367.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Best way to format a ResultSet / Row ?
The DataStax java driver has a Row object which getInt, getLong methods… However, the getString only works on string columns. That's probably reasonable… but if I have a raw Row, how the heck do I easily print it? I need a handy way do dump a ResultSet … -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
