Re: multi-node cassandra config doubt
Hi All, It looks it is know issue with Cassandra-0.8.4. So either I have to wait till 0.8.5 to be released or have to switch to 0.7.8 if this has been resolved in that. Ref: https://issues.apache.org/jira/browse/CASSANDRA-3044 Regards, Thamizhannal P --- On Thu, 25/8/11, Thamizh tceg...@yahoo.co.in wrote: From: Thamizh tceg...@yahoo.co.in Subject: Re: multi-node cassandra config doubt To: user@cassandra.apache.org Date: Thursday, 25 August, 2011, 9:01 PM Hi Aaron, Thanks a lot for your suggestions. I have got exhausted with below error. It would great if you point me what went wrong with my approach. I wanted to install cassandra-0.8.4 on 3 nodes and to run Map/Reduce job that uploads data from HDFS to Cassandra. I have installed Cassnadra on 3 nodes lab02(199.168.0.2),lab03(199.168.0.3) lab04(199.168.0.4) respectively and can create a keyspace column family and they got distributed across the cluster. When I run my map/reduce program it ended up with UnknownHostException. the same map/reduce program works well on single node cluster. Here are the steps which I have followed. 1. cassandra.yaml details lab02(199.168.0.2): (seed node) auto_bootstrap: false seeds: 199.168.0.2 listen_address: 199.168.0.2 rpc_address: 199.168.0.2 lab03(199.168.0.3): auto_bootstrap: true seeds: 199.168.0.2 listen_address: 199.168.0.3 rpc_address: 199.168.0.3 lab04(199.168.0.4): auto_bootstrap: true seeds: 199.168.0.2 listen_address: 199.168.0.4 rpc_address: 199.168.0.4 2. O/P of bin/cassandra : -- -- INFO 11:59:40,602 Node /199.168.0.2 is now part of the cluster INFO 11:59:40,604 InetAddress /199.168.0.2 is now UP INFO 11:59:55,667 Node /199.168.0.4 is now part of the cluster INFO 11:59:55,669 InetAddress /199.168.0.4 is now UP INFO 12:01:08,389 Joining: getting bootstrap token INFO 12:01:08,410 New token will be 43083119672609054510947312506340649252 to assume load from /199.168.0.2 INFO 12:01:08,412 Enqueuing flush of Memtable-LocationInfo@6824966(123/153 serialized/live bytes, 4 ops) INFO 12:01:08,413 Writing Memtable-LocationInfo@6824966(123/153 serialized/live bytes, 4 ops) INFO 12:01:08,461 Completed flushing /var/lib/cassandra/data/system/LocationInfo-g-2-Data.db (287 bytes) INFO 12:01:08,477 Node /199.168.0.3 state jump to normal INFO 12:01:08,480 Enqueuing flush of Memtable-LocationInfo@10141941(53/66 serialized/live bytes, 2 ops) INFO 12:01:08,482 Writing Memtable-LocationInfo@10141941(53/66 serialized/live bytes, 2 ops) INFO 12:01:08,514 Completed flushing /var/lib/cassandra/data/system/LocationInfo-g-3-Data.db (163 bytes) INFO 12:01:08,527 Node /199.168.0.3 state jump to normal INFO 12:01:08,652 mx4j successfuly loaded HttpAdaptor version 3.0.1 started on port 8081 3. When I run my map/reduce program it ended up with UnknownHostException Error: java.net.UnknownHostException: /199.168.0.2 at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849) at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200) at java.net.InetAddress.getAllByName0(InetAddress.java:1153) at java.net.InetAddress.getAllByName(InetAddress.java:1083) at java.net.InetAddress.getAllByName(InetAddress.java:1019) at java.net.InetAddress.getByName(InetAddress.java:969) at org.apache.cassandra.client.RingCache.refreshEndpointMap(RingCache.java:93) at org.apache.cassandra.client.RingCache.init(RingCache.java:67) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter.init(ColumnFamilyRecordWriter.java:98) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter.init(ColumnFamilyRecordWriter.java:92) at org.apache.cassandra.hadoop.ColumnFamilyOutputFormat.getRecordWriter(ColumnFamilyOutputFormat.java:132) at org.apache.cassandra.hadoop.ColumnFamilyOutputFormat.getRecordWriter(ColumnFamilyOutputFormat.java:62) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:553) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) at org.apache.hadoop.mapred.Child.main(Child.java:170) Here are the config line for map/reduce. job4.setReducerClass(TblUploadReducer.class ); job4.setOutputKeyClass(ByteBuffer.class); job4.setOutputValueClass(List.class); job4.setOutputFormatClass(ColumnFamilyOutputFormat.class); ConfigHelper.setOutputColumnFamily(job4.getConfiguration(), args[1],args[3] ); ConfigHelper.setRpcPort(job4.getConfiguration(), args[7]); // 9160 ConfigHelper.setInitialAddress(job4.getConfiguration(), args[9]); // 199.168.0.2 ConfigHelper.setPartitioner(job4.getConfiguration(), org.apache.cassandra.dht.RandomPartitioner); Steps which I have verified, 1. There is a passwordless ssh has been configured b/w lab02,lab03 lab04. All the nodes can ping each other with out any issues. 2. When I ran InetAddress.getLocalHost() from java program on lab02
NPT while get_range_slices in 0.8.1
Hi, we have 4 node Cassandra (version 0.8.1) cluster. 2 CF inside. While first CF is working properly (read/store), get_range_slices query on second CF return NPE error. Any idea why it happen? Maybe some known bug and fixed in 0.8.3 ? ERROR [pool-2-thread-51] 2011-08-25 15:02:04,360 Cassandra.java (line 3210) Internal error processing get_range_slices java.lang.NullPointerException at org.apache.cassandra.db.ColumnFamily.diff(ColumnFamily.java:298) at org.apache.cassandra.db.ColumnFamily.diff(ColumnFamily.java:406) at org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:103) at org.apache.cassandra.service.RangeSliceResponseResolver$2.getReduced(RangeSliceResponseResolver.java:120) at org.apache.cassandra.service.RangeSliceResponseResolver$2.getReduced(RangeSliceResponseResolver.java:85) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:74) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:715) at org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:617) at org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cassandra.java:3202) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
Re: Cassandra Node Requirements
Sort of. There's some fine print, such as the 50% number is only if you're manually forcing major compactions, which is not recommended, but a bigger thing to know is that 1.0 will introduce leveled compaction [1] inspired by leveldb. The free space requirement will then be a small number of megabytes. [1] https://issues.apache.org/jira/browse/CASSANDRA-1608 And in the mean time, plan for more storage as I and others have reported in other threads that repairs have caused disks to fill up (in my case, I think it's because I had multiple repairs running at the same time).
Re: Cassandra 082 - Large swap memory
On Thu, Aug 25, 2011 at 11:42 PM, King JKing beuk...@gmail.com wrote: Dear Jonathan, Cassandra process has 63.5 GB virtual size. I mention about RES column in top. RES is 8.3G. Very large than 2.5G Used Memory Used show in JConsole. https://issues.apache.org/jira/browse/CASSANDRA-2868 -Brandon
Re: Is Cassandra suitable for this use case?
On Fri, Aug 26, 2011 at 12:18 AM, Eric Evans eev...@acunu.com wrote: On Thu, Aug 25, 2011 at 6:31 AM, Ruby Stevenson ruby...@gmail.com wrote: - Although Cassandra (and other decentralized NoSQL data store) has been reported to handle very large data in total, my preliminary understanding is the individual column value is quite limited. I have read some posts saying you shouldn't store file this big in Cassandra for example, use a path instead and let file system handle it. Is this true? http://wiki.apache.org/cassandra/FAQ#large_file_and_blob_storage -- Eric Evans Acunu | http://www.acunu.com | @acunu It is important to note that even distributed storage file solutions like GlusterFS, NFS, Iscsi are not as good as local disk either. The reason is simple, best case scenario on a local file system file lives in VFS cache you maybe be talking like micro or nanoseconds to read a block. Even if not in vfs cache you are bounded by BUS speeds and disk speeds. Network disks solutions like (isci) require dedicated expensive infini-ban or ethernet networks to work well. Also that when using something like ISCI your system gets to leverage its local VFS cache so not all the read traffic has to cross the network. The best case scenario for Cassandra would be a block of data living in the row cache on a node. This data still has to traverse the network. That is going to be slower then a local file. However depending on what you are doing storing file data in cassandra could be a big win.
Re: Commit log fills up in less than a minute
Sure I can fill in the ticket. Here is what I have noticed so far, the count of HH is not going up, which is good. I think what must have happened is that after I restarted the cluster, no new hints were added just the old one's are still around and not cleaned up, is that possible? Cannot say for sure, since I only looked at this JMX bean abt 36 hours after restart. Can I just clean this up using the JMX call? I do not want to turn off HH, since that can handle the intermittent network hiccups well, right? On Thu, Aug 25, 2011 at 2:47 PM, aaron morton aa...@thelastpickle.comwrote: Could you put together some information on this in a ticket and references this one https://issues.apache.org/jira/browse/CASSANDRA-3071 The short term fix is to disable HH. You will still get consistent reads. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 25/08/2011, at 3:22 AM, Anand Somani wrote: So I have looked at the cluster from - Cassandra-client - describe cluster = shows correctly - 3 nodes - used the StorageService - JMX bean =UnreachableNodes - shows 0 If all these show the correct ring state, why are hints being maintained, looks like that is the only way to find out about phantom nodes. On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani meatfor...@gmail.comwrote: So, I restarted the cluster (not rolling), but it is still maintaining hints for the IP's that are no longer part of the ring. nodetool ring shows things correctly (as only 3 nodes). When I check thru the jmx hintedhandoff manager, it shows it is maintaining the hints for those non existent IP's. So the question is - How can I remove these IP permanently, so hints do not get saved? - Not all nodes see the same list of IP's On Sun, Aug 21, 2011 at 3:10 PM, aaron morton aa...@thelastpickle.comwrote: Yup, you can check the what HH is doing via JMX. there is a bug in 0.7 that can result in log files not been deleted https://issues.apache.org/jira/browse/CASSANDRA-2829 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 22/08/2011, at 4:56 AM, Anand Somani wrote: We have a lot of space on /data, and looks like it was flushing data fine from file timestamps. We did have a bit of goofup with IP's when bringing up a down node (and the commit files have been around since then). Wonder if that is what triggered it and we have a bunch of hinted handoff's being backed up. For hinted hand off - how do I check if the nodes are collecting hints ( I do have it turned on). I noticed console bean HintedHandManager, is that the only way to find out. On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller peter.schul...@infidyne.com wrote: When does the actual commit-data file get deleted. The flush interval on all my memtables is 60 minutes They *should* be getting deleted when they no longer contain any data that has not been flushed to disk. Are flushes definitely still happening? Is it possible flushing has started failing (e.g. out of disk)? The only way I can think of over nodes directly affecting the commit log size on your node would be e.g. hinted handoff resulting in burst of writes. -- / Peter Schuller (@scode on twitter)
Range scan
How does range scan work in Cassandra? Does the read of a key perform the read across all the SSTables that contain the key and return the row or are SSTables processed sequentially? If I have a key k and its columns are spread across N SSTables then does the read of key k return the row with all the columns spread across N SSTables or does it process each SSTable and return a row with whatever columns it finds in the processed SSTable.
