Re: lots of small nodes vs fewer big nodes
I advise you to have 7GB RAM better (more would be better if you have a lot of data per node). But the real difference is made when you use SSDs or RAIDs of SSDs, since the bottleneck is the disk throughput in most cases. We are in the cloud, we tried a lot of configurations and were comfortable only with a nodes RAM 7 GB (RAM 15 GB with lot of data / node) and see a real enhancement when we switched to SSDs (latency from 20-40 to 3-5 ms), even reducing the number of nodes from 18 to 3. This was quite impressive and I recommend SSD (or RAID of SSDs) since to anyone who can afford it. Alain 2013/8/7 Andrey Ilinykh ailin...@gmail.com You still have the same amount of RAM, so you cache the same amount of data. I don't think you gain much here. On the other side, maintenance procedures (compaction, repair) may hit your 2CPU box. I wouldn't do it. Thank you, Andrey On Wed, Aug 7, 2013 at 10:24 AM, Paul Ingalls paulinga...@gmail.comwrote: Quick question about systems architecture. Would it be better to run 5 nodes with 7GB RAM and 4CPU's or 10 nodes with 3.5GB RAM and 2CPUS? I'm currently running the former, but am considering the latter. My goal would be to improve overall performance by spreading the IO across more disks. My currently cluster has low CPU utilization but does spend a good amount of time in iowait. Would moving to more smaller nodes help with that? Or would I run into trouble with the smaller ram and cpu? Thanks! Paul
RE: lots of small nodes vs fewer big nodes
I found using a JBOD SSD setup (one per data directory) to be faster than RAID. JBOD configuration will also allow a disk to fail and the remaining disks to continue serving reads if you set disk_failure_policy: best_effort. If you do go for a RAID controller watch out for any special read/write caching options. I found these also killed performance. From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: 08 August 2013 09:30 To: user@cassandra.apache.org Subject: Re: lots of small nodes vs fewer big nodes I advise you to have 7GB RAM better (more would be better if you have a lot of data per node). But the real difference is made when you use SSDs or RAIDs of SSDs, since the bottleneck is the disk throughput in most cases. We are in the cloud, we tried a lot of configurations and were comfortable only with a nodes RAM 7 GB (RAM 15 GB with lot of data / node) and see a real enhancement when we switched to SSDs (latency from 20-40 to 3-5 ms), even reducing the number of nodes from 18 to 3. This was quite impressive and I recommend SSD (or RAID of SSDs) since to anyone who can afford it. Alain 2013/8/7 Andrey Ilinykh ailin...@gmail.com You still have the same amount of RAM, so you cache the same amount of data. I don't think you gain much here. On the other side, maintenance procedures (compaction, repair) may hit your 2CPU box. I wouldn't do it. Thank you, Andrey On Wed, Aug 7, 2013 at 10:24 AM, Paul Ingalls paulinga...@gmail.com wrote: Quick question about systems architecture. Would it be better to run 5 nodes with 7GB RAM and 4CPU's or 10 nodes with 3.5GB RAM and 2CPUS? I'm currently running the former, but am considering the latter. My goal would be to improve overall performance by spreading the IO across more disks. My currently cluster has low CPU utilization but does spend a good amount of time in iowait. Would moving to more smaller nodes help with that? Or would I run into trouble with the smaller ram and cpu? Thanks! Paul
JmxReporter.java (line 388) Error processing
Hi, I am new to Cassandra. I tried cassandra -f but i am getting the JmxReporter.java (line 388) Error processing errors. I downloaded binary version of apache-cassandra-1.2.8 which runs on Ubuntu 11.04 VM Virtual Box. I added jna.jar file to the library file too. Thanks for your help. Massoud Errors: INFO [main] 2013-08-08 07:30:53,610 CassandraDaemon.java (line 118) Logging initialized INFO [main] 2013-08-08 07:30:53,710 CassandraDaemon.java (line 142) 32bit JVM detected. It is recommended to run Cassandra on a 64bit JVM for better performance. INFO [main] 2013-08-08 07:30:53,716 CassandraDaemon.java (line 145) JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0 ERROR [main] 2013-08-08 07:30:53,717 CassandraDaemon.java (line 176) Your JVM is out of date. Please upgrade to the newest Oracle Java 6. INFO [main] 2013-08-08 07:30:53,727 CassandraDaemon.java (line 183) Heap size: 253558784/253558784 INFO [main] 2013-08-08 07:30:53,728 CassandraDaemon.java (line 184) Classpath: bin/../conf:bin/../build/ classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.2.8.jar:bin/../lib/apache-cassandra-clientutil-1.2.8.jar:bin/../lib/apache-cassandra-thrift-1.2.8.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.6.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:bin/../lib/guava-13.0.1.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jline-1.0.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/lz4-1.1.0.jar:bin/../lib/metrics-core-2.0.3.jar:bin/../lib/netty-3.5.9.Final.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.7.2.jar:bin/../lib/slf4j-log4j12-1.7.2.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.5.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar WARN [main] 2013-08-08 07:30:53,957 CLibrary.java (line 137) Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root. INFO [main] 2013-08-08 07:30:54,043 DatabaseDescriptor.java (line 132) Loading settings from file:/home/massoud/cassandra/apache-cassandra-1.2.8/conf/cassandra.yaml INFO [main] 2013-08-08 07:30:55,798 DatabaseDescriptor.java (line 150) Data files directories: [/var/lib/cassandra/data] INFO [main] 2013-08-08 07:30:55,798 DatabaseDescriptor.java (line 151) Commit log directory: /var/lib/cassandra/commitlog INFO [main] 2013-08-08 07:30:55,798 DatabaseDescriptor.java (line 191) DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard INFO [main] 2013-08-08 07:30:55,798 DatabaseDescriptor.java (line 205) disk_failure_policy is stop INFO [main] 2013-08-08 07:30:55,859 DatabaseDescriptor.java (line 273) Global memtable threshold is enabled at 80MB INFO [main] 2013-08-08 07:30:57,448 DatabaseDescriptor.java (line 401) Not using multi-threaded compaction INFO [main] 2013-08-08 07:31:00,295 CacheService.java (line 111) Initializing key cache with capacity of 12 MBs. INFO [main] 2013-08-08 07:31:00,432 CacheService.java (line 140) Scheduling key cache save to each 14400 seconds (going to save all keys). INFO [main] 2013-08-08 07:31:00,493 CacheService.java (line 154) Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO [main] 2013-08-08 07:31:00,599 CacheService.java (line 166) Scheduling row cache save to each 0 seconds (going to save all keys). org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=batchlog,name=ReadLatency javax.management.NotCompliantMBeanException: Attribute Count has more than one getter at com.sun.jmx.mbeanserver.MBeanAnalyzer.initMaps(Unknown Source) at com.sun.jmx.mbeanserver.MBeanAnalyzer.init(Unknown Source) at com.sun.jmx.mbeanserver.MBeanAnalyzer.analyzer(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.getAnalyzer(Unknown Source) at com.sun.jmx.mbeanserver.MBeanIntrospector.getPerInterface(Unknown Source) at com.sun.jmx.mbeanserver.MBeanSupport.init(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanSupport.init(Unknown Source) at com.sun.jmx.mbeanserver.Introspector.makeDynamicMBean(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(Unknown Source) at com.yammer.metrics.reporting.JmxReporter.registerBean(JmxReporter.java:451) at
Re: JmxReporter.java (line 388) Error processing
On Thu, Aug 8, 2013 at 9:35 AM, kohanm kohan.mass...@gmail.com wrote: I tried cassandra -f but i am getting the JmxReporter.java (line 388) Error processing errors. ERROR [main] 2013-08-08 07:30:53,717 CassandraDaemon.java (line 176) Your JVM is out of date. Please upgrade to the newest Oracle Java 6 Seems likely to be the relevant part of the log. =Rob
[RELEASE] Apache Cassandra 2.0.0 RC 1
The Cassandra team is pleased to announce the first release candidate for Apache Cassandra 2.0.0. As a release candidate, we're working to establish that this is relatively free of obvious bugs before calling it Done. Bottom line, if you aren't willing to test pre-production software and report bugs[3], you should probably wait for the final release. Source and binary downloads are available from the website. http://cassandra.apache.org/download The Debian repository will need to wait until Sylvain's return, I apologize (again) for the inconvenience. If you need the package before then, you can download and manually install it from: http://people.apache.org/~eevans As usual, please pay attention to the release notes[2] and let us know if you encounter any issues. Thanks, and Enjoy. [1]: http://goo.gl/ZRB5aW (CHANGES.txt) [2]: http://goo.gl/h55Zsq (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA -- Eric Evans eev...@sym-link.com
Re: [RELEASE] Apache Cassandra 2.0.0 RC 1
Is the plan to close the 1.x generation on 1.2.* branch and move to 2.0 or will 1.2 continue to 1.3, 1.4, etc while 2.0 also progresses? On Thu, Aug 8, 2013 at 1:44 PM, Eric Evans eev...@sym-link.com wrote: The Cassandra team is pleased to announce the first release candidate for Apache Cassandra 2.0.0. As a release candidate, we're working to establish that this is relatively free of obvious bugs before calling it Done. Bottom line, if you aren't willing to test pre-production software and report bugs[3], you should probably wait for the final release. Source and binary downloads are available from the website. http://cassandra.apache.org/download The Debian repository will need to wait until Sylvain's return, I apologize (again) for the inconvenience. If you need the package before then, you can download and manually install it from: http://people.apache.org/~eevans As usual, please pay attention to the release notes[2] and let us know if you encounter any issues. Thanks, and Enjoy. [1]: http://goo.gl/ZRB5aW (CHANGES.txt) [2]: http://goo.gl/h55Zsq (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA -- Eric Evans eev...@sym-link.com
Cassandra nodetool repair question
Afternoon, We are noticing nodetool repair processes are not completing after a weeks worth of time, and have resulted in some Cassandra nodes having more than one process running do to cron scheduled. We are also chasing some performance degradation after upgrading all nodes to version 1.2.8 last Friday and would like to resolve this multiple repairs running at once issue in an effort to troubleshoot our performance issues. We'd like to know more about what is happening with the repair option. Is there a way to gracefully terminate them or any adverse affect to killing the processes we should look out for? Thanks, -- Andy L
Re: JmxReporter.java (line 388) Error processing
Thanks Rob. Now I am getting listen_address in cassandra.yaml errors I mentioned before the Cassandra runs on VM VitualBox on windows 7 machine. On the ERROR text I changed the real ip address to MymachineIP. What should I put on the* listen_address:* in *cassandra.yaml* file? Thanks. short output: INFO 12:55:45,372 Cassandra version: 1.2.8 INFO 12:55:45,382 Thrift API version: 19.36.0 INFO 12:55:45,382 CQL supported versions: 2.0.0,3.0.5 (default: 3.0.5) INFO 12:55:45,677 Loading persisted ring state INFO 12:55:45,696 Starting up server gossip INFO 12:55:45,861 Enqueuing flush of Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,866 Writing Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,961 Completed flushing /var/lib/cassandra/data/system/local/system-local-ic-18-Data.db (240 bytes) for commitlog position ReplayPosition(segmentId=1375991740291, position=53949) INFO 12:55:46,663 GC for Copy: 292 ms for 1 collections, 7605200 used; max is 253558784 ERROR 12:55:46,994 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:444) at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:390) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:589) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 Fatal configuration error; unable to start server. See log for stacktrace. ERROR 12:55:47,059 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:519) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662)
Re: [RELEASE] Apache Cassandra 2.0.0 RC 1
[ David Ward ] Is the plan to close the 1.x generation on 1.2.* branch and move to 2.0 or will 1.2 continue to 1.3, 1.4, etc while 2.0 also progresses? The 1.2 branch will be continue to be maintained for sometime, but there will be no new releases = 1.3 and 2.0. On Thu, Aug 8, 2013 at 1:44 PM, Eric Evans eev...@sym-link.com wrote: The Cassandra team is pleased to announce the first release candidate for Apache Cassandra 2.0.0. As a release candidate, we're working to establish that this is relatively free of obvious bugs before calling it Done. Bottom line, if you aren't willing to test pre-production software and report bugs[3], you should probably wait for the final release. Source and binary downloads are available from the website. http://cassandra.apache.org/download The Debian repository will need to wait until Sylvain's return, I apologize (again) for the inconvenience. If you need the package before then, you can download and manually install it from: http://people.apache.org/~eevans As usual, please pay attention to the release notes[2] and let us know if you encounter any issues. Thanks, and Enjoy. [1]: http://goo.gl/ZRB5aW (CHANGES.txt) [2]: http://goo.gl/h55Zsq (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA -- Eric Evans eev...@sym-link.com -- Eric Evans eev...@sym-link.com
Re: JmxReporter.java (line 388) Error processing
What does ifconfig say? use eth0 or eth1's IP or whatever is static. And on that note, setup a static IP ( versus dhcp based ) it will make stopping/starting easier. On Thu, Aug 8, 2013 at 2:01 PM, kohanm kohan.mass...@gmail.com wrote: Thanks Rob. Now I am getting listen_address in cassandra.yaml errors I mentioned before the Cassandra runs on VM VitualBox on windows 7 machine. On the ERROR text I changed the real ip address to MymachineIP. What should I put on the* listen_address:* in *cassandra.yaml* file? Thanks. short output: INFO 12:55:45,372 Cassandra version: 1.2.8 INFO 12:55:45,382 Thrift API version: 19.36.0 INFO 12:55:45,382 CQL supported versions: 2.0.0,3.0.5 (default: 3.0.5) INFO 12:55:45,677 Loading persisted ring state INFO 12:55:45,696 Starting up server gossip INFO 12:55:45,861 Enqueuing flush of Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,866 Writing Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,961 Completed flushing /var/lib/cassandra/data/system/local/system-local-ic-18-Data.db (240 bytes) for commitlog position ReplayPosition(segmentId=1375991740291, position=53949) INFO 12:55:46,663 GC for Copy: 292 ms for 1 collections, 7605200 used; max is 253558784 ERROR 12:55:46,994 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:444) at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:390) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:589) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 Fatal configuration error; unable to start server. See log for stacktrace. ERROR 12:55:47,059 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:519) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662)
Re: [RELEASE] Apache Cassandra 2.0.0 RC 1
Congradulations on all your hard work! Can 1.2.5 run alongside of 2.0.0 RC 1? Thanks, -Tony From: Eric Evans eev...@sym-link.com To: user@cassandra.apache.org Sent: Thursday, August 8, 2013 1:44 PM Subject: [RELEASE] Apache Cassandra 2.0.0 RC 1 The Cassandra team is pleased to announce the first release candidate for Apache Cassandra 2.0.0. As a release candidate, we're working to establish that this is relatively free of obvious bugs before calling it Done. Bottom line, if you aren't willing to test pre-production software and report bugs[3], you should probably wait for the final release. Source and binary downloads are available from the website. http://cassandra.apache.org/download The Debian repository will need to wait until Sylvain's return, I apologize (again) for the inconvenience. If you need the package before then, you can download and manually install it from: http://people.apache.org/~eevans As usual, please pay attention to the release notes[2] and let us know if you encounter any issues. Thanks, and Enjoy. [1]: http://goo.gl/ZRB5aW(CHANGES.txt) [2]: http://goo.gl/h55Zsq(NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA -- Eric Evans eev...@sym-link.com
Re: JmxReporter.java (line 388) Error processing
Hi Dave Can you tell me more specific. Thanks, Here are ifconfig: ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:00:32:0c inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe00:320c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6564 errors:0 dropped:0 overruns:0 frame:0 TX packets:5588 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5649530 (5.6 MB) TX bytes:1069524 (1.0 MB) loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:20 errors:0 dropped:0 overruns:0 frame:0 TX packets:20 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1136 (1.1 KB) TX bytes:1136 (1.1 KB) On Thu, Aug 8, 2013 at 1:04 PM, David Ward da...@shareablee.com wrote: What does ifconfig say? use eth0 or eth1's IP or whatever is static. And on that note, setup a static IP ( versus dhcp based ) it will make stopping/starting easier. On Thu, Aug 8, 2013 at 2:01 PM, kohanm kohan.mass...@gmail.com wrote: Thanks Rob. Now I am getting listen_address in cassandra.yaml errors I mentioned before the Cassandra runs on VM VitualBox on windows 7 machine. On the ERROR text I changed the real ip address to MymachineIP. What should I put on the* listen_address:* in *cassandra.yaml* file? Thanks. short output: INFO 12:55:45,372 Cassandra version: 1.2.8 INFO 12:55:45,382 Thrift API version: 19.36.0 INFO 12:55:45,382 CQL supported versions: 2.0.0,3.0.5 (default: 3.0.5) INFO 12:55:45,677 Loading persisted ring state INFO 12:55:45,696 Starting up server gossip INFO 12:55:45,861 Enqueuing flush of Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,866 Writing Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,961 Completed flushing /var/lib/cassandra/data/system/local/system-local-ic-18-Data.db (240 bytes) for commitlog position ReplayPosition(segmentId=1375991740291, position=53949) INFO 12:55:46,663 GC for Copy: 292 ms for 1 collections, 7605200 used; max is 253558784 ERROR 12:55:46,994 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:444) at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:390) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:589) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 Fatal configuration error; unable to start server. See log for stacktrace. ERROR 12:55:47,059 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:519) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662) -- Massoud
Re: JmxReporter.java (line 388) Error processing
It maybe easier to manage virtualbox using vagrant ( http://www.vagrantup.com/ ). I am going to assume eth0 from your ifconfig output is a dhcp/dynamic IP address thats going to change every time you halt/start your virtualbox instance. Virtualbox is far outside the scope of C* so I'd recommend checking out the manual on Virtualbox networking http://www.virtualbox.org/manual/ch06.html plus using a dynamic IP for c* will probably yield undesirable results. On Thu, Aug 8, 2013 at 2:51 PM, kohanm kohan.mass...@gmail.com wrote: Hi Dave Can you tell me more specific. Thanks, Here are ifconfig: ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:00:32:0c inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe00:320c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6564 errors:0 dropped:0 overruns:0 frame:0 TX packets:5588 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5649530 (5.6 MB) TX bytes:1069524 (1.0 MB) loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:20 errors:0 dropped:0 overruns:0 frame:0 TX packets:20 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1136 (1.1 KB) TX bytes:1136 (1.1 KB) On Thu, Aug 8, 2013 at 1:04 PM, David Ward da...@shareablee.com wrote: What does ifconfig say? use eth0 or eth1's IP or whatever is static. And on that note, setup a static IP ( versus dhcp based ) it will make stopping/starting easier. On Thu, Aug 8, 2013 at 2:01 PM, kohanm kohan.mass...@gmail.com wrote: Thanks Rob. Now I am getting listen_address in cassandra.yaml errors I mentioned before the Cassandra runs on VM VitualBox on windows 7 machine. On the ERROR text I changed the real ip address to MymachineIP. What should I put on the* listen_address:* in *cassandra.yaml* file? Thanks. short output: INFO 12:55:45,372 Cassandra version: 1.2.8 INFO 12:55:45,382 Thrift API version: 19.36.0 INFO 12:55:45,382 CQL supported versions: 2.0.0,3.0.5 (default: 3.0.5) INFO 12:55:45,677 Loading persisted ring state INFO 12:55:45,696 Starting up server gossip INFO 12:55:45,861 Enqueuing flush of Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,866 Writing Memtable-local@28263177(251/251 serialized/live bytes, 9 ops) INFO 12:55:45,961 Completed flushing /var/lib/cassandra/data/system/local/system-local-ic-18-Data.db (240 bytes) for commitlog position ReplayPosition(segmentId=1375991740291, position=53949) INFO 12:55:46,663 GC for Copy: 292 ms for 1 collections, 7605200 used; max is 253558784 ERROR 12:55:46,994 Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:444) at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:390) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:589) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Unable to bind to address /MymachineIP:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2 Fatal configuration error; unable to start server. See log for stacktrace. ERROR 12:55:47,059 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:519) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662) -- Massoud
Re: JmxReporter.java (line 388) Error processing
Hi Dave I added *listen_address:*10.0.2.15 to the *cassandra.yaml *file and it worked. But i got another error please look at errors(short): ERROR 14:47:00,398 Exception encountered during startup java.lang.RuntimeException: No other nodes seen! Unable to bootstrap.If you intended to start a single-node cluster, you should make sure your broadcast_address (or listen_address) is listed as a seed. Otherwise, you need to determine why the seed being contacted has no knowledge of the rest of the cluster. Usually, this can be solved by giving all nodes the same seed list. Here are on details: INFO 14:46:20,654 Logging initialized INFO 14:46:20,724 32bit JVM detected. It is recommended to run Cassandra on a 64bit JVM for better performance. INFO 14:46:20,729 JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0_45 INFO 14:46:20,760 Heap size: 253558784/253558784 INFO 14:46:20,761 Classpath: bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.2.8.jar:bin/../lib/apache-cassandra-clientutil-1.2.8.jar:bin/../lib/apache-cassandra-thrift-1.2.8.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.6.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:bin/../lib/guava-13.0.1.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jline-1.0.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/lz4-1.1.0.jar:bin/../lib/metrics-core-2.0.3.jar:bin/../lib/netty-3.5.9.Final.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.7.2.jar:bin/../lib/slf4j-log4j12-1.7.2.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.5.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar WARN 14:46:21,038 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root. INFO 14:46:21,130 Loading settings from file:/home/massoud/cassandra/apache-cassandra-1.2.8/conf/cassandra.yaml INFO 14:46:22,323 Data files directories: [/var/lib/cassandra/data] INFO 14:46:22,323 Commit log directory: /var/lib/cassandra/commitlog INFO 14:46:22,325 DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard INFO 14:46:22,325 disk_failure_policy is stop INFO 14:46:22,390 Global memtable threshold is enabled at 80MB INFO 14:46:23,149 Not using multi-threaded compaction INFO 14:46:24,085 Initializing key cache with capacity of 12 MBs. INFO 14:46:24,139 Scheduling key cache save to each 14400 seconds (going to save all keys). INFO 14:46:24,152 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO 14:46:24,200 Scheduling row cache save to each 0 seconds (going to save all keys). INFO 14:46:24,781 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-3 (227 bytes) INFO 14:46:24,781 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-3 (227 bytes) INFO 14:46:24,900 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-2 (221 bytes) INFO 14:46:24,900 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-2 (221 bytes) INFO 14:46:24,953 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-7 (220 bytes) INFO 14:46:24,954 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-7 (220 bytes) INFO 14:46:24,983 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-4 (227 bytes) INFO 14:46:24,984 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-4 (227 bytes) INFO 14:46:25,011 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-6 (220 bytes) INFO 14:46:25,012 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-6 (220 bytes) INFO 14:46:25,021 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-1 (221 bytes) INFO 14:46:25,022 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-1 (221 bytes) INFO 14:46:25,053 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-5 (230 bytes) INFO 14:46:25,053 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-5 (230 bytes) INFO 14:46:25,166 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-5 (4596 bytes) INFO 14:46:25,168 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-5 (4596 bytes)
Re: Cassandra nodetool repair question
nodetool repair just triggers repair procedure. You can kill nodetool after start, it doesn't change anything. To stop repair you have to use nodetool stop VALIDATION|COMPACTION Thank you, Andrey On Thu, Aug 8, 2013 at 1:00 PM, Andy Losey and...@addthis.com wrote: Afternoon, We are noticing nodetool repair processes are not completing after a weeks worth of time, and have resulted in some Cassandra nodes having more than one process running do to cron scheduled. We are also chasing some performance degradation after upgrading all nodes to version 1.2.8 last Friday and would like to resolve this multiple repairs running at once issue in an effort to troubleshoot our performance issues. We'd like to know more about what is happening with the repair option. Is there a way to gracefully terminate them or any adverse affect to killing the processes we should look out for? Thanks, -- Andy L
Re: Cassandra nodetool repair question
But node might be streaming data as well, in that case only option is to restart node that started streaming operation Sent from my iPhone On Aug 8, 2013, at 5:56 PM, Andrey Ilinykh ailin...@gmail.com wrote: nodetool repair just triggers repair procedure. You can kill nodetool after start, it doesn't change anything. To stop repair you have to use nodetool stop VALIDATION|COMPACTION Thank you, Andrey On Thu, Aug 8, 2013 at 1:00 PM, Andy Losey and...@addthis.com wrote: Afternoon, We are noticing nodetool repair processes are not completing after a weeks worth of time, and have resulted in some Cassandra nodes having more than one process running do to cron scheduled. We are also chasing some performance degradation after upgrading all nodes to version 1.2.8 last Friday and would like to resolve this multiple repairs running at once issue in an effort to troubleshoot our performance issues. We'd like to know more about what is happening with the repair option. Is there a way to gracefully terminate them or any adverse affect to killing the processes we should look out for? Thanks, -- Andy L
Re: Any good GUI based tool to manage data in Casandra?
I've seen your email last week and saved it to ping you about DevCenter: http://www.datastax.com/dev/blog/introducing-datastax-devcenter. Please keep in mind that this is just the first version and we need suggestions/bug reports/etc. :- a) On Wed, Aug 7, 2013 at 11:18 AM, Tony Anecito adanec...@yahoo.com wrote: Thanks Nick for your reply. Good to know that. I knew OpsCenter was mainly schema management. Best Regards, -Tony -- *From:* Nick Bailey n...@datastax.com *To:* user@cassandra.apache.org *Sent:* Wednesday, August 7, 2013 12:04 PM *Subject:* Re: Any good GUI based tool to manage data in Casandra? OpsCenter allows CRUD of column families themselves (although not CQL3 column families). It only allows viewing the data inside column families though, no support for writing or updating. On Wed, Aug 7, 2013 at 12:54 PM, Aaron Morton aa...@thelastpickle.comwrote: I think on of the versions of ops centre has the feature http://www.datastax.com/what-we-offer/products-services/datastax-opscenter otherwise people use the cassandra-cli or cqlsh. Cheers - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 7/08/2013, at 1:28 AM, Tony Anecito adanec...@yahoo.com wrote: Thanks Aaron. I found that before I asked the question and Helenos seems the closest but it does not allow you to easily use CRUD like say SQL Server Management tools where you can get a list of say 1,000 records in a grid control and select rows for deletion or insert or update. I will look closer at that one since this is the reply from the team but if users on this email list have other suggestions please do not hesitate to reply. Many Thanks, -Tony *From:* Aaron Morton aa...@thelastpickle.com *To:* Cassandra User user@cassandra.apache.org *Sent:* Tuesday, August 6, 2013 1:38 AM *Subject:* Re: Any good GUI based tool to manage data in Casandra? There is a list here. http://wiki.apache.org/cassandra/Administration%20Tools Cheers - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com/ On 3/08/2013, at 6:19 AM, Tony Anecito adanec...@yahoo.com wrote: Hi All, Is there a GUI tool for managing data in Cassandra database? I have google and seen tools but they seem to be schema management or explorer to just view data. IT would be great to delete/inset rows or update values for a column via GUI. Thanks, -Tony -- :- a) Alex Popescu Sen. Product Manager @ DataStax @al3xandru
