Re: Setting up a multi-node cluster
My first node is running and second node is not running in this case I tried telnet from second node to first node. Following is the stdout # telnet 10.96.10.207 7000 Trying 10.96.10.207... Connected to 10.96.10.207. Escape character is '^]'. quit Connection closed by foreign host. I see the same output for 7199 9160 ports. is this correct. I am not seeing the telnet session as something like telnet -- But when I tried with 7001. I am seeing the below output # telnet 10.96.10.207 7001 Trying 10.96.10.207... telnet: connect to address 10.96.10.207: Connection refused I see the same output for 61620 61621 ports -- Please suggest On Thu, Aug 29, 2013 at 11:23 AM, Andrey Ilinykh ailin...@gmail.com wrote: To be sure ports are open try to connect from one node to another: telnet node ip 7000 try all ports. Andrey On Wed, Aug 28, 2013 at 10:41 PM, Dinesh dinesh.gad...@gmail.com wrote: Hi John, I had my firewall disabled in both the nodes To make sure. I checked it # rcSuSEfirewall2 status Checking the status of SuSEfirewall2 unused if it's, on it says running Please suggest the further steps, where to look and troubleshoot, if you have any idea On Thu, Aug 29, 2013 at 2:34 AM, John Pyeatt john.pye...@singlewire.comwrote: Have you verified that your firewall is configured for the cassandra traffic. At the very least you need to make certain the following ports are open between nodes: 7000, 7001, 7199, 9160, 61620 and 61621. On Wed, Aug 28, 2013 at 12:36 AM, Dinesh dinesh.gad...@gmail.comwrote: In my case rpc_address in both the nodes is set to 0.0.0.0 which means it listens on all interfaces. it has a larger scope (to listen on all localhost, ipv4, hostnames, ipv6 addresses) than providing just the hostname/ipv4 addresses anyway I initially checked that, but it's the same exception I got in this case also On Wed, Aug 28, 2013 at 10:40 AM, Naresh Yadav nyadav@gmail.comwrote: You would need to configure rpc_address also with hostname/ips on both the nodes. Naresh On Wed, Aug 28, 2013 at 10:15 AM, Dinesh dinesh.gad...@gmail.comwrote: Hi, I am trying to setup a two node Cassandra cluster Able to start the first node, but not seeing the following exception while starting the second node ERROR 17:31:34,315 Exception encountered during startup java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Exception encountered during startup: Unable to contact any seeds! ERROR 17:31:34,322 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) = My yaml configuration files have these modified first node yaml --- initial_token: -9223372036854775808 # generated this using tokengen tool seeds: 10.96.19.207 # which is the IP of first node listen_address: 10.96.19.207 # which is the IP of first node itself rpc_address: 0.0.0.0 second node yaml initial_token: 0 seeds: 10.96.19.207 # which is the IP of first node listen_address: 10.96.10.223 # which is the IP of second node rpc_address: 0.0.0.0 == Can anyone please help me what went wrong with my configuration? Regards Dinesh -- Regards
Re: Setting up a multi-node cluster
Have you tried setting rpc_address to the same as listen_address ? Andy On 29 Aug 2013, at 07:47, Dinesh dinesh.gad...@gmail.commailto:dinesh.gad...@gmail.com wrote: My first node is running and second node is not running in this case I tried telnet from second node to first node. Following is the stdout # telnet 10.96.10.207 7000 Trying 10.96.10.207... Connected to 10.96.10.207. Escape character is '^]'. quit Connection closed by foreign host. I see the same output for 7199 9160 ports. is this correct. I am not seeing the telnet session as something like telnet -- But when I tried with 7001. I am seeing the below output # telnet 10.96.10.207 7001 Trying 10.96.10.207... telnet: connect to address 10.96.10.207http://10.96.10.207/: Connection refused I see the same output for 61620 61621 ports -- Please suggest On Thu, Aug 29, 2013 at 11:23 AM, Andrey Ilinykh ailin...@gmail.commailto:ailin...@gmail.com wrote: To be sure ports are open try to connect from one node to another: telnet node ip 7000 try all ports. Andrey On Wed, Aug 28, 2013 at 10:41 PM, Dinesh dinesh.gad...@gmail.commailto:dinesh.gad...@gmail.com wrote: Hi John, I had my firewall disabled in both the nodes To make sure. I checked it # rcSuSEfirewall2 status Checking the status of SuSEfirewall2 unused if it's, on it says running Please suggest the further steps, where to look and troubleshoot, if you have any idea On Thu, Aug 29, 2013 at 2:34 AM, John Pyeatt john.pye...@singlewire.commailto:john.pye...@singlewire.com wrote: Have you verified that your firewall is configured for the cassandra traffic. At the very least you need to make certain the following ports are open between nodes: 7000, 7001, 7199, 9160, 61620 and 61621. On Wed, Aug 28, 2013 at 12:36 AM, Dinesh dinesh.gad...@gmail.commailto:dinesh.gad...@gmail.com wrote: In my case rpc_address in both the nodes is set to 0.0.0.0 which means it listens on all interfaces. it has a larger scope (to listen on all localhost, ipv4, hostnames, ipv6 addresses) than providing just the hostname/ipv4 addresses anyway I initially checked that, but it's the same exception I got in this case also On Wed, Aug 28, 2013 at 10:40 AM, Naresh Yadav nyadav@gmail.commailto:nyadav@gmail.com wrote: You would need to configure rpc_address also with hostname/ips on both the nodes. Naresh On Wed, Aug 28, 2013 at 10:15 AM, Dinesh dinesh.gad...@gmail.commailto:dinesh.gad...@gmail.com wrote: Hi, I am trying to setup a two node Cassandra cluster Able to start the first node, but not seeing the following exception while starting the second node ERROR 17:31:34,315 Exception encountered during startup java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Exception encountered during startup: Unable to contact any seeds! ERROR 17:31:34,322 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) = My yaml configuration files have these modified first node yaml --- initial_token: -9223372036854775808 # generated this using tokengen tool seeds: 10.96.19.207 # which is the IP of first node listen_address: 10.96.19.207 # which is the IP of first node itself
Re: Setting up a multi-node cluster
Facing the same issue with rpc_address set to the same as listen address on both nodes Checked with having in first node - rpc_address: 10.96.10.207 in second node - rpc_address: 10.96.10.223 On Thu, Aug 29, 2013 at 12:40 PM, Andrew Cobley a.e.cob...@dundee.ac.ukwrote: Have you tried setting rpc_address to the same as listen_address ? Andy On 29 Aug 2013, at 07:47, Dinesh dinesh.gad...@gmail.com wrote: My first node is running and second node is not running in this case I tried telnet from second node to first node. Following is the stdout # telnet 10.96.10.207 7000 Trying 10.96.10.207... Connected to 10.96.10.207. Escape character is '^]'. quit Connection closed by foreign host. I see the same output for 7199 9160 ports. is this correct. I am not seeing the telnet session as something like telnet -- But when I tried with 7001. I am seeing the below output # telnet 10.96.10.207 7001 Trying 10.96.10.207... telnet: connect to address 10.96.10.207: Connection refused I see the same output for 61620 61621 ports -- Please suggest On Thu, Aug 29, 2013 at 11:23 AM, Andrey Ilinykh ailin...@gmail.comwrote: To be sure ports are open try to connect from one node to another: telnet node ip 7000 try all ports. Andrey On Wed, Aug 28, 2013 at 10:41 PM, Dinesh dinesh.gad...@gmail.com wrote: Hi John, I had my firewall disabled in both the nodes To make sure. I checked it # rcSuSEfirewall2 status Checking the status of SuSEfirewall2 unused if it's, on it says running Please suggest the further steps, where to look and troubleshoot, if you have any idea On Thu, Aug 29, 2013 at 2:34 AM, John Pyeatt john.pye...@singlewire.com wrote: Have you verified that your firewall is configured for the cassandra traffic. At the very least you need to make certain the following ports are open between nodes: 7000, 7001, 7199, 9160, 61620 and 61621. On Wed, Aug 28, 2013 at 12:36 AM, Dinesh dinesh.gad...@gmail.comwrote: In my case rpc_address in both the nodes is set to 0.0.0.0 which means it listens on all interfaces. it has a larger scope (to listen on all localhost, ipv4, hostnames, ipv6 addresses) than providing just the hostname/ipv4 addresses anyway I initially checked that, but it's the same exception I got in this case also On Wed, Aug 28, 2013 at 10:40 AM, Naresh Yadav nyadav@gmail.comwrote: You would need to configure rpc_address also with hostname/ips on both the nodes. Naresh On Wed, Aug 28, 2013 at 10:15 AM, Dinesh dinesh.gad...@gmail.comwrote: Hi, I am trying to setup a two node Cassandra cluster Able to start the first node, but not seeing the following exception while starting the second node ERROR 17:31:34,315 Exception encountered during startup java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:451) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) Exception encountered during startup: Unable to contact any seeds! ERROR 17:31:34,322 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:370) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88) = My yaml configuration files have these modified first node yaml --- initial_token: -9223372036854775808 # generated this using tokengen tool
Re: How many seed nodes should I use?
On 29 August 2013 01:55, Ike Walker ike.wal...@flite.com wrote: What is the best practice for how many seed nodes to have in a Cassandra cluster? I remember reading a recommendation of 2 seeds per datacenter in Datastax documentation for 0.7, but I'm interested to know what other people are doing these days, especially in AWS. I'm running a cluster of 12 nodes at AWS. Each node runs Cassandra 1.2.5 on an m1.xlarge EC2 instance, and they are spread across 3 availability zones within a single region. To keep things simple I currently have all 12 nodes listed as seeds. That seems like overkill to me, but I don't know the pros and cons of too many or too few seeds. Seeds are used for bootstrapping a new node so it can discover the others. Existing nodes store a list of the other nodes it has seen so doesn't need the seeds each time it starts up. Seeds are treated slightly differently in gossip though to ensure a node keeps trying to connect to seeds in case of a partition. The best recommendations are to use the same seed list on each node and just a few. More than your replication factor is almost certainly too many, but the cost of too many is very little. Richard.
Cassandra-shuffle fails
Hi, We recently upgraded from version 1.1 to 1.2 It all went well, including setting up vnodes, but shuffle fails. We have 2 nodes, hosted on Amazon AWS The steps we took (on each of our nodes) are pretty straight forward: 1. upgrade binaries 2. adjust cassandra.yaml (keep token) 3. nodetool upgradesstables 4. change cassandra.yaml to vnodes rather than tokens 5. restart cassandra 6. cassandra-shuffle create. All the above went fine. However, the following fails: cassandra-shuffle enable *Failed to enable shuffling on 10.194.230.175!* Note: 1. The failure is immediate, and consistent. 2. Calling shuffle create on either node prepares the shuffle files for both. 3. I made sure both servers are communicating fine on both 9160 and 7199. Any help will be greatly appreciated. Tamar Tamar Rosen Senior Data Architect Correlor.com
CustomTThreadPoolServer.java: Error occurred during processing of message.
I suddenly started to encounter this weird issue after writing some data to Cassandra. Didn't know exactly what was written before this or due to which this started happening. ERROR [pool-2-thread-30] 2013-08-29 19:55:24,778 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message. java.lang.StringIndexOutOfBoundsException: String index out of range: -2147418111 at java.lang.String.checkBounds(String.java:397) at java.lang.String.init(String.java:442) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339) at org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [pool-2-thread-31] 2013-08-29 19:55:24,910 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message. java.lang.StringIndexOutOfBoundsException: String index out of range: -2147418111 at java.lang.String.checkBounds(String.java:397) at java.lang.String.init(String.java:442) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339) at org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Any ideas ??
Re: CustomTThreadPoolServer.java: Error occurred during processing of message.
Running Cassandra (1.0.0 final) single node with default configurations on Windows dev machine. Using Hector. On Thu, Aug 29, 2013 at 10:50 PM, Ertio Lew ertio...@gmail.com wrote: I suddenly started to encounter this weird issue after writing some data to Cassandra. Didn't know exactly what was written before this or due to which this started happening. ERROR [pool-2-thread-30] 2013-08-29 19:55:24,778 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message. java.lang.StringIndexOutOfBoundsException: String index out of range: - 2147418111 at java.lang.String.checkBounds(String.java:397) at java.lang.String.init(String.java:442) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339) at org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [pool-2-thread-31] 2013-08-29 19:55:24,910 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message. java.lang.StringIndexOutOfBoundsException: String index out of range: - 2147418111 at java.lang.String.checkBounds(String.java:397) at java.lang.String.init(String.java:442) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339) at org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Any ideas ??
Re: CustomTThreadPoolServer.java: Error occurred during processing of message.
On Thu, Aug 29, 2013 at 10:24 AM, Ertio Lew ertio...@gmail.com wrote: Running Cassandra (1.0.0 final) single node with default configurations on Windows dev machine. Using Hector. Why are you running such an insanely old version of Cassandra in dev? =Rob
Flush writer all time blocked
I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. http://pastebin.com/GAiu2q74 Thanks,SC
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. =Rob
RE: Flush writer all time blocked
Thanks Rob. Will it contribute to any performance problems? Thanks,SC Date: Thu, 29 Aug 2013 10:57:30 -0700 Subject: Re: Flush writer all time blocked From: rc...@eventbrite.com To: user@cassandra.apache.org On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. =Rob
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 11:04 AM, S C as...@outlook.com wrote: Thanks Rob. Will it contribute to any performance problems? Yes, though perhaps more accurate would be that it is often an effect of performance problems. In general Cassandra flushes memtables because it believes it needs to. If it is unable to do so, that's usually bad. =Rob
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 1:57 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Ken
RE: Flush writer all time blocked
Ken, What queue size are you referring to? Thanks,SC From: ken.hanc...@schange.com Date: Thu, 29 Aug 2013 14:21:04 -0400 Subject: Re: Flush writer all time blocked To: user@cassandra.apache.org On Thu, Aug 29, 2013 at 1:57 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Ken
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 11:21 AM, Ken Hancock ken.hanc...@schange.comwrote: I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Periodic flush? Between 1.0 and 2.0, there is no time based flush mechanism for memtables. FWIW, I'm glad it is being added back in 2.0, as operators often want to be able to assert how long it will take for any given write to make it to disk. https://issues.apache.org/jira/browse/CASSANDRA-4237 (background at https://issues.apache.org/jira/browse/CASSANDRA-2427 ) =Rob
Re: Flush writer all time blocked
# the number of full memtables to allow pending flush, that is, # waiting for a writer thread. At a minimum, this should be set to # the maximum number of secondary indexes created on a single CF. memtable_flush_queue_size: 4 There was an interesting thread a while back: http://mail-archives.apache.org/mod_mbox/cassandra-user/201307.mbox/%3c17c39fe466076c46b6e83f129c7b19ce2e7ec...@hkxprd0310mb352.apcprd03.prod.outlook.com%3E On Thu, Aug 29, 2013 at 2:23 PM, S C as...@outlook.com wrote: Ken, What queue size are you referring to? Thanks, SC From: ken.hanc...@schange.com Date: Thu, 29 Aug 2013 14:21:04 -0400 Subject: Re: Flush writer all time blocked To: user@cassandra.apache.org On Thu, Aug 29, 2013 at 1:57 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Ken -- Ken Hancock | System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC Office: +1 (978) 889-3329 | ken.hanc...@schange.com | hancockks | hancockks This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International.
CQL3 wide row and slow inserts - is there a single insert alternative?
Hi all, We're using a Cassandra table to store search results in a table/column family that that look like this: ++-+-+-+ || 0 | 1 | 2 | ... ++-+-+-+ | row_id | text... | text... | text... | ... The column name is the index # (an integer) of the location in the overall result set. The value is the result at that particular index. This is great because pagination becomes a simple slice query on the column name. Large result sets are split into multiple rows - we're limiting row size on disk to be around 6 or 7 MB. For our particular result entries, this means we can get around 50,000 columns in a single row. When we create the rows, we have the entire data available in the application at the time the row insert is necessary. Using CQL3, an initial implementation had one INSERT statement per column. This was killing performance (not to mention the # of tombstones it created). Here's the CQL3 table definition: create table query_results ( row_id text, shard_num int, list_index int, result text, primary key (row_id, shard_num), list_index)) with compact storage (the row key is row_id + shard_num. The 'cluster column' is list_index). I don't want to execute 50,000 INSERT statements for a single row. We have all of the data up front - I want to execute a single INSERT. Is this possible? We're using the Datastax Java Driver. Thanks for any help! Les
successful use of shuffle?
Hi! I've been wondering... is there anyone in the cassandra-user audience who has used shuffle feature successfully on a non-toy-or-testing cluster? If so, could you describe the experience you had and any problems you encountered? Thanks! =Rob
Truncate question
Do we have to run nodetool repair or nodetool cleanup after Truncating a Column Family? Thanks,SC
Re: Truncate question
On Thu, Aug 29, 2013 at 3:48 PM, S C as...@outlook.com wrote: Do we have to run nodetool repair or nodetool cleanup after Truncating a Column Family? No. Why would you? =Rob
Re: Truncate question
No. Andrey On Thu, Aug 29, 2013 at 3:48 PM, S C as...@outlook.com wrote: Do we have to run nodetool repair or nodetool cleanup after Truncating a Column Family? Thanks, SC
Re: Cluster Management
Hi Particia, Thank you for the feedback. It has been helpful. On Tue, Aug 27, 2013 at 12:02 AM, Patricia Gorla gorla.patri...@gmail.comwrote: Anthony, We use a number of tools to manage our Cassandra cluster. * Datastax OpsCenter [0] for at a glance information, and trending statistics. You can also run operations through here, though I prefer to use nodetool for any mutative operation. * nodetool for ad hoc status checks, and day-to-day node management. * puppet for setup and initialization For example, if I want to make some changes to the configuration file that resides on each node, is there a tool that will propagate the change to each node? For this, we use puppet to manage any changes to the configurations (which are stored in git). We initially had Cassandra auto-restart when the configuration changed, but you might not want the node to automatically join a cluster, so we turned this off. Puppet was the first thing that came to mind for us as well. In addition, we had the same thought about auto-restarting nodes when the configuration is changed. If a configuration on all the nodes is changed, we would want to restart one node at a time and wait for it to rejoin before restarting the next one. I am assuming in a case like this, you then manually perform the restart operation for each node? Another example is if I want to have a rolling repair (nodetool repair -pr) and clean up running on my cluster, is there a tool that will help manage/configure that? Multiple commands to the cluster are sent via clusterssh [1] (cssh for OS X). I can easily choose which nodes to control, and run those in sync. For any rolling procedures, we send commands one at a time, though we've considered sending some of these tasks to cron. Thanks again for the tip! This is quite interesting; it may help to solve our immediate problem for now. Regards, Anthony Hope this helps. Cheers, Patricia [0] http://planetcassandra.org/Download/DataStaxCommunityEdition [1] http://sourceforge.net/projects/clusterssh/
Re: Cluster Management
Thanks Nate! We will look into this one to see if we can use it. Regards, Anthony On Tue, Aug 27, 2013 at 12:22 AM, Nate McCall n...@thelastpickle.comwrote: For example, if I want to make some changes to the configuration file that resides on each node, is there a tool that will propagate the change to each node? You may also want to take a look at Priam from the Netflix folks: https://github.com/Netflix/Priam Assumes AWS (though some of this is becoming more plug-able).
Re: Cluster Management
An alternative to cssh is fabric. It's very flexible in that you can automate almost any repetitive task that you'd send to machines in a cluster, and it's written in python, meaning if you're in AWS you can mix it with boto to automate pretty much anything you want. On Thu, Aug 29, 2013 at 4:25 PM, Anthony Grasso anthony.gra...@gmail.comwrote: Hi Particia, Thank you for the feedback. It has been helpful. On Tue, Aug 27, 2013 at 12:02 AM, Patricia Gorla gorla.patri...@gmail.com wrote: Anthony, We use a number of tools to manage our Cassandra cluster. * Datastax OpsCenter [0] for at a glance information, and trending statistics. You can also run operations through here, though I prefer to use nodetool for any mutative operation. * nodetool for ad hoc status checks, and day-to-day node management. * puppet for setup and initialization For example, if I want to make some changes to the configuration file that resides on each node, is there a tool that will propagate the change to each node? For this, we use puppet to manage any changes to the configurations (which are stored in git). We initially had Cassandra auto-restart when the configuration changed, but you might not want the node to automatically join a cluster, so we turned this off. Puppet was the first thing that came to mind for us as well. In addition, we had the same thought about auto-restarting nodes when the configuration is changed. If a configuration on all the nodes is changed, we would want to restart one node at a time and wait for it to rejoin before restarting the next one. I am assuming in a case like this, you then manually perform the restart operation for each node? Another example is if I want to have a rolling repair (nodetool repair -pr) and clean up running on my cluster, is there a tool that will help manage/configure that? Multiple commands to the cluster are sent via clusterssh [1] (cssh for OS X). I can easily choose which nodes to control, and run those in sync. For any rolling procedures, we send commands one at a time, though we've considered sending some of these tasks to cron. Thanks again for the tip! This is quite interesting; it may help to solve our immediate problem for now. Regards, Anthony Hope this helps. Cheers, Patricia [0] http://planetcassandra.org/Download/DataStaxCommunityEdition [1] http://sourceforge.net/projects/clusterssh/ -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: Cluster Management
On Thu, Aug 29, 2013 at 4:25 PM, Anthony Grasso anthony.gra...@gmail.comwrote: Puppet was the first thing that came to mind for us as well. In addition, we had the same thought about auto-restarting nodes when the configuration is changed. If a configuration on all the nodes is changed, we would want to restart one node at a time and wait for it to rejoin before restarting the next one. I am assuming in a case like this, you then manually perform the restart operation for each node? FYI, potentially relevant : https://issues.apache.org/jira/browse/CASSANDRA-2356 =Rob
Re: Truncate question
You would, however, want to clear the snapshot folder afterword, right? I thought that truncate, like drop table, created a snapshot (unless that feature had been disabled in your yaml. On Thu, Aug 29, 2013 at 6:51 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 3:48 PM, S C as...@outlook.com wrote: Do we have to run nodetool repair or nodetool cleanup after Truncating a Column Family? No. Why would you? =Rob
Re: Setting up a multi-node cluster
I enabled the debug level for logging. following is the debug entry for the same exception from node2. Someone please help me DEBUG 14:53:58,372 Connection version 6 from /10.96.10.105 DEBUG 14:53:58,374 Upgrading incoming connection to be compressed DEBUG 14:53:58,378 Max version for /10.96.10.105 is 6 DEBUG 14:53:58,379 Setting version 6 for /10.96.10.105 DEBUG 14:53:58,380 set version for /10.96.10.105 to 6 DEBUG 14:53:58,380 Reseting version for /10.96.10.105 DEBUG 14:53:58,683 error writing to rhlcas1/10.96.10.105 java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336) at java.nio.channels.Channels.writeFullyImpl(Channels.java:59) at java.nio.channels.Channels.writeFully(Channels.java:81) at java.nio.channels.Channels.access$000(Channels.java:47) at java.nio.channels.Channels$1.write(Channels.java:155) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:95) at java.io.DataOutputStream.flush(DataOutputStream.java:106) at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:205) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:149) On Thu, Aug 29, 2013 at 1:48 PM, Dinesh dinesh.gad...@gmail.com wrote: Facing the same issue with rpc_address set to the same as listen address on both nodes Checked with having in first node - rpc_address: 10.96.10.207 in second node - rpc_address: 10.96.10.223 On Thu, Aug 29, 2013 at 12:40 PM, Andrew Cobley a.e.cob...@dundee.ac.ukwrote: Have you tried setting rpc_address to the same as listen_address ? Andy On 29 Aug 2013, at 07:47, Dinesh dinesh.gad...@gmail.com wrote: My first node is running and second node is not running in this case I tried telnet from second node to first node. Following is the stdout # telnet 10.96.10.207 7000 Trying 10.96.10.207... Connected to 10.96.10.207. Escape character is '^]'. quit Connection closed by foreign host. I see the same output for 7199 9160 ports. is this correct. I am not seeing the telnet session as something like telnet -- But when I tried with 7001. I am seeing the below output # telnet 10.96.10.207 7001 Trying 10.96.10.207... telnet: connect to address 10.96.10.207: Connection refused I see the same output for 61620 61621 ports -- Please suggest On Thu, Aug 29, 2013 at 11:23 AM, Andrey Ilinykh ailin...@gmail.comwrote: To be sure ports are open try to connect from one node to another: telnet node ip 7000 try all ports. Andrey On Wed, Aug 28, 2013 at 10:41 PM, Dinesh dinesh.gad...@gmail.comwrote: Hi John, I had my firewall disabled in both the nodes To make sure. I checked it # rcSuSEfirewall2 status Checking the status of SuSEfirewall2 unused if it's, on it says running Please suggest the further steps, where to look and troubleshoot, if you have any idea On Thu, Aug 29, 2013 at 2:34 AM, John Pyeatt john.pye...@singlewire.com wrote: Have you verified that your firewall is configured for the cassandra traffic. At the very least you need to make certain the following ports are open between nodes: 7000, 7001, 7199, 9160, 61620 and 61621. On Wed, Aug 28, 2013 at 12:36 AM, Dinesh dinesh.gad...@gmail.comwrote: In my case rpc_address in both the nodes is set to 0.0.0.0 which means it listens on all interfaces. it has a larger scope (to listen on all localhost, ipv4, hostnames, ipv6 addresses) than providing just the hostname/ipv4 addresses anyway I initially checked that, but it's the same exception I got in this case also On Wed, Aug 28, 2013 at 10:40 AM, Naresh Yadav nyadav@gmail.comwrote: You would need to configure rpc_address also with hostname/ips on both the nodes. Naresh On Wed, Aug 28, 2013 at 10:15 AM, Dinesh dinesh.gad...@gmail.comwrote: Hi, I am trying to setup a two node Cassandra cluster Able to start the first node, but not seeing the following exception while starting the second node ERROR 17:31:34,315 Exception encountered during startup java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:947) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:716) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:554) at
