Re: Bootstrapping a new node to a virtual node cluster
Just checking if this sorted it's self out? Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 10/02/2013, at 1:15 AM, Jouni Hartikainen jouni.hartikai...@reaktor.fi wrote: Hello all, I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration for the new node as well. My cluster looks like this before adding the new node: Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.154.111.49 GB256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1 UN 192.168.154.121.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1 UN 192.168.154.131.64 GB256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1 And corresponding gossipinfo: /192.168.154.12 RPC_ADDRESS:192.168.154.12 DC:datacenter1 STATUS:NORMAL,-1072164398478041156 LOAD:1.719425018E9 SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d NET_VERSION:6 HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6 SEVERITY:0.0 RELEASE_VERSION:1.2.1 RACK:rack1 /192.168.154.11 RPC_ADDRESS:192.168.154.11 DC:datacenter1 STATUS:NORMAL,-1158837144480089281 LOAD:1.514343678E9 SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d NET_VERSION:6 HOST_ID:234b82a4-3812-4261-adab-deb805942d63 SEVERITY:0.0 RELEASE_VERSION:1.2.1 RACK:rack1 /192.168.154.13 RPC_ADDRESS:192.168.154.13 DC:datacenter1 STATUS:NORMAL,-1135137292201587328 LOAD:1.765093695E9 SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d NET_VERSION:6 HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff SEVERITY:0.0 RELEASE_VERSION:1.2.1 RACK:rack1 I have now set the correct net addresses seeds in the cassandra.yaml of the new node (.14) and then started it with num_tokens set to 256 and initial_token commented out. Everything seems to go OK as I get the following prints on the log: On node 192.168.154.11: INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14 is now part of the cluster INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14 is now UP INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer to /192.168.154.14 And on node 192.168.154.14 (the new node): INFO 12:30:26,843 Loading persisted ring state INFO 12:30:26,846 Starting up server gossip WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This should happen exactly once per node). INFO 12:30:26,979 Starting Messaging Service on port 7000 INFO 12:30:27,014 JOINING: waiting for ring information INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap INFO 12:30:29,015 JOINING: getting bootstrap token INFO 12:30:29,157 JOINING: sleeping 3 ms for pending range setup INFO 12:30:59,159 JOINING: Starting to bootstrap... However, the new node does not show up in nodetool status (even if queried from the new node itself): Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.154.111.49 GB256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1 UN 192.168.154.121.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1 UN 192.168.154.131.64 GB256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1 It shows up in the gossip still: /192.168.154.12 RPC_ADDRESS:192.168.154.12 DC:datacenter1 STATUS:NORMAL,-1072164398478041156 LOAD:1.719430632E9 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6 SEVERITY:0.0 RELEASE_VERSION:1.2.1-SNAPSHOT RACK:rack1 /192.168.154.14 RPC_ADDRESS:192.168.154.14 DC:datacenter1 STATUS:BOOT,8077752099299332137 LOAD:105101.0 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110 RELEASE_VERSION:1.2.1-SNAPSHOT RACK:rack1 /192.168.154.11 RPC_ADDRESS:192.168.154.11 DC:datacenter1 STATUS:NORMAL,-1158837144480089281 LOAD:1.596505929E9 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:234b82a4-3812-4261-adab-deb805942d63 SEVERITY:0.0 RELEASE_VERSION:1.2.1-SNAPSHOT
Re: Bootstrapping a new node to a virtual node cluster
On Feb 12, 2013, at 18:58 , aaron morton aa...@thelastpickle.com wrote: Just checking if this sorted it's self out? Well, partially. :) Situation at the moment: 1) Is the new node not showing up in the nodetool status expected behavior? Not showing still, don't know if should. :) 2) Do I have to wait for all the data streaming to end until the new node starts to accept writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately after booting, but readable only after streaming has ended) This does not seem to be the case. Writes are indeed relayed to the bootstrapping node during the bootstrapping even if it is not shown as a part of the cluster. 3) nodetool info on the new node raises an exception Exception in thread main java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465) This is probably not normal? Still an issue. Should probably file a bug on this one? 4) Is it so that the virtual tokens for the new node are not assigned before the data streaming ends? (select * from peers; on .11 returns null for the tokens of .14) If so, how the existing nodes decide what data to stream to a new node? By reading the code it seems that the tokens are indeed generated early in the bootstrapping sequence and propagated (via gossip?) to the other nodes. However, I haven't still figured out why they are not shown in the list of peers when querying using CQL. FYI.. While playing with this I bumbed into a bug that prevents compactions from running while a node is in the bootstrapping state: https://issues.apache.org/jira/browse/CASSANDRA-5244 Might be an issue at least for those running clusters with lots of data per node (= long bootstrap times) and heavy write load (= lots of data to compacted). Cheers, -Jouni
Bootstrapping a new node to a virtual node cluster
Hello all, I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration for the new node as well. My cluster looks like this before adding the new node: Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.154.111.49 GB256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1 UN 192.168.154.121.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1 UN 192.168.154.131.64 GB256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1 And corresponding gossipinfo: /192.168.154.12 RPC_ADDRESS:192.168.154.12 DC:datacenter1 STATUS:NORMAL,-1072164398478041156 LOAD:1.719425018E9 SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d NET_VERSION:6 HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6 SEVERITY:0.0 RELEASE_VERSION:1.2.1 RACK:rack1 /192.168.154.11 RPC_ADDRESS:192.168.154.11 DC:datacenter1 STATUS:NORMAL,-1158837144480089281 LOAD:1.514343678E9 SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d NET_VERSION:6 HOST_ID:234b82a4-3812-4261-adab-deb805942d63 SEVERITY:0.0 RELEASE_VERSION:1.2.1 RACK:rack1 /192.168.154.13 RPC_ADDRESS:192.168.154.13 DC:datacenter1 STATUS:NORMAL,-1135137292201587328 LOAD:1.765093695E9 SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d NET_VERSION:6 HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff SEVERITY:0.0 RELEASE_VERSION:1.2.1 RACK:rack1 I have now set the correct net addresses seeds in the cassandra.yaml of the new node (.14) and then started it with num_tokens set to 256 and initial_token commented out. Everything seems to go OK as I get the following prints on the log: On node 192.168.154.11: INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14 is now part of the cluster INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14 is now UP INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer to /192.168.154.14 And on node 192.168.154.14 (the new node): INFO 12:30:26,843 Loading persisted ring state INFO 12:30:26,846 Starting up server gossip WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This should happen exactly once per node). INFO 12:30:26,979 Starting Messaging Service on port 7000 INFO 12:30:27,014 JOINING: waiting for ring information INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap INFO 12:30:29,015 JOINING: getting bootstrap token INFO 12:30:29,157 JOINING: sleeping 3 ms for pending range setup INFO 12:30:59,159 JOINING: Starting to bootstrap... However, the new node does not show up in nodetool status (even if queried from the new node itself): Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.154.111.49 GB256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1 UN 192.168.154.121.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1 UN 192.168.154.131.64 GB256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1 It shows up in the gossip still: /192.168.154.12 RPC_ADDRESS:192.168.154.12 DC:datacenter1 STATUS:NORMAL,-1072164398478041156 LOAD:1.719430632E9 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6 SEVERITY:0.0 RELEASE_VERSION:1.2.1-SNAPSHOT RACK:rack1 /192.168.154.14 RPC_ADDRESS:192.168.154.14 DC:datacenter1 STATUS:BOOT,8077752099299332137 LOAD:105101.0 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110 RELEASE_VERSION:1.2.1-SNAPSHOT RACK:rack1 /192.168.154.11 RPC_ADDRESS:192.168.154.11 DC:datacenter1 STATUS:NORMAL,-1158837144480089281 LOAD:1.596505929E9 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:234b82a4-3812-4261-adab-deb805942d63 SEVERITY:0.0 RELEASE_VERSION:1.2.1-SNAPSHOT RACK:rack1 /192.168.154.13 RPC_ADDRESS:192.168.154.13 DC:datacenter1 STATUS:NORMAL,-1135137292201587328 LOAD:1.765062944E9 SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 NET_VERSION:6 HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff SEVERITY:-6.776263578034403E-21 RELEASE_VERSION:1.2.1-SNAPSHOT RACK:rack1 I
