Re: Apache Cassandra meetup @ Instagram HQ
Will live stream be available for this event? Also I wish slides can be shared. Thanks! On 2/19/19 10:40 AM, Shaurya Gupta wrote: Hi, This looks very interesting to me. Can I attend this remotely? Thanks Shaurya On Tue, Feb 19, 2019 at 5:37 AM dinesh.jo...@yahoo.com.INVALID wrote: Hi all, Apologies for the cross-post. In case you're in the SF Bay Area, Instagram is hosting a meetup. Interesting talks on Cassandra Traffic management, Cassandra on Kubernetes. See details in the attached link - https://www.eventbrite.com/e/cassandra-traffic-management-at-instagram-cassandra-and-k8s-with-instaclustr-tickets-54986803008 Thanks, Dinesh -- Shaurya Gupta - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Apache Cassandra meetup @ Instagram HQ
Hi, This looks very interesting to me. Can I attend this remotely? Thanks Shaurya On Tue, Feb 19, 2019 at 5:37 AM dinesh.jo...@yahoo.com.INVALID wrote: > Hi all, > > Apologies for the cross-post. In case you're in the SF Bay Area, Instagram > is hosting a meetup. Interesting talks on Cassandra Traffic management, > Cassandra on Kubernetes. See details in the attached link - > > > https://www.eventbrite.com/e/cassandra-traffic-management-at-instagram-cassandra-and-k8s-with-instaclustr-tickets-54986803008 > > Thanks, > > Dinesh > -- Shaurya Gupta
Apache Cassandra meetup @ Instagram HQ
Hi all, Apologies for the cross-post. In case you're in the SF Bay Area, Instagram is hosting a meetup. Interesting talks on Cassandra Traffic management, Cassandra on Kubernetes. See details in the attached link - https://www.eventbrite.com/e/cassandra-traffic-management-at-instagram-cassandra-and-k8s-with-instaclustr-tickets-54986803008 Thanks, Dinesh
RE: Bootstrap stuck in JOINING state?
https://stackoverflow.com/questions/39823972/cassandra-node-cant-complete-jo ining-operation https://stackoverflow.com/questions/27251504/cassandra-2-1-2-node-stuck-on-j oining-the-cluster -Original Message- From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID] Sent: Monday, February 18, 2019 7:43 AM To: user@cassandra.apache.org Subject: RE: Bootstrap stuck in JOINING state? Hi Troels, Stackoverflow.com is a good resource for such situations. Have you seen these posts? I think they are same Cassandra version: https://stackoverflow.com/questions/39823972/cassandra-node-cant-complete-jo ining-operation https://stackoverflow.com/questions/27251504/cassandra-2-1-2-node-stuck-on-j oining-the-cluster Kenneth Brotman -Original Message- From: Troels Arvin [mailto:tro...@arvin.dk] Sent: Monday, February 18, 2019 2:54 AM To: user@cassandra.apache.org Subject: Bootstrap stuck in JOINING state? Hello, Nine days ago, I bootstrapped an Cassandra node, let's call it 10.1.2.11. The node is part of a six-node Cassandra 2.1.11 cluster spread across two datacenters. The bootstrapping was conducted the following way on the node: 1. Shut down Cassandra. 2. Deleted its data. 3. Temporarily adjusted its configuration to be in bootstrapping node: - In cassandra-env.sh: - JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=ipaddressofnode" - In cassandra.yaml: - auto_bootstrap: true - Made sure the nodes' IP address was not listed in seed_provider 4. Started Cassandra again. That seemed to go well: In system.log the following was seen: INFO [main] 2019-02-08 14:49:17,223 StorageService.java:1120 - JOINING: Starting to bootstrap... No errors seen in system.log. A graph of the server's data filesystem shows fine growth during the next ~24 hours. However, it seems the joining is stuck without completing. Parts of output from "nodetool netstats", note the "JOINING" mode: [cas@nodename ~]$ nodetool netstats Mode: JOINING Bootstrap 53d1a8a0-2ba8-11e9-9212-2d89290b6701 /10.1.2.10 /10.1.1.10 /10.1.1.12 /10.1.2.12 Receiving 225 files, 461265115573 bytes total. Already received 225 files, 461265115573 bytes total /data/cassandra/ksname/...db 43392372/43392372 bytes(100%) received from idx:0/10.1.2.12 [... lots of lines like this ...] /data/cassandra/ksname/...db 131825151/131825151 bytes(100%) received from idx:0/10.1.2.12 /10.1.1.11 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Commandsn/a 0 5 Responses n/a 03153171 All the "...received from..." lines show 100% bytes received. Parts of output from "nodetool status": [cas@nodename ~]$ nodetool status Datacenter: DC1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address LoadTokens Owns Host ID Rack UN 10.1.1.12 454.14 GB 256 ? c100e77d-... RAC1 UN 10.1.1.11 449.99 GB 256 ? 0e06dc69-... RAC1 UN 10.1.1.10 725.81 GB 256 ? f266489b-... RAC1 Datacenter: DC2 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address LoadTokens Owns Host ID Rack UN 10.1.2.12 474.59 GB 256 ? 89b71027-... RAC1 UN 10.1.2.11 455.1 GB 256 ? c08b8e99-... RAC1 UN 10.1.2.10 475 GB 256 ? e17bae55-... RAC1 How come "nodetool netstats" is still showing "Mode: JOINING"? Is there a way I can push the node to complete its joining? -- Kind regards, Troels Arvin - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
RE: Bootstrap stuck in JOINING state?
Hi Troels, Stackoverflow.com is a good resource for such situations. Have you seen these posts? I think they are same Cassandra version: https://stackoverflow.com/questions/39823972/cassandra-node-cant-complete-jo ining-operation https://stackoverflow.com/questions/27251504/cassandra-2-1-2-node-stuck-on-j oining-the-cluster Kenneth Brotman -Original Message- From: Troels Arvin [mailto:tro...@arvin.dk] Sent: Monday, February 18, 2019 2:54 AM To: user@cassandra.apache.org Subject: Bootstrap stuck in JOINING state? Hello, Nine days ago, I bootstrapped an Cassandra node, let's call it 10.1.2.11. The node is part of a six-node Cassandra 2.1.11 cluster spread across two datacenters. The bootstrapping was conducted the following way on the node: 1. Shut down Cassandra. 2. Deleted its data. 3. Temporarily adjusted its configuration to be in bootstrapping node: - In cassandra-env.sh: - JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=ipaddressofnode" - In cassandra.yaml: - auto_bootstrap: true - Made sure the nodes' IP address was not listed in seed_provider 4. Started Cassandra again. That seemed to go well: In system.log the following was seen: INFO [main] 2019-02-08 14:49:17,223 StorageService.java:1120 - JOINING: Starting to bootstrap... No errors seen in system.log. A graph of the server's data filesystem shows fine growth during the next ~24 hours. However, it seems the joining is stuck without completing. Parts of output from "nodetool netstats", note the "JOINING" mode: [cas@nodename ~]$ nodetool netstats Mode: JOINING Bootstrap 53d1a8a0-2ba8-11e9-9212-2d89290b6701 /10.1.2.10 /10.1.1.10 /10.1.1.12 /10.1.2.12 Receiving 225 files, 461265115573 bytes total. Already received 225 files, 461265115573 bytes total /data/cassandra/ksname/...db 43392372/43392372 bytes(100%) received from idx:0/10.1.2.12 [... lots of lines like this ...] /data/cassandra/ksname/...db 131825151/131825151 bytes(100%) received from idx:0/10.1.2.12 /10.1.1.11 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Commandsn/a 0 5 Responses n/a 03153171 All the "...received from..." lines show 100% bytes received. Parts of output from "nodetool status": [cas@nodename ~]$ nodetool status Datacenter: DC1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address LoadTokens Owns Host ID Rack UN 10.1.1.12 454.14 GB 256 ? c100e77d-... RAC1 UN 10.1.1.11 449.99 GB 256 ? 0e06dc69-... RAC1 UN 10.1.1.10 725.81 GB 256 ? f266489b-... RAC1 Datacenter: DC2 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address LoadTokens Owns Host ID Rack UN 10.1.2.12 474.59 GB 256 ? 89b71027-... RAC1 UN 10.1.2.11 455.1 GB 256 ? c08b8e99-... RAC1 UN 10.1.2.10 475 GB 256 ? e17bae55-... RAC1 How come "nodetool netstats" is still showing "Mode: JOINING"? Is there a way I can push the node to complete its joining? -- Kind regards, Troels Arvin - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Bootstrap stuck in JOINING state?
Hello, Nine days ago, I bootstrapped an Cassandra node, let's call it 10.1.2.11. The node is part of a six-node Cassandra 2.1.11 cluster spread across two datacenters. The bootstrapping was conducted the following way on the node: 1. Shut down Cassandra. 2. Deleted its data. 3. Temporarily adjusted its configuration to be in bootstrapping node: - In cassandra-env.sh: - JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=ipaddressofnode" - In cassandra.yaml: - auto_bootstrap: true - Made sure the nodes' IP address was not listed in seed_provider 4. Started Cassandra again. That seemed to go well: In system.log the following was seen: INFO [main] 2019-02-08 14:49:17,223 StorageService.java:1120 - JOINING: Starting to bootstrap... No errors seen in system.log. A graph of the server's data filesystem shows fine growth during the next ~24 hours. However, it seems the joining is stuck without completing. Parts of output from "nodetool netstats", note the "JOINING" mode: [cas@nodename ~]$ nodetool netstats Mode: JOINING Bootstrap 53d1a8a0-2ba8-11e9-9212-2d89290b6701 /10.1.2.10 /10.1.1.10 /10.1.1.12 /10.1.2.12 Receiving 225 files, 461265115573 bytes total. Already received 225 files, 461265115573 bytes total /data/cassandra/ksname/...db 43392372/43392372 bytes(100%) received from idx:0/10.1.2.12 [... lots of lines like this ...] /data/cassandra/ksname/...db 131825151/131825151 bytes(100%) received from idx:0/10.1.2.12 /10.1.1.11 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Commandsn/a 0 5 Responses n/a 03153171 All the "...received from..." lines show 100% bytes received. Parts of output from "nodetool status": [cas@nodename ~]$ nodetool status Datacenter: DC1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address LoadTokens Owns Host ID Rack UN 10.1.1.12 454.14 GB 256 ? c100e77d-... RAC1 UN 10.1.1.11 449.99 GB 256 ? 0e06dc69-... RAC1 UN 10.1.1.10 725.81 GB 256 ? f266489b-... RAC1 Datacenter: DC2 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address LoadTokens Owns Host ID Rack UN 10.1.2.12 474.59 GB 256 ? 89b71027-... RAC1 UN 10.1.2.11 455.1 GB 256 ? c08b8e99-... RAC1 UN 10.1.2.10 475 GB 256 ? e17bae55-... RAC1 How come "nodetool netstats" is still showing "Mode: JOINING"? Is there a way I can push the node to complete its joining? -- Kind regards, Troels Arvin - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
