When we are starting the node, it is starting bootstrap automatically and restreaming the whole data again. It is not resuming .
On Thu, May 7, 2020 at 4:47 PM Adam Scott <adam.c.sc...@gmail.com> wrote: > I think you want to run `nodetool bootstrap resume` ( > https://cassandra.apache.org/doc/latest/tools/nodetool/bootstrap.html) > to pick up where it last left off. Sorry for the late reply. > > > On Thu, May 7, 2020 at 2:22 PM Surbhi Gupta <surbhi.gupt...@gmail.com> > wrote: > >> So after failed bootstrapped , if we start cassandra again on the new >> node , will it resume bootstrap or will it start over? >> >> On Thu, 7 May 2020 at 13:32, Adam Scott <adam.c.sc...@gmail.com> wrote: >> >>> I recommend it on all nodes. This will eliminate that as a source of >>> trouble further on down the road. >>> >>> >>> On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta <surbhi.gupt...@gmail.com> >>> wrote: >>> >>>> streaming_socket_timeout_in_ms is 24 hour. >>>> So tcp settings should be changed on the new bootstrap node or on all >>>> nodes ? >>>> >>>> >>>> On Thu, 7 May 2020 at 13:23, Adam Scott <adam.c.sc...@gmail.com> wrote: >>>> >>>>> >>>>> *edit >>>>> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 >>>>> net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10* >>>>> then run sysctl -p to cause the kernel to reload the settings >>>>> >>>>> 5 minutes (300) seconds is probably too long. >>>>> >>>>> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta <surbhi.gupt...@gmail.com> >>>>> wrote: >>>>> >>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time >>>>>> >>>>>> 300 >>>>>> >>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl >>>>>> >>>>>> 30 >>>>>> >>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes >>>>>> >>>>>> 9 >>>>>> >>>>>> On Thu, 7 May 2020 at 12:32, Adam Scott <adam.c.sc...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Maybe a firewall killing a connection? >>>>>>> >>>>>>> What does the following show? >>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_time >>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl >>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes >>>>>>> >>>>>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta < >>>>>>> surbhi.gupt...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> We are trying to expand a datacenter and trying to add nodes but >>>>>>>> when node is bootstrapping , it goes half way through and then fail >>>>>>>> with >>>>>>>> below error, We have increased stremthroughput from 200 to 400 when we >>>>>>>> were >>>>>>>> trying for the 2nd time but still it failed. We are on 3.11.0 , using >>>>>>>> G1GC >>>>>>>> with 31GB heap. >>>>>>>> >>>>>>>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933 >>>>>>>> CassandraDaemon.java:228 - Exception in thread >>>>>>>> Thread[MessagingService-Incoming-/10.X.X.X,main] >>>>>>>> >>>>>>>> java.io.IOError: java.io.EOFException: Stream ended prematurely >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> Caused by: java.io.EOFException: Stream ended prematurely >>>>>>>> >>>>>>>> at >>>>>>>> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) >>>>>>>> ~[lz4-1.3.0.jar:na] >>>>>>>> >>>>>>>> at >>>>>>>> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) >>>>>>>> ~[lz4-1.3.0.jar:na] >>>>>>>> >>>>>>>> at >>>>>>>> net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) >>>>>>>> ~[lz4-1.3.0.jar:na] >>>>>>>> >>>>>>>> at >>>>>>>> java.io.DataInputStream.readFully(DataInputStream.java:195) >>>>>>>> ~[na:1.8.0_242] >>>>>>>> >>>>>>>> at >>>>>>>> java.io.DataInputStream.readFully(DataInputStream.java:169) >>>>>>>> ~[na:1.8.0_242] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at org.apache.cassandra.db.Columns.apply(Columns.java:377) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:600) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:475) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:431) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> at >>>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222) >>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0] >>>>>>>> >>>>>>>> ... 11 common frames omitted >>>>>>>> >>>>>>>> Thanks >>>>>>>> Surbhi >>>>>>>> >>>>>>>
