[
https://issues.apache.org/jira/browse/CASSANDRA-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100969#comment-16100969
]
Kurt Greaves commented on CASSANDRA-13726:
------------------------------------------
You should set {{streaming_socket_timeout_in_ms=86400000}} on all the nodes.
Both sides of the streaming connection need a larger timeout. 7200 is way too
low.
> bootstrap streaming failed suspect due to secondary index
> ---------------------------------------------------------
>
> Key: CASSANDRA-13726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13726
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.18
> Reporter: peng xiao
>
> Hi,
> We have a Cassandra DC with 31 nodes,but it's always failed when adding the
> 32nd node.
> The following is the error log.And we suspect it's due to the secondary
> index,but it never happens before.Could you please advise?
> Thanks
> {code}
> INFO [CompactionExecutor:3] 2017-07-24 22:07:40,774
> ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 149
> (0%) on-heap, 20 (0%) off-heap
> INFO [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:347 -
> Writing Memtable-compactions_in_progress@581635159(0.008KiB serialized bytes,
> 1 ops, 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:382 -
> Completed flushing
> /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1159-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309,
> position=11929032)
> INFO [CompactionExecutor:3] 2017-07-24 22:07:40,796 CompactionTask.java:274
> - Compacted 15 sstables to
> [/data/cassandra/data/ecommercedata/ecommerce_baitiao_bill-c1a7f8d0660511e781e2077cc592d79d/ecommercedata-ecommerce_baitiao_bill-ka-40,].
> 293,793 bytes to 288,251 (~98% of original) in 108ms = 2.545348MB/s. 1,024
> total partitions merged to 1,016. Partition merge counts were {1:1008, 2:8, }
> INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,351
> ColumnFamilyStore.java:917 - Enqueuing flush of
> ecommerce_baitiao_record.baitiao_state: 2228978 (0%) on-heap, 6183142 (0%)
> off-heap
> INFO [CompactionExecutor:3] 2017-07-24 22:07:43,352
> ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 349
> (0%) on-heap, 266 (0%) off-heap
> INFO [MemtableFlushWriter:287] 2017-07-24 22:07:43,353 Memtable.java:347 -
> Writing Memtable-ecommerce_baitiao_record.baitiao_state@1651345625(4.792MiB
> serialized bytes, 89948 ops, 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:347 -
> Writing Memtable-compactions_in_progress@284311350(0.197KiB serialized bytes,
> 9 ops, 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:382 -
> Completed flushing
> /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1160-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309,
> position=13336501)
> INFO [MemtableFlushWriter:287] 2017-07-24 22:07:43,542 Memtable.java:382 -
> Completed flushing
> /data/cassandra/data/ecommercedata/ecommerce_baitiao_record-c94692e012cd11e682501f0d5e9f1e54/ecommercedata-ecommerce_baitiao_record.baitiao_state-tmp-ka-15-Data.db
> (1.593MiB) for commitlog position ReplayPosition(segmentId=1500866839309,
> position=13336501)
> INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,558
> SecondaryIndexManager.java:174 - Index build of
> [ecommerce_baitiao_record.baitiao_state] complete
> INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,558
> StreamResultFuture.java:181 - [Stream #15d5ffa0-7020-11e7-8b09-d510dd9f6293]
> Session with /xxx.xxx.xxx.xxx is complete
> WARN [StreamReceiveTask:5] 2017-07-24 22:07:43,559
> StreamResultFuture.java:208 - [Stream #15d5ffa0-7020-11e7-8b09-d510dd9f6293]
> Stream failed
> ERROR [main] 2017-07-24 22:07:43,561 CassandraDaemon.java:583 - Exception
> encountered during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
> [apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
> [apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
> [apache-cassandra-2.1.18.jar:2.1.18]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> WARN [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 Gossiper.java:1462
> - No local state or state is in silent shutdown, not announcing shutdown
> INFO [StorageServiceShutdownHook] 2017-07-24 22:07:43,564
> MessagingService.java:734 - Waiting for messaging service to quiesce
> INFO [ACCEPT-/172.16.135.52] 2017-07-24 22:07:43,565
> MessagingService.java:1020 - MessagingService has terminated the accept()
> thread
> INFO [CompactionExecutor:3] 2017-07-24 22:07:43,571 CompactionTask.java:141
> - Compacting
> [SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-10-Data.db'),
>
> SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-12-Data.db'),
>
> SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-11-Data.db'),
>
> SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-14-Data.db')]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
