[
https://issues.apache.org/jira/browse/CASSANDRA-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740956#comment-14740956
]
Ara Sadoyan commented on CASSANDRA-10291:
-----------------------------------------
Have just tried to upgrade to java8, no success.
Also very strange thing is going on during bootstrap
nodetool netstats on boostrapping node shows that stream is stalled and
destination file does not exists, but source continues to send stream and shows
progress.
cassa15 14:59:00 ~ # nodetool netstats | grep 192.168.220.19 (Destination Node)
/192.168.220.19
/opt/cassandra/storage/data/REPLACED/html_2015_07-be0b6c300f5311e59d7f27679cc9890d/tmp-la-6-big-Data.db
1278964766/10319377771 bytes(12%) received from idx:0/192.168.220.19
cassa15 14:59:13 ~ # ls
/opt/cassandra/storage/data/REPLACED/html_2015_07-be0b6c300f5311e59d7f27679cc9890d/tmp-la-6-big-Data.db
ls: cannot access
/opt/cassandra/storage/data/REPLACED/html_2015_07-be0b6c300f5311e59d7f27679cc9890d/tmp-la-6-big-Data.db:
No such file or directory
cassa15 14:59:06 ~ # nodetool -h 192.168.220.19 netstats
Mode: NORMAL
Bootstrap 6e3f57f0-5893-11e5-bf5a-5f8e66db7dc7
/192.168.220.27
Sending 461 files, 146529071898 bytes total. Already sent 0 files,
7905965086 bytes total
/opt/cassandra/storage/data/REPLACED/html_2015_07-be0b6c300f5311e59d7f27679cc9890d/la-3919-big-Data.db
7905965086/10319377771 bytes(76%) sent to idx:0/192.168.220.27
after 10 minutes :
Absolutely same picture at bootstrapping node, but stuff is changed in source
node.
cassa15 15:03:18 ~ # nodetool -h 192.168.220.19 netstats
Mode: NORMAL
Bootstrap 6e3f57f0-5893-11e5-bf5a-5f8e66db7dc7
/192.168.220.27
Sending 461 files, 146529071898 bytes total. Already sent 0 files,
9994641455 bytes total
/opt/cassandra/storage/data//REPLACED/html_2015_07-be0b6c300f5311e59d7f27679cc9890d/la-3919-big-Data.db
9994641455/10319377771 bytes(96%) sent to idx:0/192.168.220.27
Read Repair Statistics:
Attempted: 48576
Mismatch (Blocking): 0
Mismatch (Background): 69
Pool Name Active Pending Completed
Large messages n/a 0 545698
Small messages n/a 0 10507575
Gossip messages n/a 0 811646
Any idea ?
> Bootstrap hangs on adding new node
> -----------------------------------
>
> Key: CASSANDRA-10291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10291
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian 7 64 bit
> HotSpot JDK 1.7.0_79
> Cassandra-2.2.1 via apt-get
> 1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive
> Reporter: Ara Sadoyan
> Attachments: nodetool.txt, system.log
>
>
> Adding new node in heavy loaded environment freeze bootstrap. No errors are
> reported in log files. Some of other other nodes throws "String didn't
> validate" error, but I;m not sure that this is related.
> After restarting node it start bootstrap again and hangs after some time .
> nodetool netstats shows :
> /data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701 bytes(27%)
> received from idx:0/192.168.220.16
> /data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671 bytes(38%)
> received from idx:0/192.168.220.22
> /data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398 bytes(48%)
> received from idx:0/192.168.220.12
> /data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610 bytes(85%)
> received from idx:0/192.168.220.18
> /data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085 bytes(48%)
> received from idx:0/192.168.220.20
> but listing files on local FS shows "No such file or directory"
> This happens only if there is significant amount of data. I have 1.5 TB per
> node on 13 node cluster, we use STCS compaction strategy and flat network
> topology .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
