Thanks for replay
How did you start the bootstrap and how are you checking if it has
> completed?
>
>
I start new node with <AutoBootstrap>true</AutoBootstrap> conf parameter,
and than check ring state with follow command:
freebsd# nodetool -h 192.168.0.37 -p 8080 ring, if bootstrap this command
must give my two nodes, but it give only one:
Address Status Load
Range Ring
128553582115057755121346842736115269027
192.168.0.37 Up 1.7 GB
43443562429493948342285492631024919735 |<--|
> What do nodetool info and nodetool streams say?
>
freebsd# nodetool -h 192.168.0.37 -p 8080 stream
Starting NodeTool
Mode: Normal
Not sending any streams.
Not receiving any streams.
freebsd# nodetool -h 192.168.0.220 -p 8080 stream
Starting NodeTool
Mode: Bootstrapping
Not sending any streams.
Not receiving any streams.
>
> The dropped message log usually indicates the node is under load.
>
> The log messages normally include the thread and file and line number with
> them. That's makes it a bit easier to follow along.
>
>
No, my node is not under heavy load, hear is top of system (192.167.0.37 -
on this node warn happens):
freebsd# top
last pid: 70295; load averages: 0.21, 0.77,
0.98 up 16+01:02:56
17:07:28
299 processes: 1 running, 297 sleeping, 1 stopped
CPU: 1.1% user, 0.0% nice, 1.5% system, 0.4% interrupt, 97.0% idle
Mem: 1664M Active, 1299M Inact, 165M Wired, 99M Cache, 112M Buf, 28M Free
Swap: 4096M Total, 1132K Used, 4095M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
83355 www 1 4 0 48060K 23788K accept 1 0:50 0.20% php-cgi
701 zabbix 1 4 5 5092K 1724K accept 0 4:52 0.00%
zabbix_agentd
702 zabbix 1 4 5 5092K 1724K accept 0 4:52 0.00%
zabbix_agentd
700 zabbix 1 4 5 5092K 1724K accept 0 4:50 0.00%
zabbix_agentd
699 zabbix 1 8 5 5092K 1424K nanslp 1 1:41 0.00%
zabbix_agentd
83349 www 1 4 0 48060K 23280K accept 0 0:50 0.00% php-cgi
83356 www 1 4 0 48060K 22804K accept 1 0:50 0.00% php-cgi
83351 www 1 4 0 48060K 23244K accept 0 0:50 0.00% php-cgi
83359 www 1 4 0 48060K 23160K accept 1 0:50 0.00% php-cgi
83358 www 1 4 0 48060K 21152K accept 0 0:49 0.00% php-cgi
83348 www 1 4 0 48060K 21300K accept 0 0:49 0.00% php-cgi
83353 www 1 4 0 48060K 22812K accept 0 0:49 0.00% php-cgi
83354 www 1 4 0 48060K 22168K accept 1 0:49 0.00% php-cgi
83352 www 1 4 0 48060K 22340K accept 1 0:49 0.00% php-cgi
83350 www 1 4 0 48060K 21504K accept 0 0:49 0.00% php-cgi
83357 www 1 4 0 48060K 22476K accept 1 0:49 0.00% php-cgi
57523 www 1 4 0 6644K 4536K kqread 0 0:38 0.00% nginx
598 root 1 44 0 4672K 1688K select 0 0:38 0.00% ntpd
703 zabbix 1 8 5 5092K 1200K nanslp 0 0:36 0.00%
zabbix_agentd
57522 www 1 4 0 5620K 4284K kqread 0 0:36 0.00% nginx
83347 root 1 4 0 47036K 6880K kqread 0 0:33 0.00% php-cgi
835 root 1 44 0 5876K 2348K select 1 0:30 0.00% sendmail
81287 root 1 44 0 9596K 4760K select 0 0:09 0.00% mc
451 root 1 44 0 3184K 876K select 1 0:09 0.00% syslogd
848 root 1 8 0 3212K 976K nanslp 1 0:07 0.00% cron
874 root 1 44 0 3240K 1052K select 0 0:05 0.00% inetd
77864 root 1 44 0 8428K 2528K select 1 0:04 0.00% sshd
30821 root 1 44 0 8428K 2556K select 0 0:01 0.00% sshd
68918 www 1 4 0 14020K 10900K kqread 1 0:01 0.00% python
28599 root 1 44 0 8428K 2556K select 1 0:01 0.00% sshd
57351 mysql 16 20 0 1124M 378M sigwai 1 0:01 0.00% mysqld
31652 www 1 44 0 9624K 6568K select 0 0:01 0.00% python
31653 www 1 44 0 9624K 6568K select 1 0:01 0.00% python
70029 root 26 44 0 1150M 59468K ucond 0 0:01 0.00% java
68919 www 1 4 0 12996K 10744K kqread 0 0:00 0.00% python
also no network heavy load.
I solve problem by recompile cassandra with folow patch on
src/java/org/apache/cassandra/net/MessageDeliveryTask.java
Index: MessageDeliveryTask.java
===================================================================
--- MessageDeliveryTask.java (revision 1025860)
+++ MessageDeliveryTask.java (working copy)
@@ -37,12 +37,6 @@
public void run()
{
- if (System.currentTimeMillis() > constructionTime_ +
DatabaseDescriptor.getRpcTimeout())
- {
- MessagingService.incrementDroppedMessages();
- return;
- }
-
StorageService.Verb verb = message_.getVerb();
IVerbHandler verbHandler =
MessagingService.instance.getVerbHandler(verb);
assert verbHandler != null : "unknown verb " + verb;
but this patch doesn't looks good for me
PS: we use:
FreeBSD 7.2-p8,
java - diablo-jdk-1.6.0.07.02_10 Java Development Kit 1.6.0_07.02
100Mbit network between nodes
