[
https://issues.apache.org/jira/browse/CASSANDRA-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881196#comment-13881196
]
Jason Brown edited comment on CASSANDRA-6614 at 1/24/14 5:46 PM:
-----------------------------------------------------------------
I also see this in the log, not sure if it's related to the initial problem,
but definitely related to MigrationManager/Migrationtask:
{code}
INFO [GossipTasks:1] 2014-01-23 20:39:14,899 Gossiper.java (line 822)
InetAddress /54.xx.yy.zz is now DOWN
ERROR [MigrationStage:1] 2014-01-23 20:39:15,823 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:17,842 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:19,862 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:20,862 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:21,091 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:23,708 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:24,597 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:25,874 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:27,879 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:28,231 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:30,784 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:31,656 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
INFO [HANDSHAKE-ec2-54.xx.yy.zz/54.xx.yy.zz] 2014-01-23 20:41:59,793
OutboundTcpConnection.java (line 399) Handshaking version with
ec2-54.xx.yy.zz/54.xx.yy.zz
INFO [GossipStage:1] 2014-01-23 20:42:00,348 Gossiper.java (line 840) Node
/54.xx.yy.zz has restarted, now UP
{code}
was (Author: jasobrown):
I also see this in the log:
{code}
INFO [GossipTasks:1] 2014-01-23 20:39:14,899 Gossiper.java (line 822)
InetAddress /54.xx.yy.zz is now DOWN
ERROR [MigrationStage:1] 2014-01-23 20:39:15,823 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:17,842 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:19,862 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:20,862 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:21,091 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:23,708 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:24,597 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:25,874 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:27,879 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:28,231 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:30,784 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
ERROR [MigrationStage:1] 2014-01-23 20:39:31,656 MigrationTask.java (line 55)
Can't send migration request: node /54.xx.yy.zz is down.
INFO [HANDSHAKE-ec2-54.xx.yy.zz/54.xx.yy.zz] 2014-01-23 20:41:59,793
OutboundTcpConnection.java (line 399) Handshaking version with
ec2-54.xx.yy.zz/54.xx.yy.zz
INFO [GossipStage:1] 2014-01-23 20:42:00,348 Gossiper.java (line 840) Node
/54.xx.yy.zz has restarted, now UP
{code}
> 2 hours loop flushing+compacting
> system/{schema_keyspaces,schema_columnfamilies,schema_columns} when upgrading
> --------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-6614
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6614
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: ubuntu 12.04
> Reporter: Cyril Scetbon
>
> It happens when we upgrade one node to 1.2.13 on a 1.2.2 cluster
> see http://pastebin.com/YZKUQLXz
> If I grep for only InternalResponseStage logs I get
> http://pastebin.com/htnXZCiT which always displays same account of ops and
> serialized/live bytes per column family.
> When I upgrade one node from 1.2.2 to 1.2.13, for 2h I get the previous
> messages with a raise of CPU (as it flushes and compacts continually) on all
> nodes
> http://picpaste.com/pics/Screen_Shot_2014-01-24_at_09.18.50-ggcCDVqd.1390551670.png
> After that, everything is fine and I can upgrade other nodes without any
> raise of cpus load. when I start the upgrade, the more nodes I upgrade at the
> same time (at the beginning), the higher the cpu load is
> http://picpaste.com/pics/Screen_Shot_2014-01-23_at_17.45.56-I3fdEQ2T.1390552036.png
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
