[jira] [Commented] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming

2018-04-18 Thread Lerh Chuan Low (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443363#comment-16443363
 ] 

Lerh Chuan Low commented on CASSANDRA-14362:


It's not blocking me because I have that workaround, but thanks for looking 
into it so quickly, much appreciated :) 

> Bind to correct local address in 4.0 streaming
> --
>
> Key: CASSANDRA-14362
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14362
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Lerh Chuan Low
>Assignee: Lerh Chuan Low
>Priority: Major
> Fix For: 4.0
>
>
> I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 
> cluster, but with a 3 node setup that works on 3.11, it fails to work on 
> {{trunk}}. I mistakenly stumbled into it while testing my own code changes. 
> My setup is as follows:
> broadcast_rpc_address: publicIP
> broadcast_address: publicIP
> listen_address: omitted. Ends up as privateIP. 
> Works on 3.11 just fine. 
> On {{trunk}} though, it never works. My node is never able to join the 
> cluster:
> {code}
> Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO  [main] 2018-04-03 
> 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range 
> (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for 
> keyspace system_traces
> Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO  [main] 2018-04-03 
> 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range 
> (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for 
> keyspace system_traces
> Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due 
> to local pause of 26215173446ns > 50ns
> Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due 
> to local pause of 10736485907ns > 50ns
> Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending 
> tasks; skipping status check (no nodes will be marked down)
> Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending 
> tasks; skipping status check (no nodes will be marked down)
> Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due 
> to local pause of 400790432954ns > 50ns
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; 
> skipping status check (no nodes will be marked down)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR 
> [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 
> StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] 
> Streaming error occurred on session with peer 52.88.241.181:7000
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed 
> to connect to 52.88.241.181:7000 (STREAM) for streaming data
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:273)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at

[jira] [Commented] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming

2018-04-17 Thread Jason Brown (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440820#comment-16440820
 ] 

Jason Brown commented on CASSANDRA-14362:
-

Opened CASSANDRA-14389 for the local address binding regression. Thanks, 
[~djoshi3] for also looking at this ticket.

> Bind to correct local address in 4.0 streaming
> --
>
> Key: CASSANDRA-14362
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14362
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Lerh Chuan Low
>Assignee: Lerh Chuan Low
>Priority: Major
> Fix For: 4.0
>
>
> I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 
> cluster, but with a 3 node setup that works on 3.11, it fails to work on 
> {{trunk}}. I mistakenly stumbled into it while testing my own code changes. 
> My setup is as follows:
> broadcast_rpc_address: publicIP
> broadcast_address: publicIP
> listen_address: omitted. Ends up as privateIP. 
> Works on 3.11 just fine. 
> On {{trunk}} though, it never works. My node is never able to join the 
> cluster:
> {code}
> Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO  [main] 2018-04-03 
> 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range 
> (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for 
> keyspace system_traces
> Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO  [main] 2018-04-03 
> 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range 
> (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for 
> keyspace system_traces
> Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due 
> to local pause of 26215173446ns > 50ns
> Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due 
> to local pause of 10736485907ns > 50ns
> Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending 
> tasks; skipping status check (no nodes will be marked down)
> Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending 
> tasks; skipping status check (no nodes will be marked down)
> Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due 
> to local pause of 400790432954ns > 50ns
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN  [GossipTasks:1] 
> 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; 
> skipping status check (no nodes will be marked down)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR 
> [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 
> StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] 
> Streaming error occurred on session with peer 52.88.241.181:7000
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed 
> to connect to 52.88.241.181:7000 (STREAM) for streaming data
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:273)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at 
>