Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Khaja, Raziuddin (NIH/NLM/NCBI) [C]]

You are welcome Martjin! Glad to have been able to help (
Best,
-Razi

On 4/13/17, 12:13 PM, "Martijn Pieters"  wrote:

On 13/04/2017, 15:06, "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
 wrote:
> Looking at your original message: 
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_user-40cassandra.apache.org_msg51736.html=DwIGaQ=5VD0RTtNlTh3ycd41b3MUw=xmrhtBIDZ_UcilJaq1SH8Q=pRFNVjsQoNpE4xWTky8s9kt-twcOChSSvnvSDW6jrfc=R0lHgyuqfgHkYPRDYmCpEWN1Q02x5moqZa25mRLdJ80=
 
>  
>   I see you edited etc/cassandra/cassandra-env.sh, by changing:
>
>   
>+#JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>
>+JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv6Addresses=true"   
>
>   First, I don’t think there is an option java.net.preferIPv6Addresses, 
so I would recommend removing that line.

The option does exist, see 
https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html.
 I had tried both with and without the IPv6 option.

> Second, I believe that starting in apache-cassandra-3.2, that 
enabling/disabling the option has been moved to a file called *jvm.options* 

[snip evidence of the option moving]

> My guess right now is that you may have upgraded Cassandra from a version 
older that 3.1 and somehow your config files are not compatible with 3.10? 

**BINGO**. Indeed, I had at some point downgraded to a different Cassandra 
version in an attempt to resolve issues with `cqlsh` (which hardcodes a CQL 
version). As a result the option was being applied **twice**, in 
/etc/cassandra/cassandra-env.sh and via /etc/cassandra/jvm.options.

Removing the switch from both locations now lets Cassandra bind to IPv6.

I now can finally drop the SSH tunnel forwarding the port in my test 
cluster.

Thanks!

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Martijn Pieters]

On 13/04/2017, 15:06, "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
 wrote:
> Looking at your original message: 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mail-2Darchive.com_user-40cassandra.apache.org_msg51736.html=DwIGaQ=5VD0RTtNlTh3ycd41b3MUw=xmrhtBIDZ_UcilJaq1SH8Q=pRFNVjsQoNpE4xWTky8s9kt-twcOChSSvnvSDW6jrfc=R0lHgyuqfgHkYPRDYmCpEWN1Q02x5moqZa25mRLdJ80=
>  
>  
>   I see you edited etc/cassandra/cassandra-env.sh, by changing:
>
>   
>+#JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>
>+JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv6Addresses=true"   
>
>   First, I don’t think there is an option java.net.preferIPv6Addresses, so I 
> would recommend removing that line.

The option does exist, see 
https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html.
 I had tried both with and without the IPv6 option.

> Second, I believe that starting in apache-cassandra-3.2, that 
> enabling/disabling the option has been moved to a file called *jvm.options* 

[snip evidence of the option moving]

> My guess right now is that you may have upgraded Cassandra from a version 
> older that 3.1 and somehow your config files are not compatible with 3.10? 

**BINGO**. Indeed, I had at some point downgraded to a different Cassandra 
version in an attempt to resolve issues with `cqlsh` (which hardcodes a CQL 
version). As a result the option was being applied **twice**, in 
/etc/cassandra/cassandra-env.sh and via /etc/cassandra/jvm.options.

Removing the switch from both locations now lets Cassandra bind to IPv6.

I now can finally drop the SSH tunnel forwarding the port in my test cluster.

Thanks!

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Khaja, Raziuddin (NIH/NLM/NCBI) [C]]

Hi Martjin, 

Looking at your original message: 
http://www.mail-archive.com/user@cassandra.apache.org/msg51736.html

I see you edited etc/cassandra/cassandra-env.sh, by changing:

+#JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
+JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv6Addresses=true"

First, I don’t think there is an option java.net.preferIPv6Addresses, so I 
would recommend removing that line.

Second, I believe that starting in apache-cassandra-3.2, that 
enabling/disabling the option has been moved to a file called *jvm.options* 

./2.1.16/apache-cassandra-2.1.16/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./2.2.4/apache-cassandra-2.2.4/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./2.2.5/apache-cassandra-2.2.5/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./2.2.6/apache-cassandra-2.2.6/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./2.2.7/apache-cassandra-2.2.7/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.0/apache-cassandra-3.0.0/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.1/apache-cassandra-3.0.1/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.2/apache-cassandra-3.0.2/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.3/apache-cassandra-3.0.3/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.4/apache-cassandra-3.0.4/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.5/apache-cassandra-3.0.5/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.6/apache-cassandra-3.0.6/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.7/apache-cassandra-3.0.7/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.8/apache-cassandra-3.0.8/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.0.9/apache-cassandra-3.0.9/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.1/apache-cassandra-3.1/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.1.1/apache-cassandra-3.1.1/conf/cassandra-env.sh:JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"
./3.2/apache-cassandra-3.2/conf/jvm.options:-Djava.net.preferIPv4Stack=true
./3.2.1/apache-cassandra-3.2.1/conf/jvm.options:-Djava.net.preferIPv4Stack=true
./3.3/apache-cassandra-3.3/conf/jvm.options:-Djava.net.preferIPv4Stack=true
./3.4/apache-cassandra-3.4/conf/jvm.options:-Djava.net.preferIPv4Stack=true
./3.5/apache-cassandra-3.5/conf/jvm.options:-Djava.net.preferIPv4Stack=true
./3.6/apache-cassandra-3.6/conf/jvm.options:-Djava.net.preferIPv4Stack=true
./3.7/apache-cassandra-3.7/conf/jvm.options:-Djava.net.preferIPv4Stack=true

My guess right now is that you may have upgraded Cassandra from a version older 
that 3.1 and somehow your config files are not compatible with 3.10? 

-Razi


On 4/13/17, 5:41 AM, "Martijn Pieters"  wrote:

From my original email: 
http://www.mail-archive.com/user@cassandra.apache.org/msg51736.html:

> My configuration changes:
>
> listen_address: 
> listen_interface_prefer_ipv6: true

listen_interface is commented out. I've just now tried again with "# 
listen_interface_prefer_ipv6: false" (option commented out), but the error 
persists. 

I've also rebooted the system, in case the upgrade from base 16.04 to 
16.04.2 left something in a funky state.

On 12/04/2017, 21:39, "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
 wrote:

Are you specifying both the listen_address and listen_interface, or 
just one of the two?

Send,  an example of the following 3 lines.  Here is what I have on my 
2.1.16 cluster that uses ipv6:

listen_address: ::hhh::h::hhh:h
# listen_interface: eth0
# listen_interface_prefer_ipv6: false

Also, looking at my config, I can confirm that it is uneccessary or 
wrong to escape the ipv6 address with \ as I suggested before.

-Razi

On 4/12/17, 4:05 PM, "Martijn Pieters"  wrote:

From: "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 

> Maybe you have to escape the IPV6 addresses in the cassandra.yaml 
in the same way.
> I think it’s worth a try.

Nope, no luck. You get an error instead:

ERROR [main] 2017-04-12 20:03:46,899 CassandraDaemon.java:752 - 
Exception encountered during startup: Unknown listen_address 
'\:\:\:\:\:h\:hh\:h'

(actual address digits replaced with h characters).

Martijn


 

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Martijn Pieters]

From my original email: 
http://www.mail-archive.com/user@cassandra.apache.org/msg51736.html:

> My configuration changes:
>
> listen_address: 
> listen_interface_prefer_ipv6: true

listen_interface is commented out. I've just now tried again with "# 
listen_interface_prefer_ipv6: false" (option commented out), but the error 
persists. 

I've also rebooted the system, in case the upgrade from base 16.04 to 16.04.2 
left something in a funky state.

On 12/04/2017, 21:39, "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
 wrote:

Are you specifying both the listen_address and listen_interface, or just 
one of the two?

Send,  an example of the following 3 lines.  Here is what I have on my 
2.1.16 cluster that uses ipv6:

listen_address: ::hhh::h::hhh:h
# listen_interface: eth0
# listen_interface_prefer_ipv6: false

Also, looking at my config, I can confirm that it is uneccessary or wrong 
to escape the ipv6 address with \ as I suggested before.

-Razi

On 4/12/17, 4:05 PM, "Martijn Pieters"  wrote:

From: "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
> Maybe you have to escape the IPV6 addresses in the cassandra.yaml in 
the same way.
> I think it’s worth a try.

Nope, no luck. You get an error instead:

ERROR [main] 2017-04-12 20:03:46,899 CassandraDaemon.java:752 - 
Exception encountered during startup: Unknown listen_address 
'\:\:\:\:\:h\:hh\:h'

(actual address digits replaced with h characters).

Martijn

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Khaja, Raziuddin (NIH/NLM/NCBI) [C]]

Are you specifying both the listen_address and listen_interface, or just one of 
the two?

Send,  an example of the following 3 lines.  Here is what I have on my 2.1.16 
cluster that uses ipv6:

listen_address: ::hhh::h::hhh:h
# listen_interface: eth0
# listen_interface_prefer_ipv6: false

Also, looking at my config, I can confirm that it is uneccessary or wrong to 
escape the ipv6 address with \ as I suggested before.

-Razi

On 4/12/17, 4:05 PM, "Martijn Pieters"  wrote:

From: "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
> Maybe you have to escape the IPV6 addresses in the cassandra.yaml in the 
same way.
> I think it’s worth a try.

Nope, no luck. You get an error instead:

ERROR [main] 2017-04-12 20:03:46,899 CassandraDaemon.java:752 - 
Exception encountered during startup: Unknown listen_address 
'\:\:\:\:\:h\:hh\:h'

(actual address digits replaced with h characters).

Martijn

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Martijn Pieters]

From: "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" 
> Maybe you have to escape the IPV6 addresses in the cassandra.yaml in the same 
> way.
> I think it’s worth a try.

Nope, no luck. You get an error instead:

ERROR [main] 2017-04-12 20:03:46,899 CassandraDaemon.java:752 - Exception 
encountered during startup: Unknown listen_address 
'\:\:\:\:\:h\:hh\:h'
    
(actual address digits replaced with h characters).

Martijn

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Khaja, Raziuddin (NIH/NLM/NCBI) [C]]

See this note in cassandra-topology.properties:



# Native IPv6 is supported, however you must escape the colon in the IPv6 
Address

# Also be sure to comment out JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"

# in cassandra-env.sh

fe80\:0\:0\:0\:202\:b3ff\:fe1e\:8329=DC1:RAC3



Maybe you have to escape the IPV6 addresses in the cassandra.yaml in the same 
way.

I think it’s worth a try.

-Razi





On 4/12/17, 3:08 PM, "Martijn Pieters"  wrote:



From: sai krishnam raju potturi 

> We have included the IPV6 address with scope GLOBAL, and not IPV6 with 
SCOPE LINK in the YAML and TOPOLOGY files.

>

> inet6 addr: 2001: *** : ** : ** : * : * :  :   Scope:Global

> inet6 addr: fe80 :: *** :  :  :  Scope:Link

>

> Not sure if this might be of relevance to the issue you are facing.



I already stated I tried both the initial email.



Martijn Pieters

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Martijn Pieters]

From: sai krishnam raju potturi 
> We have included the IPV6 address with scope GLOBAL, and not IPV6 with SCOPE 
> LINK in the YAML and TOPOLOGY files.  
>
> inet6 addr: 2001: *** : ** : ** : * : * :  :   Scope:Global
> inet6 addr: fe80 :: *** :  :  :  Scope:Link
>
> Not sure if this might be of relevance to the issue you are facing.

I already stated I tried both the initial email.

Martijn Pieters

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[sai krishnam raju potturi]

We have included the IPV6 address with scope GLOBAL, and not IPV6 with
SCOPE LINK in the YAML and TOPOLOGY files.

inet6 addr: 2001: *** : ** : ** : * : * :  :   Scope:Global

inet6 addr: fe80 :: *** :  :  :  Scope:Link


Not sure if this might be of relevance to the issue you are facing.


thanks

Sai



On Tue, Apr 11, 2017 at 10:29 AM, Martijn Pieters  wrote:

> From: sai krishnam raju potturi 
> > I got a similar error, and commenting out the below line helped.
> > JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
> >
> > Did you also include "rpc_interface_prefer_ipv6: true" in the YAML file?
>
> No luck at all here. Yes, I had commented out that line (and also tried
> replacing it with `-Djava.net.preferIPv6Addresses=true`, included in my
> email. I also included an error to make sure it was the right file).
>
> It all *should* work, but doesn’t. :-(
>
> I just tried again with “rpc_interface_prefer_ipv6: true” set as well, but
> without luck. I note that I have the default “rpc_address: localhost”, so
> it’ll bind to the lo loopback, which has IPv4 configured already. Not that
> using “rpc_address: ‘::1’” instead works (same error, so I can’t bind to
> the IPv6 localhost address either).
>
> Martijn Pieters
>
>
>
>
>

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[Martijn Pieters]

From: sai krishnam raju potturi 
> I got a similar error, and commenting out the below line helped.
> JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>
> Did you also include "rpc_interface_prefer_ipv6: true" in the YAML file?

No luck at all here. Yes, I had commented out that line (and also tried 
replacing it with `-Djava.net.preferIPv6Addresses=true`, included in my email. 
I also included an error to make sure it was the right file).

It all *should* work, but doesn’t. :-(

I just tried again with “rpc_interface_prefer_ipv6: true” set as well, but 
without luck. I note that I have the default “rpc_address: localhost”, so it’ll 
bind to the lo loopback, which has IPv4 configured already. Not that using 
“rpc_address: ‘::1’” instead works (same error, so I can’t bind to the IPv6 
localhost address either).

Martijn Pieters

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

[sai krishnam raju potturi]

I got a similar error, and commenting out the below line helped.

JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"


Did you also include "rpc_interface_prefer_ipv6: true" in the YAML file?


thanks

Sai



On Tue, Apr 11, 2017 at 6:37 AM, Martijn Pieters  wrote:

> I’m having issues getting a single-node Cassandra cluster to run on a
> Ubuntu 16.04 VM with only IPv6 available. I’m running Oracle Java 8
> (8u121-1~webupd8~2), Cassandra 3.10 (installed via the Cassandra
> http://www.apache.org/dist/cassandra/debian packages.)
>
>
>
> I consistently get a “Protocol family unavailable” exception:
>
>
>
> ERROR [main] 2017-04-11 09:54:23,991 CassandraDaemon.java:752 - Exception
> encountered during startup
>
> java.lang.RuntimeException: java.net.SocketException: Protocol family
> unavailable
>
> at 
> org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:730)
> ~[apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.net.MessagingService.listen(MessagingService.java:664)
> ~[apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.net.MessagingService.listen(MessagingService.java:648)
> ~[apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:773)
> ~[apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:666)
> ~[apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
> ~[apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:394)
> [apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
> [apache-cassandra-3.10.jar:3.10]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
> [apache-cassandra-3.10.jar:3.10]
>
> Caused by: java.net.SocketException: Protocol family unavailable
>
> at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_121]
>
> at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_121]
>
> at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_121]
>
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> ~[na:1.8.0_121]
>
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> ~[na:1.8.0_121]
>
> at 
> org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:714)
> ~[apache-cassandra-3.10.jar:3.10]
>
> ... 8 common frames omitted
>
>
>
> `lo` (loopback) has both `inet` and `inet6` addresses, but `eth0` has no
> `inet` addresses, so only inet6 addr entries (both a local and a global
> scope address are configured).
>
>
>
> My configuration changes:
>
>
>
> listen_address: 
>
> listen_interface_prefer_ipv6: true
>
>
>
> Tracing through the source code the exception shows that it is the
> listen_address value above that throws the exception, changing it back to
> 127.0.0.1 makes the server work again (but then I don’t get to use it on my
> local network). I tried both the local and the global scope IPv6 address.
>
>
>
> I tried changing the JVM configuration to prefer IPv6 by editing
> /etc/cassandra/cassandra-env.sh:
>
>
>
> --- etc/cassandra/cassandra-env.sh  2017-01-31 16:29:32.0
> +
>
> +++ /etc/cassandra/cassandra-env.sh 2017-04-11 09:52:51.45600
> +
>
> @@ -290,6 +290,9 @@
>
> # to the location of the native libraries.
>
> JVM_OPTS="$JVM_OPTS -Djava.library.path=$CASSANDRA_HOME/lib/sigar-bin"
>
>
>
> +#JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>
> +JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv6Addresses=true"
>
> +
>
> JVM_OPTS="$JVM_OPTS $MX4J_ADDRESS"
>
> JVM_OPTS="$JVM_OPTS $MX4J_PORT"
>
> JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
>
>
>
> But this makes no difference
>
>
>
> I also tried using `listen_interface` instead, but that only changes the
> error message to:
>
>
>
> ERROR [main] 2017-04-11 10:35:16,426 CassandraDaemon.java:752 -
> Exception encountered during startup: Configured listen_interface "eth0"
> could not be found
>
>
>
> What else can I do?
>
>
>
> Martijn Pieters
>