The version given by apt is 8u162-b12-1. Which I think corresponds to
openJDK-8-162. When I run jrunscript -e 'print
(javax.crypto.Cipher.getMaxAllowedKeyLength("RC5") >= 256);' the command
returns true. Not sure if that is the best way to verify JCE installed.
Michael Carlise
On Mon, Aug 26, 2019 at 5:47 PM Marc Selwan <marc.sel...@datastax.com>
wrote:
> which exact version of OpenJDK are you using? Is it possible you don't
> have JCE on those nodes? (I believe more recent versions of Java 8 has this
> baked in so that might not be it)
>
>
> *Marc Selwan | *DataStax *| *PM, Server Team *|* *(925) 413-7079* *|*
> Twitter <https://twitter.com/MarcSelwan>
>
> * Quick links | *DataStax <http://www.datastax.com> *| *Training
> <http://www.academy.datastax.com> *| *Documentation
> <http://www.datastax.com/documentation/getting_started/doc/getting_started/gettingStartedIntro_r.html>
> *| *Downloads <http://www.datastax.com/download>
>
>
>
> On Mon, Aug 26, 2019 at 1:56 PM Michael Carlise
> <mcarl...@salesforce.com.invalid> wrote:
>
>>
>> I originally opened this issue on stackoverflow (
>> https://stackoverflow.com/questions/57516660/cassandra-node-to-node-encryption-throws-unable-to-gossip-with-peers-exception
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_57516660_cassandra-2Dnode-2Dto-2Dnode-2Dencryption-2Dthrows-2Dunable-2Dto-2Dgossip-2Dwith-2Dpeers-2Dexception&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=KdhQDpMbz8v1GYrbdYL_opGq-GBPXftrEYEkgcGeMp0&s=4CR8PRQopb4FyCLj8PDI44mSouBz65Yx8THnH8tOb7o&e=>
>> ).
>>
>> However, I haven't gotten any responses in over a week. I'm going to
>> post it here and maybe someone will have an idea on where I can look.
>>
>> We currently run a multi region cassandra cluster in AWS. It runs in four
>> regions, 12 nodes per region. It runs without node to node encryption (or
>> client encryption either). We are trying to enable inter datacenter node to
>> node encryption. However, when we flip encryption over we get an exception
>> that nodes are unable to gossip with any peers.
>>
>> It could possibly be that we didn't build our jks keystore/truststores
>> correctly (more on how we built these files below). But, we additionally do
>> not see intra datacenter communication working (which should be set to
>> unencrypted communication). Additionally, cqlsh cannot connect to the node
>> either; even though we have (by default) client_auth_required set to
>> false.
>>
>> ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exception
>> encountered during startup
>> java.lang.RuntimeException: Unable to gossip with any peers
>> at
>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435)
>> ~[apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566)
>> ~[apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823)
>> ~[apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:683)
>> ~[apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:632)
>> ~[apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388)
>> [apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
>> [apache-cassandra-3.11.4.jar:3.11.4]
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
>> [apache-cassandra-3.11.4.jar:3.11.4]
>> INFO [main] 2019-08-15 18:47:07,384 YamlConfigurationLoader.java:89 -
>> Configuration location: file:/etc/cassandra/cassandra.yaml
>>
>>
>> Something to note is that this error message occurs after a few minutes
>> of the node being up. (i.e. there is a delay between start up before this
>> exception is thrown).
>>
>> *Information about our cassandra setup*
>>
>> cassandra version: 3.11.4
>> JDK version: openjdk-8.
>> Linux: Ubuntu 18.04 (bionic).
>>
>> *cassandra.yaml*
>>
>> endpoint_snitch: Ec2MultiRegionSnitch
>>
>> server_encryption_options:
>> internode_encryption: dc
>> keystore: <omitted>
>> keystore_password: <omitted>
>> truststore: <omitted>
>> truststore_password: <omitted>
>>
>> client_encryption_options:
>> enabled: false
>>
>> *cassandra-rackdc.properties*
>>
>> prefer_local=true
>>
>> *No obvious errors with SSH output*
>>
>> When starting cassandra with JVM_OPTS="$JVM_OPTS -Djavax.net.debug=ssl" added
>> to cassandra-env.sh we see SSL logs printed to stdout (*Note: Subject
>> and Issuer were omitted on purpose)*.
>>
>> found key for : cassy-us-west-2
>> adding as trusted cert:
>> Subject: ...
>> Issuer: ...
>> Algorithm: RSA; Serial number: 0xdad28d843fc73325d4c1a75207d4e74
>> Valid from Fri May 27 00:00:00 UTC 2016 until Tue May 26 23:59:59 UTC 2026
>>
>> ...
>>
>> trigger seeding of SecureRandom
>> done seeding SecureRandom
>>
>> Looking at Java SE SSL/TLS connection debugging
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.oracle.com_javase_7_docs_technotes_guides_security_jsse_ReadDebug.html&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=KdhQDpMbz8v1GYrbdYL_opGq-GBPXftrEYEkgcGeMp0&s=SR3ashwvSRxA75nBjGDwjAwq65nDuBZUaDOvHPGDrps&e=>,
>> this looks correct. But to note, we see this series of messages (along with
>> the RSA key signature output) repeated several times in rapid fire. We
>> never observe any messages about the trust store being added; however that
>> might be something that occurs only on client initiation (?)
>>
>> Additionally, we do see cassandra report that the Encrypted Messaging
>> service has been started.
>>
>> INFO [main] 2019-08-15 18:45:31,022 MessagingService.java:704 - Starting
>> Encrypted Messaging Service on SSL port 7001
>>
>> *Doesn't appear to be a cassandra.yaml configuration problem*
>>
>> We can bring the node back online by simply configuring internode_encryption:
>> none. This action seems to rule out a broadcast_address or rpc_address
>> configuration problem.
>>
>> *How we built our keystore/truststores*
>>
>> We followed the basic template datastax docs for preparing SSL
>> certificates
>> <https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/configuration/secureSSLCertWithCA.html>.
>> One minor difference was that our private key and CSRs were generated using
>> openssl. One per each region (we plan to share key/signed certs across
>> nodes in regions). This was created using a command template as:
>>
>> openssl req -new -newkey rsa:2048 -out cassy-<region>.csr -keyout
>> cassy-<region>.key -config cassy-<region>.conf -subj "..." -nodes -sha256
>>
>> The generated CSR was then signed by an internal root CA. Because we
>> generated our files using openssl, we had to build our jks files by
>> importing our certs into them.
>>
>> *Commands to generate truststore*
>>
>> We distribute this one file to all nodes.
>>
>> keytool -importcert
>> -keystore generic-server-truststore.jks
>> -alias rootCa
>> -file rootCa.crt
>> -noprompt
>> -keypass omitted
>> -storepass omitted
>>
>> *Commands to generate keystore*
>>
>> This was done one per region; but essentially we created a keystore with
>> keytool, then deleted the key entry and then imported our key entry using
>> keytool from a pkcs12 file.
>>
>> keytool -genkeypair -keyalg RSA -alias cassy-${region} -keystore
>> cassy-${region}.jks -storepass omitted -keypass omitted -validity 365
>> -keysize 2048 -dname "..."
>>
>> keytool -delete -alias cassy-${region} -keystore cassy-${region}.jks
>> -storepass omitted
>>
>> openssl pkcs12 -export -in signed_certs/${region}.pem -inkey
>> keys/cassandra.${region}.key -name cassy-${region} -out ${region}.p12
>>
>> keytool -importkeystore -deststorepass omitted -destkeystore
>> cassy-${region}.jks -srckeystore ${region}.p12 -srcstoretype PKCS12
>>
>> keytool -importcert -keystore cassy-${region}.jks -alias rootCa -file ca.crt
>> -noprompt -keypass omitted -storepass omitted
>>
>> Looking back at this, I don't remember why we used keytool to generate a
>> keypair/keystore, then deleted and imported. I think it was because the
>> keytool importkeystore command refused to run if the keystore didn't
>> already exist.
>>
>> *ca.crt and pem file*
>>
>> The ca.crt file contains the root certificate and the intermediate
>> certificate that was used to sign the CSR. The pem file contains the signed
>> CSR returned to us, the intermediate cert, and the root CA (in that order).
>>
>> *openssl verify ca.crt and pem*
>>
>> openssl verify -CAfile ca.crt us-west-2.pem
>> signed_certs/us-west-2.pem: OK
>>
>> *Command output after enabling encryption*
>>
>> *nodetool status (output truncated)*
>>
>> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> ?N 52.44.11.221 ? 256 25.4% null
>> 1c
>> ...
>> ?N 52.204.232.195 ? 256 23.2% null
>> 1d
>> Datacenter: us-west-2
>> =====================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> ?N 34.209.2.144 ? 256 26.5% null
>> 2c
>> UN 52.40.32.177 105.99 GiB 256 23.7% null
>> 2c
>> ?N 34.210.109.203 ? 256 24.7% null
>> 2a
>> ...
>>
>> With the online node being the node with encryption set.
>>
>> *cqlsh to localhost*
>>
>> cassy-node6:~$ cqlsh
>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>> Connection refused")})
>>
>> *cqlsh to remote node* Remote node is a node with encryption enabled
>>
>> cassy-node6:~$ cqlsh 10.0.2.7
>> Connection error: ('Unable to connect to any servers', {'10.0.2.7':
>> error(111, "Tried connecting to [('10.0.2.7', 9042)]. Last error: Connection
>> refused")})
>>
>>
