fasar created CASSANDRA-21023:
---------------------------------
Summary: sstableloader time-out on Cassandra 4.1.x with SSL
Configuration
Key: CASSANDRA-21023
URL: https://issues.apache.org/jira/browse/CASSANDRA-21023
Project: Apache Cassandra
Issue Type: Bug
Components: Tool/bulk load
Reporter: fasar
Hello Community !!!
I am trying to use _sstableloader_ to restore a snapshot.
Cassandra is configured with {color:#00875a}_internode_encryption_{color} and
{color:#00875a}_client_encryption_{color} enabled.
I have a test cluster with 3 nodes and a recovery cluster with 42 nodes ({_}3
racks{_} in each environment).
Both clusters have Cassandra _4.1.4_ installed.
When I try sstableloader, it terminates the connection with the error:
`{_}{color:#4c9aff}Operation timed out{color}{_}`.
I test with Cassandra _4.1.10_ and it is the same output.
Below is the command and the trace output of the sstableloader run:
```bash
/var/opt/cassandra/apache-cassandra-4.1.4/bin/sstableloader -d
cassandra-test-rack1-01,cassandra-test-rack2-01,cassandra-test-rack3-01 -f
/var/opt/cassandra/apache-cassandra-4.1.4/conf/cassandra.yaml -u user1 -pw
password /data/cassandra-snap/test/testtable
```
```log
EBUG 13:55:45,457 Loaded default ResourceLeakDetector:
com.datastax.shaded.netty.util.ResourceLeakDetector@3c7175ae
DEBUG 13:55:45,555 [id: 0x446e44c3, L:/10.10.10.214:52930 -
R:cassandra-test-rack1-01/10.10.10.214:9042] HANDSHAKEN: TLS_AES_256_GCM_SHA384
DEBUG 13:55:57,532 Defuncting
Connection[cassandra-test-rack1-01/10.10.10.214:9042-1, inFlight=0,
closed=false] because: [cassandra-test-rack1-01/10.10.10.214:9042] Operation
timed out
DEBUG 13:55:57,533 [cassandra-test-rack1-01/10.10.10.214:9042] preventing new
connections for the next 1000 ms
DEBUG 13:55:57,533 [cassandra-test-rack1-01/10.10.10.214:9042]
Connection[cassandra-test-rack1-01/10.10.10.214:9042-1, inFlight=0,
closed=false] failed, remaining = 0
DEBUG 13:55:57,533 Connection[cassandra-test-rack1-01/10.10.10.214:9042-1,
inFlight=0, closed=true] closing connection
DEBUG 13:55:57,533 Not terminating
Connection[cassandra-test-rack1-01/10.10.10.214:9042-1, inFlight=0,
closed=true]: there are still pending requests
DEBUG 13:55:57,534 [Control connection] error on
cassandra-test-rack1-01/10.10.10.214:9042 connection, trying next host
com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack1-01/10.10.10.214:9042] Operation timed out
at
com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1557)
at
com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1636)
at
com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
at
com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
at
com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
DEBUG 13:55:57,534 [cassandra-test-rack2-01/10.10.10.213:9042] preparing to
open 1 new connections, total = 1
DEBUG 13:55:57,535 Connection[cassandra-test-rack1-01/10.10.10.214:9042-1,
inFlight=0, closed=true], stream 0, Error writing request STARTUP
\{CQL_VERSION=3.0.0, DRIVER_VERSION=3.11.0, DRIVER_NAME=DataStax Java Driver}
DEBUG 13:55:57,536 Connection[cassandra-test-rack2-01/10.10.10.213:9042-1,
inFlight=0, closed=false] Connection established, initializing transport
DEBUG 13:55:57,574 [id: 0x870adc85, L:/10.10.10.214:59728 -
R:cassandra-test-rack2-01/10.10.10.213:9042] HANDSHAKEN: TLS_AES_256_GCM_SHA384
DEBUG 13:56:09,631 Defuncting
Connection[cassandra-test-rack2-01/10.10.10.213:9042-1, inFlight=0,
closed=false] because: [cassandra-test-rack2-01/10.10.10.213:9042] Operation
timed out
DEBUG 13:56:09,632 [cassandra-test-rack2-01/10.10.10.213:9042] preventing new
connections for the next 1000 ms
DEBUG 13:56:09,632 [cassandra-test-rack2-01/10.10.10.213:9042]
Connection[cassandra-test-rack2-01/10.10.10.213:9042-1, inFlight=0,
closed=false] failed, remaining = 0
DEBUG 13:56:09,632 Connection[cassandra-test-rack2-01/10.10.10.213:9042-1,
inFlight=0, closed=true] closing connection
DEBUG 13:56:09,632 Not terminating
Connection[cassandra-test-rack2-01/10.10.10.213:9042-1, inFlight=0,
closed=true]: there are still pending requests
DEBUG 13:56:09,633 [Control connection] error on
cassandra-test-rack2-01/10.10.10.213:9042 connection, trying next host
com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack2-01/10.10.10.213:9042] Operation timed out
at
com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1557)
at
com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1636)
at
com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
at
com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
at
com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
DEBUG 13:56:09,633 [cassandra-test-rack3-01/10.10.10.215:9042] preparing to
open 1 new connections, total = 1
DEBUG 13:56:09,633 Connection[cassandra-test-rack2-01/10.10.10.213:9042-1,
inFlight=0, closed=true], stream 0, Error writing request STARTUP
\{CQL_VERSION=3.0.0, DRIVER_VERSION=3.11.0, DRIVER_NAME=DataStax Java Driver}
DEBUG 13:56:09,634 Connection[cassandra-test-rack3-01/10.10.10.215:9042-1,
inFlight=0, closed=false] Connection established, initializing transport
DEBUG 13:56:09,661 [id: 0x08eded5b, L:/10.10.10.214:45150 -
R:cassandra-test-rack3-01/10.10.10.215:9042] HANDSHAKEN: TLS_AES_256_GCM_SHA384
DEBUG 13:56:21,731 Defuncting
Connection[cassandra-test-rack3-01/10.10.10.215:9042-1, inFlight=0,
closed=false] because: [cassandra-test-rack3-01/10.10.10.215:9042] Operation
timed out
DEBUG 13:56:21,732 [cassandra-test-rack3-01/10.10.10.215:9042] preventing new
connections for the next 1000 ms
DEBUG 13:56:21,732 [cassandra-test-rack3-01/10.10.10.215:9042]
Connection[cassandra-test-rack3-01/10.10.10.215:9042-1, inFlight=0,
closed=false] failed, remaining = 0
DEBUG 13:56:21,732 Connection[cassandra-test-rack3-01/10.10.10.215:9042-1,
inFlight=0, closed=true] closing connection
DEBUG 13:56:21,732 Not terminating
Connection[cassandra-test-rack3-01/10.10.10.215:9042-1, inFlight=0,
closed=true]: there are still pending requests
DEBUG 13:56:21,732 [Control connection] error on
cassandra-test-rack3-01/10.10.10.215:9042 connection, no more host to try
com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack3-01/10.10.10.215:9042] Operation timed out
at
com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1557)
at
com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1636)
at
com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
at
com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
at
com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
DEBUG 13:56:21,732 Connection[cassandra-test-rack3-01/10.10.10.215:9042-1,
inFlight=0, closed=true], stream 0, Error writing request STARTUP
\{CQL_VERSION=3.0.0, DRIVER_VERSION=3.11.0, DRIVER_NAME=DataStax Java Driver}
DEBUG 13:56:21,733 Shutting down
DEBUG 13:56:21,740 Freed 12 thread-local buffer(s) from thread:
cluster1-nio-worker-0
DEBUG 13:56:21,740 Freed 12 thread-local buffer(s) from thread:
cluster1-nio-worker-1
DEBUG 13:56:23,943 Freed 12 thread-local buffer(s) from thread:
cluster1-nio-worker-2
DEBUG 13:56:23,950 Connection[cassandra-test-rack1-01/10.10.10.214:9042-1,
inFlight=0, closed=true] has already terminated
Unable to initialise org.apache.cassandra.utils.NativeSSTableLoaderClient
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried: cassandra-test-rack2-01/10.10.10.213:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack2-01/10.10.10.213:9042] Operation timed out),
cassandra-test-rack3-01/10.10.10.215:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack3-01/10.10.10.215:9042] Operation timed out),
cassandra-test-rack1-01/10.10.10.214:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack1-01/10.10.10.214:9042] Operation timed out))
java.lang.RuntimeException: Unable to initialise
org.apache.cassandra.utils.NativeSSTableLoaderClient
at
org.apache.cassandra.utils.NativeSSTableLoaderClient.init(NativeSSTableLoaderClient.java:102)
at
org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:167)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:91)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:58)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried:
cassandra-test-rack2-01/10.10.10.213:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack2-01/10.10.10.213:9042] Operation timed out),
cassandra-test-rack3-01/10.10.10.215:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack3-01/10.10.10.215:9042] Operation timed out),
cassandra-test-rack1-01/10.10.10.214:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack1-01/10.10.10.214:9042] Operation timed out))
at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:270)
at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:109)
at
com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1813)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1726)
at com.datastax.driver.core.Cluster.init(Cluster.java:214)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:387)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:366)
at com.datastax.driver.core.Cluster.connect(Cluster.java:311)
at
org.apache.cassandra.utils.NativeSSTableLoaderClient.init(NativeSSTableLoaderClient.java:70)
... 3 more
Exception in thread "main" org.apache.cassandra.tools.BulkLoadException:
java.lang.RuntimeException: Unable to initialise
org.apache.cassandra.utils.NativeSSTableLoaderClient
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:104)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:58)
Caused by: java.lang.RuntimeException: Unable to initialise
org.apache.cassandra.utils.NativeSSTableLoaderClient
at
org.apache.cassandra.utils.NativeSSTableLoaderClient.init(NativeSSTableLoaderClient.java:102)
at
org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:167)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:91)
... 1 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried:
cassandra-test-rack2-01/10.10.10.213:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack2-01/10.10.10.213:9042] Operation timed out),
cassandra-test-rack3-01/10.10.10.215:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack3-01/10.10.10.215:9042] Operation timed out),
cassandra-test-rack1-01/10.10.10.214:9042
(com.datastax.driver.core.exceptions.OperationTimedOutException:
[cassandra-test-rack1-01/10.10.10.214:9042] Operation timed out))
at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:270)
at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:109)
at
com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1813)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1726)
at com.datastax.driver.core.Cluster.init(Cluster.java:214)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:387)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:366)
at com.datastax.driver.core.Cluster.connect(Cluster.java:311)
at
org.apache.cassandra.utils.NativeSSTableLoaderClient.init(NativeSSTableLoaderClient.java:70)
```
However, using the *sstableloader from Cassandra _5.0.6_* to load sstables on
the Cassandra Cluster 4.1.4 {*}works as expected{*}.
Based on my initial investigation, the {*}cassandra-driver-core versions 3.11.0
and 3.11.1 seem to be incompatible with my Cassandra 4.1.x configuration{*}.
Starting from cassandra-driver-core version 3.11.2, the issue is resolved.
To verify this, I created a Java project based on an example I found in the
cassandra-java-driver version 3.11.1.
(Reference: ReadTopologyAndSchemaMetadata.java)
I simply added .ssl() to the cluster constructor in the example code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
