date:20201218

Failed to open native connection to Cassandra

2020-12-18 Thread vikash rajoriya

Hi,

I am running on a 9 node cassandra cluster (cassandra version 3.11.5 , OS:
centos 7.6). We have been running this cluster from long time, but now for
a couple of days we are receiving strange errors from the
same specific node again and again, during the error i don't find anything
unusual on the node. below are the error from application side :



java.util.concurrent.CompletionException: java.io.IOException: Failed
to open native connection to Cassandra at {xx.xx.xx.xx}:9042
at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at 
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.io.IOException: Failed to open native connection to
Cassandra at {xx.xx.xx.xx}:9042
at 
com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:168)
at 
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154)
at 
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154)
at 
com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
at 
com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
at 
com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
at 
com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:79)
at 
com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
at 
com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:122)
at 
com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:332)
at 
com.datastax.spark.connector.cql.Schema$.tableFromCassandra(Schema.scala:352)
at 
com.datastax.spark.connector.writer.ReplicaLocator$.apply(ReplicaLocator.scala:62)
at 
com.datastax.spark.connector.RDDFunctions.repartitionByCassandraReplica(RDDFunctions.scala:238)
at 
com.datastax.spark.connector.japi.RDDJavaFunctions.repartitionByCassandraReplica(RDDJavaFunctions.java:157)
at 
jobs.GenerateContestSplitPDF.getLutvDataSet(GenerateContestSplitPDF.java:119)
at 
jobs.GenerateContestSplitPDF.lambda$main$0(GenerateContestSplitPDF.java:66)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
... 5 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried for query failed (tried:
cps-cassandra/xx.xx.xx.xx:9042
(com.datastax.driver.core.exceptions.TransportException:
[cps-cassandra/xx.xx.xx.xx:9042] Cannot connect))
at 
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:233)
at 
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1483)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:399)
at 
com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:161)
... 21 more
java.lang.Exception: java.util.concurrent.CompletionException:
java.io.IOException: Failed to open native connection to Cassandra at
{xx.xx.xx.xx}:9042
at xx.xx.utils.Utils.handleException(Utils.java:123)
at 
jobs.GenerateContestSplitPDF.lambda$main$3(GenerateContestSplitPDF.java:77)
at 
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at 
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

RE: Cassandra monitoring

2020-12-18 Thread Manu Chadha

Just to try my luck, I added MCAC jar in cassandra.bat in bin directory

set JAVA_OPTS=-ea^
-javaagent:"%CASSANDRA_HOME%\lib\jamm-0.3.0.jar"^
-javaagent:"%CASSANDRA_HOME%\datastax-mcac-agent-0.1.13\lib\datastax-mcac-agent.jar"^
 <-HERE
-Xms2G^
-Xmx2G^
-XX:+HeapDumpOnOutOfMemoryError^
-XX:+UseParNewGC^
-XX:+UseConcMarkSweepGC^
-XX:+CMSParallelRemarkEnabled^
-XX:SurvivorRatio=8^
-XX:MaxTenuringThreshold=1^
-XX:CMSInitiatingOccupancyFraction=75^
-XX:+UseCMSInitiatingOccupancyOnly^
-Dlogback.configurationFile=logback.xml^
-Djava.library.path="%CASSANDRA_HOME%\lib\sigar-bin"^
-Dcassandra.jmx.local.port=7199

But this is now giving error

java.lang.RuntimeException: Cannot locate 
file:\\C:\Users\manuc\Documents\manu\apache-cassandra-3.11.6-for-scalardb\datastax-mcac-agent-0.1.13/config/metric-collector.yaml.

Note the change in path separator from backslash to frontslash at the end.

To be honest, still clueless on how to get metrics on my windows machine.

Sent from Mail for Windows 10

From: Manu Chadha
Sent: 18 December 2020 10:04
To: user@cassandra.apache.org
Subject: RE: Cassandra monitoring

Thanks. I tried it but I still don’t see anything on http://localhost:9103/.

My suspicion is that I am not putting the jars or am not setting the JVM_OPTS 
correctly or maybe the path is not being picked correctly considering that I am 
on windows machine and .sh is basically a unix script. I have tried three 
exporters and they all seem to have issues (except for Cassandra exporter 
standalone which works fine - https://github.com/criteo/cassandra_exporter).

So I am getting more certain that I am not putting javaagent correctly.

I downloaded the jar from 
https://github.com/datastax/metric-collector-for-apache-cassandra/releases/tag/v0.1.13
Then unzipped it

I added path like this

MCAC_ROOT="C:\\Users\\manuc\\Documents\\manu\\datastax-mcac-agent-0.1.13"
JVM_OPTS="$JVM_OPTS -javaagent:${MCAC_ROOT}\\lib\\datastax-mcac-agent.jar"

And then restarted Cassandra but no love ☹

Sent from Mail for Windows 10

From: Ahmed Eljami
Sent: 18 December 2020 09:21
To: user@cassandra.apache.org
Subject: Re: Cassandra monitoring

Hi,

I suggest you try MCAC for DataStax, available here => 
https://github.com/datastax/metric-collector-for-apache-cassandra

Cheers,

Le ven. 18 déc. 2020 à 10:01, Manu Chadha 
mailto:manu.cha...@hotmail.com>> a écrit :
Sorry, clicked send too soon.

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

I thought I’ll test it on my Windows laptop first but I can’t connect with 
using localhost:7199.


As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
>> conf/cassandra-env.sh, I have added the following line in the 
cassandra-env.ps1 (I don’t know to be honest if I should have)

# Default JMX setup, bound to local loopback address only

$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

$env:JVM_OPTS="$env:JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
<-- HERE



$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"

}



but http://localhost:7070/metrics doesn't work

I have also tried to do the original change in cassandar.yaml i.e. 
JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml

But I still can’t access the metrics.

Any help or other suggestions about monitoring would be welcome.

Thanks
Manu

Sent from Mail for Windows 10

From: Manu Chadha
Sent: 18 December 2020 08:57
To: user@cassandra.apache.org
Subject: Cassandra monitoring

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail for Windows 10




--
Cordialement;
Ahmed ELJAMI

RE: Cassandra monitoring

2020-12-18 Thread Manu Chadha

Thanks. I tried it but I still don’t see anything on http://localhost:9103/.

My suspicion is that I am not putting the jars or am not setting the JVM_OPTS 
correctly or maybe the path is not being picked correctly considering that I am 
on windows machine and .sh is basically a unix script. I have tried three 
exporters and they all seem to have issues (except for Cassandra exporter 
standalone which works fine - https://github.com/criteo/cassandra_exporter).

So I am getting more certain that I am not putting javaagent correctly.

I downloaded the jar from 
https://github.com/datastax/metric-collector-for-apache-cassandra/releases/tag/v0.1.13
Then unzipped it

I added path like this

MCAC_ROOT="C:\\Users\\manuc\\Documents\\manu\\datastax-mcac-agent-0.1.13"
JVM_OPTS="$JVM_OPTS -javaagent:${MCAC_ROOT}\\lib\\datastax-mcac-agent.jar"

And then restarted Cassandra but no love ☹

Sent from Mail for Windows 10

From: Ahmed Eljami
Sent: 18 December 2020 09:21
To: user@cassandra.apache.org
Subject: Re: Cassandra monitoring

Hi,

I suggest you try MCAC for DataStax, available here => 
https://github.com/datastax/metric-collector-for-apache-cassandra

Cheers,

Le ven. 18 déc. 2020 à 10:01, Manu Chadha 
mailto:manu.cha...@hotmail.com>> a écrit :
Sorry, clicked send too soon.

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

I thought I’ll test it on my Windows laptop first but I can’t connect with 
using localhost:7199.


As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
>> conf/cassandra-env.sh, I have added the following line in the 
cassandra-env.ps1 (I don’t know to be honest if I should have)

# Default JMX setup, bound to local loopback address only

$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

$env:JVM_OPTS="$env:JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
<-- HERE



$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"

}



but http://localhost:7070/metrics doesn't work

I have also tried to do the original change in cassandar.yaml i.e. 
JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml

But I still can’t access the metrics.

Any help or other suggestions about monitoring would be welcome.

Thanks
Manu

Sent from Mail for Windows 10

From: Manu Chadha
Sent: 18 December 2020 08:57
To: user@cassandra.apache.org
Subject: Cassandra monitoring

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail for Windows 10




--
Cordialement;
Ahmed ELJAMI

Re: Cassandra and GCP

2020-12-18 Thread Ahmed Eljami

Hi,

First, you should deploy with the minimal configuration required for the
production. More details can be found from Cassandra doc here =>
https://cassandra.apache.org/doc/latest/operating/hardware.html

Also, to ensure reliability and fault tolerance you should configure your
cluster to have more than one replica/node (3 is the recommended
replication factor) deployed in two or more region/zone... So, in case of
upgrade/maintenance operation/ unavailable zone... you can switch your
clients to request your second Data Center

Finally, Cassandra is linear scalable, so your cluster can be increased by
simply adding new nodes when your traffic load increases :)

Cheers,

Re: Cassandra monitoring

2020-12-18 Thread Ahmed Eljami

Hi,

I suggest you try MCAC for DataStax, available here =>
https://github.com/datastax/metric-collector-for-apache-cassandra

Cheers,

Le ven. 18 déc. 2020 à 10:01, Manu Chadha  a
écrit :

> Sorry, clicked send too soon.
>
>
>
> I am trying to use Prometheus and followed this tutorial  -
> https://www.robustperception.io/monitoring-cassandra-with-prometheus
>
>
>
> I thought I’ll test it on my Windows laptop first but I can’t connect with
> using localhost:7199.
>
>
>
> As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS
> -javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"'
> >> conf/cassandra-env.sh, I have added the following line in the 
> >> cassandra-env.ps1
> (I don’t know to be honest if I should have)
>
> # Default JMX setup, bound to local loopback address only
>
> $env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"
>
> $env:JVM_OPTS="$env:JVM_OPTS 
> -javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"'
>  <-- HERE
>
>
>
> $env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"
>
> }
>
>
>
> but http://localhost:7070/metrics doesn't work
>
>
>
> I have also tried to do the original change in cassandar.yaml i.e. 
> JVM_OPTS="$JVM_OPTS
> -javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml
>
>
>
> But I still can’t access the metrics.
>
>
>
> Any help or other suggestions about monitoring would be welcome.
>
>
>
> Thanks
>
> Manu
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>
> *From: *Manu Chadha 
> *Sent: *18 December 2020 08:57
> *To: *user@cassandra.apache.org
> *Subject: *Cassandra monitoring
>
>
>
> Hi
>
>
>
> Could the group please suggest options for monitoring Cassandra in
> production (GCP)?
>
>
>
> I am trying to use Prometheus and followed this tutorial  -
> https://www.robustperception.io/monitoring-cassandra-with-prometheus
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>
>
>


-- 
Cordialement;

Ahmed ELJAMI

RE: Cassandra monitoring

2020-12-18 Thread Manu Chadha

Sorry, clicked send too soon.

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

I thought I’ll test it on my Windows laptop first but I can’t connect with 
using localhost:7199.


As I am on Windows, instead of echo 'JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
>> conf/cassandra-env.sh, I have added the following line in the 
cassandra-env.ps1 (I don’t know to be honest if I should have)

# Default JMX setup, bound to local loopback address only

$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"

$env:JVM_OPTS="$env:JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml'"' 
<-- HERE



$env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS"

}



but http://localhost:7070/metrics doesn't work

I have also tried to do the original change in cassandar.yaml i.e. 
JVM_OPTS="$JVM_OPTS 
-javaagent:'$PWD/jmx_prometheus_javaagent-0.3.0.jar=7070:$PWD/cassandra.yml

But I still can’t access the metrics.

Any help or other suggestions about monitoring would be welcome.

Thanks
Manu

Sent from Mail for Windows 10

From: Manu Chadha
Sent: 18 December 2020 08:57
To: user@cassandra.apache.org
Subject: Cassandra monitoring

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail for Windows 10

Cassandra monitoring

2020-12-18 Thread Manu Chadha

Hi

Could the group please suggest options for monitoring Cassandra in production 
(GCP)?

I am trying to use Prometheus and followed this tutorial  - 
https://www.robustperception.io/monitoring-cassandra-with-prometheus

Sent from Mail for Windows 10

Failed to open native connection to Cassandra

RE: Cassandra monitoring

RE: Cassandra monitoring

Re: Cassandra and GCP

Re: Cassandra monitoring

RE: Cassandra monitoring

Cassandra monitoring

7 matches

Site Navigation

Mail list logo

Footer information