No Host AvailableException during querying Cassandra.

[Venkata D]

Hello All,

We are using DSE 4.6.6 & Cassandra 2.0.14.425.

I am facing this exception right now. We got this exception couple of times
& repair jobs helped us temporarily.

As the data is growing significantly we are experiencing this exception
more than couple of times. Does any one have any thoughts on this ?


Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 114 in stage 17.0 failed 4 times, most recent
failure: Lost task 114.3 in stage 17.0 (TID 196, ):
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
tried for query failed (tried: [All IP addresses] - use getErrors() for
details)
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(
NoHostAvailableException.java:65)
com.datastax.driver.core.DefaultResultSetFuture.
extractCauseFromExecutionException(DefaultResultSetFuture.java:259)
com.datastax.driver.core.ArrayBackedResultSet$
MultiPage.prepareNextRow(ArrayBackedResultSet.java:279)
com.datastax.driver.core.ArrayBackedResultSet$MultiPage.isExhausted(
ArrayBackedResultSet.java:239)
com.datastax.driver.core.ArrayBackedResultSet$1.
hasNext(ArrayBackedResultSet.java:122)
com.datastax.spark.connector.rdd.reader.
PrefetchingResultSetIterator.hasNext(PrefetchingResultSetIterator.scala:16)
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
com.datastax.spark.connector.util.CountingIterator.hasNext(
CountingIterator.scala:10)
scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
org.apache.spark.storage.MemoryStore.unrollSafely(
MemoryStore.scala:235)
org.apache.spark.CacheManager.putInBlockManager(
CacheManager.scala:163)
org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(
Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)


Thanks,
Venkat.

Re: [Multi DC] Old Data Not syncing from Existing cluster to new Cluster

[kurt greaves]

What Dikang said, in your original email you are passing -dc to rebuild.
This is incorrect. Simply run nodetool rebuild  from each of the
nodes in the new dc.

On 28 Jan 2017 07:50, "Dikang Gu"  wrote:

> Have you run 'nodetool rebuild dc_india' on the new nodes?
>
> On Tue, Jan 24, 2017 at 7:51 AM, Benjamin Roth 
> wrote:
>
>> Have you also altered RF of system_distributed as stated in the tutorial?
>>
>> 2017-01-24 16:45 GMT+01:00 Abhishek Kumar Maheshwari <
>> abhishek.maheshw...@timesinternet.in>:
>>
>>> My Mistake,
>>>
>>>
>>>
>>> Both clusters are up and running.
>>>
>>>
>>>
>>> Datacenter: DRPOCcluster
>>>
>>> 
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  AddressLoad   Tokens   OwnsHost
>>> ID   Rack
>>>
>>> UN  172.29.XX.XX  1.65 GB   256  ?
>>> badf985b-37da-4735-b468-8d3a058d4b60  01
>>>
>>> UN  172.29.XX.XX  1.64 GB   256  ?
>>> 317061b2-c19f-44ba-a776-bcd91c70bbdd  03
>>>
>>> UN  172.29.XX.XX  1.64 GB   256  ?
>>> 9bf0d1dc-6826-4f3b-9c56-cec0c9ce3b6c  02
>>>
>>> Datacenter: dc_india
>>>
>>> 
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  AddressLoad   Tokens   OwnsHost
>>> ID   Rack
>>>
>>> UN  172.26.XX.XX   79.90 GB   256  ?
>>> 3e8133ed-98b5-418d-96b5-690a1450cd30  RACK1
>>>
>>> UN  172.26.XX.XX   80.21 GB   256  ?
>>> 7d3f5b25-88f9-4be7-b0f5-746619153543  RACK2
>>>
>>>
>>>
>>> *Thanks & Regards,*
>>> *Abhishek Kumar Maheshwari*
>>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>>
>>> Times Internet Ltd. | A Times of India Group Company
>>>
>>> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>>>
>>> *P** Please do not print this email unless it is absolutely necessary.
>>> Spread environmental awareness.*
>>>
>>>
>>>
>>> *From:* Benjamin Roth [mailto:benjamin.r...@jaumo.com]
>>> *Sent:* Tuesday, January 24, 2017 9:11 PM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: [Multi DC] Old Data Not syncing from Existing cluster to
>>> new Cluster
>>>
>>>
>>>
>>> I am not an expert in bootstrapping new DCs but shouldn't the OLD nodes
>>> appear as UP to be used as a streaming source in rebuild?
>>>
>>>
>>>
>>> 2017-01-24 16:32 GMT+01:00 Abhishek Kumar Maheshwari <
>>> abhishek.maheshw...@timesinternet.in>:
>>>
>>> Yes, I take all steps. While I am inserting new data is replicating on
>>> both DC. But only old data is not replication in new cluster.
>>>
>>>
>>>
>>> *Thanks & Regards,*
>>> *Abhishek Kumar Maheshwari*
>>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>>
>>> Times Internet Ltd. | A Times of India Group Company
>>>
>>> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>>>
>>> *P** Please do not print this email unless it is absolutely necessary.
>>> Spread environmental awareness.*
>>>
>>>
>>>
>>> *From:* Benjamin Roth [mailto:benjamin.r...@jaumo.com]
>>> *Sent:* Tuesday, January 24, 2017 8:55 PM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: [Multi DC] Old Data Not syncing from Existing cluster to
>>> new Cluster
>>>
>>>
>>>
>>> There is much more to it than just changing the RF in the keyspace!
>>>
>>>
>>>
>>> See here: https://docs.datastax.com/en/cassandra/3.0/cassandra/o
>>> perations/opsAddDCToCluster.html
>>>
>>>
>>>
>>> 2017-01-24 16:18 GMT+01:00 Abhishek Kumar Maheshwari <
>>> abhishek.maheshw...@timesinternet.in>:
>>>
>>> Hi All,
>>>
>>>
>>>
>>> I have Cassandra stack with 2 Dc
>>>
>>>
>>>
>>> Datacenter: DRPOCcluster
>>>
>>> 
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  AddressLoad   Tokens   OwnsHost
>>> ID   Rack
>>>
>>> UN  172.29.xx.xxx  256  MB   256  ?
>>> b6b8cbb9-1fed-471f-aea9-6a657e7ac80a  01
>>>
>>> UN  172.29.xx.xxx  240 MB   256  ?
>>> 604abbf5-8639-4104-8f60-fd6573fb2e17  03
>>>
>>> UN  172.29. xx.xxx  240 MB   256  ?
>>> 32fa79ee-93c6-4e5b-a910-f27a1e9d66c1  02
>>>
>>> Datacenter: dc_india
>>>
>>> 
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  AddressLoad   Tokens   OwnsHost
>>> ID   Rack
>>>
>>> DN  172.26. .xx.xxx  78.97 GB   256  ?
>>> 3e8133ed-98b5-418d-96b5-690a1450cd30  RACK1
>>>
>>> DN  172.26. .xx.xxx  79.18 GB   256  ?
>>> 7d3f5b25-88f9-4be7-b0f5-746619153543  RACK2
>>>
>>>
>>>
>>> dc_india is old Dc which contains all data.
>>>
>>> I update keyspace as per below:
>>>
>>>
>>>
>>> alter KEYSPACE wls WITH replication = {'class':
>>> 'NetworkTopologyStrategy', 'DRPOCcluster': '2','dc_india':'2'}  AND
>>> durable_writes = true;
>>>
>>>
>>>
>>> but old data is not updating in DRPOCcluster(which is new). Also, while
>>> running nodetool rebuild getting below exception:
>>>
>>> Cammand: ./nodetool rebuild -dc dc_india
>>>

Re: [Multi DC] Old Data Not syncing from Existing cluster to new Cluster

[Dikang Gu]

Have you run 'nodetool rebuild dc_india' on the new nodes?

On Tue, Jan 24, 2017 at 7:51 AM, Benjamin Roth 
wrote:

> Have you also altered RF of system_distributed as stated in the tutorial?
>
> 2017-01-24 16:45 GMT+01:00 Abhishek Kumar Maheshwari  timesinternet.in>:
>
>> My Mistake,
>>
>>
>>
>> Both clusters are up and running.
>>
>>
>>
>> Datacenter: DRPOCcluster
>>
>> 
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  AddressLoad   Tokens   OwnsHost
>> ID   Rack
>>
>> UN  172.29.XX.XX  1.65 GB   256  ?
>> badf985b-37da-4735-b468-8d3a058d4b60  01
>>
>> UN  172.29.XX.XX  1.64 GB   256  ?
>> 317061b2-c19f-44ba-a776-bcd91c70bbdd  03
>>
>> UN  172.29.XX.XX  1.64 GB   256  ?
>> 9bf0d1dc-6826-4f3b-9c56-cec0c9ce3b6c  02
>>
>> Datacenter: dc_india
>>
>> 
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  AddressLoad   Tokens   OwnsHost
>> ID   Rack
>>
>> UN  172.26.XX.XX   79.90 GB   256  ?
>> 3e8133ed-98b5-418d-96b5-690a1450cd30  RACK1
>>
>> UN  172.26.XX.XX   80.21 GB   256  ?
>> 7d3f5b25-88f9-4be7-b0f5-746619153543  RACK2
>>
>>
>>
>> *Thanks & Regards,*
>> *Abhishek Kumar Maheshwari*
>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>
>> Times Internet Ltd. | A Times of India Group Company
>>
>> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>>
>> *P** Please do not print this email unless it is absolutely necessary.
>> Spread environmental awareness.*
>>
>>
>>
>> *From:* Benjamin Roth [mailto:benjamin.r...@jaumo.com]
>> *Sent:* Tuesday, January 24, 2017 9:11 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: [Multi DC] Old Data Not syncing from Existing cluster to
>> new Cluster
>>
>>
>>
>> I am not an expert in bootstrapping new DCs but shouldn't the OLD nodes
>> appear as UP to be used as a streaming source in rebuild?
>>
>>
>>
>> 2017-01-24 16:32 GMT+01:00 Abhishek Kumar Maheshwari <
>> abhishek.maheshw...@timesinternet.in>:
>>
>> Yes, I take all steps. While I am inserting new data is replicating on
>> both DC. But only old data is not replication in new cluster.
>>
>>
>>
>> *Thanks & Regards,*
>> *Abhishek Kumar Maheshwari*
>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>
>> Times Internet Ltd. | A Times of India Group Company
>>
>> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>>
>> *P** Please do not print this email unless it is absolutely necessary.
>> Spread environmental awareness.*
>>
>>
>>
>> *From:* Benjamin Roth [mailto:benjamin.r...@jaumo.com]
>> *Sent:* Tuesday, January 24, 2017 8:55 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: [Multi DC] Old Data Not syncing from Existing cluster to
>> new Cluster
>>
>>
>>
>> There is much more to it than just changing the RF in the keyspace!
>>
>>
>>
>> See here: https://docs.datastax.com/en/cassandra/3.0/cassandra/
>> operations/opsAddDCToCluster.html
>>
>>
>>
>> 2017-01-24 16:18 GMT+01:00 Abhishek Kumar Maheshwari <
>> abhishek.maheshw...@timesinternet.in>:
>>
>> Hi All,
>>
>>
>>
>> I have Cassandra stack with 2 Dc
>>
>>
>>
>> Datacenter: DRPOCcluster
>>
>> 
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  AddressLoad   Tokens   OwnsHost
>> ID   Rack
>>
>> UN  172.29.xx.xxx  256  MB   256  ?
>> b6b8cbb9-1fed-471f-aea9-6a657e7ac80a  01
>>
>> UN  172.29.xx.xxx  240 MB   256  ?
>> 604abbf5-8639-4104-8f60-fd6573fb2e17  03
>>
>> UN  172.29. xx.xxx  240 MB   256  ?
>> 32fa79ee-93c6-4e5b-a910-f27a1e9d66c1  02
>>
>> Datacenter: dc_india
>>
>> 
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  AddressLoad   Tokens   OwnsHost
>> ID   Rack
>>
>> DN  172.26. .xx.xxx  78.97 GB   256  ?
>> 3e8133ed-98b5-418d-96b5-690a1450cd30  RACK1
>>
>> DN  172.26. .xx.xxx  79.18 GB   256  ?
>> 7d3f5b25-88f9-4be7-b0f5-746619153543  RACK2
>>
>>
>>
>> dc_india is old Dc which contains all data.
>>
>> I update keyspace as per below:
>>
>>
>>
>> alter KEYSPACE wls WITH replication = {'class':
>> 'NetworkTopologyStrategy', 'DRPOCcluster': '2','dc_india':'2'}  AND
>> durable_writes = true;
>>
>>
>>
>> but old data is not updating in DRPOCcluster(which is new). Also, while
>> running nodetool rebuild getting below exception:
>>
>> Cammand: ./nodetool rebuild -dc dc_india
>>
>>
>>
>> Exception : nodetool: Unable to find sufficient sources for streaming
>> range (-875697427424852,-8755484427030035332] in keyspace
>> system_distributed
>>
>>
>>
>> Cassandra version : 3.0.9
>>
>>
>>
>>
>>
>> *Thanks & Regards,*
>> *Abhishek Kumar Maheshwari*
>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>
>> Times Internet Ltd. | A Times of India Group Company
>>
>> FC - 6, Sector 16A, Film C

Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 2.1.12 - 2.1.16

[sai krishnam raju potturi]

FYI : This issue is related to CASSANDRA-10205
 (Gossiper
class) patch introduced in 2.1.11. When we roll back the changes from
CASSANDRA-10205
 (Gossiper
class) in 2.1.12 and 2.1.15, everything works as expected. Further tests
still need to be done on our end though.

One more thing observed was that the decommissioned nodes do not show up as
"UNREACHABLE" in the "nodetool describecluster" after 72 hours. Things are
normal.

thanks Pillai; but the ip-address does not exist in the system-peers table
on any of the nodes. Unsafe-assasinate is not our preferred option when we
decommission a datacenter consisting of more than 100 nodes.

Kurt; we have not tested out 2.1.7 and 2.1.8 versions yet

Pratik; i'm not sure if your issue relates to this, as we observe the node
as UNREACHABLE in the "nodetool describecluster". nodetool gossipinfo
should generally show the information of the decommissioned nodes for a
while, which is expected behaviour.

thanks
Sai




On Fri, Jan 27, 2017 at 12:54 PM, Harikrishnan Pillai <
hpil...@walmartlabs.com> wrote:

> Please remove the ips from the system.peer table of all nodes  or you can
> use unsafeassasinate from JMX.
>
>
> --
> *From:* Agrawal, Pratik 
> *Sent:* Friday, January 27, 2017 9:05:43 AM
> *To:* user@cassandra.apache.org; k...@instaclustr.com; pskraj...@gmail.com
> *Cc:* Sun, Guan
>
> *Subject:* Re: Re : Decommissioned nodes show as DOWN in Cassandra
> versions 2.1.12 - 2.1.16
>
> We are seeing the same issue with Cassandra 2.0.8. The nodetool gossipinfo
> reports a node being down even after we decommission the node from the
> cluster.
>
> Thanks,
> Pratik
>
> From: kurt greaves 
> Reply-To: "user@cassandra.apache.org" 
> Date: Friday, January 27, 2017 at 5:54 AM
> To: "user@cassandra.apache.org" 
> Subject: Re: Re : Decommissioned nodes show as DOWN in Cassandra versions
> 2.1.12 - 2.1.16
>
> we've seen this issue on a few clusters, including on 2.1.7 and 2.1.8.
> pretty sure it is an issue in gossip that's known about. in later versions
> it seems to be fixed.
>
> On 24 Jan 2017 06:09, "sai krishnam raju potturi" 
> wrote:
>
>> In the Cassandra versions 2.1.11 - 2.1.16, after we decommission a node
>> or datacenter, we observe the decommissioned nodes marked as DOWN in the
>> cluster when you do a "nodetool describecluster". The nodes however do not
>> show up in the "nodetool status" command.
>> The decommissioned node also does not show up in the "system_peers" table
>> on the nodes.
>>
>> The workaround we follow is rolling restart of the cluster, which removes
>> the decommissioned nodes from the "UNREACHABLE STATE", and shows the actual
>> state of the cluster. The workaround is tedious for huge clusters.
>>
>> We also verified the decommission process in CCM tool, and observed the
>> same issue for clusters with versions from 2.1.12 to 2.1.16. The issue was
>> not observed in versions prior to or later than the ones mentioned above.
>>
>>
>> Has anybody in the community observed similar issue? We've also raised a
>> JIRA issue regarding this.   https://issues.apache.org/jira
>> /browse/CASSANDRA-13144
>>
>>
>> Below are the observed logs from the versions without the bug, and with
>> the bug.  The one's highlighted in yellow show the expected logs. The one's
>> highlighted in red are the one's where the node is recognized as down, and
>> shows as UNREACHABLE.
>>
>>
>>
>> Cassandra 2.1.1 Logs showing the decommissioned node :  (Without the bug)
>>
>> 2017-01-19 20:18:56,415 [GossipStage:1] DEBUG ArrivalWindow Ignoring
>> interval time of 2049943233 <(204)%20994-3233> for /X.X.X.X
>> 2017-01-19 20:18:56,416 [GossipStage:1] DEBUG StorageService Node
>> /X.X.X.X state left, tokens [ 59353109817657926242901533144729725259,
>> 60254520910109313597677907197875221475, 
>> 75698727618038614819889933974570742305,
>> 84508739091270910297310401957975430578]
>> 2017-01-19 20:18:56,416 [GossipStage:1] DEBUG Gossiper adding expire
>> time for endpoint : /X.X.X.X (1485116334088)
>> 2017-01-19 20:18:56,417 [GossipStage:1] INFO StorageService Removing
>> tokens [100434964734820719895982857900842892337,
>> 114144647582686041354301802358217767299, 
>> 13209060517964702932350041942412177,
>> 138409460913927199437556572481804704749] for /X.X.X.X
>> 2017-01-19 20:18:56,418 [HintedHandoff:3] INFO HintedHandOffManager
>> Deleting any stored hints for /X.X.X.X
>> 2017-01-19 20:18:56,424 [GossipStage:1] DEBUG MessagingService Resetting
>> version for /X.X.X.X
>> 2017-01-19 20:18:56,424 [GossipStage:1] DEBUG Gossiper removing endpoint
>> /X.X.X.X
>> 2017-01-19 20:18:56,437 [GossipStage:1] DEBUG StorageService Ignoring
>> state change for dead or unknown endpoint: /X.X.X.X
>> 2017-01-19 20:19:02,022 [WRITE-/X.X.X.X] DEBUG OutboundTcpConnection
>> attempting to connect to /X.X.X.X
>> 2017-01-19 20:19:02,023 [HANDSHAKE-/X.X.X.X] INFO OutboundTcpC

Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 2.1.12 - 2.1.16

[Harikrishnan Pillai]

Please remove the ips from the system.peer table of all nodes  or you can use 
unsafeassasinate from JMX.



From: Agrawal, Pratik 
Sent: Friday, January 27, 2017 9:05:43 AM
To: user@cassandra.apache.org; k...@instaclustr.com; pskraj...@gmail.com
Cc: Sun, Guan
Subject: Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 
2.1.12 - 2.1.16

We are seeing the same issue with Cassandra 2.0.8. The nodetool gossipinfo 
reports a node being down even after we decommission the node from the cluster.

Thanks,
Pratik

From: kurt greaves mailto:k...@instaclustr.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday, January 27, 2017 at 5:54 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 
2.1.12 - 2.1.16

we've seen this issue on a few clusters, including on 2.1.7 and 2.1.8. pretty 
sure it is an issue in gossip that's known about. in later versions it seems to 
be fixed.

On 24 Jan 2017 06:09, "sai krishnam raju potturi" 
mailto:pskraj...@gmail.com>> wrote:

In the Cassandra versions 2.1.11 - 2.1.16, after we decommission a node or 
datacenter, we observe the decommissioned nodes marked as DOWN in the cluster 
when you do a "nodetool describecluster". The nodes however do not show up in 
the "nodetool status" command.
The decommissioned node also does not show up in the "system_peers" table on 
the nodes.

The workaround we follow is rolling restart of the cluster, which removes the 
decommissioned nodes from the "UNREACHABLE STATE", and shows the actual state 
of the cluster. The workaround is tedious for huge clusters.

We also verified the decommission process in CCM tool, and observed the same 
issue for clusters with versions from 2.1.12 to 2.1.16. The issue was not 
observed in versions prior to or later than the ones mentioned above.


Has anybody in the community observed similar issue? We've also raised a JIRA 
issue regarding this.   https://issues.apache.org/jira/browse/CASSANDRA-13144


Below are the observed logs from the versions without the bug, and with the 
bug.  The one's highlighted in yellow show the expected logs. The one's 
highlighted in red are the one's where the node is recognized as down, and 
shows as UNREACHABLE.



Cassandra 2.1.1 Logs showing the decommissioned node :  (Without the bug)

2017-01-19 20:18:56,415 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 2049943233 for /X.X.X.X
2017-01-19 20:18:56,416 [GossipStage:1] DEBUG StorageService Node /X.X.X.X 
state left, tokens [ 59353109817657926242901533144729725259, 
60254520910109313597677907197875221475, 75698727618038614819889933974570742305, 
84508739091270910297310401957975430578]
2017-01-19 20:18:56,416 [GossipStage:1] DEBUG Gossiper adding expire time for 
endpoint : /X.X.X.X (1485116334088)
2017-01-19 20:18:56,417 [GossipStage:1] INFO StorageService Removing tokens 
[100434964734820719895982857900842892337, 
114144647582686041354301802358217767299, 
13209060517964702932350041942412177, 
138409460913927199437556572481804704749] for /X.X.X.X
2017-01-19 20:18:56,418 [HintedHandoff:3] INFO HintedHandOffManager Deleting 
any stored hints for /X.X.X.X
2017-01-19 20:18:56,424 [GossipStage:1] DEBUG MessagingService Resetting 
version for /X.X.X.X
2017-01-19 20:18:56,424 [GossipStage:1] DEBUG Gossiper removing endpoint 
/X.X.X.X
2017-01-19 20:18:56,437 [GossipStage:1] DEBUG StorageService Ignoring state 
change for dead or unknown endpoint: /X.X.X.X
2017-01-19 20:19:02,022 [WRITE-/X.X.X.X] DEBUG OutboundTcpConnection attempting 
to connect to /X.X.X.X
2017-01-19 20:19:02,023 [HANDSHAKE-/X.X.X.X] INFO OutboundTcpConnection 
Handshaking version with /X.X.X.X
2017-01-19 20:19:02,023 [WRITE-/X.X.X.X] DEBUG MessagingService Setting version 
7 for /X.X.X.X
2017-01-19 20:19:08,096 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 2074454222 for /X.X.X.X
2017-01-19 20:19:54,407 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 4302985797 for /X.X.X.X
2017-01-19 20:19:57,405 [GossipTasks:1] DEBUG Gossiper 6 elapsed, /X.X.X.X 
gossip quarantine over
2017-01-19 20:19:57,455 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 3047826501 for /X.X.X.X
2017-01-19 20:19:57,455 [GossipStage:1] DEBUG StorageService Ignoring state 
change for dead or unknown endpoint: /X.X.X.X


Cassandra 2.1.16 Logs showing the decommissioned node :   (The logs in 2.1.16 
show the same as 2.1.1 upto "DEBUG Gossiper 6 elapsed, /X.X.X.X gossip 
quarantine over", and then is followed by "NODE is now DOWN"

017-01-19 19:52:23,687 [GossipStage:1] DEBUG StorageService.java:1883 - Node 
/X.X.X.X state left, tokens [-1112888759032625467, -228773855963737699, 
-311455042375
4381391, -4848625944949064281, -6920961603460018610, -8566729719076824066, 
1611098831406674636, 72788

Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 2.1.12 - 2.1.16

[Agrawal, Pratik]

We are seeing the same issue with Cassandra 2.0.8. The nodetool gossipinfo 
reports a node being down even after we decommission the node from the cluster.

Thanks,
Pratik

From: kurt greaves mailto:k...@instaclustr.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Friday, January 27, 2017 at 5:54 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 
2.1.12 - 2.1.16

we've seen this issue on a few clusters, including on 2.1.7 and 2.1.8. pretty 
sure it is an issue in gossip that's known about. in later versions it seems to 
be fixed.

On 24 Jan 2017 06:09, "sai krishnam raju potturi" 
mailto:pskraj...@gmail.com>> wrote:

In the Cassandra versions 2.1.11 - 2.1.16, after we decommission a node or 
datacenter, we observe the decommissioned nodes marked as DOWN in the cluster 
when you do a "nodetool describecluster". The nodes however do not show up in 
the "nodetool status" command.
The decommissioned node also does not show up in the "system_peers" table on 
the nodes.

The workaround we follow is rolling restart of the cluster, which removes the 
decommissioned nodes from the "UNREACHABLE STATE", and shows the actual state 
of the cluster. The workaround is tedious for huge clusters.

We also verified the decommission process in CCM tool, and observed the same 
issue for clusters with versions from 2.1.12 to 2.1.16. The issue was not 
observed in versions prior to or later than the ones mentioned above.


Has anybody in the community observed similar issue? We've also raised a JIRA 
issue regarding this.   https://issues.apache.org/jira/browse/CASSANDRA-13144


Below are the observed logs from the versions without the bug, and with the 
bug.  The one's highlighted in yellow show the expected logs. The one's 
highlighted in red are the one's where the node is recognized as down, and 
shows as UNREACHABLE.



Cassandra 2.1.1 Logs showing the decommissioned node :  (Without the bug)

2017-01-19 20:18:56,415 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 2049943233 for /X.X.X.X
2017-01-19 20:18:56,416 [GossipStage:1] DEBUG StorageService Node /X.X.X.X 
state left, tokens [ 59353109817657926242901533144729725259, 
60254520910109313597677907197875221475, 75698727618038614819889933974570742305, 
84508739091270910297310401957975430578]
2017-01-19 20:18:56,416 [GossipStage:1] DEBUG Gossiper adding expire time for 
endpoint : /X.X.X.X (1485116334088)
2017-01-19 20:18:56,417 [GossipStage:1] INFO StorageService Removing tokens 
[100434964734820719895982857900842892337, 
114144647582686041354301802358217767299, 
13209060517964702932350041942412177, 
138409460913927199437556572481804704749] for /X.X.X.X
2017-01-19 20:18:56,418 [HintedHandoff:3] INFO HintedHandOffManager Deleting 
any stored hints for /X.X.X.X
2017-01-19 20:18:56,424 [GossipStage:1] DEBUG MessagingService Resetting 
version for /X.X.X.X
2017-01-19 20:18:56,424 [GossipStage:1] DEBUG Gossiper removing endpoint 
/X.X.X.X
2017-01-19 20:18:56,437 [GossipStage:1] DEBUG StorageService Ignoring state 
change for dead or unknown endpoint: /X.X.X.X
2017-01-19 20:19:02,022 [WRITE-/X.X.X.X] DEBUG OutboundTcpConnection attempting 
to connect to /X.X.X.X
2017-01-19 20:19:02,023 [HANDSHAKE-/X.X.X.X] INFO OutboundTcpConnection 
Handshaking version with /X.X.X.X
2017-01-19 20:19:02,023 [WRITE-/X.X.X.X] DEBUG MessagingService Setting version 
7 for /X.X.X.X
2017-01-19 20:19:08,096 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 2074454222 for /X.X.X.X
2017-01-19 20:19:54,407 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 4302985797 for /X.X.X.X
2017-01-19 20:19:57,405 [GossipTasks:1] DEBUG Gossiper 6 elapsed, /X.X.X.X 
gossip quarantine over
2017-01-19 20:19:57,455 [GossipStage:1] DEBUG ArrivalWindow Ignoring interval 
time of 3047826501 for /X.X.X.X
2017-01-19 20:19:57,455 [GossipStage:1] DEBUG StorageService Ignoring state 
change for dead or unknown endpoint: /X.X.X.X


Cassandra 2.1.16 Logs showing the decommissioned node :   (The logs in 2.1.16 
show the same as 2.1.1 upto "DEBUG Gossiper 6 elapsed, /X.X.X.X gossip 
quarantine over", and then is followed by "NODE is now DOWN"

017-01-19 19:52:23,687 [GossipStage:1] DEBUG StorageService.java:1883 - Node 
/X.X.X.X state left, tokens [-1112888759032625467, -228773855963737699, 
-311455042375
4381391, -4848625944949064281, -6920961603460018610, -8566729719076824066, 
1611098831406674636, 7278843689020594771, 7565410054791352413, 
9166885764, 8654747784805453046]
2017-01-19 19:52:23,688 [GossipStage:1] DEBUG Gossiper.java:1520 - adding 
expire time for endpoint : /X.X.X.X (1485114743567)
2017-01-19 19:52:23,688 [GossipStage:1] INFO StorageService.java:1965 - 
Removing tokens [-1112888759032625467, -228773855963737699, 
-3114550423754381391, -48486259449
49064281, -6920961

Re: Re : Decommissioned nodes show as DOWN in Cassandra versions 2.1.12 - 2.1.16

[kurt greaves]

we've seen this issue on a few clusters, including on 2.1.7 and 2.1.8.
pretty sure it is an issue in gossip that's known about. in later versions
it seems to be fixed.

On 24 Jan 2017 06:09, "sai krishnam raju potturi" 
wrote:

> In the Cassandra versions 2.1.11 - 2.1.16, after we decommission a node or
> datacenter, we observe the decommissioned nodes marked as DOWN in the
> cluster when you do a "nodetool describecluster". The nodes however do not
> show up in the "nodetool status" command.
> The decommissioned node also does not show up in the "system_peers" table
> on the nodes.
>
> The workaround we follow is rolling restart of the cluster, which removes
> the decommissioned nodes from the "UNREACHABLE STATE", and shows the actual
> state of the cluster. The workaround is tedious for huge clusters.
>
> We also verified the decommission process in CCM tool, and observed the
> same issue for clusters with versions from 2.1.12 to 2.1.16. The issue was
> not observed in versions prior to or later than the ones mentioned above.
>
>
> Has anybody in the community observed similar issue? We've also raised a
> JIRA issue regarding this.   https://issues.apache.org/jira
> /browse/CASSANDRA-13144
>
>
> Below are the observed logs from the versions without the bug, and with
> the bug.  The one's highlighted in yellow show the expected logs. The one's
> highlighted in red are the one's where the node is recognized as down, and
> shows as UNREACHABLE.
>
>
>
> Cassandra 2.1.1 Logs showing the decommissioned node :  (Without the bug)
>
> 2017-01-19 20:18:56,415 [GossipStage:1] DEBUG ArrivalWindow Ignoring
> interval time of 2049943233 <(204)%20994-3233> for /X.X.X.X
> 2017-01-19 20:18:56,416 [GossipStage:1] DEBUG StorageService Node
> /X.X.X.X state left, tokens [ 59353109817657926242901533144729725259,
> 60254520910109313597677907197875221475, 
> 75698727618038614819889933974570742305,
> 84508739091270910297310401957975430578]
> 2017-01-19 20:18:56,416 [GossipStage:1] DEBUG Gossiper adding expire time
> for endpoint : /X.X.X.X (1485116334088)
> 2017-01-19 20:18:56,417 [GossipStage:1] INFO StorageService Removing
> tokens [100434964734820719895982857900842892337,
> 114144647582686041354301802358217767299, 
> 13209060517964702932350041942412177,
> 138409460913927199437556572481804704749] for /X.X.X.X
> 2017-01-19 20:18:56,418 [HintedHandoff:3] INFO HintedHandOffManager
> Deleting any stored hints for /X.X.X.X
> 2017-01-19 20:18:56,424 [GossipStage:1] DEBUG MessagingService Resetting
> version for /X.X.X.X
> 2017-01-19 20:18:56,424 [GossipStage:1] DEBUG Gossiper removing endpoint
> /X.X.X.X
> 2017-01-19 20:18:56,437 [GossipStage:1] DEBUG StorageService Ignoring
> state change for dead or unknown endpoint: /X.X.X.X
> 2017-01-19 20:19:02,022 [WRITE-/X.X.X.X] DEBUG OutboundTcpConnection
> attempting to connect to /X.X.X.X
> 2017-01-19 20:19:02,023 [HANDSHAKE-/X.X.X.X] INFO OutboundTcpConnection
> Handshaking version with /X.X.X.X
> 2017-01-19 20:19:02,023 [WRITE-/X.X.X.X] DEBUG MessagingService Setting
> version 7 for /X.X.X.X
> 2017-01-19 20:19:08,096 [GossipStage:1] DEBUG ArrivalWindow Ignoring
> interval time of 2074454222 <(207)%20445-4222> for /X.X.X.X
> 2017-01-19 20:19:54,407 [GossipStage:1] DEBUG ArrivalWindow Ignoring
> interval time of 4302985797 <(430)%20298-5797> for /X.X.X.X
> 2017-01-19 20:19:57,405 [GossipTasks:1] DEBUG Gossiper 6 elapsed,
> /X.X.X.X gossip quarantine over
> 2017-01-19 20:19:57,455 [GossipStage:1] DEBUG ArrivalWindow Ignoring
> interval time of 3047826501 <(304)%20782-6501> for /X.X.X.X
> 2017-01-19 20:19:57,455 [GossipStage:1] DEBUG StorageService Ignoring
> state change for dead or unknown endpoint: /X.X.X.X
>
>
> Cassandra 2.1.16 Logs showing the decommissioned node :   (The logs in
> 2.1.16 show the same as 2.1.1 upto "DEBUG Gossiper 6 elapsed, /X.X.X.X
> gossip quarantine over", and then is followed by "NODE is now DOWN"
>
> 017-01-19 19:52:23,687 [GossipStage:1] DEBUG StorageService.java:1883 -
> Node /X.X.X.X state left, tokens [-1112888759032625467,
> -228773855963737699, -311455042375
> 4381391, -4848625944949064281, -6920961603460018610, -8566729719076824066,
> 1611098831406674636, 7278843689020594771, 7565410054791352413, 9166885764
> <(916)%20688-5764>, 8654747784805453046]
> 2017-01-19 19:52:23,688 [GossipStage:1] DEBUG Gossiper.java:1520 - adding
> expire time for endpoint : /X.X.X.X (1485114743567)
> 2017-01-19 19:52:23,688 [GossipStage:1] INFO StorageService.java:1965 -
> Removing tokens [-1112888759032625467, -228773855963737699,
> -3114550423754381391, -48486259449
> 49064281, -6920961603460018610, 5690722015779071557, 6202373691525063547,
> 7191120402564284381, 7278843689020594771, 7565410054791352413,
> 8524200089166885764, 865474778
> 4805453046 <(480)%20545-3046>] for /X.X.X.X
> 2017-01-19 19:52:23,689 [HintedHandoffManager:1] INFO
> HintedHandOffManager.java:230 - Deleting any stored hints for /X.X.X.X
> 2017-01-19 19:52:23,689 [GossipStage:1] DEBUG Messa

secondary index on static column

[Micha]

Hi,

I'm quite new to cassandra and allthough there is much info on the net,
sometimes I cannot find the solution to a problem.

In this case, I have a second index on a static column and I don't
understand the answer I get from my select.

A cut down version of the table is:

create table demo (id text, id2 bigint static, added timestamp, source
text static, dest text, primary key (id, added));

create index on demo (id2);

id and id2 match one to one.

I make one insert:
insert into demo (id, id2, added, source, dest) values ('id1', 22,
'2017-01-28', 'src1', 'dst1');


The "select from demo;" gives the expected answer of the one inserted row.

But "select from demo where id2=22" gives 70 rows as result (all the same).

Why? I have read
https://www.datastax.com/dev/blog/cassandra-native-secondary-index-deep-dive

but I don't get it...

thanks for answering,
 Michael

Re: [External] Re: Cassandra ad hoc search options

[siddharth verma]

Hi
We used lucene stratio plugin with C*3.0.3

Helped to solve a lot of some read patterns. Served well for prefix.
But created problems as repairs failed repeatedly.
We might have used it sub optimally, not sure.

Later, we had to do away with it, and tried to serve most of the read
patterns with materialised views. (currently C*3.0.9)

Currently, for adhoc querries, we use spark or full scan.

Regards,

On Fri, Jan 27, 2017 at 1:03 PM, Yu, John  wrote:

> Thanks a lot. Mind sharing a couple of points where you feel it’s better
> than the alternatives.
>
>
>
> Regards,
>
> John
>
>
>
> *From:* Jonathan Haddad [mailto:j...@jonhaddad.com]
> *Sent:* Thursday, January 26, 2017 2:33 PM
> *To:* user@cassandra.apache.org
> *Subject:* [External] Re: Cassandra ad hoc search options
>
>
>
> > With Cassandra, what are the options for ad hoc query/search similar to
> RDBMS?
>
>
>
> Your best options are Spark w/ the DataStax connector or Presto.
> Cassandra isn't built for ad-hoc queries so you need to use other tools to
> make it work.
>
>
>
> On Thu, Jan 26, 2017 at 2:22 PM Yu, John  wrote:
>
> Hi All,
>
>
>
> Hope I can get some help here. We’re using Cassandra for services, and
> recently we’re adding UI support.
>
> With Cassandra, what are the options for ad hoc query/search similar to
> RDBMS? We love the features of Cassandra but it seems it’s a known
> “weakness” that it doesn’t come with strong support of indexing and ad hoc
> queries. There’re some recent development with SASI as part of secondary
> index. However I heard from a video where it says it shall not be
> extensively used.
>
>
>
> Has anyone have much experience with SASI? How does it compare to Lucene
> plugin?
>
> What is the direction of Apache Cassandra in the search area?
>
>
>
> We’re also looking into Solr or ElasticSearch integration, but it seems it
> might take more efforts, and possibly involve data duplication.
>
> For Solr, we don’t have DSE.
>
> Sorry if this has been asked before, but I haven’t seen a more complete
> answer.
>
>
>
> Thanks!
>
> John
> --
>
> NOTICE OF CONFIDENTIALITY:
> This message may contain information that is considered confidential and
> which may be prohibited from disclosure under applicable law or by
> contractual agreement. The information is intended solely for the use of
> the individual or entity named above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of the information contained in or attached to this
> message is strictly prohibited. If you have received this email
> transmission in error, please notify the sender by replying to this email
> and then delete it from your system.
>
>


-- 
Siddharth Verma
(Visit https://github.com/siddv29/cfs for a high speed cassandra full table
scan)