Decommissioned Node UNREACHABLE in describecluster but LEFT in gossipinfo

[Sergio]

Hi guys,

Cassandra 3.11.4

nodetool gossipinfo
/10.1.20.49
  generation:1571694191
  heartbeat:279800
  STATUS:279798:LEFT,-1013739435631815991,1572225050446
  LOAD:279791:3.4105213781E11
  SCHEMA:12:5cad59d2-c3d0-3a12-ad10-7578d225b082
  DC:8:live
  RACK:10:us-east-1a
  RELEASE_VERSION:4:3.11.4
  INTERNAL_IP:6:10.1.20.49
  RPC_ADDRESS:3:10.1.20.49
  NET_VERSION:1:11
  HOST_ID:2:be5a0193-56e7-4d42-8cc8-5d2141ab4872
  RPC_READY:29:true
  TOKENS:15:

The node is not shown in nodetool status

and it is displayed as UNREACHABLE in nodetool describecluster

I found this old conversation
https://grokbase.com/t/cassandra/user/162gwp6pz6/decommissioned-nodes-shows-up-in-nodetool-describecluster-as-unreachable-in-2-1-12-version

Is there something that I should do to fix this?

Best,

Sergio

Re: Cassandra Rack - Datacenter Load Balancing relations

[Sergio]

Thanks Reid!

I agree with all the things that you said!

Best,
Sergio

Il giorno gio 24 ott 2019 alle ore 09:25 Reid Pinchback <
rpinchb...@tripadvisor.com> ha scritto:

> Two different AWS AZs are in two different physical locations.  Typically
> different cities.  Which means that you’re trying to manage the risk of an
> AZ going dark, so you use more than one AZ just in case.  The downside is
> that you will have some degree of network performance difference between
> AZs because of whatever WAN pipe AWS owns/leased to connect between them.
>
>
>
> Having a DC in one AZ is easy to reason about.  The AZ is there, or it is
> not.  If you have two DCs in your cluster, and you lose an AZ, it means you
> still have a functioning cluster with one DC and you still have quorum.
> Yay, even in an outage, you know you can still do business.  You would only
> have to route any traffic normally sent to the other DC to the remaining
> one, so as long as there is resource headroom planning in how you provision
> your hardware, you’re in a safe state.
>
>
>
> If you start splitting a DC across AZs without using racks to organize
> nodes on a per-AZ basis, off the top of my head I don’t know how you reason
> about your risks for losing quorum without pausing to really think through
> vnodes and token distribution and whatnot.  I’m not a fan of topologies I
> can’t reason about when paged at 3 in the morning and I’m half asleep.  I
> prefer simple until the workload motivates complex.
>
>
>
> R
>
>
>
>
>
> *From: *Sergio 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Thursday, October 24, 2019 at 12:06 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Cassandra Rack - Datacenter Load Balancing relations
>
>
>
> *Message from External Sender*
>
> Thanks Reid and Jon!
>
>
>
> Yes I will stick with one rack per DC for sure and I will look at the
> Vnodes problem later on.
>
>
>
>
>
> What's the difference in terms of reliability between
>
> A) spreading 2 Datacenters across 3 AZ
>
> B) having 2 Datacenters in 2 separate AZ
>
> ?
>
>
>
>
>
> Best,
>
>
>
> Sergio
>
>
>
> On Thu, Oct 24, 2019, 7:36 AM Reid Pinchback 
> wrote:
>
> Hey Sergio,
>
>
>
> Forgive but I’m at work and had to skim the info quickly.
>
>
>
> When in doubt, simplify.  So 1 rack per DC.  Distributed systems get
> rapidly harder to reason about the more complicated you make them.  There’s
> more than enough to learn about C* without jumping into the complexity too
> soon.
>
>
>
> To deal with the unbalancing issue, pay attention to Jon Haddad’s advice
> on vnode count and how to fairly distribute tokens with a small vnode
> count.  I’d rather point you to his information, as I haven’t dug into
> vnode counts and token distribution in detail; he’s got a lot more time in
> C* than I do.  I come at this more as a traditional RDBMS and Java guy who
> has slowly gotten up to speed on C* over the last few years, and dealt with
> DynamoDB a lot so have lived with a lot of similarity in data modelling
> concerns.  Detailed internals I only know in cases where I had reason to
> dig into C* source.
>
>
>
> There are so many knobs to turn in C* that it can be very easy to
> overthink things.  Simplify where you can.  Remove GC pressure wherever you
> can.  Negotiate with your consumers to have data models that make sense for
> C*.  If you have those three criteria foremost in mind, you’ll likely be
> fine for quite some time.  And in the times where something isn’t going
> well, simpler is easier to investigate.
>
>
> R
>
>
>
> *From: *Sergio 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Wednesday, October 23, 2019 at 3:34 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Cassandra Rack - Datacenter Load Balancing relations
>
>
>
> *Message from External Sender*
>
> Hi Reid,
>
> Thank you very much for clearing these concepts for me.
> https://community.datastax.com/comments/1133/view.html
> 
> I posted this question on the datastax forum regarding our cluster that it
> is unbalanced and the reply was related that the *number of racks should
> be a multiplier of the replication factor *in order to be balanced or 1.
> I thought then if I have 3 availability zones I should have 3 racks for
> each datacenter and not 2 (us-east-1b, us-east-1a) as I have right now or
> in the easiest way, I should have a rack for each datacenter.
>
>
> 1.  Datacenter: live
> 
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   OwnsHost ID
> Rack
> UN  10.1.20.49   289.75 GiB  256  ?
> be5a0193-56e7-4d42-8cc8-5d2141ab4872  us-east-1a
> UN  10.1.30.112  103.03 GiB  256  ?
> 

Re: Repair Issues

[Ben Mills]

Thanks Jon!

This is very helpful - allow me to follow-up and ask a question.

(1) Yes, incremental repairs will never be used (unless it becomes viable
in Cassandra 4.x someday).
(2) I hear you on the JVM - will look into that.
(3) Been looking at Cassandra version 3.11.x though was unaware that 3.7 is
considered non-viable for production use.

For (4) - Question/Request:

Note that with:

-XX:MaxRAMFraction=2

the actual amount of memory allocated for heap space is effectively 2Gi
(i.e. half of the 4Gi allocated on the machine type). We can definitely
increase memory (for heap and nonheap), though can you expand a bit on your
heap comment to help my understanding (as this is such a small cluster with
such a small amount of data at rest)?

Thanks again.

On Thu, Oct 24, 2019 at 5:11 PM Jon Haddad  wrote:

> There's some major warning signs for me with your environment.  4GB heap
> is too low, and Cassandra 3.7 isn't something I would put into production.
>
> Your surface area for problems is massive right now.  Things I'd do:
>
> 1. Never use incremental repair.  Seems like you've already stopped doing
> them, but it's worth mentioning.
> 2. Upgrade to the latest JVM, that version's way out of date.
> 3. Upgrade to Cassandra 3.11.latest (we're voting on 3.11.5 right now).
> 4. Increase memory to 8GB minimum, preferably 12.
>
> I usually don't like making a bunch of changes without knowing the root
> cause of a problem, but in your case there's so many potential problems I
> don't think it's worth digging into, especially since the problem might be
> one of the 500 or so bugs that were fixed since this release.
>
> Once you've done those things it'll be easier to narrow down the problem.
>
> Jon
>
>
> On Thu, Oct 24, 2019 at 4:59 PM Ben Mills  wrote:
>
>> Hi Sergio,
>>
>> No, not at this time.
>>
>> It was in use with this cluster previously, and while there were no
>> reaper-specific issues, it was removed to help simplify investigation of
>> the underlying repair issues I've described.
>>
>> Thanks.
>>
>> On Thu, Oct 24, 2019 at 4:21 PM Sergio  wrote:
>>
>>> Are you using Cassandra reaper?
>>>
>>> On Thu, Oct 24, 2019, 12:31 PM Ben Mills  wrote:
>>>
 Greetings,

 Inherited a small Cassandra cluster with some repair issues and need
 some advice on recommended next steps. Apologies in advance for a long
 email.

 Issue:

 Intermittent repair failures on two non-system keyspaces.

 - platform_users
 - platform_management

 Repair Type:

 Full, parallel repairs are run on each of the three nodes every five
 days.

 Repair command output for a typical failure:

 [2019-10-18 00:22:09,109] Starting repair command #46, repairing
 keyspace platform_users with repair options (parallelism: parallel, primary
 range: false, incremental: false, job threads: 1, ColumnFamilies: [],
 dataCenters: [], hosts: [], # of ranges: 12)
 [2019-10-18 00:22:09,242] Repair session
 5282be70-f13d-11e9-9b4e-7f6db768ba9a for range
 [(-1890954128429545684,2847510199483651721],
 (8249813014782655320,-8746483007209345011],
 (4299912178579297893,6811748355903297393],
 (-8746483007209345011,-8628999431140554276],
 (-5865769407232506956,-4746990901966533744],
 (-4470950459111056725,-1890954128429545684],
 (4001531392883953257,4299912178579297893],
 (6811748355903297393,6878104809564599690],
 (6878104809564599690,8249813014782655320],
 (-4746990901966533744,-4470950459111056725],
 (-8628999431140554276,-5865769407232506956],
 (2847510199483651721,4001531392883953257]] failed with error [repair
 #5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2,
 [(-1890954128429545684,2847510199483651721],
 (8249813014782655320,-8746483007209345011],
 (4299912178579297893,6811748355903297393],
 (-8746483007209345011,-8628999431140554276],
 (-5865769407232506956,-4746990901966533744],
 (-4470950459111056725,-1890954128429545684],
 (4001531392883953257,4299912178579297893],
 (6811748355903297393,6878104809564599690],
 (6878104809564599690,8249813014782655320],
 (-4746990901966533744,-4470950459111056725],
 (-8628999431140554276,-5865769407232506956],
 (2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x
 (progress: 26%)
 [2019-10-18 00:22:09,246] Some repair failed
 [2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds

 Additional Notes:

 Repairs encounter above failures more often than not. Sometimes on one
 node only, though occasionally on two. Sometimes just one of the two
 keyspaces, sometimes both. Apparently the previous repair schedule for
 this cluster included incremental repairs (script alternated between
 incremental and full repairs). After reading this TLP article:


 

Re: Repair Issues

[Ben Mills]

Hi Reid,

Many thanks - I have seen that article though will definitely give it
another read.

Note that nodetool scrub has been tried (no effect) and sstablescrub cannot
currently be run with the Cassandra image in use (though certainly a new
image that allows the server to be stopped but keeps the operating
environment available to use this utility can be built - just haven't done
so yet). Note also that none of the logs are indicating that a corrupt
data file (or files) is in play here. Noting that because the article
includes a solution where a specific data file is manually deleted and then
repairs are run to restore the file from a different node in the cluster.
Also, the way persistent volumes are mounted onto [Kubernetes] nodes
prevents this solution (manual deletion of an offending data file) from
being viable because the PV mount on the node's filesystem is detached when
the pods are down. This is a subtlety of running Cassandra in Kubernetes.

On Thu, Oct 24, 2019 at 4:24 PM Reid Pinchback 
wrote:

> Ben, you may find this helpful:
>
>
>
> https://blog.pythian.com/so-you-have-a-broken-cassandra-sstable-file/
>
>
>
>
>
> *From: *Ben Mills 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Thursday, October 24, 2019 at 3:31 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Repair Issues
>
>
>
> *Message from External Sender*
>
> Greetings,
>
> Inherited a small Cassandra cluster with some repair issues and need some
> advice on recommended next steps. Apologies in advance for a long email.
>
> Issue:
>
> Intermittent repair failures on two non-system keyspaces.
>
> - platform_users
> - platform_management
>
> Repair Type:
>
> Full, parallel repairs are run on each of the three nodes every five days.
>
> Repair command output for a typical failure:
>
> [2019-10-18 00:22:09,109] Starting repair command #46, repairing keyspace
> platform_users with repair options (parallelism: parallel, primary range:
> false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
> [], hosts: [], # of ranges: 12)
> [2019-10-18 00:22:09,242] Repair session
> 5282be70-f13d-11e9-9b4e-7f6db768ba9a for range
> [(-1890954128429545684,2847510199483651721],
> (8249813014782655320,-8746483007209345011],
> (4299912178579297893,6811748355903297393],
> (-8746483007209345011,-8628999431140554276],
> (-5865769407232506956,-4746990901966533744],
> (-4470950459111056725,-1890954128429545684],
> (4001531392883953257,4299912178579297893],
> (6811748355903297393,6878104809564599690],
> (6878104809564599690,8249813014782655320],
> (-4746990901966533744,-4470950459111056725],
> (-8628999431140554276,-5865769407232506956],
> (2847510199483651721,4001531392883953257]] failed with error [repair
> #5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2,
> [(-1890954128429545684,2847510199483651721],
> (8249813014782655320,-8746483007209345011],
> (4299912178579297893,6811748355903297393],
> (-8746483007209345011,-8628999431140554276],
> (-5865769407232506956,-4746990901966533744],
> (-4470950459111056725,-1890954128429545684],
> (4001531392883953257,4299912178579297893],
> (6811748355903297393,6878104809564599690],
> (6878104809564599690,8249813014782655320],
> (-4746990901966533744,-4470950459111056725],
> (-8628999431140554276,-5865769407232506956],
> (2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x
> (progress: 26%)
> [2019-10-18 00:22:09,246] Some repair failed
> [2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds
>
> Additional Notes:
>
> Repairs encounter above failures more often than not. Sometimes on one
> node only, though occasionally on two. Sometimes just one of the two
> keyspaces, sometimes both. Apparently the previous repair schedule for
> this cluster included incremental repairs (script alternated between
> incremental and full repairs). After reading this TLP article:
>
>
> https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
> 
>
> the repair script was replaced with cassandra-reaper (v1.4.0), which was
> run with its default configs. Reaper was fine but only obscured the ongoing
> issues (it did not resolve them) and complicated the debugging process and
> so was then removed. The current repair schedule is as described above
> under Repair Type.
>
> Attempts at Resolution:
>
> (1) nodetool scrub was attempted on the offending keyspaces/tables to no
> effect.
>
> (2) sstablescrub has not been attempted due to the current design of the
> Docker image that runs Cassandra in each Kubernetes pod - i.e. there is no
> way to stop the server to run this utility without killing the only pid
> running in the 

Re: Repair Issues

[Jon Haddad]

There's some major warning signs for me with your environment.  4GB heap is
too low, and Cassandra 3.7 isn't something I would put into production.

Your surface area for problems is massive right now.  Things I'd do:

1. Never use incremental repair.  Seems like you've already stopped doing
them, but it's worth mentioning.
2. Upgrade to the latest JVM, that version's way out of date.
3. Upgrade to Cassandra 3.11.latest (we're voting on 3.11.5 right now).
4. Increase memory to 8GB minimum, preferably 12.

I usually don't like making a bunch of changes without knowing the root
cause of a problem, but in your case there's so many potential problems I
don't think it's worth digging into, especially since the problem might be
one of the 500 or so bugs that were fixed since this release.

Once you've done those things it'll be easier to narrow down the problem.

Jon


On Thu, Oct 24, 2019 at 4:59 PM Ben Mills  wrote:

> Hi Sergio,
>
> No, not at this time.
>
> It was in use with this cluster previously, and while there were no
> reaper-specific issues, it was removed to help simplify investigation of
> the underlying repair issues I've described.
>
> Thanks.
>
> On Thu, Oct 24, 2019 at 4:21 PM Sergio  wrote:
>
>> Are you using Cassandra reaper?
>>
>> On Thu, Oct 24, 2019, 12:31 PM Ben Mills  wrote:
>>
>>> Greetings,
>>>
>>> Inherited a small Cassandra cluster with some repair issues and need
>>> some advice on recommended next steps. Apologies in advance for a long
>>> email.
>>>
>>> Issue:
>>>
>>> Intermittent repair failures on two non-system keyspaces.
>>>
>>> - platform_users
>>> - platform_management
>>>
>>> Repair Type:
>>>
>>> Full, parallel repairs are run on each of the three nodes every five
>>> days.
>>>
>>> Repair command output for a typical failure:
>>>
>>> [2019-10-18 00:22:09,109] Starting repair command #46, repairing
>>> keyspace platform_users with repair options (parallelism: parallel, primary
>>> range: false, incremental: false, job threads: 1, ColumnFamilies: [],
>>> dataCenters: [], hosts: [], # of ranges: 12)
>>> [2019-10-18 00:22:09,242] Repair session
>>> 5282be70-f13d-11e9-9b4e-7f6db768ba9a for range
>>> [(-1890954128429545684,2847510199483651721],
>>> (8249813014782655320,-8746483007209345011],
>>> (4299912178579297893,6811748355903297393],
>>> (-8746483007209345011,-8628999431140554276],
>>> (-5865769407232506956,-4746990901966533744],
>>> (-4470950459111056725,-1890954128429545684],
>>> (4001531392883953257,4299912178579297893],
>>> (6811748355903297393,6878104809564599690],
>>> (6878104809564599690,8249813014782655320],
>>> (-4746990901966533744,-4470950459111056725],
>>> (-8628999431140554276,-5865769407232506956],
>>> (2847510199483651721,4001531392883953257]] failed with error [repair
>>> #5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2,
>>> [(-1890954128429545684,2847510199483651721],
>>> (8249813014782655320,-8746483007209345011],
>>> (4299912178579297893,6811748355903297393],
>>> (-8746483007209345011,-8628999431140554276],
>>> (-5865769407232506956,-4746990901966533744],
>>> (-4470950459111056725,-1890954128429545684],
>>> (4001531392883953257,4299912178579297893],
>>> (6811748355903297393,6878104809564599690],
>>> (6878104809564599690,8249813014782655320],
>>> (-4746990901966533744,-4470950459111056725],
>>> (-8628999431140554276,-5865769407232506956],
>>> (2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x
>>> (progress: 26%)
>>> [2019-10-18 00:22:09,246] Some repair failed
>>> [2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds
>>>
>>> Additional Notes:
>>>
>>> Repairs encounter above failures more often than not. Sometimes on one
>>> node only, though occasionally on two. Sometimes just one of the two
>>> keyspaces, sometimes both. Apparently the previous repair schedule for
>>> this cluster included incremental repairs (script alternated between
>>> incremental and full repairs). After reading this TLP article:
>>>
>>>
>>> https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
>>>
>>> the repair script was replaced with cassandra-reaper (v1.4.0), which was
>>> run with its default configs. Reaper was fine but only obscured the ongoing
>>> issues (it did not resolve them) and complicated the debugging process and
>>> so was then removed. The current repair schedule is as described above
>>> under Repair Type.
>>>
>>> Attempts at Resolution:
>>>
>>> (1) nodetool scrub was attempted on the offending keyspaces/tables to no
>>> effect.
>>>
>>> (2) sstablescrub has not been attempted due to the current design of the
>>> Docker image that runs Cassandra in each Kubernetes pod - i.e. there is no
>>> way to stop the server to run this utility without killing the only pid
>>> running in the container.
>>>
>>> Related Error:
>>>
>>> Not sure if this is related, though sometimes, when either:
>>>
>>> (a) Running nodetool snapshot, or
>>> (b) Rolling a pod that runs a Cassandra node, which 

Re: Repair Issues

[Ben Mills]

Hi Sergio,

No, not at this time.

It was in use with this cluster previously, and while there were no
reaper-specific issues, it was removed to help simplify investigation of
the underlying repair issues I've described.

Thanks.

On Thu, Oct 24, 2019 at 4:21 PM Sergio  wrote:

> Are you using Cassandra reaper?
>
> On Thu, Oct 24, 2019, 12:31 PM Ben Mills  wrote:
>
>> Greetings,
>>
>> Inherited a small Cassandra cluster with some repair issues and need some
>> advice on recommended next steps. Apologies in advance for a long email.
>>
>> Issue:
>>
>> Intermittent repair failures on two non-system keyspaces.
>>
>> - platform_users
>> - platform_management
>>
>> Repair Type:
>>
>> Full, parallel repairs are run on each of the three nodes every five days.
>>
>> Repair command output for a typical failure:
>>
>> [2019-10-18 00:22:09,109] Starting repair command #46, repairing keyspace
>> platform_users with repair options (parallelism: parallel, primary range:
>> false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
>> [], hosts: [], # of ranges: 12)
>> [2019-10-18 00:22:09,242] Repair session
>> 5282be70-f13d-11e9-9b4e-7f6db768ba9a for range
>> [(-1890954128429545684,2847510199483651721],
>> (8249813014782655320,-8746483007209345011],
>> (4299912178579297893,6811748355903297393],
>> (-8746483007209345011,-8628999431140554276],
>> (-5865769407232506956,-4746990901966533744],
>> (-4470950459111056725,-1890954128429545684],
>> (4001531392883953257,4299912178579297893],
>> (6811748355903297393,6878104809564599690],
>> (6878104809564599690,8249813014782655320],
>> (-4746990901966533744,-4470950459111056725],
>> (-8628999431140554276,-5865769407232506956],
>> (2847510199483651721,4001531392883953257]] failed with error [repair
>> #5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2,
>> [(-1890954128429545684,2847510199483651721],
>> (8249813014782655320,-8746483007209345011],
>> (4299912178579297893,6811748355903297393],
>> (-8746483007209345011,-8628999431140554276],
>> (-5865769407232506956,-4746990901966533744],
>> (-4470950459111056725,-1890954128429545684],
>> (4001531392883953257,4299912178579297893],
>> (6811748355903297393,6878104809564599690],
>> (6878104809564599690,8249813014782655320],
>> (-4746990901966533744,-4470950459111056725],
>> (-8628999431140554276,-5865769407232506956],
>> (2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x
>> (progress: 26%)
>> [2019-10-18 00:22:09,246] Some repair failed
>> [2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds
>>
>> Additional Notes:
>>
>> Repairs encounter above failures more often than not. Sometimes on one
>> node only, though occasionally on two. Sometimes just one of the two
>> keyspaces, sometimes both. Apparently the previous repair schedule for
>> this cluster included incremental repairs (script alternated between
>> incremental and full repairs). After reading this TLP article:
>>
>>
>> https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
>>
>> the repair script was replaced with cassandra-reaper (v1.4.0), which was
>> run with its default configs. Reaper was fine but only obscured the ongoing
>> issues (it did not resolve them) and complicated the debugging process and
>> so was then removed. The current repair schedule is as described above
>> under Repair Type.
>>
>> Attempts at Resolution:
>>
>> (1) nodetool scrub was attempted on the offending keyspaces/tables to no
>> effect.
>>
>> (2) sstablescrub has not been attempted due to the current design of the
>> Docker image that runs Cassandra in each Kubernetes pod - i.e. there is no
>> way to stop the server to run this utility without killing the only pid
>> running in the container.
>>
>> Related Error:
>>
>> Not sure if this is related, though sometimes, when either:
>>
>> (a) Running nodetool snapshot, or
>> (b) Rolling a pod that runs a Cassandra node, which calls nodetool drain
>> prior shutdown,
>>
>> the following error is thrown:
>>
>> -- StackTrace --
>> java.lang.RuntimeException: Last written key
>> DecoratedKey(10df3ba1-6eb2-4c8e-bddd-c0c7af586bda,
>> 10df3ba16eb24c8ebdddc0c7af586bda) >= current key
>> DecoratedKey(----,
>> 17343121887f480c9ba87c0e32206b74) writing into
>> /cassandra_data/data/platform_management/device_by_tenant_v2-e91529202ccf11e7ab96d5693708c583/.device_by_tenant_tags_idx/mb-45-big-Data.db
>> at
>> org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:114)
>> at
>> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
>> at
>> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
>> at
>> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:441)
>> at
>> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:477)

Re: Repair Issues

[Reid Pinchback]

Ben, you may find this helpful:

https://blog.pythian.com/so-you-have-a-broken-cassandra-sstable-file/


From: Ben Mills 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, October 24, 2019 at 3:31 PM
To: "user@cassandra.apache.org" 
Subject: Repair Issues

Message from External Sender
Greetings,

Inherited a small Cassandra cluster with some repair issues and need some 
advice on recommended next steps. Apologies in advance for a long email.

Issue:

Intermittent repair failures on two non-system keyspaces.

- platform_users
- platform_management

Repair Type:

Full, parallel repairs are run on each of the three nodes every five days.

Repair command output for a typical failure:

[2019-10-18 00:22:09,109] Starting repair command #46, repairing keyspace 
platform_users with repair options (parallelism: parallel, primary range: 
false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters: [], 
hosts: [], # of ranges: 12)
[2019-10-18 00:22:09,242] Repair session 5282be70-f13d-11e9-9b4e-7f6db768ba9a 
for range [(-1890954128429545684,2847510199483651721], 
(8249813014782655320,-8746483007209345011], 
(4299912178579297893,6811748355903297393], 
(-8746483007209345011,-8628999431140554276], 
(-5865769407232506956,-4746990901966533744], 
(-4470950459111056725,-1890954128429545684], 
(4001531392883953257,4299912178579297893], 
(6811748355903297393,6878104809564599690], 
(6878104809564599690,8249813014782655320], 
(-4746990901966533744,-4470950459111056725], 
(-8628999431140554276,-5865769407232506956], 
(2847510199483651721,4001531392883953257]] failed with error [repair 
#5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2, 
[(-1890954128429545684,2847510199483651721], 
(8249813014782655320,-8746483007209345011], 
(4299912178579297893,6811748355903297393], 
(-8746483007209345011,-8628999431140554276], 
(-5865769407232506956,-4746990901966533744], 
(-4470950459111056725,-1890954128429545684], 
(4001531392883953257,4299912178579297893], 
(6811748355903297393,6878104809564599690], 
(6878104809564599690,8249813014782655320], 
(-4746990901966533744,-4470950459111056725], 
(-8628999431140554276,-5865769407232506956], 
(2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x 
(progress: 26%)
[2019-10-18 00:22:09,246] Some repair failed
[2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds

Additional Notes:

Repairs encounter above failures more often than not. Sometimes on one node 
only, though occasionally on two. Sometimes just one of the two keyspaces, 
sometimes both. Apparently the previous repair schedule for this cluster 
included incremental repairs (script alternated between incremental and full 
repairs). After reading this TLP article:

https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html

the repair script was replaced with cassandra-reaper (v1.4.0), which was run 
with its default configs. Reaper was fine but only obscured the ongoing issues 
(it did not resolve them) and complicated the debugging process and so was then 
removed. The current repair schedule is as described above under Repair Type.

Attempts at Resolution:

(1) nodetool scrub was attempted on the offending keyspaces/tables to no effect.

(2) sstablescrub has not been attempted due to the current design of the Docker 
image that runs Cassandra in each Kubernetes pod - i.e. there is no way to stop 
the server to run this utility without killing the only pid running in the 
container.

Related Error:

Not sure if this is related, though sometimes, when either:

(a) Running nodetool snapshot, or
(b) Rolling a pod that runs a Cassandra node, which calls nodetool drain prior 
shutdown,

the following error is thrown:

-- StackTrace --
java.lang.RuntimeException: Last written key 
DecoratedKey(10df3ba1-6eb2-4c8e-bddd-c0c7af586bda, 
10df3ba16eb24c8ebdddc0c7af586bda) >= current key 
DecoratedKey(----, 
17343121887f480c9ba87c0e32206b74) writing into 
/cassandra_data/data/platform_management/device_by_tenant_v2-e91529202ccf11e7ab96d5693708c583/.device_by_tenant_tags_idx/mb-45-big-Data.db
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:114)
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
at 
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
at 
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:441)
at 
org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:477)
  

Re: Repair Issues

[Sergio]

Are you using Cassandra reaper?

On Thu, Oct 24, 2019, 12:31 PM Ben Mills  wrote:

> Greetings,
>
> Inherited a small Cassandra cluster with some repair issues and need some
> advice on recommended next steps. Apologies in advance for a long email.
>
> Issue:
>
> Intermittent repair failures on two non-system keyspaces.
>
> - platform_users
> - platform_management
>
> Repair Type:
>
> Full, parallel repairs are run on each of the three nodes every five days.
>
> Repair command output for a typical failure:
>
> [2019-10-18 00:22:09,109] Starting repair command #46, repairing keyspace
> platform_users with repair options (parallelism: parallel, primary range:
> false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
> [], hosts: [], # of ranges: 12)
> [2019-10-18 00:22:09,242] Repair session
> 5282be70-f13d-11e9-9b4e-7f6db768ba9a for range
> [(-1890954128429545684,2847510199483651721],
> (8249813014782655320,-8746483007209345011],
> (4299912178579297893,6811748355903297393],
> (-8746483007209345011,-8628999431140554276],
> (-5865769407232506956,-4746990901966533744],
> (-4470950459111056725,-1890954128429545684],
> (4001531392883953257,4299912178579297893],
> (6811748355903297393,6878104809564599690],
> (6878104809564599690,8249813014782655320],
> (-4746990901966533744,-4470950459111056725],
> (-8628999431140554276,-5865769407232506956],
> (2847510199483651721,4001531392883953257]] failed with error [repair
> #5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2,
> [(-1890954128429545684,2847510199483651721],
> (8249813014782655320,-8746483007209345011],
> (4299912178579297893,6811748355903297393],
> (-8746483007209345011,-8628999431140554276],
> (-5865769407232506956,-4746990901966533744],
> (-4470950459111056725,-1890954128429545684],
> (4001531392883953257,4299912178579297893],
> (6811748355903297393,6878104809564599690],
> (6878104809564599690,8249813014782655320],
> (-4746990901966533744,-4470950459111056725],
> (-8628999431140554276,-5865769407232506956],
> (2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x
> (progress: 26%)
> [2019-10-18 00:22:09,246] Some repair failed
> [2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds
>
> Additional Notes:
>
> Repairs encounter above failures more often than not. Sometimes on one
> node only, though occasionally on two. Sometimes just one of the two
> keyspaces, sometimes both. Apparently the previous repair schedule for
> this cluster included incremental repairs (script alternated between
> incremental and full repairs). After reading this TLP article:
>
>
> https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html
>
> the repair script was replaced with cassandra-reaper (v1.4.0), which was
> run with its default configs. Reaper was fine but only obscured the ongoing
> issues (it did not resolve them) and complicated the debugging process and
> so was then removed. The current repair schedule is as described above
> under Repair Type.
>
> Attempts at Resolution:
>
> (1) nodetool scrub was attempted on the offending keyspaces/tables to no
> effect.
>
> (2) sstablescrub has not been attempted due to the current design of the
> Docker image that runs Cassandra in each Kubernetes pod - i.e. there is no
> way to stop the server to run this utility without killing the only pid
> running in the container.
>
> Related Error:
>
> Not sure if this is related, though sometimes, when either:
>
> (a) Running nodetool snapshot, or
> (b) Rolling a pod that runs a Cassandra node, which calls nodetool drain
> prior shutdown,
>
> the following error is thrown:
>
> -- StackTrace --
> java.lang.RuntimeException: Last written key
> DecoratedKey(10df3ba1-6eb2-4c8e-bddd-c0c7af586bda,
> 10df3ba16eb24c8ebdddc0c7af586bda) >= current key
> DecoratedKey(----,
> 17343121887f480c9ba87c0e32206b74) writing into
> /cassandra_data/data/platform_management/device_by_tenant_v2-e91529202ccf11e7ab96d5693708c583/.device_by_tenant_tags_idx/mb-45-big-Data.db
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:114)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:441)
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:477)
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:363)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 

Repair Issues

[Ben Mills]

Greetings,

Inherited a small Cassandra cluster with some repair issues and need some
advice on recommended next steps. Apologies in advance for a long email.

Issue:

Intermittent repair failures on two non-system keyspaces.

- platform_users
- platform_management

Repair Type:

Full, parallel repairs are run on each of the three nodes every five days.

Repair command output for a typical failure:

[2019-10-18 00:22:09,109] Starting repair command #46, repairing keyspace
platform_users with repair options (parallelism: parallel, primary range:
false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
[], hosts: [], # of ranges: 12)
[2019-10-18 00:22:09,242] Repair session
5282be70-f13d-11e9-9b4e-7f6db768ba9a for range
[(-1890954128429545684,2847510199483651721],
(8249813014782655320,-8746483007209345011],
(4299912178579297893,6811748355903297393],
(-8746483007209345011,-8628999431140554276],
(-5865769407232506956,-4746990901966533744],
(-4470950459111056725,-1890954128429545684],
(4001531392883953257,4299912178579297893],
(6811748355903297393,6878104809564599690],
(6878104809564599690,8249813014782655320],
(-4746990901966533744,-4470950459111056725],
(-8628999431140554276,-5865769407232506956],
(2847510199483651721,4001531392883953257]] failed with error [repair
#5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2,
[(-1890954128429545684,2847510199483651721],
(8249813014782655320,-8746483007209345011],
(4299912178579297893,6811748355903297393],
(-8746483007209345011,-8628999431140554276],
(-5865769407232506956,-4746990901966533744],
(-4470950459111056725,-1890954128429545684],
(4001531392883953257,4299912178579297893],
(6811748355903297393,6878104809564599690],
(6878104809564599690,8249813014782655320],
(-4746990901966533744,-4470950459111056725],
(-8628999431140554276,-5865769407232506956],
(2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x
(progress: 26%)
[2019-10-18 00:22:09,246] Some repair failed
[2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds

Additional Notes:

Repairs encounter above failures more often than not. Sometimes on one node
only, though occasionally on two. Sometimes just one of the two keyspaces,
sometimes both. Apparently the previous repair schedule for this cluster
included incremental repairs (script alternated between incremental and
full repairs). After reading this TLP article:

https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html

the repair script was replaced with cassandra-reaper (v1.4.0), which was
run with its default configs. Reaper was fine but only obscured the ongoing
issues (it did not resolve them) and complicated the debugging process and
so was then removed. The current repair schedule is as described above
under Repair Type.

Attempts at Resolution:

(1) nodetool scrub was attempted on the offending keyspaces/tables to no
effect.

(2) sstablescrub has not been attempted due to the current design of the
Docker image that runs Cassandra in each Kubernetes pod - i.e. there is no
way to stop the server to run this utility without killing the only pid
running in the container.

Related Error:

Not sure if this is related, though sometimes, when either:

(a) Running nodetool snapshot, or
(b) Rolling a pod that runs a Cassandra node, which calls nodetool drain
prior shutdown,

the following error is thrown:

-- StackTrace --
java.lang.RuntimeException: Last written key
DecoratedKey(10df3ba1-6eb2-4c8e-bddd-c0c7af586bda,
10df3ba16eb24c8ebdddc0c7af586bda) >= current key
DecoratedKey(----,
17343121887f480c9ba87c0e32206b74) writing into
/cassandra_data/data/platform_management/device_by_tenant_v2-e91529202ccf11e7ab96d5693708c583/.device_by_tenant_tags_idx/mb-45-big-Data.db
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:114)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:441)
at
org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:477)
at
org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:363)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Here are some details on the environment and configs in the event that
something is relevant.

Environment: Kubernetes
Environment Config: Stateful set of 3 replicas
Storage: Persistent Volumes
Storage Class: SSD
Node OS: Container-Optimized OS
Container OS: Ubuntu 

Re: [RELEASE] Apache Cassandra 4.0-alpha1 released

[Abdul Patel]

Have anyone used this yet?
Any intial thoughts?
When will be full and final ready fornprod version coming in?

On Sunday, September 8, 2019, Michael Shuler  wrote:

> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 4.0-alpha1.
>
> Apache Cassandra is a fully distributed database. It is the right choice
> when you need scalability and high availability without compromising
> performance.
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions for 4.0-alpha1:
>
>
> http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha1/
> apache-cassandra-4.0-alpha1-bin.tar.gz
>
> http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha1/
> apache-cassandra-4.0-alpha1-src.tar.gz
>
> Debian and Redhat configurations
>
>  sources.list:
>  deb http://www.apache.org/dist/cassandra/debian 40x main
>
>  yum config:
>  baseurl=https://www.apache.org/dist/cassandra/redhat/40x/
>
> See http://cassandra.apache.org/download/ for full install instructions.
> Since this is the first alpha release, it will not be present on the
> download page.
>
> This is an ALPHA version! It is not intended for production use, however
> the project would appreciate your testing and feedback to make the final
> release better. As always, please pay attention to the release notes[2] and
> Let us know[3] if you were to encounter any problem.
>
> Enjoy!
>
> [1]: CHANGES.txt: https://gitbox.apache.org/repo
> s/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/
> tags/cassandra-4.0-alpha1
> [2]: NEWS.txt: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_
> plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha1
> [3]: JIRA: https://issues.apache.org/jira/browse/CASSANDRA
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Duplicates columns which are backed by LIST collection types

[Muralikrishna Gutha]

If data was TTL'ed & tombstoned, shouldn't select be returning just 3
elements from the list column ?

Thanks,
Murali

On Thu, Oct 24, 2019 at 12:03 PM ZAIDI, ASAD  wrote:

> Guess TTL’ed data is lurking around?  If so , you can try get rid of
> tombstones (by reducing gc_grace_seconds to zero? ) and let compaction take
> care of tombstone before sstable  migration. Do keep an eye on hinted
> handoffs  because of  zero’ed gc_grace_second property.
>
>
>
>
>
> *From:* Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
> *Sent:* Thursday, October 24, 2019 10:39 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Duplicates columns which are backed by LIST collection
> types
>
>
>
> Thanks, List datatype has been in-use for this table almost over a few
> years now and never had issues. We ran into this issue recently when we did
> the keyspace migration.
>
>
>
> Thanks,
>
> Murali
>
>
>
> On Thu, Oct 24, 2019 at 11:36 AM ZAIDI, ASAD  wrote:
>
> Have you chosen correct datatype to begin with, if you don’t want
> duplicates?
>
>
>
> Generally speaking:
>
>
>
> A set and a list both represent multiple values but do so differently.
>
> A set doesn’t save ordering and values are sorted in ascending order. No
> duplicates are allowed.
>
>
>
> A list saves ordering where you append or prepend the value into the list.
> A list allows duplicates.
>
>
>
>
>
>
>
> *From:* Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
> *Sent:* Thursday, October 24, 2019 10:27 AM
> *To:* user@cassandra.apache.org
> *Cc:* Muralikrishna Gutha 
> *Subject:* Duplicates columns which are backed by LIST collection types
>
>
>
> Hello Guys,
>
>
>
> We started noticing strange behavior after we migrated one keyspace from
> existing cluster to new cluster.
>
>
>
> We expanded our source cluster from 18 node to 36 nodes and Didn't run
> "nodetool cleanup".
>
> We took sstable backups on source cluster and restored which has duplicate
> data and restored (sstableloader) it on to new cluster. Apparently
> applications started seeing duplicate data mostly on list backed columns.
> Below is sstable2json output for one of the list backed columns.
>
>
>
> Clustering Column1:Clustering Column2:mods (List collection type
>
> ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233
>
>
>
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],
>
>
>
> Below is the select statement i would expect Cassandra to return data with
> latest timestamp rather it returns duplicate values.
>
>
>
> select mods from keyspace.table where partition_key ='1117302' and
> type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;
>
>
>
> [image: image.png]
>
>
>
> Any help or guidance is greatly appreciated.
>
>
>
> --
>
> Thanks & Regards
>   Murali K Gutha
>
>
>
>
> --
>
> Thanks & Regards
>   Murali K Gutha
>


-- 
Thanks & Regards
  Murali K Gutha

Re: Cassandra Rack - Datacenter Load Balancing relations

[Reid Pinchback]

Two different AWS AZs are in two different physical locations.  Typically 
different cities.  Which means that you’re trying to manage the risk of an AZ 
going dark, so you use more than one AZ just in case.  The downside is that you 
will have some degree of network performance difference between AZs because of 
whatever WAN pipe AWS owns/leased to connect between them.

Having a DC in one AZ is easy to reason about.  The AZ is there, or it is not.  
If you have two DCs in your cluster, and you lose an AZ, it means you still 
have a functioning cluster with one DC and you still have quorum.  Yay, even in 
an outage, you know you can still do business.  You would only have to route 
any traffic normally sent to the other DC to the remaining one, so as long as 
there is resource headroom planning in how you provision your hardware, you’re 
in a safe state.

If you start splitting a DC across AZs without using racks to organize nodes on 
a per-AZ basis, off the top of my head I don’t know how you reason about your 
risks for losing quorum without pausing to really think through vnodes and 
token distribution and whatnot.  I’m not a fan of topologies I can’t reason 
about when paged at 3 in the morning and I’m half asleep.  I prefer simple 
until the workload motivates complex.

R


From: Sergio 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, October 24, 2019 at 12:06 PM
To: "user@cassandra.apache.org" 
Subject: Re: Cassandra Rack - Datacenter Load Balancing relations

Message from External Sender
Thanks Reid and Jon!

Yes I will stick with one rack per DC for sure and I will look at the Vnodes 
problem later on.


What's the difference in terms of reliability between
A) spreading 2 Datacenters across 3 AZ
B) having 2 Datacenters in 2 separate AZ
?


Best,

Sergio

On Thu, Oct 24, 2019, 7:36 AM Reid Pinchback 
mailto:rpinchb...@tripadvisor.com>> wrote:
Hey Sergio,

Forgive but I’m at work and had to skim the info quickly.

When in doubt, simplify.  So 1 rack per DC.  Distributed systems get rapidly 
harder to reason about the more complicated you make them.  There’s more than 
enough to learn about C* without jumping into the complexity too soon.

To deal with the unbalancing issue, pay attention to Jon Haddad’s advice on 
vnode count and how to fairly distribute tokens with a small vnode count.  I’d 
rather point you to his information, as I haven’t dug into vnode counts and 
token distribution in detail; he’s got a lot more time in C* than I do.  I come 
at this more as a traditional RDBMS and Java guy who has slowly gotten up to 
speed on C* over the last few years, and dealt with DynamoDB a lot so have 
lived with a lot of similarity in data modelling concerns.  Detailed internals 
I only know in cases where I had reason to dig into C* source.

There are so many knobs to turn in C* that it can be very easy to overthink 
things.  Simplify where you can.  Remove GC pressure wherever you can.  
Negotiate with your consumers to have data models that make sense for C*.  If 
you have those three criteria foremost in mind, you’ll likely be fine for quite 
some time.  And in the times where something isn’t going well, simpler is 
easier to investigate.

R

From: Sergio mailto:lapostadiser...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, October 23, 2019 at 3:34 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Cassandra Rack - Datacenter Load Balancing relations

Message from External Sender
Hi Reid,

Thank you very much for clearing these concepts for me.
https://community.datastax.com/comments/1133/view.html
 I posted this question on the datastax forum regarding our cluster that it is 
unbalanced and the reply was related that the number of racks should be a 
multiplier of the replication factor in order to be balanced or 1. I thought 
then if I have 3 availability zones I should have 3 racks for each datacenter 
and not 2 (us-east-1b, us-east-1a) as I have right now or in the easiest way, I 
should have a rack for each datacenter.



1.  Datacenter: live

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load   Tokens   OwnsHost ID
   Rack
UN  10.1.20.49   289.75 GiB  256  ?   
be5a0193-56e7-4d42-8cc8-5d2141ab4872  us-east-1a
UN  10.1.30.112  103.03 GiB  256  ?   
e5108a8e-cc2f-4914-a86e-fccf770e3f0f  us-east-1b
UN  10.1.19.163  129.61 GiB  256  ?   
3c2efdda-8dd4-4f08-b991-9aff062a5388  us-east-1a
UN  10.1.26.181  145.28 GiB  256  ?   

Re: Cassandra Rack - Datacenter Load Balancing relations

[Sergio]

Thanks Reid and Jon!

Yes I will stick with one rack per DC for sure and I will look at the
Vnodes problem later on.


What's the difference in terms of reliability between
A) spreading 2 Datacenters across 3 AZ
B) having 2 Datacenters in 2 separate AZ
?


Best,

Sergio

On Thu, Oct 24, 2019, 7:36 AM Reid Pinchback 
wrote:

> Hey Sergio,
>
>
>
> Forgive but I’m at work and had to skim the info quickly.
>
>
>
> When in doubt, simplify.  So 1 rack per DC.  Distributed systems get
> rapidly harder to reason about the more complicated you make them.  There’s
> more than enough to learn about C* without jumping into the complexity too
> soon.
>
>
>
> To deal with the unbalancing issue, pay attention to Jon Haddad’s advice
> on vnode count and how to fairly distribute tokens with a small vnode
> count.  I’d rather point you to his information, as I haven’t dug into
> vnode counts and token distribution in detail; he’s got a lot more time in
> C* than I do.  I come at this more as a traditional RDBMS and Java guy who
> has slowly gotten up to speed on C* over the last few years, and dealt with
> DynamoDB a lot so have lived with a lot of similarity in data modelling
> concerns.  Detailed internals I only know in cases where I had reason to
> dig into C* source.
>
>
>
> There are so many knobs to turn in C* that it can be very easy to
> overthink things.  Simplify where you can.  Remove GC pressure wherever you
> can.  Negotiate with your consumers to have data models that make sense for
> C*.  If you have those three criteria foremost in mind, you’ll likely be
> fine for quite some time.  And in the times where something isn’t going
> well, simpler is easier to investigate.
>
>
> R
>
>
>
> *From: *Sergio 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Wednesday, October 23, 2019 at 3:34 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Cassandra Rack - Datacenter Load Balancing relations
>
>
>
> *Message from External Sender*
>
> Hi Reid,
>
> Thank you very much for clearing these concepts for me.
> https://community.datastax.com/comments/1133/view.html
> 
> I posted this question on the datastax forum regarding our cluster that it
> is unbalanced and the reply was related that the *number of racks should
> be a multiplier of the replication factor *in order to be balanced or 1.
> I thought then if I have 3 availability zones I should have 3 racks for
> each datacenter and not 2 (us-east-1b, us-east-1a) as I have right now or
> in the easiest way, I should have a rack for each datacenter.
>
>
>
> 1.  Datacenter: live
> 
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   OwnsHost ID
> Rack
> UN  10.1.20.49   289.75 GiB  256  ?
> be5a0193-56e7-4d42-8cc8-5d2141ab4872  us-east-1a
> UN  10.1.30.112  103.03 GiB  256  ?
> e5108a8e-cc2f-4914-a86e-fccf770e3f0f  us-east-1b
> UN  10.1.19.163  129.61 GiB  256  ?
> 3c2efdda-8dd4-4f08-b991-9aff062a5388  us-east-1a
> UN  10.1.26.181  145.28 GiB  256  ?
> 0a8f07ba-a129-42b0-b73a-df649bd076ef  us-east-1b
> UN  10.1.17.213  149.04 GiB  256  ?
> 71563e86-b2ae-4d2c-91c5-49aa08386f67  us-east-1a
> DN  10.1.19.198  52.41 GiB  256  ?
> 613b43c0-0688-4b86-994c-dc772b6fb8d2  us-east-1b
> UN  10.1.31.60   195.17 GiB  256  ?
> 3647fcca-688a-4851-ab15-df36819910f4  us-east-1b
> UN  10.1.25.206  100.67 GiB  256  ?
> f43532ad-7d2e-4480-a9ce-2529b47f823d  us-east-1b
> So each rack label right now matches the availability zone and we have 3
> Datacenters and 2 Availability Zone with 2 racks per DC but the above is
> clearly unbalanced
> If I have a keyspace with a replication factor = 3 and I want to minimize
> the number of nodes to scale up and down the cluster and keep it balanced
> should I consider an approach like OPTION A)
>
> 2.  Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1a
>
> 3.  3 read ONE us-east-1a
>
> 4.  4 write ONE us-east-1b 5 write ONE us-east-1b
>
> 5.  6 write ONE us-east-1b
>
> 6.  OPTION B)
>
> 7.  Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1a
>
> 8.  3 read ONE us-east-1a
>
> 9.  4 write TWO us-east-1b 5 write TWO us-east-1b
>
> 10.6 write TWO us-east-1b
>
> 11.*7 read ONE us-east-1c 8 write TWO us-east-1c*
>
> 12.*9 read ONE us-east-1c* Option B looks to be unbalanced and I would
> exclude it OPTION C)
>
> 13.Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1b
>
> 14.3 read ONE us-east-1c
>
> 15.4 write TWO us-east-1a 5 write TWO us-east-1b
>
> 16.6 write TWO us-east-1c
>
> 17.
>
> so I am thinking of A if I have the restriction of 2 AZ but I guess that
> option C would be the best. If I have to 

RE: Duplicates columns which are backed by LIST collection types

[ZAIDI, ASAD]

Guess TTL’ed data is lurking around?  If so , you can try get rid of tombstones 
(by reducing gc_grace_seconds to zero? ) and let compaction take care of 
tombstone before sstable  migration. Do keep an eye on hinted handoffs  because 
of  zero’ed gc_grace_second property.


From: Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
Sent: Thursday, October 24, 2019 10:39 AM
To: user@cassandra.apache.org
Subject: Re: Duplicates columns which are backed by LIST collection types

Thanks, List datatype has been in-use for this table almost over a few years 
now and never had issues. We ran into this issue recently when we did the 
keyspace migration.

Thanks,
Murali

On Thu, Oct 24, 2019 at 11:36 AM ZAIDI, ASAD 
mailto:az1...@att.com>> wrote:
Have you chosen correct datatype to begin with, if you don’t want duplicates?

Generally speaking:

A set and a list both represent multiple values but do so differently.
A set doesn’t save ordering and values are sorted in ascending order. No 
duplicates are allowed.

A list saves ordering where you append or prepend the value into the list. A 
list allows duplicates.



From: Muralikrishna Gutha 
[mailto:muralikgu...@gmail.com]
Sent: Thursday, October 24, 2019 10:27 AM
To: user@cassandra.apache.org
Cc: Muralikrishna Gutha mailto:muralikgu...@gmail.com>>
Subject: Duplicates columns which are backed by LIST collection types

Hello Guys,

We started noticing strange behavior after we migrated one keyspace from 
existing cluster to new cluster.

We expanded our source cluster from 18 node to 36 nodes and Didn't run 
"nodetool cleanup".
We took sstable backups on source cluster and restored which has duplicate data 
and restored (sstableloader) it on to new cluster. Apparently applications 
started seeing duplicate data mostly on list backed columns. Below is 
sstable2json output for one of the list backed columns.

Clustering Column1:Clustering Column2:mods (List collection type
ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],

Below is the select statement i would expect Cassandra to return data with 
latest timestamp rather it returns duplicate values.

select mods from keyspace.table where partition_key ='1117302' and 
type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;

[image.png]

Any help or guidance is greatly appreciated.

--
Thanks & Regards
  Murali K Gutha


--
Thanks & Regards
  Murali K Gutha

Re: Duplicates columns which are backed by LIST collection types

[Muralikrishna Gutha]

Thanks, List datatype has been in-use for this table almost over a few
years now and never had issues. We ran into this issue recently when we did
the keyspace migration.

Thanks,
Murali

On Thu, Oct 24, 2019 at 11:36 AM ZAIDI, ASAD  wrote:

> Have you chosen correct datatype to begin with, if you don’t want
> duplicates?
>
>
>
> Generally speaking:
>
>
>
> A set and a list both represent multiple values but do so differently.
>
> A set doesn’t save ordering and values are sorted in ascending order. No
> duplicates are allowed.
>
>
>
> A list saves ordering where you append or prepend the value into the list.
> A list allows duplicates.
>
>
>
>
>
>
>
> *From:* Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
> *Sent:* Thursday, October 24, 2019 10:27 AM
> *To:* user@cassandra.apache.org
> *Cc:* Muralikrishna Gutha 
> *Subject:* Duplicates columns which are backed by LIST collection types
>
>
>
> Hello Guys,
>
>
>
> We started noticing strange behavior after we migrated one keyspace from
> existing cluster to new cluster.
>
>
>
> We expanded our source cluster from 18 node to 36 nodes and Didn't run
> "nodetool cleanup".
>
> We took sstable backups on source cluster and restored which has duplicate
> data and restored (sstableloader) it on to new cluster. Apparently
> applications started seeing duplicate data mostly on list backed columns.
> Below is sstable2json output for one of the list backed columns.
>
>
>
> Clustering Column1:Clustering Column2:mods (List collection type
>
> ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233
>
>
>
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],
>
>  
> ["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],
>
>
>
> Below is the select statement i would expect Cassandra to return data with
> latest timestamp rather it returns duplicate values.
>
>
>
> select mods from keyspace.table where partition_key ='1117302' and
> type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;
>
>
>
> [image: image.png]
>
>
>
> Any help or guidance is greatly appreciated.
>
>
>
> --
>
> Thanks & Regards
>   Murali K Gutha
>


-- 
Thanks & Regards
  Murali K Gutha

RE: Duplicates columns which are backed by LIST collection types

[ZAIDI, ASAD]

Have you chosen correct datatype to begin with, if you don’t want duplicates?

Generally speaking:

A set and a list both represent multiple values but do so differently.
A set doesn’t save ordering and values are sorted in ascending order. No 
duplicates are allowed.

A list saves ordering where you append or prepend the value into the list. A 
list allows duplicates.



From: Muralikrishna Gutha [mailto:muralikgu...@gmail.com]
Sent: Thursday, October 24, 2019 10:27 AM
To: user@cassandra.apache.org
Cc: Muralikrishna Gutha 
Subject: Duplicates columns which are backed by LIST collection types

Hello Guys,

We started noticing strange behavior after we migrated one keyspace from 
existing cluster to new cluster.

We expanded our source cluster from 18 node to 36 nodes and Didn't run 
"nodetool cleanup".
We took sstable backups on source cluster and restored which has duplicate data 
and restored (sstableloader) it on to new cluster. Apparently applications 
started seeing duplicate data mostly on list backed columns. Below is 
sstable2json output for one of the list backed columns.

Clustering Column1:Clustering Column2:mods (List collection type
ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],
   
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],

Below is the select statement i would expect Cassandra to return data with 
latest timestamp rather it returns duplicate values.

select mods from keyspace.table where partition_key ='1117302' and 
type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;

[image.png]

Any help or guidance is greatly appreciated.

--
Thanks & Regards
  Murali K Gutha

Duplicates columns which are backed by LIST collection types

[Muralikrishna Gutha]

Hello Guys,

We started noticing strange behavior after we migrated one keyspace from
existing cluster to new cluster.

We expanded our source cluster from 18 node to 36 nodes and Didn't run
"nodetool cleanup".
We took sstable backups on source cluster and restored which has duplicate
data and restored (sstableloader) it on to new cluster. Apparently
applications started seeing duplicate data mostly on list backed columns.
Below is sstable2json output for one of the list backed columns.

Clustering Column1:Clustering Column2:mods (List collection type
ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b050eac811e9ab2729ea208ce219","eb25d0b13a6611e980b22102e728a233",1570648383445000],

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b051eac811e9ab2729ea208ce219","eb26bb113a6611e980b22102e728a233",1570648383445000],

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:d120b052eac811e9ab2729ea208ce219","a4fcf1f1eac811e99664732b9302ab46",1570648383445000],

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973560ead811e98bf68711844fec13","eb25d0b13a6611e980b22102e728a233",1570654999478000],

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973561ead811e98bf68711844fec13","eb26bb113a6611e980b22102e728a233",1570654999478000],

 
["ModifierList:eb26e221-3a66-11e9-80b2-2102e728a233:mods:38973562ead811e98bf68711844fec13","a4fcf1f1eac811e99664732b9302ab46",1570654999478000],

Below is the select statement i would expect Cassandra to return data with
latest timestamp rather it returns duplicate values.

select mods from keyspace.table where partition_key ='1117302' and
type='ModifierList' and id=eb26e221-3a66-11e9-80b2-2102e728a233;

[image: image.png]

Any help or guidance is greatly appreciated.

-- 
Thanks & Regards
  Murali K Gutha

Re: Cassandra Rack - Datacenter Load Balancing relations

[Reid Pinchback]

Hey Sergio,

Forgive but I’m at work and had to skim the info quickly.

When in doubt, simplify.  So 1 rack per DC.  Distributed systems get rapidly 
harder to reason about the more complicated you make them.  There’s more than 
enough to learn about C* without jumping into the complexity too soon.

To deal with the unbalancing issue, pay attention to Jon Haddad’s advice on 
vnode count and how to fairly distribute tokens with a small vnode count.  I’d 
rather point you to his information, as I haven’t dug into vnode counts and 
token distribution in detail; he’s got a lot more time in C* than I do.  I come 
at this more as a traditional RDBMS and Java guy who has slowly gotten up to 
speed on C* over the last few years, and dealt with DynamoDB a lot so have 
lived with a lot of similarity in data modelling concerns.  Detailed internals 
I only know in cases where I had reason to dig into C* source.

There are so many knobs to turn in C* that it can be very easy to overthink 
things.  Simplify where you can.  Remove GC pressure wherever you can.  
Negotiate with your consumers to have data models that make sense for C*.  If 
you have those three criteria foremost in mind, you’ll likely be fine for quite 
some time.  And in the times where something isn’t going well, simpler is 
easier to investigate.

R

From: Sergio 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, October 23, 2019 at 3:34 PM
To: "user@cassandra.apache.org" 
Subject: Re: Cassandra Rack - Datacenter Load Balancing relations

Message from External Sender
Hi Reid,

Thank you very much for clearing these concepts for me.
https://community.datastax.com/comments/1133/view.html
 I posted this question on the datastax forum regarding our cluster that it is 
unbalanced and the reply was related that the number of racks should be a 
multiplier of the replication factor in order to be balanced or 1. I thought 
then if I have 3 availability zones I should have 3 racks for each datacenter 
and not 2 (us-east-1b, us-east-1a) as I have right now or in the easiest way, I 
should have a rack for each datacenter.




1.  Datacenter: live

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load   Tokens   OwnsHost ID
   Rack
UN  10.1.20.49   289.75 GiB  256  ?   
be5a0193-56e7-4d42-8cc8-5d2141ab4872  us-east-1a
UN  10.1.30.112  103.03 GiB  256  ?   
e5108a8e-cc2f-4914-a86e-fccf770e3f0f  us-east-1b
UN  10.1.19.163  129.61 GiB  256  ?   
3c2efdda-8dd4-4f08-b991-9aff062a5388  us-east-1a
UN  10.1.26.181  145.28 GiB  256  ?   
0a8f07ba-a129-42b0-b73a-df649bd076ef  us-east-1b
UN  10.1.17.213  149.04 GiB  256  ?   
71563e86-b2ae-4d2c-91c5-49aa08386f67  us-east-1a
DN  10.1.19.198  52.41 GiB  256  ?   
613b43c0-0688-4b86-994c-dc772b6fb8d2  us-east-1b
UN  10.1.31.60   195.17 GiB  256  ?   
3647fcca-688a-4851-ab15-df36819910f4  us-east-1b
UN  10.1.25.206  100.67 GiB  256  ?   
f43532ad-7d2e-4480-a9ce-2529b47f823d  us-east-1b
So each rack label right now matches the availability zone and we have 3 
Datacenters and 2 Availability Zone with 2 racks per DC but the above is 
clearly unbalanced
If I have a keyspace with a replication factor = 3 and I want to minimize the 
number of nodes to scale up and down the cluster and keep it balanced should I 
consider an approach like OPTION A)

2.  Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1a

3.  3 read ONE us-east-1a

4.  4 write ONE us-east-1b 5 write ONE us-east-1b

5.  6 write ONE us-east-1b

6.  OPTION B)

7.  Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1a

8.  3 read ONE us-east-1a

9.  4 write TWO us-east-1b 5 write TWO us-east-1b

10.6 write TWO us-east-1b

11.7 read ONE us-east-1c 8 write TWO us-east-1c

12.9 read ONE us-east-1c Option B looks to be unbalanced and I would exclude it 
OPTION C)

13.Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1b

14.3 read ONE us-east-1c

15.4 write TWO us-east-1a 5 write TWO us-east-1b

16.6 write TWO us-east-1c

17.
so I am thinking of A if I have the restriction of 2 AZ but I guess that option 
C would be the best. If I have to add another DC for reads because we want to 
assign a new DC for each new microservice it would look like:
OPTION EXTRA DC For Reads

1.  Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1b

2.  3 read ONE us-east-1c

3.  4 write TWO us-east-1a 5 write TWO us-east-1b

4.  6 write TWO us-east-1c 7 extra-read THREE us-east-1a

5.  8 extra-read THREE us-east-1b

6.

7.

1.  9 extra-read THREE us-east-1c

2.
The DC for write will replicate the data in the other 

Re: [EXTERNAL] Cassandra Export error in COPY command

[Hossein Ghiyasi Mehr]

I tested dsbulk too. But there are many errors:

"[1710949318] Error writing cancel request. This is not critical (the
request will eventually time out server-side)."
"Forcing termination of Connection[/127.0.0.1:9042-14, inFlight=1,
closed=true]. This should not happen and is likely a bug, please report."

I tried with "--executor.maxPerSecond 1000" and
"--driver.socket.readTimeout 3600" options but errors didn't resolve.

How can I export a table data?

On Mon, Sep 23, 2019 at 6:30 AM Durity, Sean R 
wrote:

> Copy command tries to export all rows in the table, not just the ones on
> the node. It will eventually timeout if the table is large. It is really
> built for something under 5 million rows or so. Dsbulk (from DataStax) is
> great for this, if you are a customer. Otherwise, you will probably need to
> write an extract of some kind. You can get keys from the sstables, then
> dedupe, then export rows one by one using the keys (kind of painful). How
> large is the table you are trying to export?
>
>
>
> Sean Durity
>
>
>
> *From:* Hossein Ghiyasi Mehr 
> *Sent:* Saturday, September 21, 2019 8:02 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra Export error in COPY command
>
>
>
> Hi all members,
>
> I want to export (pk, another_int_column) from single node using COPY
> command. But after about 1h 45m, I've got a lot of read errors:
>
>
>
>
>
> I tried this action many times but after maximum 2h, it failed with the
> errors.
>
>
>
> Any idea may help me!
>
> Thanks.
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>