Re: Basic query in setting up secure inter-dc cluster

[Ajay Garg]

Thanks everyone for the reply.

I actually have a fair bit of questions, but it will be nice if someone
could please tell me the flow (implementation-wise), as to how node-to-node
encryption works in a cluster.

Let's say node1 from DC1, wishes to talk securely to node 2 from DC2
(with *"require_client_auth:
false*").
I presume it would be like below (please correct me if am wrong) ::

a)
node1 tries to connect to node2, using the certificate *as defined on node1*
in cassandra.yaml.

b)
node2 will confirm if the certificate being offered by node1 is in the
truststore *as defined on node2* in cassandra.yaml.
if it is, secure-communication is allowed.


Is my thinking right?
I

On Wed, Jan 6, 2016 at 1:55 PM, Neha Dave  wrote:

> Hi Ajay,
> Have a look here :
> https://docs.datastax.com/en/cassandra/1.2/cassandra/security/secureSSLNodeToNode_t.html
>
> You can configure for DC level Security:
>
> Procedure
>
> On each node under sever_encryption_options:
>
>- Enable internode_encryption.
>The available options are:
>   - all
>   - none
>   - dc: Cassandra encrypts the traffic between the data centers.
>   - rack: Cassandra encrypts the traffic between the racks.
>
> regards
>
> Neha
>
>
>
> On Wed, Jan 6, 2016 at 12:48 PM, Singh, Abhijeet 
> wrote:
>
>> Security is a very wide concept. What exactly do you want to achieve ?
>>
>>
>>
>> *From:* Ajay Garg [mailto:ajaygargn...@gmail.com]
>> *Sent:* Wednesday, January 06, 2016 11:27 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Basic query in setting up secure inter-dc cluster
>>
>>
>>
>> Hi All.
>>
>> We have a 2*2 cluster deployed, but no security as of now.
>>
>> As a first stage, we wish to implement inter-dc security.
>>
>> Is it possible to enable security one machine at a time?
>>
>> For example, let's say the machines are DC1M1, DC1M2, DC2M1, DC2M2.
>>
>> If I make the changes JUST IN DC2M2 and restart it, will the traffic
>> between DC1M1/DC1M2 and DC2M2 be secure? Or security will kick in ONLY
>> AFTER the changes are made in all the 4 machines?
>>
>> Asking here, because I don't want to screw up a live cluster due to my
>> lack of experience.
>>
>> Looking forward to some pointers.
>>
>>
>> --
>>
>> Regards,
>> Ajay
>>
>
>


-- 
Regards,
Ajay

Re: Node stuck when joining a Cassandra 2.2.0 cluster

[Herbert Fischer]

Hi,

Thanks for the tip.

I found that one keyspace was kinda corrupted. It was previously
scrubbed/deleted but there where files left in the servers, so it was in a
strange state. After removing it from the filesystem I was able to add the
new node to the cluster. Since this keyspace was in an unknown state, I
could not find it through the cfId from the error messages.

best

On 5 January 2016 at 22:33, Robert Coli  wrote:

> On Tue, Jan 5, 2016 at 3:01 AM, Herbert Fischer <
> herbert.fisc...@crossengage.io> wrote:
>
>> We run a small Cassandra 2.2.0 cluster, with 5 nodes, on bare-metal
>> servers and we are going to replace those nodes with other nodes. I planned
>> to add all the new nodes first, one-by-one, and later remove the old ones,
>> one-by-one.
>>
>
> It sounds like your bootstraps are hanging. Your streams should restart
> after an hour, but probably you want to figure out why they're hanging...
>
> You can also use the auto_bootstrap=false+hibernate repair method for this
> process. That's probably what I'd do if I was upgrading the hardware of
> nodes in place.
>
> =Rob
>
>



-- 
Herbert Fischer | Senior IT Architect
CrossEngage GmbH (haftungsbeschränkt) | Julie-Wolfthorn-Straße 1 | 10115
Berlin

E-Mail: herbert.fisc...@crossengage.io
Web: www.crossengage.io

Amtsgericht Berlin-Charlottenburg | HRB 169537 B
Geschäftsführer: Dr. Markus Wübben, Manuel Hinz | USt-IdNr.: DE301504202

Re: Basic query in setting up secure inter-dc cluster

[Neha Dave]

Hi Ajay,
Have a look here :
https://docs.datastax.com/en/cassandra/1.2/cassandra/security/secureSSLNodeToNode_t.html

You can configure for DC level Security:

Procedure

On each node under sever_encryption_options:

   - Enable internode_encryption.
   The available options are:
  - all
  - none
  - dc: Cassandra encrypts the traffic between the data centers.
  - rack: Cassandra encrypts the traffic between the racks.

regards

Neha



On Wed, Jan 6, 2016 at 12:48 PM, Singh, Abhijeet 
wrote:

> Security is a very wide concept. What exactly do you want to achieve ?
>
>
>
> *From:* Ajay Garg [mailto:ajaygargn...@gmail.com]
> *Sent:* Wednesday, January 06, 2016 11:27 AM
> *To:* user@cassandra.apache.org
> *Subject:* Basic query in setting up secure inter-dc cluster
>
>
>
> Hi All.
>
> We have a 2*2 cluster deployed, but no security as of now.
>
> As a first stage, we wish to implement inter-dc security.
>
> Is it possible to enable security one machine at a time?
>
> For example, let's say the machines are DC1M1, DC1M2, DC2M1, DC2M2.
>
> If I make the changes JUST IN DC2M2 and restart it, will the traffic
> between DC1M1/DC1M2 and DC2M2 be secure? Or security will kick in ONLY
> AFTER the changes are made in all the 4 machines?
>
> Asking here, because I don't want to screw up a live cluster due to my
> lack of experience.
>
> Looking forward to some pointers.
>
>
> --
>
> Regards,
> Ajay
>

Re: Data Modeling: Partition Size and Query Efficiency

[Jim Ancona]

On Tue, Jan 5, 2016 at 5:52 PM, Jonathan Haddad  wrote:

> You could keep a "num_buckets" value associated with the client's account,
> which can be adjusted accordingly as usage increases.
>

Yes, but the adjustment problem is tricky when there are multiple
concurrent writers. What happens when you change the number of buckets?
Does existing data have to be re-written into new buckets? If so, how do
you make sure that's only done once for each bucket size increase? Or
perhaps I'm misunderstanding your suggestion?

Jim


> On Tue, Jan 5, 2016 at 2:17 PM Jim Ancona  wrote:
>
>> On Tue, Jan 5, 2016 at 4:56 PM, Clint Martin <
>> clintlmar...@coolfiretechnologies.com> wrote:
>>
>>> What sort of data is your clustering key composed of? That might help
>>> some in determining a way to achieve what you're looking for.
>>>
>> Just a UUID that acts as an object identifier.
>>
>>>
>>> Clint
>>> On Jan 5, 2016 2:28 PM, "Jim Ancona"  wrote:
>>>
 Hi Nate,

 Yes, I've been thinking about treating customers as either small or
 big, where "small" ones have a single partition and big ones have 50 (or
 whatever number I need to keep sizes reasonable). There's still the problem
 of how to handle a small customer who becomes too big, but that will happen
 much less frequently than a customer filling a partition.

 Jim

 On Tue, Jan 5, 2016 at 12:21 PM, Nate McCall 
 wrote:

>
>> In this case, 99% of my data could fit in a single 50 MB partition.
>> But if I use the standard approach, I have to split my partitions into 50
>> pieces to accommodate the largest data. That means that to query the 700
>> rows for my median case, I have to read 50 partitions instead of one.
>>
>> If you try to deal with this by starting a new partition when an old
>> one fills up, you have a nasty distributed consensus problem, along with
>> read-before-write. Cassandra LWT wasn't available the last time I dealt
>> with this, but might help with the consensus part today. But there are
>> still some nasty corner cases.
>>
>> I have some thoughts on other ways to solve this, but they all have
>> drawbacks. So I thought I'd ask here and hope that someone has a better
>> approach.
>>
>>
> Hi Jim - good to see you around again.
>
> If you can segment this upstream by customer/account/whatever,
> handling the outliers as an entirely different code path (potentially
> different cluster as the workload will be quite different at that point 
> and
> have different tuning requirements) would be your best bet. Then a
> read-before-write makes sense given it is happening on such a small number
> of API queries.
>
>
> --
> -
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

New node has high network and disk usage.

[Vickrum Loi]

Hi,

We recently added a new node to our cluster in order to replace a node that
died (hardware failure we believe). For the next two weeks it had high disk
and network activity. We replaced the server, but it's happened again.
We've looked into memory allowances, disk performance, number of
connections, and all the nodetool stats, but can't find the cause of the
issue.

`nodetool tpstats`[0] shows a lot of active and pending threads, in
comparison to the rest of the cluster, but that's likely a symptom, not a
cause.

`nodetool status`[1] shows the cluster isn't quite balanced. The bad node
(D) has less data.

Disk Activity[2] and Network activity[3] on this node is far higher than
the rest.

The only other difference this node has to the rest of the cluster is that
its on the ext4 filesystem, whereas the rest are ext3, but we've done
plenty of testing there and can't see how that would affect performance on
this node so much.

Nothing of note in system.log.

What should our next step be in trying to diagnose this issue?

Best wishes,
Vic

[0] `nodetool tpstats` output:

Good node:
Pool NameActive   Pending  Completed   Blocked
All time blocked
ReadStage 0 0   46311521
0 0
RequestResponseStage  0 0   23817366
0 0
MutationStage 0 0   47389269
0 0
ReadRepairStage   0 0  11108
0 0
ReplicateOnWriteStage 0 0  0
0 0
GossipStage   0 05259908
0 0
CacheCleanupExecutor  0 0  0
0 0
MigrationStage0 0 30
0 0
MemoryMeter   0 0  16563
0 0
FlushWriter   0 0  39637
026
ValidationExecutor0 0  19013
0 0
InternalResponseStage 0 0  9
0 0
AntiEntropyStage  0 0  38026
0 0
MemtablePostFlusher   0 0  81740
0 0
MiscStage 0 0  19196
0 0
PendingRangeCalculator0 0 23
0 0
CompactionExecutor0 0  61629
0 0
commitlog_archiver0 0  0
0 0
HintedHandoff 0 0 63
0 0

Message type   Dropped
RANGE_SLICE  0
READ_REPAIR  0
PAGED_RANGE  0
BINARY   0
READ   640
MUTATION 0
_TRACE   0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0

Bad node:
Pool NameActive   Pending  Completed   Blocked
All time blocked
ReadStage32   113  52216
0 0
RequestResponseStage  0 0   4167
0 0
MutationStage 0 0 127559
0 0
ReadRepairStage   0 0125
0 0
ReplicateOnWriteStage 0 0  0
0 0
GossipStage   0 0   9965
0 0
CacheCleanupExecutor  0 0  0
0 0
MigrationStage0 0  0
0 0
MemoryMeter   0 0 24
0 0
FlushWriter   0 0 27
0 1
ValidationExecutor0 0  0
0 0
InternalResponseStage 0 0  0
0 0
AntiEntropyStage  0 0  0
0 0
MemtablePostFlusher   0 0 96
0 0
MiscStage 0 0  0
0 0
PendingRangeCalculator0 0 10
0 0
CompactionExecutor1 1 73
0 0
commitlog_archiver0 0  0
0 0
HintedHandoff 0 0 15
0 0

Message type   Dropped
RANGE_SLICE130
READ_REPAIR  1
PAGED_RANGE   

Re: New node has high network and disk usage.

[Vickrum Loi]

I should probably have mentioned that we're on Cassandra 2.0.10.

On 6 January 2016 at 15:26, Vickrum Loi 
wrote:

> Hi,
>
> We recently added a new node to our cluster in order to replace a node
> that died (hardware failure we believe). For the next two weeks it had high
> disk and network activity. We replaced the server, but it's happened again.
> We've looked into memory allowances, disk performance, number of
> connections, and all the nodetool stats, but can't find the cause of the
> issue.
>
> `nodetool tpstats`[0] shows a lot of active and pending threads, in
> comparison to the rest of the cluster, but that's likely a symptom, not a
> cause.
>
> `nodetool status`[1] shows the cluster isn't quite balanced. The bad node
> (D) has less data.
>
> Disk Activity[2] and Network activity[3] on this node is far higher than
> the rest.
>
> The only other difference this node has to the rest of the cluster is that
> its on the ext4 filesystem, whereas the rest are ext3, but we've done
> plenty of testing there and can't see how that would affect performance on
> this node so much.
>
> Nothing of note in system.log.
>
> What should our next step be in trying to diagnose this issue?
>
> Best wishes,
> Vic
>
> [0] `nodetool tpstats` output:
>
> Good node:
> Pool NameActive   Pending  Completed
> Blocked  All time blocked
> ReadStage 0 0   46311521
> 0 0
> RequestResponseStage  0 0   23817366
> 0 0
> MutationStage 0 0   47389269
> 0 0
> ReadRepairStage   0 0  11108
> 0 0
> ReplicateOnWriteStage 0 0  0
> 0 0
> GossipStage   0 05259908
> 0 0
> CacheCleanupExecutor  0 0  0
> 0 0
> MigrationStage0 0 30
> 0 0
> MemoryMeter   0 0  16563
> 0 0
> FlushWriter   0 0  39637
> 026
> ValidationExecutor0 0  19013
> 0 0
> InternalResponseStage 0 0  9
> 0 0
> AntiEntropyStage  0 0  38026
> 0 0
> MemtablePostFlusher   0 0  81740
> 0 0
> MiscStage 0 0  19196
> 0 0
> PendingRangeCalculator0 0 23
> 0 0
> CompactionExecutor0 0  61629
> 0 0
> commitlog_archiver0 0  0
> 0 0
> HintedHandoff 0 0 63
> 0 0
>
> Message type   Dropped
> RANGE_SLICE  0
> READ_REPAIR  0
> PAGED_RANGE  0
> BINARY   0
> READ   640
> MUTATION 0
> _TRACE   0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
>
> Bad node:
> Pool NameActive   Pending  Completed
> Blocked  All time blocked
> ReadStage32   113  52216
> 0 0
> RequestResponseStage  0 0   4167
> 0 0
> MutationStage 0 0 127559
> 0 0
> ReadRepairStage   0 0125
> 0 0
> ReplicateOnWriteStage 0 0  0
> 0 0
> GossipStage   0 0   9965
> 0 0
> CacheCleanupExecutor  0 0  0
> 0 0
> MigrationStage0 0  0
> 0 0
> MemoryMeter   0 0 24
> 0 0
> FlushWriter   0 0 27
> 0 1
> ValidationExecutor0 0  0
> 0 0
> InternalResponseStage 0 0  0
> 0 0
> AntiEntropyStage  0 0  0
> 0 0
> MemtablePostFlusher   0 0 96
> 0 0
> MiscStage 0 0  0
> 0 0
> PendingRangeCalculator0 0 10
> 0 0
> 

Formal EOL for 2.0.17 and 2.1.12

[Anuj Wadehra]

Hi,
Can someone help me by providing formal dates of EOL for Cassandra 2.0.17 and 
2.1.12?

ThanksAnuj



Sent from Yahoo Mail on Android

Re: New node has high network and disk usage.

[Jeff Ferland]

What’s your output of `nodetool compactionstats`?

> On Jan 6, 2016, at 7:26 AM, Vickrum Loi  wrote:
> 
> Hi,
> 
> We recently added a new node to our cluster in order to replace a node that 
> died (hardware failure we believe). For the next two weeks it had high disk 
> and network activity. We replaced the server, but it's happened again. We've 
> looked into memory allowances, disk performance, number of connections, and 
> all the nodetool stats, but can't find the cause of the issue.
> 
> `nodetool tpstats`[0] shows a lot of active and pending threads, in 
> comparison to the rest of the cluster, but that's likely a symptom, not a 
> cause.
> 
> `nodetool status`[1] shows the cluster isn't quite balanced. The bad node (D) 
> has less data.
> 
> Disk Activity[2] and Network activity[3] on this node is far higher than the 
> rest.
> 
> The only other difference this node has to the rest of the cluster is that 
> its on the ext4 filesystem, whereas the rest are ext3, but we've done plenty 
> of testing there and can't see how that would affect performance on this node 
> so much.
> 
> Nothing of note in system.log.
> 
> What should our next step be in trying to diagnose this issue?
> 
> Best wishes,
> Vic
> 
> [0] `nodetool tpstats` output:
> 
> Good node:
> Pool NameActive   Pending  Completed   Blocked  
> All time blocked
> ReadStage 0 0   46311521 0
>  0
> RequestResponseStage  0 0   23817366 0
>  0
> MutationStage 0 0   47389269 0
>  0
> ReadRepairStage   0 0  11108 0
>  0
> ReplicateOnWriteStage 0 0  0 0
>  0
> GossipStage   0 05259908 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> MigrationStage0 0 30 0
>  0
> MemoryMeter   0 0  16563 0
>  0
> FlushWriter   0 0  39637 0
> 26
> ValidationExecutor0 0  19013 0
>  0
> InternalResponseStage 0 0  9 0
>  0
> AntiEntropyStage  0 0  38026 0
>  0
> MemtablePostFlusher   0 0  81740 0
>  0
> MiscStage 0 0  19196 0
>  0
> PendingRangeCalculator0 0 23 0
>  0
> CompactionExecutor0 0  61629 0
>  0
> commitlog_archiver0 0  0 0
>  0
> HintedHandoff 0 0 63 0
>  0
> 
> Message type   Dropped
> RANGE_SLICE  0
> READ_REPAIR  0
> PAGED_RANGE  0
> BINARY   0
> READ   640
> MUTATION 0
> _TRACE   0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
> 
> Bad node:
> Pool NameActive   Pending  Completed   Blocked  
> All time blocked
> ReadStage32   113  52216 0
>  0
> RequestResponseStage  0 0   4167 0
>  0
> MutationStage 0 0 127559 0
>  0
> ReadRepairStage   0 0125 0
>  0
> ReplicateOnWriteStage 0 0  0 0
>  0
> GossipStage   0 0   9965 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> MigrationStage0 0  0 0
>  0
> MemoryMeter   0 0 24 0
>  0
> FlushWriter   0 0 27 0
>  1
> ValidationExecutor0 0  0 0
>  0
> InternalResponseStage 0 0  0 0
>  0
> AntiEntropyStage  0 0  0 

Re: New node has high network and disk usage.

[Anuj Wadehra]

Hi Vickrum,
I would have proceeded with diagnosis as follows:
1. Analysis of sar report to check system health -cpu memory swap disk etc. 
System seems to be overloaded. This is evident from mutation drops.
2. Make sure that  all recommended Cassandra production settings available at 
Datastax site are applied ,disable zone reclaim and THP.
3.Run full Repair on bad node and check data size. Node is owner of maximum 
token range but has significant lower data.I doubt that bootstrapping happened 
properly.
4.Compactionstats shows 22 pending compactions. Try throttling compactions via 
reducing cincurent compactors or compaction throughput.
5.Analyze logs to make sure bootstrapping happened without errors.
6. Look for other common performance problems such as GC pauses to make sure 
that dropped mutations are not caused by GC pauses.

ThanksAnuj

Sent from Yahoo Mail on Android 
 
  On Wed, 6 Jan, 2016 at 10:12 pm, Vickrum Loi 
wrote:   # nodetool compactionstats
pending tasks: 22
  compaction type    keyspace   table   completed   
    total  unit  progress
   Compactionproduction_analytics    interactions   240410213   
 161172668724 bytes 0.15%
   Compactionproduction_decisionsdecisions.decisions_q_idx   
120815385   226295183 bytes    53.39%
Active compaction remaining time :   2h39m58s

Worth mentioning that compactions haven't been running on this node 
particularly often. The node's been performing badly regardless of whether it's 
compacting or not.

On 6 January 2016 at 16:35, Jeff Ferland  wrote:

What’s your output of `nodetool compactionstats`?

On Jan 6, 2016, at 7:26 AM, Vickrum Loi  wrote:
Hi,

We recently added a new node to our cluster in order to replace a node that 
died (hardware failure we believe). For the next two weeks it had high disk and 
network activity. We replaced the server, but it's happened again. We've looked 
into memory allowances, disk performance, number of connections, and all the 
nodetool stats, but can't find the cause of the issue.

`nodetool tpstats`[0] shows a lot of active and pending threads, in comparison 
to the rest of the cluster, but that's likely a symptom, not a cause.

`nodetool status`[1] shows the cluster isn't quite balanced. The bad node (D) 
has less data.

Disk Activity[2] and Network activity[3] on this node is far higher than the 
rest.

The only other difference this node has to the rest of the cluster is that its 
on the ext4 filesystem, whereas the rest are ext3, but we've done plenty of 
testing there and can't see how that would affect performance on this node so 
much.

Nothing of note in system.log.

What should our next step be in trying to diagnose this issue?

Best wishes,
Vic

[0] `nodetool tpstats` output:

Good node:
    Pool Name    Active   Pending  Completed   Blocked  All 
time blocked
    ReadStage 0 0   46311521 0  
   0
    RequestResponseStage  0 0   23817366 0  
   0
    MutationStage 0 0   47389269 0  
   0
    ReadRepairStage   0 0  11108 0  
   0
    ReplicateOnWriteStage 0 0  0 0  
   0
    GossipStage   0 0    5259908 0  
   0
    CacheCleanupExecutor  0 0  0 0  
   0
    MigrationStage    0 0 30 0  
   0
    MemoryMeter   0 0  16563 0  
   0
    FlushWriter   0 0  39637 0  
  26
    ValidationExecutor    0 0  19013 0  
   0
    InternalResponseStage 0 0  9 0  
   0
    AntiEntropyStage  0 0  38026 0  
   0
    MemtablePostFlusher   0 0  81740 0  
   0
    MiscStage 0 0  19196 0  
   0
    PendingRangeCalculator    0 0 23 0  
   0
    CompactionExecutor    0 0  61629 0  
   0
    commitlog_archiver    0 0  0 0  
   0
    HintedHandoff 0 0 63 0  
   0

    Message type   Dropped
    RANGE_SLICE  0
    READ_REPAIR  0
    PAGED_RANGE  0
    BINARY   0
    READ   640
    

Re: Cassandra Performance on a Single Machine

[John Schulz]

Anurag,

Unless you are planning on continuing to use only one machine with RF=1
benchmarking a single system using RF=Consistancy=1 is mostly a waste of
time. If you are going to use RF=1 and a single host then why use Cassandra
at all. Plain old relational dbs should do the job just fine.

Cassandra is designed to be distributed. You won't get the full impact of
how it scales and the limits on scaling unless you benchmark a distributed
system. For example the scaling impact of secondary indexes will not be
visible on a single node.

John




On Tue, Jan 5, 2016 at 3:16 PM, Anurag Khandelwal 
wrote:

> Hi,
>
> I’ve been benchmarking Cassandra to get an idea of how the performance
> scales with more data on a single machine. I just wanted to get some
> feedback to whether these are the numbers I should expect.
>
> The benchmarks are quite simple — I measure the latency and throughput for
> two kinds of queries:
>
> 1. get() queries - These fetch an entire row for a given primary key.
> 2. search() queries - These fetch all the primary keys for rows where a
> particular column matches a particular value (e.g., “name” is “John
> Smith”).
>
> Indexes are constructed for all columns that are queried.
>
> *Dataset*
>
> The dataset used comprises of ~1.5KB records (on an average) when
> represented as CSV; there are 105 attributes in each record.
>
> *Queries*
>
> For get() queries, randomly generated primary keys are used.
>
> For search() queries, column values are selected such that their total
> number of occurrences in the dataset is between 1 - 4000. For example, a
> query for  “name” = “John Smith” would only be performed if the number of
> rows that contain the same lies between 1-4000.
>
> The results for the benchmarks are provided below:
>
> *Latency Measurements*
>
> The latency measurements are an average of 1 queries.
>
>
>
>
>
> *Throughput Measurements*
>
> The throughput measurements were repeated for 1-16 client threads, and the
> numbers reported for each input size is for the configuration (i.e., #
> client threads) with the highest throughput.
>
>
>
>
>
> Any feedback here would be greatly appreciated!
>
> Thanks!
> Anurag
>
>


-- 

John H. Schulz

Principal Consultant

Pythian - Love your data


sch...@pythian.com |  Linkedin www.linkedin.com/pub/john-schulz/13/ab2/930/

Mobile: 248-376-3380

*www.pythian.com *

-- 


--

Re: opscenter doesn't work with cassandra 3.0

[Michael Shuler]

On 01/06/2016 01:47 AM, Wills Feng wrote:
> Looks like opscenter doesn't support cassandra 3.0?

This is correct. OpsCenter does not support Cassandra >= 3.0.

-- 
Michael

Re: opscenter doesn't work with cassandra 3.0

[Michael Shuler]

On 01/06/2016 10:55 AM, Michael Shuler wrote:
> On 01/06/2016 01:47 AM, Wills Feng wrote:
>> Looks like opscenter doesn't support cassandra 3.0?
> 
> This is correct. OpsCenter does not support Cassandra >= 3.0.

It took me a minute to find the correct document:

http://docs.datastax.com/en/upgrade/doc/upgrade/opscenter/opscCompatibility.html

According to this version table, OpsCenter does not officially support
Cassandra > 2.1.

-- 
Michael

Re: New node has high network and disk usage.

[Vickrum Loi]

# nodetool compactionstats
pending tasks: 22
  compaction typekeyspace   table
completed   total  unit  progress
   Compactionproduction_analyticsinteractions
240410213161172668724 bytes 0.15%

Compactionproduction_decisionsdecisions.decisions_q_idx
120815385   226295183 bytes53.39%
Active compaction remaining time :   2h39m58s

Worth mentioning that compactions haven't been running on this node
particularly often. The node's been performing badly regardless of whether
it's compacting or not.

On 6 January 2016 at 16:35, Jeff Ferland  wrote:

> What’s your output of `nodetool compactionstats`?
>
> On Jan 6, 2016, at 7:26 AM, Vickrum Loi 
> wrote:
>
> Hi,
>
> We recently added a new node to our cluster in order to replace a node
> that died (hardware failure we believe). For the next two weeks it had high
> disk and network activity. We replaced the server, but it's happened again.
> We've looked into memory allowances, disk performance, number of
> connections, and all the nodetool stats, but can't find the cause of the
> issue.
>
> `nodetool tpstats`[0] shows a lot of active and pending threads, in
> comparison to the rest of the cluster, but that's likely a symptom, not a
> cause.
>
> `nodetool status`[1] shows the cluster isn't quite balanced. The bad node
> (D) has less data.
>
> Disk Activity[2] and Network activity[3] on this node is far higher than
> the rest.
>
> The only other difference this node has to the rest of the cluster is that
> its on the ext4 filesystem, whereas the rest are ext3, but we've done
> plenty of testing there and can't see how that would affect performance on
> this node so much.
>
> Nothing of note in system.log.
>
> What should our next step be in trying to diagnose this issue?
>
> Best wishes,
> Vic
>
> [0] `nodetool tpstats` output:
>
> Good node:
> Pool NameActive   Pending  Completed
> Blocked  All time blocked
> ReadStage 0 0   46311521
> 0 0
> RequestResponseStage  0 0   23817366
> 0 0
> MutationStage 0 0   47389269
> 0 0
> ReadRepairStage   0 0  11108
> 0 0
> ReplicateOnWriteStage 0 0  0
> 0 0
> GossipStage   0 05259908
> 0 0
> CacheCleanupExecutor  0 0  0
> 0 0
> MigrationStage0 0 30
> 0 0
> MemoryMeter   0 0  16563
> 0 0
> FlushWriter   0 0  39637
> 026
> ValidationExecutor0 0  19013
> 0 0
> InternalResponseStage 0 0  9
> 0 0
> AntiEntropyStage  0 0  38026
> 0 0
> MemtablePostFlusher   0 0  81740
> 0 0
> MiscStage 0 0  19196
> 0 0
> PendingRangeCalculator0 0 23
> 0 0
> CompactionExecutor0 0  61629
> 0 0
> commitlog_archiver0 0  0
> 0 0
> HintedHandoff 0 0 63
> 0 0
>
> Message type   Dropped
> RANGE_SLICE  0
> READ_REPAIR  0
> PAGED_RANGE  0
> BINARY   0
> READ   640
> MUTATION 0
> _TRACE   0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
>
> Bad node:
> Pool NameActive   Pending  Completed
> Blocked  All time blocked
> ReadStage32   113  52216
> 0 0
> RequestResponseStage  0 0   4167
> 0 0
> MutationStage 0 0 127559
> 0 0
> ReadRepairStage   0 0125
> 0 0
> ReplicateOnWriteStage 0 0  0
> 0 0
> GossipStage   0 0   9965
> 0 0
> CacheCleanupExecutor  0 0  0
> 0 0
> MigrationStage0 0  0
> 0 0
> MemoryMeter   0 0 24
> 0

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Peddi, Praveen]

Hi Rob,
I want to make sure I am not missing anything on my side before creating a jira 
ticket with Cassandra. I will wait for others to respond before filing a 
ticket. May be somebody can give me a clue of what might be going wrong.

This blog 
claims Cassandra is now 50% faster. We are obviously not seeing that.

Praveen

From: Robert Coli >
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, January 6, 2016 at 3:49 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11

On Wed, Jan 6, 2016 at 11:49 AM, Peddi, Praveen 
> wrote:
2nd column is replication factor (RF). I have 2 rows for reads and 2 for 
writes. First row is RF=1 and 2nd row is RF=3. So when I said increasing RF , I 
meant from 1 to 3. Sorry the table is probably not clear.

Ah, I see now, I was mis-aligning visually.

With RF=1, QUORUM is ONE.
With RF=3, QUORUM is .

I agree that your results look suspicious. I would probably file a Jira ticket 
if I were you, and let the list know what the id is.

=Rob

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Michael Shuler]

On 01/06/2016 03:57 PM, Peddi, Praveen wrote:
> This blog
> 
> claims Cassandra is now 50% faster. We are obviously not seeing that.

That post compared cassandra-stress write on versions 2.0 and 2.1, each
on a single AWS c3.8xlarge instance, so just be sure to compare apples
to apples in observations.

-- 
Kind regards,
Michael

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Peddi, Praveen]

Hi Michael,
I am not comparing my results with results on that page. I mentioned only in 
the context of improved performance in 2.1 compared to 2.0.

That page compares performance between 2.0 and 2.1 with same hardware. I am 
doing the same exact thing (running 2.0.9 and 2.1.11 on same hardware, same 
load test, same datastax driver code etc) and expected same or better 
performance.

BTW, I just came across an issue with datastax driver 
https://datastax-oss.atlassian.net/plugins/servlet/mobile#issue/JAVA-803. Not 
sure if we are hitting this but we are going to test with 2.1.5 driver and see 
if it's any better.

Praveen

On Jan 6, 2016, at 6:07 PM, Michael Shuler 
> wrote:

On 01/06/2016 03:57 PM, Peddi, Praveen wrote:
This blog

claims Cassandra is now 50% faster. We are obviously not seeing that.

That post compared cassandra-stress write on versions 2.0 and 2.1, each
on a single AWS c3.8xlarge instance, so just be sure to compare apples
to apples in observations.

--
Kind regards,
Michael

Re: Revisit Cassandra EOL Policy

[Anuj Wadehra]

I would appreciate if you guys share your thoughts on the concerns I expressed 
regarding Cassandra End of Life policy. I think these concerns are quite 
genuine and should be openly discussed so that EOL is more predictable and 
generates less overhead for the users.
I would like to understand how various users are dealing with the situation. 
Are you upgrading Cassandra every 3-6 mths? How do you cut short your 
planning,test and release cycles for Cassandra upgrades in your 
application/products?



ThanksAnuj


 
 
  On Tue, 5 Jan, 2016 at 8:04 pm, Anuj Wadehra wrote:   
Hi,
As per my understanding, a Cassandra version n is implicitly declared EOL when 
two major versions are released after the version n i.e. when version n + 2 is 
released.
I think the EOL policy must be revisted in interest of the expanding Cassandra 
user base. 
Concerns with current EOL Policy:
In March 2015, Apache web site mentioned that 2.0.14 is the most stable version 
of the Cassandra recommended for Production. So, one would push its clients to 
upgrade to 2.0.14 in Mar 2015. It takes months to roll out a Cassandra upgrade 
to all your clients and by the time all your clients get the upgrade, the 
version is declared EOL with the release of 2.2 in Aug 2015 (within 6 mths of 
being declared production ready). I completely understand that supporting 
multiple versions is tougher but at the same time it is very painful and 
somewhat unrealistic for users to push Cassandra upgrades to all thier clients 
after every few months.
One proposed solution could be to declare a version n as EOL one year after n+1 
was declared Production Ready. E.g. if 2.1.7 is the first production ready 
release of 2.1 which is released in Jun 2015, I would declare 2.0 EOL in Jun 
2016. This gives reasonable time for users to plan upgrades.
Moreover, I think the EOL policy and declarations must be documented explicitly 
on Apache web site.
Please share your feedback on revisting the EOL policy.
ThanksAnuj
  

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Jeff Jirsa]

Anecdotal evidence typically agrees that 2.1 is faster than 2.0 (our experience 
was anywhere from 20-60%, depending on workload).

However, it’s not necessarily true that everything behaves exactly the same – 
in particular, memtables are different, commitlog segment handling is 
different, and GC params may need to be tuned differently for 2.1 than 2.0.

When the system is busy, what’s it actually DOING? Cassandra exposes a TON of 
metrics – have you plugged any into a reporting system to see what’s going on? 
Is your latency due to pegged cpu, iowait/disk queues or gc pauses? 

My colleagues spent a lot of time validating different AWS EBS configs (video 
from reinvent at https://www.youtube.com/watch?v=1R-mgOcOSd4), 2.1 was faster 
in almost every case, but you’re using an instance size I don’t believe we 
tried (too little RAM to be viable in production).  c3.2xl only gives you 15G 
of ram – most “performance” based systems want 2-4x that (people running G1 
heaps usually start at 16G heaps and leave another 16-30G for page cache), 
you’re running fairly small hardware – it’s possible that 2.1 isn’t “as good” 
on smaller hardware. 

(I do see your domain, presumably you know all of this, but just to be sure):

You’re using c3, so presumably you’re using EBS – are you using GP2? Which 
volume sizes? Are they the same between versions? Are you hitting your iops 
limits? Running out of burst tokens? Do you have enhanced networking enabled? 
At load, what part of your system is stressed? Are you cpu bound? Are you 
seeing GC pauses hurt latency? Have you tried changing memtable_allocation_type 
-> offheap objects  (available in 2.1, not in 2.0)? 

Tuning gc_grace is weird – do you understand what it does? Are you overwriting 
or deleting a lot of data in your test (that’d be unusual)? Are you doing a lot 
of compaction?


From:  "Peddi, Praveen"
Reply-To:  "user@cassandra.apache.org"
Date:  Wednesday, January 6, 2016 at 11:41 AM
To:  "user@cassandra.apache.org"
Subject:  Slow performance after upgrading from 2.0.9 to 2.1.11

Hi,
We have upgraded Cassandra from 2.0.9 to 2.1.11 in our loadtest environment 
with pretty much same yaml settings in both (removed unused yaml settings and 
renamed few others) and we have noticed performance on 2.1.11 is worse compared 
to 2.0.9. After more investigation we found that the performance gets worse as 
we increase replication factor on 2.1.11 where as on 2.0.9 performance is more 
or less same. Has anything architecturally changed as far as replication is 
concerned in 2.1.11?

All googling only suggested 2.1.11 should be FASTER than 2.0.9 so we are 
obviously doing something different. However the client code, load test is all 
identical in both cases.

Details:
Nodes: 3 ec2 c3.2x large
R/W Consistency: QUORUM
Renamed memtable_total_space_in_mb to memtable_heap_space_in_mb and removed 
unused properties from yaml file.
We run compaction aggressive compaction with low gc_grace (15 mins) but this is 
true for both 2.0.9 and 2.1.11.

As you can see, all p50, p90 and p99 latencies stayed with in 10% difference on 
2.0.9 when we increased RF from 1 to 3, where as on 2.1.11 latencies almost 
doubled (especially reads are much slower than writes).

# Nodes RF# of rows2.0.92.1.11
READ
   P50P90P99P50P90P99
314503065947474258491085
3345035863487770812742642
 
WRITE
3110268017937131196
3310319618446166468
Any pointers on how to debug performance issues will be appreciated.

Praveen



smime.p7s
Description: S/MIME cryptographic signature

confusion about migrating to incremental repair

[Kai Wang]

Hi,

I am running a cluster with 2.2.4. I have some table on LCS and plan to use
incremental repair. I read the post at
http://www.datastax.com/dev/blog/anticompaction-in-cassandra-2-1 and am a
little confused.

especially:
"This means that *once you do an incremental repair you will have to
continue doing them* (there are ways to clear out the repair-state to
revert this, more about that later). Otherwise you will not run leveled
compaction, just size tiered."

Does this mean once I start doing inc repair, I have to always run inc
repair to keep using LCS? What about full repair once a week or month? Is
that still recommended? Will it stop LCS?

Re: Re: Fail to export all data in C* cluster

[Jack Krupansky]

Cassandra does not have any automatic rollback of partial work for a failed
write request, if with CL=ALL - that just assures that you will get an
error indication if not all the writes succeed. You do need to keep
retrying failed writes until they succeed. Otherwise the partial work will
still be in the database until you run repair. Note my step 1 specified the
need for retries.

If you are getting just a handful of failures and retries, that's probably
okay, but if you are getting tons of failures then some manual throttling
(sleep between requests) may be the better approach, but even then your app
should be prepared for occasional failures, like due to Java GC or network
latency or whatever.

If you can do the retries at CL=ALL until they succeed, then you should be
set and not need repair.

And as I set in approach #3, if you throttle back enough then you can
probably avoid the need for CL=ALL.


-- Jack Krupansky

On Tue, Jan 5, 2016 at 2:43 AM, xutom  wrote:

> Dear Jack,
> Thanks!
> My keyspace is such as:
> test@cqlsh> DESC KEYSPACE sky ;
> CREATE KEYSPACE sky WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'}  AND durable_writes = true;
> CREATE TABLE sky.user1 (pati int, uuid text, name text,  name2 text,
> PRIMARY KEY (pati, uuid))
>
> Now I am using CL=ALL during inserting and set the retry policy by
> such following codes:
> RetryPolicy rp = new CustomRetryPolicy(3, 3, 2);
> Cluster cluster =
> Cluster.builder().addContactPoint(seedIp).withCredentials(
> "test", "test")
> .withRetryPolicy(rp)
> .withLoadBalancingPolicy(
> new TokenAwarePolicy(new
> DCAwareRoundRobinPolicy()))  /*Here My Cluster has 6 nodes, all in the
> same DataCenter*/
> .build();
> PreparedStatement insertStatement = session
> .prepare("INSERT INTO  " + tableName
> + "(" + columns + ") "
> + "VALUES (?, ?, ?, ?);");
> insertStatement.setConsistencyLevel(ConsistencyLevel.ALL); /* Here I set
> the CL to ALL */
> I start 30 threads to insert datas, each thread uses BatchStatement to
> insert 100 rows with the same partition key but different primary key every
> time, and run 10 times. After Inserting about 99084500 rows into C*
> cluster with many timeout exceptions I stop the inserting progress and then
> use following codes to export all the datas into local file:
> String cqlstr = " select * from " + this.tableName
> + " where pati = " + this.partition[i];
> PreparedStatement Statement = session
> .prepare(cqlstr);
> BoundStatement bStatement = new BoundStatement(
> Statement);
> bStatement.setFetchSize(1);
> iter = session.execute(bStatement).iterator();
> then I write the results( in iter) to localfile. I run  3 times and all
> three results are different。
>
> I have set the CL to ALL when inserting datas, but why I get the
> different results when I export all datas everytime?
> By the way, I have set :  "hinted_handoff_enabled: true", is it the
> problem when the C* cluster is overloaded even if I have set the CL to ALL?
>
> Best Regrads
> jerry
>
> At 2016-01-04 23:37:20, "Jack Krupansky"  wrote:
>
> You have three choices:
>
> 1. Insert with CL=ALL, with client-level retries if the write fails due to
> the cluster being overloaded.
> 2. Insert with CL=QUORUM and then run repair after all data has been
> inserted.
> 3. Lower your insert rate in your client so that the cluster can keep up
> with your inserts.
>
> Yes, Cassandra supports eventual consistency, but if you overload the
> cluster, the hinted handoff for nodes beyond the requested CL may timeout
> and be discarded, hence the need for repair.
>
> What CL are you currently using for inserts?
>
>
> -- Jack Krupansky
>
> On Mon, Jan 4, 2016 at 9:52 AM, xutom  wrote:
>
>> Hi all,
>>
>> I have a C* cluster with 6 nodes. My cassandra version is 2.1.1. I
>> start 50 threads to insert datas into C* cluster, each thread inserts about
>> up to 100 million rows with the same partition key. After inserting all the
>> datas, I start another app with 50 threads to export all the datas into
>> localfile, I using such cqlsh: select * from table where
>> partition_id=xxx(each partition has about 100 million rows). But 
>> unfortunately
>> I fail to export all the datas: I run 3 times, and each time I get the
>> different number of results. If I successfully export all datas, everytime
>> I should get the same number of results, is it right?
>>
>> Best Regards
>>
>>
>>
>>
>
>
>
>
>

Slow performance after upgrading from 2.0.9 to 2.1.11

[Peddi, Praveen]

Hi,
We have upgraded Cassandra from 2.0.9 to 2.1.11 in our loadtest environment 
with pretty much same yaml settings in both (removed unused yaml settings and 
renamed few others) and we have noticed performance on 2.1.11 is worse compared 
to 2.0.9. After more investigation we found that the performance gets worse as 
we increase replication factor on 2.1.11 where as on 2.0.9 performance is more 
or less same. Has anything architecturally changed as far as replication is 
concerned in 2.1.11?

All googling only suggested 2.1.11 should be FASTER than 2.0.9 so we are 
obviously doing something different. However the client code, load test is all 
identical in both cases.

Details:
Nodes: 3 ec2 c3.2x large
R/W Consistency: QUORUM
Renamed memtable_total_space_in_mb to memtable_heap_space_in_mb and removed 
unused properties from yaml file.
We run compaction aggressive compaction with low gc_grace (15 mins) but this is 
true for both 2.0.9 and 2.1.11.

As you can see, all p50, p90 and p99 latencies stayed with in 10% difference on 
2.0.9 when we increased RF from 1 to 3, where as on 2.1.11 latencies almost 
doubled (especially reads are much slower than writes).

# Nodes RF  # of rows   2.0.9   2.1.11
READ
P50 P90 P99 P50 P90 P99
3   1   450 306 594 747 425 849 1085
3   3   450 358 634 877 708 12742642

WRITE
3   1   10  26  80  179 37  131 196
3   3   10  31  96  184 46  166 468

Any pointers on how to debug performance issues will be appreciated.

Praveen

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Robert Coli]

On Wed, Jan 6, 2016 at 11:41 AM, Peddi, Praveen  wrote:

> We have upgraded Cassandra from 2.0.9 to 2.1.11 in our loadtest
> environment with pretty much same yaml settings in both (removed unused
> yaml settings and renamed few others) and we have noticed performance on
> 2.1.11 is worse compared to 2.0.9. *After more investigation we found
> that the performance gets worse as we increase replication factor on 2.1.11
> where as on 2.0.9 performance is more or less same.* Has anything
> architecturally changed as far as replication is concerned in 2.1.11?
>

What does "as we increase replication factor" mean in your sentence?

The only value I see for replication factor in the rest of your mail is 3?

=Rob

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Peddi, Praveen]

2nd column is replication factor (RF). I have 2 rows for reads and 2 for 
writes. First row is RF=1 and 2nd row is RF=3. So when I said increasing RF , I 
meant from 1 to 3. Sorry the table is probably not clear.

Praveen








From: Robert Coli >
Reply-To: "user@cassandra.apache.org" 
>
Date: Wednesday, January 6, 2016 at 2:44 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11

On Wed, Jan 6, 2016 at 11:41 AM, Peddi, Praveen 
> wrote:
We have upgraded Cassandra from 2.0.9 to 2.1.11 in our loadtest environment 
with pretty much same yaml settings in both (removed unused yaml settings and 
renamed few others) and we have noticed performance on 2.1.11 is worse compared 
to 2.0.9. After more investigation we found that the performance gets worse as 
we increase replication factor on 2.1.11 where as on 2.0.9 performance is more 
or less same. Has anything architecturally changed as far as replication is 
concerned in 2.1.11?

What does "as we increase replication factor" mean in your sentence?

The only value I see for replication factor in the rest of your mail is 3?

=Rob

Re: Formal EOL for 2.0.17 and 2.1.12

[Jack Krupansky]

Back in June, Jonathan posted: "After 2.2.0 is released, 2.0 will reach
end-of-life as planned. After 3.0.0 is released, 2.1 will also reach end of
life.  This is earlier than expected, but 2.2 will be very close
to as stable as 2.1 and users will be well served by upgrading.  We will
maintain the 2.2 stability series until 4.0 is released, and 3.0 for six
months after that."

I can't recall seeing any more formal articulation of EOL/EOSL for
Cassandra in recent years, other than the DataStax DSE support policy:
http://www.datastax.com/support-policy/supported-software
http://www.datastax.com/support-policy

For DataStax Cerified Cassandra releases there is this stated policy:
"End of Life Support - Certified Cassandra customers receive full
commercial software lifecycle support that includes both certified software
updates and expert support for earlier versions of Certified Cassandra that
they still have deployed in production." But that is a little vague and has
no hard date or time intervals. I mean, it sounds as if the mere fact that
you have it in production means DataStax will support it forever. See:
http://www.datastax.com/products/datastax-enterprise-production-certified-cassandra

Cassandra 2.0.17 is in DSE 4.6.x which has a stated EOL of June 2, 2016 and
stated EOSL of December 2, 2016.
Cassandra 2.1.11 (not 2.1.12) is in DSE 4.8.x (and 4.7) which has a stated
EOL of March 23, 2017 and stated EOSL of September 23, 2017.

If you really care about 2.1.12 vs. 2.1.11, DataStax tends to include
additional bug fixes if they seem critical. That list is here:
https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/RNcassChanges.html?scroll=RNcassFixes__483_unique_1


-- Jack Krupansky

On Wed, Jan 6, 2016 at 12:41 PM, Anuj Wadehra 
wrote:

> Hi,
>
> Can someone help me by providing formal dates of EOL for Cassandra 2.0.17
> and 2.1.12?
>
>
> Thanks
> Anuj
>
>
>
>
> Sent from Yahoo Mail on Android
> 
>

Re: CQL Composite Key Seen After Table Creation

[Robert Coli]

On Wed, Jan 6, 2016 at 12:54 PM, Chris Burroughs 
wrote:

> I work with Amir and further experimentation I can shed a little more
> light on what exactly is going on under the hood.  For background our goal
> is to take data that is currently being read and written to via thrift,
> switch reads to CQL, and then switch writes to CQL.  This is in alternative
> to deleting all of our data and starting over, or being forever struck on
> super old thrift clients (both of those options obviously suck.)  The data
> models involved are absurdly simple (and single key with a handful of
> static columns).

 ...

> The problem with that approach is that manually editing the local schema
> tables in live cluster is wildly dangerous. I *think* this would work:

 * Make triple sure no schema changes are happening on the cluster.

 * Update schema tables on each node --> drain --> restart


I think that would work too, and probably be lower risk than modifying on
one and trying to get the others to pull via resetlocalschema. But I agree
it seems "wildly dangerous".

FWIW, I think your case may be the case the project hopes to handle in
https://issues.apache.org/jira/browse/CASSANDRA-10857 ?

=Rob

Re: Slow performance after upgrading from 2.0.9 to 2.1.11

[Robert Coli]

On Wed, Jan 6, 2016 at 11:49 AM, Peddi, Praveen  wrote:

> 2nd column is replication factor (RF). I have 2 rows for reads and 2 for
> writes. First row is RF=1 and 2nd row is RF=3. So when I said increasing RF
> , I meant from 1 to 3. Sorry the table is probably not clear.
>

Ah, I see now, I was mis-aligning visually.

With RF=1, QUORUM is ONE.
With RF=3, QUORUM is .

I agree that your results look suspicious. I would probably file a Jira
ticket if I were you, and let the list know what the id is.

=Rob

Re: CQL Composite Key Seen After Table Creation

[Chris Burroughs]

I work with Amir and further experimentation I can shed a little more light on 
what exactly is going on under the hood.  For background our goal is to take 
data that is currently being read and written to via thrift, switch reads to 
CQL, and then switch writes to CQL.  This is in alternative to deleting all of 
our data and starting over, or being forever struck on super old thrift clients 
(both of those options obviously suck.)  The data models involved are absurdly 
simple (and single key with a handful of static columns).

TLDR: Metadata is complicated.  What is the least dangerous way to make direct 
changes to system.schema_columnfamilies and system.schema_columns?

Anyway, given some super simple Foo and Bar column families:

create keyspace Test with  placement_strategy = 
'org.apache.cassandra.locator.SimpleStrategy' and strategy_options = 
{replication_factor:1};
use Test;
create column family Foo with comparator = UTF8Type and 
key_validation_class=UTF8Type and column_metadata = [ {column_name: title, 
validation_class: UTF8Type}];
create column family Bar with comparator = UTF8Type and 
key_validation_class=UTF8Type;
update column family Bar with column_metadata = [ {column_name: title, 
validation_class: UTF8Type}];

(The salient difference as described by Amir is when the column_metadata is 
set; at the same time as creation or later.)

Now we can inject a little data and see that from thrift everything looks fine:

[default@Test] set Foo['testkey']['title']='mytitle';
Value inserted.
Elapsed time: 19 msec(s).
[default@Test] set Bar['testkey']['title']='mytitle';
Value inserted.
Elapsed time: 4.47 msec(s).

[default@Test] list Foo;
Using default limit of 100
Using default cell limit of 100
---
RowKey: testkey
=> (name=title, value=mytitle, timestamp=1452108082972000)

1 Row Returned.
Elapsed time: 268 msec(s).
[default@Test] list Bar;
Using default limit of 100
Using default cell limit of 100
---
RowKey: testkey
=> (name=title, value=mytitle, timestamp=1452108093739000)

1 Row Returned.
Elapsed time: 9.3 msec(s).

But from cql the Bar column does not look like the data we wrote:

cqlsh> select * from "Test"."Foo";

 key | title
-+-
 testkey | mytitle

(1 rows)


cqlsh> select * from "Test"."Bar";

 key | column1 | value| title
-+-+--+-
 testkey |   title | 0x6d797469746c65 | mytitle


It's not just that these phantom columns are ugly, cql thinks column1 is part 
of a composite primary key.  Since there **is no column1**, that renderes the 
data un-query-able with WHERE clauses.

Just to make sure it's not thrift that is doing something unexpected, the 
sstables show the expected structure:

$ ./tools/bin/sstable2json 
/data/sstables/data/Test/Foo-d3348860b4af11e5b456639406f48f1b/Test-Foo-ka-1-Data.db
 
[
{"key": "testkey",
 "cells": [["title","mytitle",1452110466924000]]}
]


$ ./tools/bin/sstable2json 
/data/sstables/data/Test/Foo-d3348860b4af11e5b456639406f48f1b/Test-Foo-ka-1-Data.db
 
[
{"key": "testkey",
 "cells": [["title","mytitle",1452110466924000]]}
]


So, what appeared as innocent variation made years ago when the thrift schema 
was written causes very different results to cql.

Digging into the schema tables shows what is going on in more detail:

> select 
> keyspace_name,columnfamily_name,column_aliases,comparator,is_dense,key_aliases,value_alias
>  from system.schema_columnfamilies where keyspace_name='Test';

 keyspace_name | columnfamily_name | column_aliases | comparator
  | is_dense | key_aliases | value_alias
---+---++ 
+--+-+-
  Test |   Bar | ["column1"]   | 
org.apache.cassandra.db.marshal.UTF8Type | True | ["key"] |   value
  Test |   Foo |  []   | 
org.apache.cassandra.db.marshal.UTF8Type |False | ["key"] |null

> select keyspace_name,columnfamily_name,column_name,validator from 
> system.schema_columns where keyspace_name='Test';

 keyspace_name | columnfamily_name | column_name | validator
---+---+-+---
  Test |   Bar | column1 |  
org.apache.cassandra.db.marshal.UTF8Type
  Test |   Bar | key |  
org.apache.cassandra.db.marshal.UTF8Type
  Test |   Bar |   title |  
org.apache.cassandra.db.marshal.UTF8Type
  Test |   Bar |   value | 
org.apache.cassandra.db.marshal.BytesType
  Test |   Foo | key |  
org.apache.cassandra.db.marshal.UTF8Type
  Test |   Foo |   title |  
org.apache.cassandra.db.marshal.UTF8Type


Now the interesting bit is that the metadata can  be manually "fixed":

UPDATE