Re: 1, 2, 3...

[Jeff Jirsa]

SELECT COUNT(*) probably works (with internal paging) on many datasets with 
enough time and assuming you don’t have any partitions that will kill you.

No, it doesn’t count extra replicas / duplicates.

The old way to do this (before paging / fetch size) was to use manual paging 
based on tokens/clustering keys:

https://docs.datastax.com/en/cql/3.1/cql/cql_using/paging_c.html – SELECT’s 
WHERE clause can use token(), which is what you’d want to use to page through 
the whole token space. 

You could, in theory, issue thousands of queries in parallel, all for different 
token ranges, and then sum the results. That’s what something like spark would 
be doing. If you want to determine rows per node, limit the token range to that 
owned by the node (easier with 1 token than vnodes, with vnodes repeat 
num_tokens times).



From:  Jack Krupansky
Reply-To:  "user@cassandra.apache.org"
Date:  Friday, April 8, 2016 at 3:48 PM
To:  "user@cassandra.apache.org"
Subject:  1, 2, 3...

I'm afraid I don't have the solid answer to this obvious question: How do I get 
a fairly accurate count of (CQL) rows in a Cassandra table? 

Does SELECT COUNT (*) FROM  actually do it?

Does it really count (CQL) rows across all nodes and exclude replicated rows?

Is there a better/preferred technique? For example, is it more efficient to 
query the row count one node at a time?

And for bonus points: How do you count (CQL) rows for each node? Again, 
excluding replication.

-- Jack Krupansky



smime.p7s
Description: S/MIME cryptographic signature

Re: Efficiently filtering results directly in CS

[kurt Greaves]

If you're using C* 3.0 you can probably achieve this with UDFs.
http://www.planetcassandra.org/blog/user-defined-functions-in-cassandra-3-0/

On 9 April 2016 at 00:22, Kevin Burton  wrote:

> Ha..  Yes... C*...  I guess I need something like coprocessors in
> bigtable.
>
> On Fri, Apr 8, 2016 at 1:49 AM, vincent gromakowski <
> vincent.gromakow...@gmail.com> wrote:
>
>> c* I suppose
>>
>> 2016-04-07 19:30 GMT+02:00 Jonathan Haddad :
>>
>>> What is CS?
>>>
>>> On Thu, Apr 7, 2016 at 10:03 AM Kevin Burton  wrote:
>>>
 I have a paging model whereby we stream data from CS by fetching
 'pages' thereby reading (sequentially) entire datasets.

 We're using the bucket approach where we write data for 5 minutes, then
 we can just fetch the bucket for that range.

 Our app now has TONS of data and we have a piece of middleware that
 filters it based on the client requests.

 So if they only want english they just get english and filter away
 about 60% of our data.

 but it doesn't support condition pushdown.  So ALL this data has to be
 sent from our CS boxes to our middleware and filtered there (wasting a lot
 of network IO).

 Is there away (including refactoring the code) that I could push this
 this into CS?  Maybe some way I could discovery the CS topology and put
 daemons on each of our CS boxes and fetch from CS directly (doing the
 filtering there).

 Thoughts?

 --

 We’re hiring if you know of any awesome Java Devops or Linux Operations
 Engineers!

 Founder/CEO Spinn3r.com
 Location: *San Francisco, CA*
 blog: http://burtonator.wordpress.com
 … or check out my Google+ profile
 


>>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>


-- 
Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

Re: Efficiently filtering results directly in CS

[Kevin Burton]

Ha..  Yes... C*...  I guess I need something like coprocessors in bigtable.


On Fri, Apr 8, 2016 at 1:49 AM, vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> c* I suppose
>
> 2016-04-07 19:30 GMT+02:00 Jonathan Haddad :
>
>> What is CS?
>>
>> On Thu, Apr 7, 2016 at 10:03 AM Kevin Burton  wrote:
>>
>>> I have a paging model whereby we stream data from CS by fetching 'pages'
>>> thereby reading (sequentially) entire datasets.
>>>
>>> We're using the bucket approach where we write data for 5 minutes, then
>>> we can just fetch the bucket for that range.
>>>
>>> Our app now has TONS of data and we have a piece of middleware that
>>> filters it based on the client requests.
>>>
>>> So if they only want english they just get english and filter away about
>>> 60% of our data.
>>>
>>> but it doesn't support condition pushdown.  So ALL this data has to be
>>> sent from our CS boxes to our middleware and filtered there (wasting a lot
>>> of network IO).
>>>
>>> Is there away (including refactoring the code) that I could push this
>>> this into CS?  Maybe some way I could discovery the CS topology and put
>>> daemons on each of our CS boxes and fetch from CS directly (doing the
>>> filtering there).
>>>
>>> Thoughts?
>>>
>>> --
>>>
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> 
>>>
>>>
>


-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

1, 2, 3...

[Jack Krupansky]

I'm afraid I don't have the solid answer to this obvious question: How do I
get a fairly accurate count of (CQL) rows in a Cassandra table?

Does SELECT COUNT (*) FROM  actually do it?

Does it really count (CQL) rows across all nodes and exclude replicated
rows?

Is there a better/preferred technique? For example, is it more efficient to
query the row count one node at a time?

And for bonus points: How do you count (CQL) rows for each node? Again,
excluding replication.

-- Jack Krupansky

consistency level ALL?

[Henry M]

I was wondering if there are users in this list using consistency level ALL
and their reasons for doing so?

For example, would the errors for deleting a financial transaction due to
an error be reason enough to use consistency level of ALL? Are there other
strategies people would use to avoid inconsistent/stale results which could
contain the deleted financial transaction?

Thank you,
Henry

Re: disable compaction if all data are read-only?

[Robert Wille]

You still need compaction. Compaction is what organizes your data into levels. 
Without compaction, every query would have to look at every SSTable.

Also, due to commit log rotation, your memtable may get flushed from time to 
time before it is full, resulting in small SSTables that would benefit from 
compaction.

On Apr 8, 2016, at 5:49 AM, Yatong Zhang 
> wrote:

I am using leveled strategy. What if my data are 'append-only'? I mean there 
are always new data but will be never changed once written to cassandra?

On Fri, Apr 8, 2016 at 6:33 PM, Pedro Gordo 
> wrote:
Hi Yatong

My understanding is that if you have a table whichi read-only and hence doesn't 
receive any writes, then no SSTables will be created, and hence, no compaction 
will happen. What compaction strategy do you have on your table?

Best regards

Pedro Gordo

On 8 April 2016 at 10:42, Yatong Zhang 
> wrote:
Hi there,
I am wondering if it is possible to disable compaction when all my data are 
read-only?

Re: disable compaction if all data are read-only?

[Yatong Zhang]

I am using leveled strategy. What if my data are 'append-only'? I mean
there are always new data but will be never changed once written to
cassandra?

On Fri, Apr 8, 2016 at 6:33 PM, Pedro Gordo 
wrote:

> Hi Yatong
>
> My understanding is that if you have a table whichi read-only and hence
> doesn't receive any writes, then no SSTables will be created, and hence, no
> compaction will happen. What compaction strategy do you have on your table?
>
> Best regards
>
> Pedro Gordo
>
> On 8 April 2016 at 10:42, Yatong Zhang  wrote:
>
>> Hi there,
>> I am wondering if it is possible to disable compaction when all my data
>> are read-only?
>>
>
>

Re: all the nost are not reacheable when running massive deletes

[Alain RODRIGUEZ]

It looks like a complex issue that might worth having a close look at your
data model, configurations and machines.

It is hard to help you from the mailing list. Yet here are some thoughts,
some might be irrelevant or wrong, but some other might point you to your
issue, hope we will get lucky there :-):

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>1,000,000,400,030,00   98,57
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
> avgqu-sz   await  svctm  %util
>
> sda   0,00 0,000,000,20 0,00 0,00
> 8,00 0,000,00   0,00   0,00
>
> sdb   0,00 0,000,000,00 0,00 0,00
> 0,00 0,000,00   0,00   0,00
>
> sdc   0,00 0,000,000,00 0,00 0,00
> 0,00 0,000,00   0,00   0,00
>
> sdd   0,00 0,200,000,40 0,00 0,00
> 12,00 0,002,50   2,50   0,10
>
>
>
CPU:
>
>
>
> -  General use: 1 – 4 %
>
> -  Worst case: 98% .It is when the problem comes, running massive
> deletes(even in a different machine which is receiving the deletes) or
> running a repair.
>

First, the cluster is definitely not overloaded. You are having an issue
with some nodes from time to time. This looks like an imbalanced cluster.
It can be due to some wide rows or bad partition key. Make sure writes are
well balanced at any time with the partition you are using and try to spot
some warnings about large row compactions in the logs. Yet, I don't think
this is what you face as you then should have 2 or 3 nodes going crazy at
the same time because of RF (2 or 3).

Also, can we have an 'iostat -mx 5 100' on when a node goes mad?
An other good troubleshooting tool would be using
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#ttop-command.
It would be interesting to see what Cassandra threads are consuming the CPU
power. This is definitely something I would try on a high load node/time.


About the client, some comments, clearly unrelated to your issue, but
probably worth it to be told:

.setConsistencyLevel(ConsistencyLevel.QUORUM))

 [...]

 .withRetryPolicy(new LoggingRetryPolicy(DowngradingConsistencyRetryPol
> icy.INSTANCE))


I advice people to never do this. Basically, consistency level means: even
in the worst case, I want to make sure that at least (RF / 2) + 1 got the
read / write to consider it valid, if not drop the operation. If used for
both writes & reads, this provide you a strong and 'immediate' consistency
(no locks though, so excepted for some races). Data will always be sent to
all the nodes in charge of the token (generally 2 or 3 nodes, depending on
RF).

Then you say, if I can't have quorum, then go for one. Meaning you prefer
availability, rather than consistency. Then, why not use one from the start
as the consistency level? I would go for CL ONE or remove the '
DowngradingConsistencyRetryPolicy'.

Also, I would go with 'LOCAL_ONE/QUORUM', using Local is not an issue when
using only one DC as you do, but avoid some surprises when adding a new DC.
If you don't change it, keep it in mind for the day you add a new DC.

Yet, this client does a probably well balanced use of the cluster.

About your table:

I think the problem is related with one specific column family:
>
>
>
> CREATE TABLE snpaware.snpsearch ...
>

First thing is that this table is using a secondary index. I must say I
never used them, because it never worked very well and I did not want to
operate this kind of tables. I preferred maintaining my own indexes in the
past. In the future I might rather use Materialised View (C* 3.0+). Though,
I am not sure how performant they are yet.

>From what I heard, indexes are quite efficient on low cardinality. Is that
your case?

Also indexes are hold locally, no distribution. That would fit with your
'one node at the time and rotating' issue. Also, when deleting data from
there, index need to be updated. Delete operation is probably quite heavy.

Plus you say:

Which holds a lot of data. It is normally a cf which needs to be read but
> sometimes updated and deleted and I think the problem is there.


And I believe you're right. Could you work around this index somehow?

I wanted to change the compaction strategy but that means that a compaction
> will be executed and then timeouts will appear and I can not do that on the
> live cluster right now.


Not sure what new strategy you wanted to use, but LCS could make things a
lot worse, as LCS uses far more resources than STCS at compaction time.
Plus, at start, all your data would have to go through an heavy process.

Honestly, from what I know now, I would blame, the index, but again, I can
be wrong.

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-04-07 9:18 GMT+02:00 Paco Trujillo 

Re: disable compaction if all data are read-only?

[Pedro Gordo]

Hi Yatong

My understanding is that if you have a table whichi read-only and hence
doesn't receive any writes, then no SSTables will be created, and hence, no
compaction will happen. What compaction strategy do you have on your table?

Best regards

Pedro Gordo

On 8 April 2016 at 10:42, Yatong Zhang  wrote:

> Hi there,
> I am wondering if it is possible to disable compaction when all my data
> are read-only?
>

Re: Returning an UDT from a user defined function (UDF)

[Robert Stupp]

Hi Henry,

there’s https://issues.apache.org/jira/browse/CASSANDRA-10818 
 to allow creation of 
UDTs and tuples.

Robert

—
Robert Stupp
@snazy

> On Apr 8, 2016, at 07:12, Henry M  wrote:
> 
> Whatever I wanted to do does not seem to be possible (probably a limitation 
> or a bug)... I see a way to get the KeyspaceMetadata and from that get the 
> UserType instance (code lines 1 & 2 below).
> 
> 1.)
> org.apache.cassandra.schema.KeyspaceMetadata ksm = 
> org.apache.cassandra.config.Schema.instance.getKSMetaData("test_ks");
> 
> 2.)
> com.datastax.driver.core.UserType myUdt = ksm.types.get("my_other_udt").get();
> 
> But this fails with the below error because Schema is not a whitelisted 
> package. It probably should not be whitelisted but there should be a way to 
> create and return a user defined type.
> 
> :88:InvalidRequest: code=2200 [Invalid query] message="Could not 
> compile function 'test_ks.transform_udt' from Java source: 
> org.apache.cassandra.exceptions.InvalidRequestException: Java source 
> compilation failed:
> Line 4: org.apache.cassandra.schema.KeyspaceMetadata cannot be resolved to a 
> type
> Line 4: org.apache.cassandra.config.Schema.instance cannot be resolved to a 
> type
> "
> :90:InvalidRequest: code=2200 [Invalid query] message="Unknown 
> function 'extract_text_field_sample_udt'"
> 
> My updated UDF for complete context.
> 
> CREATE OR REPLACE FUNCTION test_ks.transform_udt (val my_udt)
>  RETURNS NULL ON NULL INPUT
>  RETURNS my_other_udt
>  LANGUAGE java
>   AS '
> String fieldA = val.getString("field_a");
> 
> org.apache.cassandra.schema.KeyspaceMetadata ksm = 
> org.apache.cassandra.config.Schema.instance.getKSMetaData("test_ks");
> 
> com.datastax.driver.core.UserType myUdt = 
> ksm.types.get("my_other_udt").get();
> 
> com.datastax.driver.core.UDTValue transformedValue = myUdt.newValue();
> 
> transformedValue.setUUID("id", java.util.UUID.randomUUID());
> transformedValue.setString("field_a", fieldA);
> transformedValue.setString("field_b", "value b");
> 
> return transformedValue;
>   ';
> 
> 
> On Thu, Apr 7, 2016 at 7:40 PM Henry M  > wrote:
> I was wondering if it is possible to create an UDT and return it within a 
> user defined function.
> 
> I looked at this documentation 
> http://docs.datastax.com/en/cql/3.3/cql/cql_using/useCreateUDF.html 
>  but the 
> examples are only for basic types.
> 
> This is my pseudo code I came up with... the part I think I am missing is how 
> to get an instance of the UserType so that I can invoke newValue to create a 
> UDTValue.
> 
> Has anyone done this and know how to get the keyspace in order to call 
> getUserType? Or know of an alternate approach?
> 
> CREATE OR REPLACE FUNCTION test_ks.transform_udt (val my_udt)
>  RETURNS NULL ON NULL INPUT
>  RETURNS my_other_udt
>  LANGUAGE java
>   AS '
> String fieldA = val.getString("field_a");
> 
> // How do you get a reference the user type?
> UserType myUdt = ?keyspace?.getUserType("my_other_udt");
> 
> UDTValue transformedValue = myUdt.newValue();
> 
> transformedValue.setUUID("id", UUID.randomUUID());
> transformedValue.setString("field_a", fieldA);
> transformedValue.setString("field_b", "value b");
> 
> return transformedValue;
>   ';
> 
> 
> Thank you,
> Henry
> 
> 
> P.S. This is the setup for my sample table and types.
> drop keyspace test_ks;
> 
> create keyspace test_ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 
> 'replication_factor' : 1 };
> 
> use test_ks;
> 
> CREATE TYPE IF NOT EXISTS test_ks.my_udt (field_a text, field_b text);
> CREATE TYPE IF NOT EXISTS test_ks.my_other_udt (id uuid, field_a text, 
> field_b text);
> 
> CREATE TABLE IF NOT EXISTS test_ks.sample_table(id uuid primary key, col_a 
> frozen);
> 
> INSERT INTO sample_table(id, col_a) VALUES ( now() , { field_a: 'value 1', 
> field_b: 'value 2'} );
> INSERT INTO sample_table(id) VALUES ( now() );
> 
> 
> 
> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

C*2.1 using timestamp with same timestamp

[aeljami.ext]

Hello,

I try to insert two lines for the same partition with the same timestamp. 
prmiere the line without TTL. the 2nd line with TTL.
As shown in the example below:

INSERT INTO tab1 (a,b) VALUES ('YYY','') USING TIMESTAMP 1460088862025940
INSERT INTO tab1 (a,b) VALUES ('YYY','') USING TIMESTAMP 1460088862025940 
AND TTL 1000

With C*2.0, when I select from the table, I get the row with the TTL (), 
therefore the second row

a| b | ttl(b) | writetime(b)
++-+---
YYY | | 999 |  1460088862025940

but with C*2.1 for the same insertions I get the row without TTL, therefore the 
first row.

a| b | ttl(b) | writetime(b)
++-+---
YYY | | null |  1460088862025940

I know it is related to the WriteTime except that I do not understand the 
difference between 2.0 and 2.1 in this case

Any idea ?

Thx.

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Re: Efficiently filtering results directly in CS

[vincent gromakowski]

c* I suppose

2016-04-07 19:30 GMT+02:00 Jonathan Haddad :

> What is CS?
>
> On Thu, Apr 7, 2016 at 10:03 AM Kevin Burton  wrote:
>
>> I have a paging model whereby we stream data from CS by fetching 'pages'
>> thereby reading (sequentially) entire datasets.
>>
>> We're using the bucket approach where we write data for 5 minutes, then
>> we can just fetch the bucket for that range.
>>
>> Our app now has TONS of data and we have a piece of middleware that
>> filters it based on the client requests.
>>
>> So if they only want english they just get english and filter away about
>> 60% of our data.
>>
>> but it doesn't support condition pushdown.  So ALL this data has to be
>> sent from our CS boxes to our middleware and filtered there (wasting a lot
>> of network IO).
>>
>> Is there away (including refactoring the code) that I could push this
>> this into CS?  Maybe some way I could discovery the CS topology and put
>> daemons on each of our CS boxes and fetch from CS directly (doing the
>> filtering there).
>>
>> Thoughts?
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>