Re: Cassandra crashes....

[Alain Rastoul]

On 08/22/2017 05:39 PM, Thakrar, Jayesh wrote:

Surbhi and Fay,

I agree we have plenty of RAM to spare.



Hi

At the very beginning of system.log there is a
INFO  [CompactionExecutor:487] 2017-08-21 23:21:01,684 
NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot 
allocate chunk of 1.000MiB

who comes from BufferPool exhaustion (several messages)
From the source
file_cache_size_in_mb
(Default: Smaller of 1/4 heap or 512) Total memory to use for 
SSTable-reading buffers.


So here in your configuration it is 512M, may be you should set it to a 
higher value in your cassandra.yaml (1/4 => 4G) ?
(also see https://issues.apache.org/jira/browse/CASSANDRA-11681, the 
default value may not be accurate)


Another strange thing is the number of prepared statements which also 
gives errors: lot of messages like
WARN  [ScheduledTasks:1] 2017-08-22 07:09:25,009 QueryProcessor.java:105 
- 1 prepared statements discarded in the last minute because cache limit 
reached (64 MB)

...
on startup you see:
INFO  [main] 2017-08-22 12:50:13,787 QueryProcessor.java:162 - Preloaded 
13357 prepared statements


13K different prepared statements sounds a lot...
an issue about that seems to be fixed in 3.11 
https://issues.apache.org/jira/browse/CASSANDRA-13641
May be youc should truncate your system.prepared_statements and restart 
your node



HTH


--
best,
Alain

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra-count gives wrong results

[Alain Rastoul]

Hi,

I use cassandra-count (github 
https://github.com/brianmhess/cassandra-count) to count records in a 
table, but I have wrong results.


When I export data with cqlsh /copy to csv, I have 1M records in my test 
table, when I use cassandra-count I have different results for each node :
build/cassandra-count -host cstar1 -user cassandra -pw cassandra 
-keyspace metrics -table datapoints

metrics.datapoints: 379285

build/cassandra-count -host cstar2 -user cassandra -pw cassandra 
-keyspace metrics -table datapoints

metrics.datapoints: 324856

build/cassandra-count -host cstar3 -user cassandra -pw cassandra 
-keyspace metrics -table datapoints

metrics.datapoints: 340615

It used to work in previous runs, but suddenly, the results went  wrong, 
I can't understand why.


I downloaded the cassandra-count project, built and debugged it, but 
still can't understand.


the program reads partitions in system.size_estimates, then for each 
partition execute
SELECT COUNT(*) FROM keyspaceName.tableName WHERE Token("path") > ? AND 
Token("path") <= ?

with start and end ranges.
The ring is correct, each node has the same ring.


Any clue about this ?

TIA


--
best,
Alain


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: How can I scale my read rate?

[Alain Rastoul]

On 24/03/2017 01:00, Eric Stevens wrote:

Assuming an even distribution of data in your cluster, and an even
distribution across those keys by your readers, you would not need to
increase RF with cluster size to increase read performance.  If you have
3 nodes with RF=3, and do 3 million reads, with good distribution, each
node has served 1 million read requests.  If you increase to 6 nodes and
keep RF=3, then each node now owns half as much data and serves only
500,000 reads.  Or more meaningfully in the same time it takes to do 3
million reads under the 3 node cluster you ought to be able to do 6
million reads under the 6 node cluster since each node is just
responsible for 1 million total reads.


Hi Eric,

I think I got your point.
In case of really evenly distributed  reads it may (or should?) not make 
any difference,


But when you do not distribute well the reads (and in that case only),
my understanding about RF was that it could help spreading the load :
In that case, with RF= 4 instead of 3,  with several clients accessing keys
same key ranges, a coordinator could pick up one node to handle the request
in 4 replicas instead of picking up one node in 3 , thus having
more "workers" to handle a request ?

Am I wrong here ?

Thank you for the clarification


--
best,
Alain

Re: Grouping time series data into blocks of times

[Alain Rastoul]

Kairosdb (OpenTSDB clone forCassandra) is a TSDB that does this.
MAy be you could have a look at it ?
It has a daemon process that collects and groups data points into blobs 
before writing to cassandra.


--
best,
Alain

Re: How can I scale my read rate?

[Alain Rastoul]

On 20/03/2017 22:05, Michael Wojcikiewicz wrote:

Not sure if someone has suggested this, but I believe it's not
sufficient to simply add nodes to a cluster to increase read
performance: you also need to alter the ReplicationFactor of the
keyspace to a larger value as you increase your cluster gets larger.

ie. data is available from more nodes in the cluster for each query.

Yes, good point in case of cluster growth, there would be more replica 
to handle same key ranges.

And also readjust token ranges :
https://cassandra.apache.org/doc/latest/operating/topo_changes.html

SG, can you give some information (or share your code) about how you 
generate your data and how you read it ?


--
best,
Alain

Re: How can I scale my read rate?

[Alain Rastoul]

On 20/03/2017 02:35, S G wrote:

2)
https://docs.datastax.com/en/developer/java-driver/3.1/manual/statements/prepared/
tells me to avoid preparing select queries if I expect a change of
columns in my table down the road.
The problem is also related to select * which is considered bad practice 
with most databases...



I did some more testing to see if my client machines were the bottleneck.
For a 6-node Cassandra cluster (each VM having 8-cores), I got 26,000
reads/sec for all of the following:
1) Client nodes:1, Threads: 60
2) Client nodes:3, Threads: 180
3) Client nodes:5, Threads: 300
4) Client nodes:10, Threads: 600
5) Client nodes:20, Threads: 1200

So adding more client nodes or threads to those client nodes is not
having any effect.
I am suspecting Cassandra is simply not allowing me to go any further.

> Primary keys for my schema are:
>  PRIMARY KEY((name, phone), age)
> name: text
> phone: int
> age: int

Yes with such a PK data must be spread on the whole cluster (also taking 
into account the partitioner), strange that the throughput doesn't scale.

I guess you also have verified that you select data randomly?

May be you could have a look at the system traces to see the query plan 
for some requests:
If you are on a test cluster you can truncate the tables before 
(truncate system_traces.sessions; and truncate system_traces.events;), 
run a test then select * from system_traces.events

where session_id = 
xxx being one of the sessions you pick in trace.sessions.

Try to see if you are not always hitting the same nodes.


--
best,
Alain

Re: How can I scale my read rate?

[Alain Rastoul]

On 19/03/2017 02:54, S G wrote:

Forgot to mention that this vmstat picture is for the client-cluster
reading from Cassandra.



Hi SG,


Your numbers are low, 15k req/sec would be ok for a single node, for a 
12 nodes cluster, something goes wrong... how do you measure the 
throughput?


As suggested by others, to achieve good results you have to add threads 
and client VMs: Cassandra scales horizontally, not vertically, ie each 
single node performance will not goes up, but if you spread the load, by 
adding nodes the global cluster performance will.


Theorically,
assuming the data and the load is spread on the cluster (*1)
from your saying, with each request at 2ms avg (*2)
you should have 500 req/sec in each thread,
40 threads should go 20k req/sec on each client VM stress application (*3)
and 10 client VMs should go 200k req/sec on the whole cluster (*4)

=
(*1) the partition key (first PK column) must spread data on all nodes
and your testing code must spread the load by selecting evenly spread data.
(This point is very important: can you give information on your schema 
and your data ?)


(*2) to achieve better single client throughtput, may be you could 
prepare the requests, since you are always executing the same requests


(*3) => run more client tests application on each VM

(*4) add more client  VMs (Patrick's suggestion)

with (3) and (4) the throughput of each client will not be better, but 
the global cluster throughput will.

=

There are other factors to take into account if you are also writing to 
the cluster : read path, tombstones, replication, repairs etc. but 
that's not the case here?


Performance testing goes to the limit of our understanding of the system
 and is very difficult
... hence interesting :)



--
best,
Alain

Re: weird jvm metrics

[Alain Rastoul]

On 01/05/2017 07:59 PM, Edward Capriolo wrote:

Good troubleshooting. I would open a Jira. It seems like a good solution
would be to replace '..' with '.' somehow. It seems like no one would
every want ..


In my case with logstash it was not really a problem, in case of the OP, 
may be it is a  problem and he cannot simply replace the '..'?
I guess it depends of the collector used, and may be it is a real issue 
with graphite/carbon or others.


I created the issue https://issues.apache.org/jira/browse/CASSANDRA-13103

--
best,
Alain

Re: weird jvm metrics

[Alain Rastoul]

On 01/04/2017 11:12 PM, Edward Capriolo wrote:

The metric-reporter is actually leveraged from another project.

https://github.com/addthis/metrics-reporter-config

Check the version of metric-reporter (in cassandra/lib) and see if it
has changed from your old version to your new version.




On Wed, Jan 4, 2017 at 12:02 PM, Mike Torra > wrote:

Just bumping - has anyone seen this before?


http://stackoverflow.com/questions/41446352/cassandra-3-9-jvm-metrics-have-bad-name



From: Mike Torra >
Reply-To: "user@cassandra.apache.org
" >
Date: Wednesday, December 28, 2016 at 4:49 PM
To: "user@cassandra.apache.org "
>
Subject: weird jvm metrics

Hi There -

I recently upgraded from cassandra 3.5 to 3.9 (DDC), and I noticed
that the "new" jvm metrics are reporting with an extra '.' character
in them. Here is a snippet of what I see from one of my nodes:

ubuntu@ip-10-0-2-163:~$ sudo tcpdump -i eth0 -v dst port 2003 -A |
grep 'jvm'

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture
size 65535 bytes

.Je..l>.pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.capacity
762371494 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.count 3054
1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..direct.used
762371496 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.capacity
515226631134 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.count 45572
1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.buffers..mapped.used
515319762610 1482960946

pi.cassandra.us-east-1.cassy-node1.jvm.fd.usage 0.00 1482960946


My metrics.yaml looks like this:

graphite:
   -
 period: 60
 timeunit: 'SECONDS'
 prefix: 'pi.cassandra.us-east-1.cassy-node1'
 hosts:
  - host: '#RELAY_HOST#'
port: 2003
 predicate:
   color: "white"
   useQualifiedName: true
   patterns:
 - "^org.+"
 - "^jvm.+"
 - "^java.lang.+"

All the org.* metrics come through fine, and the jvm.fd.usage metric
strangely comes through fine, too. The rest of the jvm.* metrics
have this extra '.' character that causes them to not show up in
graphite.

Am I missing something silly here? Appreciate any help or suggestions.

- Mike



Hi,

I also noticed this problem recently.
Some jvm metrics have a double dot in name like:
jvm.memory..total.max , jvm.memory..total.init (etc).
Investigating a bit, it seems that an extra dot is added at the end of 
the name in CassandraDaemon.java, around line 367 (in 3.0.10):

...
// enable metrics provided by metrics-jvm.jar

CassandraMetricsRegistry.Metrics.register("jvm.buffers.", new 
BufferPoolMetricSet(ManagementFactory.getPlatformMBeanServer()));
CassandraMetricsRegistry.Metrics.register("jvm.gc.", 
new GarbageCollectorMetricSet());


CassandraMetricsRegistry.Metrics.register("jvm.memory.", new 
MemoryUsageGaugeSet());


and also added in append method of MetricRegistry.
Call stack is:
MetricRegistry>>registerAll(String prefix, MetricSet metrics)
MetricRegistry>>static String name(String name, String... names)
MetricRegistry>>static void append(StringBuilder builder, String part)

in append the dot is also added:
...
if(builder.length() > 0) {
builder.append('.');
}
builder.append(part);
...

The codahale MetricRegistry class seems to have no recent modification 
of name or append methods, so it looks like a small (but annoying) bug.

May be the fix could be to simply remove the final dot ?

Is it worth opening an issue in JIRA ?


--
best,
Alain

Re: Improving performance where a lot of updates and deletes are required?

[Alain Rastoul]

On 11/08/2016 08:52 PM, Alain Rastoul wrote:

For example if you had to track the position of a lot of objects,
instead of updating the object records, each second you could insert a
new event with : (object: object_id, event_type: position_move, position
: x, y ).



and add a timestamp of course
and eventually TTL the data, with a decreasing clustering sort order


--
best,
Alain

Re: Improving performance where a lot of updates and deletes are required?

[Alain Rastoul]

On 11/08/2016 11:05 AM, DuyHai Doan wrote:

Are you sure Cassandra is a good fit for this kind of heavy update &
delete scenario ?


+1
this sounds like relational thinking scenario... (no offense, I like 
relational systems)
As if you want to maintain the state of a lot of entities with updates & 
deletes, and you have a lot of state changes for your entities.


May be an eventstore/DDD approach would be a better model for that?

You could have an aggregate for each entity (ie. a record) you have in 
your system and insert a new event record on each update of this agregate.


For example if you had to track the position of a lot of objects, 
instead of updating the object records, each second you could insert a 
new event with : (object: object_id, event_type: position_move, position 
: x, y ).


Just a suggestion.

--
best,
Alain

Re: A difficult data model with C*

[Alain Rastoul]

On 11/08/2016 03:54 AM, ben ben wrote:

Hi guys,
   CREATE TABLE recent (
 user_name text,
 vedio_id text,
 position int,
 last_time timestamp,
 PRIMARY KEY (user_name, vedio_id)
)



Hi Ben,

May be a clustering columns order would help
CREATE TABLE recent (
...
) WITH CLUSTERING ORDER BY (last_time DESC);
So you can query only the last 10 records
SELECT * FROM recent WHERE vedio_id = xxx  LIMIT 10

See here http://www.datastax.com/dev/blog/we-shall-have-order
--
best,
Alain

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

[Alain Rastoul]

On 15/06/2016 06:40, linbo liao wrote:

I am not sure, but looks it will cause the update other than insert. If
it is true, the only way is request includes IF NOT EXISTS, inform the
client it failed?

Thanks,
Linbo


Hi Linbo,

+1 with what Ben said, timestamp has a millisecond precision and is a 
bad choice for making PK unicity.
If your client and server are on the same physical machine (both on same 
computer or different vms on same hypervisor), insert duration can go 
down to very few microseconds (2~3 on a recent computer).

Your insert will/should often become "update".
The reason is that update does not exists in cassandra, neither delete, 
they are just "appends":  append with same key for update or append of a 
tombstone for delete.
You should try to use a timeuuid instead, it has a node, clock sequence, 
a counter plus the timestamp part that you can get with cql functions, 
and it exists for that use.

see here for the functions
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/timeuuid_functions_r.html


--
best,
Alain

Re: Cassandra

[Alain Rastoul]

On 25/05/2016 17:56, bastien dine wrote:

Hi,

I'm running a 3 nodes Cassandra 2.1.x cluster. Each node has 8vCPU and 30
Go RAM.
Replication factor = 3 for my keyspace.


...



Is there a problem with the Java Driver ? The load balancing is not

Hi Bastien,

A replication factor of 3 for a 3 node cluster does not balance the 
load: since you ask for 3 copies of the data (rf=3) on 3 nodes cluster,

each node will have a copy of the data and you are overloading all nodes.
May be you should try with a rf = 2 or add nodes to your cluster ?


"working" ? How can I list connections on a node ?

For a 3.x (I think also 2.x) you can trace  requests at the query level 
with enableTracing() method.

something like : (uncomment the line with .enableTracing() )

session.execute( boundedInsertEventStatement.bind( aggregateId, 
aggregateType, eventType, payload )

.setConsistencyLevel(ConsistencyLevel.ONE)
//.enableTracing()
);
see the doc for other classes and tracing or consistency options,
and have a look at nodetool settraceprobability if you cannot change the 
code


The queries and query plans appear in the system_traces.sessions and 
system_traces.events tables. It can be very verbose for query plans 
(events table), may be you should truncate the sessions and events 
tables before running your load (on 3.x tables are  truncated on startup)




Regards,
Bastien



HTH,
--
best,
Alain