Re: cassandra.yaml configuration for large machines (scale up vs. scale out)

2017-11-04 Thread Jeff Jirsa 
SSDs definitely help. I think JBOD works correctly in newer versions
(CASSANDRA-6696 that separates data by token, so you dont have to rebuild
the whole node, not sure off the top of my head if there's followup
tickets).
10GB probably more useful if you use vnodes than single tokens (likely to
change significantly in future versions as streaming gets more efficient).
Most people run fine on 8-16G heaps, roughly the same offheap, and use the
rest for os page cache (again, newer versions have some extra chunk
caches). Going from 128 to 256G of RAM is useful if the extra RAM will give
you some data access benefits (ie: if you have ~200G of hot data, going
from 128->256G of ram probably helps keep all the hot data in cache, that's
a huge help; if you have 4T of data and it's randomly accessed, the extra
128G of data is probably going to be mostly wasted).

At a past employer we ran thousands of nodes with 8 virtual cores (probably
4 with hyper-threading), 4T of ssd, and 32G of RAM - lots of cheap'ish
machines.

For the original post, the baseline looks reasonable. Key cache may benefit
from being larger (since you're using the full 2G), or it may be that
you're invalidating frequently so you're not benefiting as much - hard to
know which it is without knowing your access patterns. For all of those
values, you can glance at (
https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html ) for
help tuning - that post is pretty darn reasonable and well annotated.


On Sat, Nov 4, 2017 at 6:28 PM, James Briggs  wrote:

> > I know that Cassandra is built for scale out on commodity hardware
>
> The term "commodity hardware" is not very useful, though the average
> server-class machine bought in 2017 can work.
>
> Netflix found that SSD helped greatly with compactions in production.
> Generally servers use 10 GB networking in 2017.
>
> 128 GB is commonly used, but I would use 256+ GB in new servers.
>
> I don't recommend the Cassandra JBOD configuration since losing
> one drive means rebuilding the node immediately, which many
> organizations aren't responsive enough to do.
>
> Thanks, James.
> --
> Cassandra/MySQL DBA. Available in San Jose area or remote.
> cass_top: https://github.com/jamesbriggs/cassandra-top
>
>
> --
> *From:* "Steinmaurer, Thomas" 
> *To:* "user@cassandra.apache.org" 
> *Sent:* Friday, November 3, 2017 6:34 AM
> *Subject:* cassandra.yaml configuration for large machines (scale up vs.
> scale out)
>
> Hello,
>
> I know that Cassandra is built for scale out on commodity hardware, but I
> wonder if anyone can share some experience when running Cassandra on rather
> capable machines.
>
> Let’s say we have a 3 node cluster with 128G RAM, 32 physical cores (16
> per CPU socket), Large Raid with Spinning Disks (so somewhere beyond 2000
> IOPS).
>
> What are some recommended cassandra.yaml configuration / JVM settings,
> e.g. we have been using with something like that as a first baseline:
> · 31G heap, G1, -XX:MaxGCPauseMillis=2000
> · concurrent_compactors: 8
> · compaction_throughput_mb_per_sec: 128
> · key_cache_size_in_mb: 2048
> · concurrent_reads: 256
> · concurrent_writes: 256
> · native_transport_max_threads: 256
>
> Anything else we should add to our first baseline of settings?
>
> E.g. although we have a key cache of 2G, nodetool info gives me only 0.451
> as hit rate:
>
> Key Cache  : entries 2919619, size 1.99 GB, capacity 2 GB,
> 71493172 hits, 158411217 requests, 0.451 recent hit rate, 14400 save period
> in seconds
>
>
> Thanks,
> Thomas
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>
>
>

Re: cassandra.yaml configuration for large machines (scale up vs. scale out)

2017-11-04 Thread James Briggs 
> I know that Cassandra is built for scale out on commodity hardware
The term "commodity hardware" is not very useful, though the 
averageserver-class machine bought in 2017 can work.

Netflix found that SSD helped greatly with compactions in production.Generally 
servers use 10 GB networking in 2017.
128 GB is commonly used, but I would use 256+ GB in new servers.
 I don't recommend the Cassandra JBOD configuration since losingone drive means 
rebuilding the node immediately, which manyorganizations aren't responsive 
enough to do.

Thanks, James.
--
Cassandra/MySQL DBA. Available in San Jose area or remote.
cass_top: https://github.com/jamesbriggs/cassandra-top

  From: "Steinmaurer, Thomas" 
 To: "user@cassandra.apache.org"  
 Sent: Friday, November 3, 2017 6:34 AM
 Subject: cassandra.yaml configuration for large machines (scale up vs. scale 
out)
   
 Hello,    I know that Cassandra is built for scale out 
on commodity hardware, but I wonder if anyone can share some experience when 
running Cassandra on rather capable machines.    Let’s say we have a 3 node 
cluster with 128G RAM, 32 physical cores (16 per CPU socket), Large Raid with 
Spinning Disks (so somewhere beyond 2000 IOPS).    What are some recommended 
cassandra.yaml configuration / JVM settings, e.g. we have been using with 
something like that as a first baseline: ·31G heap, G1, 
-XX:MaxGCPauseMillis=2000 ·concurrent_compactors: 8 ·
compaction_throughput_mb_per_sec: 128 ·key_cache_size_in_mb: 2048 · 
   concurrent_reads: 256 ·concurrent_writes: 256 ·
native_transport_max_threads: 256    Anything else we should add to our first 
baseline of settings?    E.g. although we have a key cache of 2G, nodetool info 
gives me only 0.451 as hit rate:    Key Cache  : entries 2919619, 
size 1.99 GB, capacity 2 GB, 71493172 hits, 158411217 requests, 0.451 recent 
hit rate, 14400 save period in seconds       Thanks, Thomas    The contents of 
this e-mail are intended for the named addressee only. It contains information 
that may be confidential. Unless you are the named addressee or an authorized 
designee, you may not copy or use it, or disclose it to anyone else. If you 
received it in error please notify us immediately and then destroy it. 
Dynatrace Austria GmbH (registration number FN 91482h) is a company registered 
in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313

[no subject]

2017-11-04 Thread vbhang...@gmail.com 
Kishore, Here is the table dean and cfstats o/p  ---
===
CREATE TABLE ks1.table1 (
key text,
column1 
'org.apache.cassandra.db.marshal.DynamicCompositeType(org.apache.cassandra.db.marshal.UTF8Type)',
value blob,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'sstable_size_in_mb': '256', 'class': 
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 86400
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.1
AND speculative_retry = '99.0PERCENTILE';
==
SSTable count: 261
SSTables in each level: [0, 6, 40, 215, 0, 0, 0, 0, 0]
Space used (live): 129255873809
Space used (total): 129255873809
Space used by snapshots (total): 0
Off heap memory used (total): 20977830
SSTable Compression Ratio: 0.7879224917729545
Number of keys (estimate): 71810
Memtable cell count: 2010
Memtable data size: 226253
Memtable off heap memory used: 1327192
Memtable switch count: 47
Local read count: 11688546
Local read latency: 0.195 ms
Local write count: 225262
Local write latency: 0.055 ms
Pending flushes: 0
Bloom filter false positives: 146072
Bloom filter false ratio: 0.01543
Bloom filter space used: 35592
Bloom filter off heap memory used: 33504
Index summary off heap memory used: 26686
Compression metadata off heap memory used: 19590448
Compacted partition minimum bytes: 25
Compacted partition maximum bytes: 10299432635
Compacted partition mean bytes: 2334776
Average live cells per slice (last five minutes): 
4.346574725773759
Maximum live cells per slice (last five minutes): 2553.0
Average tombstones per slice (last five minutes): 
0.3096773382165276
Maximum tombstones per slice (last five minutes): 804.0
=

On 2017-10-24 14:39, "Mohapatra, Kishore"  wrote: 
> Hi Vedant,
>   I was actually referring to command line select query 
> with Consistency level=ALL . This will force a read repair in the background.
> But as I can see, you have tried with consistency level = one and and it is 
> still timing out. SO what error you see in the system.log ?
> Streaming error ?
> 
> Can you also check how many sstables are there for that table . Seems like 
> your compaction may not be working.
> Is your repair job running fine ?
> 
> Thanks
> 
> Kishore Mohapatra
> Principal Operations DBA
> Seattle, WA
> Ph : 425-691-6417 (cell)
> Email : kishore.mohapa...@nuance.com
> 
> 
> -Original Message-
> From: vbhang...@gmail.com [mailto:vbhang...@gmail.com] 
> Sent: Monday, October 23, 2017 6:59 PM
> To: user@cassandra.apache.org
> Subject: [EXTERNAL] 
> 
> It is RF=3 and 12 nodes in 3 regions and 6 in other 2, so total 48 nodes. Are 
> you suggesting forced read repair by reading consistency of ONE or by bumping 
> up read_repair_chance? 
> 
> We have tried from command  line with ONE but that times out. 
> On 2017-10-23 10:18, "Mohapatra, Kishore"  
> wrote: 
> > What is your RF for the keyspace and how many nodes are there in each DC ?
> > 
> > Did you force a Read Repair to see, if you are getting the data or getting 
> > an error ?
> > 
> > Thanks
> > 
> > Kishore Mohapatra
> > Principal Operations DBA
> > Seattle, WA
> > Email : kishore.mohapa...@nuance.com
> > 
> > 
> > -Original Message-
> > From: vbhang...@gmail.com [mailto:vbhang...@gmail.com]
> > Sent: Sunday, October 22, 2017 11:31 PM
> > To: user@cassandra.apache.org
> > Subject: [EXTERNAL]
> > 
> > -- Consistency level  LQ
> > -- It started happening approximately couple of months back.  Issue is very 
> > inconsistent and can't be reproduced.  It used rarely happen earlier (since 
> > last few years).
> > -- There are very few GC pauses but  they don't coincide with the issue. 
> > -- 99% latency is less than 80ms and 75% is less than 5ms.
> > 
> > - Vedant
> > On 2017-10-22 21:29, Jeff Jirsa  wrote: 
> > > What consistency level do you use on writes?
> > > Did this just start or has it always happened ?
> > >

poolingOptions not serializable?

2017-11-04 Thread Andrea Giordano 
Hi, 
I’m using datastax driver to use Cassandra as sink for some data streams with 
Apache Flink:
I have a problem executing my application raising an error about the full 
queue. I discovered that the default value is 256, probably too low for my 
load, so I have raised it using poolingOptions setting maxRequestsPerConnection 
as suggested here: 
http://docs.datastax.com/en/developer/java-driver/3.1/manual/pooling/ 
.

Unfortunately with the following code I obtain the following error when I 
launch it:

The implementation of the ClusterBuilder is not serializable. 
The object probably contains or references non serializable fields.


My code:


PoolingOptions poolingOptions = new PoolingOptions();
poolingOptions
  .setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)
  .setMaxRequestsPerConnection(HostDistance.REMOTE, 1);


ClusterBuilder cassandraBuilder = new ClusterBuilder() {
private static final long serialVersionUID = 1L;

@Override
public Cluster buildCluster(Cluster.Builder builder) {
return 
builder.addContactPoint(CASSANDRA_ADDRESS).withPort(CASSANDRA_PORT)..withPoolingOptions(poolingOptions).build();
}
};


sinkBuilderNormalStream
.setQuery("INSERT INTO keyspace_local.values_by_sensors_users"
+ " (user, sensor, timestamp, rdf_stream, observed_value, 
value)"
+ " VALUES (?, ?, ?, ?, ?, ?);")
.setClusterBuilder(cassandraBuilder)
.build();


How can I deal with it?