Re: yet another benchmark bottleneck

[Michael Burman]

Although low amount of updates, it's possible that you hit a contention 
bug. A simple test would be to add multiple Cassandra nodes on the same 
physical node (like split your 20 cores to 5 instances of Cassandra). If 
you get much higher throughput, then you have an answer..


I don't think a single-instance Cassandra 3.11.2 scales to 20 cores (at 
least with the stress-test pattern). There's few known issues in the 
write-path at least that prevent scaling with high CPU core count.


  - Micke


On 03/12/2018 03:14 PM, onmstester onmstester wrote:
I mentioned that already tested increasing client threads + many 
stress-client instances in one node + two stress-client in two 
separate nodes, in all of them the sum of throughputs is less than 
130K. I've been tuning all aspects of OS and Cassandra (whatever I've 
seen in config files!) for two days, still no luck!


Sent using Zoho Mail 



 On Mon, 12 Mar 2018 16:38:22 +0330 *Jacques-Henri Berthemet 
* wrote 


What happens if you increase number of client threads?

Can you add another instance of cassandra-stress on another host?


*--*

*Jacques-Henri Berthemet*


*From:* onmstester onmstester [mailto:onmstes...@zoho.com
]
*Sent:* Monday, March 12, 2018 12:50 PM
*To:* user >
*Subject:* RE: yet another benchmark bottleneck


no luck even with 320 threads for write


Sent using Zoho Mail 



 On Mon, 12 Mar 2018 14:44:15 +0330 *Jacques-Henri Berthemet
>* wrote 


It makes more sense now, 130K is not that bad.


According to cassandra.yaml you should be able to increase
your number of write threads in Cassandra:

# On the other hand, since writes are almost never IO bound,
the ideal

# number of "concurrent_writes" is dependent on the number of
cores in

# your system; (8 * number_of_cores) is a good rule of thumb.

concurrent_reads: 32

concurrent_writes: 32

concurrent_counter_writes: 32


Jumping directly to 160 would be a bit high with spinning
disks, maybe start with 64 just to see if it gets better.


*--*

*Jacques-Henri Berthemet*


*From:*onmstester onmstester [mailto:onmstes...@zoho.com
]

*Sent:*Monday, March 12, 2018 12:08 PM

*To:*user >

*Subject:*RE: yet another benchmark bottleneck


RF=1

No errors or warnings.

Actually its 300 Mbit/seconds and 130K OP/seconds. I missed a
'K' in first mail, but anyway! the point is: More than half of
node resources (cpu, mem, disk, network) is unused and i can't
increase write throughput.


Sent using Zoho Mail 



 On Mon, 12 Mar 2018 14:25:12 +0330 *Jacques-Henri
Berthemet >* wrote 


Any errors/warning in Cassandra logs? What’s your RF?

Using 300MB/s of network bandwidth for only 130 op/s looks
very high.


*--*

*Jacques-Henri Berthemet*


*From:*onmstester onmstester [mailto:onmstes...@zoho.com
]

*Sent:*Monday, March 12, 2018 11:38 AM

*To:*user >

*Subject:*RE: yet another benchmark bottleneck


1.2 TB 15K

latency reported by stress tool is 7.6 ms. disk latency is
2.6 ms


Sent using Zoho Mail 



 On Mon, 12 Mar 2018 14:02:29 +0330 *Jacques-Henri
Berthemet >* wrote 


What’s your disk latency? What kind of disk is it?


*--*

*Jacques-Henri Berthemet*


*From:*onmstester onmstester
[mailto:onmstes...@zoho.com ]

*Sent:*Monday, March 12, 2018 10:48 AM

*To:*user >

*Subject:*Re: yet another benchmark bottleneck


Running two instance of Apache Cassandra on same
server, each having their own commit log disk dis not
help. Sum of cpu/ram usage for both instances would be
less than half of all available resources. 

Re: configure pooling options to avoid BusyPoolException

[Michael Burman]

Hi,

You could try something like (I tried to cleanup the code from other 
stuff in the gist-editor, so it might not compile directly) the 
following for your scheduling:


https://gist.github.com/burmanm/230c306f88c69c62dfe73799fc01

That should prevent pool getting full, instead using the old blocking 
behavior for your code. Add your own twist of backpressure handling to 
the code obviously.


  - Micke


On 08/23/2017 09:40 PM, Avi Levi wrote:

Hi ,
I need to execute large amount (millions) of select queries. but I am 
getting BusyPoolExcption how can I avoid that ? I tried to configure 
the pooling options but couldn't see that it had any impact

Any advice ?
|Failed to execute query SELECT * FROM my_table WHERE id = 'some_uuid' 
AND x >= 1503501104 AND y < 1503501224; 
com.datastax.driver.core.exceptions.NoHostAvailableException: All 
host(s) tried for query failed (tried: localhost/0:0:0:0:0:0:0:1:9042 
(com.datastax.driver.core.exceptions.BusyPoolException: 
[localhost/0:0:0:0:0:0:0:1] Pool is busy (no available connection and 
the queue has reached its max size 256)))|



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Composite partition key token

[Michael Burman]

Hi,

How about taking it from the BoundStatement directly?

ByteBuffer routingKey = 
b.getRoutingKey(ProtocolVersion.NEWEST_SUPPORTED, codecRegistry);

Token token = metadata.newToken(routingKey);

In this case the b is the "BoundStatement". Replace codecRegistry & 
ProtocolVersion with what you have. codecRegistry for example from the 
codecRegistry = session.getCluster().getConfiguration().getCodecRegistry();


  - Micke


On 02/08/2017 08:58 PM, Branislav Janosik -T (bjanosik - AAP3 INC at 
Cisco) wrote:


Hi,

I would like to ask how to calculate token for composite partition key 
using java api?


For partition key made of one column I use 
cluster.getMetadata().newToken(newBuffer);


But what if my key looks like this PRIMARY KEY 
((parentResourceId,timeRT), childName)?


I read that “:” is a separator but it doesn’t seem to be the case.

How can I create ByteBuffer with multiple values so that the token 
would be actually correct?


Thank you,

Branislav

Re: Merging cells in compaction / compression?

[Michael Burman]

Hi,

For storing time series data, storage disk usage is quite significant factor - 
time series applications generate a lot of data (and of course the newest data 
is most important). Given that even DateTiered compaction was designed in 
keeping mind of these specialities of time series data, wouldn't it make sense 
to also improve the storage efficiency? Cassandra 3.x's one of the key 
improvements was that improved storage engine - but it's still far away from 
being efficient with time series data.

Efficient compression methods for both floating points & integers have a lot of 
research behind them and can be applied to time series data. I wish to apply 
these methods to improve storage efficiency - and performance*

* In my experience, storing blocks of data and decompressing them on the client 
side instead of letting Cassandra read more rows improves performance by 
several times. The query patterns for time series data are often in requesting 
a range of data (instead of single datapoint).

And I wasn't comparing Cassandra & Hadoop, but the combination of 
Spark+Cassandra+distributed-scheduler+other stuff vs. a Hadoop installation. At 
that point they are quite comparable in many cases, with latter being easier to 
manage in the end. I don't want either for a simple time series storage 
solution as I have no need for other components than data storage.

  - Micke

- Original Message -
From: "Jonathan Haddad" <j...@jonhaddad.com>
To: user@cassandra.apache.org
Sent: Friday, August 5, 2016 5:22:58 PM
Subject: Re: Merging cells in compaction / compression?

Hadoop and Cassandra have very different use cases.  If the ability to
write a custom compression system is the primary factor in how you choose
your database I suspect you may run into some trouble.

Jon

On Fri, Aug 5, 2016 at 6:14 AM Michael Burman <mibur...@redhat.com> wrote:

> Hi,
>
> As Spark is an example of something I really don't want. It's resource
> heavy, it involves copying data and it involves managing yet another
> distributed system. Actually I would also need a distributed system to
> schedule the spark jobs also.
>
> Sounds like a nightmare to implement a compression method. Might as well
> run Hadoop.
>
>   - Micke
>
> - Original Message -
> From: "DuyHai Doan" <doanduy...@gmail.com>
> To: user@cassandra.apache.org
> Sent: Thursday, August 4, 2016 11:26:09 PM
> Subject: Re: Merging cells in compaction / compression?
>
> Look like you're asking for some sort of ETL on your C* data, why not use
> Spark to compress those data into blobs and use User-Defined-Function to
> explode them when reading ?
>
> On Thu, Aug 4, 2016 at 10:08 PM, Michael Burman <mibur...@redhat.com>
> wrote:
>
> > Hi,
> >
> > No, I don't want to lose precision (if that's what you meant), but if you
> > meant just storing them in a larger bucket (which I could decompress
> either
> > on client side or server side). To clarify, it could be like:
> >
> > 04082016T230215.1234, value
> > 04082016T230225.4321, value
> > 04082016T230235.2563, value
> > 04082016T230245.1145, value
> > 04082016T230255.0204, value
> >
> > ->
> >
> > 04082016T230200 -> blob (that has all the points for this minute stored -
> > no data is lost to aggregated avgs or sums or anything).
> >
> > That's acceptable, of course the prettiest solution would be to keep this
> > hidden from a client so it would see while decompressing the original
> rows
> > (like with byte[] compressors), but this is acceptable for my use-case.
> If
> > this is what you meant, then yes.
> >
> >   -  Micke
> >
> > - Original Message -
> > From: "Eric Stevens" <migh...@gmail.com>
> > To: user@cassandra.apache.org
> > Sent: Thursday, August 4, 2016 10:26:30 PM
> > Subject: Re: Merging cells in compaction / compression?
> >
> > When you say merge cells, do you mean re-aggregating the data into
> courser
> > time buckets?
> >
> > On Thu, Aug 4, 2016 at 5:59 AM Michael Burman <mibur...@redhat.com>
> wrote:
> >
> > > Hi,
> > >
> > > Considering the following example structure:
> > >
> > > CREATE TABLE data (
> > > metric text,
> > > value double,
> > > time timestamp,
> > > PRIMARY KEY((metric), time)
> > > ) WITH CLUSTERING ORDER BY (time DESC)
> > >
> > > The natural inserting order is metric, value, timestamp pairs, one
> > > metric/value pair per second for example. That means creating more and
> > more
> > > cells to the same partition, which creates a large amount of overhead
> an

Re: Merging cells in compaction / compression?

[Michael Burman]

Hi,

As Spark is an example of something I really don't want. It's resource heavy, 
it involves copying data and it involves managing yet another distributed 
system. Actually I would also need a distributed system to schedule the spark 
jobs also.

Sounds like a nightmare to implement a compression method. Might as well run 
Hadoop.

  - Micke

- Original Message -
From: "DuyHai Doan" <doanduy...@gmail.com>
To: user@cassandra.apache.org
Sent: Thursday, August 4, 2016 11:26:09 PM
Subject: Re: Merging cells in compaction / compression?

Look like you're asking for some sort of ETL on your C* data, why not use
Spark to compress those data into blobs and use User-Defined-Function to
explode them when reading ?

On Thu, Aug 4, 2016 at 10:08 PM, Michael Burman <mibur...@redhat.com> wrote:

> Hi,
>
> No, I don't want to lose precision (if that's what you meant), but if you
> meant just storing them in a larger bucket (which I could decompress either
> on client side or server side). To clarify, it could be like:
>
> 04082016T230215.1234, value
> 04082016T230225.4321, value
> 04082016T230235.2563, value
> 04082016T230245.1145, value
> 04082016T230255.0204, value
>
> ->
>
> 04082016T230200 -> blob (that has all the points for this minute stored -
> no data is lost to aggregated avgs or sums or anything).
>
> That's acceptable, of course the prettiest solution would be to keep this
> hidden from a client so it would see while decompressing the original rows
> (like with byte[] compressors), but this is acceptable for my use-case. If
> this is what you meant, then yes.
>
>   -  Micke
>
> - Original Message -
> From: "Eric Stevens" <migh...@gmail.com>
> To: user@cassandra.apache.org
> Sent: Thursday, August 4, 2016 10:26:30 PM
> Subject: Re: Merging cells in compaction / compression?
>
> When you say merge cells, do you mean re-aggregating the data into courser
> time buckets?
>
> On Thu, Aug 4, 2016 at 5:59 AM Michael Burman <mibur...@redhat.com> wrote:
>
> > Hi,
> >
> > Considering the following example structure:
> >
> > CREATE TABLE data (
> > metric text,
> > value double,
> > time timestamp,
> > PRIMARY KEY((metric), time)
> > ) WITH CLUSTERING ORDER BY (time DESC)
> >
> > The natural inserting order is metric, value, timestamp pairs, one
> > metric/value pair per second for example. That means creating more and
> more
> > cells to the same partition, which creates a large amount of overhead and
> > reduces the compression ratio of LZ4 & Deflate (LZ4 reaches ~0.26 and
> > Deflate ~0.10 ratios in some of the examples I've run). Now, to improve
> > compression ratio, how could I merge the cells on the actual Cassandra
> > node? I looked at ICompress and it provides only byte-level compression.
> >
> > Could I do this on the compaction phase, by extending the
> > DateTieredCompaction for example? It has SSTableReader/Writer facilities
> > and it seems to be able to see the rows? I'm fine with the fact that
> repair
> > run might have to do some conflict resolution as the final merged rows
> > would be quite "small" (50kB) in size. The naive approach is of course to
> > fetch all the rows from Cassandra - merge them on the client and send
> back
> > to the Cassandra, but this seems very wasteful and has its own problems.
> > Compared to table-LZ4 I was able to reduce the required size to 1/20th
> > (context-aware compression is sometimes just so much better) so there are
> > real benefits to this approach, even if I would probably violate multiple
> > design decisions.
> >
> > One approach is of course to write to another storage first and once the
> > blocks are ready, write them to Cassandra. But that again seems idiotic
> (I
> > know some people are using Kafka in front of Cassandra for example, but
> > that means maintaining yet another distributed solution and defeats the
> > benefit of Cassandra's easy management & scalability).
> >
> > Has anyone done something similar? Even planned? If I need to extend
> > something in Cassandra I can accept that approach also - but as I'm not
> > that familiar with Cassandra source code I could use some hints.
> >
> >   - Micke
> >
>

Re: Merging cells in compaction / compression?

[Michael Burman]

Hi,

No, I don't want to lose precision (if that's what you meant), but if you meant 
just storing them in a larger bucket (which I could decompress either on client 
side or server side). To clarify, it could be like:

04082016T230215.1234, value
04082016T230225.4321, value
04082016T230235.2563, value
04082016T230245.1145, value
04082016T230255.0204, value

-> 

04082016T230200 -> blob (that has all the points for this minute stored - no 
data is lost to aggregated avgs or sums or anything).

That's acceptable, of course the prettiest solution would be to keep this 
hidden from a client so it would see while decompressing the original rows 
(like with byte[] compressors), but this is acceptable for my use-case. If this 
is what you meant, then yes.

  -  Micke

- Original Message -
From: "Eric Stevens" <migh...@gmail.com>
To: user@cassandra.apache.org
Sent: Thursday, August 4, 2016 10:26:30 PM
Subject: Re: Merging cells in compaction / compression?

When you say merge cells, do you mean re-aggregating the data into courser
time buckets?

On Thu, Aug 4, 2016 at 5:59 AM Michael Burman <mibur...@redhat.com> wrote:

> Hi,
>
> Considering the following example structure:
>
> CREATE TABLE data (
> metric text,
> value double,
> time timestamp,
> PRIMARY KEY((metric), time)
> ) WITH CLUSTERING ORDER BY (time DESC)
>
> The natural inserting order is metric, value, timestamp pairs, one
> metric/value pair per second for example. That means creating more and more
> cells to the same partition, which creates a large amount of overhead and
> reduces the compression ratio of LZ4 & Deflate (LZ4 reaches ~0.26 and
> Deflate ~0.10 ratios in some of the examples I've run). Now, to improve
> compression ratio, how could I merge the cells on the actual Cassandra
> node? I looked at ICompress and it provides only byte-level compression.
>
> Could I do this on the compaction phase, by extending the
> DateTieredCompaction for example? It has SSTableReader/Writer facilities
> and it seems to be able to see the rows? I'm fine with the fact that repair
> run might have to do some conflict resolution as the final merged rows
> would be quite "small" (50kB) in size. The naive approach is of course to
> fetch all the rows from Cassandra - merge them on the client and send back
> to the Cassandra, but this seems very wasteful and has its own problems.
> Compared to table-LZ4 I was able to reduce the required size to 1/20th
> (context-aware compression is sometimes just so much better) so there are
> real benefits to this approach, even if I would probably violate multiple
> design decisions.
>
> One approach is of course to write to another storage first and once the
> blocks are ready, write them to Cassandra. But that again seems idiotic (I
> know some people are using Kafka in front of Cassandra for example, but
> that means maintaining yet another distributed solution and defeats the
> benefit of Cassandra's easy management & scalability).
>
> Has anyone done something similar? Even planned? If I need to extend
> something in Cassandra I can accept that approach also - but as I'm not
> that familiar with Cassandra source code I could use some hints.
>
>   - Micke
>

Merging cells in compaction / compression?

[Michael Burman]

Hi,

Considering the following example structure:

CREATE TABLE data (
metric text,
value double,
time timestamp,
PRIMARY KEY((metric), time)
) WITH CLUSTERING ORDER BY (time DESC)

The natural inserting order is metric, value, timestamp pairs, one metric/value 
pair per second for example. That means creating more and more cells to the 
same partition, which creates a large amount of overhead and reduces the 
compression ratio of LZ4 & Deflate (LZ4 reaches ~0.26 and Deflate ~0.10 ratios 
in some of the examples I've run). Now, to improve compression ratio, how could 
I merge the cells on the actual Cassandra node? I looked at ICompress and it 
provides only byte-level compression.

Could I do this on the compaction phase, by extending the DateTieredCompaction 
for example? It has SSTableReader/Writer facilities and it seems to be able to 
see the rows? I'm fine with the fact that repair run might have to do some 
conflict resolution as the final merged rows would be quite "small" (50kB) in 
size. The naive approach is of course to fetch all the rows from Cassandra - 
merge them on the client and send back to the Cassandra, but this seems very 
wasteful and has its own problems. Compared to table-LZ4 I was able to reduce 
the required size to 1/20th (context-aware compression is sometimes just so 
much better) so there are real benefits to this approach, even if I would 
probably violate multiple design decisions. 

One approach is of course to write to another storage first and once the blocks 
are ready, write them to Cassandra. But that again seems idiotic (I know some 
people are using Kafka in front of Cassandra for example, but that means 
maintaining yet another distributed solution and defeats the benefit of 
Cassandra's easy management & scalability).

Has anyone done something similar? Even planned? If I need to extend something 
in Cassandra I can accept that approach also - but as I'm not that familiar 
with Cassandra source code I could use some hints. 

  - Micke