Re: Cassandra decompress already compressed data

2017-02-21 Thread Minh Do 
Hi Alive,

I think your observation is aligned to what I saw earlier during my
debugging session.

What you have is the compression in the wire between Java Driver and the C*
Server.

C* storage has a different flag and mechanism to compress its stored data
in the SSTable files
and as you can see, it is set at the table level.

I think C* server needs to decompress the compressed data from the socket
to understand the data before processing it.





On Tue, Feb 21, 2017 at 11:59 AM, Мириан Джачвадзе  wrote:

> The question related to C* driver and C* server itself. I'll ask it here
> and at C* driver developers group too. Hope somebody can give me an answer
> or maybe give an explanation why it happens as it happens.
>
>
> This is the story about cassandra 2.2.7(I've checked it with the C* 3.* and
> result is the same) and driver 3.0.2.
>
>
> I am playing with blob in this table:
>
> CREATE TABLE compression_test (
>
> id uuid,
>
> chunk int,
>
> data blob,
>
> PRIMARY KEY (id, chunk))
>
> WITH compression = { 'sstable_compression' : '' }"
>
>
> Java cassandra driver has a compression option out of the box. It can
> compress data before send it to socket. I am thinking that driver compress
> source byte array, send it to C* and after processing it will be stored as
> is to SSTable. That is why the table configured without compression on
> SSTable level. But it's only my proposal. During investigation I found out
> that I was wrong.
>
>
> In my case I choose LZ4 compression in driver and 0.5G data was compressed
> down to 40Mb. Before send data to socket driver compress it, set
> COMPRESSED flag
> at the beginning of the compressed array and sent to C*  server. Through
> debug I see that exactly 40Mb was written to socket.
>
>
> Meanwhile C* server has Server.Initial class that initialize netty
> ChannelPipelineand append severalChannelHandler's to it. One of the
> ChannelHandleris Frame.Decompressor. It check that given frame has a
> COMPRESSED flag and decompress data in case it exist.  Also debugging I see
> that C* received exactly 40Mb chunk, find COMPRESSED flag, decompress it
> and process it. It leads to extra memory and disk consumption. I'm not sure
> how it's stored in mem table but absolutely sure that 0.5G of decompressed
> data is stored on disk driver. And it's not what I want. I can set LZ4
> compressor for SSTable and the received chunk will be stored compressed but
> it's also not what I want.
>
>
> For what purpose C* decompress already compressed data? I am miss something
> in C* or driver configuration?
>

Cassandra decompress already compressed data

2017-02-21 Thread Мириан Джачвадзе 
The question related to C* driver and C* server itself. I'll ask it here
and at C* driver developers group too. Hope somebody can give me an answer
or maybe give an explanation why it happens as it happens.


This is the story about cassandra 2.2.7(I've checked it with the C* 3.* and
result is the same) and driver 3.0.2.


I am playing with blob in this table:

CREATE TABLE compression_test (

id uuid,

chunk int,

data blob,

PRIMARY KEY (id, chunk))

WITH compression = { 'sstable_compression' : '' }"


Java cassandra driver has a compression option out of the box. It can
compress data before send it to socket. I am thinking that driver compress
source byte array, send it to C* and after processing it will be stored as
is to SSTable. That is why the table configured without compression on
SSTable level. But it's only my proposal. During investigation I found out
that I was wrong.


In my case I choose LZ4 compression in driver and 0.5G data was compressed
down to 40Mb. Before send data to socket driver compress it, set
COMPRESSED flag
at the beginning of the compressed array and sent to C*  server. Through
debug I see that exactly 40Mb was written to socket.


Meanwhile C* server has Server.Initial class that initialize netty
ChannelPipelineand append severalChannelHandler's to it. One of the
ChannelHandleris Frame.Decompressor. It check that given frame has a
COMPRESSED flag and decompress data in case it exist.  Also debugging I see
that C* received exactly 40Mb chunk, find COMPRESSED flag, decompress it
and process it. It leads to extra memory and disk consumption. I'm not sure
how it's stored in mem table but absolutely sure that 0.5G of decompressed
data is stored on disk driver. And it's not what I want. I can set LZ4
compressor for SSTable and the received chunk will be stored compressed but
it's also not what I want.


For what purpose C* decompress already compressed data? I am miss something
in C* or driver configuration?

[RELEASE] Apache Cassandra 2.2.9 released

2017-02-21 Thread Michael Shuler 
The Cassandra team is pleased to announce the release of Apache
Cassandra version 2.2.9.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.2 series. As always,
please pay attention to the release notes[2] and Let us know[3] if you
were to encounter any problem.

Enjoy!

[1]: (CHANGES.txt)
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-2.2.9
[2]: (NEWS.txt)
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-2.2.9
[3]: https://issues.apache.org/jira/browse/CASSANDRA



signature.asc
Description: OpenPGP digital signature

[RELEASE] Apache Cassandra 2.1.17 released

2017-02-21 Thread Michael Shuler 
The Cassandra team is pleased to announce the release of Apache
Cassandra version 2.1.17.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.1 series. As always,
please pay attention to the release notes[2] and Let us know[3] if you
were to encounter any problem.

Enjoy!

[1]: (CHANGES.txt)
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-2.1.17
[2]: (NEWS.txt)
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-2.1.17
[3]: https://issues.apache.org/jira/browse/CASSANDRA



signature.asc
Description: OpenPGP digital signature

[RELEASE] Apache Cassandra 3.0.11 released

2017-02-21 Thread Michael Shuler 
The Cassandra team is pleased to announce the release of Apache
Cassandra version 3.0.11.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 3.0 series. As always,
please pay attention to the release notes[2] and Let us know[3] if you
were to encounter any problem.

Enjoy!

[1]: (CHANGES.txt)
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-3.0.11
[2]: (NEWS.txt)
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-3.0.11
[3]: https://issues.apache.org/jira/browse/CASSANDRA



signature.asc
Description: OpenPGP digital signature

Re: Pluggable throttling of read and write queries

2017-02-21 Thread vincent gromakowski 
Aren't you using mesos Cassandra framework to manage your multiple clusters
? (Seen a presentation in cass summit)
What's wrong with your current mesos approach ?
I am also thinking it's better to split a large cluster into smallers
except if you also manage client layer that query cass and you can put some
backpressure or rate limit in it.

Le 21 févr. 2017 2:46 AM, "Edward Capriolo"  a
écrit :

> Older versions had a request scheduler api.
>
> On Monday, February 20, 2017, Ben Slater 
> wrote:
>
>> We’ve actually had several customers where we’ve done the opposite -
>> split large clusters apart to separate uses cases. We found that this
>> allowed us to better align hardware with use case requirements (for example
>> using AWS c3.2xlarge for very hot data at low latency, m4.xlarge for more
>> general purpose data) we can also tune JVM settings, etc to meet those uses
>> cases.
>>
>> Cheers
>> Ben
>>
>> On Mon, 20 Feb 2017 at 22:21 Oleksandr Shulgin <
>> oleksandr.shul...@zalando.de> wrote:
>>
>>> On Sat, Feb 18, 2017 at 3:12 AM, Abhishek Verma  wrote:
>>>
 Cassandra is being used on a large scale at Uber. We usually create
 dedicated clusters for each of our internal use cases, however that is
 difficult to scale and manage.

 We are investigating the approach of using a single shared cluster with
 100s of nodes and handle 10s to 100s of different use cases for different
 products in the same cluster. We can define different keyspaces for each of
 them, but that does not help in case of noisy neighbors.

 Does anybody in the community have similar large shared clusters and/or
 face noisy neighbor issues?

>>>
>>> Hi,
>>>
>>> We've never tried this approach and given my limited experience I would
>>> find this a terrible idea from the perspective of maintenance (remember the
>>> old saying about basket and eggs?)
>>>
>>> What potential benefits do you see?
>>>
>>> Regards,
>>> --
>>> Alex
>>>
>>> --
>> 
>> Ben Slater
>> Chief Product Officer
>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>> +61 437 929 798 <+61%20437%20929%20798>
>>
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>

Re: [VOTE PASSED] Release Apache Cassandra 3.0.11

2017-02-21 Thread Michael Shuler 
Vote result:
8 +1 binding votes
1 +1 non-binding votes
0 +0 or -1 votes

This release vote passed, so I'll get the artifacts published.

-- 
Kind regards,
Michael

On 02/15/2017 07:15 PM, Michael Shuler wrote:
> I propose the following artifacts for release as 3.0.11.
> 
> sha1: 338226e042a22242645ab54a372c7c1459e78a01
> Git:
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.11-tentative
> Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1138/org/apache/cassandra/apache-cassandra/3.0.11/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1138/
> 
> The Debian packages are available here: http://people.apache.org/~mshuler
> 
> The vote will be open for 72 hours (longer if needed).
> 
> [1]: (CHANGES.txt) https://goo.gl/ztSHQJ
> [2]: (NEWS.txt) https://goo.gl/nrengr
> 




signature.asc
Description: OpenPGP digital signature

Re: [VOTE PASSED] Release Apache Cassandra 2.2.9

2017-02-21 Thread Michael Shuler 
Vote result:
8 +1 binding votes
2 +1 non-binding votes
0 +0 or -1 votes

This release vote passed, so I'll get the artifacts published.

-- 
Kind regards,
Michael

On 02/15/2017 07:16 PM, Michael Shuler wrote:
> I propose the following artifacts for release as 2.2.9.
> 
> sha1: 70a08f1c35091a36f7d9cc4816259210c2185267
> Git:
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.2.9-tentative
> Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1139/org/apache/cassandra/apache-cassandra/2.2.9/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1139/
> 
> The Debian packages are available here: http://people.apache.org/~mshuler
> 
> The vote will be open for 72 hours (longer if needed).
> 
> [1]: (CHANGES.txt) https://goo.gl/AYblr5
> [2]: (NEWS.txt) https://goo.gl/gIXxgR
> 




signature.asc
Description: OpenPGP digital signature

Re: [VOTE PASSED] Release Apache Cassandra 2.1.17

2017-02-21 Thread Michael Shuler 
Vote result:
8 +1 binding votes
1 +1 non-binding votes
0 +0 or -1 votes

This release vote passed, so I'll get the artifacts published.

-- 
Kind regards,
Michael

On 02/15/2017 07:16 PM, Michael Shuler wrote:
> I propose the following artifacts for release as 2.1.17.
> 
> sha1: 943db2488c8b62e1fbe03b132102f0e579c9ae17
> Git:
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.1.17-tentative
> Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1140/org/apache/cassandra/apache-cassandra/2.1.17/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1140/
> 
> The Debian packages are available here: http://people.apache.org/~mshuler
> 
> The vote will be open for 72 hours (longer if needed).
> 
> [1]: (CHANGES.txt) https://goo.gl/17RivH
> [2]: (NEWS.txt) https://goo.gl/axKXys
> 




signature.asc
Description: OpenPGP digital signature

Re: Does C* coordinator writes to replicas in same order or different order?

2017-02-21 Thread Kant Kodali 
Agreed that async performs better than sync in general but the catch here
to me is the "order".

The whole point of async is to do out of order processing by which I mean
say if a request 1 comes in at time t1 and a request 2 comes in at time t2
where t1 < t2 and say now that t1 is taking longer to process than t2 in
which case request 2 should get a response first and subsequently a
response for request 1. This is where I would imagine all the benefits of
async come in but the moment you introduce order by saying for Last Write
Wins all the async requests should be processed in order I would imagine
all the benefits of async are lost.

Let's see if anyone can comment about how it works inside C*.

Thanks!



On Mon, Feb 20, 2017 at 10:54 PM, Dor Laor  wrote:

> Could be. Let's stay tuned to see if someone else pick it up.
> Anyway, if it's synchronous, you'll have a large penalty for latency.
>
> On Mon, Feb 20, 2017 at 10:11 PM, Kant Kodali  wrote:
>
>> Thanks again for the response! if they mean it between client and server
>> I am not sure why they would use the word "replication" in the statement
>> below since there is no replication between client and server( coordinator).
>>
>> "Choose between synchronous or asynchronous replication for each update."
>>>
>>
>> Sent from my iPhone
>>
>> On Feb 20, 2017, at 5:30 PM, Dor Laor  wrote:
>>
>> I think they mean the client to server and not among the servers
>>
>> On Mon, Feb 20, 2017 at 5:28 PM, Kant Kodali  wrote:
>>
>>> Also here is a statement from C* website
>>>
>>> "Choose between synchronous or asynchronous replication for each update.
>>> "
>>>
>>> http://cassandra.apache.org/
>>>
>>> Looks like we can choose then either sync or async then?
>>>
>>> On Mon, Feb 20, 2017 at 5:25 PM, Kant Kodali  wrote:
>>>
 Hi Dor,

 Great response! My comments are inline.

 Thanks a lot,
 kant


 On Mon, Feb 20, 2017 at 4:41 PM, Dor Laor  wrote:

> I sent this answer but it bounced off the user@apache.
> Here is the email anyway:
>
> -- Forwarded message --
> From: Dor Laor 
> Date: Mon, Feb 20, 2017 at 4:37 PM
> Subject: Re: Does C* coordinator writes to replicas in same order or
> different order?
> To: dev@cassandra.apache.org
> Cc: u...@cassandra.apache.org
>
>
> + The C* coordinator send async write requests to the replicas.
>This is very important since it allows it to return a low latency
>reply to the client once the CL is reached. You wouldn't want
>to serialize the replicas one after the other.
>

 *so coordinator wont wait until a CL is reached before it
 process another request? *

>
>  + The client <-> server sync/async isn't related to the coordinator
> in this case.
>
>  + In the case of concurrent writes (always the case...), the time
> stamp
> sets the order. Note that it's possible to work with client
> timestamps or
> server timestamps. The client ones are usually the best choice.
>

  *In theory, Why we say concurrent writes they should have the same
 timestamp right?  What I am really looking for is that if I send write
 request concurrently for record 1 and record 2 are they guaranteed to be
 inserted in the same order across replicas? (Whatever order coordinator may
 choose is fine but I want the same order across all replicas and with async
 replication I am not sure how that is possible ? for example,  if a request
 arrives with timestamp t1 and another request arrives with a timestamp t2
 where t1 < t2...with async replication what if one replica chooses to
 execute t2 first and then t1 simply because t1 is slow while another
 replica choose to execute t1 first and then t2..how would that work?  )*

>
> Note that C* each node can be a coordinator (one per request) and its
> the desired case in order to load balance the incoming requests. Once
> again,
> timestamps determine the order among the requests.
>
> Cheers,
> Dor
>
> On Mon, Feb 20, 2017 at 4:12 PM, Kant Kodali 
> wrote:
>
>> Hi,
>>
>> when C* coordinator writes to replicas does it write it in same order
>> or
>> different order? other words, Does the replication happen
>> synchronously or
>> asynchrnoulsy ? Also does this depend sync or async client? What
>> happens in
>> the case of concurrent writes to a coordinator ?
>>
>> Thanks,
>> kant
>>
>
>
>

>>>
>>
>