Re: Async queries

[Charulata Sharma (charshar)]

Thanks Andy for the detailed explanation. I have added the callback feature and 
am going to test if this helps.

Charu

From: Andy Tolbert 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, October 2, 2017 at 5:48 PM
To: "user@cassandra.apache.org" 
Subject: Re: Async queries

Hi Charu,

Since the driver uses Guava futures, you can use some of the methods in 
Futures
 to add listeners, callbacks and transformers that are invoked when the future 
completes without blocking the calling thread like getUninterruptibly does.  
For example, the following registers a callback whose onSuccess or onFailure 
method is called based on the success of the query.

Futures.addCallback(future, new FutureCallback() {
public void onSuccess(ResultSet result) {
// process result
}

public void onFailure(Throwable t) {
// log exception
}
});

You can read more about using the driver's async features 
here.

Since executeAsync does not block, you'll want to be careful of is not 
submitting too many requests at a time as this can degrade performance and may 
explain what you are observing.  One simple (although somewhat crude) way of 
handling this is to use a Semaphore with a fixed number of permits.  You would 
acquire a Semaphore permit before you execute a query, and then release a 
permit in a callback on completion of the request.  This would cause your 
calling thread to block whenever you run out of permits, and then continue when 
a query completes and releases a permit.

The upcoming version (4.0) of the java driver uses 
CompletionStage/CompletableFuture (java 8 futures), although we'll probably 
provide a guava extension as well for those who still want to use 
ListenableFuture.

Thanks,
Andy


On Mon, Oct 2, 2017 at 6:44 PM Charulata Sharma (charshar) 
> wrote:
Hi ,


We are observing some performance issues when executing a large number of 
read/write queries.
We use executeAsync query for most of our read and write requests and then 
future.getUninterruptibly() methods before returning to the client application.


Code snippet is:  (In the bind portion we have some GSON object conversions.

 List futures = new ArrayList<>();

  BoundStatement bind ;
  For(loop condition) {
bind =PreparedStatement.bind(….) //The PreparedStatement is prepared 
outside the loop.
   resultSetFuture = SESSION.executeAsync(bind);
 futures.add(resultSetFu ture);
  }

for(ResultSetFuture future: futures){
   future.getUninterruptibly();
}


Reading through the documents , I found that although the queries are executed 
in an async fashion, the future. getUninterruptibly(), is a blocking call.
I am trying to implement a callable future, but wanted to know from the 
community if there is any better way of doing this and if changing to callable 
future will help.


Thanks,
Charu

Re: Async queries

[Andy Tolbert]

Hi Charu,

Since the driver uses Guava futures, you can use some of the methods in
Futures

to
add listeners, callbacks and transformers that are invoked when the future
completes without blocking the calling thread like getUninterruptibly
does.  For example, the following registers a callback whose onSuccess or
onFailure method is called based on the success of the query.

Futures.addCallback(future, new FutureCallback() {
public void onSuccess(ResultSet result) {
// process result
}

public void onFailure(Throwable t) {
// log exception
}
});

You can read more about using the driver's async features here
.

Since executeAsync does not block, you'll want to be careful of is not
submitting too many requests at a time as this can degrade performance and
may explain what you are observing.  One simple (although somewhat crude)
way of handling this is to use a Semaphore with a fixed number of permits.
You would acquire a Semaphore permit before you execute a query, and then
release a permit in a callback on completion of the request.  This would
cause your calling thread to block whenever you run out of permits, and
then continue when a query completes and releases a permit.

The upcoming version (4.0) of the java driver uses
CompletionStage/CompletableFuture (java 8 futures), although we'll probably
provide a guava extension as well for those who still want to use
ListenableFuture.

Thanks,
Andy


On Mon, Oct 2, 2017 at 6:44 PM Charulata Sharma (charshar) <
chars...@cisco.com> wrote:

> Hi ,
>
>
>
>
>
> We are observing some performance issues when executing a large number of
> read/write queries.
>
> We use executeAsync query for most of our read and write requests and then
> future.getUninterruptibly() methods before returning to the client
> application.
>
>
>
>
>
> Code snippet is:  (In the bind portion we have some GSON object
> conversions.
>
>
>
>  List futures = *new* ArrayList<>();
>
>
>
>   BoundStatement bind ;
>
>   For(loop condition) {
>
> bind =PreparedStatement.bind(….) //The PreparedStatement is
> prepared outside the loop.
>
>resultSetFuture = *SESSION*.executeAsync(bind);
>
>
>  futures.add(resultSetFu ture);
>
>   }
>
>
>
> *for*(ResultSetFuture future: futures){
>
>future.getUninterruptibly();
>
> }
>
>
>
>
>
> Reading through the documents , I found that although the queries are
> executed in an async fashion, the future. getUninterruptibly(), is a
> blocking call.
>
> I am trying to implement a callable future, but wanted to know from the
> community if there is any better way of doing this and if changing to
> callable future will help.
>
>
>
>
>
> Thanks,
>
> Charu
>

Async queries

[Charulata Sharma (charshar)]

Hi ,


We are observing some performance issues when executing a large number of 
read/write queries.
We use executeAsync query for most of our read and write requests and then 
future.getUninterruptibly() methods before returning to the client application.


Code snippet is:  (In the bind portion we have some GSON object conversions.

 List futures = new ArrayList<>();

  BoundStatement bind ;
  For(loop condition) {
bind =PreparedStatement.bind(….) //The PreparedStatement is prepared 
outside the loop.
   resultSetFuture = SESSION.executeAsync(bind);
 futures.add(resultSetFu ture);
  }

for(ResultSetFuture future: futures){
   future.getUninterruptibly();
}


Reading through the documents , I found that although the queries are executed 
in an async fashion, the future. getUninterruptibly(), is a blocking call.
I am trying to implement a callable future, but wanted to know from the 
community if there is any better way of doing this and if changing to callable 
future will help.


Thanks,
Charu

Re:

[Dan Kinder]

Created https://issues.apache.org/jira/browse/CASSANDRA-13923

On Mon, Oct 2, 2017 at 12:06 PM, Dan Kinder  wrote:

> Sure will do.
>
> On Mon, Oct 2, 2017 at 11:48 AM, Jeff Jirsa  wrote:
>
>> You're right, sorry I didnt read the full stack (gmail hid it from me)
>>
>> Would you open a JIRA with your stack traces, and note (somewhat loudly)
>> that this is a regression?
>>
>>
>> On Mon, Oct 2, 2017 at 11:43 AM, Dan Kinder  wrote:
>>
>>> Right, I just meant that calling it at all results in holding a read
>>> lock, which unfortunately is blocking these read threads.
>>>
>>> On Mon, Oct 2, 2017 at 11:40 AM, Jeff Jirsa  wrote:
>>>


 On Mon, Oct 2, 2017 at 11:27 AM, Dan Kinder 
 wrote:

> (As a side note, it seems silly to call shouldDefragment at all on a
> read if the compaction strategy is not STSC)
>
>
>
 It defaults to false:

 https://github.com/apache/cassandra/blob/cassandra-3.0/src/j
 ava/org/apache/cassandra/db/compaction/AbstractCompactionStr
 ategy.java#L302

 And nothing else other than STCS overrides it to true.



>>>
>>>
>>> --
>>> Dan Kinder
>>> Principal Software Engineer
>>> Turnitin – www.turnitin.com
>>> dkin...@turnitin.com
>>>
>>
>>
>
>
> --
> Dan Kinder
> Principal Software Engineer
> Turnitin – www.turnitin.com
> dkin...@turnitin.com
>



-- 
Dan Kinder
Principal Software Engineer
Turnitin – www.turnitin.com
dkin...@turnitin.com

Re: Alter table gc_grace_seconds

[Gábor Auth]

Hi,

On Mon, Oct 2, 2017 at 1:43 PM Varun Barala  wrote:

> Either you can change min_threshold to three in your case or you can
> change compaction strategy for this table.
>

I've changed:
alter table number_item with compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '2'};

The list of database files of `number_item` on one node:
-rw-r--r-- 1 cassandra cassandra 71353717 Oct  2 11:53 mc-48399-big-Data.db
-rw-r--r-- 1 cassandra cassandra 32843468 Oct  2 19:15 mc-48435-big-Data.db
-rw-r--r-- 1 cassandra cassandra 24734857 Oct  2 19:49 mc-48439-big-Data.db

I've initiated a compaction on the `number_item` CF. After the compaction:
-rw-r--r-- 1 cassandra cassandra 71353717 Oct  2 11:53 mc-48399-big-Data.db
-rw-r--r-- 1 cassandra cassandra 32843468 Oct  2 19:15 mc-48435-big-Data.db
-rw-r--r-- 1 cassandra cassandra 24734857 Oct  2 19:53 mc-48440-big-Data.db

Two of them untouched and one rewritten with the same content. :/

Bye,
Gábor Auth

Re:

[Dan Kinder]

Sure will do.

On Mon, Oct 2, 2017 at 11:48 AM, Jeff Jirsa  wrote:

> You're right, sorry I didnt read the full stack (gmail hid it from me)
>
> Would you open a JIRA with your stack traces, and note (somewhat loudly)
> that this is a regression?
>
>
> On Mon, Oct 2, 2017 at 11:43 AM, Dan Kinder  wrote:
>
>> Right, I just meant that calling it at all results in holding a read
>> lock, which unfortunately is blocking these read threads.
>>
>> On Mon, Oct 2, 2017 at 11:40 AM, Jeff Jirsa  wrote:
>>
>>>
>>>
>>> On Mon, Oct 2, 2017 at 11:27 AM, Dan Kinder 
>>> wrote:
>>>
 (As a side note, it seems silly to call shouldDefragment at all on a
 read if the compaction strategy is not STSC)



>>> It defaults to false:
>>>
>>> https://github.com/apache/cassandra/blob/cassandra-3.0/src/j
>>> ava/org/apache/cassandra/db/compaction/AbstractCompactionStr
>>> ategy.java#L302
>>>
>>> And nothing else other than STCS overrides it to true.
>>>
>>>
>>>
>>
>>
>> --
>> Dan Kinder
>> Principal Software Engineer
>> Turnitin – www.turnitin.com
>> dkin...@turnitin.com
>>
>
>


-- 
Dan Kinder
Principal Software Engineer
Turnitin – www.turnitin.com
dkin...@turnitin.com

Re:

[Jeff Jirsa]

You're right, sorry I didnt read the full stack (gmail hid it from me)

Would you open a JIRA with your stack traces, and note (somewhat loudly)
that this is a regression?


On Mon, Oct 2, 2017 at 11:43 AM, Dan Kinder  wrote:

> Right, I just meant that calling it at all results in holding a read lock,
> which unfortunately is blocking these read threads.
>
> On Mon, Oct 2, 2017 at 11:40 AM, Jeff Jirsa  wrote:
>
>>
>>
>> On Mon, Oct 2, 2017 at 11:27 AM, Dan Kinder  wrote:
>>
>>> (As a side note, it seems silly to call shouldDefragment at all on a
>>> read if the compaction strategy is not STSC)
>>>
>>>
>>>
>> It defaults to false:
>>
>> https://github.com/apache/cassandra/blob/cassandra-3.0/src/
>> java/org/apache/cassandra/db/compaction/AbstractCompactionS
>> trategy.java#L302
>>
>> And nothing else other than STCS overrides it to true.
>>
>>
>>
>
>
> --
> Dan Kinder
> Principal Software Engineer
> Turnitin – www.turnitin.com
> dkin...@turnitin.com
>

Re:

[Dan Kinder]

Right, I just meant that calling it at all results in holding a read lock,
which unfortunately is blocking these read threads.

On Mon, Oct 2, 2017 at 11:40 AM, Jeff Jirsa  wrote:

>
>
> On Mon, Oct 2, 2017 at 11:27 AM, Dan Kinder  wrote:
>
>> (As a side note, it seems silly to call shouldDefragment at all on a read
>> if the compaction strategy is not STSC)
>>
>>
>>
> It defaults to false:
>
> https://github.com/apache/cassandra/blob/cassandra-3.0/
> src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.
> java#L302
>
> And nothing else other than STCS overrides it to true.
>
>
>


-- 
Dan Kinder
Principal Software Engineer
Turnitin – www.turnitin.com
dkin...@turnitin.com

Re:

[Jeff Jirsa]

On Mon, Oct 2, 2017 at 11:27 AM, Dan Kinder  wrote:

> (As a side note, it seems silly to call shouldDefragment at all on a read
> if the compaction strategy is not STSC)
>
>
>
It defaults to false:

https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java#L302

And nothing else other than STCS overrides it to true.

Re: How do I install Cassandra on AWS

[Michael Shuler]

On 10/02/2017 10:53 AM, Lutaya Shafiq Holmes wrote:
> 
> How do I install Cassandra on AWS- Amazon web services

Follow the installation instructions on the following page, relevant to
your OS of choice:

  http://cassandra.apache.org/download/

Let the list know if you have any problems!

-- 
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Materialized views stability

[Carlos Rolo]

I've been dealing with MV extensively, and I second Blake. MVs are not
suitable for production. Unless you're ready for the pain (The out of sync
is a major pain point), I would not go that way.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Mon, Oct 2, 2017 at 4:50 PM, Blake Eggleston 
wrote:

> Hi Hannu,
>
> There are more than a few committers that don't think MVs are currently
> suitable for production use. I'm not involved with MV development, so this
> may not be 100% accurate, but the problems as I understand them are:
>
> There's no way to determine if a view is out of sync with the base table.
> If you do determine that a view is out of sync, the only way to fix it is
> to drop and rebuild the view.
> There are liveness issues with updates being reflected in the view.
>
> Any one of these issues makes it difficult to recommend for general
> application development. I'd say that if you're not super familiar with
> their shortcomings and are confident you can fit your use case in them,
> you're probably better off not using them.
>
> Thanks,
>
> Blake
>
> On October 2, 2017 at 6:55:52 AM, Hannu Kröger (hkro...@gmail.com) wrote:
>
> Hello,
>
> I have seen some discussions around Materialized Views and stability of
> that functionality.
>
> There are some open issues around repairs:
> https://issues.apache.org/jira/browse/CASSANDRA-13810?
> jql=project%20%3D%20CASSANDRA%20AND%20issuetype%20%3D%20Bug%
> 20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%
> 20Reopened%2C%20%22Patch%20Available%22%2C%20Testing%
> 2C%20%22Ready%20to%20Commit%22%2C%20%22Awaiting%20Feedback%22)%20AND%
> 20component%20%3D%20%22Materialized%20Views%22
>
> Is it so that the current problems are mostly related to incremental
> repairs or are there also other major concerns why some people don’t
> consider them to be safe for production use?
>
> Cheers,
> Hannu
>
>

-- 


--

How do I install Cassandra on AWS

[Lutaya Shafiq Holmes]

Hi folks,

How do I install Cassandra on AWS- Amazon web services
-- 
Lutaaya Shafiq
Web: www.ronzag.com | i...@ronzag.com
Mobile: +256702772721 | +256783564130
Twitter: @lutayashafiq
Skype: lutaya5
Blog: lutayashafiq.com
http://www.fourcornersalliancegroup.com/?a=shafiqholmes

"The most beautiful people we have known are those who have known defeat,
known suffering, known struggle, known loss and have found their way out of
the depths. These persons have an appreciation, a sensitivity and an
understanding of life that fills them with compassion, gentleness and a
deep loving concern. Beautiful people do not just happen." - *Elisabeth
Kubler-Ross*

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Materialized views stability

[Blake Eggleston]

Hi Hannu,

There are more than a few committers that don't think MVs are currently 
suitable for production use. I'm not involved with MV development, so this may 
not be 100% accurate, but the problems as I understand them are: 

There's no way to determine if a view is out of sync with the base table.
If you do determine that a view is out of sync, the only way to fix it is to 
drop and rebuild the view.
There are liveness issues with updates being reflected in the view. 

Any one of these issues makes it difficult to recommend for general application 
development. I'd say that if you're not super familiar with their shortcomings 
and are confident you can fit your use case in them, you're probably better off 
not using them.

Thanks,

Blake

On October 2, 2017 at 6:55:52 AM, Hannu Kröger (hkro...@gmail.com) wrote:

Hello,

I have seen some discussions around Materialized Views and stability of that 
functionality.

There are some open issues around repairs:
https://issues.apache.org/jira/browse/CASSANDRA-13810?jql=project%20%3D%20CASSANDRA%20AND%20issuetype%20%3D%20Bug%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22%2C%20Testing%2C%20%22Ready%20to%20Commit%22%2C%20%22Awaiting%20Feedback%22)%20AND%20component%20%3D%20%22Materialized%20Views%22

Is it so that the current problems are mostly related to incremental repairs or 
are there also other major concerns why some people don’t consider them to be 
safe for production use?

Cheers,
Hannu

Creating materialized view for bulk selections

[Avi Levi]

I have metrics streaming in load of 10mil per minute.
I am using ttl in order to maintain the retention.
I have a cleanup process that needs to clean resources from other places
(not cassandra) when that key is no longer exists (i.e the it's ttl reach
it's due).

for that I thought to maintain table metric_retention((key), expire_time)
this table will keep getting inserts will either add new record or update

*insert into metric_retention (key, expire_time) values ([metric key],
[metric timestamp]+ttl)*

I thought of creating materialized view

CREATE MATERIALIZED VIEW metricsdb.mv_metric_retention AS
SELECT exp_time, mkey
FROM *metric_retention*
WHERE exp_time IS NOT NULL
PRIMARY KEY (exp_time, mkey)
WITH CLUSTERING ORDER BY (mkey ASC)

most of queries  will have the following shape
*"select key from metric_retention where Token(expire_time ) < Token(now)" *

of course once the resources are cleared I need also to delete those keys
from this table as well.

I do expect that reads will not be very often (couple of times a day) but
inserts/updates as I stated will suffer heavy load (10mil per minute) .

creating a materialized view over the metric_retaention table is the proper
approach here ?
are there any pitfalls to look out from ?

Thanks
Avi

Materialized views stability

[Hannu Kröger]

Hello,

I have seen some discussions around Materialized Views and stability of
that functionality.

There are some open issues around repairs:
https://issues.apache.org/jira/browse/CASSANDRA-13810?jql=project%20%3D%20CASSANDRA%20AND%20issuetype%20%3D%20Bug%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22%2C%20Testing%2C%20%22Ready%20to%20Commit%22%2C%20%22Awaiting%20Feedback%22)%20AND%20component%20%3D%20%22Materialized%20Views%22

Is it so that the current problems are mostly related to incremental
repairs or are there also other major concerns why some people don’t
consider them to be safe for production use?

Cheers,
Hannu

Re: Alter table gc_grace_seconds

[Varun Barala]

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsConfigureCompaction.html

As pe STCS:
*This strategy triggers a minor compaction when there are a number of
similar sized SSTables on disk as configured by the table subproperty,
min_threshold.*
In your case you have *min_threshold=4 and 3 Big sstable and one small
sstable.*
Either you can change min_threshold to three in your case or you can change
compaction strategy for this table.

Thanks!!

On Mon, Oct 2, 2017 at 4:48 PM, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello Justin,
>
>
>
> yes, but in real-world, hard to accomplish for high volume column families
> >= 3-digit GB. Even with the default of 10 days grace period, this may get
> a real challenge, to complete a full repair. J
>
>
>
> Possibly back again to the discussion that incremental repair has some
> flaws, full repair (-full option) in 3.0+ (2.2+?) isn’t behaving the same
> way as in 2.1 anymore, due to troubles when kicking off with –pr on several
> nodes at once.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Justin Cameron [mailto:jus...@instaclustr.com]
> *Sent:* Montag, 02. Oktober 2017 08:32
> *To:* user@cassandra.apache.org
> *Subject:* Re: Alter table gc_grace_seconds
>
>
>
> >> * You should not try on real clusters directly.
>
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
> See http://thelastpickle.com/blog/2016/07/27/about-deletes-
> and-tombstones.html for a good overview of the problem
>
>
>
>
>
> On Mon, 2 Oct 2017 at 16:51 Gábor Auth  wrote:
>
> Hi,
>
> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala 
> wrote:
>
> *select gc_grace_seconds from system_schema.tables where keyspace_name =
> 'keyspace' and table_name = 'number_item;*
>
>
>
> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>
>
> CREATE TABLE mat.number_item (
>nodeid uuid,
>type text,
>created timeuuid,
>value float,
>PRIMARY KEY (nodeid, type, created)
> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
>AND bloom_filter_fp_chance = 0.01
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>AND cdc = false
>AND comment = ''
>AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>AND crc_check_chance = 1.0
>AND dclocal_read_repair_chance = 0.1
>AND default_time_to_live = 0
>AND gc_grace_seconds = 3600
>AND max_index_interval = 2048
>AND memtable_flush_period_in_ms = 0
>AND min_index_interval = 128
>AND read_repair_chance = 0.0
>AND speculative_retry = '99PERCENTILE';
>
> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
> where keyspace_name = 'mat' and table_name = 'number_item';
>
> *gc_grace_seconds*
> --
> *3600*
>
> (1 rows)
>
>
> Bye,
> Gábor Auth
>
> --
>
> *Justin Cameron*
> Senior Software Engineer
>
>
>
> [image: Image removed by sender.] 
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>

Re: space left for compaction

[Avi Levi]

Got it. Thanks

On Mon, Oct 2, 2017 at 4:54 AM, Justin Cameron 
wrote:

> Hi Avi,
>
> Actually, in Thomas' example you would need an additional 100G of free
> disk space to complete the compaction, in the worst-case situation (the
> worst-case would be that neither input SSTable contains any overlapping
> data or tombstones, therefore the output SSTable would also be roughly
> 100G).
>
> STCS progressively compacts SSTables of similar size together, with the
> output being a single SSTable containing the data of the input SSTables.
>
> Eventually you may end up with some very large SSTables that combined will
> take up 50% of your total disk space. In order to compact those SSTables
> together, STCS requires an equal amount of free disk space, which would be
> the other (unused) 50% of your total disk space.
>
> Cheers,
> Justin
>
> On Mon, 2 Oct 2017 at 12:42 Avi Levi  wrote:
>
>> Hi Thomas ,
>> So IIUC in this case you should leave at least 50G for compaction  (half
>> of the sstables size). Is that makes sense?
>> Cheers
>> Avi
>>
>>
>> On Oct 1, 2017 11:39 AM, "Steinmaurer, Thomas" <
>> thomas.steinmau...@dynatrace.com> wrote:
>>
>> Hi,
>>
>>
>>
>> half of free space does not make sense. Imagine your SSTables need 100G
>> space and you have 20G free disk. Compaction won’t be able to do its job
>> with 10G.
>>
>>
>>
>> Half free of total disk makes more sense and is what you need for a major
>> compaction worst case.
>>
>>
>>
>> Thomas
>>
>>
>>
>> *From:* Peng Xiao [mailto:2535...@qq.com]
>> *Sent:* Samstag, 30. September 2017 10:21
>> *To:* user 
>> *Subject:* space left for compaction
>>
>>
>>
>> Dear All,
>>
>>
>>
>> As for STCS,datastax suggest us to keep half of the free space for
>> compaction,this is not strict,could anyone advise how many space should we
>> left for one node?
>>
>>
>>
>> Thanks,
>>
>> Peng Xiao
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freist
>> 
>> ädterstra
>> 
>> ße 313
>> 
>>
>>
>> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> 
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>

RE: Alter table gc_grace_seconds

[Steinmaurer, Thomas]

Hello Justin,

yes, but in real-world, hard to accomplish for high volume column families >= 
3-digit GB. Even with the default of 10 days grace period, this may get a real 
challenge, to complete a full repair. ☺

Possibly back again to the discussion that incremental repair has some flaws, 
full repair (-full option) in 3.0+ (2.2+?) isn’t behaving the same way as in 
2.1 anymore, due to troubles when kicking off with –pr on several nodes at once.

Regards,
Thomas

From: Justin Cameron [mailto:jus...@instaclustr.com]
Sent: Montag, 02. Oktober 2017 08:32
To: user@cassandra.apache.org
Subject: Re: Alter table gc_grace_seconds

>> * You should not try on real clusters directly.
>Why not? :)

It's highly recommended that you complete a full repair before the GC grace 
period expires, otherwise it's possible you could experience zombie data (i.e. 
data that was previously deleted coming back to life)

See http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html 
for a good overview of the problem


On Mon, 2 Oct 2017 at 16:51 Gábor Auth 
> wrote:
Hi,
On Mon, Oct 2, 2017 at 5:55 AM Varun Barala 
> wrote:
select gc_grace_seconds from system_schema.tables where keyspace_name = 
'keyspace' and table_name = 'number_item;

cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;

CREATE TABLE mat.number_item (
   nodeid uuid,
   type text,
   created timeuuid,
   value float,
   PRIMARY KEY (nodeid, type, created)
) WITH CLUSTERING ORDER BY (type ASC, created ASC)
   AND bloom_filter_fp_chance = 0.01
   AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
   AND cdc = false
   AND comment = ''
   AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
   AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
   AND crc_check_chance = 1.0
   AND dclocal_read_repair_chance = 0.1
   AND default_time_to_live = 0
   AND gc_grace_seconds = 3600
   AND max_index_interval = 2048
   AND memtable_flush_period_in_ms = 0
   AND min_index_interval = 128
   AND read_repair_chance = 0.0
   AND speculative_retry = '99PERCENTILE';

cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables where 
keyspace_name = 'mat' and table_name = 'number_item';

gc_grace_seconds
--
3600

(1 rows)

Bye,
Gábor Auth
--
Justin Cameron
Senior Software Engineer

[Image removed by sender.]

This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and 
Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally privileged 
information.  If you are not the intended recipient, do not copy or disclose 
its content, but please reply to this email immediately and highlight the error 
to the sender and then immediately delete the message.
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Alter table gc_grace_seconds

[Gábor Auth]

Hi,

On Mon, Oct 2, 2017 at 8:41 AM Varun Barala  wrote:

> Might be possible C* is not compacting the sstables [
> https://stackoverflow.com/questions/28437301/cassandra-not-compacting-sstables
> ]
>

Oh, the other CF-s in the same keyspace are compacted, but the
`number_item` not.

[cassandra@dc02-rack01-cass01
number_item-524bf49001d911e798503511c5f98764]$ ls -l
total 172704
drwxr-xr-x 4 cassandra cassandra 4096 Oct  1 10:43 backups
-rw-r--r-- 1 cassandra cassandra15227 Oct  2 01:15
mc-48278-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 46562318 Oct  2 01:15 mc-48278-big-Data.db
-rw-r--r-- 1 cassandra cassandra   10 Oct  2 01:15
mc-48278-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra  176 Oct  2 01:15
mc-48278-big-Filter.db
-rw-r--r-- 1 cassandra cassandra   119665 Oct  2 01:15
mc-48278-big-Index.db
-rw-r--r-- 1 cassandra cassandra 6368 Oct  2 01:15
mc-48278-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:15
mc-48278-big-Summary.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:15 mc-48278-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra20643 Oct  2 01:16
mc-48279-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 62681705 Oct  2 01:16 mc-48279-big-Data.db
-rw-r--r-- 1 cassandra cassandra   10 Oct  2 01:16
mc-48279-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra  176 Oct  2 01:16
mc-48279-big-Filter.db
-rw-r--r-- 1 cassandra cassandra   162571 Oct  2 01:16
mc-48279-big-Index.db
-rw-r--r-- 1 cassandra cassandra 6375 Oct  2 01:16
mc-48279-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:16
mc-48279-big-Summary.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:16 mc-48279-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra20099 Oct  2 01:16
mc-48280-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 62587865 Oct  2 01:16 mc-48280-big-Data.db
-rw-r--r-- 1 cassandra cassandra   10 Oct  2 01:16
mc-48280-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra  176 Oct  2 01:16
mc-48280-big-Filter.db
-rw-r--r-- 1 cassandra cassandra   158379 Oct  2 01:16
mc-48280-big-Index.db
-rw-r--r-- 1 cassandra cassandra 6375 Oct  2 01:16
mc-48280-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:16
mc-48280-big-Summary.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:16 mc-48280-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra   51 Oct  2 01:16
mc-48281-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra   50 Oct  2 01:16 mc-48281-big-Data.db
-rw-r--r-- 1 cassandra cassandra   10 Oct  2 01:16
mc-48281-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra  176 Oct  2 01:16
mc-48281-big-Filter.db
-rw-r--r-- 1 cassandra cassandra   20 Oct  2 01:16
mc-48281-big-Index.db
-rw-r--r-- 1 cassandra cassandra 4640 Oct  2 01:16
mc-48281-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:16
mc-48281-big-Summary.db
-rw-r--r-- 1 cassandra cassandra   92 Oct  2 01:16 mc-48281-big-TOC.txt

Now check both the list results. If they have some common sstables then we
> can say that C* is not compacting sstables.
>

Yes, exactly. How can I fix it?

Bye,
Gábor Auth

Re: Alter table gc_grace_seconds

[Gábor Auth]

Hi,

On Mon, Oct 2, 2017 at 8:32 AM Justin Cameron 
wrote:

> >> * You should not try on real clusters directly.
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>

It is a test cluster with test keyspaces. :)

Bye,
Gábor Auth

>

Re: Alter table gc_grace_seconds

[Varun Barala]

+1 Justing,

Might be possible C* is not compacting the sstables [
https://stackoverflow.com/questions/28437301/cassandra-not-compacting-sstables
].

You can this fact by doing below procedure:-

*Run this before compaction:-*
ls /var/lib/cassandra/data/mat/number_item-*/

Store result to some file [just to keep track of sstables]


*Run compaction:-*nodetool compact mat  number_item

*List directories again:-*
ls /var/lib/cassandra/data/mat/number_item-*/

Now check both the list results. If they have some common sstables then we
can say that C* is not compacting sstables.

Thanks!!

On Mon, Oct 2, 2017 at 2:32 PM, Justin Cameron 
wrote:

> >> * You should not try on real clusters directly.
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
> See http://thelastpickle.com/blog/2016/07/27/about-deletes-
> and-tombstones.html for a good overview of the problem
>
>
> On Mon, 2 Oct 2017 at 16:51 Gábor Auth  wrote:
>
>> Hi,
>>
>> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala 
>> wrote:
>>
>>> *select gc_grace_seconds from system_schema.tables where keyspace_name =
>>> 'keyspace' and table_name = 'number_item;*
>>>
>>
>> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>>
>>
>> CREATE TABLE mat.number_item (
>>nodeid uuid,
>>type text,
>>created timeuuid,
>>value float,
>>PRIMARY KEY (nodeid, type, created)
>> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
>>AND bloom_filter_fp_chance = 0.01
>>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>AND cdc = false
>>AND comment = ''
>>AND compaction = {'class': 
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>>AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>AND crc_check_chance = 1.0
>>AND dclocal_read_repair_chance = 0.1
>>AND default_time_to_live = 0
>>AND gc_grace_seconds = 3600
>>AND max_index_interval = 2048
>>AND memtable_flush_period_in_ms = 0
>>AND min_index_interval = 128
>>AND read_repair_chance = 0.0
>>AND speculative_retry = '99PERCENTILE';
>>
>> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
>> where keyspace_name = 'mat' and table_name = 'number_item';
>>
>> gc_grace_seconds
>> --
>> 3600
>>
>> (1 rows)
>>
>> Bye,
>> Gábor Auth
>>
> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> 
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>

Re: Alter table gc_grace_seconds

[Justin Cameron]

>> * You should not try on real clusters directly.
>Why not? :)

It's highly recommended that you complete a full repair before the GC grace
period expires, otherwise it's possible you could experience zombie data
(i.e. data that was previously deleted coming back to life)

See
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html for
a good overview of the problem


On Mon, 2 Oct 2017 at 16:51 Gábor Auth  wrote:

> Hi,
>
> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala 
> wrote:
>
>> *select gc_grace_seconds from system_schema.tables where keyspace_name =
>> 'keyspace' and table_name = 'number_item;*
>>
>
> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>
>
> CREATE TABLE mat.number_item (
>nodeid uuid,
>type text,
>created timeuuid,
>value float,
>PRIMARY KEY (nodeid, type, created)
> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
>AND bloom_filter_fp_chance = 0.01
>AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>AND cdc = false
>AND comment = ''
>AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>AND crc_check_chance = 1.0
>AND dclocal_read_repair_chance = 0.1
>AND default_time_to_live = 0
>AND gc_grace_seconds = 3600
>AND max_index_interval = 2048
>AND memtable_flush_period_in_ms = 0
>AND min_index_interval = 128
>AND read_repair_chance = 0.0
>AND speculative_retry = '99PERCENTILE';
>
> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
> where keyspace_name = 'mat' and table_name = 'number_item';
>
> gc_grace_seconds
> --
> 3600
>
> (1 rows)
>
> Bye,
> Gábor Auth
>
-- 


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.