Re: Cassandra Compactions Speed increase

[Jeff Jirsa]

Check your settings for concurrent compactors (in your yaml) and throughput 
(can check and set with nodetool) - both of those can be adjusted

Beyond that you need to give a bit more info to help us help you - are you CPU 
bound, disk bound, is the read load the limiter or is compaction? We need 
graphs or thread pool stats or something - there's a hundred or a thousand 
things to tune, and you've given almost no info for us to tell you which to 
look at first


-- 
Jeff Jirsa


> On Sep 15, 2017, at 9:29 PM, Ashish Pandey  wrote:
> 
> I haven't throttled cassandra compactions. My assumption is that because we
> are running lot of batch jobs which can updates or delete partition key,
> number of compactions are too many to finish in time which i see in
> nodetool compactionstats pending tasks.
> 
> Multiple Batch write (and delete) of partition keys. Each batch can have
> overlap of partition keys. High throughput read after couple of hours
> multiple batch writes
> 
>> On Fri, Sep 15, 2017 at 9:07 PM, Jeff Jirsa  wrote:
>> 
>> Do you have compaction throttled now?
>> What exactly takes 6-8 hours?
>> Are you writing all the keys at one time, then reading, then deleting them
>> all, or is it a constant stream of writes and reads?
>> 
>> 
>> --
>> Jeff Jirsa
>> 
>> 
>>> On Sep 15, 2017, at 8:40 PM, Ashish Pandey  wrote:
>>> 
>>> Hi All,
>>> 
>>> We are using cassandra 3.9 and experiencing read latency issues.
>>> 
>>> We are using LeveledCompactionStrategy after experiencing read latency
>>> issue with SizeTieredCompactionStrategy. For our use case, table keys are
>>> getting batch updated and batch delete of column values for keys. We
>> almost
>>> write to entire set of keys everyday.
>>> After switching to leveledcompactionstrategy read latency for the table
>>> has improved but we see that compaction takes lot of time (6-8hrs) and if
>>> the reads have to happen before compactions are reasonably done, latency
>> is
>>> severly impacted. Is there way to increase the speed of compactions as
>> the
>>> system resources are available in our use case for compaction to run
>> faster
>>> after batch update and delete are run?
>>> 
>>> This article
>>> https://www.datastax.com/dev/blog/cassandra-anti-patterns-
>> queues-and-queue-like-datasets
>>> suggests
>>> this is an anti-pattern for cassandra for such use cases, but I would
>> like
>>> to try if increase in compaction speed could help.
>>> 
>>> Thanks very much, appreciate responses.
>>> 
>>> Ashish
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Cassandra Compactions Speed increase

[Ashish Pandey]

I haven't throttled cassandra compactions. My assumption is that because we
are running lot of batch jobs which can updates or delete partition key,
number of compactions are too many to finish in time which i see in
nodetool compactionstats pending tasks.

Multiple Batch write (and delete) of partition keys. Each batch can have
overlap of partition keys. High throughput read after couple of hours
multiple batch writes

On Fri, Sep 15, 2017 at 9:07 PM, Jeff Jirsa  wrote:

> Do you have compaction throttled now?
> What exactly takes 6-8 hours?
> Are you writing all the keys at one time, then reading, then deleting them
> all, or is it a constant stream of writes and reads?
>
>
> --
> Jeff Jirsa
>
>
> > On Sep 15, 2017, at 8:40 PM, Ashish Pandey  wrote:
> >
> > Hi All,
> >
> > We are using cassandra 3.9 and experiencing read latency issues.
> >
> > We are using LeveledCompactionStrategy after experiencing read latency
> > issue with SizeTieredCompactionStrategy. For our use case, table keys are
> > getting batch updated and batch delete of column values for keys. We
> almost
> > write to entire set of keys everyday.
> > After switching to leveledcompactionstrategy read latency for the table
> > has improved but we see that compaction takes lot of time (6-8hrs) and if
> > the reads have to happen before compactions are reasonably done, latency
> is
> > severly impacted. Is there way to increase the speed of compactions as
> the
> > system resources are available in our use case for compaction to run
> faster
> > after batch update and delete are run?
> >
> > This article
> > https://www.datastax.com/dev/blog/cassandra-anti-patterns-
> queues-and-queue-like-datasets
> > suggests
> > this is an anti-pattern for cassandra for such use cases, but I would
> like
> > to try if increase in compaction speed could help.
> >
> > Thanks very much, appreciate responses.
> >
> > Ashish
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Cassandra Compactions Speed increase

[Jeff Jirsa]

Do you have compaction throttled now?
What exactly takes 6-8 hours? 
Are you writing all the keys at one time, then reading, then deleting them all, 
or is it a constant stream of writes and reads?


-- 
Jeff Jirsa


> On Sep 15, 2017, at 8:40 PM, Ashish Pandey  wrote:
> 
> Hi All,
> 
> We are using cassandra 3.9 and experiencing read latency issues.
> 
> We are using LeveledCompactionStrategy after experiencing read latency
> issue with SizeTieredCompactionStrategy. For our use case, table keys are
> getting batch updated and batch delete of column values for keys. We almost
> write to entire set of keys everyday.
> After switching to leveledcompactionstrategy read latency for the table
> has improved but we see that compaction takes lot of time (6-8hrs) and if
> the reads have to happen before compactions are reasonably done, latency is
> severly impacted. Is there way to increase the speed of compactions as the
> system resources are available in our use case for compaction to run faster
> after batch update and delete are run?
> 
> This article
> https://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
> suggests
> this is an anti-pattern for cassandra for such use cases, but I would like
> to try if increase in compaction speed could help.
> 
> Thanks very much, appreciate responses.
> 
> Ashish

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Cassandra Compactions Speed increase

[Ashish Pandey]

Hi All,

We are using cassandra 3.9 and experiencing read latency issues.

We are using LeveledCompactionStrategy after experiencing read latency
issue with SizeTieredCompactionStrategy. For our use case, table keys are
getting batch updated and batch delete of column values for keys. We almost
write to entire set of keys everyday.
 After switching to leveledcompactionstrategy read latency for the table
has improved but we see that compaction takes lot of time (6-8hrs) and if
the reads have to happen before compactions are reasonably done, latency is
severly impacted. Is there way to increase the speed of compactions as the
system resources are available in our use case for compaction to run faster
after batch update and delete are run?

This article
https://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
suggests
this is an anti-pattern for cassandra for such use cases, but I would like
to try if increase in compaction speed could help.

Thanks very much, appreciate responses.

Ashish

Re: Review request: CASSANDRA-13799

[Jeff Jirsa]

On 2017-09-06 00:58, Malcolm Taylor  wrote: 
> Can someone please review my patch on CASSANDRA-13799 ? The issue fixes
> several alerts raised by lgtm.com, which runs a deep analysis of the code
> in trunk. Some of the alerts refer to clear bugs, while others are
> straightforward improvements such as renaming a variable where the current
> name may cause confusion. All the fixes are very small changes, so it
> should be relatively easy to review. I have only generated a patch for
> trunk, because that is the branch that is analysed by lgtm.com, but please
> let me know if I should provide additional information.
> Thanks,
> Malcolm
> 

I'll try to take it soon'ish.



-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

RE: question on the code formatter

[Tyagi, Preetika]

Thank you for the info! I tried it and it worked as expected.


-Original Message-
From: Murukesh Mohanan [mailto:murukesh.moha...@gmail.com] 
Sent: Thursday, September 14, 2017 11:10 PM
To: dev@cassandra.apache.org
Subject: Re: question on the code formatter

The wiki seems to be outdated. See
https://github.com/apache/cassandra/blob/trunk/doc/source/development/ide.rst
:


> The project generated by the ant task ``generate-idea-files`` contains
nearly everything
> you need to debug Cassandra and execute unit tests.
>
> * Run/debug defaults for JUnit
> * Run/debug configuration for Cassandra daemon
> * License header for Java source files
> * Cassandra code style
> * Inspections

You can just run `generate-idea-files` and then open the project in IDEA.
Code style settings should be automatically picked up by IDEA.

On Fri, 15 Sep 2017 at 14:46 Tyagi, Preetika 
wrote:

> Hi all,
>
> I was trying to configure the Cassandra code formatter and downloaded 
> IntelliJ-codestyle.jar from this link:
> https://wiki.apache.org/cassandra/CodeStyle
>
> After extracting this JAR, I was able to import 
> codestyle/Default_1_.xml into my project and formatting seemed to work.
>
> However, I'm wondering what options/code.style.schemes.xml file is 
> exactly used for? Could anyone please give me an idea if I need to 
> configure this as well?
>
> Thanks,
> Preetika
>
>
> --

Murukesh Mohanan,
Yahoo! Japan

RE: Proposal: Closing old, unable-to-repro JIRAs

[Tyagi, Preetika]

+ 1 

This is a good idea.

-Original Message-
From: beggles...@apple.com [mailto:beggles...@apple.com] 
Sent: Friday, September 15, 2017 8:29 AM
To: dev@cassandra.apache.org
Subject: Re: Proposal: Closing old, unable-to-repro JIRAs

+1 to that


On September 14, 2017 at 4:50:54 PM, Jeff Jirsa (jji...@gmail.com) wrote:

There's a number of JIRAs that are old - sometimes very old - that represent 
bugs that either don't exist in modern versions, or don't have sufficient 
information for us to repro, but the reporter has gone away. 

Would anyone be offended if I start tagging these with the label 
'UnableToRepro' or 'Unresponsive' and start a 30 day timer to close them? 
Anyone have a better suggestion? 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposal: Closing old, unable-to-repro JIRAs

[Blake Eggleston]

+1 to that


On September 14, 2017 at 4:50:54 PM, Jeff Jirsa (jji...@gmail.com) wrote:

There's a number of JIRAs that are old - sometimes very old - that 
represent bugs that either don't exist in modern versions, or don't have 
sufficient information for us to repro, but the reporter has gone away. 

Would anyone be offended if I start tagging these with the label 
'UnableToRepro' or 'Unresponsive' and start a 30 day timer to close them? 
Anyone have a better suggestion? 

materialise view creation taking too much time with 12 million data set

[Pankaj Gajjar]

Hi everyone,

recently we are in big trouble while creating the materialise view(mv) on
12 million data set and then we wait for almost 72 hours.

with same data model with half of data set 7 million it took only 10-15
mins to prepare same mv.

please welcome any input on this..


-- 
Regards,

Pankaj G.
ContentSphere

Re: question on the code formatter

[Murukesh Mohanan]

The wiki seems to be outdated. See
https://github.com/apache/cassandra/blob/trunk/doc/source/development/ide.rst
:


> The project generated by the ant task ``generate-idea-files`` contains
nearly everything
> you need to debug Cassandra and execute unit tests.
>
> * Run/debug defaults for JUnit
> * Run/debug configuration for Cassandra daemon
> * License header for Java source files
> * Cassandra code style
> * Inspections

You can just run `generate-idea-files` and then open the project in IDEA.
Code style settings should be automatically picked up by IDEA.

On Fri, 15 Sep 2017 at 14:46 Tyagi, Preetika 
wrote:

> Hi all,
>
> I was trying to configure the Cassandra code formatter and downloaded
> IntelliJ-codestyle.jar from this link:
> https://wiki.apache.org/cassandra/CodeStyle
>
> After extracting this JAR, I was able to import codestyle/Default_1_.xml
> into my project and formatting seemed to work.
>
> However, I'm wondering what options/code.style.schemes.xml file is exactly
> used for? Could anyone please give me an idea if I need to configure this
> as well?
>
> Thanks,
> Preetika
>
>
> --

Murukesh Mohanan,
Yahoo! Japan