Re: Reg:- CQL SOLR Query Not gives result

[Jonathan Haddad]

This is a question for datastax support, not the Apache mailing list. Folks
here are more than happy to help with open source, Apache Cassandra
questions, if you've got one.
On Thu, May 11, 2017 at 9:06 PM @Nandan@ 
wrote:

> Hi ,
>
> In my table, I am having few records and implemented SOLR for partial
> search but not able to retrieve data.
>
> SELECT * from revall_book_by_title where solr_query = 'language:中';
> SELECT * from revall_book_by_title where solr_query = 'language:中*';
>
> None of them are working.
> Any suggestions.
>

Reg:- CQL SOLR Query Not gives result

[@Nandan@]

Hi ,

In my table, I am having few records and implemented SOLR for partial
search but not able to retrieve data.

SELECT * from revall_book_by_title where solr_query = 'language:中';
SELECT * from revall_book_by_title where solr_query = 'language:中*';

None of them are working.
Any suggestions.

Re: cassandra 3.10

[Anthony Grasso]

Hi Dhruva,

There are definitely some performance improvements to Storage Engine in
Cassandra 3.10 which make it worth the upgrade. Note that Cassandra 3.11
has further bug fixes and it may be worth considering a migration to that
version.

Regarding the issue of building a Cassandra 3.10 RPM, it sounds like the
team have built their own custom spec file? Has the team looked at using
the project spec file and associated instructions in Apache Cassandra
GitHub mirror?

https://github.com/apache/cassandra/tree/cassandra-3.10/redhat

Kind regards,
Anthony


On 11 May 2017 at 14:20, Gopal, Dhruva  wrote:

> Hi –
>
>   We’re currently on 3.9 and have been told that Cassandra 3.10 is a more
> stable version to be on. We’ve been using the datastax-ddc rpms in our
> production and dev environments (on 3.9) and it appears there is no 3.10
> rpm version out yet. We tried to build our own rpm (our devops processes
> use rpms, so changing to using tarballs is not easily done) and found that
> the build process fails (to do with the byteman-3.0.3 jar) that we manage
> to patch and get working (with rpmbuild). My concerns/questions are these:
>
> -  Is the 3.10 version actually stable enough given that the
> build failed (we obtained the source from this location:
> http://apache.mirrors.tds.net/cassandra/3.10/apache-
> cassandra-3.10-src.tar.gz) and used the attached patch file for byteman
> during the build process)?
>
> -  Are there any other issues with the binaries that we need to
> be aware of (other patches)?
>
>
>
> I’m concerned that there may be other issues and that we really won’t know
> since we’re not Cassandra experts, so looking for feedback from this group
> on whether we should just stay with 3.9 or if it’s safe to proceed with
> this approach. I can share the spec file and patch files that we’ve setup
> for the build process, if desired.
>
>
>
>
>
> Regards,
>
> *DHRUVA GOPAL*
>
> *sr. MANAGER, ENGINEERING*
>
> *REPORTING, ANALYTICS AND BIG DATA*
>
> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>
> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>
> *UNITED STATES*
>
> *dhruva.go...@aspect.com  *
>
> *aspect.com *
>
> [image: escription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Nodes stopping

[Daniel Steuernol]

I'm switching the instances to machines with 61G of RAM, in this case would you still recommend using 8G of heap space?Here is a gist of my heap settings from jvm.optionshttps://gist.github.com/dlsteuer/40e80280029897e6bb5fd12f2a86cbbe
  

On May 11 2017, at 3:08 pm, Alain RODRIGUEZ  wrote:


  For some context, I'm trying to get regular repairs going but am having issues with it.You're not the only one, repairs are a real concern for many people.For what it is worth, my team is actively working on this project initiated at Spotify: https://github.com/thelastpickle/cassandra-reaper.C*heers,---Alain Rodriguez - @arodream - al...@thelastpickle.comFranceThe Last Pickle - Apache Cassandra Consultinghttp://www.thelastpickle.com2017-05-11 23:04 GMT+01:00 Alain RODRIGUEZ :Hi Daniel,Could you paste the exact GC options in use?Also 30 GB is not much. I would not use more than 8 GB for the JVM and probably CMS in those conditions for what it is worth. The thing is if memtables, bloom filter, caches, indexes, etc are off heap, then you probably ran out of Native memory. In any case it is good to have some space for page cache.As a reminder you can try new GC option in a canary node, see how it goes.C*heers,---Alain Rodriguez - @arodream - al...@thelastpickle.comFranceThe Last Pickle - Apache Cassandra Consultinghttp://www.thelastpickle.com2017-05-11 22:29 GMT+01:00 Daniel Steuernol :Thank you, it's an Out of memory crash according to dmesg. I have the heap size set to 15G in the jvm.options for cassandra, and there is 30G on the machine.
  

On May 11 2017, at 2:22 pm, Cogumelos Maravilha  wrote:


  
  

  
  
Have a look at dmesg. It have already happened to me regarding
  type i instances at AWS.

On 11-05-2017 22:17, Daniel Steuernol
  wrote:

I had 2 nodes go down today, here is the ERRORs from
  the system log on both nodes
  
https://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218

For some context, I'm trying to get regular repairs going
  but am having issues with it.


  

On May 11 2017, at 2:10 pm, Cogumelos Maravilha
 wrote: 

  
  Can you grep ERROR system.log
  
  
  On 11-05-2017 21:52, Daniel Steuernol wrote:
  
  There is nothing in the system log
about it being drained or shutdown, I'm not sure how else it
would be pre-empted. No one else on the team is on the
servers and I haven't been shutting them down. There also is
no java memory dump on the server either. It appears that
the process just died.

 
  On May 11 2017, at 1:36 pm, Varun Gupta 
  wrote: 
  

  What do you mean by "no obvious error in the
logs", do you see node was drained or shutdown. Are
you sure, no other process is calling nodetool drain
or shutdown, OR pre-empting cassandra process?


  On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol

wrote:

  I have a 6 node cassandra cluster running, and
  frequently a node will go down with no obvious
  error in the logs. This is starting to happen
  quite often, almost daily now. Any suggestions on
  how to track down what is causing the node to
  stop? -
  To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
  For additional commands, e-mail: user-h...@cassandra.apache.org

  
  

  

- To
unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
  
  

  
-
  To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
  For additional commands, e-mail: user-h...@cassandra.apache.org


  



  

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, 

Re: Nodes stopping

[Alain RODRIGUEZ]

Hi Daniel,

Could you paste the exact GC options in use?

Also 30 GB is not much. I would not use more than 8 GB for the JVM and
probably CMS in those conditions for what it is worth. The thing is if
memtables, bloom filter, caches, indexes, etc are off heap, then you
probably ran out of Native memory. In any case it is good to have some
space for page cache.

As a reminder you can try new GC option in a canary node, see how it goes.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-05-11 22:29 GMT+01:00 Daniel Steuernol :

> Thank you, it's an Out of memory crash according to dmesg. I have the heap
> size set to 15G in the jvm.options for cassandra, and there is 30G on the
> machine.
>
>
>
> On May 11 2017, at 2:22 pm, Cogumelos Maravilha <
> cogumelosmaravi...@sapo.pt> wrote:
>
>> Have a look at dmesg. It have already happened to me regarding type i
>> instances at AWS.
>>
>> On 11-05-2017 22:17, Daniel Steuernol wrote:
>>
>> I had 2 nodes go down today, here is the ERRORs from the system log on
>> both nodes
>> https://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218
>> For some context, I'm trying to get regular repairs going but am having
>> issues with it.
>>
>>
>> On May 11 2017, at 2:10 pm, Cogumelos Maravilha
>>   wrote:
>>
>> Can you grep ERROR system.log
>>
>> On 11-05-2017 21:52, Daniel Steuernol wrote:
>>
>> There is nothing in the system log about it being drained or shutdown,
>> I'm not sure how else it would be pre-empted. No one else on the team is on
>> the servers and I haven't been shutting them down. There also is no java
>> memory dump on the server either. It appears that the process just died.
>>
>>
>> On May 11 2017, at 1:36 pm, Varun Gupta 
>>  wrote:
>>
>>
>> What do you mean by "no obvious error in the logs", do you see node was
>> drained or shutdown. Are you sure, no other process is calling nodetool
>> drain or shutdown, OR pre-empting cassandra process?
>>
>> On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol 
>> wrote:
>>
>>
>> I have a 6 node cassandra cluster running, and frequently a node will go
>> down with no obvious error in the logs. This is starting to happen quite
>> often, almost daily now. Any suggestions on how to track down what is
>> causing the node to stop? --
>> --- To unsubscribe, e-mail:
>> user-unsubscr...@cassandra.apache.org For additional commands, e-mail:
>> user-h...@cassandra.apache.org
>>
>>
>> - To
>> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
>> additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>> - To
>> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
>> additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>> - To
> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional
> commands, e-mail: user-h...@cassandra.apache.org
>

Re: Drop tables takes too long

[Alain RODRIGUEZ]

Hi

We were trying to overcome OOM crashes.


Fair enough :-).

We changed settings to default on one node. GC times became about two times
> smaller on that node.
>

That's encouraging! Looks like even if the number of tables is really high,
there is still space for optimization. Have you made the change on the
entire cluster by now? How are things going?

Also you can continue to play around with a canary node to find the sweet
spot, the right tuning for your use case and hardware. Taking a day to do
this is sometimes very worth it ;-).

Number of sstables is not constant. During about 2.5 hours number of tables
> was changed for 26 tables, e.g. [4, 4, 4, 4, 4, 4, 4, 4, 4, 4] => [6, 6,
> 6, 6, 6, 6, 6, 6, 6, 6] or [4, 7, 7, 7, 5, 5, 4, 4, 4, 4] => [4, 4, 4, 4,
> 4, 4, 4, 4, 4, 4] (each list is number of sstables on each of 10 nodes for
> one table).
> Number of sstables is balanced for almost all tables. But for some tables
> number of sstables is not really balanced, like [11, 2, 4, 4, 4, 2, 2, 2,
> 2, 5] or [439, 558, 346, 521, 490, 553, 500, 515, 522, 495]
>

Sounds like reasonable numbers of SSTables. Imbalances are not that big. I
would say compaction is running normally

 We run incremental repairs

 We use LCS for all tables and MVs. We don't do manual compactions (or
> trigger any anti-compactions)


If you run incremental repairs, it means you trigger anti-compactions.
Actually the first repair might completely have produce the increase in the
number of SSTables. If it was not the first time, you might have run into
one of the corner cases still not fixed on incremental repairs.

Also, using LCS for such a high number of tables is probably putting some
pressure on disks. Is it a real need? Would STCS or TWCS not be a better
fit on some of the table? That being said, compactions look good. Are you
seeing pending compactions under standard conditions or on peak hours?

 What number of MemtableFlushWriter are you using?
>
> We do not specify if, so default is use
>

https://github.com/apache/cassandra/blob/cassandra-3.10/conf/cassandra.yaml#L538

So it is 2. If disks IO is not a bottleneck you might want to consider
increasing this number to 4, it should be a safe value. To know if Flush
Writers is an issue, use 'watch -d nodetool tpstats' and see if there are
any 'FlushWriter' pending threads.

One column for each node
> READ:   2609 7   0   0   2   1   2   0   1   1
>

The only non negligible value is about read being dropped and on only one
node. If this value is not growing anymore, you might have faced a punctual
issue.

This cluster looks relatively healthy excepted for GC activity (that can
explain read drops). I would persevere on GC tuning, continuing to monitor
things you have been sharing with us so far, to see how it evolves. Having
a closer look at repairs impact might be worth it as well. Good luck!

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-05-08 14:21 GMT+01:00 Bohdan Tantsiura :

> Hi,
>
> > Why did you move from defaults that much?
> We were trying to overcome OOM crashes.
>
> > Would you consider giving defaults a try on a canary node and monitor /
> compare GC times to other nodes?
> We changed settings to default on one node. GC times became about two
> times smaller on that node.
>
> > What do you mean from time to time? For how long are this task pending,
> what frequency is this happening?
>  CompactionExecutor pending tasks appeared on nodes once during 3 or more
> hours. Tasks were pending during about 5-15 minutes.
>
> > Is the number of sstables constant and balanced between nodes?
> Number of sstables is not constant. During about 2.5 hours number of
> tables was changed for 26 tables, e.g. [4, 4, 4, 4, 4, 4, 4, 4, 4, 4] =>
> [6, 6, 6, 6, 6, 6, 6, 6, 6, 6] or [4, 7, 7, 7, 5, 5, 4, 4, 4, 4] => [4,
> 4, 4, 4, 4, 4, 4, 4, 4, 4] (each list is number of sstables on each of 10
> nodes for one table).
> Number of sstables is balanced for almost all tables. But for some tables
> number of sstables is not really balanced, like [11, 2, 4, 4, 4, 2, 2, 2,
> 2, 5] or [439, 558, 346, 521, 490, 553, 500, 515, 522, 495]
>
> > Also do you run full or incremental repairs?
> We run incremental repairs
>
> > Do you use LCS or do some manual compactions (or trigger any
> anti-compactions)?
> We use LCS for all tables and MVs. We don't do manual compactions (or
> trigger any anti-compactions)
>
> > How is CPU doing, is there any burst in CPU that could be related to
> these errors?
> Unfortunately, stats for period when there were InternalResponseState
> pending tasks is lost.
>
> > What number of MemtableFlushWriter are you using
> We do not specify if, so default is used
>
> > What tasks were dropped
> One column for each node
> READ:   2609 7   0   0   2   1   2   0   1   1
> HINT:   00   0   0   

Re: Nodes stopping

[Daniel Steuernol]

Thank you, it's an Out of memory crash according to dmesg. I have the heap size set to 15G in the jvm.options for cassandra, and there is 30G on the machine.
  

On May 11 2017, at 2:22 pm, Cogumelos Maravilha  wrote:


  
  

  
  
Have a look at dmesg. It have already happened to me regarding
  type i instances at AWS.

On 11-05-2017 22:17, Daniel Steuernol
  wrote:

I had 2 nodes go down today, here is the ERRORs from
  the system log on both nodes
  
https://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218

For some context, I'm trying to get regular repairs going
  but am having issues with it.


  

On May 11 2017, at 2:10 pm, Cogumelos Maravilha
 wrote: 

  
  Can you grep ERROR system.log
  
  
  On 11-05-2017 21:52, Daniel Steuernol wrote:
  
  There is nothing in the system log
about it being drained or shutdown, I'm not sure how else it
would be pre-empted. No one else on the team is on the
servers and I haven't been shutting them down. There also is
no java memory dump on the server either. It appears that
the process just died.

 
  On May 11 2017, at 1:36 pm, Varun Gupta 
  wrote: 
  

  What do you mean by "no obvious error in the
logs", do you see node was drained or shutdown. Are
you sure, no other process is calling nodetool drain
or shutdown, OR pre-empting cassandra process?


  On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol

wrote:

  I have a 6 node cassandra cluster running, and
  frequently a node will go down with no obvious
  error in the logs. This is starting to happen
  quite often, almost daily now. Any suggestions on
  how to track down what is causing the node to
  stop? -
  To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
  For additional commands, e-mail: user-h...@cassandra.apache.org

  
  

  

- To
unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
  
  

  
-
  To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
  For additional commands, e-mail: user-h...@cassandra.apache.org


  



  

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Nodes stopping

[Daniel Steuernol]

I had 2 nodes go down today, here is the ERRORs from the system log on both nodeshttps://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218For some context, I'm trying to get regular repairs going but am having issues with it.
  

On May 11 2017, at 2:10 pm, Cogumelos Maravilha  wrote:


  
  

  
  
Can you grep ERROR system.log


On 11-05-2017 21:52, Daniel Steuernol
  wrote:

There is nothing in the system log about it being
  drained or shutdown, I'm not sure how else it would be pre-empted.
  No one else on the team is on the servers and I haven't been
  shutting them down. There also is no java memory dump on the
  server either. It appears that the process just died.
  
  
  

On May 11 2017, at 1:36 pm, Varun Gupta 
wrote: 

  
What do you mean by "no obvious error in the logs", do
  you see node was drained or shutdown. Are you sure, no
  other process is calling nodetool drain or shutdown, OR
  pre-empting cassandra process?
  
  
On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol 
  wrote:
  
I have a 6 node cassandra cluster running, and
frequently a node will go down with no obvious error in
the logs. This is starting to happen quite often, almost
daily now. Any suggestions on how to track down what is
causing the node to stop?
-
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
  


  

  
-
  To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
  For additional commands, e-mail: user-h...@cassandra.apache.org


  



  

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Nodes stopping

[Cogumelos Maravilha]

Can you grep ERROR system.log


On 11-05-2017 21:52, Daniel Steuernol wrote:
> There is nothing in the system log about it being drained or shutdown,
> I'm not sure how else it would be pre-empted. No one else on the team
> is on the servers and I haven't been shutting them down. There also is
> no java memory dump on the server either. It appears that the process
> just died.
>
>
> On May 11 2017, at 1:36 pm, Varun Gupta  wrote:
>
>
> What do you mean by "no obvious error in the logs", do you see
> node was drained or shutdown. Are you sure, no other process is
> calling nodetool drain or shutdown, OR pre-empting cassandra process?
>
> On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol
> > wrote:
>
>
> I have a 6 node cassandra cluster running, and frequently a
> node will go down with no obvious error in the logs. This is
> starting to happen quite often, almost daily now. Any
> suggestions on how to track down what is causing the node to
> stop?
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>  For additional
> commands, e-mail: user-h...@cassandra.apache.org
>  
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
> additional commands, e-mail: user-h...@cassandra.apache.org 

Re: Nodes stopping

[Varun Gupta]

Maybe this article helps you.

http://stackoverflow.com/questions/26285133/who-sends-a-sigkill-to-my-process-mysteriously-on-ubuntu-server

On Thu, May 11, 2017 at 1:52 PM, Daniel Steuernol 
wrote:

> There is nothing in the system log about it being drained or shutdown, I'm
> not sure how else it would be pre-empted. No one else on the team is on the
> servers and I haven't been shutting them down. There also is no java memory
> dump on the server either. It appears that the process just died.
>
>
>
> On May 11 2017, at 1:36 pm, Varun Gupta  wrote:
>
>>
>> What do you mean by "no obvious error in the logs", do you see node was
>> drained or shutdown. Are you sure, no other process is calling nodetool
>> drain or shutdown, OR pre-empting cassandra process?
>>
>> On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol 
>> wrote:
>>
>>
>> I have a 6 node cassandra cluster running, and frequently a node will go
>> down with no obvious error in the logs. This is starting to happen quite
>> often, almost daily now. Any suggestions on how to track down what is
>> causing the node to stop? --
>> --- To unsubscribe, e-mail:
>> user-unsubscr...@cassandra.apache.org For additional commands, e-mail:
>> user-h...@cassandra.apache.org
>>
>>
>>

Re: AWS Cassandra backup/Restore tools

[Manikandan Srinivasan]

Blake is correct. OpsCenter 6.0 and up doesn't work with OSS C*. @Nitan: We
have made some substantial changes to the Opscenter 6.1 backup service,
specifically when it comes to S3 backups. Having said this, I am not going
to be sale-sy here. If folks need some help or need more clarity to know
more about these improvements, please send me an email directly:
msriniva...@datastax.com

Regards
Mani

On Thu, May 11, 2017 at 1:54 PM, Nitan Kainth  wrote:

> Also , Opscenter backup/restore does not work for large databases
>
> Sent from my iPhone
>
> On May 11, 2017, at 3:41 PM, Blake Eggleston  wrote:
>
> OpsCenter 6.0 and up don't work with Cassandra.
>
> On May 11, 2017 at 12:31:08 PM, cass savy (casss...@gmail.com) wrote:
>
> AWS Backup/Restore process/tools for C*/DSE C*:
>
> Has anyone used Opscenter 6.1 backup tool to backup/restore data for
> larger datasets online ?
>
> If yes, did you run into issues using that tool to backup/restore data in
> PROD that caused any performance or any other impact to the cluster?
>
> If no, what are other tools that people have used or recommended for
> backup and restore of Cassandra keyspaces?
>
> Please advice.
>
>
>


-- 
Regards,

Manikandan Srinivasan

Director, Product Management| +1.408.887.3686 |
manikandan.sriniva...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]

Re: AWS Cassandra backup/Restore tools

[Nitan Kainth]

Also , Opscenter backup/restore does not work for large databases 

Sent from my iPhone

> On May 11, 2017, at 3:41 PM, Blake Eggleston  wrote:
> 
> OpsCenter 6.0 and up don't work with Cassandra.
> 
>> On May 11, 2017 at 12:31:08 PM, cass savy (casss...@gmail.com) wrote:
>> 
>> AWS Backup/Restore process/tools for C*/DSE C*:
>> 
>> Has anyone used Opscenter 6.1 backup tool to backup/restore data for larger 
>> datasets online ?
>> 
>> If yes, did you run into issues using that tool to backup/restore data in 
>> PROD that caused any performance or any other impact to the cluster?
>> 
>> If no, what are other tools that people have used or recommended for backup 
>> and restore of Cassandra keyspaces?
>> 
>> Please advice.
>> 
>> 

Re: Nodes stopping

[Daniel Steuernol]

There is nothing in the system log about it being drained or shutdown, I'm not sure how else it would be pre-empted. No one else on the team is on the servers and I haven't been shutting them down. There also is no java memory dump on the server either. It appears that the process just died.
  

On May 11 2017, at 1:36 pm, Varun Gupta  wrote:


  What do you mean by "no obvious error in the logs", do you see node was drained or shutdown. Are you sure, no other process is calling nodetool drain or shutdown, OR pre-empting cassandra process?On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol  wrote:I have a 6 node cassandra cluster running, and frequently a node will go down with no obvious error in the logs. This is starting to happen quite often, almost daily now. Any suggestions on how to track down what is causing the node to stop?

-
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




  

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra Snapshots and directories

[Varun Gupta]

I did not get your question completely, with "snapshot files are mixed with
files and backup files".

When you call nodetool snapshot, it will create a directory with snapshot
name if specified or current timestamp at
/data///backup/. This directory will
have all sstables, metadata files and schema.cql (if using 3.0.9 or higher).


On Thu, May 11, 2017 at 2:37 AM, Daniel Hölbling-Inzko <
daniel.hoelbling-in...@bitmovin.com> wrote:

> Hi,
> I am going through this guide to do backup/restore of cassandra data to a
> new cluster:
> http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_
> snapshot_restore_t.html#task_ds_cmf_11r_gk
>
> When creating a snapshot I get the snapshot files mixed in with the normal
> data files and backup files, so it's all over the place and very hard
> (especially with lots of tables per keyspace) to transfer ONLY the snapshot.
> (Mostly since there is a snapshot directory per table..)
>
> Am I missing something or is there some arcane shell command that filters
> out only the snapshots?
> Because this way it's much easier to just backup the whole data directory.
>
> greetings Daniel
>

Re: AWS Cassandra backup/Restore tools

[Blake Eggleston]

OpsCenter 6.0 and up don't work with Cassandra.

On May 11, 2017 at 12:31:08 PM, cass savy (casss...@gmail.com) wrote:

AWS Backup/Restore process/tools for C*/DSE C*:

Has anyone used Opscenter 6.1 backup tool to backup/restore data for larger 
datasets online ?

If yes, did you run into issues using that tool to backup/restore data in PROD 
that caused any performance or any other impact to the cluster?

If no, what are other tools that people have used or recommended for backup and 
restore of Cassandra keyspaces?

Please advice.

Re: Nodes stopping

[Varun Gupta]

What do you mean by "no obvious error in the logs", do you see node was
drained or shutdown. Are you sure, no other process is calling nodetool
drain or shutdown, OR pre-empting cassandra process?

On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol 
wrote:

>
> I have a 6 node cassandra cluster running, and frequently a node will go
> down with no obvious error in the logs. This is starting to happen quite
> often, almost daily now. Any suggestions on how to track down what is
> causing the node to stop? --
> --- To unsubscribe, e-mail:
> user-unsubscr...@cassandra.apache.org For additional commands, e-mail:
> user-h...@cassandra.apache.org

Re: repair question (-dc option)

[Varun Gupta]

If there was no node down during that period, and you are using
LOCAL_QUORUM read/write, then yes above command works.

On Thu, May 11, 2017 at 11:59 AM, Gopal, Dhruva 
wrote:

> Hi –
>
>   I have a question on running a repair after bringing up a node that was
> down (brought down gracefully) for a few days within a data center. Can we
> just run nodetool repair –dc  on a single node (within that DC –
> specifically the downed node, after it is brought online) and have that
> entire DC repaired?
>
>
>
> Regards,
>
> Dhruva
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

Re: Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

ok thank you

On Thu, May 11, 2017 at 1:11 PM, Jeff Jirsa  wrote:

> No, it's not expected, but it's pretty obvious from reading the code
> what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-
> 13526
>
>
>
>
>
> On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Yes I have many keyspaces which are not spread across all the data
>> centers(expected by design).
>> In this case, is this the expected behavior cleanup will not work for all
>> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
>> versions?
>>
>> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
>> keyspacename"
>>
>> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa  wrote:
>>
>>> If you didn't explicitly remove a keyspace from one of your datacenters,
>>> the next most likely cause is that you have one keyspace that's NOT
>>> replicated to one of the datacenters. You can work around this by running
>>> 'nodetool cleanup ' on all of your other keyspaces individually,
>>> skipping the one that isn't replicated to that datacenter.
>>>
>>>
>>>
>>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 Thanks Jeff,

 I have a C* cluster spread across multiple datacenter.
 reason for cleanup : I added multiple nodes to cluster and need to run
 cleanup on old nodes so that the redundant data is cleaned-up.

 On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:

>
>
> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
> > Hello,
> >
> > I am running into an issue where *nodetool cleanup *fails to cleanup
> data.
> > We are running 2.1.16 version of Cassandra.
> >
> >
> > [user@host ~]$ nodetool cleanup
> > Aborted cleaning up atleast one column family in keyspace user, check
> > server logs for more information.
> > Aborted cleaning up atleast one column family in keyspace org, check
> server
> > logs for more information.
> > error: nodetool failed, check server logs
> > -- StackTrace --
> > java.lang.RuntimeException: nodetool failed, check server logs
> > at
> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool
> .java:294)
> > at org.apache.cassandra.tools.Nod
> eTool.main(NodeTool.java:206)
> >
> > *Logs:*
> >
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> > CompactionManager.java:415 - Cleanup cannot run before a node has
> joined
> > the ring
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> > CompactionManager.java:415 - Cleanup cannot run before a node has
> joined
> > the ring
> >
> > All the nodes in the cluster are up and running. We tried doing a
> rolling
> > restart of all nodes and no luck.
> >
> > After looking at the Cassandra JIRA :
> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like
> the issue
> > is fixed with 2.2.6 and 3.0 version.
> > While we have plans to upgrade to the latest versions(which might
> take
> > longer time), does any know if there is any work around to mitigate
> the
> > issue?
> >
>
> Are you running multiple datacenters, and you just removed a specific
> datacenter from a keyspace (and that's why you want to run cleanup)? If
> that's the case, I fear the fix for 10991 isn't really going to fix it in
> the way you hope (we may need a follow-up jira). What you'll almost
> certainly need to do is remove the data on disk manually, which is quite
> unfortunate as it'll require you to 
> stop+delete-data-for-that-keyspace+start
> each node in the datacenter for which you removed replication.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

>>>
>>
>

Nodes stopping

[Daniel Steuernol]

I have a 6 node cassandra cluster running, and frequently a node will go down with no obvious error in the logs. This is starting to happen quite often, almost daily now. Any suggestions on how to track down what is causing the node to stop?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Nodetool cleanup doesn't work

[Jeff Jirsa]

No, it's not expected, but it's pretty obvious from reading the code
what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-13526





On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Yes I have many keyspaces which are not spread across all the data
> centers(expected by design).
> In this case, is this the expected behavior cleanup will not work for all
> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
> versions?
>
> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
> keyspacename"
>
> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa  wrote:
>
>> If you didn't explicitly remove a keyspace from one of your datacenters,
>> the next most likely cause is that you have one keyspace that's NOT
>> replicated to one of the datacenters. You can work around this by running
>> 'nodetool cleanup ' on all of your other keyspaces individually,
>> skipping the one that isn't replicated to that datacenter.
>>
>>
>>
>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>> Thanks Jeff,
>>>
>>> I have a C* cluster spread across multiple datacenter.
>>> reason for cleanup : I added multiple nodes to cluster and need to run
>>> cleanup on old nodes so that the redundant data is cleaned-up.
>>>
>>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:
>>>


 On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
 jaibheem...@gmail.com> wrote:
 > Hello,
 >
 > I am running into an issue where *nodetool cleanup *fails to cleanup
 data.
 > We are running 2.1.16 version of Cassandra.
 >
 >
 > [user@host ~]$ nodetool cleanup
 > Aborted cleaning up atleast one column family in keyspace user, check
 > server logs for more information.
 > Aborted cleaning up atleast one column family in keyspace org, check
 server
 > logs for more information.
 > error: nodetool failed, check server logs
 > -- StackTrace --
 > java.lang.RuntimeException: nodetool failed, check server logs
 > at
 > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool
 .java:294)
 > at org.apache.cassandra.tools.Nod
 eTool.main(NodeTool.java:206)
 >
 > *Logs:*
 >
 > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
 > CompactionManager.java:415 - Cleanup cannot run before a node has
 joined
 > the ring
 > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
 > CompactionManager.java:415 - Cleanup cannot run before a node has
 joined
 > the ring
 >
 > All the nodes in the cluster are up and running. We tried doing a
 rolling
 > restart of all nodes and no luck.
 >
 > After looking at the Cassandra JIRA :
 > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
 issue
 > is fixed with 2.2.6 and 3.0 version.
 > While we have plans to upgrade to the latest versions(which might take
 > longer time), does any know if there is any work around to mitigate
 the
 > issue?
 >

 Are you running multiple datacenters, and you just removed a specific
 datacenter from a keyspace (and that's why you want to run cleanup)? If
 that's the case, I fear the fix for 10991 isn't really going to fix it in
 the way you hope (we may need a follow-up jira). What you'll almost
 certainly need to do is remove the data on disk manually, which is quite
 unfortunate as it'll require you to 
 stop+delete-data-for-that-keyspace+start
 each node in the datacenter for which you removed replication.

 -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org


>>>
>>
>

Re: Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

Yes I have many keyspaces which are not spread across all the data
centers(expected by design).
In this case, is this the expected behavior cleanup will not work for all
the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
versions?

P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
keyspacename"

On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa  wrote:

> If you didn't explicitly remove a keyspace from one of your datacenters,
> the next most likely cause is that you have one keyspace that's NOT
> replicated to one of the datacenters. You can work around this by running
> 'nodetool cleanup ' on all of your other keyspaces individually,
> skipping the one that isn't replicated to that datacenter.
>
>
>
> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Thanks Jeff,
>>
>> I have a C* cluster spread across multiple datacenter.
>> reason for cleanup : I added multiple nodes to cluster and need to run
>> cleanup on old nodes so that the redundant data is cleaned-up.
>>
>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:
>>
>>>
>>>
>>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>> > Hello,
>>> >
>>> > I am running into an issue where *nodetool cleanup *fails to cleanup
>>> data.
>>> > We are running 2.1.16 version of Cassandra.
>>> >
>>> >
>>> > [user@host ~]$ nodetool cleanup
>>> > Aborted cleaning up atleast one column family in keyspace user, check
>>> > server logs for more information.
>>> > Aborted cleaning up atleast one column family in keyspace org, check
>>> server
>>> > logs for more information.
>>> > error: nodetool failed, check server logs
>>> > -- StackTrace --
>>> > java.lang.RuntimeException: nodetool failed, check server logs
>>> > at
>>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
>>> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
>>> >
>>> > *Logs:*
>>> >
>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>> joined
>>> > the ring
>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>> joined
>>> > the ring
>>> >
>>> > All the nodes in the cluster are up and running. We tried doing a
>>> rolling
>>> > restart of all nodes and no luck.
>>> >
>>> > After looking at the Cassandra JIRA :
>>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
>>> issue
>>> > is fixed with 2.2.6 and 3.0 version.
>>> > While we have plans to upgrade to the latest versions(which might take
>>> > longer time), does any know if there is any work around to mitigate the
>>> > issue?
>>> >
>>>
>>> Are you running multiple datacenters, and you just removed a specific
>>> datacenter from a keyspace (and that's why you want to run cleanup)? If
>>> that's the case, I fear the fix for 10991 isn't really going to fix it in
>>> the way you hope (we may need a follow-up jira). What you'll almost
>>> certainly need to do is remove the data on disk manually, which is quite
>>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
>>> each node in the datacenter for which you removed replication.
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>

RE: Repairs on 2.1.12

[Mark Furlong]

The repair on the DC has completed in 308 hours. What would be helpful is if 
anyone has a good way of monitoring a manual ‘antientropy’ repair.

Thanks
Mark
801-705-7115 office

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Thursday, May 11, 2017 1:06 AM
To: Mark Furlong 
Cc: user@cassandra.apache.org
Subject: Re: Repairs on 2.1.12

to clarify, what exactly was your repair command, and in reference to a ring 
did you mean the DC or the cluster, and has the repair been running for 2 weeks 
or is that in reference to the "ring"?

It would be helpful if you provided the relevant logs as well, also, the 
cassandra version you are running.
​

AWS Cassandra backup/Restore tools

[cass savy]

AWS Backup/Restore process/tools for C*/DSE C*:

Has anyone used Opscenter 6.1 backup tool to backup/restore data for larger
datasets online ?

If yes, did you run into issues using that tool to backup/restore data in
PROD that caused any performance or any other impact to the cluster?

If no, what are other tools that people have used or recommended for backup
and restore of Cassandra keyspaces?

Please advice.

Re: Nodetool cleanup doesn't work

[Jeff Jirsa]

If you didn't explicitly remove a keyspace from one of your datacenters,
the next most likely cause is that you have one keyspace that's NOT
replicated to one of the datacenters. You can work around this by running
'nodetool cleanup ' on all of your other keyspaces individually,
skipping the one that isn't replicated to that datacenter.



On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thanks Jeff,
>
> I have a C* cluster spread across multiple datacenter.
> reason for cleanup : I added multiple nodes to cluster and need to run
> cleanup on old nodes so that the redundant data is cleaned-up.
>
> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:
>
>>
>>
>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am running into an issue where *nodetool cleanup *fails to cleanup
>> data.
>> > We are running 2.1.16 version of Cassandra.
>> >
>> >
>> > [user@host ~]$ nodetool cleanup
>> > Aborted cleaning up atleast one column family in keyspace user, check
>> > server logs for more information.
>> > Aborted cleaning up atleast one column family in keyspace org, check
>> server
>> > logs for more information.
>> > error: nodetool failed, check server logs
>> > -- StackTrace --
>> > java.lang.RuntimeException: nodetool failed, check server logs
>> > at
>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
>> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
>> >
>> > *Logs:*
>> >
>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
>> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
>> > the ring
>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
>> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
>> > the ring
>> >
>> > All the nodes in the cluster are up and running. We tried doing a
>> rolling
>> > restart of all nodes and no luck.
>> >
>> > After looking at the Cassandra JIRA :
>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
>> issue
>> > is fixed with 2.2.6 and 3.0 version.
>> > While we have plans to upgrade to the latest versions(which might take
>> > longer time), does any know if there is any work around to mitigate the
>> > issue?
>> >
>>
>> Are you running multiple datacenters, and you just removed a specific
>> datacenter from a keyspace (and that's why you want to run cleanup)? If
>> that's the case, I fear the fix for 10991 isn't really going to fix it in
>> the way you hope (we may need a follow-up jira). What you'll almost
>> certainly need to do is remove the data on disk manually, which is quite
>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
>> each node in the datacenter for which you removed replication.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

repair question (-dc option)

[Gopal, Dhruva]

Hi –
  I have a question on running a repair after bringing up a node that was down 
(brought down gracefully) for a few days within a data center. Can we just run 
nodetool repair –dc  on a single node (within that DC – specifically the 
downed node, after it is brought online) and have that entire DC repaired?

Regards,
Dhruva

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Re: LCS, range tombstones, and eviction

[Blake Eggleston]

Hi Stefano,

Based on what I understood reading the docs, if the ratio of garbage 
collectable tomstones exceeds the "tombstone_threshold", C* should start 
compacting and evicting.

If there are no other normal compaction tasks to be run, LCS will attempt to 
compact the sstables it estimates it will be able to drop the most tombstones 
from. It does this by estimating the number of tombstones an sstable has that 
have passed the gc grace period. Whether or not a tombstone will actually be 
evicted is more complicated. Even if a tombstone has passed gc grace, it can't 
be dropped if the data it's deleting still exists in another sstable, otherwise 
the data would appear to return. So, a tombstone won't be dropped if there is 
data for the same partition in other sstables that is older than the tombstone 
being evaluated for eviction.

I am quite puzzled however by what might happen when dealing with range 
tombstones. In that case a single tombstone might actually stand for an 
arbitrary number of normal tombstones. In other words, do range tombstones 
contribute to the "tombstone_threshold"? If so, how?

From what I can tell, each end of the range tombstone is counted as a single 
tombstone tombstone. So a range tombstone effectively contributes '2' to the 
count of tombstones for an sstable. I'm not 100% sure, but I haven't seen any 
sstable writing logic that tracks open tombstones and counts covered cells as 
tombstones. So, it's likely that the effect of range tombstones covering many 
rows are under represented in the droppable tombstone estimate.

I am also a bit confused by the "tombstone_compaction_interval". If I am 
dealing with a big partition in LCS which is receiving new records every day, 
and a weekly incremental repair job continously anticompacting the data and 
thus creating SStables, what is the likelhood of the default interval 
(10 days) to be actually hit?

It will be hit, but probably only in the repaired data. Once the data is marked 
repaired, it shouldn't be anticompacted again, and should get old enough to 
pass the compaction interval. That shouldn't be an issue though, because you 
should be running repair often enough that data is repaired before it can ever 
get past the gc grace period. Otherwise you'll have other problems. Also, keep 
in mind that tombstone eviction is a part of all compactions, it's just that 
occasionally a compaction is run specifically for that purpose. Finally, you 
probably shouldn't run incremental repair on data that is deleted. There is a 
design flaw in the incremental repair used in pre-4.0 of cassandra that can 
cause consistency issues. It can also cause a *lot* of over streaming, so you 
might want to take a look at how much streaming your cluster is doing with full 
repairs, and incremental repairs. It might actually be more efficient to run 
full repairs.

Hope that helps,

Blake

On May 11, 2017 at 7:16:26 AM, Stefano Ortolani (ostef...@gmail.com) wrote:

Hi all,

I am trying to wrap my head around how C* evicts tombstones when using LCS.
Based on what I understood reading the docs, if the ratio of garbage 
collectable tomstones exceeds the "tombstone_threshold", C* should start 
compacting and evicting.

I am quite puzzled however by what might happen when dealing with range 
tombstones. In that case a single tombstone might actually stand for an 
arbitrary number of normal tombstones. In other words, do range tombstones 
contribute to the "tombstone_threshold"? If so, how?

I am also a bit confused by the "tombstone_compaction_interval". If I am 
dealing with a big partition in LCS which is receiving new records every day, 
and a weekly incremental repair job continously anticompacting the data and 
thus creating SStables, what is the likelhood of the default interval 
(10 days) to be actually hit?

Hopefully somebody will be able to shed some lights here!

Thanks in advance! 
Stefano 

Re: Node containing all data of the cluster

[Igor Leão]

Thank you Varun and DuyHai!

2017-05-10 20:57 GMT-03:00 Varun Gupta :

> Hi Igor,
>
> You can setup cluster with configuration as below.
>
> Replication: DC1: 3 and DC2: 1.
>
> If you are using datastax java driver, then use dcaware load balancing
> policy and pass DC1, as input. As well as add DC2 node in ignore nodes, so
> request never goes to that node.
>
> Thanks,
> Varun
>
> On Wed, May 10, 2017 at 1:21 PM, Igor Leão  wrote:
>
>> Hey everyone,
>>
>> Imagine a have Cassandra cluster with 4 nodes.
>>
>> Is it possible to have a separate node which would not receive requests
>> but would by in sync with the rest of the cluster? Ideally this super node
>> would have all data of the cluster.
>>
>> I want to take a snapshot of this node from time to time in order to
>> reproduce scenarios that are happening in production.
>>
>> Thanks in advance!
>>
>>
>>
>>
>>
>>
>>
>>
>


-- 
Igor Leão  Site Reliability Engineer

Mobile: +55 81 99727-1083 
Skype: *igorvpcleao*
Office: +55 81 4042-9757 
Website: inlocomedia.com 
[image: inlocomedia]

 [image: LinkedIn]

 [image: Facebook]  [image: Twitter]

Re: Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

Thanks Jeff,

I have a C* cluster spread across multiple datacenter.
reason for cleanup : I added multiple nodes to cluster and need to run
cleanup on old nodes so that the redundant data is cleaned-up.

On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:

>
>
> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
> > Hello,
> >
> > I am running into an issue where *nodetool cleanup *fails to cleanup
> data.
> > We are running 2.1.16 version of Cassandra.
> >
> >
> > [user@host ~]$ nodetool cleanup
> > Aborted cleaning up atleast one column family in keyspace user, check
> > server logs for more information.
> > Aborted cleaning up atleast one column family in keyspace org, check
> server
> > logs for more information.
> > error: nodetool failed, check server logs
> > -- StackTrace --
> > java.lang.RuntimeException: nodetool failed, check server logs
> > at
> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
> >
> > *Logs:*
> >
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
> > the ring
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
> > the ring
> >
> > All the nodes in the cluster are up and running. We tried doing a rolling
> > restart of all nodes and no luck.
> >
> > After looking at the Cassandra JIRA :
> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
> issue
> > is fixed with 2.2.6 and 3.0 version.
> > While we have plans to upgrade to the latest versions(which might take
> > longer time), does any know if there is any work around to mitigate the
> > issue?
> >
>
> Are you running multiple datacenters, and you just removed a specific
> datacenter from a keyspace (and that's why you want to run cleanup)? If
> that's the case, I fear the fix for 10991 isn't really going to fix it in
> the way you hope (we may need a follow-up jira). What you'll almost
> certainly need to do is remove the data on disk manually, which is quite
> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
> each node in the datacenter for which you removed replication.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Nodetool cleanup doesn't work

[Jeff Jirsa]

On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada  
wrote: 
> Hello,
> 
> I am running into an issue where *nodetool cleanup *fails to cleanup data.
> We are running 2.1.16 version of Cassandra.
> 
> 
> [user@host ~]$ nodetool cleanup
> Aborted cleaning up atleast one column family in keyspace user, check
> server logs for more information.
> Aborted cleaning up atleast one column family in keyspace org, check server
> logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
> 
> *Logs:*
> 
> INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> CompactionManager.java:415 - Cleanup cannot run before a node has joined
> the ring
> INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> CompactionManager.java:415 - Cleanup cannot run before a node has joined
> the ring
> 
> All the nodes in the cluster are up and running. We tried doing a rolling
> restart of all nodes and no luck.
> 
> After looking at the Cassandra JIRA :
> https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the issue
> is fixed with 2.2.6 and 3.0 version.
> While we have plans to upgrade to the latest versions(which might take
> longer time), does any know if there is any work around to mitigate the
> issue?
> 

Are you running multiple datacenters, and you just removed a specific 
datacenter from a keyspace (and that's why you want to run cleanup)? If that's 
the case, I fear the fix for 10991 isn't really going to fix it in the way you 
hope (we may need a follow-up jira). What you'll almost certainly need to do is 
remove the data on disk manually, which is quite unfortunate as it'll require 
you to stop+delete-data-for-that-keyspace+start each node in the datacenter for 
which you removed replication. 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Strange Cassandra startup error

[Jeff Jirsa]

On 2017-05-11 07:26 (-0700), William Boutin  
wrote: 
> I've been running apache-cassandra 2.2.6 for a few years. Recently when I 
> configured a new cassandra node and started Cassandra via "service start 
> cassandra" on LINUX 6.5, I got the following exception. After cleaning out my 
> data_file_directories I was able to successfully restart. Any ideas? Thanks
> ERROR [main] 2017-05-10 02:12:09,935 CassandraDaemon.java:638 - Exception 
> encountered during startup
> org.apache.cassandra.exceptions.ConfigurationException: Unable to check disk 
> space available to /usr/share/cassandra/data/commitlog. Perhaps the Cassandra 
> user does not have the necessary permissions
> Caused by: java.io.IOException: Mount point not found

Is "/usr/share/cassandra/data" a local drive, or NFS? Anyone doing any changes 
to fstab / mounted volumes? 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: DSE 5.0 Upgrade

[Ben Bromhead]

Hi Cass Savy

I would suggest directing your question to Datastax, or you could look at
an open source alternative such as Apache Cassandra :)

Ben

On Wed, 10 May 2017 at 22:50 cass savy  wrote:

> Team,
>
> 1. What is the stable version for DSE 5.0 to upgrade from DSE 4.8.x?
>
> 2. Is anybody switched to using DSE Unified auth model which enfores to
> use one auth policy as primary and other secondary?
>
> 3. Do I need to use multi-auth/DSE Unified auth for me upgrade to DSE 5.0
> or higher?
> Our old clusters are using internal authentication and we learnt that
> we have to go with DSE Unifiedauth model as part of upgrade?
>
> 4. Does anybody migrate from Java driver to Datastax enterprise Java
> driver 1.1 or 1.2 recently which will be only driver that will support DSE
> 5.0 and above and new auth policies added by DSE?
>
> Or
>
> Has anybody made Java driver version 3.2 work with DSE 5.0 or 5.1?
>
> 4. For AWS, what is prod recommended AMI with CentOS and for DSE 5.x
> versions?
>
> --
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra 2.1.13: Using JOIN_RING=False

[Ben Bromhead]

A rough guess on the best way to do this with cql, without looking too
deeply into it:

You would likely have to implement a custom load balancing policy within
the driver that only uses coordinator nodes and/or modify
refreshNodeListAndTokenMap() behaviour in ControlConnection.java (there
should also be an equivalent class in the python driver).

On Thu, 11 May 2017 at 08:19 Dikang Gu  wrote:

> 1. The coordinator still store system data and hints, but they should not
> store any user data since they are not part of ring.
> 2. We are using coordinator for thrift client. For cql based drivers, they
> needs to talk to nodes in the ring, so I think coordinator mode won't work
> for them.
>
> -Dikang
>
> On Tue, May 9, 2017 at 2:01 PM, Anubhav Kale <
> anubhav.k...@microsoft.com.invalid> wrote:
>
>> Hello,
>>
>>
>>
>> With some inspiration from the Cassandra Summit talk from last year, we
>> are trying to setup a cluster with coordinator-only nodes. We setup
>> join_ring=false in env.sh, disabled auth in YAML and the nodes are able to
>> start just fine. However, we’re running into a few problems
>>
>>
>>
>> 1] The nodes marked with join_ring=false continue to store data. Why ?
>>
>> 2] We tried Python driver’s whitelistedpolicy. But we notice message like
>> below, so we are not able to send queries to all nodes marked as
>> coordinators. We also changed the Scala driver to support whitelisting, but
>> see the same thing. What are we missing ?
>>
>> 3] Is there any way to concretely tell that only coordinator nodes are
>> getting requests from clients ? We don’t have OpsCenter.
>>
>>
>>
>> Thanks !
>>
>>
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: 
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.128 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.128
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: 
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.127 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.127
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: 
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.129 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.129
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Finished fetching ring info
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Rebuilding token map due to topology changes
>>
>> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table
>> not found
>>
>> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table
>> not found
>>
>> 2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection
>> created
>>
>> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
>> for host 10.80.10.125
>>
>> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
>> for host 10.80.10.126
>>
>>
>>
>
>
>
> --
> Dikang
>
> --
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Reg:- Apache Solr with DSE Query

[Ben Bromhead]

Hi Nandan

I would suggest referring your question to Datastax, otherwise you could
look at some open source alternatives such as:

https://github.com/strapdata/elassandra
https://github.com/Stratio/cassandra-lucene-index

Cheers

Ben

On Thu, 11 May 2017 at 03:06 @Nandan@ 
wrote:

> Hi,
>
> Details are as below:-
> 1) Have table:- video_info
> 2) PRIMARY KEY:- video_id UUID
> 3) having records around 5.
> 4) Table is having around 30-35 columns
> 5) Using DSE 4.8
>
> Need clarifications and suggestions:-
> 1) I need to search by few 3-4 columns like Video_title, video_actor etc..
> 2) If I am implementing Solr indexing on this single table, then we can
> able to do a query from other columns and much more.. but is it going to
> effect my READ and WRITE speed.
> 3) is it will be a good idea or not to implement SOLR directly.
>
> Please suggest on above.
> Thanks.
>
-- 
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra 2.1.13: Using JOIN_RING=False

[Dikang Gu]

1. The coordinator still store system data and hints, but they should not
store any user data since they are not part of ring.
2. We are using coordinator for thrift client. For cql based drivers, they
needs to talk to nodes in the ring, so I think coordinator mode won't work
for them.

-Dikang

On Tue, May 9, 2017 at 2:01 PM, Anubhav Kale <
anubhav.k...@microsoft.com.invalid> wrote:

> Hello,
>
>
>
> With some inspiration from the Cassandra Summit talk from last year, we
> are trying to setup a cluster with coordinator-only nodes. We setup
> join_ring=false in env.sh, disabled auth in YAML and the nodes are able to
> start just fine. However, we’re running into a few problems
>
>
>
> 1] The nodes marked with join_ring=false continue to store data. Why ?
>
> 2] We tried Python driver’s whitelistedpolicy. But we notice message like
> below, so we are not able to send queries to all nodes marked as
> coordinators. We also changed the Scala driver to support whitelisting, but
> see the same thing. What are we missing ?
>
> 3] Is there any way to concretely tell that only coordinator nodes are
> getting requests from clients ? We don’t have OpsCenter.
>
>
>
> Thanks !
>
>
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Removing host not found in peers metadata: 
>
> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
> 10.80.10.128 removed
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
> 10.80.10.128
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Removing host not found in peers metadata: 
>
> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
> 10.80.10.127 removed
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
> 10.80.10.127
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Removing host not found in peers metadata: 
>
> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
> 10.80.10.129 removed
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
> 10.80.10.129
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Finished fetching ring info
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Rebuilding token map due to topology changes
>
> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table
> not found
>
> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table
> not found
>
> 2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection
> created
>
> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
> for host 10.80.10.125
>
> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
> for host 10.80.10.126
>
>
>



-- 
Dikang

Re: NoSE: Automated schema design for Cassandra

[Michael Mior]

Thanks for the feedback! I did change column families to tables. I agree
the documentation could use some work. If you're interested in seeing what
the input and output look like, here's a sample:

https://michael.mior.ca/projects/nose/rubis

So far we haven't had any schemas used directly for production although it
has provided some advice on design alternatives. NoSE actually already
contains a mechanism to execute different schema alternatives which is what
we used during our evaluation. However, it does not currently directly
provide mechanism for synthetic data generation. It would definitely be
possible however to add automated generation of test data in the future.

Cheers,
--
Michael Mior
mm...@uwaterloo.ca

2017-05-10 3:55 GMT-04:00 Jacques-Henri Berthemet <
jacques-henri.berthe...@genesys.com>:

> Hi,
>
>
>
> This is interesting, I’d just advise to put full examples and more
> documentation on how to use it (the articles are a bit too detailed).
>
> Also, you should not mention “column families” but just tables.
>
>
>
> Was this used to generate a schema used for production?
>
> Do you think it’s possible to generate test code to validate the workload?
>
>
>
> *--*
>
> *Jacques-Henri Berthemet*
>
>
>
> *From:* michael.m...@gmail.com [mailto:michael.m...@gmail.com] *On Behalf
> Of *Michael Mior
> *Sent:* mardi 9 mai 2017 17:30
> *To:* user 
> *Subject:* NoSE: Automated schema design for Cassandra
>
>
>
> Hi all,
>
>
>
> I wanted to share a tool I've been working on that tries to help automate
> the schema design process for Cassandra. The short description is that you
> provide information on the kind of data you want to store and the queries
> and updates you want to issue, and NoSE will perform a cost-based analysis
> to suggest an optimal schema.
>
>
>
> There's lots of room for improvement and many Cassandra features which are
> not currently supported, but hopefully some in the community may still find
> it useful as a starting point.
>
>
>
> Link to more details and the source code below:
>
>
>
> https://michael.mior.ca/projects/nose/
>
>
>
> If you're interested in trying it out, don't hesitate to reach out and I'm
> happy to help!
>
>
>
> Cheers,
>
> --
>
> Michael Mior
>
> mm...@uwaterloo.ca
>

Strange Cassandra startup error

[William Boutin]

I've been running apache-cassandra 2.2.6 for a few years. Recently when I 
configured a new cassandra node and started Cassandra via "service start 
cassandra" on LINUX 6.5, I got the following exception. After cleaning out my 
data_file_directories I was able to successfully restart. Any ideas? Thanks
ERROR [main] 2017-05-10 02:12:09,935 CassandraDaemon.java:638 - Exception 
encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: Unable to check disk 
space available to /usr/share/cassandra/data/commitlog. Perhaps the Cassandra 
user does not have the necessary permissions
at 
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:522)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:119)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:491) 
[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) 
[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
Caused by: java.io.IOException: Mount point not found
at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) 
~[na:1.7.0_141]
at sun.nio.fs.UnixFileStore.(UnixFileStore.java:65) 
~[na:1.7.0_141]
at sun.nio.fs.LinuxFileStore.(LinuxFileStore.java:44) 
~[na:1.7.0_141]
at 
sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:49)
 ~[na:1.7.0_141]
at 
sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:37)
 ~[na:1.7.0_141]
at 
sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) 
~[na:1.7.0_141]
at java.nio.file.Files.getFileStore(Files.java:1413) ~[na:1.7.0_141]
at 
org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:685)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:517)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
... 3 common frames omitted

[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

LCS, range tombstones, and eviction

[Stefano Ortolani]

Hi all,

I am trying to wrap my head around how C* evicts tombstones when using LCS.
Based on what I understood reading the docs, if the ratio of garbage
collectable tomstones exceeds the "tombstone_threshold", C* should start
compacting and evicting.

I am quite puzzled however by what might happen when dealing with range
tombstones. In that case a single tombstone might actually stand for an
arbitrary number of normal tombstones. In other words, do range tombstones
contribute to the "tombstone_threshold"? If so, how?

I am also a bit confused by the "tombstone_compaction_interval". If I am
dealing with a big partition in LCS which is receiving new records every
day,
and a weekly incremental repair job continously anticompacting the data and
thus creating SStables, what is the likelhood of the default interval
(10 days) to be actually hit?

Hopefully somebody will be able to shed some lights here!

Thanks in advance!
Stefano

Reg:- Apache Solr with DSE Query

[@Nandan@]

Hi,

Details are as below:-
1) Have table:- video_info
2) PRIMARY KEY:- video_id UUID
3) having records around 5.
4) Table is having around 30-35 columns
5) Using DSE 4.8

Need clarifications and suggestions:-
1) I need to search by few 3-4 columns like Video_title, video_actor etc..
2) If I am implementing Solr indexing on this single table, then we can
able to do a query from other columns and much more.. but is it going to
effect my READ and WRITE speed.
3) is it will be a good idea or not to implement SOLR directly.

Please suggest on above.
Thanks.

Cassandra Snapshots and directories

[Daniel Hölbling-Inzko]

Hi,
I am going through this guide to do backup/restore of cassandra data to a
new cluster:
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_snapshot_restore_t.html#task_ds_cmf_11r_gk

When creating a snapshot I get the snapshot files mixed in with the normal
data files and backup files, so it's all over the place and very hard
(especially with lots of tables per keyspace) to transfer ONLY the snapshot.
(Mostly since there is a snapshot directory per table..)

Am I missing something or is there some arcane shell command that filters
out only the snapshots?
Because this way it's much easier to just backup the whole data directory.

greetings Daniel

Re: Difference between yum and git

[Yuji Ito]

Thanks Jonathan, Joaquin,

Sorry, I found logback.xml caused the difference.
I changed logging level TRACE and the maxFileSize of debug.log in
conf/logback.xml.

logback.xml of C* by yum (changed):
  logginglevel: TRACE
  maxFileSize of debug.log: 500MB

logback.xml of C* by git (default):
  logginglevel: DEBUG
  maxFileSize of debug.log: 20MB


I have another question.
When the size of debug.log becomes the maxFileSize, Cassandra stops
temporarily?
Some operations failed at that time due to Read Timeout.
According to vmstat, the process didn't run as below.
Is it normal??

$ vmstat 1
procs ---memory-- ---swap-- -io --system--
-cpu-
 r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
wa st
22  1  0 1645528 477624 343747600 0  1460 11908 23589 81 12
 4  4  0
 8  1  0 1642520 477624 344023200 0  1396 12080 24284 82  7
 6  5  1
21  1  0 1639732 477624 344305200 0  1404 11331 22967 82 10
 4  3  0
 1  1  0 1636848 477624 344566400 0  1440 12085 24791 81  9
 5  5  0
 0  0  0 1635144 477624 344753200 0  1044 12982 24702 57  5
27 10  1  < debug.log became maxFileSize
 0  0  0 1635176 477624 344770000 016  628 1128  0  0
100  0  0
 0  0  0 1635176 477624 344770400 0 0  612 1108  1  0
100  0  0
 0  0  0 1635176 477624 344770800 0 0  603 1103  0  0
100  0  0
 0  0  0 1635176 477624 344770800 0 0  636 1136  1  0
99  0  0
 0  0  0 1635176 477624 344771600 0 0  582 1088  0  0
100  0  0
15  1  0 1634492 477624 344816000 0  1024 8069 12479 16  2
79  3  0
27  1  0 1631732 477624 345077200 0  1620 12144 28098 81 10
 7  3  0
 9  1  0 1629036 477624 345370000 0  1416 12003 23268 79 10
 6  5  0
 8  1  0 1626524 477624 345612400 0  1372 14024 22991 78  8
 7  6  1

Thanks,
Yuji


On Thu, May 11, 2017 at 12:23 PM, Jonathan Haddad  wrote:

> Where are you getting Cassandra 2.2 built from yum?
> On Wed, May 10, 2017 at 9:54 PM Yuji Ito  wrote:
>
>> Hi Joaquin,
>>
>> > Were both tests run from the same machine at close the same time?
>> Yes. I run the both tests within 30 min.
>> I retried them today. The result was the same as yesterday.
>>
>> The test run on the same instances and the same Java.
>>
>> Thanks,
>> Yuji
>>
>>
>> On Thu, May 11, 2017 at 3:27 AM, Joaquin Casares <
>> joaq...@thelastpickle.com> wrote:
>>
>>> Hi Yuji,
>>>
>>> Were both tests run from the same machine at close the same time? If
>>> not, noisy neighbors may be affecting your performance on different AWS
>>> instances.
>>>
>>> You should verify that you're using the same version of Java during both
>>> tests.
>>>
>>> Also, ensure that you're using the same test instance (that is not
>>> running Cassandra) to connect to both Cassandra clusters.
>>>
>>> Cheers,
>>>
>>> Joaquin
>>>
>>> Joaquin Casares
>>> Consultant
>>> Austin, TX
>>>
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>> On Wed, May 10, 2017 at 5:01 AM, Yuji Ito  wrote:
>>>
 Hi all,

 I'm trying a simple performance test.
 The test requests select operations (CL.SERIAL or CL.QUORUM) by
 increasing the number of threads.
 There is the difference of the performance between C* installed by yum
 and C* which I built by myself.
 What causes the difference?

 I use C* 2.2.8.
 One of them was installed by yum (# yum install cassandra22).
 Another was acquired by git from https://github.com/apache/
 cassandra/tree/cassandra-2.2.8 and built it by myself.
 I changed cassandra.yaml to set `commitlog_sync: batch` and
 `commitlog_sync_batch_window_in_ms: 2`.

 My environment:
 - a cluster has 3 nodes
 - node: AWS EC2 m4.large with 200 IOPS EBS volume
 - Replication Factor: 3
 - 1 rows

 Result:
 ** yum
  select (CL.SERIAL) 
 threads  operations/sec
 1   188
 2   156
 4   434
 8   396
 16  837
 32  1176
 64  2206
 128 4115
 256 7272

 ** git
  select (CL.SERIAL) 
 threads  operations/sec
 1   192
 2   162
 4   264
 8   446
 16  733
 32  1114
 64  1715
 128 2776
 256 3920

 ** yum
  select (CL.QUORUM) 
 threads  operations/sec
 1   434
 2   909
 4   1481
 8   1904
 16  2666
 32  3106
 64  3555
 128 5000
 256 9014

 ** git
  select (CL.QUORUM) 
 threads  operations/sec
 1   666
 2   1538
 4   2500
 8   
 16  4210
 32  5333
 64  6597
 128 7356
 256 8075

 Thanks,
 Yuji


>>>
>>

Re: Repairs on 2.1.12

[kurt greaves]

to clarify, what exactly was your repair command, and in reference to a
ring did you mean the DC or the cluster, and has the repair been running
for 2 weeks or is that in reference to the "ring"?

It would be helpful if you provided the relevant logs as well, also, the
cassandra version you are running.
​