from:"Cogumelos Maravilha"

Re: TWCS and autocompaction

2018-01-16 Thread Cogumelos Maravilha

Hi,

My read_repair_chance is 0 (AND read_repair_chance = 0.0)

When I bootstrap a new node there is around 700 sstables, but after auto
compaction the number drop to around 100.

I'm using C* 3.11.1. To solve the problem I've already changed to
'unchecked_tombstone_compaction': 'true'. Now should I run nodetool compact?

And for the future crontab nodetool disableautocompaction?

Thanks


On 16-01-2018 11:35, Alexander Dejanovski wrote:
>
> Hi,
>
> The overlaps you're seeing on time windows aren't due to automatic
> compactions, but to read repairs.
> You must be reading at quorum or local_quorum which can perform
> foreground read repair in case of digest mismatch.
>
> You can set unchecked_tombstone_compaction to true if you want to
> perform single sstable compaction to purge tombstones and a patch has
> recently been merged in to allow twcs to delete fully expired data
> even in case of overlap between time windows (I can't remember if it's
> been merged in 3.11.1).
> Just so you know, the timestamp considered for time windows is the max
> timestamp. You can have old data in recent time windows, but not the
> opposite.
>
> Cheers,
>
>
> Le mar. 16 janv. 2018 à 12:07, Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>> a écrit :
>
> Hi list,
>
> My settings:
>
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '4', 'compaction_window_unit': 'HOURS',
> 'enabled': 'true', 'max_threshold': '64', 'min_threshold': '2',
> 'tombstone_compaction_interval': '15000', 'tombstone_threshold':
> '0.2',
> 'unchecked_tombstone_compaction': 'false'}
>     AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io
> <http://org.apache.cassandra.io>.compress.LZ4Compressor'}
>     AND crc_check_chance = 0.0
>     AND dclocal_read_repair_chance = 0.0
>     AND default_time_to_live = 1555200
>     AND gc_grace_seconds = 10800
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99PERCENTILE';
>
> Running this script:
>
> for f in *Data.db; do
>    ls -lrt $f
>    output=$(sstablemetadata $f 2>/dev/null)
>    max=$(echo "$output" | grep Maximum\ timestamp | cut -d" " -f3
> | cut
> -c 1-10)
>    min=$(echo "$output" | grep Minimum\ timestamp | cut -d" " -f3
> | cut
> -c 1-10)
>    date -d @$max +'%d/%m/%Y %H:%M:%S'
>    date -d @$min +'%d/%m/%Y %H:%M:%S'
> done
>
> on sstables I'm getting values like these:
>
> -rw-r--r-- 1 cassandra cassandra 12137573577 Jan 14 20:08
> mc-22750-big-Data.db
> 14/01/2018 19:57:41
> 31/12/2017 19:06:48
>
> -rw-r--r-- 1 cassandra cassandra 4669422106 Jan 14 06:55
> mc-22322-big-Data.db
> 12/01/2018 07:59:57
> 28/12/2017 19:08:42
>
> My goal is using TWCS for sstables expired fast because lots of
> new data
> is coming in. What is the best approach to archive that? Should I
> disable auto compaction?
> Thanks in advance.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org
> <mailto:user-h...@cassandra.apache.org>
>
> -- 
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com <http://www.thelastpickle.com/>

TWCS and autocompaction

2018-01-16 Thread Cogumelos Maravilha

Hi list,

My settings:

AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '4', 'compaction_window_unit': 'HOURS',
'enabled': 'true', 'max_threshold': '64', 'min_threshold': '2',
'tombstone_compaction_interval': '15000', 'tombstone_threshold': '0.2',
'unchecked_tombstone_compaction': 'false'}
    AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 0.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 1555200
    AND gc_grace_seconds = 10800
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Running this script:

for f in *Data.db; do
   ls -lrt $f
   output=$(sstablemetadata $f 2>/dev/null)
   max=$(echo "$output" | grep Maximum\ timestamp | cut -d" " -f3 | cut
-c 1-10)
   min=$(echo "$output" | grep Minimum\ timestamp | cut -d" " -f3 | cut
-c 1-10)
   date -d @$max +'%d/%m/%Y %H:%M:%S'
   date -d @$min +'%d/%m/%Y %H:%M:%S'
done

on sstables I'm getting values like these:

-rw-r--r-- 1 cassandra cassandra 12137573577 Jan 14 20:08
mc-22750-big-Data.db
14/01/2018 19:57:41
31/12/2017 19:06:48

-rw-r--r-- 1 cassandra cassandra 4669422106 Jan 14 06:55
mc-22322-big-Data.db
12/01/2018 07:59:57
28/12/2017 19:08:42

My goal is using TWCS for sstables expired fast because lots of new data
is coming in. What is the best approach to archive that? Should I
disable auto compaction?
Thanks in advance.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

version 3.11.1 number_of_keys_estimate is missing

2017-10-11 Thread Cogumelos Maravilha

Hi list,

After upgrading from 3.11.0 to 3.11.1 I've notice in nodetool tablestats
that the number_of_keys_estimate is missing. How can I get this value now?

Thanks in advance.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

decommission mode with background compactors running

2017-10-04 Thread Cogumelos Maravilha

Hi list,

I've decommission a node but in the background with nodetool status I've
checked and there was 4 compactors running and simultaneously the
SSTables sent to other nodes. Is this safe or we should disable all
background process before decommission like:

nodetool disableautocompaction
nodetool stop COMPACTION
nodetool stop VALIDATION
nodetool stop INDEX_BUILD
nodetool stop VERIFY
nodetool stop SCRUB
nodetool stop CLEANUP

Thanks in advance


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: From SimpleStrategy to DCs approach

2017-09-16 Thread Cogumelos Maravilha

If zone a or b goes dark I want to keep my cluster alive with QUORUM on
reading. That's why I'm imagine solve this using another node in a
different location.

Thanks


On 15-09-2017 22:32, kurt greaves wrote:
> You can add a tiny node with 3 tokens. it will own a very small amount
> of data and be responsible for replicas of that data and thus included
> in quorum queries for that data. What is the use case? This won't give
> you any real improvement in meeting consistency.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: From SimpleStrategy to DCs approach

2017-09-15 Thread Cogumelos Maravilha

I've finished the migration to NetworkTopologyStrategy using
GossipingPropertyFileSnitch.

Now I have 4 nodes at zone a (rack1) and another 4 nodes at zone b
(rack2) only one dc, there's no zone c at Frankfurt.

Can I get QUORUM consistency for reading (for writing I'm using ANY)
adding a tiny node using only num_tokens = 3 in another place or it must
be a node like the others with vnodes = 256?

I only make inserts and queries there's no updates or direct deletes
only deletes made by the TTL.

Thanks in advance.


On 05-09-2017 13:41, kurt greaves wrote:
> data will be distributed amongst racks correctly, however only if you
> are using a snitch that understands racks and also
> NetworkTopologyStrategy. SimpleStrategy doesn't understand racks or
> DCs. You should use a snitch that understands racks and then
> transition to a 2 rack cluster, keeping only 1 DC. The whole DC per
> rack thing isn't necessary and will make your clients overly complicated.
>
> On 5 Sep. 2017 21:01, "Cogumelos Maravilha"
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
> Hi list,
>
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '2'}  AND durable_writes = true;
>
> I'm using C* 3.11.0 with 8 nodes at aws, 4 nodes at zone a and the
> other
> 4 nodes at zone b. The idea is to keep the cluster alive if zone a
> or b
> goes dark and keep QUORUM for reading. For writing I'm using ANY.
>
> Using getendpoints I can see that lots of keys are in the same
> zone. As
> fair as I understand a rack solution does not grant full data
> replication between racks.
>
> My idea to reach this goal is:
>
> - Change replication_factor to 1
>
> - Start decommission nodes one by node in one zone.
>
> - When only 4 nodes are up and running in one zone change keyspace
> configuration to DC using actual data as DC1 and the other 4 nodes
> as DC2
>
>
> Is this the best approach?
>
>
> Thanks in advance.
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org
> <mailto:user-h...@cassandra.apache.org>
>

Re: truncate table in C* 3.11.0

2017-09-07 Thread Cogumelos Maravilha

Now it's working but nothing changed!

In the first attempt (I've tried more that 6 times) that I've reported
all the nodes were up, snapshot is disable, and I didn't check the
sstables files.

All the nodes are up and working hard, another table with 15M
inserts/hour and 12k/h queries plus the normal maintenances tasks.

Thanks

On 07-09-2017 11:16, Carlos Rolo wrote:
> If you waited less than 60s, no warning/error was issued. Do the
> following:
>
> * Check if all nodes are up (truncate fails if not)
> * Check if you got a snapshot generated (unless you have auto_snapshot
> disabled)
> * Check if you have still the sstables in the directories (you shouldn't)
>
> If it didn't happen, open a ticket.
>
> Tested right now on a 3 nodes 3.11.0 cluster and it worked fine. So if
> you can trace the query to provide additional detail, it would help.
>
>
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> _linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>_
> Mobile: +351 918 918 100
> www.pythian.com <http://www.pythian.com/>
>
> On Thu, Sep 7, 2017 at 10:07 AM, Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
> Hi list,
>
> Using cqlsh:
>
> consistency all;
>
> select count(*) table1;
>
> 219871
>
> truncate table1;
>
> select count(*) table1;
>
> 219947
>
>
> There is a consumer reading data from kafka and inserting in C*
> but the
> rate is around 50 inserts/minute.
>
>
> Cheers
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org
> <mailto:user-h...@cassandra.apache.org>
>
>
>
> --
>
>
>

truncate table in C* 3.11.0

2017-09-07 Thread Cogumelos Maravilha

Hi list,

Using cqlsh:

consistency all;

select count(*) table1;

219871

truncate table1;

select count(*) table1;

219947


There is a consumer reading data from kafka and inserting in C* but the
rate is around 50 inserts/minute.


Cheers



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: C* 3 node issue -Urgent

2017-09-06 Thread Cogumelos Maravilha

After insert a new node we should:

ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
'replication_factor' : x };

x = number of nodes in dc

The default user and password should work:
-u cassandra -p cassandra

Cheers.

On 23-08-2017 11:14, kurt greaves wrote:
> The cassandra user requires QUORUM consistency to be achieved for
> authentication. Normal users only require ONE. I suspect your
> system_auth keyspace has an RF of 1, and the node that owns the
> cassandra users data is down.
>
> Steps to recover:
> 1. Turn off authentication on all the nodes
> 2. Restart the nodes and make sure they are UN
> 3. Alter system_auth to have a higher RF than 1 (3 is probably
> appropriate)
> 4. Turn auth back on and restart
> 5. Create a new user and use that from now on.
>
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

From SimpleStrategy to DCs approach

2017-09-05 Thread Cogumelos Maravilha

Hi list,

CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '2'}  AND durable_writes = true;

I'm using C* 3.11.0 with 8 nodes at aws, 4 nodes at zone a and the other
4 nodes at zone b. The idea is to keep the cluster alive if zone a or b
goes dark and keep QUORUM for reading. For writing I'm using ANY.

Using getendpoints I can see that lots of keys are in the same zone. As
fair as I understand a rack solution does not grant full data
replication between racks.

My idea to reach this goal is:

- Change replication_factor to 1

- Start decommission nodes one by node in one zone.

- When only 4 nodes are up and running in one zone change keyspace
configuration to DC using actual data as DC1 and the other 4 nodes as DC2


Is this the best approach?


Thanks in advance.



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Adding a new node with the double of disk space

2017-08-17 Thread Cogumelos Maravilha

Hi all,

I need to add a new node to my cluster but this time the new node will
have the double of disk space comparing to the other nodes.

I'm using the default vnodes (num_tokens: 256). To fully use the disk
space in the new node I just have to configure num_tokens: 512?

Thanks in advance.



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Deflate compressor

2017-07-08 Thread Cogumelos Maravilha

*"Decrease tombstone_compaction_interval and decrease
tombstone_threshold, or set unchecked_tombstone_compaction to true to
ignore both conditions and collect based /purely on gc grace/."*

Is this actually true for C* version 3.11.0?

AND compaction = {'class': 
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
'compaction_window_size': '1', 'compaction_window_unit': 'HOURS', 
'max_threshold': '128', 'min_threshold': 
'2',|*'unchecked_tombstone_compaction': 'true'*|}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.DeflateCompressor'}

Is this approach enough?

Thanks.

On 07/06/2017 06:27 PM, Jeff Jirsa wrote:
>
> On 2017-07-06 01:37 (-0700), Cogumelos Maravilha  
> wrote: 
>> Hi Jeff,
>>
>> Thanks for your reply. But I've already changed from LZ4 to Deflate to 
>> get higher compression level. Can I in the Deflate compressor do the 
>> same, set for a higher level of compression?
>
> Not at this time; if it's important to you, please open a JIRA (as always, 
> patches from the community are welcome)
>
>
>> Another question to the creator of the TimeWindowsCompactionStrategy:
>>
>> AND compaction = {'class': 
>> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
>> 'compaction_window_size': '1', 'compaction_window_unit': 'HOURS', 
>> 'max_threshold': '128', 'min_threshold': '2'}
>> AND compression = {'chunk_length_in_kb': '64', 'class': 
>> 'org.apache.cassandra.io.compress.DeflateCompressor'}
>>
>> There are some days that I have exactly 24 SSTables:
>> ls -alFh *Data*|grep 'Jul  3'|wc
>>   24
>> Others no:
>> ls -alFh *Data*|grep 'Jul  2'|wc
>>   59
>>
>> Is this normal?
> "Maybe", you could use sstablemetadata to get the maxTimestamp from the 
> table, that's what TWCS will use to group data files together. Keep in mind 
> that repaired sstables won't compact with unrepaired (if you're using 
> incremental repair), and that tombstone compaction subproperties (which I 
> don't see in your options, but maybe you had set before) can cause single 
> sstable compactions that change the timestamp of the FILE, but the data 
> within it may continue to be much older.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Deflate compressor

2017-07-06 Thread Cogumelos Maravilha


Hi Jeff,

Thanks for your reply. But I've already changed from LZ4 to Deflate to 
get higher compression level. Can I in the Deflate compressor do the 
same, set for a higher level of compression?


Another question to the creator of the TimeWindowsCompactionStrategy:

AND compaction = {'class': 
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
'compaction_window_size': '1', 'compaction_window_unit': 'HOURS', 
'max_threshold': '128', 'min_threshold': '2'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.DeflateCompressor'}


There are some days that I have exactly 24 SSTables:
ls -alFh *Data*|grep 'Jul  3'|wc
 24
Others no:
ls -alFh *Data*|grep 'Jul  2'|wc
 59

Is this normal?
Thanks in advance.


On 06-07-2017 06:30, Jeff Jirsa wrote:


On 2017-07-01 02:50 (-0700), Cogumelos Maravilha  
wrote:

Hi list,

Is there a way to set Deflate level of compression?

Brotli sounds good but unstable.

I just need more compression ratio.

I'm using C* 3.11.0


LZ4 compression levels are configurable as of cassandra 3.6 ( 
https://issues.apache.org/jira/browse/CASSANDRA-11051 ) .

"The attached patch adds the following optional parameters 'lz4_compressor_type' and 
'lz4_high_compressor_level' to the LZ4Compressor."

Glancing at the source: 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/LZ4Compressor.java#L41-L49

lz4_high_compressor_level defaults to 9.
lz4_compressor_type defaults to 'fast' ('high' is also available).




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Deflate compressor

2017-07-01 Thread Cogumelos Maravilha

Hi list,

Is there a way to set Deflate level of compression?

Brotli sounds good but unstable.

I just need more compression ratio.

I'm using C* 3.11.0

Cheers.



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Cogumelos Maravilha

Simplest way of all, if you are using RF>=2 simple terminate the old
instance and create a new one.

Cheers.

On 13-06-2017 18:01, Rutvij Bhatt wrote:
> Nevermind, I misunderstood the first link. In this case, the
> replacement would just be leaving the listen_address as is (to
> InetAddress.getLocalHost()) and just start the new instance up as you
> pointed out in your original answer Hannu.
>
> Thanks.
>
> On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt  > wrote:
>
> Hannu/Nitan,
>
> Thanks for your help so far! From what you said in your first
> response, I can get away with just attaching the EBS volume to
> Cassandra and starting it with the old node's private IP as my
> listen_address because it will take over the token assignment from
> the old node using the data files? With regards to "Cassandra
> automatically realizes that have just effectively changed IP
> address.", it says in the first link to change this manually to
> the desired address - does this not apply in my case if I'm
> replacing the old node?
>
> As for the plan I outlined earlier, is this more for DR scenarios
> where I have lost a node due to hardware failure and I need to
> recover the data in a safe manner by requesting a stream from the
> other replicas?  Am I understanding this right?
>
>
> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  > wrote:
>
> Hello,
>
> So the local information about tokens is stored in the system
> keyspace. Also the host id and all that.
>
> Also documented here:
> 
> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>
> If for any reason that causes issues, you can also check this:
> https://issues.apache.org/jira/browse/CASSANDRA-8382
>
> If you copy all cassandra data, you are on the safe side. Good
> point in the links is that if you have IP addresses in topolgy
> or other files, then update those as well.
>
> Hannu
>
> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com
> ) wrote:
>
>> Hannu, 
>>
>> "Cassandra automatically realizes that have just effectively
>> changed IP address” —> are you sure C* will take care of IP
>> change as is? How will it know which token range to be
>> assigned to this new IP address?
>>
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger
>>> mailto:hkro...@gmail.com>> wrote:
>>>
>>> Cassandra automatically realizes that have just effectively
>>> changed IP address
>>

Re: Cassandra Server 3.10 unable to Start after crash - commitlog needs to be removed

2017-06-01 Thread Cogumelos Maravilha

You can also manually delete the corrupt log file. Just check is name on
the logs.

Of course you are losing some data or not!

Cheers


On 01-06-2017 20:01, Peter Reilly wrote:
> Please, how do you do this?
>
> Peter
>
>
> On Fri, May 19, 2017 at 7:13 PM, Varun Gupta  > wrote:
>
> Yes the bugs need to be fixed, but as a work around on dev
> environment, you can enable cassandra.yaml option to override any
> corrupted commit log file.
>
>
> Thanks,
> Varun
>
> > On May 19, 2017, at 11:31 AM, Jeff Jirsa  > wrote:
> >
> >
> >
> >> On 2017-05-19 08:13 (-0700), Haris Altaf  > wrote:
> >> Hi All,
> >> I am using Cassandra 3.10 for my project and whenever my local
> windows
> >> system, which is my development environment, crashes then
> cassandra server
> >> is unable to start. I have to delete commitlog directory after
> every system
> >> crash. This is actually annoying and what's the purpose of
> commitlog if it
> >> itself gets crashed. I have uploaded the entire dump of
> Cassandra Server
> >> (along with logs, commitlogs, data, configs etc) at the link
> below. Kindly
> >> share its solution. I believe it needs to be fixed.
> >>
> >
> > You need to share the exact stack trace. In cassandra 3.0+, we
> became much less tolerant of surprises in commitlog state -
> perhaps a bit too aggressive, failing to start in many cases when
> only minor things were wrong. We've recently fixed a handful of
> these, but they may not be released yet for the version you're using.
> >
> >
> >
> >
> -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> 
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> 
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 
>
>

Re: EC2 instance recommendations

2017-05-23 Thread Cogumelos Maravilha

Exactly.


On 23-05-2017 23:55, Gopal, Dhruva wrote:
>
> By that do you mean it’s like bootstrapping a node if it fails or is
> shutdown and with a RF that is 2 or higher, data will get replicated
> when it’s brought up?
>
>  
>
> *From: *Cogumelos Maravilha 
> *Date: *Tuesday, May 23, 2017 at 1:52 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> Yes we can only reboot.
>
> But using rf=2 or higher it's only a node fresh restart.
>
> EBS is a network attached disk. Spinning disk or SSD is almost the same.
>
> It's better take the "risk" and use type i instances.
>
> Cheers.
>
>  
>
> On 23-05-2017 21:39, sfesc...@gmail.com <mailto:sfesc...@gmail.com> wrote:
>
> I think this is overstating it. If the instance ever stops you'll
> lose the data. That means if the server crashes for example, or if
> Amazon decides your instance requires maintenance.
>
>  
>
> On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva
> mailto:dhruva.go...@aspect.com>> wrote:
>
> Thanks! So, I assume that as long we make sure we never
> explicitly “shutdown” the instance, we are good. Are you also
> saying we won’t be able to snapshot a directory with ephemeral
> storage and that is why EBS is better? We’re just finding that
> to get a reasonable amount of IOPS (gp2) out of EBS at a
> reasonable rate, it gets more expensive than an I3.
>
>  
>
> *From: *Jonathan Haddad  <mailto:j...@jonhaddad.com>>
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva" 
> <mailto:dhruva.go...@aspect.com>, Matija Gobec
> mailto:matija0...@gmail.com>>, Bhuvan
> Rawal mailto:bhu1ra...@gmail.com>>
> *Cc: *"user@cassandra.apache.org
> <mailto:user@cassandra.apache.org>"  <mailto:user@cassandra.apache.org>>
>
>
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> > Oh, so all the data is lost if the instance is shutdown or
> restarted (for that instance)? 
>
>  
>
> When you restart the OS, you're technically not shutting down
> the instance.  As long as the instance isn't stopped /
> terminated, your data is fine.  I ran my databases on
> ephemeral storage for years without issue.  In general,
> ephemeral storage is going to give you lower latency since
> there's no network overhead.  EBS is generally cheaper than
> ephemeral, is persistent, and you can take snapshots easily.
>
>  
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva
> mailto:dhruva.go...@aspect.com>> wrote:
>
> Oh, so all the data is lost if the instance is shutdown or
> restarted (for that instance)? If we take a naïve approach
> to backing up the directory, and restoring it, if we ever
> have to bring down the instance and back up, will that
> work as a strategy? Data is only kept around for 2 days
> and is TTL’d after.
>
>  
>
> *From: *Matija Gobec  <mailto:matija0...@gmail.com>>
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal  <mailto:bhu1ra...@gmail.com>>
> *Cc: *"Gopal, Dhruva" 
> <mailto:dhruva.go...@aspect.com>,
> "user@cassandra.apache.org
> <mailto:user@cassandra.apache.org>"
> mailto:user@cassandra.apache.org>>
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> We are running on I3s since they came out. NVMe SSDs are
> really fast and I managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you
> can work around it and plan for failure recovery you are
> good to go.
>
>  
>
> I ran Cassandra on m4s before and had no problems with EBS
> volumes (gp2) even in low latency use cases. With the cost
> of M4 instances and EBS volumes that make sense in IOPs, I
> would recommend going with more i3s and working around the
> ephemeral issue (if its an issue).
>
>  
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal
> mailto

Re: EC2 instance recommendations

2017-05-23 Thread Cogumelos Maravilha

Yes we can only reboot.

But using rf=2 or higher it's only a node fresh restart.

EBS is a network attached disk. Spinning disk or SSD is almost the same.

It's better take the "risk" and use type i instances.

Cheers.


On 23-05-2017 21:39, sfesc...@gmail.com wrote:
> I think this is overstating it. If the instance ever stops you'll lose
> the data. That means if the server crashes for example, or if Amazon
> decides your instance requires maintenance.
>
> On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva
> mailto:dhruva.go...@aspect.com>> wrote:
>
> Thanks! So, I assume that as long we make sure we never explicitly
> “shutdown” the instance, we are good. Are you also saying we won’t
> be able to snapshot a directory with ephemeral storage and that is
> why EBS is better? We’re just finding that to get a reasonable
> amount of IOPS (gp2) out of EBS at a reasonable rate, it gets more
> expensive than an I3.
>
>  
>
> *From: *Jonathan Haddad mailto:j...@jonhaddad.com>>
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva" , Matija Gobec
> mailto:matija0...@gmail.com>>, Bhuvan Rawal
> mailto:bhu1ra...@gmail.com>>
> *Cc: *"user@cassandra.apache.org
> "  >
>
>
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> > Oh, so all the data is lost if the instance is shutdown or
> restarted (for that instance)? 
>
>  
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated,
> your data is fine.  I ran my databases on ephemeral storage for
> years without issue.  In general, ephemeral storage is going to
> give you lower latency since there's no network overhead.  EBS is
> generally cheaper than ephemeral, is persistent, and you can take
> snapshots easily.
>
>  
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva
> mailto:dhruva.go...@aspect.com>> wrote:
>
> Oh, so all the data is lost if the instance is shutdown or
> restarted (for that instance)? If we take a naïve approach to
> backing up the directory, and restoring it, if we ever have to
> bring down the instance and back up, will that work as a
> strategy? Data is only kept around for 2 days and is TTL’d after.
>
>  
>
> *From: *Matija Gobec  >
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal  >
> *Cc: *"Gopal, Dhruva" ,
> "user@cassandra.apache.org "
> mailto:user@cassandra.apache.org>>
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> We are running on I3s since they came out. NVMe SSDs are
> really fast and I managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can
> work around it and plan for failure recovery you are good to go.
>
>  
>
> I ran Cassandra on m4s before and had no problems with EBS
> volumes (gp2) even in low latency use cases. With the cost of
> M4 instances and EBS volumes that make sense in IOPs, I would
> recommend going with more i3s and working around the ephemeral
> issue (if its an issue).
>
>  
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal
> mailto:bhu1ra...@gmail.com>> wrote:
>
> i3 instances will undoubtedly give you more meat for buck
> - easily 40K+ iops whereas on the other hand EBS maxes out
> at 20K PIOPS which is highly expensive (at times they can
> cost you significantly more than cost of instance). 
>
> But they have ephemeral local storage and data is lost
> once instance is stopped, you need to be prudent in case
> of i series, it is generally used for large persistent caches.
>
>  
>
> Regards,
>
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva
> mailto:dhruva.go...@aspect.com>>
> wrote:
>
> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3
> TB of storage and have been comparing this to
> I-3.2xlarge, which seems more cost effective when
> dealing with this amount of storage and from an IOPS
> perspective. Does anyone have any recommendations/ on
> the I-3s and how it performs overall, compared to the
> M4 equivalent? On the surface, without us having taken
> it through its paces performance-wise, it does seem to
> be pretty powerful. We just ran through an exercise
> with a RA

Re: InternalResponseStage low on some nodes

2017-05-23 Thread Cogumelos Maravilha

This is really atypical.

What about nodetool compactionstats?

crontab jobs in each node like nodetool repair, etc?

Also security these 2 nodes have the same ports open?

Same configuration, same JVM params?

nodetool ring normal?

Cheers.


On 23-05-2017 20:11, Andrew Jorgensen wrote:
> Hello,
>
> I will preface this and say that all of the nodes have been running
> for about the same amount of time and were not restarted before
> running nodetool tpstats.
>
> This is more for my understanding that anything else but I have a 20
> node cassandra cluster running cassandra 3.0.3. I have 0 read and 0
> writes going to the cluster and I see a strange banding of load
> average for some of the nodes.
>
> 18 out of 20 of the nodes sit around an 15 min LA of 0.3 while 2 nodes
> are at 0.08. Once I applied writes to the cluster all of the load
> averages increased by the same proportion, so 18 out of 20 nodes
> increased to ~0.4 and 2 nodes increased to 0.1
>
> When I look at what is different between these nodes, all of which
> have been running for the same amount of time, the only numerical
> difference in nodetool tpstats is the InternalResponseStage. 18 out of
> 20 of the nodes are in the range of 20,000 completed while 2 are only
> at 300. Interestingly it also looks like the 2 nodes that are on the
> low side are exhibiting symptoms
> of https://issues.apache.org/jira/browse/CASSANDRA-11090.
>
> I restarted one of the low nodes and it immediately jumped up to match
> the other 18 nodes in the cluster and settled around 0.4 15 minute LA.
>
> I am curious of two things, first why the InternalResponseStage is so
> low on two of the nodes in the cluster and what that means. Second is
> it atypical for the other 18 nodes in the cluster to have such a high
> number of InternalResponseStage completed tasks or do these numbers
> seems reasonable for a idle cluster?
>
> Thanks!
> Andrew Jorgensen
> @ajorgensen

Re: Slowness in C* cluster after implementing multiple network interface configuration.

2017-05-23 Thread Cogumelos Maravilha

Hi,

I never used version 2.0.x but I think port 7000 isn't enough.

Try enable:

7000 inter-node

7001 SSL inter-node

9042 CQL

9160 Thrift is enable in that version


And

**In Cassandra.yaml, add property “broadcast_address”.  = local ipv4

**In Cassandra.yaml, change “listen_address” to private IP. = local ipv4


As a starting point.


Cheers.


On 22-05-2017 12:36, Prakash Chauhan wrote:
>
> Hi All ,
>
>  
>
> Need Help !!!
>
>  
>
> *Setup Details:*
>
> Cassandra 2.0.14
>
> Geo Red setup
>
> · DC1 - 3 nodes
>
> · DC2 - 3 nodes
>
>  
>
>  
>
> We were trying to implement multiple network interfaces with Cassandra
> 2.0.14
>
> After doing all the steps mentioned in DataStax doc
> http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configMultiNetworks.html,
> we observed that nodes were not able to see each other (checked using
> nodetool status).
>
>  
>
> To resolve this issue, we followed the comment
> 
> mentioned in the JIRA : CASSANDRA-9748
> 
>
>  
>
> Exact steps that we followed are :
>
> 
>
> *1.   *Stop Cassandra
>
> *2.   *Add rule to “iptables” to forward all packets on the public
> interface to the private interface.
>
>  
>
> COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d
>  --dport 7000 -j DNAT --to-destination :7000
>
>  
>
> *3.   *In Cassandra.yaml, add property “broadcast_address”.
>
> *4.   *In Cassandra.yaml, change “listen_address” to private IP.
>
> *5.   *Clear the data from directory “peers”.
>
> *6.   *Change Snitch to GossipingPropertyFileSnitch.
>
> *7.   *Append following property to the file
> “/etc/cassandra/conf/cassandra-env.sh” to purge gossip state.
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
>  
>
> *8.   *Start Cassandra
>
> *9.   *After node has been started, remove following property from
> the file “/etc/cassandra/conf/cassandra-env.sh” (previously added in
> step 7)
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
> *10.   *Delete file “/etc/cassandra/conf/cassandra-topology.properties”
>
>  
>
>  
>
> Now We have an observation that after multiple restarts of Cassandra
> on multiple nodes, slowness is observed in the cluster.
>
> The problem gets resolved when we revert the steps mentioned above.
>
>  
>
> *Do u think there is any step that can cause the problem ?*
>
> We are suspecting Step 2(iptable rule) but not very sure about it.
>
>  
>
>  
>
> Regards,
>
> Prakash Chauhan.
>

Re: Bottleneck for small inserts?

2017-05-23 Thread Cogumelos Maravilha

Hi,

Change to *|durable_writes = false|*

And please post the results.

Thanks.

On 05/22/2017 10:08 PM, Jonathan Haddad wrote:
> How many CPUs are you using for interrupts?
>  
> http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
>
> Have you tried making a flame graph to see where Cassandra is spending
> its
> time? http://www.brendangregg.com/blog/2014-06-12/java-flame-graphs.html
>
> Are you tracking GC pauses?
>
> Jon
>
> On Mon, May 22, 2017 at 2:03 PM Eric Pederson  > wrote:
>
> Hi all:
>
> I'm new to Cassandra and I'm doing some performance testing.  One
> of things that I'm testing is ingestion throughput.   My server
> setup is:
>
>   * 3 node cluster
>   * SSD data (both commit log and sstables are on the same disk)
>   * 64 GB RAM per server
>   * 48 cores per server
>   * Cassandra 3.0.11
>   * 48 Gb heap using G1GC
>   * 1 Gbps NICs
>
> Since I'm using SSD I've tried tuning the following (one at a
> time) but none seemed to make a lot of difference:
>
>   * concurrent_writes=384
>   * memtable_flush_writers=8
>   * concurrent_compactors=8
>
> I am currently doing ingestion tests sending data from 3 clients
> on the same subnet.  I am using cassandra-stress to do some
> ingestion testing.  The tests are using CL=ONE and RF=2.
>
> Using cassandra-stress (3.10) I am able to saturate the disk using
> a large enough column size and the standard five column
> cassandra-stress schema.  For example, -col size=fixed(400) will
> saturate the disk and compactions will start falling behind. 
>
> One of our main tables has a row size that approximately 200
> bytes, across 64 columns.  When ingesting this table I don't see
> any resource saturation.  Disk utilization is around 10-15% per
> iostat.  Incoming network traffic on the servers is around 100-300
> Mbps.  CPU utilization is around 20-70%.  nodetool tpstats shows
> mostly zeros with occasional spikes around 500 in MutationStage.  
>
> The stress run does 10,000,000 inserts per client, each with a
> separate range of partition IDs.  The run with 200 byte rows takes
> about 4 minutes, with mean Latency 4.5ms, Total GC time of 21
> secs, Avg GC time 173 ms.   
>
> The overall performance is good - around 120k rows/sec ingested. 
> But I'm curious to know where the bottleneck is.  There's no
> resource saturation andnodetool tpstats shows only occasional
> brief queueing.  Is the rest just expected latency inside of
> Cassandra?
>
> Thanks,
>
> -- Eric
>

Re: Is it safe to upgrade 2.2.6 to 3.0.13?

2017-05-20 Thread Cogumelos Maravilha

It's better wait for 3.0.14

https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/12340362/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel

Cheers.

On 05/20/2017 11:31 AM, Stefano Ortolani wrote:
> Hi Varun, 
>
> can you elaborate a bit more? I have seen a schema change being
> pushed, but that was just the first restart. After that, everything
> has been smooth so far (including several schema changes, all of them
> verified via "describe").
>
> Thanks!
> Stefano
>
> On Sat, May 20, 2017 at 2:10 AM, Varun Gupta  > wrote:
>
> We upgraded from 2.2.5 to 3.0.11 and it works fine. I will suggest
> not to go with 3.013, we are seeing some issues with schema
> mismatch due to which we had to rollback to 3.0.11.
>
> Thanks,
> Varun
>
> On May 19, 2017, at 7:43 AM, Stefano Ortolani  > wrote:
>
>> Here
>> (https://github.com/apache/cassandra/blob/cassandra-3.0/NEWS.txt
>> )
>> is stated that the minimum supported version for the 2.2.X branch
>> is 2.2.2.
>>
>> On Fri, May 19, 2017 at 2:16 PM, Nicolas Guyomar
>> mailto:nicolas.guyo...@gmail.com>> wrote:
>>
>> Hi Xihui,
>>
>> I was looking for this documentation also, but I believe
>> datastax removed it, and it is not available yet on the
>> apache website
>>
>> As far as I remember, intermediate version was needed if  C*
>> Version < 2.1.7.
>>
>> You should be safe starting from 2.2.6, but testing the
>> upgrade on a dedicated platform is always a good idea.
>>
>> Nicolas
>>
>> On 19 May 2017 at 09:02, Xihui He > > wrote:
>>
>> Hi All,
>>
>> We are planning to upgrade our production cluster to 3.x,
>> but I can't find the upgrade guide anymore.
>> Can I upgrade to 3.0.13 from 2.2.6 directly? Is a interim
>> version necessary?
>>
>> Thanks,
>> Xihui
>>
>>
>>
>

Re: Nodes stopping

2017-05-11 Thread Cogumelos Maravilha

Have a look at dmesg. It have already happened to me regarding type i
instances at AWS.


On 11-05-2017 22:17, Daniel Steuernol wrote:
> I had 2 nodes go down today, here is the ERRORs from the system log on
> both nodes
> https://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218
> For some context, I'm trying to get regular repairs going but am
> having issues with it.
>
>
> On May 11 2017, at 2:10 pm, Cogumelos Maravilha
>  wrote:
>
> Can you grep ERROR system.log
>
>
> On 11-05-2017 21:52, Daniel Steuernol wrote:
>> There is nothing in the system log about it being drained or
>> shutdown, I'm not sure how else it would be pre-empted. No one
>> else on the team is on the servers and I haven't been shutting
>> them down. There also is no java memory dump on the server
>> either. It appears that the process just died.
>>
>>
>> On May 11 2017, at 1:36 pm, Varun Gupta 
>> <mailto:var...@uber.com> wrote:
>>
>>
>> What do you mean by "no obvious error in the logs", do you
>> see node was drained or shutdown. Are you sure, no other
>> process is calling nodetool drain or shutdown, OR pre-empting
>> cassandra process?
>>
>> On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol
>> mailto:dan...@sendwithus.com>> wrote:
>>
>>
>> I have a 6 node cassandra cluster running, and frequently
>> a node will go down with no obvious error in the logs.
>> This is starting to happen quite often, almost daily now.
>> Any suggestions on how to track down what is causing the
>> node to stop?
>> 
>> -
>> To unsubscribe, e-mail:
>> user-unsubscr...@cassandra.apache.org
>> <mailto:user-unsubscr...@cassandra.apache.org> For
>> additional commands, e-mail:
>> user-h...@cassandra.apache.org
>> <mailto:user-h...@cassandra.apache.org> 
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> <mailto:user-unsubscr...@cassandra.apache.org> For additional
>> commands, e-mail: user-h...@cassandra.apache.org
>> <mailto:user-h...@cassandra.apache.org> 
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
> additional commands, e-mail: user-h...@cassandra.apache.org

Re: Nodes stopping

2017-05-11 Thread Cogumelos Maravilha

Can you grep ERROR system.log


On 11-05-2017 21:52, Daniel Steuernol wrote:
> There is nothing in the system log about it being drained or shutdown,
> I'm not sure how else it would be pre-empted. No one else on the team
> is on the servers and I haven't been shutting them down. There also is
> no java memory dump on the server either. It appears that the process
> just died.
>
>
> On May 11 2017, at 1:36 pm, Varun Gupta  wrote:
>
>
> What do you mean by "no obvious error in the logs", do you see
> node was drained or shutdown. Are you sure, no other process is
> calling nodetool drain or shutdown, OR pre-empting cassandra process?
>
> On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol
> mailto:dan...@sendwithus.com>> wrote:
>
>
> I have a 6 node cassandra cluster running, and frequently a
> node will go down with no obvious error in the logs. This is
> starting to happen quite often, almost daily now. Any
> suggestions on how to track down what is causing the node to
> stop?
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>  For additional
> commands, e-mail: user-h...@cassandra.apache.org
>  
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
> additional commands, e-mail: user-h...@cassandra.apache.org

Try version 3.11

2017-05-06 Thread Cogumelos Maravilha

Hi all,

deb http://www.apache.org/dist/cassandra/debian 310x main
deb http://www.apache.org/dist/cassandra/debian 311x main
deb http://www.apache.org/dist/cassandra/debian sid main
deb http://www.apache.org/dist/cassandra/debian unstable main

Is there a way to try C* version 3.11 binary before release date?

Thanks


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Totally unbalanced cluster

2017-05-05 Thread Cogumelos Maravilha

nals is
> very important to operate Apache Cassandra correctly, I mean playing
> with it to learn can put you in some undesirable situations. That's
> why I keep mentioning some blog posts, talks or documentations that I
> think could be helpful to know Apache Cassandra internals and
> processes a bit more.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> <mailto:al...@thelastpickle.com>
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2017-05-04 17:10 GMT+01:00 Jon Haddad  <mailto:jonathan.had...@gmail.com>>:
>
> Adding nodes with NTS is easier, in my opinion.  You don’t need to
> worry about replica placement, if you do it right.
>
>> On May 4, 2017, at 7:43 AM, Cogumelos Maravilha
>> mailto:cogumelosmaravi...@sapo.pt>>
>> wrote:
>>
>> Hi Alain thanks for your kick reply.
>>
>>
>> Regarding SimpleStrategy perhaps you are right but it's so easy
>> to add nodes.
>>
>> I'm *not* using vnodes and the default 256. The information that
>> I've posted it a regular nodetool status keyspace.
>>
>> My partition key is a sequencial big int but nodetool cfstatus
>> shows that the number of keys are not balanced (data from 3 nodes):
>>
>> Number of keys (estimate): 442779640
>>
>> Number of keys (estimate): 736380940
>>
>> Number of keys (estimate): 451097313
>>
>> *Should I use nodetool rebuild?*
>>
>> Running:
>>
>> nodetool getendpoints mykeyspace data 9213395123941039285
>>
>> 10.1.1.52
>> 10.1.1.185
>>
>> nodetool getendpoints mykeyspace data 9213395123941039286
>>
>> 10.1.1.161
>> 10.1.1.19
>>
>> All nodes are working hard because my TTL is for 18 days and
>> daily data ingestion is around 120,000,000 records:
>>
>> nodetool compactionstats -H
>> pending tasks: 3
>> - mykeyspace.data: 3
>>
>> id   compaction type
>> keyspace  table completed  total  unit  progress
>> c49599b1-308d-11e7-ba5b-67e232f1bee1 Remove deleted data
>> mykeyspace data 133.89 GiB 158.33 GiB bytes 84.56% 
>> c49599b0-308d-11e7-ba5b-67e232f1bee1 Remove deleted data
>> mykeyspace data 136.2 GiB  278.96 GiB bytes 48.83%
>>
>> Active compaction remaining time :   0h00m00s
>>
>>
>> nodetool compactionstats -H
>> pending tasks: 2
>> - mykeyspace.data: 2
>>
>> id   compaction type keyspace 
>> table completed total  unit  progress
>> b6e8ce80-30d4-11e7-a2be-9b830f114108 Compaction  mykeyspace
>> data 4.05 GiB  133.02 GiB bytes 3.04%  
>> Active compaction remaining time :   2h17m34s
>>
>>
>> The nodetool repair by default in this C* version is incremental
>> and since the repair is run in all nodes in different hours and I
>> don't want snapshots that's why I'm cleaning twice a day (not
>> sure that with -pr a snapshot is created).
>>
>> The cleanup was already remove was there because last node was
>> created a few days ago.
>>
>> I'm using garbagecollect to force the cleanup since I'm running
>> out of space.
>>
>>
>> Regards.
>>
>>
>> On 05/04/2017 12:50 PM, Alain RODRIGUEZ wrote:
>>> Hi,
>>>
>>> CREATE KEYSPACE mykeyspace WITH replication = {'class':
>>> 'SimpleStrategy', 'replication_factor': '2'}  AND
>>> durable_writes = false;
>>>
>>>
>>> The SimpleStrategy is never recommended for production clusters
>>> as it does not recognise racks or datacenter, inducing possible
>>> availability issues and unpredictable latency when using those.
>>> I would not even use it for testing purposes, I see no point in
>>> most cases.
>>>
>>> Even if this should be changed, carefully but as soon as
>>> possible imho, it is probably not related to your main issue at
>>> hand.
>>>
>>> If nodes are imbalanced, there are 3 mains questions that come
>>> to my mind:
>>>
>>>  1. Are the token well distributed among the available nodes?
>>>  2. Is the data correctly balanced on

Re: DTCS to TWCS

2017-05-04 Thread Cogumelos Maravilha

Hi,

Take a look to https://issues.apache.org/jira/browse/CASSANDRA-13038

Regards


On 04-05-2017 18:22, vasu gunja wrote:
> Hi All,
>
> We are currently on C* 2.1.13 version and we are using DTCS for our
> tables.
> We planning to move to TWCS. 
>
> My questions
> From which versions TWCS is available ?
> Do we need to perform upgrade of cassandra before changing DTCS to TWCS ?
>
>
> Thanks

Re: Totally unbalanced cluster

2017-05-04 Thread Cogumelos Maravilha

y is often a good idea, but it depends on your
> needs as well, of course. You could look at the distribution on the
> distinct nodes through: 'nodetool cfstats'.
> *
> *
> *Point 3* : even if the tokens are perfectly distributed and the
> primary key perfectly randomized, some node can have some disk issue
> or any other reason having the compactions falling behind. This would
> lead to this node to hold more data and note evicting tombstones
> properly in some cases, increasing disk space used. Other than that,
> you can have a big SSTable being compacted on a node, having the size
> of the node growing quite suddenly (that's why 50 to 20% of the disk
> should always be free, depending on the compaction strategy in use and
> the number of concurrent compactions). Here, running 'nodetool
> compactionstats -H' on all the nodes would probably help you to
> troubleshoot.
> *
> *
> *About crontab*
>  
>
> 08 05   * * *   rootnodetool repair -pr
> 11 11   * * *   rootfstrim -a
> 04 12   * * *   rootnodetool clearsnapshot
> 33 13   * * 2   rootnodetool cleanup
> 35 15   * * *   rootnodetool garbagecollect
> 46 19   * * *   rootnodetool clearsnapshot
> 50 23   * * *   rootnodetool flush*
> *
>
> *
> *
> I don't understand what you try to achieve with some of the commands:
>
> nodetool repair -pr
>
>
> Repairing the cluster regularly is good in most cases, but as default
> changes with version, I would specify if the repair is supposed to be
> 'incremental' or 'full', if it is supposed to be 'sequential' or
> 'parallel' for example. Also, as the dataset growth, some issue will
> appear with repairs.Just search for 'repairs cassandra' on google or
> any search engine you are using and you will see that repair is a
> complex topic. Look for videos and you will find a lot of informations
> about it from nice talks like these 2 from the last summit:
>
> https://www.youtube.com/watch?v=FrF8wQuXXks
> <https://www.youtube.com/watch?v=FrF8wQuXXks>
> https://www.youtube.com/watch?v=1Sz_K8UID6E
> <https://www.youtube.com/watch?v=1Sz_K8UID6E>
>
> Also some nice tools exist to help with repairs:
>
> The Reaper (originally made at Spotify now maintained by The Last
> Pickle): https://github.com/thelastpickle/cassandra-reaper
> <https://github.com/thelastpickle/cassandra-reaper>
> 'cassandra_range_repair.py':
>  https://github.com/BrianGallew/cassandra_range_repair
> <https://github.com/BrianGallew/cassandra_range_repair>
>
> 11 11   * * *   rootfstrim -a
>
>
> I am not really sure about this one but it looks good as long as the
> 'fstrim' do not create performance issue while it is running it seems
> fine.
>
> 04 12   * * *   rootnodetool clearsnapshot
>
>
> This will automatically erase any snapshot you might want to keep. It
> might be good to specify what snapshot you want to remove and name it.
> Some snapshots will be created and not removed when using a sequential
> repair. So I believe clearing specific snapshots is a good idea to
> save disk space.
>
> 33 13   * * 2   rootnodetool cleanup
>
>
> This is to be ran on all the nodes after adding a new node. It will
> just remove data from existing node that 'gave' some token ranges to
> the new node. To do so it will compact all the SSTables. It doesn't
> seem to be a good idea to 'cron' that.
>
> 35 15   * * *   rootnodetool garbagecollect
>
>
> This is also an heavy operation that you should not need in a regular
> basis:
> http://cassandra.apache.org/doc/latest/tools/nodetool/garbagecollect.html.
> What problem are you trying to solve here? Your data uses TTLs and
> TWCS, so expired SSTable should be going away without any issue.
>
> 46 19   * * *   rootnodetool clearsnapshot
>
>
> Again? What for?
>
> 50 23   * * *   rootnodetool flush
>
>
> This will produce tables to be flushed at the same time, no matter
> their sizes or any other considerations. It is not to be used unless
> you are doing some testing, debugging or on your way to shut down the
> node.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> <mailto:al...@thelastpickle.com>
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2017-05-04 11:38 GMT+01:00 Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>>:
>
> Hi all,
>
> I'm using C* 3.10.
>
> CREATE KEYSPACE m

Totally unbalanced cluster

2017-05-04 Thread Cogumelos Maravilha

Hi all,

I'm using C* 3.10.

CREATE KEYSPACE mykeyspace WITH replication = {'class':
'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = false;

CREATE TABLE mykeyspace.data (
id bigint PRIMARY KEY,
kafka text
) WITH bloom_filter_fp_chance = 0.5
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '10', 'compaction_window_unit': 'HOURS',
'max_threshold': '32', 'min_threshold': '6'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 0.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 1555200
AND gc_grace_seconds = 10800
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

UN  10.1.1.161  398.39 GiB  256  28.9%
UN  10.1.1.19   765.32 GiB  256  29.9%
UN  10.1.1.52   574.24 GiB  256  28.2%
UN  10.1.1.213  817.56 GiB  256  28.2%
UN  10.1.1.85   638.82 GiB  256  28.2%
UN  10.1.1.245  408.95 GiB  256  28.7%
UN  10.1.1.185  574.63 GiB  256  27.9%

At crontab in all nodes (only changes the time):

08 05   * * *   rootnodetool repair -pr
11 11   * * *   rootfstrim -a
04 12   * * *   rootnodetool clearsnapshot
33 13   * * 2   rootnodetool cleanup
35 15   * * *   rootnodetool garbagecollect
46 19   * * *   rootnodetool clearsnapshot
50 23   * * *   rootnodetool flush

I can I fixed this?

Thanks in advance.



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Node always dieing

2017-04-11 Thread Cogumelos Maravilha

"system_auth" not my table.


On 04/11/2017 07:12 AM, Oskar Kjellin wrote:
> You changed to 6 nodes because you were running out of disk? But you
> still replicate 100% to all so you don't gain anything 
>
>
>
> On 10 Apr 2017, at 13:48, Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
>> No.
>>
>> nodetool status, nodetool describecluster also nodetool ring shows a
>> correct cluster.
>>
>> Not all nodes needs to be a seed, but can be.
>>
>> I had also ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' :
>> 'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;
>>
>> And the first command on the new node was  |nodetool repair system_auth|
>>
>>
>> On 04/10/2017 12:37 PM, Chris Mawata wrote:
>>> Notice
>>> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85,
>>> 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>>>
>>> Why do you have all six of your nodes as seeds? is it possible that
>>> the last one you added used itself as the seed and is isolated?
>>>
>>> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha
>>> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>>>
>>> Yes C* is running as cassandra:
>>>
>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
>>> -XX:Threa...
>>>
>>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>>> configuration:[allocate_tokens_for_keyspace=null;
>>> authenticator=PasswordAuthenticator;
>>> authorizer=CassandraAuthorizer; auto_bootstrap=true;
>>> auto_snapshot=true; back_pressure_enabled=false;
>>> back_pressure_strategy=org.apache.cassandra.net
>>> <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
>>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>>> batch_size_warn_threshold_in_kb=5;
>>> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
>>> broadcast_rpc_address=null;
>>> buffer_pool_use_heap_if_exhausted=true;
>>> cas_contention_timeout_in_ms=600; cdc_enabled=false;
>>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
>>> cdc_total_space_in_mb=0; client_encryption_options=;
>>> cluster_name=company; column_index_cache_size_in_kb=2;
>>> column_index_size_in_kb=64; commit_failure_policy=ignore;
>>> commitlog_compression=null;
>>> commitlog_directory=/mnt/cassandra/commitlog;
>>> commitlog_max_compression_buffers_in_pool=3;
>>> commitlog_periodic_queue_size=-1;
>>> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
>>> commitlog_sync_batch_window_in_ms=NaN;
>>> commitlog_sync_period_in_ms=1;
>>> commitlog_total_space_in_mb=null;
>>> compaction_large_partition_warning_threshold_mb=100;
>>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
>>> concurrent_counter_writes=32;
>>> concurrent_materialized_view_writes=32; concurrent_reads=32;
>>> concurrent_replicates=null; concurrent_writes=32;
>>> counter_cache_keys_to_save=2147483647;
>>> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
>>> counter_write_request_timeout_in_ms=600;
>>> credentials_cache_max_entries=1000;
>>> credentials_update_interval_in_ms=-1;
>>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>>> data_file_directories=[Ljava.lang.String;@223f3642;
>>> disk_access_mode=auto; disk_failure_policy=ignore;
>>> disk_optimization_estimate_percentile=0.95;
>>> disk_optimization_page_cross_chance=0.1;
>>> disk_optimization_strategy=ssd; dynamic_snitch=true;
>>> dynamic_snitch_badness_threshold=0.1;
>>> dynamic_snitch_reset_interval_in_ms=60;
>>> dynamic_snitch_update_interval_in_ms=100;
>>> enable_scripted_user_defined_functions=false;
>>> enable_user_defined_functions=false;
>>> enable_user_defined_functions_threads=true;
>>> encryption_options=null; endpoint_snitch=SimpleSnitch;
>>> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>>> gc_warn_threshold_in_ms=1000;
>>> hinted_handoff_disabled_datacenters=[];
>>> hinted_handoff_enabled=true; hinted_handoff_thrott

Re: Node always dieing

2017-04-10 Thread Cogumelos Maravilha

No.

nodetool status, nodetool describecluster also nodetool ring shows a
correct cluster.

Not all nodes needs to be a seed, but can be.

I had also ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' :
'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;

And the first command on the new node was  |nodetool repair system_auth|


On 04/10/2017 12:37 PM, Chris Mawata wrote:
> Notice
> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85,
> 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>
> Why do you have all six of your nodes as seeds? is it possible that
> the last one you added used itself as the seed and is isolated?
>
> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
> Yes C* is running as cassandra:
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
> configuration:[allocate_tokens_for_keyspace=null;
> authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_bootstrap=true;
> auto_snapshot=true; back_pressure_enabled=false;
> back_pressure_strategy=org.apache.cassandra.net
> <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
> broadcast_rpc_address=null;
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=600; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=company; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=ignore;
> commitlog_compression=null;
> commitlog_directory=/mnt/cassandra/commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1;
> commitlog_total_space_in_mb=null;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
> concurrent_counter_writes=32;
> concurrent_materialized_view_writes=32; concurrent_reads=32;
> concurrent_replicates=null; concurrent_writes=32;
> counter_cache_keys_to_save=2147483647;
> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> counter_write_request_timeout_in_ms=600;
> credentials_cache_max_entries=1000;
> credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@223f3642;
> disk_access_mode=auto; disk_failure_policy=ignore;
> disk_optimization_estimate_percentile=0.95;
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false;
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=SimpleSnitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000;
> hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=/mnt/cassandra/hints;
> hints_flush_period_in_ms=1; incremental_backups=false;
> index_interval=null; index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; initial_token=null;
> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
> inter_dc_tcp_nodelay=false; internode_authenticator=null;
> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
> internode_send_buff_size_in_bytes=0;
> key_cache_keys_to_save=2147483647; key_cache_save_period=14400;
> key_cache_size_in_mb=null; listen_address=10.100.100.213;
> listen_interface=null; listen_interface_prefer_ipv6=false;
> listen_on_broadcast_address=false; max_hint_window_in_ms=1080;
> max_hints_delivery_threads=2; max_hints_file_size_in_mb=128;
> max_mutation_size_in_kb=null; max_streaming_retries=3;
> max_valu

Re: Node always dieing

2017-04-07 Thread Cogumelos Maravilha

There's a tweak.

I've forgot to put this in the new instance:

At /lib/udev/rules.d/

|cat ||40-vm-hotadd.rules**||# On Hyper-V and Xen Virtual Machines we
want to add memory and cpus as soon as they appear|

|ATTR{[dmi/id]sys_vendor}=="Microsoft Corporation",
ATTR{[dmi/id]product_name}=="Virtual Machine", GOTO="vm_hotadd_apply"|
|ATTR{[dmi/id]sys_vendor}=="Xen", GOTO="vm_hotadd_apply"|
|GOTO="vm_hotadd_end"|
|LABEL="vm_hotadd_apply"|
|# Memory hotadd request|
|#SUBSYSTEM=="memory", ACTION=="add",
DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state",
ATTR{state}="online"|
|# CPU hotadd request|
|SUBSYSTEM=="cpu", ACTION=="add",
DEVPATH=="/devices/system/cpu/cpu[0-9]*", TEST=="online", ATTR{online}="1"|
|LABEL="vm_hotadd_end"|
|#SUBSYSTEM=="memory", ACTION=="add",
DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state",
ATTR{state}="online"|

Don't ask from where this code came from! Sharing workarounds.
On 04/06/2017 06:13 PM, Carlos Rolo wrote:
> i3 are having those issues more than the other instances it seems. Not
> the first report I heard about.
> Regards,
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> _linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>_
> Mobile: +351 918 918 100
> www.pythian.com <http://www.pythian.com/>
> On Thu, Apr 6, 2017 at 5:36 PM, Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
> Yes but this time I going to give lots of time between killing and
> pickup.
>
> Thanks a lot.
> On 04/06/2017 05:31 PM, Avi Kivity wrote:
>>
>> Your disk is bad.  Kill that instance and hope someone else gets it.
>>
>> On 04/06/2017 07:27 PM, Cogumelos Maravilha wrote:
>>>
>>> Interesting
>>>
>>> [  720.693768] blk_update_request: I/O error, dev nvme0n1,
>>> sector 1397303056 [  750.698840] blk_update_request: I/O error,
>>> dev nvme0n1, sector 1397303080 [ 1416.202103]
>>> blk_update_request: I/O error, dev nvme0n1, sector 1397303080
>>>
>>> On 04/06/2017 05:26 PM, Avi Kivity wrote:
>>>>
>>>> Is there anything in dmesg?
>>>>
>>>> On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:
>>>>>
>>>>> Now dies and restart (systemd) without logging why
>>>>>
>>>>> system.log
>>>>>
>>>>> INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
>>>>> AuthCache.java:172 - (Re)initializing RolesCache (validity
>>>>> period /update interval/max entries) (2000/2000/1000) INFO 
>>>>> [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
>>>>> - Configuration location: file:/etc/cassandra/cassandra. yaml
>>>>>
>>>>> debug.log DEBUG [GossipStage:1] 2017-04-06 16:16:56,272
>>>>> FailureDetector.java:457 - Ignoring interval time of
>>>>> 2496703934 for /10.100.120.52 <http://10.100.120.52> DEBUG
>>>>> [GossipStage:1] 2017-04-06 16:16:59,090
>>>>> FailureDetector.java:457 - Ignoring interval time of
>>>>> 2818071981 for /10.100.120.161 <http://10.100.120.161> INFO 
>>>>> [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
>>>>> - Configuration location: file:/etc/cassandra/cassandra.yaml
>>>>> DEBUG [main] 2017-04-06 16:17:42,540
>>>>> YamlConfigurationLoader.java:108 - Loading settings from
>>>>> file:/etc/cassandra/cassandra.yaml
>>>>> On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
>>>>>> find //mnt/cassandra// \! -user cassandra
>>>>>> nothing
>>>>>>
>>>>>> I've found some "strange" solutions on Internet
>>>>>> chmod -R 2777 /tmp
>>>>>> chmod -R 2775 cassandra folder
>>>>>>
>>>>>> Lets give some time to see the result
>>>>>> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>>>>>>> All it takes is one frustrated `sudo cassandra` run. Checking only 
>>>>>>> the
>>>>>>> top lev

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

Yes but this time I going to give lots of time between killing and pickup.

Thanks a lot.

On 04/06/2017 05:31 PM, Avi Kivity wrote:
>
> Your disk is bad.  Kill that instance and hope someone else gets it.
>
>
> On 04/06/2017 07:27 PM, Cogumelos Maravilha wrote:
>>
>> Interesting
>>
>> [  720.693768] blk_update_request: I/O error, dev nvme0n1, sector
>> 1397303056
>> [  750.698840] blk_update_request: I/O error, dev nvme0n1, sector
>> 1397303080
>> [ 1416.202103] blk_update_request: I/O error, dev nvme0n1, sector
>> 1397303080
>>
>>
>> On 04/06/2017 05:26 PM, Avi Kivity wrote:
>>>
>>> Is there anything in dmesg?
>>>
>>>
>>> On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:
>>>>
>>>> Now dies and restart (systemd) without logging why
>>>>
>>>> system.log
>>>>
>>>> INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
>>>> AuthCache.java:172 - (Re)initializing RolesCache (validity period
>>>> /update interval/max entries) (2000/2000/1000)
>>>> INFO  [main] 2017-04-06 16:17:42,535
>>>> YamlConfigurationLoader.java:89 - Configuration location:
>>>> file:/etc/cassandra/cassandra.
>>>> yaml
>>>>
>>>>
>>>> debug.log
>>>> DEBUG [GossipStage:1] 2017-04-06 16:16:56,272
>>>> FailureDetector.java:457 - Ignoring interval time of 2496703934 for
>>>> /10.100.120.52
>>>> DEBUG [GossipStage:1] 2017-04-06 16:16:59,090
>>>> FailureDetector.java:457 - Ignoring interval time of 2818071981 for
>>>> /10.100.120.161
>>>> INFO  [main] 2017-04-06 16:17:42,535
>>>> YamlConfigurationLoader.java:89 - Configuration location:
>>>> file:/etc/cassandra/cassandra.yaml
>>>> DEBUG [main] 2017-04-06 16:17:42,540
>>>> YamlConfigurationLoader.java:108 - Loading settings from
>>>> file:/etc/cassandra/cassandra.yaml
>>>>
>>>>
>>>> On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
>>>>> find /mnt/cassandra/ \! -user cassandra
>>>>> nothing
>>>>>
>>>>> I've found some "strange" solutions on Internet
>>>>> chmod -R 2777 /tmp
>>>>> chmod -R 2775 cassandra folder
>>>>>
>>>>> Lets give some time to see the result
>>>>>
>>>>> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>>>>>> All it takes is one frustrated `sudo cassandra` run. Checking only the
>>>>>> top level directory ownership is insufficient, since root could own
>>>>>> files/dirs created below the top level. Find all files not owned by user
>>>>>> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>>>>>>
>>>>>> Just another thought.
>>>>>>
>>>>>> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>>>>>>> From cassandra.yaml:
>>>>>>>
>>>>>>> hints_directory: /mnt/cassandra/hints
>>>>>>> data_file_directories:
>>>>>>> - /mnt/cassandra/data
>>>>>>> commitlog_directory: /mnt/cassandra/commitlog
>>>>>>> saved_caches_directory: /mnt/cassandra/saved_caches
>>>>>>>
>>>>>>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>>>>>>
>>>>>>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>>>>>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>>>>>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>>>>>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>>>>>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>>>>>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>>>>>>
>>>>>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>>>>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities 
>>>>>>> -XX:Threa...
>>>>>>>
>>>>>>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>>>>>>
>>>>>>> On /etc/security/limits.conf
>>>>>>>
>>>>>>> *   -   memlock      unlimited
>>>>>>> *   -  nofile  10
>>>>

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

Interesting

[  720.693768] blk_update_request: I/O error, dev nvme0n1, sector 1397303056
[  750.698840] blk_update_request: I/O error, dev nvme0n1, sector 1397303080
[ 1416.202103] blk_update_request: I/O error, dev nvme0n1, sector 1397303080


On 04/06/2017 05:26 PM, Avi Kivity wrote:
>
> Is there anything in dmesg?
>
>
> On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:
>>
>> Now dies and restart (systemd) without logging why
>>
>> system.log
>>
>> INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
>> AuthCache.java:172 - (Re)initializing RolesCache (validity period
>> /update interval/max entries) (2000/2000/1000)
>> INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
>> - Configuration location: file:/etc/cassandra/cassandra.
>> yaml
>>
>>
>> debug.log
>> DEBUG [GossipStage:1] 2017-04-06 16:16:56,272
>> FailureDetector.java:457 - Ignoring interval time of 2496703934 for
>> /10.100.120.52
>> DEBUG [GossipStage:1] 2017-04-06 16:16:59,090
>> FailureDetector.java:457 - Ignoring interval time of 2818071981 for
>> /10.100.120.161
>> INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
>> - Configuration location: file:/etc/cassandra/cassandra.yaml
>> DEBUG [main] 2017-04-06 16:17:42,540 YamlConfigurationLoader.java:108
>> - Loading settings from file:/etc/cassandra/cassandra.yaml
>>
>>
>> On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
>>> find /mnt/cassandra/ \! -user cassandra
>>> nothing
>>>
>>> I've found some "strange" solutions on Internet
>>> chmod -R 2777 /tmp
>>> chmod -R 2775 cassandra folder
>>>
>>> Lets give some time to see the result
>>>
>>> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>>>> All it takes is one frustrated `sudo cassandra` run. Checking only the
>>>> top level directory ownership is insufficient, since root could own
>>>> files/dirs created below the top level. Find all files not owned by user
>>>> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>>>>
>>>> Just another thought.
>>>>
>>>> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>>>>> From cassandra.yaml:
>>>>>
>>>>> hints_directory: /mnt/cassandra/hints
>>>>> data_file_directories:
>>>>> - /mnt/cassandra/data
>>>>> commitlog_directory: /mnt/cassandra/commitlog
>>>>> saved_caches_directory: /mnt/cassandra/saved_caches
>>>>>
>>>>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>>>>
>>>>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>>>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>>>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>>>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>>>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>>>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>>>>
>>>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities 
>>>>> -XX:Threa...
>>>>>
>>>>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>>>>
>>>>> On /etc/security/limits.conf
>>>>>
>>>>> *   -   memlock  unlimited
>>>>> *   -  nofile  10
>>>>> *   -  nproc  32768
>>>>> *   -  as   unlimited
>>>>>
>>>>> On /etc/security/limits.d/cassandra.conf
>>>>>
>>>>> cassandra  -  memlock  unlimited
>>>>> cassandra  -  nofile   10
>>>>> cassandra  -  as   unlimited
>>>>> cassandra  -  nproc    32768
>>>>>
>>>>> On /etc/sysctl.conf
>>>>>
>>>>> vm.max_map_count = 1048575
>>>>>
>>>>> On /etc/systcl.d/cassanda.conf
>>>>>
>>>>> vm.max_map_count = 1048575
>>>>> net.ipv4.tcp_keepalive_time=600
>>>>>
>>>>> On /etc/pam.d/su
>>>>> ...
>>>>> sessionrequired   pam_limits.so
>>>>> ...
>>>>>
>>>>> Distro is the currently Ubuntu LTS.
>>>>> Thanks
>>>>>
>>>>>
>>>

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

Now dies and restart (systemd) without logging why

system.log

INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
AuthCache.java:172 - (Re)initializing RolesCache (validity period
/update interval/max entries) (2000/2000/1000)
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/cassandra.
yaml


debug.log
DEBUG [GossipStage:1] 2017-04-06 16:16:56,272 FailureDetector.java:457 -
Ignoring interval time of 2496703934 for /10.100.120.52
DEBUG [GossipStage:1] 2017-04-06 16:16:59,090 FailureDetector.java:457 -
Ignoring interval time of 2818071981 for /10.100.120.161
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/cassandra.yaml
DEBUG [main] 2017-04-06 16:17:42,540 YamlConfigurationLoader.java:108 -
Loading settings from file:/etc/cassandra/cassandra.yaml


On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
> find /mnt/cassandra/ \! -user cassandra
> nothing
>
> I've found some "strange" solutions on Internet
> chmod -R 2777 /tmp
> chmod -R 2775 cassandra folder
>
> Lets give some time to see the result
>
> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>> All it takes is one frustrated `sudo cassandra` run. Checking only the
>> top level directory ownership is insufficient, since root could own
>> files/dirs created below the top level. Find all files not owned by user
>> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>>
>> Just another thought.
>>
>> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>>> From cassandra.yaml:
>>>
>>> hints_directory: /mnt/cassandra/hints
>>> data_file_directories:
>>> - /mnt/cassandra/data
>>> commitlog_directory: /mnt/cassandra/commitlog
>>> saved_caches_directory: /mnt/cassandra/saved_caches
>>>
>>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>>
>>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>>
>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>>>
>>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>>
>>> On /etc/security/limits.conf
>>>
>>> *   -   memlock  unlimited
>>> *   -  nofile  10
>>> *   -  nproc  32768
>>> *   -  as   unlimited
>>>
>>> On /etc/security/limits.d/cassandra.conf
>>>
>>> cassandra  -  memlock  unlimited
>>> cassandra  -  nofile   10
>>> cassandra  -  as   unlimited
>>> cassandra  -  nproc32768
>>>
>>> On /etc/sysctl.conf
>>>
>>> vm.max_map_count = 1048575
>>>
>>> On /etc/systcl.d/cassanda.conf
>>>
>>> vm.max_map_count = 1048575
>>> net.ipv4.tcp_keepalive_time=600
>>>
>>> On /etc/pam.d/su
>>> ...
>>> sessionrequired   pam_limits.so
>>> ...
>>>
>>> Distro is the currently Ubuntu LTS.
>>> Thanks
>>>
>>>
>>> On 04/06/2017 10:39 AM, benjamin roth wrote:
>>>> Cassandra cannot write an SSTable to disk. Are you sure the
>>>> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
>>>> is writeable for the CS user and has enough free space?
>>>> The CDC warning also implies that.
>>>> The other warnings indicate you are probably not running CS as root
>>>> and you did not set an appropriate limit for max open files. Running
>>>> out of open files can also be a reason for the IO error.
>>>>
>>>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>>>> mailto:cogumelosmaravi...@sapo.pt>>:
>>>>
>>>> Hi list,
>>>>
>>>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>>>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
>>>> I have
>>>> one node that is always dieing and I don't understand why. Can anyone
>>>> give me some hints please. All nodes using the same configuration.
>>>>
>>>> Thanks

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

find /mnt/cassandra/ \! -user cassandra
nothing

I've found some "strange" solutions on Internet
chmod -R 2777 /tmp
chmod -R 2775 cassandra folder

Lets give some time to see the result


On 04/06/2017 03:14 PM, Michael Shuler wrote:
> All it takes is one frustrated `sudo cassandra` run. Checking only the
> top level directory ownership is insufficient, since root could own
> files/dirs created below the top level. Find all files not owned by user
> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>
> Just another thought.
>
> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>> From cassandra.yaml:
>>
>> hints_directory: /mnt/cassandra/hints
>> data_file_directories:
>> - /mnt/cassandra/data
>> commitlog_directory: /mnt/cassandra/commitlog
>> saved_caches_directory: /mnt/cassandra/saved_caches
>>
>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>
>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>
>> cassand+  2267 1 99 10:18 ?00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>>
>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>
>> On /etc/security/limits.conf
>>
>> *   -   memlock  unlimited
>> *   -  nofile  10
>> *   -  nproc  32768
>> *   -  as   unlimited
>>
>> On /etc/security/limits.d/cassandra.conf
>>
>> cassandra  -  memlock  unlimited
>> cassandra  -  nofile   10
>> cassandra  -  as   unlimited
>> cassandra  -  nproc32768
>>
>> On /etc/sysctl.conf
>>
>> vm.max_map_count = 1048575
>>
>> On /etc/systcl.d/cassanda.conf
>>
>> vm.max_map_count = 1048575
>> net.ipv4.tcp_keepalive_time=600
>>
>> On /etc/pam.d/su
>> ...
>> sessionrequired   pam_limits.so
>> ...
>>
>> Distro is the currently Ubuntu LTS.
>> Thanks
>>
>>
>> On 04/06/2017 10:39 AM, benjamin roth wrote:
>>> Cassandra cannot write an SSTable to disk. Are you sure the
>>> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
>>> is writeable for the CS user and has enough free space?
>>> The CDC warning also implies that.
>>> The other warnings indicate you are probably not running CS as root
>>> and you did not set an appropriate limit for max open files. Running
>>> out of open files can also be a reason for the IO error.
>>>
>>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>>> mailto:cogumelosmaravi...@sapo.pt>>:
>>>
>>> Hi list,
>>>
>>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
>>> I have
>>> one node that is always dieing and I don't understand why. Can anyone
>>> give me some hints please. All nodes using the same configuration.
>>>
>>> Thanks in advance.
>>>
>>> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:22,5,main]
>>> org.apache.cassandra.io
>>> <http://org.apache.cassandra.io>.FSWriteError:
>>> java.io.IOException: Input/output
>>> error
>>> at
>>> 
>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> 
>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io
>>> 
>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io
>>> 
>>> &l

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

We tested in c4 instances but EBS is too slow. So we deployed for
production in i3.

It was running with 5 nodes without problems but we started running out
of space so we added another node. And is this last node that is giving
problems. I've already terminated the instance and created another once,
but the problem remains. The configuration is the same for all
instances. Runs nice for a few time but then

ERROR [MemtablePostFlush:11] 2017-04-06 11:47:40,840
CassandraDaemon.java:229 - Exception in thread
Thread[MemtablePostFlush:11,5,main]

Thanks.

On 04/06/2017 01:18 PM, Carlos Rolo wrote:
> There was some issue with the i3 instances and Cassandra. Did you had
> this cluster running always on i3? 
>
> On Apr 6, 2017 13:06, "Cogumelos Maravilha"
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
> Limit Soft Limit   Hard
> Limit   Units
> Max cpu time  unlimited   
> unlimitedseconds  
> Max file size unlimited   
> unlimitedbytes
> Max data size unlimited   
> unlimitedbytes
> Max stack size8388608 
> unlimitedbytes
> Max core file size0   
> unlimitedbytes
> Max resident set  unlimited   
> unlimitedbytes
> Max processes 122575  
> 122575   processes
> Max open files10  
> 10   files
> Max locked memory unlimited   
> unlimitedbytes
> Max address space unlimited   
> unlimitedbytes
> Max file locksunlimited   
> unlimitedlocks
> Max pending signals   122575  
> 122575   signals  
> Max msgqueue size 819200  
> 819200   bytes
> Max nice priority 00   
> Max realtime priority 00   
> Max realtime timeout  unlimited   
> unlimitedus   
>
> Please find something wrong there!
>
> Thanks.
>
>     On 04/06/2017 11:50 AM, benjamin roth wrote:
>> Limits: You should check them in /proc/$pid/limits
>>
>> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha
>> mailto:cogumelosmaravi...@sapo.pt>>:
>>
>> Yes C* is running as cassandra:
>>
>> cassand+  2267 1 99 10:18 ?00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea
>> -XX:+UseThreadPriorities -XX:Threa...
>>
>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>> configuration:[allocate_tokens_for_keyspace=null;
>> authenticator=PasswordAuthenticator;
>> authorizer=CassandraAuthorizer; auto_bootstrap=true;
>> auto_snapshot=true; back_pressure_enabled=false;
>> back_pressure_strategy=org.apache.cassandra.net
>> <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>> batch_size_warn_threshold_in_kb=5;
>> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
>> broadcast_rpc_address=null;
>> buffer_pool_use_heap_if_exhausted=true;
>> cas_contention_timeout_in_ms=600; cdc_enabled=false;
>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
>> cdc_total_space_in_mb=0;
>> client_encryption_options=; cluster_name=company;
>> column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>> commit_failure_policy=ignore; commitlog_compression=null;
>> commitlog_directory=/mnt/cassandra/commitlog;
>> commitlog_max_compression_buffers_in_pool=3;
>> commitlog_periodic_queue_size=-1;
>> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
>> commitlog_sync_batch_window_in_ms=NaN;
>> commitlog_sync_period_in_ms=1;
>> commitlog_total_space_in_mb=null;
>> compaction_large_partition_warning_threshold_mb=100;
>> compaction_throughput_mb_per_sec=16;
>> concurrent_compactors=null; concurrent_counter_writes=32;
>> concurrent_materialized_view_writes=32; concurrent_reads=32;
>> concurrent_replicates=null; concurrent_writes=32;
>>

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

Limit Soft Limit   Hard Limit  
Units
Max cpu time  unlimitedunlimited   
seconds  
Max file size unlimitedunlimited   
bytes
Max data size unlimitedunlimited   
bytes
Max stack size8388608  unlimited   
bytes
Max core file size0unlimited   
bytes
Max resident set  unlimitedunlimited   
bytes
Max processes 122575   122575  
processes
Max open files10   10  
files
Max locked memory unlimitedunlimited   
bytes
Max address space unlimitedunlimited   
bytes
Max file locksunlimitedunlimited   
locks
Max pending signals   122575   122575  
signals  
Max msgqueue size 819200   819200  
bytes
Max nice priority 00   
Max realtime priority 00   
Max realtime timeout  unlimitedunlimitedus   

Please find something wrong there!

Thanks.

On 04/06/2017 11:50 AM, benjamin roth wrote:
> Limits: You should check them in /proc/$pid/limits
>
> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>>:
>
> Yes C* is running as cassandra:
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
> configuration:[allocate_tokens_for_keyspace=null;
> authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_bootstrap=true;
> auto_snapshot=true; back_pressure_enabled=false;
> back_pressure_strategy=org.apache.cassandra.net
> <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
> broadcast_rpc_address=null;
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=600; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=company; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=ignore;
> commitlog_compression=null;
> commitlog_directory=/mnt/cassandra/commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1;
> commitlog_total_space_in_mb=null;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
> concurrent_counter_writes=32;
> concurrent_materialized_view_writes=32; concurrent_reads=32;
> concurrent_replicates=null; concurrent_writes=32;
> counter_cache_keys_to_save=2147483647;
> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> counter_write_request_timeout_in_ms=600;
> credentials_cache_max_entries=1000;
> credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@223f3642;
> disk_access_mode=auto; disk_failure_policy=ignore;
> disk_optimization_estimate_percentile=0.95;
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false;
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=SimpleSnitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000;
> hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=/mnt/cassandra/hints;
> hints_flush_period_in_ms=1; incremental_backups=false;
> index_interval=null; index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; initi

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

=/mnt/cassandra/saved_caches;
seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19,
10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52,
10.100.1000.213}; server_encryption_options=;
slow_query_log_timeout_in_ms=600; snapshot_before_compaction=false;
ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
start_native_transport=true; start_rpc=false; storage_port=7000;
stream_throughput_outbound_megabits_per_sec=200;
streaming_keep_alive_period_in_secs=300;
streaming_socket_timeout_in_ms=8640;
thrift_framed_transport_size_in_mb=15;
thrift_max_message_length_in_mb=16;
thrift_prepared_statements_cache_size_mb=null;
tombstone_failure_threshold=10; tombstone_warn_threshold=1000;
tracetype_query_ttl=86400; tracetype_repair_ttl=604800;
transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@38c5cc4c;
trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
truncate_request_timeout_in_ms=600;
unlogged_batch_across_partitions_warn_threshold=10;
user_defined_function_fail_timeout=1500;
user_defined_function_warn_timeout=500;
user_function_timeout_policy=die; windows_timer_interval=1;
write_request_timeout_in_ms=600]

Thanks

On 04/06/2017 11:30 AM, benjamin roth wrote:
> Have you checked the effective limits of a running CS process?
> Is CS run as Cassandra? Just to rule out missing file perms. 
>
>
> Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha"
> mailto:cogumelosmaravi...@sapo.pt>>:
>
> From cassandra.yaml:
>
> hints_directory: /mnt/cassandra/hints
> data_file_directories:
> - /mnt/cassandra/data
> commitlog_directory: /mnt/cassandra/commitlog
> saved_caches_directory: /mnt/cassandra/saved_caches
>
> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>
> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>
> On /etc/security/limits.conf
>
> *   -   memlock  unlimited
> *   -  nofile  10
> *   -  nproc  32768
> *   -  as   unlimited
>
> On /etc/security/limits.d/cassandra.conf
>
> cassandra  -  memlock  unlimited
> cassandra  -  nofile   10
> cassandra  -  as   unlimited
> cassandra  -  nproc32768
>
> On /etc/sysctl.conf
>
> vm.max_map_count = 1048575
>
> On /etc/systcl.d/cassanda.conf
>
> vm.max_map_count = 1048575
> net.ipv4.tcp_keepalive_time=600
>
> On /etc/pam.d/su
> ...
> sessionrequired   pam_limits.so
> ...
>
> Distro is the currently Ubuntu LTS.
> Thanks
>
>
>
> On 04/06/2017 10:39 AM, benjamin roth wrote:
>> Cassandra cannot write an SSTable to disk. Are you sure the
>> disk/volume where SSTables reside (normally
>> /var/lib/cassandra/data) is writeable for the CS user and has
>> enough free space?
>> The CDC warning also implies that.
>> The other warnings indicate you are probably not running CS as
>> root and you did not set an appropriate limit for max open files.
>> Running out of open files can also be a reason for the IO error.
>>
>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>> mailto:cogumelosmaravi...@sapo.pt>>:
>>
>> Hi list,
>>
>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated
>> 885G. I have
>> one node that is always dieing and I don't understand why.
>> Can anyone
>> give me some hints please. All nodes using the same
>> configuration.
>>
>> Thanks in advance.
>>
>> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>> IndexSummaryRedistribution.java:75 - Redistributing index
>> summaries
>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>> CassandraDaemon.java:229 - Exception in thread
>> Thread[MemtablePostFlush:22,5,main]
>> org.apache.cassandra.io
>> <http://or

Re: Node always dieing

2017-04-06 Thread Cogumelos Maravilha

From cassandra.yaml:

hints_directory: /mnt/cassandra/hints
data_file_directories:
- /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches

drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/

drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/

cassand+  2267 1 99 10:18 ?00:02:56 java
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

/dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt

On /etc/security/limits.conf

*   -   memlock  unlimited
*   -  nofile  10
*   -  nproc  32768
*   -  as   unlimited

On /etc/security/limits.d/cassandra.conf

cassandra  -  memlock  unlimited
cassandra  -  nofile   10
cassandra  -  as   unlimited
cassandra  -  nproc32768

On /etc/sysctl.conf

vm.max_map_count = 1048575

On /etc/systcl.d/cassanda.conf

vm.max_map_count = 1048575
net.ipv4.tcp_keepalive_time=600

On /etc/pam.d/su
...
sessionrequired   pam_limits.so
...

Distro is the currently Ubuntu LTS.
Thanks


On 04/06/2017 10:39 AM, benjamin roth wrote:
> Cassandra cannot write an SSTable to disk. Are you sure the
> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
> is writeable for the CS user and has enough free space?
> The CDC warning also implies that.
> The other warnings indicate you are probably not running CS as root
> and you did not set an appropriate limit for max open files. Running
> out of open files can also be a reason for the IO error.
>
> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>>:
>
> Hi list,
>
> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
> I have
> one node that is always dieing and I don't understand why. Can anyone
> give me some hints please. All nodes using the same configuration.
>
> Thanks in advance.
>
> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
> IndexSummaryRedistribution.java:75 - Redistributing index summaries
> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
> CassandraDaemon.java:229 - Exception in thread
> Thread[MemtablePostFlush:22,5,main]
> org.apache.cassandra.io
> <http://org.apache.cassandra.io>.FSWriteError:
> java.io.IOException: Input/output
> error
> at
> 
> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.jav

Node always dieing

2017-04-06 Thread Cogumelos Maravilha

Hi list,

I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G. I have
one node that is always dieing and I don't understand why. Can anyone
give me some hints please. All nodes using the same configuration.

Thanks in advance.

INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
IndexSummaryRedistribution.java:75 - Redistributing index summaries
ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
CassandraDaemon.java:229 - Exception in thread
Thread[MemtablePostFlush:22,5,main]
org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
error
at
org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
~[apache-cassandra-3.10.jar:3.10]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_121]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
Caused by: java.io.IOException: Input/output error
at sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[na:1.8.0_121]
at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
~[na:1.8.0_121]
at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
~[na:1.8.0_121]
at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169)
~[apache-cassandra-3.10.jar:3.10]
... 15 common frames omitted
INFO  [IndexSummaryManager:1] 2017-04-06 06:22:18,366
IndexSummaryRedistribution.java:75 - Redistributing index summaries
ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
CassandraDaemon.java:229 - Exception in thread
Thread[MemtablePostFlush:31,5,main]
org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
error
at
org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
~[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.i

How to add a node with zero downtime

2017-03-21 Thread Cogumelos Maravilha

Hi list,

I'm using C* 3.10;

authenticator: PasswordAuthenticator and authorizer: CassandraAuthorizer

When adding a node and before |nodetool repair system_auth| finished all
my clients die with:

cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
{'10.100.100.19': AuthenticationFailed('Failed to authenticate to ...

Thanks in advance.

Re: Count(*) is not working

2017-02-16 Thread Cogumelos Maravilha

With C* 3.10

cqlsh ip --request-timeout=60
Connected to x at 10.10.10.10:9042.
[cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> USE  ;
cqlsh:> SELECT count(*) from table;
   
 count
-
 3572579


On 02/16/2017 12:27 PM, Selvam Raman wrote:
> I am using cassandra 3.9.
>
> Primary Key:
> id text;
>
> On Thu, Feb 16, 2017 at 12:25 PM, Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>> wrote:
>
> C* version please and partition key.
>
>
> On 02/16/2017 12:18 PM, Selvam Raman wrote:
>> Hi,
>>
>> I want to know the total records count in table.
>>
>> I fired the below query:
>>select count(*) from tablename;
>>
>> and i have got the below output
>>
>> Read 100 live rows and 1423 tombstone cells for query SELECT *
>> FROM keysace.table WHERE token(id) >
>> token(test:ODP0144-0883E-022R-002/047-052) LIMIT 100 (see
>> tombstone_warn_threshold)
>>
>> Read 100 live rows and 1435 tombstone cells for query SELECT *
>> FROM keysace.table WHERE token(id) > token(test:2565-AMK-2) LIMIT
>> 100 (see tombstone_warn_threshold)
>>
>> Read 96 live rows and 1385 tombstone cells for query SELECT *
>> FROM keysace.table WHERE token(id) > token(test:-2220-UV033/04)
>> LIMIT 100 (see tombstone_warn_threshold).
>>
>>
>>
>>
>> Can you please help me to get the total count of the table.
>>
>> -- 
>> Selvam Raman
>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>
>
>
>
> -- 
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"

Re: Count(*) is not working

2017-02-16 Thread Cogumelos Maravilha

C* version please and partition key.


On 02/16/2017 12:18 PM, Selvam Raman wrote:
> Hi,
>
> I want to know the total records count in table.
>
> I fired the below query:
>select count(*) from tablename;
>
> and i have got the below output
>
> Read 100 live rows and 1423 tombstone cells for query SELECT * FROM
> keysace.table WHERE token(id) >
> token(test:ODP0144-0883E-022R-002/047-052) LIMIT 100 (see
> tombstone_warn_threshold)
>
> Read 100 live rows and 1435 tombstone cells for query SELECT * FROM
> keysace.table WHERE token(id) > token(test:2565-AMK-2) LIMIT 100 (see
> tombstone_warn_threshold)
>
> Read 96 live rows and 1385 tombstone cells for query SELECT * FROM
> keysace.table WHERE token(id) > token(test:-2220-UV033/04) LIMIT 100
> (see tombstone_warn_threshold).
>
>
>
>
> Can you please help me to get the total count of the table.
>
> -- 
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"

Re: inconsistent results

2017-02-14 Thread Cogumelos Maravilha

A little of python code also would help to debug;

query = SimpleStatement(
consistency_level=ConsistencyLevel.ANY)


On 14-02-2017 21:43, Josh England wrote:
> I'll try it the repair.  Using quorum tends to lead to too many
> timeout problems though.  :(
>
> -JE
>
>
> On Tue, Feb 14, 2017 at 1:39 PM, Oskar Kjellin
> mailto:oskar.kjel...@gmail.com>> wrote:
>
> Repair might help. But you will end up in this situation again
> unless you read/write using quorum (may be local)
>
> Sent from my iPhone
>
> On 14 Feb 2017, at 22:37, Josh England  > wrote:
>
>> All client interactions are from python (python-driver 3.7.1)
>> using default consistency (LOCAL_ONE I think).  Should I try
>> repairing all nodes to make sure all data is consistent?
>>
>> -JE
>>
>>
>> On Tue, Feb 14, 2017 at 1:32 PM, Oskar Kjellin
>> mailto:oskar.kjel...@gmail.com>> wrote:
>>
>> What consistency levels are you using for reads/writes?
>>
>> Sent from my iPhone
>>
>> > On 14 Feb 2017, at 22:27, Josh England > > wrote:
>> >
>> > I'm running Cassandra 3.9 on CentOS 6.7 in a 6-node
>> cluster.  I've got a situation where the same query sometimes
>> returns 2 records (correct), and sometimes only returns 1
>> record (incorrect).  I've ruled out the application and the
>> indexing since this is reproducible directly from a cqlsh
>> shell with a simple select statement.  What is the best way
>> to debug what is happening here?
>> >
>> > -JE
>> >
>>
>>
>

Extract big data to file

2017-02-08 Thread Cogumelos Maravilha

Hi list,

My database stores data from Kafka. Using C* 3.0.10

In my cluster I'm using:
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.LZ4Compressor'}

The result of extract one day of data uncompressed is around 360G.

I've find these approaches:

echo "SELECT kafka from red where datetimestamp >= '2017-02-02 00:00:00'
and datetimestamp < '2017-02-02 15:00:01';" | cqlsh 100.100.221.146 9042
> result.txt
Here by default I get 100 rows.

Using CAPTURE result.csv with paging off I always get the error out of
memory. With paging on I need to put something heavy in the top of the
Enter key. Crazy thing need to enable paging to get ride of out of
memory! I've take a look to the result file and is empty, perhaps is
cooking the result in memory to in the end past to disk.

Is there another approach like this on ACID databases:
copy (select kafka from red where datetimestamp >= '2017-02-02 00:00:00'
and datetimestamp < '2017-02-02 15:00:01') to 'result.csv' WITH CSV HEADER;

Thanks in advance.

Re: Global TTL vs Insert TTL

2017-02-01 Thread Cogumelos Maravilha

Thank you all, for your answers.


On 02/01/2017 01:06 PM, Carlos Rolo wrote:
> To reinforce Alain statement:
>
> "I would say that the unsafe part is more about using C* 3.9" this is
> key. You would be better on 3.0.x unless you need features on the 3.x
> series.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> _linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>_
> Mobile: +351 918 918 100
> www.pythian.com <http://www.pythian.com/>
>
> On Wed, Feb 1, 2017 at 8:32 AM, Alain RODRIGUEZ  <mailto:arodr...@gmail.com>> wrote:
>
> Is it safe to use TWCS in C* 3.9?
>
>
> I would say that the unsafe part is more about using C* 3.9 than
> using TWCS in C*3.9 :-). I see no reason to say 3.9 would be
> specifically unsafe in C*3.9, but I might be missing something.
>
> Going from STCS to TWCS is often smooth, from LCS you might expect
> an extra load compacting a lot (all?) of the SSTable from what we
> saw from the field. In this case, be sure that your compaction
> options are safe enough to handle this.
>
> TWCS is even easier to use on C*3.0.8+ and C*3.8+ as it became the
> new default replacing TWCS, so no extra jar is needed, you can
> enable TWCS as any other default compaction strategy.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> <mailto:al...@thelastpickle.com>
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2017-01-31 23:29 GMT+01:00 Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>>:
>
> Hi Alain,
>
> Thanks for your response and the links.
>
> I've also checked "Time series data model and tombstones".
>
> Is it safe to use TWCS in C* 3.9?
>
> Thanks in advance.
>
>
> On 31-01-2017 11:27, Alain RODRIGUEZ wrote:
>>
>> Is there a overhead using line by line option or wasted disk 
>> space?
>>
>>  There is a very recent topic about that in the mailing list,
>> look for "Time series data model and tombstones". I believe
>> DuyHai answer your question there with more details :).
>>
>> *tl;dr:*
>>
>> Yes, if you know the TTL in advance, and it is fixed, you
>> might want to go with the table option instead of adding the
>> TTL in each insert. Also you might want consider using TWCS
>> compaction strategy.
>>
>> Here are some blogposts my coworkers recently wrote about
>> TWCS, it might be useful:
>>
>> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html
>> <http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html>
>> http://thelastpickle.com/blog/2017/01/10/twcs-part2.html
>> <http://thelastpickle.com/blog/2017/01/10/twcs-part2.html>
>>
>> C*heers,
>> ---
>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>> <mailto:al...@thelastpickle.com>
>> France
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>>
>>
>> 2017-01-31 10:43 GMT+01:00 Cogumelos Maravilha
>> mailto:cogumelosmaravi...@sapo.pt>>:
>>
>> Hi I'm just wondering what option is fastest:
>>
>> Global:***create table xxx (.|AND 
>> |**|default_time_to_live = |**|XXX|**|;|**||and**UPDATE xxx USING 
>> TTL XXX;*
>>
>> Line by line:
>>
>> *INSERT INTO xxx (...USING TTL xxx;*
>>
>> Is there a overhead using line by line option or wasted disk 
>> space?
>>
>> Thanks in advance.
>>
>>
>

Re: Global TTL vs Insert TTL

2017-01-31 Thread Cogumelos Maravilha

Hi Alain,

Thanks for your response and the links.

I've also checked "Time series data model and tombstones".

Is it safe to use TWCS in C* 3.9?

Thanks in advance.


On 31-01-2017 11:27, Alain RODRIGUEZ wrote:
>
> Is there a overhead using line by line option or wasted disk space?
>
>  There is a very recent topic about that in the mailing list, look for
> "Time series data model and tombstones". I believe DuyHai answer your
> question there with more details :).
>
> *tl;dr:*
>
> Yes, if you know the TTL in advance, and it is fixed, you might want
> to go with the table option instead of adding the TTL in each insert.
> Also you might want consider using TWCS compaction strategy.
>
> Here are some blogposts my coworkers recently wrote about TWCS, it
> might be useful:
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html
> http://thelastpickle.com/blog/2017/01/10/twcs-part2.html
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> <mailto:al...@thelastpickle.com>
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
> 2017-01-31 10:43 GMT+01:00 Cogumelos Maravilha
> mailto:cogumelosmaravi...@sapo.pt>>:
>
> Hi I'm just wondering what option is fastest:
>
> Global:***create table xxx (.|AND |**|default_time_to_live = 
> |**|XXX|**|;|**||and**UPDATE xxx USING TTL XXX;*
>
> Line by line:
>
> *INSERT INTO xxx (...USING TTL xxx;*
>
> Is there a overhead using line by line option or wasted disk space?
>
> Thanks in advance.
>
>

Global TTL vs Insert TTL

2017-01-31 Thread Cogumelos Maravilha

Hi I'm just wondering what option is fastest:

Global:***create table xxx (.|AND |**|default_time_to_live = 
|**|XXX|**|;|**||and**UPDATE xxx USING TTL XXX;*

Line by line:

*INSERT INTO xxx (...USING TTL xxx;*

Is there a overhead using line by line option or wasted disk space?

Thanks in advance.

Kill queries

2017-01-23 Thread Cogumelos Maravilha

Hi,

I'm using cqlsh --request-timeout=1 but because I've more than
600.000.000 rows some times I get blocked and I kill the cqlsh. But what
about the running query in Cassandra? How can I check that?

Thanks in advance.

Re: Is this normal!?

2017-01-11 Thread Cogumelos Maravilha

Nodetool repair always list lots of data and never stays repaired. I think.

Cheers


On 01/11/2017 02:15 PM, Hannu Kröger wrote:
> Just to understand:
>
> What exactly is the problem?
>
> Cheers,
> Hannu
>
>> On 11 Jan 2017, at 16.07, Cogumelos Maravilha  
>> wrote:
>>
>> Cassandra 3.9.
>>
>> nodetool status
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address   Load   Tokens   Owns (effective)  Host
>> ID   Rack
>> UN  10.0.120.145  1.21 MiB   256  49.5%
>> da6683cd-c3cf-4c14-b3cc-e7af4080c24f  rack1
>> UN  10.0.120.179  1020.51 KiB  256  48.1%
>> fb695bea-d5e8-4bde-99db-9f756456a035  rack1
>> UN  10.0.120.55   1.02 MiB   256  53.3%
>> eb911989-3555-4aef-b11c-4a684a89a8c4  rack1
>> UN  10.0.120.46   1.01 MiB   256  49.1%
>> 8034c30a-c1bc-44d4-bf84-36742e0ec21c  rack1
>>
>> nodetool repair
>> [2017-01-11 13:58:27,274] Replication factor is 1. No repair is needed
>> for keyspace 'system_auth'
>> [2017-01-11 13:58:27,284] Starting repair command #4, repairing keyspace
>> system_traces with repair options (parallelism: parallel, primary range:
>> false, incremental: true, job threads: 1, ColumnFamilies: [],
>> dataCenters: [], hosts: [], # of ranges: 515)
>> [2017-01-11 14:01:55,628] Repair session
>> 82a25960-d806-11e6-8ac4-73b93fe4986d for range
>> [(-1278992819359672027,-1209509957304098060],
>> (-2593749995021251600,-2592266543457887959],
>> (-6451044457481580778,-6438233936014720969],
>> (-1917989291840804877,-1912580903456869648],
>> (-3693090304802198257,-3681923561719364766],
>> (-380426998894740867,-350094836653869552],
>> (1890591246410309420,1899294587910578387],
>> (6561031217224224632,6580230317350171440],
>> ... 4 pages of data
>> , (6033828815719998292,6079920177089043443]] finished (progress: 1%)
>> [2017-01-11 13:58:27,986] Repair completed successfully
>> [2017-01-11 13:58:27,988] Repair command #4 finished in 0 seconds
>>
>> nodetool gcstats
>> Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed
>> (ms)   GC Reclaimed (MB) Collections  Direct Memory Bytes
>>360134  23 
>> 23   0   333975216  
>> 1   -1
>>
>> (wait)
>> nodetool gcstats
>> Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed
>> (ms)   GC Reclaimed (MB) Collections  Direct Memory Bytes
>>   60016   0   0
>> NaN   0   0   -1
>>
>> nodetool repair
>> [2017-01-11 14:00:45,888] Replication factor is 1. No repair is needed
>> for keyspace 'system_auth'
>> [2017-01-11 14:00:45,896] Starting repair command #5, repairing keyspace
>> system_traces with repair options (parallelism: parallel, primary range:
>> false, incremental: true, job threads: 1, ColumnFamilies: [],
>> dataCenters: [], hosts: [], # of ranges: 515)
>> ... 4 pages of data
>> , (94613607632078948,219237792837906432],
>> (6033828815719998292,6079920177089043443]] finished (progress: 1%)
>> [2017-01-11 14:00:46,567] Repair completed successfully
>> [2017-01-11 14:00:46,576] Repair command #5 finished in 0 seconds
>>
>> nodetool gcstats
>> Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed
>> (ms)   GC Reclaimed (MB) Collections  Direct Memory Bytes
>>   9169  25  25  
>> 0   330518688   1   -1
>>
>>
>> Always in loop, I think!
>>
>> Thanks in advance.
>>

Is this normal!?

2017-01-11 Thread Cogumelos Maravilha

Cassandra 3.9.

nodetool status
Datacenter: dc1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load   Tokens   Owns (effective)  Host
ID   Rack
UN  10.0.120.145  1.21 MiB   256  49.5%
da6683cd-c3cf-4c14-b3cc-e7af4080c24f  rack1
UN  10.0.120.179  1020.51 KiB  256  48.1%
fb695bea-d5e8-4bde-99db-9f756456a035  rack1
UN  10.0.120.55   1.02 MiB   256  53.3%
eb911989-3555-4aef-b11c-4a684a89a8c4  rack1
UN  10.0.120.46   1.01 MiB   256  49.1%
8034c30a-c1bc-44d4-bf84-36742e0ec21c  rack1

nodetool repair
[2017-01-11 13:58:27,274] Replication factor is 1. No repair is needed
for keyspace 'system_auth'
[2017-01-11 13:58:27,284] Starting repair command #4, repairing keyspace
system_traces with repair options (parallelism: parallel, primary range:
false, incremental: true, job threads: 1, ColumnFamilies: [],
dataCenters: [], hosts: [], # of ranges: 515)
[2017-01-11 14:01:55,628] Repair session
82a25960-d806-11e6-8ac4-73b93fe4986d for range
[(-1278992819359672027,-1209509957304098060],
(-2593749995021251600,-2592266543457887959],
(-6451044457481580778,-6438233936014720969],
(-1917989291840804877,-1912580903456869648],
(-3693090304802198257,-3681923561719364766],
(-380426998894740867,-350094836653869552],
(1890591246410309420,1899294587910578387],
(6561031217224224632,6580230317350171440],
... 4 pages of data
, (6033828815719998292,6079920177089043443]] finished (progress: 1%)
[2017-01-11 13:58:27,986] Repair completed successfully
[2017-01-11 13:58:27,988] Repair command #4 finished in 0 seconds

nodetool gcstats
Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed
(ms)   GC Reclaimed (MB) Collections  Direct Memory Bytes
360134  23 
23   0   333975216  
1   -1

(wait)
nodetool gcstats
Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed
(ms)   GC Reclaimed (MB) Collections  Direct Memory Bytes
   60016   0   0
NaN   0   0   -1

nodetool repair
[2017-01-11 14:00:45,888] Replication factor is 1. No repair is needed
for keyspace 'system_auth'
[2017-01-11 14:00:45,896] Starting repair command #5, repairing keyspace
system_traces with repair options (parallelism: parallel, primary range:
false, incremental: true, job threads: 1, ColumnFamilies: [],
dataCenters: [], hosts: [], # of ranges: 515)
... 4 pages of data
, (94613607632078948,219237792837906432],
(6033828815719998292,6079920177089043443]] finished (progress: 1%)
[2017-01-11 14:00:46,567] Repair completed successfully
[2017-01-11 14:00:46,576] Repair command #5 finished in 0 seconds

nodetool gcstats
Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed
(ms)   GC Reclaimed (MB) Collections  Direct Memory Bytes
   9169  25  25  
0   330518688   1   -1


Always in loop, I think!

Thanks in advance.

52 matches

Mail list logo