Cassandra adding node issue (no UJ status)

[Rock Zhang]

Hi All,


Now I got a problem everything when add new node, the data is not balance
again, just add it as new node.


Originally I saw the UJ status, anybody experienced this kind of issue ? I
don't know what changed .


ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata

Datacenter: DC1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  AddressLoad   Tokens  Owns (effective)  Host ID
  Rack

UN  172.31.16.191  742.15 GB  256 21.1%
08638815-b721-46c4-b77c-af08285226db  RAC1

UN  172.31.30.145  774.42 GB  256 21.2%
bde31dd9-ff1d-4f2f-b28d-fe54d0531c51  RAC1

UN  172.31.6.79592.9 GB   256 19.8%
15795fca-5425-41cd-909c-c1756715442a  RAC2

UN  172.31.27.186  674.42 GB  256 18.4%
9685a476-1da7-4c6f-819e-dd4483e3345e  RAC1

UN  172.31.7.31642.47 GB  256 19.9%
f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b  RAC1

*UN  172.31.15.242  37.4 MB256 19.8%
c3eff010-9904-49a0-83cd-258dc5a98525  RAC1*

UN  172.31.24.32   780.59 GB  256 20.1%
ffa58bd1-3188-440d-94c9-97166ee4b735  RAC1

*UN  172.31.3.4080.75 GB   256 18.9%
01ce3f96-ebc0-4128-9ec3-ddd1a9845d51  RAC1*

UN  172.31.31.238  756.59 GB  256 19.9%
82d34a3b-4f12-4874-816c-7d89a0535577  RAC1

UN  172.31.31.99   583.68 GB  256 20.8%
2b10194f-23d2-4bdc-bcfa-7961a149cd11  RAC2


Thanks

Rock

RE: Using DTCS, TTL but old SSTables not being removed

[Jacques-Henri Berthemet]

Hi,

Any idea when 2.2.2 will be released?
I see there are still 3 issues left to fix:
https://issues.apache.org/jira/browse/CASSANDRA/fixforversion/1219/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel


--
Jacques-Henri Berthemet

-Original Message-
From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com] 
Sent: dimanche 13 septembre 2015 20:34
To: user@cassandra.apache.org
Subject: Re: Using DTCS, TTL but old SSTables not being removed

2.2.1 has a pretty significant bug in compaction: 
https://issues.apache.org/jira/browse/CASSANDRA-10270

That prevents it from compacting files after 60 minutes. It may or may not be 
the cause of the problem you’re seeing, but it seems like it may be possibly 
related, and you can try the workaround in that ticket to see if it helps.





On 9/13/15, 10:54 AM, "Phil Budne"  wrote:

>Running Cassandra 2.2.1 on 3 nodes (on EC2, from Datastax AMI, then
>upgraded).  Inserting time-series data; All entries with TTL to expire
>3 hours after the "actual_time" of the observation.  Entries arrive
>with varied delay, and often in duplicate. Data is expiring (no longer
>visible from CQL), but old SSTables are not being removed (except on
>restart).
>
>CREATE KEYSPACE thing
>WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'}
>AND durable_writes = true;
>
>CREATE TABLE thing.thing_ia (
>id int,
>actual_time timestamp,
>data text,
>PRIMARY KEY (id, actual_time)
>) WITH CLUSTERING ORDER BY (actual_time ASC)
>AND bloom_filter_fp_chance = 0.01
>AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>AND comment = ''
>AND compaction = {'tombstone_threshold': '0.1', 
> 'tombstone_compaction_interval': '600', 'class': 
> 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'}
>AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>AND dclocal_read_repair_chance = 0.1
>AND default_time_to_live = 0
>AND gc_grace_seconds = 60
>AND max_index_interval = 2048
>AND memtable_flush_period_in_ms = 0
>AND min_index_interval = 128
>AND read_repair_chance = 0.0
>AND speculative_retry = '99.0PERCENTILE';
>
>All times shown in UTC:
>
>$ python -c 'import time; print int(time.time())'
>1442166347
>
>$ date
>Sun Sep 13 17:46:19 UTC 2015
>
>$ cat ~/mmm.sh
>for x in la-*Data.db; do
>ls -l $x
>~/meta.sh $x >/tmp/mmm/$x
>head < /tmp/mmm/$x
>echo 
>grep Ances /tmp/mmm/$x
>echo ''
>done
>
>$ sh ~/mmm.sh 
>-rw-r--r-- 1 cassandra cassandra 31056032 Sep 12 05:41 la-203-big-Data.db
>SSTable: /raid0/cassandra/data/thing.thing_ia-.../la-203-big
>Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>Bloom Filter FP chance: 0.01
>Minimum timestamp: 1442025790163000
>Maximum timestamp: 1442034620451000
>SSTable max local deletion time: 1442045239
>Compression ratio: -1.0
>Estimated droppable tombstones: 0.946418951062831
>SSTable Level: 0
>Repaired at: 0
>
>Ancestors: [202]
>
>-rw-r--r-- 1 cassandra cassandra 23647585 Sep 12 06:09 la-204-big-Data.db
>SSTable: /raid0/cassandra/data/thing.thing_ia-.../la-204-big
>Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>Bloom Filter FP chance: 0.01
>Minimum timestamp: 1442034620472000
>Maximum timestamp: 1442038188419002
>SSTable max local deletion time: 1442073136
>Compression ratio: -1.0
>Estimated droppable tombstones: 0.9163514458998852
>SSTable Level: 0
>Repaired at: 0
>
>Ancestors: []
>
>-rw-r--r-- 1 cassandra cassandra 23456946 Sep 12 07:25 la-205-big-Data.db
>SSTable: /raid0/cassandra/data/thing.thing_ia-.../la-205-big
>Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>Bloom Filter FP chance: 0.01
>Minimum timestamp: 1442038188472000
>Maximum timestamp: 1442042703834001
>SSTable max local deletion time: 1442053303
>Compression ratio: -1.0
>Estimated droppable tombstones: 0.9442594560554178
>SSTable Level: 0
>Repaired at: 0
>
>Ancestors: []
>
>-rw-r--r-- 1 cassandra cassandra 23331024 Sep 12 08:11 la-206-big-Data.db
>SSTable: /raid0/cassandra/data/thing.thing_ia-.../la-206-big
>Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>Bloom Filter FP chance: 0.01
>Minimum timestamp: 1442042703845000
>Maximum timestamp: 1442045482391000
>SSTable max local deletion time: 1442056194
>Compression ratio: -1.0
>Estimated droppable tombstones: 0.922422134865437
>SSTable Level: 0
>Repaired at: 0
>
>Ancestors: []
>
>-rw-r--r-- 1 cassandra cassandra 23699494 Sep 12 09:11 la-207-big-Data.db
>SSTable: /raid0/cassandra/data/thing.thing_ia-.../la-207-big
>Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>Bloom Filter FP chance: 0.01
>Minimum timestamp: 1442045482398001
>Maximum timestamp: 144204909216
>SSTable max local deletion time: 1442059681
>Compression ratio: -1.0
>Estimated droppable tombstones: 0.9327568753815364
>SSTable Level: 0
>Repaired at: 0
>
>Ances

Re: Tag filtering data model

[Carlos Alonso]

Really interesting question Artur. Have you gone any further?

I think, based on my experience and recalling Cassandra's good practices,
that full denormalisation is the Cassandra way to go.

Cheers

Carlos Alonso | Software Engineer | @calonso 

On 11 September 2015 at 08:53, Artur Siekielski  wrote:

> I store documents submitted by users, with optional tags (lists of
> strings):
>
> CREATE TABLE doc (
>   user_id uuid,
>   date text, // part of partition key, to distribute data better
>   doc_id uuid,
>   tags list,
>   contents text,
>   PRIMARY KEY((user_id, date), doc_id)
> );
>
> What is the best way to implement tag filtering? A user can select a list
> of tags and get documents with the tags. I thought about:
>
> 1) Full denormalization - include tags in the primary key and insert a doc
> for each subset of specified tags. This will however lead to large disk
> space usage, because there are 2**n subsets (for 10 tags and a 1MB doc
> 1000MB would be written).
>
> 2) Secondary index on 'tags' collection, and using queries like:
> SELECT * FROM doc WHERE user_id=? AND date=? AND tags CONTAINS=? AND tags
> CONTAINS=? ...
>
> Since I will supply partition key value, I assume there will be no
> problems with contacting multiple nodes. But how well will it work for
> hundreds of thousands of results? I think intersection of tag matches needs
> to be performed in memory so it will not scale well.
>
> 3) Partial denormalization - do inserts for each single tag and then
> manually compute intersection. However in the worst case it can lead to
> scanning almost the whole table.
>
> 4) Full denormalization but without contents. I would get correct doc_ids
> fast, then I would need to use '... WHERE doc_id IN ?' with potentially a
> very large list of doc_ids.
>
>
> What's Cassandra's way to implement this?
>

Currupt sstables when upgrading from 2.1.8 to 2.1.9

[George Sigletos]

Hello,

I tried to upgrade two of our clusters from 2.1.8 to 2.1.9. In some, but
not all nodes, I got errors about corrupt sstables when restarting. I
downgraded back to 2.1.8 for now.

Has anybody else faced the same problem? Should sstablescrub fix the
problem? I ddin't tried that yet.

Kind regards,
George

ERROR [SSTableBatchOpen:3] 2015-09-14 10:16:03,296 FileUtils.java:447 -
Exiting forcefully due to file system exception on startup, disk failure
policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
~[apache-cassandra-2.1.9.jar:2.1.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) [na:1.7.0_75]
at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_75]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_75]
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readUnsignedShort(Unknown Source)
~[na:1.7.0_75]
at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75]
at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75]
at
org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106)
~[apache-cassandra-2.1.9.jar:2.1.9]

Re: Cassandra adding node issue (no UJ status)

[Mark Greene]

Hey Rock,

I've seen this occur as well. I've come to learn that in some cases, like a
network blip, the join can fail. There is usually something in the log to
the effect of "Stream failed"

When I encounter this issue, I make an attempt to bootstrap the new node
again. If that doesn't help, I run a repair on the new node.

On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang  wrote:

> Hi All,
>
>
> Now I got a problem everything when add new node, the data is not balance
> again, just add it as new node.
>
>
> Originally I saw the UJ status, anybody experienced this kind of issue ? I
> don't know what changed .
>
>
> ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata
>
> Datacenter: DC1
>
> ===
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  AddressLoad   Tokens  Owns (effective)  Host ID
> Rack
>
> UN  172.31.16.191  742.15 GB  256 21.1%
> 08638815-b721-46c4-b77c-af08285226db  RAC1
>
> UN  172.31.30.145  774.42 GB  256 21.2%
> bde31dd9-ff1d-4f2f-b28d-fe54d0531c51  RAC1
>
> UN  172.31.6.79592.9 GB   256 19.8%
> 15795fca-5425-41cd-909c-c1756715442a  RAC2
>
> UN  172.31.27.186  674.42 GB  256 18.4%
> 9685a476-1da7-4c6f-819e-dd4483e3345e  RAC1
>
> UN  172.31.7.31642.47 GB  256 19.9%
> f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b  RAC1
>
> *UN  172.31.15.242  37.4 MB256 19.8%
> c3eff010-9904-49a0-83cd-258dc5a98525  RAC1*
>
> UN  172.31.24.32   780.59 GB  256 20.1%
> ffa58bd1-3188-440d-94c9-97166ee4b735  RAC1
>
> *UN  172.31.3.4080.75 GB   256 18.9%
> 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51  RAC1*
>
> UN  172.31.31.238  756.59 GB  256 19.9%
> 82d34a3b-4f12-4874-816c-7d89a0535577  RAC1
>
> UN  172.31.31.99   583.68 GB  256 20.8%
> 2b10194f-23d2-4bdc-bcfa-7961a149cd11  RAC2
>
>
> Thanks
>
> Rock
>

Re: Cassandra adding node issue (no UJ status)

[Sebastian Estevez]

Check https://issues.apache.org/jira/browse/CASSANDRA-8611

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]





DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Tue, Sep 15, 2015 at 8:44 AM, Mark Greene  wrote:

> Hey Rock,
>
> I've seen this occur as well. I've come to learn that in some cases, like
> a network blip, the join can fail. There is usually something in the log to
> the effect of "Stream failed"
>
> When I encounter this issue, I make an attempt to bootstrap the new node
> again. If that doesn't help, I run a repair on the new node.
>
>
> On Tue, Sep 15, 2015 at 3:14 AM, Rock Zhang  wrote:
>
>> Hi All,
>>
>>
>> Now I got a problem everything when add new node, the data is not balance
>> again, just add it as new node.
>>
>>
>> Originally I saw the UJ status, anybody experienced this kind of issue ?
>> I don't know what changed .
>>
>>
>> ubuntu@ip-172-31-15-242:/etc/cassandra$ nodetool status rawdata
>>
>> Datacenter: DC1
>>
>> ===
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  AddressLoad   Tokens  Owns (effective)  Host ID
>> Rack
>>
>> UN  172.31.16.191  742.15 GB  256 21.1%
>> 08638815-b721-46c4-b77c-af08285226db  RAC1
>>
>> UN  172.31.30.145  774.42 GB  256 21.2%
>> bde31dd9-ff1d-4f2f-b28d-fe54d0531c51  RAC1
>>
>> UN  172.31.6.79592.9 GB   256 19.8%
>> 15795fca-5425-41cd-909c-c1756715442a  RAC2
>>
>> UN  172.31.27.186  674.42 GB  256 18.4%
>> 9685a476-1da7-4c6f-819e-dd4483e3345e  RAC1
>>
>> UN  172.31.7.31642.47 GB  256 19.9%
>> f7c8c6fb-ab37-4124-ba1a-a9a1beaecc1b  RAC1
>>
>> *UN  172.31.15.242  37.4 MB256 19.8%
>> c3eff010-9904-49a0-83cd-258dc5a98525  RAC1*
>>
>> UN  172.31.24.32   780.59 GB  256 20.1%
>> ffa58bd1-3188-440d-94c9-97166ee4b735  RAC1
>>
>> *UN  172.31.3.4080.75 GB   256 18.9%
>> 01ce3f96-ebc0-4128-9ec3-ddd1a9845d51  RAC1*
>>
>> UN  172.31.31.238  756.59 GB  256 19.9%
>> 82d34a3b-4f12-4874-816c-7d89a0535577  RAC1
>>
>> UN  172.31.31.99   583.68 GB  256 20.8%
>> 2b10194f-23d2-4bdc-bcfa-7961a149cd11  RAC2
>>
>>
>> Thanks
>>
>> Rock
>>
>
>

Re: LTCS Strategy Resulting in multiple SSTables

[Saladi Naidu]

We are on 2.1.2 and planning to upgrade to 2.1.9  Naidu Saladi 

  From: Marcus Eriksson 
 To: user@cassandra.apache.org; Saladi Naidu  
 Sent: Tuesday, September 15, 2015 1:53 AM
 Subject: Re: LTCS Strategy Resulting in multiple SSTables
   
if you are on Cassandra 2.2, it is probably this: 
https://issues.apache.org/jira/browse/CASSANDRA-10270


On Tue, Sep 15, 2015 at 4:37 AM, Saladi Naidu  wrote:

We are using Level Tiered Compaction Strategy on a Column Family. Below are 
CFSTATS from two nodes in same cluster, one node has 880 SStables in L0 whereas 
one node just has 1 SSTable in L0. In the node where there are multiple 
SStables, all of them are small size and created same time stamp. We ran 
Compaction, it did not result in much change, node remained with huge number of 
SStables. Due to this large number of SSTables, Read performance is being 
impacted
In same cluster, under same keyspace, we are observing this discrepancy in 
other column families as well. What is going wrong? What is the solution to fix 
this
---NODE1---   Table: category_ranking_dedup 
  SSTable count: 1   SSTables in each level: 
[1, 0, 0, 0, 0, 0, 0, 0, 0]   Space used (live): 
2012037   Space used (total): 2012037   
Space used by snapshots (total): 0  
 SSTable Compression Ratio: 0.07677216119569073   
Memtable cell count: 990   Memtable data size: 
32082   Memtable switch count: 11   
Local read count: 2842   Local read 
latency: 3.215 ms   Local write count: 18309
   Local write latency: 5.008 ms
   Pending flushes: 0   Bloom filter false 
positives: 0   Bloom filter false ratio: 0.0
   Bloom filter space used: 816 
  Compacted partition minimum bytes: 87   
Compacted partition maximum bytes: 25109160   
Compacted partition mean bytes: 22844   Average 
live cells per slice (last five minutes): 338.84588318085855
   Maximum live cells per slice (last five minutes): 10002.0
   Average tombstones per slice (last five minutes): 
36.53307529908515   Maximum tombstones per slice 
(last five minutes): 36895.0 NODE2---  Table: category_ranking_dedup
   SSTable count: 808   
SSTables in each level: [808/4, 0, 0, 0, 0, 0, 0, 0, 0] 
  Space used (live): 291641980   Space used 
(total): 291641980   Space used by snapshots 
(total): 0   SSTable Compression Ratio: 
0.1431106696818256   Memtable cell count: 4365293   
Memtable data size: 3742375 
  Memtable switch count: 44   Local read count: 
2061   Local read latency: 31.983 ms
   Local write count: 30096   Local 
write latency: 27.449 ms   Pending flushes: 0   
Bloom filter false positives: 0 
  Bloom filter false ratio: 0.0   Bloom 
filter space used: 54544   Compacted partition 
minimum bytes: 87   Compacted partition maximum 
bytes: 25109160   Compacted partition mean bytes: 
634491   Average live cells per slice (last five 
minutes): 416.1780688985929   Maximum live cells 
per slice (last five minutes): 10002.0   Average 
tombstones per slice (last five minutes): 45.11547792333818 
  Maximum tombstones per slice (last five minutes): 36895.0


 Naidu Saladi 




   

Secondary index is causing high CPU load

[Tom van den Berge]

Read queries on a secondary index are somehow causing an excessively high
CPU load on all nodes in my DC.

The table has some 60K records, and the cardinality of the index is very
low (~10 distinct values). The returned result set typically contains
10-30K records.
The same queries on nodes in another DC are working fine. The nodes with
the high CPU are in a newly set up DC (see my previous message below). The
hardware in both DCs is the same, as well as the C* version (2.1.6). The
only difference in the C* setup is that the new DC is using vnodes (256),
while the old DC is not. Both DCs have 4 nodes, and RF=2.

I've rebuilt the index, but that didn't help.

It looks a bit like CASSANDRA-8530
 (unresolved).

What really surprised me is that executing a single query on this secondary
index makes the "Local read count" in the cfstats for the index go up with
almost 20! When doing the same query on one of my "good" nodes, it only
increases with a small number, as I would expect.

Could it be that the use of vnodes is causing these problems?

Regards,
Tom



On Mon, Sep 14, 2015 at 8:09 PM, Tom van den Berge <
tom.vandenbe...@gmail.com> wrote:

> I have a DC of 4 nodes that must be expanded to accommodate an expected
> growth in data. Since the DC is not using vnodes, we have decided to set up
> a new DC with vnodes enabled, start using the new DC, and decommission the
> old DC.
>
> Both DCs have 4 nodes. The idea is to add additional nodes to the new DC
> later on.
> The servers in both DCs are very similar: quad-core machines with 8GB.
>
> We have bootstrapped/rebuilt the nodes in the new DC. When that finished,
> the nodes in the new DC were showing little CPU activity, as you would
> expect, because they are receiving writes from the other DC. So far, so
> good.
>
> Then we switched the clients from the old DC to the new DC. The CPU load
> on all nodes in the new DC immediately rose to excessively high levels (15
> - 25), which made the servers effectively unavailable. The load did not
> drop structurally within 20 minutes, so we had to switch the clients back
> to the old DC. Then the load dropped again.
>
> What can be the reason for the high CPU loads on the new nodes?
>
> Performance test shows that the servers in the new DC perform slightly
> better (both IO and CPU) than the servers in the old DC.
> I did not see anything abnormal in the Cassandra logs, like garbage
> collection warnings. I also did not see any strange things in the tpstats.
> The only difference I'm aware of between the old and new DC is the use of
> vnodes.
>
> Any help is appreciated!
> Thanks,
> Tom
>

Re: Currupt sstables when upgrading from 2.1.8 to 2.1.9

[Nate McCall]

You have a/some corrupt SSTables. 2.1.9 is doing strict checking at startup
and reacting based on "disk_failure_policy" per the stack trace.

For details, see:
https://issues.apache.org/jira/browse/CASSANDRA-9686

Either way, you are going to have to run nodetool scrub. I'm not sure if
it's better to do this from 2.1.8 or from 2.1.9 with "disk_failure_policy:
ignore"

It feels like that option got overloaded a bit strangely with the changes
in CASSANDRA-9686 and I have not yet tried it with it's new meaning.

On Tue, Sep 15, 2015 at 5:26 AM, George Sigletos 
wrote:

> Hello,
>
> I tried to upgrade two of our clusters from 2.1.8 to 2.1.9. In some, but
> not all nodes, I got errors about corrupt sstables when restarting. I
> downgraded back to 2.1.8 for now.
>
> Has anybody else faced the same problem? Should sstablescrub fix the
> problem? I ddin't tried that yet.
>
> Kind regards,
> George
>
> ERROR [SSTableBatchOpen:3] 2015-09-14 10:16:03,296 FileUtils.java:447 -
> Exiting forcefully due to file system exception on startup, disk failure
> policy "stop"
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException
> at
> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at
> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) [na:1.7.0_75]
> at java.util.concurrent.FutureTask.run(Unknown Source)
> [na:1.7.0_75]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) [na:1.7.0_75]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) [na:1.7.0_75]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_75]
> Caused by: java.io.EOFException: null
> at java.io.DataInputStream.readUnsignedShort(Unknown Source)
> ~[na:1.7.0_75]
> at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75]
> at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106)
> ~[apache-cassandra-2.1.9.jar:2.1.9]
>



-- 
-
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Concurrency in Cassandra

[Thouraya TH]

Hi all;


Please, is it possible to use insert queries in parallel at the same time
in two different nodes ? Or, i will a concurrency problem ?



Thanks you so much for answers.
Best regards.

Re: Concurrency in Cassandra

[Jonathan Haddad]

You may want to take an hour and watch a video on Cassandra fundamentals.
It'll answer a lot of the questions you're likely to ask next, including
this one.

https://academy.datastax.com/courses/ds101-introduction-cassandra

On Tue, Sep 15, 2015 at 2:04 PM Thouraya TH  wrote:

> Hi all;
>
>
> Please, is it possible to use insert queries in parallel at the same time
> in two different nodes ? Or, i will a concurrency problem ?
>
>
>
> Thanks you so much for answers.
> Best regards.
>

Re: LTCS Strategy Resulting in multiple SSTables

[Nate McCall]

That's an early 2.1/known buggy version. There have been several issues
fixed since which could cause that behavior. Most likely
https://issues.apache.org/jira/browse/CASSANDRA-9592 ?

Upgrade to 2.1.9 and see if the problem persists.

On Tue, Sep 15, 2015 at 8:31 AM, Saladi Naidu  wrote:

> We are on 2.1.2 and planning to upgrade to 2.1.9
>
> Naidu Saladi
>
> --
> *From:* Marcus Eriksson 
> *To:* user@cassandra.apache.org; Saladi Naidu 
> *Sent:* Tuesday, September 15, 2015 1:53 AM
> *Subject:* Re: LTCS Strategy Resulting in multiple SSTables
>
> if you are on Cassandra 2.2, it is probably this:
> https://issues.apache.org/jira/browse/CASSANDRA-10270
>
>
>
> On Tue, Sep 15, 2015 at 4:37 AM, Saladi Naidu 
> wrote:
>
> We are using Level Tiered Compaction Strategy on a Column Family. Below
> are CFSTATS from two nodes in same cluster, one node has 880 SStables in L0
> whereas one node just has 1 SSTable in L0. In the node where there are
> multiple SStables, all of them are small size and created same time stamp.
> We ran Compaction, it did not result in much change, node remained with
> huge number of SStables. Due to this large number of SSTables, Read
> performance is being impacted
>
> In same cluster, under same keyspace, we are observing this discrepancy in
> other column families as well. What is going wrong? What is the solution to
> fix this
>
> *---*NODE1*---*
> *Table: category_ranking_dedup*
> *SSTable count: 1*
> *SSTables in each level: [1, 0, 0, 0, 0,
> 0, 0, 0, 0]*
> *Space used (live): 2012037*
> *Space used (total): 2012037*
> *Space used by snapshots (total): 0*
> *SSTable Compression Ratio:
> 0.07677216119569073*
> *Memtable cell count: 990*
> *Memtable data size: 32082*
> *Memtable switch count: 11*
> *Local read count: 2842*
> *Local read latency: 3.215 ms*
> *Local write count: 18309*
> *Local write latency: 5.008 ms*
> *Pending flushes: 0*
> *Bloom filter false positives: 0*
> *Bloom filter false ratio: 0.0*
> *Bloom filter space used: 816*
> *Compacted partition minimum bytes: 87*
> *Compacted partition maximum bytes:
> 25109160*
> *Compacted partition mean bytes: 22844*
> *Average live cells per slice (last five
> minutes): 338.84588318085855*
> *Maximum live cells per slice (last five
> minutes): 10002.0*
> *Average tombstones per slice (last five
> minutes): 36.53307529908515*
> *Maximum tombstones per slice (last five
> minutes): 36895.0*
>
> *NODE2---  *
> *Table: category_ranking_dedup*
> *SSTable count: 808*
> *SSTables in each level: [808/4, 0, 0, 0,
> 0, 0, 0, 0, 0]*
> *Space used (live): 291641980*
> *Space used (total): 291641980*
> *Space used by snapshots (total): 0*
> *SSTable Compression Ratio:
> 0.1431106696818256*
> *Memtable cell count: 4365293*
> *Memtable data size: 3742375*
> *Memtable switch count: 44*
> *Local read count: 2061*
> *Local read latency: 31.983 ms*
> *Local write count: 30096*
> *Local write latency: 27.449 ms*
> *Pending flushes: 0*
> *Bloom filter false positives: 0*
> *Bloom filter false ratio: 0.0*
> *Bloom filter space used: 54544*
> *Compacted partition minimum bytes: 87*
> *Compacted partition maximum bytes:
> 25109160*
> *Compacted partition mean bytes: 634491*
> *Average live cells per slice (last five
> minutes): 416.1780688985929*
> *Maximum live cells per slice (last five
> minutes): 10002.0*
> *Average tombstones per slice (last five
> minutes): 45.11547792333818*
> *Maximu