Re: Duplicate records returned

[Eric Stevens]

Can you give us an example of the duplicate records that comes back?  How
reliable is it (i.e. is it every record, is it one record per read, etc)?
By any chance is it just the `data` field that duplicates while the other
fields change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause
it to happen.  Not to say it's impossible, but Cassandra's standard read
path involves collapsing duplicate or otherwise overlapping answers from
multiple replicas; such a thing would be a pretty substantial deviation.
Especially since you don't see the duplicates in cqlsh, I have a hunch this
is an application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille  wrote:

> When I run the query "SELECT image FROM roll WHERE roll = :roll“ against
> this table
>
> CREATE TABLE roll (
> roll INT,
> image BIGINT,
> data VARCHAR static,
> mid VARCHAR,
> imp_st VARCHAR,
> PRIMARY KEY ((roll), image)
> ) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' :
> 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };
>
> I often get duplicate records back. Seems like a very simple query to
> botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1.
> I don’t see duplicates in cqlsh. Any thoughts?
>
> Thanks
>
> Robert
>
>

Manually expire OpsCenter metric data in cassandra

[John Wong]

Hi.

I am using Cassandra 1.2.15 and OpsCenter 5.0.2 (non-enterprise). We are
planning to upgrade in the upcoming week.

I found some old opscenter tables still lurking around.

/mnt/data/OpsCenter# du -h
68K./events_timeline
18M./rollups86400
172K./pdps
269M./rollups7200
68K./events
4.0K./bestpractice_results
228K./settings
5.0G./rollups300
28G./rollups60
34G.

and if we look at rollups60

/mnt/data/OpsCenter/rollups60# ll
total 29195188
drwxr-xr-x  2 cassandra cassandra  20480 Oct  3 19:59 ./
drwxr-xr-x 11 cassandra cassandra   4096 Jun 13 04:54 ../
-rw-r--r--  1 cassandra cassandra1064886 Jul 15 01:33
OpsCenter-rollups60-ic-212-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra 2980474664 Jul 15 01:33
OpsCenter-rollups60-ic-212-Data.db
-rw-r--r--  1 cassandra cassandra   1776 Jul 15 01:33
OpsCenter-rollups60-ic-212-Filter.db
-rw-r--r--  1 cassandra cassandra2002960 Jul 15 01:33
OpsCenter-rollups60-ic-212-Index.db

-rw-r--r--  1 cassandra cassandra 5706306736 Oct  3 20:42
OpsCenter-rollups60-tmp-ic-672-Data.db
-rw-r--r--  1 cassandra cassandra3670016 Oct  3 20:42
OpsCenter-rollups60-tmp-ic-672-Index.db


But we have been using the expire technique as described in
http://docs.datastax.com/en/opscenter/5.1/opsc/configure/opscChangingPerformanceDataExpiration_t.html#opsc_changing_performance_data_expiration_times_t__clusternameconf_unique_1
.

Our config is expire 2 hour after 28 days.
[cassandra_metrics]
1min_ttl = 86400
5min_ttl = 604800
2hr_ttl = 2419200

Looking at one of the rollup tables:

cqlsh:OpsCenter> select * from rollups300 limit 10;

 key|
column1   | value
+---+
 10.0.1.1-SOME_CF-notifications-getRecentBloomFilterFalseRatio | 703578847
| 0x
 10.0.1.1-SOME_CF-getRecentBloomFilterFalseRatio | 703579147 |
0x


Here is my TTL query:
cqlsh:OpsCenter> select TTL (value) from rollups300 limit 2;

 ttl(value)

 604500
 604194

I can see TTL is going down each time I query.

How do I expire these rolllup data myself? Or to be exactly, why do I still
have data from July? Also, how can I confirm the TTL is actually set and
working?
I am okay with truncating the entire keyspace once but I'd like to know if
there are more granular options in case this occurs again.

Thanks.
John

Re: Duplicate records returned

[Robert Wille]

Oops, I was trimming out some irrelevant stuff, and trimmed out too much. The 
second snippet should be this:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;


for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

if (imageId == lastImageId)
{
logger.warn("Cassandra duplicated " + imageId);
continue;
}

total++;
lastImageId = imageId;
}


On Oct 3, 2015, at 10:54 AM, Robert Wille 
> wrote:

I don’t think its an application problem. The following simple snippets produce 
different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

-

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem 
(maybe 20% do this). But the ones that do repeat records, do so 
deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data 
field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens 
> wrote:

Can you give us an example of the duplicate records that comes back?  How 
reliable is it (i.e. is it every record, is it one record per read, etc)?  By 
any chance is it just the `data` field that duplicates while the other fields 
change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it 
to happen.  Not to say it's impossible, but Cassandra's standard read path 
involves collapsing duplicate or otherwise overlapping answers from multiple 
replicas; such a thing would be a pretty substantial deviation.  Especially 
since you don't see the duplicates in cqlsh, I have a hunch this is an 
application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille 
> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this 
table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 
'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. 
I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t 
see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

[Robert Wille]

I don’t think its an application problem. The following simple snippets produce 
different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

-

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem 
(maybe 20% do this). But the ones that do repeat records, do so 
deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data 
field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens 
> wrote:

Can you give us an example of the duplicate records that comes back?  How 
reliable is it (i.e. is it every record, is it one record per read, etc)?  By 
any chance is it just the `data` field that duplicates while the other fields 
change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it 
to happen.  Not to say it's impossible, but Cassandra's standard read path 
involves collapsing duplicate or otherwise overlapping answers from multiple 
replicas; such a thing would be a pretty substantial deviation.  Especially 
since you don't see the duplicates in cqlsh, I have a hunch this is an 
application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille 
> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this 
table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 
'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. 
I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t 
see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

[Robert Wille]

It's a paging bug. I ALWAYS get a duplicated record every fetchSize records. 
Easily duplicated 100% of the time.

I’ve logged a bug: https://issues.apache.org/jira/browse/CASSANDRA-10442

Robert

On Oct 3, 2015, at 10:59 AM, Robert Wille 
> wrote:

Oops, I was trimming out some irrelevant stuff, and trimmed out too much. The 
second snippet should be this:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;

for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

if (imageId == lastImageId)
{
logger.warn("Cassandra duplicated " + imageId);
continue;
}

total++;
lastImageId = imageId;
}


On Oct 3, 2015, at 10:54 AM, Robert Wille 
> wrote:

I don’t think its an application problem. The following simple snippets produce 
different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

-

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem 
(maybe 20% do this). But the ones that do repeat records, do so 
deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data 
field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens 
> wrote:

Can you give us an example of the duplicate records that comes back?  How 
reliable is it (i.e. is it every record, is it one record per read, etc)?  By 
any chance is it just the `data` field that duplicates while the other fields 
change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it 
to happen.  Not to say it's impossible, but Cassandra's standard read path 
involves collapsing duplicate or otherwise overlapping answers from multiple 
replicas; such a thing would be a pretty substantial deviation.  Especially 
since you don't see the duplicates in cqlsh, I have a hunch this is an 
application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille 
> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this 
table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 
'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. 
I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t 
see duplicates in cqlsh. Any thoughts?

Thanks

Robert