[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154085#comment-13154085
]
Aaron Morton commented on CASSANDRA-3510:
-
To be clear, I do not think this is an issue with super CF's. The Schema CF is
a standard, it was noticed on a super CF.
I added some more logging and removed the call to cut the query short by
checking maxTimestamp in CollationController.collectTimeOrderedData(). The
query returned the result value (password col with timestamp 1319563673493000)
and this was the output
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
//reduceNameFilter(reducedFilter, container, currentMaxTs);
//if (((NamesQueryFilter)
reducedFilter.filter).columns.isEmpty())
//break;
IColumnIterator iter = reducedFilter.getSSTableColumnIterator(sstable);
iterators.add(iter);
if (iter.getColumnFamily() != null)
{
container.delete(iter.getColumnFamily());
sstablesIterated++;
while (iter.hasNext())
{
IColumn col = iter.next();
if (col instanceof SuperColumn)
{
for (IColumn subcol : ((SuperColumn)col).columns)
{
if (subcol.name().equals(ByteBufferUtil.bytes(password)))
logger.debug(String.format(Add Sub Column %s,
subcol.getString(cfs.metadata.subcolumnComparator)));
}
}
else
{
logger.debug(String.format(Add Column %s,
col.getString(cfs.metadata.comparator)));
}
container.addColumn(col);
}
{code}
{noformat}
DEBUG 22:05:20,748 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and max
TS 1321824847534000
DEBUG 22:05:20,749 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and max
TS 1321813380793000
DEBUG 22:05:20,753 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,754 Add Sub Column password:false:8@1307352647576000
DEBUG 22:05:20,755 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,757 Add Sub Column password:false:8@1307352647576000
DEBUG 22:05:20,758 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12327-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,760 Add Sub Column password:false:8@1307352647576000
DEBUG 22:05:20,761 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12328-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,762 Add Sub Column password:false:8@1319563673493000
DEBUG 22:05:20,763 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12326-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,765 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12331-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,767 Add Sub Column password:false:8@1307352647576000
DEBUG 22:05:20,768 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12332-Data.db') and max
TS -9223372036854775808
DEBUG 22:05:20,774 Read: 27 ms.
{noformat}
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154101#comment-13154101 ] Eric Evans commented on CASSANDRA-3507: --- This would be an improvement over the current situation I think. bq. Maybe we even ought to take some minor steps to discourage its use for other purposes. One option would be to script generate some obfuscation. This could be as simple as prepending (or appending) something to each of the methods. Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Minor Labels: cql, cqlsh Fix For: 1.0.4 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Morton updated CASSANDRA-3510:
Attachment: 0002-3510-update-maxTimestamp-during-repair.patch
0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch
patch 0001 is a proof of concept hack based on my first comment, it generated
this output when using the extra logging
{noformat}
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 77)
collectTimeOrderedData
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,583 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,584 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,585 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12325-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,587 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,588 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12327-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,590 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,591 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12328-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,592 CollationController.java (line 130)
Add Sub Column password:false:8@1319563673493000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,593 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12326-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,595 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12331-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,596 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,597 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12332-Data.db') and
max TS -9223372036854775808
DEBUG [pool-2-thread-1] 2011-11-21 23:12:28,604 StorageProxy.java (line 694)
Read: 26 ms.
{noformat}
patch 0002 makes repair update the maxTimestamp and when I ran my test
afterwards it created this output:
{noformat}
DEBUG [ReadStage:33] 2011-11-21 23:20:32,032 CollationController.java (line 77)
collectTimeOrderedData
DEBUG [ReadStage:33] 2011-11-21 23:20:32,033 CollationController.java (line
111) Got sstable
SSTableReader(path='/private/var/lib/cassandra/data/fmm/Users-hc-12357-Data.db')
and max TS 1321824847534000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,033 CollationController.java (line
111) Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hc-12352-Data.db') and
max TS 1321813380793000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,063 CollationController.java (line
111) Got sstable
SSTableReader(path='/private/var/lib/cassandra/data/fmm/Users-hc-12356-Data.db')
and max TS 1321560509938000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,064 CollationController.java (line
111) Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hc-12349-Data.db') and
max TS 1319813295567000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,105 CollationController.java (line
130) Add Sub Column password:false:8@1319563673493000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,105 CollationController.java (line
111) Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hc-12350-Data.db') and
max TS 1318523190222000
DEBUG [pool-2-thread-2] 2011-11-21 23:20:32,106 StorageProxy.java (line 694)
Read: 74 ms.
{noformat}
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
[jira] [Updated] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Morton updated CASSANDRA-3510:
Description:
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a lot
of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to 1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable name
and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and max
TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and max
TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and max
TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and max
TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see the
first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column read
from the 12330 file has a time stamp of 1307352647576000 and the 12325 file
incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta data
collector according to the maxTimestamp in the meta data for the file(s) that
will be scrubbed / compacted. But for pre 1.0 format files the default in
SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and the
ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the SSTable
is originally written. So the error from the old files will be carried along.
e.g. If the files a,b and c have the maxTimestamps 10, 100 and Long.MIN_VALUE
compaction will write a SSTable with maxTimestamp 100. However file c may
actually contain columns with a timestamp 100 which will be in the compacted
file.
h1. Reproduce
1. Start a clean 0.8.7
2. Add a schema (details of the schema do not matter):
{noformat}
[default@unknown] create keyspace dev;
5f834620-140b-11e1--242d50cf1fdf
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown]
[default@unknown] use dev;
Authenticated to keyspace: dev
[default@dev]
[default@dev] create column family super_dev with column_type = 'Super'
... and key_validation_class = 'AsciiType' and comparator = 'AsciiType' and
... subcomparator = 'AsciiType' and default_validation_class = 'AsciiType';
60490720-140b-11e1--242d50cf1fdf
Waiting for schema agreement...
... schemas agree across the cluster
{noformat}
3. Shutdown 0.8.7
4.
[jira] [Issue Comment Edited] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154111#comment-13154111
]
Aaron Morton edited comment on CASSANDRA-3510 at 11/21/11 10:48 AM:
patch 0001 is a proof of concept hack based on my first comment, it generated
this output when using the extra logging
{noformat}
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 77)
collectTimeOrderedData
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,583 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,584 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,585 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12325-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,587 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,588 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12327-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,590 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,591 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12328-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,592 CollationController.java (line 130)
Add Sub Column password:false:8@1319563673493000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,593 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12326-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,595 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12331-Data.db') and
max TS -9223372036854775808
DEBUG [ReadStage:1] 2011-11-21 23:12:28,596 CollationController.java (line 130)
Add Sub Column password:false:8@1307352647576000
DEBUG [ReadStage:1] 2011-11-21 23:12:28,597 CollationController.java (line 111)
Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hb-12332-Data.db') and
max TS -9223372036854775808
DEBUG [pool-2-thread-1] 2011-11-21 23:12:28,604 StorageProxy.java (line 694)
Read: 26 ms.
{noformat}
patch 0002 makes scrub update the maxTimestamp and when I ran my test
afterwards it created this output:
**NOTE** patch 0002 has incorrect file name, it modifies scrub.
{noformat}
DEBUG [ReadStage:33] 2011-11-21 23:20:32,032 CollationController.java (line 77)
collectTimeOrderedData
DEBUG [ReadStage:33] 2011-11-21 23:20:32,033 CollationController.java (line
111) Got sstable
SSTableReader(path='/private/var/lib/cassandra/data/fmm/Users-hc-12357-Data.db')
and max TS 1321824847534000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,033 CollationController.java (line
111) Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hc-12352-Data.db') and
max TS 1321813380793000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,063 CollationController.java (line
111) Got sstable
SSTableReader(path='/private/var/lib/cassandra/data/fmm/Users-hc-12356-Data.db')
and max TS 1321560509938000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,064 CollationController.java (line
111) Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hc-12349-Data.db') and
max TS 1319813295567000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,105 CollationController.java (line
130) Add Sub Column password:false:8@1319563673493000
DEBUG [ReadStage:33] 2011-11-21 23:20:32,105 CollationController.java (line
111) Got sstable
SSTableReader(path='/var/lib/cassandra/data/fmm/Users-hc-12350-Data.db') and
max TS 1318523190222000
DEBUG [pool-2-thread-2] 2011-11-21 23:20:32,106 StorageProxy.java (line 694)
Read: 74 ms.
{noformat}
was (Author: amorton):
patch 0001 is a proof of concept hack based on my first comment, it
generated this output when using the extra logging
{noformat}
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 77)
collectTimeOrderedData
DEBUG [ReadStage:1] 2011-11-21 23:12:28,578 CollationController.java (line 111)
Got sstable
[jira] [Updated] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-3510:
Attachment: 3510.patch
The fix of the first patch looks. For the second patch, even though this is an
option, I think I'd rather make this parts of compaction, because I don't like
too much having potential subtle bug that needs scrub to be run (without really
anything telling you that you potentially have a problem btw).
Attaching a patch that does this (make it work for compaction in general as
long as we're not using an EchoedRow (which makes it work for scrub in
particular)). The patch also adds a few comments and a unit test. It includes
the fix of the first patch.
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp
svn commit: r1204486 - /cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/dht/RandomPartitioner.java
Author: slebresne
Date: Mon Nov 21 13:21:12 2011
New Revision: 1204486
URL: http://svn.apache.org/viewvc?rev=1204486view=rev
Log:
Fix typo
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/dht/RandomPartitioner.java
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/dht/RandomPartitioner.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/dht/RandomPartitioner.java?rev=1204486r1=1204485r2=1204486view=diff
==
---
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/dht/RandomPartitioner.java
(original)
+++
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/dht/RandomPartitioner.java
Mon Nov 21 13:21:12 2011
@@ -168,7 +168,7 @@ public class RandomPartitioner implement
// n-case
else {
// NOTE: All divisions must take place in BigDecimals, and all
modulo operators must take place in BigIntegers.
-final BigInteger ri = MAXIMUM(127); //
(used for addition later)
+final BigInteger ri = MAXIMUM;
// (used for addition later)
final BigDecimal r = new BigDecimal(ri);
// The entire range, 2**127
Token start = (Token)i.next(); BigInteger ti =
((BigIntegerToken)start).token; // The first token and its value
Token t; BigInteger tim1 = ti;
// The last token and its value (after loop)
[jira] [Commented] (CASSANDRA-3511) Key caches are not saved
[ https://issues.apache.org/jira/browse/CASSANDRA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154220#comment-13154220 ] Jonathan Ellis commented on CASSANDRA-3511: --- What is the difference between the CF where saving works and the one where it does not? Is the non-saving behavior reproducible? Are there any exceptions in the log? Is anything logged on startup about the cache that didn't load? Key caches are not saved Key: CASSANDRA-3511 URL: https://issues.apache.org/jira/browse/CASSANDRA-3511 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2, 1.0.3 Reporter: Radim Kolar Priority: Minor cache saving seems to be broken in 1.0.2 and 1.0.3 i have 2 CF in keyspace with enabled cache saving and only one gets its key cache saved. It worked perfectly in 0.8, both were saved. This one works: create column family query2 with column_type = 'Standard' and comparator = 'AsciiType' and default_validation_class = 'BytesType' and key_validation_class = 'UTF8Type' and rows_cached = 500.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 14400 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 5 and max_compaction_threshold = 10 and replicate_on_write = false and row_cache_provider = 'ConcurrentLinkedHashCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' This does not create column family dkb13 with column_type = 'Super' and comparator = 'LongType' and subcomparator = 'AsciiType' and default_validation_class = 'BytesType' and key_validation_class = 'UTF8Type' and rows_cached = 600.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 14400 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 5 and max_compaction_threshold = 10 and replicate_on_write = false and row_cache_provider = 'ConcurrentLinkedHashCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' in second test system i created these 2 column families and none of them got single cache key saved. Both have save period 30 seoonds - their cache should save often. Its not that standard column family works while super does not. create column family test1 with column_type = 'Standard' and comparator = 'BytesType' and default_validation_class = 'BytesType' and key_validation_class = 'BytesType' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 30 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 4 and max_compaction_threshold = 32 and replicate_on_write = true and row_cache_provider = 'SerializingCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'; create column family test2 with column_type = 'Standard' and comparator = 'BytesType' and default_validation_class = 'BytesType' and key_validation_class = 'BytesType' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 30 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 4 and max_compaction_threshold = 32 and replicate_on_write = true and row_cache_provider = 'SerializingCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'; If this is done on purpose for example cassandra 1.0 is doing some heuristic decision if cache should be saved or not then it should be removed. Saving cache is fast. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154222#comment-13154222 ] Jonathan Ellis commented on CASSANDRA-3507: --- bq. One option would be to script generate some obfuscation Ugh. Should we just spin cqlsh out to a separate project? I don't think it belongs in the dbapi tree either. Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Minor Labels: cql, cqlsh Fix For: 1.0.4 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1204529 - /cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java
Author: jbellis
Date: Mon Nov 21 15:01:17 2011
New Revision: 1204529
URL: http://svn.apache.org/viewvc?rev=1204529view=rev
Log:
inline filed initialization that doesn't depend on constructor params
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java?rev=1204529r1=1204528r2=1204529view=diff
==
---
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java
(original)
+++
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java
Mon Nov 21 15:01:17 2011
@@ -104,20 +104,11 @@ public class SSTableMetadata
public static class Collector
{
-protected EstimatedHistogram estimatedRowSize;
-protected EstimatedHistogram estimatedColumnCount;
-protected ReplayPosition replayPosition;
-protected long maxTimestamp;
-protected double compressionRatio;
-
-private Collector()
-{
-this.estimatedRowSize = defaultRowSizeHistogram();
-this.estimatedColumnCount = defaultColumnCountHistogram();
-this.replayPosition = ReplayPosition.NONE;
-this.maxTimestamp = Long.MIN_VALUE;
-this.compressionRatio = Double.MIN_VALUE;
-}
+protected EstimatedHistogram estimatedRowSize =
defaultRowSizeHistogram();
+protected EstimatedHistogram estimatedColumnCount =
defaultColumnCountHistogram();
+protected ReplayPosition replayPosition = ReplayPosition.NONE;
+protected long maxTimestamp = Long.MIN_VALUE;
+protected double compressionRatio = Double.MIN_VALUE;
public void addRowSize(long rowSize)
{
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154228#comment-13154228
]
Jonathan Ellis commented on CASSANDRA-3510:
---
bq.
{noformat}
+ * However, for old sstables without timestamp, we still want to
update the timestamp (and we know
+ * that in this case we will not use EchoedRow).
{noformat}
Where is the don't use echoedrow for old sstables logic?
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have
[jira] [Updated] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-2749: --- Attachment: 2749_not_backwards.tar.gz updated; * sstablemover can now reverse changes (-r) and delete old files (-d). * LegacySSTableTest fixed, hopefully * added comments to cassandra.yaml fine-grained control over data directories -- Key: CASSANDRA-2749 URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Priority: Minor Fix For: 1.1 Attachments: 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, 2749.tar.gz, 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, 2749_backwards_compatible_v4.patch, 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz Currently Cassandra supports multiple data directories but no way to control what sstables are placed where. Particularly for systems with mixed SSDs and rotational disks, it would be nice to pin frequently accessed columnfamilies to the SSDs. Postgresql does this with tablespaces (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we should probably avoid using that name because of confusing similarity to keyspaces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3505) tombstone appears after truncate
[
https://issues.apache.org/jira/browse/CASSANDRA-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-3505:
--
Reviewer: jbellis
Fix Version/s: 1.0.4
Assignee: Yuki Morishita (was: T Jake Luciani)
tombstone appears after truncate
Key: CASSANDRA-3505
URL: https://issues.apache.org/jira/browse/CASSANDRA-3505
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Cathy Daw
Assignee: Yuki Morishita
Fix For: 1.0.4
This bug is regarding the select after the 'truncate'. In 1.0.1 no rows
would ever be returned, but now we are seeing a tombstone when querying for
user1. Jake mentioned this may be related to CASSANDRA-2855.
{code}
cqlsh CREATE KEYSPACE ks1 with
... strategy_class =
... 'org.apache.cassandra.locator.SimpleStrategy'
... and strategy_options:replication_factor=1;
cqlsh use ks1;
cqlsh:ks1
cqlsh:ks1 CREATE COLUMNFAMILY users (
... KEY varchar PRIMARY KEY, password varchar, gender varchar,
... session_token varchar, state varchar, birth_year bigint);
cqlsh:ks1 INSERT INTO users (KEY, password) VALUES ('user1', 'ch@ngem3a');
cqlsh:ks1 UPDATE users SET gender = 'm', birth_year = '1980' WHERE KEY =
'user1';
cqlsh:ks1 SELECT * FROM users WHERE key='user1';
KEY | birth_year | gender | password |
user1 | 1980 | m | ch@ngem3a |
cqlsh:ks1 TRUNCATE users;
// Expected, no rows returned
cqlsh:ks1 SELECT * FROM users WHERE key='user1';
KEY |
user1 |
// Expected, no rows returned
cqlsh:ks1 SELECT * FROM users;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3508) requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option)
[
https://issues.apache.org/jira/browse/CASSANDRA-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154241#comment-13154241
]
Jonathan Ellis commented on CASSANDRA-3508:
---
We need to be a little judicious here. Simply printing stacktraces for *all*
exceptions is not right either. For instance, it looks to me like
{code}
sessionState.err.println(e.getWhy());
-if (sessionState.debug)
-e.printStackTrace();
+e.printStackTrace(sessionState.err);
{code}
{code}
-if (sessionState.debug)
-e.printStackTrace();
-
sessionState.err.println(Login failure. Did you specify
'keyspace', 'username' and 'password'?);
+e.printStackTrace(sessionState.err);
{code}
should not be stacktraces by default.
requiring --debug to see stack traces for failures in cassandra-cli is a
terrible idea (aka silent failure is never a valid option)
---
Key: CASSANDRA-3508
URL: https://issues.apache.org/jira/browse/CASSANDRA-3508
Project: Cassandra
Issue Type: Bug
Components: Tools
Affects Versions: 1.0.0
Reporter: Matthew F. Dennis
Assignee: Pavel Yaskevich
Priority: Minor
Fix For: 1.0.3
Attachments: CASSANDRA-3508.patch
this manifests itself in cassandra-cli by returning null to the user. In
order to see what the problem was (and in many cases, just to know there was
a problem at all) requires running cassandra-cli with --debug
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154246#comment-13154246
]
Sylvain Lebresne commented on CASSANDRA-3510:
-
bq. Where is the don't use echoedrow for old sstables logic?
In CompactionController.needDeserialize(), but I agree that the comment could
be improved to recall it.
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have the maxTimestamps 10, 100 and Long.MIN_VALUE
compaction will write a SSTable
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3442: -- Reviewer: jbellis (was: slebresne) TTL histogram for sstable metadata -- Key: CASSANDRA-3442 URL: https://issues.apache.org/jira/browse/CASSANDRA-3442 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Yuki Morishita Priority: Minor Labels: compaction Fix For: 1.1 Attachments: 3442.txt Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation. If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154250#comment-13154250 ] Jonathan Ellis commented on CASSANDRA-3442: --- Should we do single-sstable compactions *after* the bucket compactions? Doing them first means we might compact them twice, when the bucket-based compaction would have been adequate. It looks like this will never stop recompacting sstables with high expiring column counts, until they finally expire and are expunged. I think we need to address this somehow, possibly by waiting until some fraction of gc_grace_seconds has elapsed since sstable creation (which we can just get from mtime). If we can reasonably test this in CompactionsTest I'd like to add that. TTL histogram for sstable metadata -- Key: CASSANDRA-3442 URL: https://issues.apache.org/jira/browse/CASSANDRA-3442 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Yuki Morishita Priority: Minor Labels: compaction Fix For: 1.1 Attachments: 3442.txt Under size-tiered compaction, you can generate large sstables that compact infrequently. With expiring columns mixed in, we could waste a lot of space in this situation. If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a single-sstable compaction aginst sstables with over 20% (?) expired data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154252#comment-13154252
]
Jonathan Ellis commented on CASSANDRA-3510:
---
Referring to the isLatestVersion check?
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have the maxTimestamps 10, 100 and Long.MIN_VALUE
compaction will write a SSTable with maxTimestamp 100. However file c may
actually contain columns with a timestamp 100 which will be in the
compacted file.
h1.
[jira] [Commented] (CASSANDRA-3508) requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option)
[ https://issues.apache.org/jira/browse/CASSANDRA-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154259#comment-13154259 ] Pavel Yaskevich commented on CASSANDRA-3508: I agree that we can remove it from the first one (describeRing) but I think we should keep stack-trace for login as it could be handy to have. requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option) --- Key: CASSANDRA-3508 URL: https://issues.apache.org/jira/browse/CASSANDRA-3508 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.0.0 Reporter: Matthew F. Dennis Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.0.3 Attachments: CASSANDRA-3508.patch this manifests itself in cassandra-cli by returning null to the user. In order to see what the problem was (and in many cases, just to know there was a problem at all) requires running cassandra-cli with --debug -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3512) Getting Started instructions don't work in README.txt - wrong version of jamm, wrong path
Getting Started instructions don't work in README.txt - wrong version of jamm,
wrong path
-
Key: CASSANDRA-3512
URL: https://issues.apache.org/jira/browse/CASSANDRA-3512
Project: Cassandra
Issue Type: Bug
Components: Packaging
Affects Versions: 1.0.3
Environment: Ubuntu 11.04
Reporter: David Allsopp
Priority: Minor
Download latest release from
http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.0.3/apache-cassandra-1.0.3-bin.tar.gz
Unpack the tarball.
Follow instructions in README.txt, concluding with:
{noformat}
dna@master:~/code/apache-cassandra-1.0.3$ bin/cassandra -f
Error opening zip file or JAR manifest missing : /lib/jamm-0.2.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument
{noformat}
Firstly, the version of jamm packaged with Cassandra 1.0.3 is jamm-0.2.5, not
jamm-0.2.1.
Both bin/cassandra.bat and conf/cassandra-env.sh reference jamm-0.2.5 so not
sure where jamm-0.2.1 is being referenced from - nothing obvious using grep.
Secondly, /lib/jamm-0.2.1.jar is the wrong path - should be set relative to
working directory, not filesystem root
(Incidentally, Cassandra v1.0.3 is still listed as unreleased on JIRA.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3512) Getting Started instructions don't work in README.txt - wrong version of jamm, wrong path
[
https://issues.apache.org/jira/browse/CASSANDRA-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154278#comment-13154278
]
David Allsopp commented on CASSANDRA-3512:
--
I think the problem is that bin/cassandra.bat is detecting a cassandra.in.sh
from a previous install of Cassandra. Setting CASSANDRA_INCLUDE to point
explicitly to bin/cassandra.in.sh fixes the problem.
Getting Started instructions don't work in README.txt - wrong version of
jamm, wrong path
-
Key: CASSANDRA-3512
URL: https://issues.apache.org/jira/browse/CASSANDRA-3512
Project: Cassandra
Issue Type: Bug
Components: Packaging
Affects Versions: 1.0.3
Environment: Ubuntu 11.04
Reporter: David Allsopp
Priority: Minor
Download latest release from
http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.0.3/apache-cassandra-1.0.3-bin.tar.gz
Unpack the tarball.
Follow instructions in README.txt, concluding with:
{noformat}
dna@master:~/code/apache-cassandra-1.0.3$ bin/cassandra -f
Error opening zip file or JAR manifest missing : /lib/jamm-0.2.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument
{noformat}
Firstly, the version of jamm packaged with Cassandra 1.0.3 is jamm-0.2.5, not
jamm-0.2.1.
Both bin/cassandra.bat and conf/cassandra-env.sh reference jamm-0.2.5 so not
sure where jamm-0.2.1 is being referenced from - nothing obvious using grep.
Secondly, /lib/jamm-0.2.1.jar is the wrong path - should be set relative to
working directory, not filesystem root
(Incidentally, Cassandra v1.0.3 is still listed as unreleased on JIRA.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rene kochen reopened CASSANDRA-2786: Tested with 0.8.2 and 0.8.7, but still does not work. On 0.7.x it works fine. After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rene kochen updated CASSANDRA-2786: --- Affects Version/s: 0.8.7 After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0, 0.8.7 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154289#comment-13154289 ] rene kochen commented on CASSANDRA-2786: One note: I tested with several grace-periods. With a grace-period of one minute, it is easier to reproduce. On our production site (with grace-priod of 24 hours), the data resurrects after several days. After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0, 0.8.7 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3513) CQL Schema metadata does not contain Type information for the key unless the key is selected
CQL Schema metadata does not contain Type information for the key unless the
key is selected
Key: CASSANDRA-3513
URL: https://issues.apache.org/jira/browse/CASSANDRA-3513
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.2
Reporter: Kelley Reynolds
Priority: Minor
The CQL ResultSchema does not contain type information for the key unless the
key is actually part of the CQL query. It's available in the column family
schema but that's what we're trying to avoid by having a ResultSchema in the
first place.
CREATE COLUMNFAMILY test (id varchar PRIMARY KEY, serial int)
INSERT INTO test (id, serial) VALUES ('test', 12345)
SELECT serial FROM test where id='test'
# The Result Schema
--- !ruby/object:CassandraCQL::ResultSchema
names: {}
values:
serial: Int32Type
# The row with no way to properly cast the key (though I used varchar here to
make a functional example)
--- !ruby/object:CassandraCQL::Thrift::CqlRow
columns:
- !ruby/object:CassandraCQL::Thrift::Column
name: serial
timestamp: 1321893998457001
value: !binary |
AAAwOQ==
key: test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1034) Remove assumption that Key to Token is one-to-one
[
https://issues.apache.org/jira/browse/CASSANDRA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154293#comment-13154293
]
Sylvain Lebresne commented on CASSANDRA-1034:
-
bq. Put another way: in my ideal world, DK.token would be purely an
optimization to avoid calling partitioner.getToken(key) over and over.
I understand that, but I think there is two different things and I want to know
exactly where the disagreement/problem is.
The first problem, which is imho the core of this ticket, is that the code
needs to be able somehow to deal with things like (where I use k for keys and t
for tokens, and the term range for either Range or Bounds):
* Is key k in the range [k', t] (or (t', k''])? Because when you do a
range_slice of [k', k''] and there is multiple nodes and [k', k''] spans
multiple replica, we will end up requesting all keys in [k', t] (for some t) or
(t', k''].
* Is key k in range (t, t']? Because we're allowed to range query keys by a
token range, but also a few other reason, like the fact that during validation
compaction we hashes together keys within a token range.
Note that those are not trivial questions, because for instance [k', t], while
we intuitively understand what it represents is a weird beast in that is a
range a point and a segment?!
Or in other words, as much as I'd like the operations on Tokens and the ones on
Keys to be two completely orthogonal sets of operation with no interaction
whatsoever, it is not the case and we have to deal with it.
Dealing with the case where we need tokens and we have keys is trivial (we just
call Key.getToken() and boom, we're back in the case with only tokens).
The problem is when we fundamentally work on keys, but have only token to start
with. Today (i.e. before this ticket), we take a simplification by doing
essentially the same thing that in the 'needs token but got keys' case by
having a sort of Token.getKey() (it's more ugly in practice, we inline calls to
new DecoratedKey(t, null), but that's the same thing). But doing that forces in
itself the fact that key an token are in bijection and we want to lift that.
One solution could be to try to keep Token as long as we can, even in places
where we really need a key and have the code deal with that. I can understand
that on the surface that could look clean, but in practice the code to do that
correctly would a pure nightmare. Just trying to implement a Range that would
mix token and keys (like the [k', t] range above) is a complete mess.
So what this patch does is realizing that you could characterize the set of
keys that a token t represents with only two keys: the smallest key having
token t, and the biggest key having token t.
Now, supposing we agree on what is above, the rest is implementation details
and that's probably a much simpler discussion. Note that above I'm not talking
of DecoratedKey, only key. But the question is, how do you represent those two
new keys (for each token). The patch uses special values of the key field of DK
to deal with those. I can agree this is not the cleanest thing ever and I'm
fine looking for a different encoding, but I just don't have a much better
idea, and frankly I don't find that horrible either.
bq. I think I'd rather have these in the partitioner
Good idea.
bq. his feels odd when we go ahead and construct DKs with null key anyway in
the other constructor.
The goal here is to avoid constructing one of the two 'fake' keys by accident
For that the second constructor is dedicated to their construction and as the
commnet says, you're not even supposed to use this second constructor, but use
Token.{upper|lower}Bound instead. Actually, the assert should check for the
EMPTY_BYTE_BUFFER.
bq. Could we update the methods for whose benefits we're performing the Token
- DK conversion, to accept RingPosition instead?
Frankly, and as argumented above, no, not without *huge* pain. We only do that
conversion in places where we will have to do it at some point, and trying to
push Tokens deeper would only serve in having operations that make no real
sense for Tokens be able to actually deal with Token. As one example, we would
have to make Range with a mix of Token and Keys, and frankly that will be a
total mess to code.
bq. I don't see a good reason to not use a real hashcode implementation
(Objects.hashCode is useful here)
Not sure I follow but ByteBuffer.hashCode() does hash the content of the buffer
if that was what you meant.
bq. There's clearly a lot of places in this patch where it's causing special
case ugliness though, independent of its status as max.
Again, I would be open to better encoding. But is there really that much
places? The patch tried to make it so that no code outside of DecoratedKey
really have to deal with it. If not perfect, I actually think it's better
contained that before the
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154294#comment-13154294
]
Sylvain Lebresne commented on CASSANDRA-3510:
-
bq. Referring to the isLatestVersion check?
Yes.
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have the maxTimestamps 10, 100 and Long.MIN_VALUE
compaction will write a SSTable with maxTimestamp 100. However file c may
actually contain columns with a timestamp 100 which will be in the
[jira] [Commented] (CASSANDRA-3508) requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option)
[ https://issues.apache.org/jira/browse/CASSANDRA-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154313#comment-13154313 ] Jonathan Ellis commented on CASSANDRA-3508: --- I'd be okay with keeping --debug around for that. requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option) --- Key: CASSANDRA-3508 URL: https://issues.apache.org/jira/browse/CASSANDRA-3508 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.0.0 Reporter: Matthew F. Dennis Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.0.3 Attachments: CASSANDRA-3508.patch this manifests itself in cassandra-cli by returning null to the user. In order to see what the problem was (and in many cases, just to know there was a problem at all) requires running cassandra-cli with --debug -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3513) CQL Schema metadata does not contain Type information for the key unless the key is selected
[
https://issues.apache.org/jira/browse/CASSANDRA-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-3513.
---
Resolution: Won't Fix
If you want the key in the resultset you need to ask for it. It's more of a
bug than a feature that you always get the key back in the Thrift struct.
CQL Schema metadata does not contain Type information for the key unless the
key is selected
Key: CASSANDRA-3513
URL: https://issues.apache.org/jira/browse/CASSANDRA-3513
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.2
Reporter: Kelley Reynolds
Priority: Minor
Labels: cql
The CQL ResultSchema does not contain type information for the key unless the
key is actually part of the CQL query. It's available in the column family
schema but that's what we're trying to avoid by having a ResultSchema in the
first place.
CREATE COLUMNFAMILY test (id varchar PRIMARY KEY, serial int)
INSERT INTO test (id, serial) VALUES ('test', 12345)
SELECT serial FROM test where id='test'
# The Result Schema
--- !ruby/object:CassandraCQL::ResultSchema
names: {}
values:
serial: Int32Type
# The row with no way to properly cast the key (though I used varchar here to
make a functional example)
--- !ruby/object:CassandraCQL::Thrift::CqlRow
columns:
- !ruby/object:CassandraCQL::Thrift::Column
name: serial
timestamp: 1321893998457001
value: !binary |
AAAwOQ==
key: test
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3508) requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option)
[ https://issues.apache.org/jira/browse/CASSANDRA-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-3508: --- Attachment: CASSANDRA-3508-v2.patch this patch keeps the --debug option and shows exceptions in the places your mentioned only when that option is set. requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option) --- Key: CASSANDRA-3508 URL: https://issues.apache.org/jira/browse/CASSANDRA-3508 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.0.0 Reporter: Matthew F. Dennis Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.0.3 Attachments: CASSANDRA-3508-v2.patch, CASSANDRA-3508.patch this manifests itself in cassandra-cli by returning null to the user. In order to see what the problem was (and in many cases, just to know there was a problem at all) requires running cassandra-cli with --debug -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3508) requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option)
[ https://issues.apache.org/jira/browse/CASSANDRA-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154329#comment-13154329 ] Jonathan Ellis commented on CASSANDRA-3508: --- +1 requiring --debug to see stack traces for failures in cassandra-cli is a terrible idea (aka silent failure is never a valid option) --- Key: CASSANDRA-3508 URL: https://issues.apache.org/jira/browse/CASSANDRA-3508 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.0.0 Reporter: Matthew F. Dennis Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.0.3 Attachments: CASSANDRA-3508-v2.patch, CASSANDRA-3508.patch this manifests itself in cassandra-cli by returning null to the user. In order to see what the problem was (and in many cases, just to know there was a problem at all) requires running cassandra-cli with --debug -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1034) Remove assumption that Key to Token is one-to-one
[ https://issues.apache.org/jira/browse/CASSANDRA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154335#comment-13154335 ] Jonathan Ellis commented on CASSANDRA-1034: --- bq. in practice, we inline calls to new DecoratedKey(t, null) Right. I must be missing something crucial, because that's exactly what it looks like we're still doing in this patch, only with a special constructor. Remove assumption that Key to Token is one-to-one - Key: CASSANDRA-1034 URL: https://issues.apache.org/jira/browse/CASSANDRA-1034 Project: Cassandra Issue Type: Bug Reporter: Stu Hood Assignee: Sylvain Lebresne Priority: Minor Fix For: 1.1 Attachments: 0001-Generify-AbstractBounds.patch, 0002-Remove-assumption-that-token-and-keys-are-one-to-one.patch, 0003-unit-test.patch, 1034_v1.txt, CASSANDRA-1034.patch get_range_slices assumes that Tokens do not collide and converts a KeyRange to an AbstractBounds. For RandomPartitioner, this assumption isn't safe, and would lead to a very weird heisenberg. Converting AbstractBounds to use a DecoratedKey would solve this, because the byte[] key portion of the DecoratedKey can act as a tiebreaker. Alternatively, we could make DecoratedKey extend Token, and then use DecoratedKeys in places where collisions are unacceptable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1204637 - in /cassandra/branches/cassandra-1.0: CHANGES.txt src/java/org/apache/cassandra/cli/CliClient.java src/java/org/apache/cassandra/cli/CliMain.java src/java/org/apache/cassandra/c
Author: xedin
Date: Mon Nov 21 18:25:16 2011
New Revision: 1204637
URL: http://svn.apache.org/viewvc?rev=1204637view=rev
Log:
skip --debug requirement to see common exceptions in CLI
patch by Pavel Yaskevich; reviewed by Jonathan Ellis for CASSANDRA-3508
Modified:
cassandra/branches/cassandra-1.0/CHANGES.txt
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliClient.java
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliMain.java
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliOptions.java
Modified: cassandra/branches/cassandra-1.0/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/CHANGES.txt?rev=1204637r1=1204636r2=1204637view=diff
==
--- cassandra/branches/cassandra-1.0/CHANGES.txt (original)
+++ cassandra/branches/cassandra-1.0/CHANGES.txt Mon Nov 21 18:25:16 2011
@@ -5,7 +5,7 @@
* add bloom filter FP rates to nodetool cfstats (CASSANDRA-3347)
* record partitioner in sstable metadata component (CASSANDRA-3407)
* add new upgradesstables nodetool command (CASSANDRA-3406)
-
+ * skip --debug requirement to see common exceptions in CLI (CASSANDRA-3508)
1.0.3
* revert name-based query defragmentation aka CASSANDRA-2503 (CASSANDRA-3491)
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliClient.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliClient.java?rev=1204637r1=1204636r2=1204637view=diff
==
---
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliClient.java
(original)
+++
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliClient.java
Mon Nov 21 18:25:16 2011
@@ -1482,7 +1482,7 @@ public class CliClient
sessionState.err.println(e.getWhy());
if (sessionState.debug)
-e.printStackTrace();
+e.printStackTrace(sessionState.err);
}
}
@@ -1956,8 +1956,8 @@ public class CliClient
catch (TException e)
{
if (sessionState.debug)
-e.printStackTrace();
-
+e.printStackTrace(sessionState.err);
+
sessionState.err.println(Login failure. Did you specify
'keyspace', 'username' and 'password'?);
}
}
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliMain.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliMain.java?rev=1204637r1=1204636r2=1204637view=diff
==
---
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliMain.java
(original)
+++
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliMain.java
Mon Nov 21 18:25:16 2011
@@ -81,8 +81,7 @@ public class CliMain
}
catch (Exception e)
{
-if (sessionState.debug)
-e.printStackTrace();
+e.printStackTrace(sessionState.err);
String error = (e.getCause() == null) ? e.getMessage() :
e.getCause().getMessage();
throw new RuntimeException(Exception connecting to + server +
/ + port + . Reason: + error + .);
@@ -161,8 +160,7 @@ public class CliMain
{
sessionState.err.println(Exception retrieving information about
the cassandra node, check you have connected to the thrift port.);
-if (sessionState.debug)
-e.printStackTrace();
+e.printStackTrace(sessionState.err);
return;
}
@@ -228,10 +226,8 @@ public class CliMain
String message = (e.getCause() == null) ? e.getMessage() :
e.getCause().getMessage();
sessionState.err.println(errorTemplate + message);
-if (sessionState.debug)
-{
+if (!(e instanceof RuntimeException))
e.printStackTrace(sessionState.err);
-}
if (sessionState.batch || sessionState.inFileMode())
{
Modified:
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliOptions.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliOptions.java?rev=1204637r1=1204636r2=1204637view=diff
==
---
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliOptions.java
(original)
+++
cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/cli/CliOptions.java
Mon Nov 21 18:25:16 2011
@@ -67,7 +67,7 @@ public class CliOptions
// options without argument
options.addOption(B, BATCH_OPTION, enabled batch mode (suppress
output;
svn commit: r1204649 - in /cassandra/trunk: ./ bin/ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/cli/ src/java/org/apache/cassandra/dht/ src/java/org/a
Author: xedin Date: Mon Nov 21 18:59:41 2011 New Revision: 1204649 URL: http://svn.apache.org/viewvc?rev=1204649view=rev Log: merge from 1.0 Modified: cassandra/trunk/ (props changed) cassandra/trunk/CHANGES.txt cassandra/trunk/bin/cassandra.bat cassandra/trunk/contrib/ (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/trunk/src/java/org/apache/cassandra/cli/CliClient.java cassandra/trunk/src/java/org/apache/cassandra/cli/CliMain.java cassandra/trunk/src/java/org/apache/cassandra/cli/CliOptions.java cassandra/trunk/src/java/org/apache/cassandra/dht/RandomPartitioner.java cassandra/trunk/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java cassandra/trunk/src/java/org/apache/cassandra/io/sstable/SSTableReader.java cassandra/trunk/src/java/org/apache/cassandra/service/StorageService.java cassandra/trunk/test/unit/org/apache/cassandra/io/sstable/SSTableMetadataSerializerTest.java cassandra/trunk/test/unit/org/apache/cassandra/utils/MerkleTreeTest.java Propchange: cassandra/trunk/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Mon Nov 21 18:59:41 2011 @@ -4,7 +4,7 @@ /cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1198724,1198726-1199259,1199284,1200226,1200471 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 -/cassandra/branches/cassandra-1.0:1167085-1202912,1203117 +/cassandra/branches/cassandra-1.0:1167085-1204646 /cassandra/branches/cassandra-1.0.0:1167104-1167229,1167232-1181093,1181741,1181816,1181820,1182951,1183243 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/tags/cassandra-0.8.0-rc1:1102511-1125020 Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1204649r1=1204648r2=1204649view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Mon Nov 21 18:59:41 2011 @@ -10,11 +10,13 @@ 1.0.4 + * enforce RP tokens 0..2**127 (CASSANDRA-3501) + * canonicalize paths exposed through JMX (CASSANDRA-3504) * fix liveSize stat when sstables are removed (CASSANDRA-3496) * add bloom filter FP rates to nodetool cfstats (CASSANDRA-3347) * record partitioner in sstable metadata component (CASSANDRA-3407) * add new upgradesstables nodetool command (CASSANDRA-3406) - + * skip --debug requirement to see common exceptions in CLI (CASSANDRA-3508) 1.0.3 * revert name-based query defragmentation aka CASSANDRA-2503 (CASSANDRA-3491) @@ -31,7 +33,8 @@ * (CQL) fix for counter decrement syntax (CASSANDRA-3418) * Fix race introduced by CASSANDRA-2503 (CASSANDRA-3482) * Fix incomplete deletion of delivered hints (CASSANDRA-3466) - * Avoid reschuling compactions when no compaction was executed (CASSANDRA-3484) + * Avoid rescheduling compactions when no compaction was executed + (CASSANDRA-3484) * fix handling of the chunk_length_kb compression options (CASSANDRA-3492) Merged from 0.8: * Make counter shard merging thread safe (CASSANDRA-3178) Modified: cassandra/trunk/bin/cassandra.bat URL: http://svn.apache.org/viewvc/cassandra/trunk/bin/cassandra.bat?rev=1204649r1=1204648r2=1204649view=diff == --- cassandra/trunk/bin/cassandra.bat (original) +++ cassandra/trunk/bin/cassandra.bat Mon Nov 21 18:59:41 2011 @@ -15,6 +15,7 @@ @REM limitations under the License. @echo off +if %OS% == Windows_NT setlocal set ARG=%1 set INSTALL=INSTALL Propchange: cassandra/trunk/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Mon Nov 21 18:59:41 2011 @@ -4,7 +4,7 @@ /cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1198724,1198726-1199259,1199284,1200226,1200471 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 -/cassandra/branches/cassandra-1.0/contrib:1167085-1202912,1203117 +/cassandra/branches/cassandra-1.0/contrib:1167085-1204646 /cassandra/branches/cassandra-1.0.0/contrib:1167104-1167229,1167232-1181093,1181741,1181816,1181820,1182951,1183243 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689
[jira] [Commented] (CASSANDRA-3045) Update ColumnFamilyOutputFormat to use new bulkload API
[ https://issues.apache.org/jira/browse/CASSANDRA-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154371#comment-13154371 ] Brandon Williams commented on CASSANDRA-3045: - Final version is forthcoming, just struggling to pass some unit tests. Update ColumnFamilyOutputFormat to use new bulkload API --- Key: CASSANDRA-3045 URL: https://issues.apache.org/jira/browse/CASSANDRA-3045 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Jonathan Ellis Assignee: Brandon Williams Priority: Minor Fix For: 1.1 Attachments: 0001-Remove-gossip-SS-requirement-from-BulkLoader.txt, 0002-Allow-DD-loading-without-yaml.txt, 0003-hadoop-output-support-for-bulk-loading.txt The bulk loading interface added in CASSANDRA-1278 is a great fit for Hadoop jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3514) CounterColumnFamily Compaction error (ArrayIndexOutOfBoundsException)
CounterColumnFamily Compaction error (ArrayIndexOutOfBoundsException) -- Key: CASSANDRA-3514 URL: https://issues.apache.org/jira/browse/CASSANDRA-3514 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.3 Reporter: Eric Falcao On a single node, I'm seeing the following error when trying to compact a CounterColumnFamily. This appears to have started with version 1.0.3. nodetool -h localhost compact TRProd MetricsAllTime Error occured during compaction java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:250) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1471) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1523) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.ArrayIndexOutOfBoundsException at org.apache.cassandra.utils.ByteBufferUtil.arrayCopy(ByteBufferUtil.java:292) at org.apache.cassandra.db.context.CounterContext$ContextState.copyTo(CounterContext.java:792) at org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterContext.java:709) at org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:260) at org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:306) at org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:271) at org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:86) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:102) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:133) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:102) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:87) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116) at
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154378#comment-13154378
]
Aaron Morton commented on CASSANDRA-3510:
-
Thanks, will test shortly.
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have the maxTimestamps 10, 100 and Long.MIN_VALUE
compaction will write a SSTable with maxTimestamp 100. However file c may
actually contain columns with a timestamp 100 which will be in the
compacted file.
h1. Reproduce
1.
[jira] [Commented] (CASSANDRA-3045) Update ColumnFamilyOutputFormat to use new bulkload API
[
https://issues.apache.org/jira/browse/CASSANDRA-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154387#comment-13154387
]
Brandon Williams commented on CASSANDRA-3045:
-
Specifically, on line 97 FileStreamTask has a dangling semicolon that causes
the 'if' to always fire. However, removing that still causes a secondary index
check to fail in StreamingTransferTest. Here is a debug log of a working test:
{noformat}
INFO [Thread-3] 2011-11-18 17:36:04,039 SecondaryIndexManager.java (line 115)
Submitting index build of 626972746864617465, for data in
SSTableReader(path='build/test/cassandra/data/Keyspace1/Indexed1-hb-4-Data.db')
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,041 Table.java (line 515)
Indexing row key1
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,042 CollationController.java
(line 76) collectTimeOrderedData
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,042 KeysIndex.java (line 100)
applying index row 3288498 in ColumnFamily(Indexed1.626972746864617465
[6b657931:false:0@1234,])
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,043 SlabAllocator.java (line
105) 1 regions now allocated in org.apache.cassandra.utils.SlabAllocator@1716fa0
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,043 Table.java (line 515)
Indexing row key3
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,043 CollationController.java
(line 76) collectTimeOrderedData
DEBUG [CompactionExecutor:1] 2011-11-18 17:36:04,044 KeysIndex.java (line 100)
applying index row 3288500 in ColumnFamily(Indexed1.626972746864617465
[6b657933:false:0@1234,])
DEBUG [Thread-3] 2011-11-18 17:36:04,044 ColumnFamilyStore.java (line 671)
flush position is ReplayPosition(segmentId=1321659363183, position=354)
INFO [Thread-3] 2011-11-18 17:36:04,045 ColumnFamilyStore.java (line 685)
Enqueuing flush of Memtable-Indexed1.626972746864617465@5734522(38/47
serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2011-11-18 17:36:04,045 Memtable.java (line 239) Writing
Memtable-Indexed1.626972746864617465@5734522(38/47 serialized/live bytes, 2 ops)
DEBUG [FlushWriter:1] 2011-11-18 17:36:04,047 DatabaseDescriptor.java (line
783) expected data files size is 84; largest free partition has 19441123328
bytes free
INFO [FlushWriter:1] 2011-11-18 17:36:04,062 Memtable.java (line 275)
Completed flushing
build/test/cassandra/data/Keyspace1/Indexed1.626972746864617465-hb-2-Data.db
(154 bytes)
DEBUG [FlushWriter:1] 2011-11-18 17:36:04,063 IntervalNode.java (line 45)
Creating IntervalNode from [Interval(DecoratedKey(3288498, 00322db2),
DecoratedKey(3288500, 00322db4))]
DEBUG [FlushWriter:1] 2011-11-18 17:36:04,063 DataTracker.java (line 331)
adding build/test/cassandra/data/Keyspace1/Indexed1.626972746864617465-hb-2 to
list of files tracked for Keyspace1.Indexed1.626972746864617465
DEBUG [COMMIT-LOG-WRITER] 2011-11-18 17:36:04,064 CommitLog.java (line 459)
discard completed log segments for ReplayPosition(segmentId=1321659363183,
position=354), column family 1047.
DEBUG [COMMIT-LOG-WRITER] 2011-11-18 17:36:04,065 CommitLog.java (line 498) Not
safe to delete commit log
CommitLogSegment(/srv/encrypted/project/cassandra/build/test/cassandra/commitlog/CommitLog-1321659363183.log);
dirty is ; hasNext: false
INFO [Thread-3] 2011-11-18 17:36:04,065 SecondaryIndexManager.java (line 134)
Index build of 626972746864617465, complete
INFO [Thread-3] 2011-11-18 17:36:04,066 StreamInSession.java (line 162)
Finished streaming session 778312411854932 from /127.0.0.1
DEBUG [MiscStage:1] 2011-11-18 17:36:04,066 StreamReplyVerbHandler.java (line
47) Received StreamReply StreamReply(sessionId=778312411854932, file='',
action=SESSION_FINISHED)
{noformat}
and here is a failing one (with 3045 applied):
{noformat}
INFO [Thread-3] 2011-11-18 17:20:02,669 SecondaryIndexManager.java (line 117)
Submitting index build of 626972746864617465, for data in
SSTableReader(path='build/test/cassandra/data/Keyspace1/Indexed1-h-4-Data.db')
DEBUG [Streaming:1] 2011-11-18 17:20:02,669 MmappedSegmentedFile.java (line
139) All segments have been unmapped successfully
DEBUG [NonPeriodicTasks:1] 2011-11-18 17:20:02,671 FileUtils.java (line 51)
Deleting Indexed1-h-2-Statistics.db
DEBUG [NonPeriodicTasks:1] 2011-11-18 17:20:02,671 FileUtils.java (line 51)
Deleting Indexed1-h-2-Filter.db
DEBUG [NonPeriodicTasks:1] 2011-11-18 17:20:02,671 FileUtils.java (line 51)
Deleting Indexed1-h-2-Index.db
DEBUG [NonPeriodicTasks:1] 2011-11-18 17:20:02,672 SSTable.java (line 143)
Deleted build/test/cassandra/data/Keyspace1/Indexed1-h-2
DEBUG [CompactionExecutor:1] 2011-11-18 17:20:02,674 Table.java (line 516)
Indexing row key1
DEBUG [CompactionExecutor:1] 2011-11-18 17:20:02,674 CollationController.java
(line 74) collectTimeOrderedData
DEBUG [CompactionExecutor:1] 2011-11-18 17:20:02,675 KeysIndex.java (line 100)
applying index row 3288498 in
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154397#comment-13154397 ] paul cannon commented on CASSANDRA-3507: bq. Should we just spin cqlsh out to a separate project? I expect that if cqlsh is in a different project, but cassandra-cli is distributed with c* proper, cassandra-cli will remain the de facto standard interactive interface, even if cqlsh is quite simple to install alongside. I've heard from several people that we want to transition to cqlsh over time, so that would make this an important issue. But if we don't mind sticking with cassandra-cli (I don't really know), then it makes sense to split cqlsh into a different project. Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Minor Labels: cql, cqlsh Fix For: 1.0.4 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154425#comment-13154425
]
Aaron Morton commented on CASSANDRA-3510:
-
Tested against the current 1.0 head, with the test case I had and the query
works as expected.
I agree doing it in compaction is safer, putting it in repair just got me there
faster last night.
Thanks for cleaning up the patch.
+1
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have the maxTimestamps
[jira] [Commented] (CASSANDRA-3510) Incorrect query results due to invalid SSTable.maxTimestamp
[
https://issues.apache.org/jira/browse/CASSANDRA-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154433#comment-13154433
]
Jonathan Ellis commented on CASSANDRA-3510:
---
+1
Incorrect query results due to invalid SSTable.maxTimestamp
---
Key: CASSANDRA-3510
URL: https://issues.apache.org/jira/browse/CASSANDRA-3510
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.3
Reporter: Aaron Morton
Priority: Critical
Attachments: 0001-3510-ignore-maxTimestamp-if-Long.MIN_VALUE.patch,
0002-3510-update-maxTimestamp-during-repair.patch, 3510.patch
related to CASSANDRA-3446
(sorry this is so long, took me a bit to work through it all and there is a
lot of new code :) )
h1. Summary
SSTable.maxTimestamp for files created before 1.0 defaults to Long.MIN_VALUE,
and this means the wrong data is returned from queries.
h2. Details
Noticed on a cluster that was upgraded from 0.8.X to 1.X, it then had trouble
similar to CASSANDRA-3446. It was rolled back to 0.8 and the migrated to
1.0.3.
4 Node cluster, all files upgraded to hb format.
In a super CF there are situations where a get for a sub columns returns a
different value than a get for the column. .e.g.
{noformat}
[default@XXX] get Users[ascii('username')]['meta']['password'];
= (column=password, value=3130323130343130, timestamp=1307352647576000)
[default@XX] get Users[ascii('username')]['meta'];
(snip)
= (column=password, value=3034323131303034, timestamp=1319563673493000)
{noformat}
The correct value is the second one.
I added logging after line 109 in
o.a.c.db.CollectionController.collectTimeOrderedData() to log the sstable
name and the file max timestamp, this is what I got:
{code:java}
for (SSTableReader sstable : view.sstables)
{
long currentMaxTs = sstable.getMaxTimestamp();
logger.debug(String.format(Got sstable %s and max TS %d, sstable,
currentMaxTs));
reduceNameFilter(reducedFilter, container, currentMaxTs);
{code}
{noformat}
DEBUG 14:08:46,012 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12348-Data.db') and
max TS 1321824847534000
DEBUG 14:08:47,231 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12346-Data.db') and
max TS 1321813380793000
DEBUG 14:08:49,879 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12330-Data.db') and
max TS -9223372036854775808
DEBUG 14:08:49,880 Got sstable
SSTableReader(path='/var/lib/cassandra/data/X/Users-hb-12325-Data.db') and
max TS -9223372036854775808
{noformat}
The key I was reading is present in files 12330 and 12325, the first contains
the *old / wrong* value with timestamp 1307352647576000 above. The second
contains the *new / correct* value with timestamp 1319563673493000.
**Updated:** Incorrect, it was a later file that had the correct value, see
the first comment.
When CollectionController.collectTimeOrderedData() processes the 12325 file
(after processing the 12330 file) while looping over the sstables the call to
reduceNameFilter() removes the column from the filter because the column
read from the 12330 file has a time stamp of 1307352647576000 and the 12325
file incorrectly has a max time stamp of -9223372036854775808 .
SSTableMetadata is reading the max time stamp from the stats file, but it is
Long.MIN_VALUE. I think this happens because scrub creates the SSTableWriter
using cfs.createCompactionWriter() which sets the maxTimestamp in the meta
data collector according to the maxTimestamp in the meta data for the file(s)
that will be scrubbed / compacted. But for pre 1.0 format files the default
in SSTableMetadata is Long.MIN_VALUE, (see SSTableMetaData.deserialize() and
the ctor). So scrubbing a pre 1.0 file will write stats files that have
maxTimestamp as Long.MIN_VALUE.
During scrubbing the SSTableWriter does not update the maxTimestamp because
append(AbstractCompactedRow) is called which expects the that
cfs.createCompactionWriter() was able to set the correct maxTimestamp on the
meta data. Compaction also uses append(AbstractCompactedRow) so may create an
SSTable with an incorrect maxTimestamp if one of the input files started life
as a pre 1.0 file and has a bad maxTimestamp.
It looks like the only time the maxTimestamp is calculated is when the
SSTable is originally written. So the error from the old files will be
carried along.
e.g. If the files a,b and c have the maxTimestamps 10, 100 and Long.MIN_VALUE
compaction will write a SSTable with maxTimestamp 100. However file c may
actually contain columns with a timestamp 100 which will be in the
compacted file.
h1. Reproduce
1. Start a clean 0.8.7
[jira] [Updated] (CASSANDRA-3514) CounterColumnFamily Compaction error (ArrayIndexOutOfBoundsException)
[ https://issues.apache.org/jira/browse/CASSANDRA-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3514: -- Component/s: Core Fix Version/s: 1.0.4 Assignee: Sylvain Lebresne Labels: compaction (was: ) CounterColumnFamily Compaction error (ArrayIndexOutOfBoundsException) -- Key: CASSANDRA-3514 URL: https://issues.apache.org/jira/browse/CASSANDRA-3514 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.3 Reporter: Eric Falcao Assignee: Sylvain Lebresne Labels: compaction Fix For: 1.0.4 On a single node, I'm seeing the following error when trying to compact a CounterColumnFamily. This appears to have started with version 1.0.3. nodetool -h localhost compact TRProd MetricsAllTime Error occured during compaction java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:250) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1471) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1523) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.ArrayIndexOutOfBoundsException at org.apache.cassandra.utils.ByteBufferUtil.arrayCopy(ByteBufferUtil.java:292) at org.apache.cassandra.db.context.CounterContext$ContextState.copyTo(CounterContext.java:792) at org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterContext.java:709) at org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:260) at org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:306) at org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:271) at org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:86) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:102) at
[jira] [Updated] (CASSANDRA-3045) Update ColumnFamilyOutputFormat to use new bulkload API
[ https://issues.apache.org/jira/browse/CASSANDRA-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3045: Attachment: (was: 0001-Remove-gossip-SS-requirement-from-BulkLoader.txt) Update ColumnFamilyOutputFormat to use new bulkload API --- Key: CASSANDRA-3045 URL: https://issues.apache.org/jira/browse/CASSANDRA-3045 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Jonathan Ellis Assignee: Brandon Williams Priority: Minor Fix For: 1.1 The bulk loading interface added in CASSANDRA-1278 is a great fit for Hadoop jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3045) Update ColumnFamilyOutputFormat to use new bulkload API
[ https://issues.apache.org/jira/browse/CASSANDRA-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3045: Attachment: (was: 0002-Allow-DD-loading-without-yaml.txt) Update ColumnFamilyOutputFormat to use new bulkload API --- Key: CASSANDRA-3045 URL: https://issues.apache.org/jira/browse/CASSANDRA-3045 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Jonathan Ellis Assignee: Brandon Williams Priority: Minor Fix For: 1.1 The bulk loading interface added in CASSANDRA-1278 is a great fit for Hadoop jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3045) Update ColumnFamilyOutputFormat to use new bulkload API
[ https://issues.apache.org/jira/browse/CASSANDRA-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3045: Attachment: (was: 0003-hadoop-output-support-for-bulk-loading.txt) Update ColumnFamilyOutputFormat to use new bulkload API --- Key: CASSANDRA-3045 URL: https://issues.apache.org/jira/browse/CASSANDRA-3045 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Jonathan Ellis Assignee: Brandon Williams Priority: Minor Fix For: 1.1 The bulk loading interface added in CASSANDRA-1278 is a great fit for Hadoop jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3045) Update ColumnFamilyOutputFormat to use new bulkload API
[ https://issues.apache.org/jira/browse/CASSANDRA-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3045: Attachment: 0003-hadoop-output-support-for-bulk-loading.txt 0002-Allow-DD-loading-without-yaml.txt 0001-Remove-gossip-SS-requirement-from-BulkLoader.txt Updated/rebased patches include the dangling semicolon fix. Update ColumnFamilyOutputFormat to use new bulkload API --- Key: CASSANDRA-3045 URL: https://issues.apache.org/jira/browse/CASSANDRA-3045 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Jonathan Ellis Assignee: Brandon Williams Priority: Minor Fix For: 1.1 Attachments: 0001-Remove-gossip-SS-requirement-from-BulkLoader.txt, 0002-Allow-DD-loading-without-yaml.txt, 0003-hadoop-output-support-for-bulk-loading.txt The bulk loading interface added in CASSANDRA-1278 is a great fit for Hadoop jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3512) Getting Started instructions don't work in README.txt - wrong version of jamm, wrong path
[
https://issues.apache.org/jira/browse/CASSANDRA-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reassigned CASSANDRA-3512:
-
Assignee: Eric Evans
Right. That's by design. Here's the relevant part of bin/cassandra:
{noformat}
# If an include wasn't specified in the environment, then search for one...
if [ x$CASSANDRA_INCLUDE = x ]; then
# Locations (in order) to use when searching for an include file.
for include in /usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh \
$HOME/.cassandra.in.sh \
`dirname $0`/cassandra.in.sh; do
if [ -r $include ]; then
. $include
break
fi
done
# ...otherwise, source the specified include.
elif [ -r $CASSANDRA_INCLUDE ]; then
. $CASSANDRA_INCLUDE
fi
{noformat}
However, my experience matches yours, that virtually everyone when running
bin/cassandra manually expects it to use the files in bin/ and conf/ rather
than global package-installed ones. (Assigning to Eric for comment since he
wrote this.)
Getting Started instructions don't work in README.txt - wrong version of
jamm, wrong path
-
Key: CASSANDRA-3512
URL: https://issues.apache.org/jira/browse/CASSANDRA-3512
Project: Cassandra
Issue Type: Bug
Components: Packaging
Affects Versions: 1.0.3
Environment: Ubuntu 11.04
Reporter: David Allsopp
Assignee: Eric Evans
Priority: Minor
Download latest release from
http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.0.3/apache-cassandra-1.0.3-bin.tar.gz
Unpack the tarball.
Follow instructions in README.txt, concluding with:
{noformat}
dna@master:~/code/apache-cassandra-1.0.3$ bin/cassandra -f
Error opening zip file or JAR manifest missing : /lib/jamm-0.2.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument
{noformat}
Firstly, the version of jamm packaged with Cassandra 1.0.3 is jamm-0.2.5, not
jamm-0.2.1.
Both bin/cassandra.bat and conf/cassandra-env.sh reference jamm-0.2.5 so not
sure where jamm-0.2.1 is being referenced from - nothing obvious using grep.
Secondly, /lib/jamm-0.2.1.jar is the wrong path - should be set relative to
working directory, not filesystem root
(Incidentally, Cassandra v1.0.3 is still listed as unreleased on JIRA.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new datacenter to existing cluster without repair
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154502#comment-13154502 ] Jonathan Ellis commented on CASSANDRA-3483: --- So, am I understanding correctly that we're talking about two different scenarios? - Add new DC without repair - Add many nodes to existing DC without RING_DELAY in between I think Sylvain's proposal addresses the first nicely. So what I need help with is understanding what problem you're trying to solve with the second part. Dealing with overlapping ranges in node movement basically requires a rewrite of that subsystem (CASSANDRA-2434). But I suspect there is a good enough solution that we could find if I understood better what your pain point is here. Support bringing up a new datacenter to existing cluster without repair --- Key: CASSANDRA-3483 URL: https://issues.apache.org/jira/browse/CASSANDRA-3483 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair. I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154526#comment-13154526 ] Brandon Williams commented on CASSANDRA-2786: - bq. With a grace-period of one minute, it is easier to reproduce. This seems to me to suggest you aren't running repair often enough and are encountering the same effect as CASSANDRA-1316. After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0, 0.8.7 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154534#comment-13154534 ] Jonathan Ellis commented on CASSANDRA-2786: --- bq. This seems to me to suggest you aren't running repair often enough If you can only reproduce on multiple nodes, that is probably the issue here. After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0, 0.8.7 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1034) Remove assumption that Key to Token is one-to-one
[
https://issues.apache.org/jira/browse/CASSANDRA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154533#comment-13154533
]
Sylvain Lebresne commented on CASSANDRA-1034:
-
No, no, the patch does use the same think. I merely said that the patch does
some attempt at a better encapsulation, as it seems better to use the
Token.{upper|lower}BoundKey to creates those fake keys that inlining the call
to the constructor all over the code (which we do now). It makes the use of
null more of an internal detail of DecoratedKey (not completely, granted, but
it's a little bit better). It also makes it simpler to check we don't
accidentally construct a DK with a null key by accident (the goal of the
assertion in the first DK constructor in the patch).
But let it be clear that I'm not making any claim that this patch cleans some
ugliness in the current code. It mainly try to solve the problem at hand, which
is basically to be able to do range_slices and getting the right result even
when multiple keys have the same token.
This is not saying it wouldn't be good to fix any current ugliness at the same
time if possible, but in truth, I don't find that using special DK to represent
special keys is such an ugly hack (not either claiming it's super beautiful, I
just don't have a particular hatred of this). Besides, I don't have tons of
ideas to fix the issue at end (the priority) and make the code clearly cleaner.
And I do think that whatever ugliness the current have, this patch doesn't make
it worst.
Anyway, I'll try to see if I can improve the encapsulation of the
Token.{upper|lower}BoundKey representation and see if I can come with something
slightly cleaner.
Remove assumption that Key to Token is one-to-one
-
Key: CASSANDRA-1034
URL: https://issues.apache.org/jira/browse/CASSANDRA-1034
Project: Cassandra
Issue Type: Bug
Reporter: Stu Hood
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 1.1
Attachments: 0001-Generify-AbstractBounds.patch,
0002-Remove-assumption-that-token-and-keys-are-one-to-one.patch,
0003-unit-test.patch, 1034_v1.txt, CASSANDRA-1034.patch
get_range_slices assumes that Tokens do not collide and converts a KeyRange
to an AbstractBounds. For RandomPartitioner, this assumption isn't safe, and
would lead to a very weird heisenberg.
Converting AbstractBounds to use a DecoratedKey would solve this, because the
byte[] key portion of the DecoratedKey can act as a tiebreaker.
Alternatively, we could make DecoratedKey extend Token, and then use
DecoratedKeys in places where collisions are unacceptable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3411) Pre-allocated, Recycled Commitlog Segment Files
[ https://issues.apache.org/jira/browse/CASSANDRA-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rick Branson updated CASSANDRA-3411: Attachment: 3411-v6-retry.txt Here's a fixed version of the v6 patch. Pre-allocated, Recycled Commitlog Segment Files --- Key: CASSANDRA-3411 URL: https://issues.apache.org/jira/browse/CASSANDRA-3411 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Rick Branson Priority: Minor Attachments: 001-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 003-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 004-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 006-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 3411-cleaned.txt, 3411-v5.txt, 3411-v6-retry.txt An approach for improving commitlog performance is to pre-allocate the full 128MB segment files and reuse them once all the mutations have been flushed. Pre-allocation allows writes to be performed without modifying the file size metadata, and should (in theory) allow the filesystem to allocate a contiguous block of space for the file. Recycling the segment files prevents the overhead of pre-allocation from impacting overall performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1034) Remove assumption that Key to Token is one-to-one
[
https://issues.apache.org/jira/browse/CASSANDRA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154553#comment-13154553
]
Jonathan Ellis commented on CASSANDRA-1034:
---
bq. the patch does some attempt at a better encapsulation, as it seems better
to use the Token.{upper|lower}BoundKey to creates those fake keys that inlining
the call to the constructor all over the code (which we do now).
Okay, I'll buy that. It's an awful lot of code churn for IMO a relatively
minor win, but I see where you're going with that.
Help me understand this patchset a different way: which is the part without
which CASSANDRA-1600 is impossible?
Remove assumption that Key to Token is one-to-one
-
Key: CASSANDRA-1034
URL: https://issues.apache.org/jira/browse/CASSANDRA-1034
Project: Cassandra
Issue Type: Bug
Reporter: Stu Hood
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 1.1
Attachments: 0001-Generify-AbstractBounds.patch,
0002-Remove-assumption-that-token-and-keys-are-one-to-one.patch,
0003-unit-test.patch, 1034_v1.txt, CASSANDRA-1034.patch
get_range_slices assumes that Tokens do not collide and converts a KeyRange
to an AbstractBounds. For RandomPartitioner, this assumption isn't safe, and
would lead to a very weird heisenberg.
Converting AbstractBounds to use a DecoratedKey would solve this, because the
byte[] key portion of the DecoratedKey can act as a tiebreaker.
Alternatively, we could make DecoratedKey extend Token, and then use
DecoratedKeys in places where collisions are unacceptable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3411) Pre-allocated, Recycled Commitlog Segment Files
[ https://issues.apache.org/jira/browse/CASSANDRA-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154578#comment-13154578 ] Jonathan Ellis commented on CASSANDRA-3411: --- One more concern. I'm not a huge fan of END_OF_SEGMENT_MARKER: - Zero is a very common value. With a recycled log, it doesn't seem impossible to me that we'd encounter a zero in a partially synced record prematurely - More generally, writing a value and then seeking back feels like a bug waiting to happen (if we forget the seek-back step) - Finally and most importantly, the marker approach generalizes poorly to CASSANDRA-622 (should we ever bite that bullet) The postgresql approach addresses these without much complexity: add an incrementing transaction id to each record, and if we encounter a lower transaction id than the previous on replay, then that's the end of segment. Pre-allocated, Recycled Commitlog Segment Files --- Key: CASSANDRA-3411 URL: https://issues.apache.org/jira/browse/CASSANDRA-3411 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Rick Branson Priority: Minor Attachments: 001-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 003-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 004-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 006-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 3411-cleaned.txt, 3411-v5.txt, 3411-v6-retry.txt An approach for improving commitlog performance is to pre-allocate the full 128MB segment files and reuse them once all the mutations have been flushed. Pre-allocation allows writes to be performed without modifying the file size metadata, and should (in theory) allow the filesystem to allocate a contiguous block of space for the file. Recycling the segment files prevents the overhead of pre-allocation from impacting overall performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3411) Pre-allocated, Recycled Commitlog Segment Files
[ https://issues.apache.org/jira/browse/CASSANDRA-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154580#comment-13154580 ] Jonathan Ellis commented on CASSANDRA-3411: --- Oops, hit submit prematurely. Still reviewing the rest. Pre-allocated, Recycled Commitlog Segment Files --- Key: CASSANDRA-3411 URL: https://issues.apache.org/jira/browse/CASSANDRA-3411 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Rick Branson Priority: Minor Attachments: 001-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 003-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 004-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 006-pre-allocated-recycled-commitlog-segment-files-CASSANDRA-3411.patch, 3411-cleaned.txt, 3411-v5.txt, 3411-v6-retry.txt An approach for improving commitlog performance is to pre-allocate the full 128MB segment files and reuse them once all the mutations have been flushed. Pre-allocation allows writes to be performed without modifying the file size metadata, and should (in theory) allow the filesystem to allocate a contiguous block of space for the file. Recycling the segment files prevents the overhead of pre-allocation from impacting overall performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of FAQ by BrandonWilliams
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The FAQ page has been changed by BrandonWilliams: http://wiki.apache.org/cassandra/FAQ?action=diffrev1=139rev2=140 * [[#schema_disagreement|What are schema disagreement errors and how do I fix them?]] * [[#dropped_messages|Why do I see ... messages dropped.. in the logs?]] * [[#cli_keys|Why does the 0.8 cli not assume keys are strings anymore?]] + * [[#memlock|Cassandra dies with java.lang.OutOfMemoryError: Map failed]] Anchor(cant_listen_on_ip_any) @@ -541, +542 @@ If all your keys are of the same type, you should add information to the CF metadata, e.g., update column family cf with key_validation_class = 'UTF8Type'. If you have heterogeneous keys, you can tell the cli what type to use on case-by-case basis, as in, assume cf keys as utf8. + Anchor(memlock) + == Cassandra dies with java.lang.OutOfMemoryError: Map failed == + '''IF''' Cassandra is dying specifically with the Map failed message it means the OS is denying java the ability to lock more memory. In linux, this typically means memlock is limited. Check /proc/pid of cassandra/limits to verify this and raise it (eg, via ulimit in bash.) Note that the debian and redhat packages handle this for you automatically. +
[jira] [Commented] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154740#comment-13154740 ] rene kochen commented on CASSANDRA-2786: With the attached program I'm able to reproduce it on a single node. After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0, 0.8.7 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1740) Nodetool commands to query and stop compaction, repair, cleanup and scrub
[ https://issues.apache.org/jira/browse/CASSANDRA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-1740: - Attachment: 0001-Patch-to-Stop-compactions-v3.patch Hi Jonathan, Moved instanceof to DTPE.logExceptionsAfterExecute Thanks! Nodetool commands to query and stop compaction, repair, cleanup and scrub - Key: CASSANDRA-1740 URL: https://issues.apache.org/jira/browse/CASSANDRA-1740 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Chip Salzenberg Assignee: Vijay Priority: Minor Labels: compaction Fix For: 1.0.4 Attachments: 0001-Patch-to-Stop-compactions-v2.patch, 0001-Patch-to-Stop-compactions-v3.patch, 0001-Patch-to-Stop-compactions.patch, CASSANDRA-1740.patch Original Estimate: 24h Remaining Estimate: 24h The only way to stop compaction, repair, cleanup, or scrub in progress is to stop and restart the entire Cassandra server. Please provide nodetool commands to query whether such things are running, and stop them if they are. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3189) redhat init script can report success (exit 0) for startup failures
[ https://issues.apache.org/jira/browse/CASSANDRA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shyamal Prasad updated CASSANDRA-3189: -- Attachment: 3189.patch Fixes redhat init script for multiple small issues, most notably support multiple 'start' calls and return valid status codes. redhat init script can report success (exit 0) for startup failures --- Key: CASSANDRA-3189 URL: https://issues.apache.org/jira/browse/CASSANDRA-3189 Project: Cassandra Issue Type: Bug Components: Packaging Affects Versions: 0.8.5 Reporter: Eric Evans Assignee: paul cannon Priority: Minor Labels: patch Fix For: 1.0.4 Attachments: 3189.patch The Redhat init script's start target merely returns the status code of the JVM. In a perfect world that would be adequate, but the JVM does not always return a non-zero status when it should. The most obvious fix is to test that the process is live before exiting. See the Debian init script for an example of this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[ https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154784#comment-13154784 ] Matthew F. Dennis commented on CASSANDRA-2786: -- Per previous phone conversation, I tested this against 0.8.7 twice. Both times the attached java test ran to completion against a single node and output Done. When was this last reproduced? After a minor compaction, deleted key-slices are visible again -- Key: CASSANDRA-2786 URL: https://issues.apache.org/jira/browse/CASSANDRA-2786 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0, 0.8.7 Environment: Reproduced on single Cassandra node (CentOS 5.5) Reproduced on single Cassandra node (Windows Server 2008) Reporter: rene kochen Assignee: Sylvain Lebresne Fix For: 0.8.1, 0.8.2 Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch, 2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip After a minor compaction, deleted key-slices are visible again. Steps to reproduce: 1) Insert a row named test. 2) Insert 50 rows. During this step, row test is included in a major compaction: file-1, file-2, file-3 and file-4 compacted to file-5 (includes test). 3) Delete row named test. 4) Insert 50 rows. During this step, row test is included in a minor compaction: file-6, file-7, file-8 and file-9 compacted to file-10 (should include tombstoned test). After step 4, row test is live again. Test environment: Single node with empty database. Standard configured super-column-family (I see this behavior with several gc_grace settings (big and small values): create column family Customers with column_type = 'Super' and comparator = 'BytesType; In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the row is still deleted. I've included a .NET program to reproduce the problem. I will add a Java version later on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2786) After a minor compaction, deleted key-slices are visible again
[
https://issues.apache.org/jira/browse/CASSANDRA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154786#comment-13154786
]
Matthew F. Dennis commented on CASSANDRA-2786:
--
I should mention that I did change one line in the test. I changed
{noformat}column.timestamp = getTimestamp();{noformat} to
{noformat}column.setTimestamp(getTimestamp());{noformat} because otherwise
thrift complained that the timestamp wasn't set.
After a minor compaction, deleted key-slices are visible again
--
Key: CASSANDRA-2786
URL: https://issues.apache.org/jira/browse/CASSANDRA-2786
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.0, 0.8.7
Environment: Reproduced on single Cassandra node (CentOS 5.5)
Reproduced on single Cassandra node (Windows Server 2008)
Reporter: rene kochen
Assignee: Sylvain Lebresne
Fix For: 0.8.1, 0.8.2
Attachments: 0001-Fix-wrong-purge-of-deleted-cf.patch,
2786_part2.patch, CassandraIssue.zip, CassandraIssueJava.zip
After a minor compaction, deleted key-slices are visible again.
Steps to reproduce:
1) Insert a row named test.
2) Insert 50 rows. During this step, row test is included in a major
compaction:
file-1, file-2, file-3 and file-4 compacted to file-5 (includes test).
3) Delete row named test.
4) Insert 50 rows. During this step, row test is included in a minor
compaction:
file-6, file-7, file-8 and file-9 compacted to file-10 (should include
tombstoned test).
After step 4, row test is live again.
Test environment:
Single node with empty database.
Standard configured super-column-family (I see this behavior with several
gc_grace settings (big and small values):
create column family Customers with column_type = 'Super' and comparator =
'BytesType;
In Cassandra 0.7.6 I observe the expected behavior, i.e. after step 4, the
row is still deleted.
I've included a .NET program to reproduce the problem. I will add a Java
version later on.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3514) CounterColumnFamily Compaction error (ArrayIndexOutOfBoundsException)
[ https://issues.apache.org/jira/browse/CASSANDRA-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154822#comment-13154822 ] Eric Falcao commented on CASSANDRA-3514: This also just started happening on another node (without user-triggered major compaction). Not sure what's causing this (old 0.8.* counter data perhaps?) CounterColumnFamily Compaction error (ArrayIndexOutOfBoundsException) -- Key: CASSANDRA-3514 URL: https://issues.apache.org/jira/browse/CASSANDRA-3514 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.3 Reporter: Eric Falcao Assignee: Sylvain Lebresne Labels: compaction Fix For: 1.0.4 On a single node, I'm seeing the following error when trying to compact a CounterColumnFamily. This appears to have started with version 1.0.3. nodetool -h localhost compact TRProd MetricsAllTime Error occured during compaction java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:250) at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1471) at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1523) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.ArrayIndexOutOfBoundsException at org.apache.cassandra.utils.ByteBufferUtil.arrayCopy(ByteBufferUtil.java:292) at org.apache.cassandra.db.context.CounterContext$ContextState.copyTo(CounterContext.java:792) at org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterContext.java:709) at org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:260) at org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:306) at org.apache.cassandra.db.CounterColumn.mergeAndRemoveOldShards(CounterColumn.java:271) at org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:86) at
[jira] [Created] (CASSANDRA-3515) add tombstones thats been filtered in cfhistograms
add tombstones thats been filtered in cfhistograms
--
Key: CASSANDRA-3515
URL: https://issues.apache.org/jira/browse/CASSANDRA-3515
Project: Cassandra
Issue Type: Improvement
Reporter: Jackson Chung
Priority: Minor
i think summary says it all, but see further:
{noformat}
driftx x: I guess we could add a histogram of how many tombstones we filtered
on a read
driftx x: @action: go for it in that context
action jackson: thx!
mdennis x: +1 on histogram of filtered tombstones
driftx x: definitely useful for diagnosing the queue-like case
driftx x: the histogram I mea
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new datacenter to existing cluster without repair
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154898#comment-13154898 ] Chris Goffinet commented on CASSANDRA-3483: --- Jonathan, You are correct. Sylvain's proposal does satisfy this ticket. It doesn't solve the case of (2) where if you want to add lots of nodes in an existing DC, and you know the tokens they should be at, and want to join them all at once. Our use case is, we actually add 60-100 nodes in one big capacity add. We would like to avoid the 120 second per node time frame. It's not a deal breaker though. We actually realized though if we are adding that many nodes to our cluster, with a large cluster already, we need to rebalance heavily anyway. Peter is almost done with the 'rebuild' patch, I'm assigning him to this ticket. Our next big focus is improving the rebalancing of a cluster. We have a very large cluster and after adding 100 nodes every month or so, this becomes painful. Almost all of our nodes have over 600GB+ each. We have an application that will require us to be rebalancing at all times to reduce our hot spots. Support bringing up a new datacenter to existing cluster without repair --- Key: CASSANDRA-3483 URL: https://issues.apache.org/jira/browse/CASSANDRA-3483 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair. I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3483) Support bringing up a new datacenter to existing cluster without repair
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Goffinet reassigned CASSANDRA-3483: - Assignee: Peter Schuller Support bringing up a new datacenter to existing cluster without repair --- Key: CASSANDRA-3483 URL: https://issues.apache.org/jira/browse/CASSANDRA-3483 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet Assignee: Peter Schuller Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair. I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3517) Make auto bootstrap smarter
Make auto bootstrap smarter --- Key: CASSANDRA-3517 URL: https://issues.apache.org/jira/browse/CASSANDRA-3517 Project: Cassandra Issue Type: Bug Reporter: Chris Goffinet Priority: Minor Just wanted to create this before I forget. One of the things we noticed when we were looking over the auto bootstrap code: When the bootstrapping node tries to determine its token, it looks at the ring for the node with the most 'load'. It then sends a message to that node to split its range. The function on that node will sample the keys to figure out where it should split the range. This can actually be very bad. If you have a data model where you have very few keys, and very wide rows the token it picks could be very bad. We should make this more auto bootstrap more intelligent than just looking at the node with the most disk space. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3518) Back pressure users by request/s instead of concurrent reads/writes
Back pressure users by request/s instead of concurrent reads/writes --- Key: CASSANDRA-3518 URL: https://issues.apache.org/jira/browse/CASSANDRA-3518 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet We are running into use cases where it makes a lot of sense to have QoS at the request level per user. Imagine this case: I have a cluster that can do 100,000 req/s. But I want to limit the user to only being able to do either 50,000 read or write/s per second in the cluster. I rather give back pressure to the user then make the cluster fall down because the user tried to take down my cluster. Also another case we have is where you have experimental features and want to give access to certain group of customers and let them run experiments on data. You *dont* want them taking down the cluster, you rather make them fail fast, or slow them down. If I could limit a user to N req/s for reads or writes, instead of adding back pressure based on # of concurrent requests in each stage, this would go a long way for us. We have had a few incidents where spinning up new features caused unexpected load and we couldn't stop them without turning the feature off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3518) Back pressure users by request/s instead of concurrent reads/writes
[ https://issues.apache.org/jira/browse/CASSANDRA-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154909#comment-13154909 ] Radim Kolar commented on CASSANDRA-3518: cassandra has request scheduler, look for _request_scheduler_ in config. You can even write yours. Back pressure users by request/s instead of concurrent reads/writes --- Key: CASSANDRA-3518 URL: https://issues.apache.org/jira/browse/CASSANDRA-3518 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet We are running into use cases where it makes a lot of sense to have QoS at the request level per user. Imagine this case: I have a cluster that can do 100,000 req/s. But I want to limit the user to only being able to do either 50,000 read or write/s per second in the cluster. I rather give back pressure to the user then make the cluster fall down because the user tried to take down my cluster. Also another case we have is where you have experimental features and want to give access to certain group of customers and let them run experiments on data. You *dont* want them taking down the cluster, you rather make them fail fast, or slow them down. If I could limit a user to N req/s for reads or writes, instead of adding back pressure based on # of concurrent requests in each stage, this would go a long way for us. We have had a few incidents where spinning up new features caused unexpected load and we couldn't stop them without turning the feature off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3516) Make bootstrapping smarter instead of using 120 seconds to stabilize the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-3516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154912#comment-13154912 ] Radim Kolar commented on CASSANDRA-3516: how many nodes you have? We need here about 40 seconds to stabilize ring. Make bootstrapping smarter instead of using 120 seconds to stabilize the ring - Key: CASSANDRA-3516 URL: https://issues.apache.org/jira/browse/CASSANDRA-3516 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet We run a very large cluster, and the 120s doesn't really make sense. We see gossip take anywhere from 30 to 60 seconds for the ring to actually mark nodes up or down (stabilize). We need to come up with a better way instead of picking an arbitrary number to wait. Our clusters are growing so large where 120s won't be enough time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3516) Make bootstrapping smarter instead of using 120 seconds to stabilize the ring
[ https://issues.apache.org/jira/browse/CASSANDRA-3516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154919#comment-13154919 ] Chris Goffinet commented on CASSANDRA-3516: --- 168 nodes to stabilize in 30-60 seconds. Make bootstrapping smarter instead of using 120 seconds to stabilize the ring - Key: CASSANDRA-3516 URL: https://issues.apache.org/jira/browse/CASSANDRA-3516 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet We run a very large cluster, and the 120s doesn't really make sense. We see gossip take anywhere from 30 to 60 seconds for the ring to actually mark nodes up or down (stabilize). We need to come up with a better way instead of picking an arbitrary number to wait. Our clusters are growing so large where 120s won't be enough time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3518) Back pressure users by request/s instead of concurrent reads/writes
[
https://issues.apache.org/jira/browse/CASSANDRA-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154920#comment-13154920
]
Chris Goffinet edited comment on CASSANDRA-3518 at 11/22/11 7:08 AM:
-
Radim,
I know what the Request Scheduler is.. please read my ticket more carefully.
{noformat}
# Scheduler Options vary based on the type of scheduler
# NoScheduler - Has no options
# RoundRobin
# - throttle_limit -- The throttle_limit is the number of in-flight
# requests per client. Requests beyond
# that limit are queued up until
# running requests can complete.
# The value of 80 here is twice the number of
# concurrent_reads + concurrent_writes.
{noformat}
number of in-flight requests per client. I want to a) do this per user b) break
it down so it's request/s not how many concurrent reads/writes in flight.
was (Author: lenn0x):
Radim,
I know what the Request Scheduler is.. please read my ticket more carefully.
# Scheduler Options vary based on the type of scheduler
# NoScheduler - Has no options
# RoundRobin
# - throttle_limit -- The throttle_limit is the number of in-flight
# requests per client. Requests beyond
# that limit are queued up until
# running requests can complete.
# The value of 80 here is twice the number of
# concurrent_reads + concurrent_writes.
number of in-flight requests per client. I want to a) do this per user b) break
it down so it's request/s not how many concurrent reads/writes in flight.
Back pressure users by request/s instead of concurrent reads/writes
---
Key: CASSANDRA-3518
URL: https://issues.apache.org/jira/browse/CASSANDRA-3518
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.2
Reporter: Chris Goffinet
We are running into use cases where it makes a lot of sense to have QoS at
the request level per user. Imagine this case:
I have a cluster that can do 100,000 req/s. But I want to limit the user to
only being able to do either 50,000 read or write/s per second in the
cluster. I rather give back pressure to the user then make the cluster fall
down because the user tried to take down my cluster.
Also another case we have is where you have experimental features and want to
give access to certain group of customers and let them run experiments on
data. You *dont* want them taking down the cluster, you rather make them fail
fast, or slow them down. If I could limit a user to N req/s for reads or
writes, instead of adding back pressure based on # of concurrent requests in
each stage, this would go a long way for us.
We have had a few incidents where spinning up new features caused unexpected
load and we couldn't stop them without turning the feature off.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3518) Back pressure users by request/s instead of concurrent reads/writes
[ https://issues.apache.org/jira/browse/CASSANDRA-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154920#comment-13154920 ] Chris Goffinet commented on CASSANDRA-3518: --- Radim, I know what the Request Scheduler is.. please read my ticket more carefully. # Scheduler Options vary based on the type of scheduler # NoScheduler - Has no options # RoundRobin # - throttle_limit -- The throttle_limit is the number of in-flight # requests per client. Requests beyond # that limit are queued up until # running requests can complete. # The value of 80 here is twice the number of # concurrent_reads + concurrent_writes. number of in-flight requests per client. I want to a) do this per user b) break it down so it's request/s not how many concurrent reads/writes in flight. Back pressure users by request/s instead of concurrent reads/writes --- Key: CASSANDRA-3518 URL: https://issues.apache.org/jira/browse/CASSANDRA-3518 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.2 Reporter: Chris Goffinet We are running into use cases where it makes a lot of sense to have QoS at the request level per user. Imagine this case: I have a cluster that can do 100,000 req/s. But I want to limit the user to only being able to do either 50,000 read or write/s per second in the cluster. I rather give back pressure to the user then make the cluster fall down because the user tried to take down my cluster. Also another case we have is where you have experimental features and want to give access to certain group of customers and let them run experiments on data. You *dont* want them taking down the cluster, you rather make them fail fast, or slow them down. If I could limit a user to N req/s for reads or writes, instead of adding back pressure based on # of concurrent requests in each stage, this would go a long way for us. We have had a few incidents where spinning up new features caused unexpected load and we couldn't stop them without turning the feature off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3519) ConcurrentModificationException in FailureDetector
ConcurrentModificationException in FailureDetector
--
Key: CASSANDRA-3519
URL: https://issues.apache.org/jira/browse/CASSANDRA-3519
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.1
Environment: Free BSD 8.2
/java -version
java version 1.6.0_07
Diablo Java(TM) SE Runtime Environment (build 1.6.0_07-b02)
Diablo Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode)
Reporter: Aaron Morton
Assignee: Aaron Morton
Priority: Minor
Noticed in a 2 DC cluster, error was on node in DC 2 streaming to a node in DC
1.
{code:java}
INFO [GossipTasks:1] 2011-11-20 18:36:05,153 Gossiper.java (line 759)
InetAddress /10.6.130.70 is now dead.
ERROR [GossipTasks:1] 2011-11-20 18:36:25,252 StreamOutSession.java (line 232)
StreamOutSession /10.6.130.70 failed because {} died or was restarted/removed
ERROR [AntiEntropySessions:21] 2011-11-20 18:36:25,252 AntiEntropyService.java
(line 688) [repair #7fb5b1b0-11f1-11e1--baed0a2090fe] session completed
with the following err
or
java.io.IOException: Endpoint /10.6.130.70 died
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:725)
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:762)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:192)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:559)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:62)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [GossipTasks:1] 2011-11-20 18:36:25,256 Gossiper.java (line 172) Gossip
error
java.util.ConcurrentModificationException
at
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:190)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:559)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:62)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [AntiEntropySessions:21] 2011-11-20 18:36:25,256
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[AntiEntropySessions:21,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Endpoint /10.6.130.70 died
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Endpoint /10.6.130.70 died
at
[jira] [Updated] (CASSANDRA-3519) ConcurrentModificationException in FailureDetector
[
https://issues.apache.org/jira/browse/CASSANDRA-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Morton updated CASSANDRA-3519:
Attachment: 3519.patch
Use a CopyOnWriteArrayList in the FailureDetector to track listeners, like the
Gossiper does.
ConcurrentModificationException in FailureDetector
--
Key: CASSANDRA-3519
URL: https://issues.apache.org/jira/browse/CASSANDRA-3519
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.1
Environment: Free BSD 8.2
/java -version
java version 1.6.0_07
Diablo Java(TM) SE Runtime Environment (build 1.6.0_07-b02)
Diablo Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode)
Reporter: Aaron Morton
Assignee: Aaron Morton
Priority: Minor
Attachments: 3519.patch
Noticed in a 2 DC cluster, error was on node in DC 2 streaming to a node in
DC 1.
{code:java}
INFO [GossipTasks:1] 2011-11-20 18:36:05,153 Gossiper.java (line 759)
InetAddress /10.6.130.70 is now dead.
ERROR [GossipTasks:1] 2011-11-20 18:36:25,252 StreamOutSession.java (line
232) StreamOutSession /10.6.130.70 failed because {} died or was
restarted/removed
ERROR [AntiEntropySessions:21] 2011-11-20 18:36:25,252
AntiEntropyService.java (line 688) [repair
#7fb5b1b0-11f1-11e1--baed0a2090fe] session completed with the following
err
or
java.io.IOException: Endpoint /10.6.130.70 died
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:725)
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:762)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:192)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:559)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:62)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [GossipTasks:1] 2011-11-20 18:36:25,256 Gossiper.java (line 172) Gossip
error
java.util.ConcurrentModificationException
at
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:190)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:559)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:62)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [AntiEntropySessions:21] 2011-11-20 18:36:25,256
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[AntiEntropySessions:21,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Endpoint /10.6.130.70 died
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
