[jira] [Updated] (CASSANDRA-3885) Support multiple ranges in SliceQueryFilter
[ https://issues.apache.org/jira/browse/CASSANDRA-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Alves updated CASSANDRA-3885: --- Attachment: CASSANDRA-3885.patch much simple and readable patch (imo). - adds a column slice object (gets rid of the nasty unchecked cast warnings) - gets rid of the isColumnNeeded method and diminishes the number of bb comparisons Support multiple ranges in SliceQueryFilter --- Key: CASSANDRA-3885 URL: https://issues.apache.org/jira/browse/CASSANDRA-3885 Project: Cassandra Issue Type: Sub-task Components: Core Reporter: Jonathan Ellis Assignee: David Alves Fix For: 1.2 Attachments: CASSANDRA-3885.patch, CASSANDRA-3885.patch, CASSANDRA-3885.patch, CASSANDRA-3885.patch This is logically a subtask of CASSANDRA-2710, but Jira doesn't allow sub-sub-tasks. We need to support multiple ranges in a SliceQueryFilter, and we want querying them to be efficient, i.e., one pass through the row to get all of the ranges, rather than one pass per range. Supercolumns are irrelevant since the goal is to replace them anyway. Ignore supercolumn-related code or rip it out, whichever is easier. This is ONLY dealing with the storage engine part, not the StorageProxy and Command intra-node messages or the Thrift or CQL client APIs. Thus, a unit test should be added to ColumnFamilyStoreTest to demonstrate that it works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3885) Support multiple ranges in SliceQueryFilter
[ https://issues.apache.org/jira/browse/CASSANDRA-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Alves updated CASSANDRA-3885: --- Attachment: CASSANDRA-3885.patch corrected a bug where in reverse the index was being looked up based on the start of the slice and not on the finish. Support multiple ranges in SliceQueryFilter --- Key: CASSANDRA-3885 URL: https://issues.apache.org/jira/browse/CASSANDRA-3885 Project: Cassandra Issue Type: Sub-task Components: Core Reporter: Jonathan Ellis Assignee: David Alves Fix For: 1.2 Attachments: CASSANDRA-3885.patch, CASSANDRA-3885.patch, CASSANDRA-3885.patch, CASSANDRA-3885.patch, CASSANDRA-3885.patch This is logically a subtask of CASSANDRA-2710, but Jira doesn't allow sub-sub-tasks. We need to support multiple ranges in a SliceQueryFilter, and we want querying them to be efficient, i.e., one pass through the row to get all of the ranges, rather than one pass per range. Supercolumns are irrelevant since the goal is to replace them anyway. Ignore supercolumn-related code or rip it out, whichever is easier. This is ONLY dealing with the storage engine part, not the StorageProxy and Command intra-node messages or the Thrift or CQL client APIs. Thus, a unit test should be added to ColumnFamilyStoreTest to demonstrate that it works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4233) overlapping sstables in leveled compaction strategy
[ https://issues.apache.org/jira/browse/CASSANDRA-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290893#comment-13290893 ] Yanis Biziuk commented on CASSANDRA-4233: - I already have this bug in 1.1.1 version java.lang.AssertionError: Last written key DecoratedKey(144093708553026671072703572185472027831, 003eef36) = current key DecoratedKey(50602742485951681279829043048319220646, 004506bf) writing into /cassandra/data/XX/YY/AAA-BBB-tmp-hd-5217-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153) ... any help? overlapping sstables in leveled compaction strategy --- Key: CASSANDRA-4233 URL: https://issues.apache.org/jira/browse/CASSANDRA-4233 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Reporter: Jonathan Ellis Assignee: Sylvain Lebresne Fix For: 1.1.1 Attachments: 0001-Make-sure-Leveled-compaction-always-use-the-right-min-.txt, 4233-assert.txt, 4233-v2.txt, 4233-v3.txt, 4233-v4.txt, system.log.bz2 CASSANDRA-4142 introduces test failures, that are caused by overlapping tables within a level, which Shouldn't Happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4233) overlapping sstables in leveled compaction strategy
[ https://issues.apache.org/jira/browse/CASSANDRA-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290895#comment-13290895 ] Sylvain Lebresne commented on CASSANDRA-4233: - I may or may not be related to this issue. It would help to get the full stack trace and a bit more information on when that happened, if that was a fresh 1.1.1 cluster or an upgraded one from 1.1.0 and the compaction strategy used. And it's probably worth creating a new ticket. overlapping sstables in leveled compaction strategy --- Key: CASSANDRA-4233 URL: https://issues.apache.org/jira/browse/CASSANDRA-4233 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Reporter: Jonathan Ellis Assignee: Sylvain Lebresne Fix For: 1.1.1 Attachments: 0001-Make-sure-Leveled-compaction-always-use-the-right-min-.txt, 4233-assert.txt, 4233-v2.txt, 4233-v3.txt, 4233-v4.txt, system.log.bz2 CASSANDRA-4142 introduces test failures, that are caused by overlapping tables within a level, which Shouldn't Happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4233) overlapping sstables in leveled compaction strategy
[
https://issues.apache.org/jira/browse/CASSANDRA-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290906#comment-13290906
]
Yanis Biziuk commented on CASSANDRA-4233:
-
Stack trace of this error
ERROR 04:09:21,216 Exception in thread Thread[CompactionExecutor:410,1,main]
java.lang.AssertionError: Last written key
DecoratedKey(144093708553026671072703572185472027831, 003eef36) =
current key DecoratedKey(50602742485951681279829043048319220646,
004506bf) writing into
/cassandra/data/Server/Messages/Server-Messages-tmp-hd-5380-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
at
org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Linux version 2.6.32-5-amd64 (Debian 2.6.32-45)
32GB ram, 24 threads (12 cores)
We are upgraded Cassandra server from repository(config file was overwritten,
but we change it later).
And we REcreated keyspace with all CF (we in not a production stage yet).
all us CFs
UPDATE COLUMN FAMILY Messages WITH
compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:1};
UPDATE COLUMN FAMILY Messages WITH
compaction_strategy=LeveledCompactionStrategy AND
compaction_strategy_options={sstable_size_in_mb: 10};
UPDATE COLUMN FAMILY Messages WITH gc_grace = 0;
This CF has many inserts in one time.
Over 1000-3000 keys with 9 columns each in one sec.
And this CF has 4 secondary indexes (numbers ~ from 1 to 20)
More info about Cassandra server
partitioner: org.apache.cassandra.dht.RandomPartitioner
key_cache_size_in_mb - default
row_cache_size_in_mb - default
row_cache_provider: SerializingCacheProvider
MAX_HEAP_SIZE=8G
HEAP_NEWSIZE=2G
Thanks for any help.
overlapping sstables in leveled compaction strategy
---
Key: CASSANDRA-4233
URL: https://issues.apache.org/jira/browse/CASSANDRA-4233
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Jonathan Ellis
Assignee: Sylvain Lebresne
Fix For: 1.1.1
Attachments:
0001-Make-sure-Leveled-compaction-always-use-the-right-min-.txt,
4233-assert.txt, 4233-v2.txt, 4233-v3.txt, 4233-v4.txt, system.log.bz2
CASSANDRA-4142 introduces test failures, that are caused by overlapping
tables within a level, which Shouldn't Happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4316) Compaction Throttle too bursty with large rows
Wayne Lewis created CASSANDRA-4316: -- Summary: Compaction Throttle too bursty with large rows Key: CASSANDRA-4316 URL: https://issues.apache.org/jira/browse/CASSANDRA-4316 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0, 1.0.1 Reporter: Wayne Lewis In org.apache.cassandra.db.compaction.CompactionIterable the check for compaction throttling occurs once every 1000 rows. In our workload this is much too large as we have many large rows (16 - 100 MB). With a 100 MB row, about 100 GB is read (and possibly written) before the compaction throttle sleeps. This causes bursts of essentially unthrottled compaction IO followed by a long sleep which yields inconsistence performance and high error rates during the bursts. We applied a workaround to check throttle every row which solved our performance and error issues: line 116 in org.apache.cassandra.db.compaction.CompactionIterable: if ((row++ % 1000) == 0) replaced with if ((row++ % 1) == 0) I think the better solution is to calculate how often throttle should be checked based on the throttle rate to apply sleeps more consistently. E.g. if 16MB/sec is the limit then check for sleep after every 16MB is read so sleeps are spaced out about every second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4316) Compaction Throttle too bursty with large rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wayne Lewis updated CASSANDRA-4316: --- Affects Version/s: (was: 1.0.1) 1.0.9 Compaction Throttle too bursty with large rows -- Key: CASSANDRA-4316 URL: https://issues.apache.org/jira/browse/CASSANDRA-4316 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.9, 1.1.0 Reporter: Wayne Lewis In org.apache.cassandra.db.compaction.CompactionIterable the check for compaction throttling occurs once every 1000 rows. In our workload this is much too large as we have many large rows (16 - 100 MB). With a 100 MB row, about 100 GB is read (and possibly written) before the compaction throttle sleeps. This causes bursts of essentially unthrottled compaction IO followed by a long sleep which yields inconsistence performance and high error rates during the bursts. We applied a workaround to check throttle every row which solved our performance and error issues: line 116 in org.apache.cassandra.db.compaction.CompactionIterable: if ((row++ % 1000) == 0) replaced with if ((row++ % 1) == 0) I think the better solution is to calculate how often throttle should be checked based on the throttle rate to apply sleeps more consistently. E.g. if 16MB/sec is the limit then check for sleep after every 16MB is read so sleeps are spaced out about every second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2356) make the debian package never start by default
[
https://issues.apache.org/jira/browse/CASSANDRA-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290933#comment-13290933
]
Eric Evans commented on CASSANDRA-2356:
---
{quote}
1) node A has token X
2) node A dies and the OS crashes
3) node A is replaced by node B, also at token X (say by restoring a full
snapshot)
4) node A is rebooted (say as part of RMA process)
5) cluster accepts node A as the rightful owner of token X, because it has a
later generation number by virtue of having been more recently started
6) you have a two nodes which contain the same but desychronized dataset, at
the same token, and no straightforward way to unify them
{quote}
The best way to deal with a situation like this is to make Cassandra do the
Right Thing. Barring that, you could hack the init script to check for this
condition and make startup a noop (or error), which would work even if someone
had overridden the start policy.
For what it's worth, I remember running across code recently that I think would
safeguard against this; Is this a failure scenario that you've tested against
recent versions?
bq. Are there any circumstances where auto-starting, esp. on packaged install
or upgrade, is actually desirable?
This is considered configuration, and with any configuration all you can do is
provide a reasonable default, something that will work for most of the people,
most of the time. Starting a service by default is generally considered The
Way[*] on Debian systems, (and hence the least surprising choice for our Debian
package).
[*] The oft stated reason being that if the user didn't want the service
running, they wouldn't have installed it.
make the debian package never start by default
--
Key: CASSANDRA-2356
URL: https://issues.apache.org/jira/browse/CASSANDRA-2356
Project: Cassandra
Issue Type: Improvement
Components: Packaging
Reporter: Jeremy Hanna
Priority: Minor
Labels: debian, packaging
Attachments: 2356.txt
Currently the debian package that installs cassandra starts cassandra by
default. It sounds like that is a standard debian packaging convention.
However, if you want to bootstrap a new node and want to configure it before
it creates any sort of state information, it's a pain. I would think that
the common use case would be to have it install all of the init scripts and
such but *not* have it start up by default. That way an admin can configure
cassandra with seed, token, host, etc. information and then start it. That
makes it easier to programmatically do this as well - have chef/puppet
install cassandra, do some configuration, then do the service start.
With the current setup, it sounds like cassandra creates state on startup
that has to be cleaned before a new configuration can take effect. So the
process of installing turns into:
* install debian package
* shutdown cassandra
* clean out state (data/log dirs)
* configure cassandra
* start cassandra
That seems suboptimal for the default case, especially when trying to
automate new nodes being bootstrapped.
Another case might be when a downed node comes back up and starts by default
and tries to claim a token that has already been claimed by another newly
bootstrapped node. Rob is more familiar with that case so I'll let him
explain it in the comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4150) Allow larger cache capacities than 2GB
[ https://issues.apache.org/jira/browse/CASSANDRA-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290967#comment-13290967 ] Chris Burroughs commented on CASSANDRA-4150: I'm having trouble following what happened from this ticket and 4315 (just open a new ticket for updating to CLHM 1.3 in 1.2... ). Isn't 02672936f635c93e84ed6625bb994e1628da5a9b in the 1.1 branch and we already upgraded to CLHM 1.3? https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=02672936f635c93e84ed6625bb994e1628da5a9b Allow larger cache capacities than 2GB -- Key: CASSANDRA-4150 URL: https://issues.apache.org/jira/browse/CASSANDRA-4150 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0 Reporter: Vijay Assignee: Vijay Fix For: 1.1.1 Attachments: 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4150.patch, 0002-Use-EntryWeigher-for-HeapCache.patch, 0002-add-bytes-written-metric-v2.patch, concurrentlinkedhashmap-lru-1.3.jar The problem is that capacity is a Integer which can maximum hold 2 GB, I will post a fix to CLHM in the mean time we might want to remove the maximumWeightedCapacity code path (atleast for Serializing cache) and implement it in our code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4228) Exception while reading from cassandra via ColumnFamilyInputFormat and OrderPreservingPartitioner
[
https://issues.apache.org/jira/browse/CASSANDRA-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290965#comment-13290965
]
bert Passek commented on CASSANDRA-4228:
Well, i just checked the cluster configuration and i had a misconfiguration
with one of our test clusters. We fixed this and it works like a charm :)
Exception while reading from cassandra via ColumnFamilyInputFormat and
OrderPreservingPartitioner
-
Key: CASSANDRA-4228
URL: https://issues.apache.org/jira/browse/CASSANDRA-4228
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Affects Versions: 1.1.0
Environment: Debian Squeeze
Reporter: bert Passek
Attachments: CassandraTest.java
We recently updated cassandra from verison 1.0.8 to 1.1.0 on a debian squeeze
system. After that we can not use ColumnFamilyInputFormat anymore due to
exceptions in cassandra. A simple unit test is provided via attachement.
Here are some details about our simple setup:
Ring:
Address DC RackStatus State LoadOwns
Token
127.0.0.1 datacenter1 rack1 Up Normal 859.36 KB
100,00% 55894951196891831822413178196787984716
Schema Definition:
create column family TestSuper
with column_type = 'Super'
and comparator = 'BytesType'
and subcomparator = 'BytesType'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
While running the test we face following exception on client side:
12/05/09 10:18:22 INFO junit.TestRunner:
testColumnFamilyInputFormat(de.unister.cpc.tests.CassandraTest):
org.apache.thrift.transport.TTransportException
12/05/09 10:18:22 INFO junit.TestRunner: java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:391)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:397)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:323)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:188)
at
de.unister.cpc.tests.CassandraTest.testColumnFamilyInputFormat(CassandraTest.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
[jira] [Commented] (CASSANDRA-4311) clean up messagingservice protocol limitations
[ https://issues.apache.org/jira/browse/CASSANDRA-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13290969#comment-13290969 ] Marcus Eriksson commented on CASSANDRA-4311: looks good to me, just a bit confusing calling it a 'header' when it is essentially a handshake packet could we negotiate SSL the way we do compression? nothing 'secret' is shared during the handshake phase anyway, after that we could upgrade sockets clean up messagingservice protocol limitations -- Key: CASSANDRA-4311 URL: https://issues.apache.org/jira/browse/CASSANDRA-4311 Project: Cassandra Issue Type: Bug Reporter: Jonathan Ellis Assignee: Jonathan Ellis Labels: jmx Fix For: 1.2 Attachments: 4311-skeleton.txt Weaknesses of the existing protocol: - information asymmetry: node A can know what version node B expects, but not vice versa (see CASSANDRA-4101) - delayed information: node A will often not know what version node B expects, until after first contacting node B -- forcing it to throw that first message away and retry for the next one - protocol cannot handle both cross-dc forwarding and broadcast_address != socket address (see bottom of CASSANDRA-4099) - version is partly global, partly per-connection, and partly per-message, resulting in some interesting hacks (CASSANDRA-3166) and difficulty layering more sophisticated OutputStreams on the socket (CASSANDRA-3127, CASSANDRA-4139) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4261) [patch] Support consistency-latency prediction in nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291008#comment-13291008
]
Jonathan Ellis commented on CASSANDRA-4261:
---
stress on ec2 would be a reasonable smoke test.
would definitely prefer something tunable via JMX, even if not exposed to
nodetool; cassandra.yaml changes require a node restart to take effect.
[patch] Support consistency-latency prediction in nodetool
--
Key: CASSANDRA-4261
URL: https://issues.apache.org/jira/browse/CASSANDRA-4261
Project: Cassandra
Issue Type: New Feature
Components: Tools
Affects Versions: 1.2
Reporter: Peter Bailis
Attachments: demo-pbs-v2.sh, pbs-nodetool-v2.patch
h3. Introduction
Cassandra supports a variety of replication configurations: ReplicationFactor
is set per-ColumnFamily and ConsistencyLevel is set per-request. Setting
{{ConsistencyLevel}} to {{QUORUM}} for reads and writes ensures strong
consistency, but {{QUORUM}} is often slower than {{ONE}}, {{TWO}}, or
{{THREE}}. What should users choose?
This patch provides a latency-consistency analysis within {{nodetool}}. Users
can accurately predict Cassandra's behavior in their production environments
without interfering with performance.
What's the probability that we'll read a write t seconds after it completes?
What about reading one of the last k writes? This patch provides answers via
{{nodetool predictconsistency}}:
{{nodetool predictconsistency ReplicationFactor TimeAfterWrite Versions}}
\\ \\
{code:title=Example output|borderStyle=solid}
//N == ReplicationFactor
//R == read ConsistencyLevel
//W == write ConsistencyLevel
user@test:$ nodetool predictconsistency 3 100 1
Performing consistency prediction
100ms after a given write, with maximum version staleness of k=1
N=3, R=1, W=1
Probability of consistent reads: 0.678900
Average read latency: 5.377900ms (99.900th %ile 40ms)
Average write latency: 36.971298ms (99.900th %ile 294ms)
N=3, R=1, W=2
Probability of consistent reads: 0.791600
Average read latency: 5.372500ms (99.900th %ile 39ms)
Average write latency: 303.630890ms (99.900th %ile 357ms)
N=3, R=1, W=3
Probability of consistent reads: 1.00
Average read latency: 5.426600ms (99.900th %ile 42ms)
Average write latency: 1382.650879ms (99.900th %ile 629ms)
N=3, R=2, W=1
Probability of consistent reads: 0.915800
Average read latency: 11.091000ms (99.900th %ile 348ms)
Average write latency: 42.663101ms (99.900th %ile 284ms)
N=3, R=2, W=2
Probability of consistent reads: 1.00
Average read latency: 10.606800ms (99.900th %ile 263ms)
Average write latency: 310.117615ms (99.900th %ile 335ms)
N=3, R=3, W=1
Probability of consistent reads: 1.00
Average read latency: 52.657501ms (99.900th %ile 565ms)
Average write latency: 39.949799ms (99.900th %ile 237ms)
{code}
h3. Demo
Here's an example scenario you can run using
[ccm|https://github.com/pcmanus/ccm]. The prediction is fast:
{code:borderStyle=solid}
cd cassandra-source-dir with patch applied
ant
# turn on consistency logging
sed -i .bak 's/log_latencies_for_consistency_prediction:
false/log_latencies_for_consistency_prediction: true/' conf/cassandra.yaml
ccm create consistencytest --cassandra-dir=.
ccm populate -n 5
ccm start
# if start fails, you might need to initialize more loopback interfaces
# e.g., sudo ifconfig lo0 alias 127.0.0.2
# use stress to get some sample latency data
tools/bin/stress -d 127.0.0.1 -l 3 -n 1 -o insert
tools/bin/stress -d 127.0.0.1 -l 3 -n 1 -o read
bin/nodetool -h 127.0.0.1 -p 7100 predictconsistency 3 100 1
{code}
h3. What and Why
We've implemented [Probabilistically Bounded
Staleness|http://pbs.cs.berkeley.edu/#demo], a new technique for predicting
consistency-latency trade-offs within Cassandra. Our
[paper|http://arxiv.org/pdf/1204.6082.pdf] will appear in [VLDB
2012|http://www.vldb2012.org/], and, in it, we've used PBS to profile a range
of Dynamo-style data store deployments at places like LinkedIn and Yammer in
addition to profiling our own Cassandra deployments. In our experience,
prediction is both accurate and much more lightweight than profiling and
manually testing each possible replication configuration (especially in
production!).
This analysis is important for the many users we've talked to and heard about
who use partial quorum operation (e.g., non-{{QUORUM}}
{{ConsistencyLevel}}). Should they use CL={{ONE}}? CL={{TWO}}? It likely
depends on their runtime environment and, short of profiling in production,
there's no existing way to answer these questions. (Keep in mind, Cassandra
defaults to CL={{ONE}}, meaning users don't know how stale their data will
be.)
We outline
[jira] [Resolved] (CASSANDRA-4228) Exception while reading from cassandra via ColumnFamilyInputFormat and OrderPreservingPartitioner
[
https://issues.apache.org/jira/browse/CASSANDRA-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-4228.
---
Resolution: Not A Problem
Thanks for the followup, bert.
Exception while reading from cassandra via ColumnFamilyInputFormat and
OrderPreservingPartitioner
-
Key: CASSANDRA-4228
URL: https://issues.apache.org/jira/browse/CASSANDRA-4228
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Affects Versions: 1.1.0
Environment: Debian Squeeze
Reporter: bert Passek
Attachments: CassandraTest.java
We recently updated cassandra from verison 1.0.8 to 1.1.0 on a debian squeeze
system. After that we can not use ColumnFamilyInputFormat anymore due to
exceptions in cassandra. A simple unit test is provided via attachement.
Here are some details about our simple setup:
Ring:
Address DC RackStatus State LoadOwns
Token
127.0.0.1 datacenter1 rack1 Up Normal 859.36 KB
100,00% 55894951196891831822413178196787984716
Schema Definition:
create column family TestSuper
with column_type = 'Super'
and comparator = 'BytesType'
and subcomparator = 'BytesType'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
While running the test we face following exception on client side:
12/05/09 10:18:22 INFO junit.TestRunner:
testColumnFamilyInputFormat(de.unister.cpc.tests.CassandraTest):
org.apache.thrift.transport.TTransportException
12/05/09 10:18:22 INFO junit.TestRunner: java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:391)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:397)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:323)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:188)
at
de.unister.cpc.tests.CassandraTest.testColumnFamilyInputFormat(CassandraTest.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at
[jira] [Commented] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291009#comment-13291009
]
Jonathan Ellis commented on CASSANDRA-4314:
---
bq. I have done no deletes
Then either you're using TTLs -- expired columns are basically the same as
deletes, in this respect -- or you're doing a lot of indexed column overwrites,
which also generate deletes in the index CF.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage
[jira] [Commented] (CASSANDRA-3707) Add KeyspaceChange CqlResultType
[ https://issues.apache.org/jira/browse/CASSANDRA-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291015#comment-13291015 ] Rick Shaw commented on CASSANDRA-3707: -- Returning in a special type that only gets set when the script calls USE really only tells you what the DEFAULT KS is. Not what KS was used in the request (other than USE of course). That is because you can say SELECT x,y from KS.CF where key=10; now; where KS can override the currently set default from USE. Add KeyspaceChange CqlResultType Key: CASSANDRA-3707 URL: https://issues.apache.org/jira/browse/CASSANDRA-3707 Project: Cassandra Issue Type: Sub-task Components: API Reporter: Jonathan Ellis Assignee: satish babu krishnamoorthy Labels: cql Fix For: 1.1.2 High level clients want to handle failover and load balancing transparently to the application, which means not just connection pooling but moving an existing connection to another server if necessary. When this happens, the client needs to know what the active keyspace was before failover, so it can set it to the same one in the new connection. Currently some clients handle this by checking for SET KEYSPACE queries, which violates the design principle that clients shouldn't have to parse CQL. Adding a new CqlResultType (that is set in response to a SET KEYSPACE command) would make this unnecessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291077#comment-13291077
]
Wade Poziombka commented on CASSANDRA-4314:
---
I am doing the latter. However, I have run repair which I would expect to
clear up the tombstones no? I still observe the same problem after repair.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage 0 0 0 0
0
[jira] [Updated] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wade Poziombka updated CASSANDRA-4314:
--
Attachment: 2012-06-07.zip
Here are log files taken showing running repair then running the query in
question demonstrating the out of memory condition.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: 2012-06-07.zip, oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage 0 0 0 0
0
InternalResponseStage 0 0
[jira] [Commented] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291086#comment-13291086
]
Wade Poziombka commented on CASSANDRA-4314:
---
I guess repair may not be the ticket... it should be done during compaction I
guess. But compaction should run automatically. I have heeded the note on
Tuning Compaction and not run a major compaction:
Also, once you run a major compaction, automatic minor compactions are no
longer triggered frequently forcing you to manually run major compactions on a
routine basis. So while read performance will be good immediately following a
major compaction, it will continually degrade until the next major compaction
is manually invoked. For this reason, major compaction is NOT recommended by
DataStax.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: 2012-06-07.zip, oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
[jira] [Commented] (CASSANDRA-4150) Allow larger cache capacities than 2GB
[ https://issues.apache.org/jira/browse/CASSANDRA-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291102#comment-13291102 ] Vijay commented on CASSANDRA-4150: -- Yes we did upgrade it to CLHM 1.3 in both Cassandra 1.1 and Cassandra 1.2 to fix the original issue mentioned in the ticket. Ben (author of CLHM) also added EntryWeightK, V to CLHM 1.3 on our request. Which is 0002 part of the patch originally submitted. To use EntryWeights we needed to calculate the memory size's (We where trying out multiple options using MemoryMeter or using other mechanisms) of the Key and Value, we where trying that in this ticket. Hence CASSANDRA-4315 was opened to continue the work of updating CLHM WeigherV to CLHM EntryWeightK, V. Allow larger cache capacities than 2GB -- Key: CASSANDRA-4150 URL: https://issues.apache.org/jira/browse/CASSANDRA-4150 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0 Reporter: Vijay Assignee: Vijay Fix For: 1.1.1 Attachments: 0001-CASSANDRA-4139-v2.patch, 0001-CASSANDRA-4150.patch, 0002-Use-EntryWeigher-for-HeapCache.patch, 0002-add-bytes-written-metric-v2.patch, concurrentlinkedhashmap-lru-1.3.jar The problem is that capacity is a Integer which can maximum hold 2 GB, I will post a fix to CLHM in the mean time we might want to remove the maximumWeightedCapacity code path (atleast for Serializing cache) and implement it in our code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4317) AssertionError in handleStateNormal in a mixed cluster
Brandon Williams created CASSANDRA-4317:
---
Summary: AssertionError in handleStateNormal in a mixed cluster
Key: CASSANDRA-4317
URL: https://issues.apache.org/jira/browse/CASSANDRA-4317
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2
Reporter: Brandon Williams
Fix For: 1.2
In a 3 node cluster with one seed on trunk, a member on trunk, and another
member on a previous version, the following occurs only on the non-seed trunk
member:
{noformat}
ERROR 16:44:18,708 Exception in thread Thread[GossipStage:1,5,main]
java.lang.AssertionError
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1072)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:995)
at
org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1568)
at
org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:819)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:897)
at
org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:43)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:57)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This doesn't repro if a non-trunk member is the seed, however upgrading the
seed first should still be valid.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291125#comment-13291125
]
Brandon Williams commented on CASSANDRA-4314:
-
Repair doesn't remove tombstones, and compactions only remove them if they are
older than gc_grace_seconds, so you're always holding however many occurred in
that time period.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: 2012-06-07.zip, oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage 0 0 0
[jira] [Commented] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291126#comment-13291126
]
Jonathan Ellis commented on CASSANDRA-4314:
---
Hmm. Looks like we don't override index gcgs to 0. I can't think of any
reason to keep tombstones around on a purely local table.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: 2012-06-07.zip, oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage 0 0 0 0
0
[jira] [Updated] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wade Poziombka updated CASSANDRA-4314:
--
Attachment: 2012-06-07-compact.zip
I have run nodetool compact on each node then reran the query. Still run out
of memory. Attached is logs.
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: 2012-06-07-compact.zip, 2012-06-07.zip, oom.zip,
yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage 0 0 0 0
0
InternalResponseStage
[jira] [Commented] (CASSANDRA-4311) clean up messagingservice protocol limitations
[
https://issues.apache.org/jira/browse/CASSANDRA-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291160#comment-13291160
]
Brandon Williams commented on CASSANDRA-4311:
-
MS.setVersion needs a minor fix to prevent NPE:
{noformat}
public Integer setVersion(InetAddress address, int version)
{
logger.debug(Setting version {} for {}, version, address);
return versions.put(address, version);
Integer v = versions.put(address, version);
return v == null ? version : v;
}
{noformat}
+1 otherwise.
clean up messagingservice protocol limitations
--
Key: CASSANDRA-4311
URL: https://issues.apache.org/jira/browse/CASSANDRA-4311
Project: Cassandra
Issue Type: Bug
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Labels: jmx
Fix For: 1.2
Attachments: 4311-skeleton.txt
Weaknesses of the existing protocol:
- information asymmetry: node A can know what version node B expects, but not
vice versa (see CASSANDRA-4101)
- delayed information: node A will often not know what version node B
expects, until after first contacting node B -- forcing it to throw that
first message away and retry for the next one
- protocol cannot handle both cross-dc forwarding and broadcast_address !=
socket address (see bottom of CASSANDRA-4099)
- version is partly global, partly per-connection, and partly per-message,
resulting in some interesting hacks (CASSANDRA-3166) and difficulty layering
more sophisticated OutputStreams on the socket (CASSANDRA-3127,
CASSANDRA-4139)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4311) clean up messagingservice protocol limitations
[
https://issues.apache.org/jira/browse/CASSANDRA-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291160#comment-13291160
]
Brandon Williams edited comment on CASSANDRA-4311 at 6/7/12 5:50 PM:
-
MS.setVersion needs a minor fix to prevent NPE:
{noformat}
public Integer setVersion(InetAddress address, int version)
{
logger.debug(Setting version {} for {}, version, address);
Integer v = versions.put(address, version);
return v == null ? version : v;
}
{noformat}
+1 otherwise.
was (Author: brandon.williams):
MS.setVersion needs a minor fix to prevent NPE:
{noformat}
public Integer setVersion(InetAddress address, int version)
{
logger.debug(Setting version {} for {}, version, address);
return versions.put(address, version);
Integer v = versions.put(address, version);
return v == null ? version : v;
}
{noformat}
+1 otherwise.
clean up messagingservice protocol limitations
--
Key: CASSANDRA-4311
URL: https://issues.apache.org/jira/browse/CASSANDRA-4311
Project: Cassandra
Issue Type: Bug
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Labels: jmx
Fix For: 1.2
Attachments: 4311-skeleton.txt
Weaknesses of the existing protocol:
- information asymmetry: node A can know what version node B expects, but not
vice versa (see CASSANDRA-4101)
- delayed information: node A will often not know what version node B
expects, until after first contacting node B -- forcing it to throw that
first message away and retry for the next one
- protocol cannot handle both cross-dc forwarding and broadcast_address !=
socket address (see bottom of CASSANDRA-4099)
- version is partly global, partly per-connection, and partly per-message,
resulting in some interesting hacks (CASSANDRA-3166) and difficulty layering
more sophisticated OutputStreams on the socket (CASSANDRA-3127,
CASSANDRA-4139)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4299) Update read paths to ensure ColumnFamilyStore.removeDeleted is called
[
https://issues.apache.org/jira/browse/CASSANDRA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated CASSANDRA-4299:
-
Attachment: trunk-4299.txt
Update read paths to ensure ColumnFamilyStore.removeDeleted is called
-
Key: CASSANDRA-4299
URL: https://issues.apache.org/jira/browse/CASSANDRA-4299
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2
Reporter: Kirk True
Assignee: Kirk True
Fix For: 1.2
Attachments: trunk-4299.txt
In some cases, {{RowIteratorFactory}}'s {{getReduced}} calls the
filter.collateColumns path and doesn't remove columns TTL-ed out (by
CASSANDRA-3974).
Sylvain suggested that we should put {{removeDeleted}} on every read path,
making it the only method concerned with tombstone and CF-TTL checks. He also
suggested that {{QueryFilter.isRelelevant}} might then be able to be removed.
See comments in CASSANDRA-3974 for more information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2356) make the debian package never start by default
[ https://issues.apache.org/jira/browse/CASSANDRA-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291209#comment-13291209 ] paul cannon commented on CASSANDRA-2356: Robert, are you proposing not having the Cassandra service automatically start, even on boot? That seems like it would run counter to the interests of most users. make the debian package never start by default -- Key: CASSANDRA-2356 URL: https://issues.apache.org/jira/browse/CASSANDRA-2356 Project: Cassandra Issue Type: Improvement Components: Packaging Reporter: Jeremy Hanna Priority: Minor Labels: debian, packaging Attachments: 2356.txt Currently the debian package that installs cassandra starts cassandra by default. It sounds like that is a standard debian packaging convention. However, if you want to bootstrap a new node and want to configure it before it creates any sort of state information, it's a pain. I would think that the common use case would be to have it install all of the init scripts and such but *not* have it start up by default. That way an admin can configure cassandra with seed, token, host, etc. information and then start it. That makes it easier to programmatically do this as well - have chef/puppet install cassandra, do some configuration, then do the service start. With the current setup, it sounds like cassandra creates state on startup that has to be cleaned before a new configuration can take effect. So the process of installing turns into: * install debian package * shutdown cassandra * clean out state (data/log dirs) * configure cassandra * start cassandra That seems suboptimal for the default case, especially when trying to automate new nodes being bootstrapped. Another case might be when a downed node comes back up and starts by default and tries to claim a token that has already been claimed by another newly bootstrapped node. Rob is more familiar with that case so I'll let him explain it in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4308) Promote the use of IFilter for internal commands
[
https://issues.apache.org/jira/browse/CASSANDRA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291256#comment-13291256
]
Yuki Morishita commented on CASSANDRA-4308:
---
LGTM.
nit: You can now replace QueryFilter.getFilter(SlicePredicate, AbstractType)
which is only used in deprecated IndexScanCommand and ColumnFamilyStoreTest,
with newly added ThriftValidation.asIFilter.
Promote the use of IFilter for internal commands
-
Key: CASSANDRA-4308
URL: https://issues.apache.org/jira/browse/CASSANDRA-4308
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 1.2
Attachments: 4308.txt
All read commands (the IReadCommand) work on slice and names filters, but
none of them uses the SliceQueryFilter and NamesQueryFilter classes
(RangeSliceCommand uses SlicePrediate from thrift and {SliceFrom,
SliceByNames}ReadReadCommand interns the arguments).
The main problem of that is that it follows that those command don't share
the serialization code for the column filters. Which isn't good for code
reuse, but also makes it a pain to add new fields to the filter
SliceQueryFilter (which CASSANDRA-3885 will need, but probably CASSANDRA-3647
too).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of SecondaryIndexes by JimAncona
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The SecondaryIndexes page has been changed by JimAncona: http://wiki.apache.org/cassandra/SecondaryIndexes?action=diffrev1=2rev2=3 Comment: Add question about Consistency Level and cluster availability * Q: Why is it necessary to always have at least one EQ comparison on secondary indices? A: Inequalities on secondary indices are always done in memory, so without at least one EQ on another secondary index you will be loading every row in the database, which with a massive database isn't a good idea. So by requiring at least one EQ on an index, you hopefully limit the set of rows that need to be read into memory to a manageable size. Although obviously you can still get into trouble with that as well. + * Q: How does choice of Consistency Level affect cluster availability when using secondary indexes? +A: Because secondary indexes are distributed, you must have CL level nodes available for all token ranges in the cluster in order to complete a query. For example, with RF = 3, when two out of three consecutive nodes in the ring are unavailable, all secondary index queries at CL = QUORUM will fail, however secondary index queries at CL = ONE will succeed. This is true regardless of cluster size. +
[Cassandra Wiki] Trivial Update of SecondaryIndexes by JimAncona
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The SecondaryIndexes page has been changed by JimAncona: http://wiki.apache.org/cassandra/SecondaryIndexes?action=diffrev1=3rev2=4 A: Inequalities on secondary indices are always done in memory, so without at least one EQ on another secondary index you will be loading every row in the database, which with a massive database isn't a good idea. So by requiring at least one EQ on an index, you hopefully limit the set of rows that need to be read into memory to a manageable size. Although obviously you can still get into trouble with that as well. * Q: How does choice of Consistency Level affect cluster availability when using secondary indexes? -A: Because secondary indexes are distributed, you must have CL level nodes available for all token ranges in the cluster in order to complete a query. For example, with RF = 3, when two out of three consecutive nodes in the ring are unavailable, all secondary index queries at CL = QUORUM will fail, however secondary index queries at CL = ONE will succeed. This is true regardless of cluster size. +A: Because secondary indexes are distributed, you must have CL level nodes available for '''all''' token ranges in the cluster in order to complete a query. For example, with RF = 3, when two out of three consecutive nodes in the ring are unavailable, '''all''' secondary index queries at CL = QUORUM will fail, however secondary index queries at CL = ONE will succeed. This is true regardless of cluster size.
[jira] [Created] (CASSANDRA-4318) Nodetool compactionstats fails with NullPointerException
David B created CASSANDRA-4318:
--
Summary: Nodetool compactionstats fails with NullPointerException
Key: CASSANDRA-4318
URL: https://issues.apache.org/jira/browse/CASSANDRA-4318
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.1
Environment: 2 node cluster running on Ubuntu 10.10 64-bit
16GB Max Heap allocated to each node.
Test keyspace using replication factor 2
Reporter: David B
Test uses Column family C defined as follows:
create column family C with caching = 'keys_only' and key_validation_class =
'LongType' and compression_options = { sstable_compression: SnappyCompressor,
chunk_length_kb: 64 } and max_compaction_threshold=0;
max_compaction_threshold is set to 0 to disable auto compaction.
SSTables are streamed via sstableloader, after which a major compaction is
triggered using nodetool compact MyKeyspace C.
Thereafter, attempts to request compaction stats via nodetool compactionstats
fail with the following exception:
Exception in thread main java.lang.NullPointerException
at
org.apache.cassandra.db.compaction.CompactionInfo.asMap(CompactionInfo.java:103)
at
org.apache.cassandra.db.compaction.CompactionManager.getCompactions(CompactionManager.java:1115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4319) ex msg for cql3 order by constraints says primary filter can be an IN clause which is misleading
Dave Brosius created CASSANDRA-4319:
---
Summary: ex msg for cql3 order by constraints says primary filter
can be an IN clause which is misleading
Key: CASSANDRA-4319
URL: https://issues.apache.org/jira/browse/CASSANDRA-4319
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2
Reporter: Dave Brosius
Priority: Trivial
Fix For: 1.2
Attachments: better_order_restriction_error_msg.txt
select * from visitor_audit where visitor_id in ('1_172892701', '1_316755431')
order by audit_id desc limit 6;
gives error
Bad Request: Ordering is only supported is the first part of the PRIMARY KEY is
restricted by an Equal or a IN
which seems to be what the query is doing.
But it only supports Equal, fix error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4319) ex msg for cql3 order by constraints says primary filter can be an IN clause which is misleading
[
https://issues.apache.org/jira/browse/CASSANDRA-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Brosius updated CASSANDRA-4319:
Attachment: better_order_restriction_error_msg.txt
ex msg for cql3 order by constraints says primary filter can be an IN clause
which is misleading
-
Key: CASSANDRA-4319
URL: https://issues.apache.org/jira/browse/CASSANDRA-4319
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2
Reporter: Dave Brosius
Priority: Trivial
Fix For: 1.2
Attachments: better_order_restriction_error_msg.txt
select * from visitor_audit where visitor_id in ('1_172892701',
'1_316755431') order by audit_id desc limit 6;
gives error
Bad Request: Ordering is only supported is the first part of the PRIMARY KEY
is restricted by an Equal or a IN
which seems to be what the query is doing.
But it only supports Equal, fix error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4319) ex msg for cql3 order by constraints says primary filter can be an IN clause which is misleading
[
https://issues.apache.org/jira/browse/CASSANDRA-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Brosius updated CASSANDRA-4319:
Assignee: Eric Evans
ex msg for cql3 order by constraints says primary filter can be an IN clause
which is misleading
-
Key: CASSANDRA-4319
URL: https://issues.apache.org/jira/browse/CASSANDRA-4319
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2
Reporter: Dave Brosius
Assignee: Eric Evans
Priority: Trivial
Fix For: 1.2
Attachments: better_order_restriction_error_msg.txt
select * from visitor_audit where visitor_id in ('1_172892701',
'1_316755431') order by audit_id desc limit 6;
gives error
Bad Request: Ordering is only supported is the first part of the PRIMARY KEY
is restricted by an Equal or a IN
which seems to be what the query is doing.
But it only supports Equal, fix error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4319) ex msg for cql3 order by constraints says primary filter can be an IN clause which is misleading
[
https://issues.apache.org/jira/browse/CASSANDRA-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Brosius reassigned CASSANDRA-4319:
---
Assignee: Dave Brosius (was: Eric Evans)
ex msg for cql3 order by constraints says primary filter can be an IN clause
which is misleading
-
Key: CASSANDRA-4319
URL: https://issues.apache.org/jira/browse/CASSANDRA-4319
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2
Reporter: Dave Brosius
Assignee: Dave Brosius
Priority: Trivial
Fix For: 1.2
Attachments: better_order_restriction_error_msg.txt
select * from visitor_audit where visitor_id in ('1_172892701',
'1_316755431') order by audit_id desc limit 6;
gives error
Bad Request: Ordering is only supported is the first part of the PRIMARY KEY
is restricted by an Equal or a IN
which seems to be what the query is doing.
But it only supports Equal, fix error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4319) ex msg for cql3 order by constraints says primary filter can be an IN clause which is misleading
[
https://issues.apache.org/jira/browse/CASSANDRA-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Brosius updated CASSANDRA-4319:
Reviewer: urandom
ex msg for cql3 order by constraints says primary filter can be an IN clause
which is misleading
-
Key: CASSANDRA-4319
URL: https://issues.apache.org/jira/browse/CASSANDRA-4319
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2
Reporter: Dave Brosius
Assignee: Dave Brosius
Priority: Trivial
Fix For: 1.2
Attachments: better_order_restriction_error_msg.txt
select * from visitor_audit where visitor_id in ('1_172892701',
'1_316755431') order by audit_id desc limit 6;
gives error
Bad Request: Ordering is only supported is the first part of the PRIMARY KEY
is restricted by an Equal or a IN
which seems to be what the query is doing.
But it only supports Equal, fix error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4305) CF serialization failure when working with custom secondary indices.
[
https://issues.apache.org/jira/browse/CASSANDRA-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291431#comment-13291431
]
Pavel Yaskevich commented on CASSANDRA-4305:
Also if the main argument against this being that we don't want to use by-value
CommitLogSegment.write(byte[]) than we can wrap row serialized mutation into
ByteBuffer in LogRecordAdder and pass it to write method by-reference, we have
to do serialization anyway so I see no reason to hold RM object around all the
time for CL as we really want only a serialized value there.
CF serialization failure when working with custom secondary indices.
Key: CASSANDRA-4305
URL: https://issues.apache.org/jira/browse/CASSANDRA-4305
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.10
Reporter: Pavel Yaskevich
Labels: datastax_qa
Attachments: CASSANDRA-4305.patch
Assertion (below) was triggered when client was adding new rows to
Solr-backed secondary indices (1000-row batch without any timeout).
{noformat}
ERROR [COMMIT-LOG-WRITER] 2012-05-30 16:39:02,896
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[COMMIT-LOG-WRITER,5,main]
java.lang.AssertionError: Final buffer length 176 to accomodate data size of
123 (predicted 87) for RowMutation(keyspace='solrTest1338395932411',
key='6b6579383039', modifications=[ColumnFamily(cf1
[long:false:8@1338395942384024,stringId:false:13@1338395940586003,])])
at
org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
at
org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
at
org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:600)
at
org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
{noformat}
After investigation it was clear that it was happening because we were
holding instances of RowMutation queued to the addition to CommitLog to the
actual write moment which is redundant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4320) Assertion error while delivering the hints.
Vijay created CASSANDRA-4320: Summary: Assertion error while delivering the hints. Key: CASSANDRA-4320 URL: https://issues.apache.org/jira/browse/CASSANDRA-4320 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0, 1.2 Reporter: Vijay Priority: Minor Fix For: 1.1.2, 1.2 java.lang.AssertionError at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:351) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:269) at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did some digging and looks like we just need to skip the deleted columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4320) Assertion error while delivering the hints.
[ https://issues.apache.org/jira/browse/CASSANDRA-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay reassigned CASSANDRA-4320: Assignee: Vijay Assertion error while delivering the hints. --- Key: CASSANDRA-4320 URL: https://issues.apache.org/jira/browse/CASSANDRA-4320 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0, 1.2 Reporter: Vijay Assignee: Vijay Priority: Minor Fix For: 1.1.2, 1.2 java.lang.AssertionError at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:351) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:269) at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did some digging and looks like we just need to skip the deleted columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4320) Assertion error while delivering the hints.
[ https://issues.apache.org/jira/browse/CASSANDRA-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-4320: - Attachment: 0001-CASSANDRA-4320.patch Assertion error while delivering the hints. --- Key: CASSANDRA-4320 URL: https://issues.apache.org/jira/browse/CASSANDRA-4320 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0, 1.2 Reporter: Vijay Assignee: Vijay Priority: Minor Fix For: 1.1.2, 1.2 Attachments: 0001-CASSANDRA-4320.patch java.lang.AssertionError at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:351) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:269) at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did some digging and looks like we just need to skip the deleted columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4321) stackoverflow building interval tree possible sstable corruptions
Anton Winter created CASSANDRA-4321: --- Summary: stackoverflow building interval tree possible sstable corruptions Key: CASSANDRA-4321 URL: https://issues.apache.org/jira/browse/CASSANDRA-4321 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Reporter: Anton Winter After upgrading to 1.1.1 (from 1.1.0) I have started experiencing StackOverflowError's resulting in compaction backlog and failure to restart. The ring currently consists of 6 DC's and 22 nodes using LCS compression. This issue was first noted on 2 nodes in one DC and then appears to have spread to various other nodes in the other DC's. When the first occurrence of this was found I restarted the instance but it failed to start so I cleared its data and treated it as a replacement node for the token it was previously responsible for. This node successfully streamed all the relevant data back but failed again a number of hours later with the same StackOverflowError and again was unable to restart. The initial stack overflow error on a running instance looks like this: ERROR [CompactionExecutor:314] 2012-06-07 09:59:43,017 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:314,1,main] java.lang.StackOverflowError at java.util.Arrays.mergeSort(Arrays.java:1157) at java.util.Arrays.sort(Arrays.java:1092) at java.util.Collections.sort(Collections.java:134) at org.apache.cassandra.utils.IntervalTree.IntervalNode.findMinMedianMax(IntervalNode.java:114) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:49) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) [snip - this repeats until stack overflow. Compactions stop from this point onwards] I restarted this failing instance with DEBUG logging enabled and it throws the following exception part way through startup: ERROR 11:37:51,046 Exception in thread Thread[OptionalTasks:1,5,main] java.lang.StackOverflowError at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:307) at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:276) at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:230) at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:124) at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:228) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:45) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) [snip - this repeats until stack overflow] at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalTree.init(IntervalTree.java:39) at org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:560) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:617) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:320) at org.apache.cassandra.db.DataTracker.addInitialSSTables(DataTracker.java:259) at org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:234) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:331) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:309) at org.apache.cassandra.db.Table.initCf(Table.java:367) at org.apache.cassandra.db.Table.init(Table.java:299) at
[jira] [Commented] (CASSANDRA-4320) Assertion error while delivering the hints.
[ https://issues.apache.org/jira/browse/CASSANDRA-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291507#comment-13291507 ] Jonathan Ellis commented on CASSANDRA-4320: --- So... I guess the deletion during one page, doesn't get purged by removeDeleted in the next since it's happening in the same ms? What if we just increased the gcBefore in the removeDeleted call to MAX_VALUE? Assertion error while delivering the hints. --- Key: CASSANDRA-4320 URL: https://issues.apache.org/jira/browse/CASSANDRA-4320 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0, 1.2 Reporter: Vijay Assignee: Vijay Priority: Minor Fix For: 1.1.2, 1.2 Attachments: 0001-CASSANDRA-4320.patch java.lang.AssertionError at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:351) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:269) at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did some digging and looks like we just need to skip the deleted columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4321) stackoverflow building interval tree possible sstable corruptions
[ https://issues.apache.org/jira/browse/CASSANDRA-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291511#comment-13291511 ] Jonathan Ellis commented on CASSANDRA-4321: --- bq. SSTable first key DecoratedKey(100294972947100949193477090306072672386, 4fcf051ef5067d7f17d9fc35) last key DecoratedKey(90250429663386465697464050082134975058, 4fce996e3c1eed8c4b17dd66) Cassandra checks key ordering for correctness with every row that is added at write time, so unless you changed your partitioner (which is emphatically Not Supported), I'm not sure how this can happen. Whatever it is, it's unlikely to be related to the 1.1.1 upgrade. Scrub checks that the sstable content is well-formed and readable. It doesn't check for out-of-order rows. You can use a tool like sstablekeys to do that. stackoverflow building interval tree possible sstable corruptions --- Key: CASSANDRA-4321 URL: https://issues.apache.org/jira/browse/CASSANDRA-4321 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Reporter: Anton Winter After upgrading to 1.1.1 (from 1.1.0) I have started experiencing StackOverflowError's resulting in compaction backlog and failure to restart. The ring currently consists of 6 DC's and 22 nodes using LCS compression. This issue was first noted on 2 nodes in one DC and then appears to have spread to various other nodes in the other DC's. When the first occurrence of this was found I restarted the instance but it failed to start so I cleared its data and treated it as a replacement node for the token it was previously responsible for. This node successfully streamed all the relevant data back but failed again a number of hours later with the same StackOverflowError and again was unable to restart. The initial stack overflow error on a running instance looks like this: ERROR [CompactionExecutor:314] 2012-06-07 09:59:43,017 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:314,1,main] java.lang.StackOverflowError at java.util.Arrays.mergeSort(Arrays.java:1157) at java.util.Arrays.sort(Arrays.java:1092) at java.util.Collections.sort(Collections.java:134) at org.apache.cassandra.utils.IntervalTree.IntervalNode.findMinMedianMax(IntervalNode.java:114) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:49) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) [snip - this repeats until stack overflow. Compactions stop from this point onwards] I restarted this failing instance with DEBUG logging enabled and it throws the following exception part way through startup: ERROR 11:37:51,046 Exception in thread Thread[OptionalTasks:1,5,main] java.lang.StackOverflowError at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:307) at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:276) at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:230) at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:124) at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:228) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:45) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) [snip - this repeats until stack overflow] at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at
[jira] [Commented] (CASSANDRA-4321) stackoverflow building interval tree possible sstable corruptions
[ https://issues.apache.org/jira/browse/CASSANDRA-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291520#comment-13291520 ] Anton Winter commented on CASSANDRA-4321: - The partitioner (RP) was not changed. stackoverflow building interval tree possible sstable corruptions --- Key: CASSANDRA-4321 URL: https://issues.apache.org/jira/browse/CASSANDRA-4321 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Reporter: Anton Winter After upgrading to 1.1.1 (from 1.1.0) I have started experiencing StackOverflowError's resulting in compaction backlog and failure to restart. The ring currently consists of 6 DC's and 22 nodes using LCS compression. This issue was first noted on 2 nodes in one DC and then appears to have spread to various other nodes in the other DC's. When the first occurrence of this was found I restarted the instance but it failed to start so I cleared its data and treated it as a replacement node for the token it was previously responsible for. This node successfully streamed all the relevant data back but failed again a number of hours later with the same StackOverflowError and again was unable to restart. The initial stack overflow error on a running instance looks like this: ERROR [CompactionExecutor:314] 2012-06-07 09:59:43,017 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:314,1,main] java.lang.StackOverflowError at java.util.Arrays.mergeSort(Arrays.java:1157) at java.util.Arrays.sort(Arrays.java:1092) at java.util.Collections.sort(Collections.java:134) at org.apache.cassandra.utils.IntervalTree.IntervalNode.findMinMedianMax(IntervalNode.java:114) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:49) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) [snip - this repeats until stack overflow. Compactions stop from this point onwards] I restarted this failing instance with DEBUG logging enabled and it throws the following exception part way through startup: ERROR 11:37:51,046 Exception in thread Thread[OptionalTasks:1,5,main] java.lang.StackOverflowError at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:307) at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:276) at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:230) at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:124) at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:228) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:45) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) [snip - this repeats until stack overflow] at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:64) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalNode.init(IntervalNode.java:62) at org.apache.cassandra.utils.IntervalTree.IntervalTree.init(IntervalTree.java:39) at org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:560) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:617) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:320) at org.apache.cassandra.db.DataTracker.addInitialSSTables(DataTracker.java:259) at
[jira] [Updated] (CASSANDRA-4314) OOM errors on key slice
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4314:
--
Attachment: 4314-1.1.txt
4314-1.0.txt
bq. Looks like we don't override index gcgs to 0
Patches to do this attached for 1.0 and 1.1. If you compact the *index* CF
with this patch applied, that should get rid of the tombstones. (compacting the
data CF won't do anything.)
OOM errors on key slice
---
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: 2012-06-07-compact.zip, 2012-06-07.zip, 4314-1.0.txt,
4314-1.1.txt, oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0
[jira] [Updated] (CASSANDRA-4314) Index CF tombstones can cause OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4314:
--
Priority: Critical (was: Major)
Summary: Index CF tombstones can cause OOM (was: OOM errors on key slice)
Index CF tombstones can cause OOM
-
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Assignee: Jonathan Ellis
Priority: Critical
Fix For: 1.0.11, 1.1.2
Attachments: 2012-06-07-compact.zip, 2012-06-07.zip, 4314-1.0.txt,
4314-1.1.txt, oom.zip, yourkitsnapshot.png
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
my cluster
Selector s = new Selector(this.pool);
ListIndexExpression indexExpressions = new
ArrayListIndexExpression();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap(encryptionSettingsID.getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
ListKeySlice slices = s.getKeySlices(CF_TOKEN,
indexClause, predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT,
strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 3 3107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage
[jira] [Updated] (CASSANDRA-4318) Nodetool compactionstats fails with NullPointerException
[
https://issues.apache.org/jira/browse/CASSANDRA-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4318:
--
Attachment: 4318.txt
patch attached to deal with cfm directly instead of looking up by id, and
create a dummy cfm for cache saving instead of leaving null
Nodetool compactionstats fails with NullPointerException
Key: CASSANDRA-4318
URL: https://issues.apache.org/jira/browse/CASSANDRA-4318
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.1
Environment: 2 node cluster running on Ubuntu 10.10 64-bit
16GB Max Heap allocated to each node.
Test keyspace using replication factor 2
Reporter: David B
Labels: caching
Fix For: 1.1.2
Attachments: 4318.txt
Test uses Column family C defined as follows:
create column family C with caching = 'keys_only' and key_validation_class =
'LongType' and compression_options = { sstable_compression: SnappyCompressor,
chunk_length_kb: 64 } and max_compaction_threshold=0;
max_compaction_threshold is set to 0 to disable auto compaction.
SSTables are streamed via sstableloader, after which a major compaction is
triggered using nodetool compact MyKeyspace C.
Thereafter, attempts to request compaction stats via nodetool
compactionstats fail with the following exception:
Exception in thread main java.lang.NullPointerException
at
org.apache.cassandra.db.compaction.CompactionInfo.asMap(CompactionInfo.java:103)
at
org.apache.cassandra.db.compaction.CompactionManager.getCompactions(CompactionManager.java:1115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4318) Nodetool compactionstats fails with NullPointerException
[
https://issues.apache.org/jira/browse/CASSANDRA-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4318:
--
Priority: Minor (was: Major)
Nodetool compactionstats fails with NullPointerException
Key: CASSANDRA-4318
URL: https://issues.apache.org/jira/browse/CASSANDRA-4318
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.1
Environment: 2 node cluster running on Ubuntu 10.10 64-bit
16GB Max Heap allocated to each node.
Test keyspace using replication factor 2
Reporter: David B
Assignee: Jonathan Ellis
Priority: Minor
Labels: caching
Fix For: 1.1.2
Attachments: 4318.txt
Test uses Column family C defined as follows:
create column family C with caching = 'keys_only' and key_validation_class =
'LongType' and compression_options = { sstable_compression: SnappyCompressor,
chunk_length_kb: 64 } and max_compaction_threshold=0;
max_compaction_threshold is set to 0 to disable auto compaction.
SSTables are streamed via sstableloader, after which a major compaction is
triggered using nodetool compact MyKeyspace C.
Thereafter, attempts to request compaction stats via nodetool
compactionstats fail with the following exception:
Exception in thread main java.lang.NullPointerException
at
org.apache.cassandra.db.compaction.CompactionInfo.asMap(CompactionInfo.java:103)
at
org.apache.cassandra.db.compaction.CompactionManager.getCompactions(CompactionManager.java:1115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4320) Assertion error while delivering the hints.
[ https://issues.apache.org/jira/browse/CASSANDRA-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-4320: - Attachment: 0001-CASSANDRA-4320-v2.patch ahaaa that works too :) Plz see the attached. Thanks! Assertion error while delivering the hints. --- Key: CASSANDRA-4320 URL: https://issues.apache.org/jira/browse/CASSANDRA-4320 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0, 1.2 Reporter: Vijay Assignee: Vijay Priority: Minor Fix For: 1.1.2, 1.2 Attachments: 0001-CASSANDRA-4320-v2.patch, 0001-CASSANDRA-4320.patch java.lang.AssertionError at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:351) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:269) at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Did some digging and looks like we just need to skip the deleted columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4012) Load-from-flat-file data import tool
[ https://issues.apache.org/jira/browse/CASSANDRA-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] paul cannon updated CASSANDRA-4012: --- Attachment: 4012.patch.txt Patch attached, or see the 4012 branch in my github. Current version tagged as pending/4012. https://github.com/thepaul/cassandra/tree/4012 Load-from-flat-file data import tool Key: CASSANDRA-4012 URL: https://issues.apache.org/jira/browse/CASSANDRA-4012 Project: Cassandra Issue Type: New Feature Components: Tools Reporter: Jonathan Ellis Assignee: paul cannon Labels: cqlsh Fix For: 1.1.2 Attachments: 4012.patch.txt Would like to be able to import data from a comma-delimited or tab-delimited file, one row per line. (That is, row can be logical row as in http://www.datastax.com/dev/blog/schema-in-cassandra-1-1.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4261) [patch] Support consistency-latency prediction in nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291541#comment-13291541
]
Peter Bailis commented on CASSANDRA-4261:
-
I agree that JMX would better. I'll work on changing this configuration and
will post performance numbers shortly. I should be able to have this done in a
week or so (latency due to my schedule, not due to task difficulty).
[patch] Support consistency-latency prediction in nodetool
--
Key: CASSANDRA-4261
URL: https://issues.apache.org/jira/browse/CASSANDRA-4261
Project: Cassandra
Issue Type: New Feature
Components: Tools
Affects Versions: 1.2
Reporter: Peter Bailis
Attachments: demo-pbs-v2.sh, pbs-nodetool-v2.patch
h3. Introduction
Cassandra supports a variety of replication configurations: ReplicationFactor
is set per-ColumnFamily and ConsistencyLevel is set per-request. Setting
{{ConsistencyLevel}} to {{QUORUM}} for reads and writes ensures strong
consistency, but {{QUORUM}} is often slower than {{ONE}}, {{TWO}}, or
{{THREE}}. What should users choose?
This patch provides a latency-consistency analysis within {{nodetool}}. Users
can accurately predict Cassandra's behavior in their production environments
without interfering with performance.
What's the probability that we'll read a write t seconds after it completes?
What about reading one of the last k writes? This patch provides answers via
{{nodetool predictconsistency}}:
{{nodetool predictconsistency ReplicationFactor TimeAfterWrite Versions}}
\\ \\
{code:title=Example output|borderStyle=solid}
//N == ReplicationFactor
//R == read ConsistencyLevel
//W == write ConsistencyLevel
user@test:$ nodetool predictconsistency 3 100 1
Performing consistency prediction
100ms after a given write, with maximum version staleness of k=1
N=3, R=1, W=1
Probability of consistent reads: 0.678900
Average read latency: 5.377900ms (99.900th %ile 40ms)
Average write latency: 36.971298ms (99.900th %ile 294ms)
N=3, R=1, W=2
Probability of consistent reads: 0.791600
Average read latency: 5.372500ms (99.900th %ile 39ms)
Average write latency: 303.630890ms (99.900th %ile 357ms)
N=3, R=1, W=3
Probability of consistent reads: 1.00
Average read latency: 5.426600ms (99.900th %ile 42ms)
Average write latency: 1382.650879ms (99.900th %ile 629ms)
N=3, R=2, W=1
Probability of consistent reads: 0.915800
Average read latency: 11.091000ms (99.900th %ile 348ms)
Average write latency: 42.663101ms (99.900th %ile 284ms)
N=3, R=2, W=2
Probability of consistent reads: 1.00
Average read latency: 10.606800ms (99.900th %ile 263ms)
Average write latency: 310.117615ms (99.900th %ile 335ms)
N=3, R=3, W=1
Probability of consistent reads: 1.00
Average read latency: 52.657501ms (99.900th %ile 565ms)
Average write latency: 39.949799ms (99.900th %ile 237ms)
{code}
h3. Demo
Here's an example scenario you can run using
[ccm|https://github.com/pcmanus/ccm]. The prediction is fast:
{code:borderStyle=solid}
cd cassandra-source-dir with patch applied
ant
# turn on consistency logging
sed -i .bak 's/log_latencies_for_consistency_prediction:
false/log_latencies_for_consistency_prediction: true/' conf/cassandra.yaml
ccm create consistencytest --cassandra-dir=.
ccm populate -n 5
ccm start
# if start fails, you might need to initialize more loopback interfaces
# e.g., sudo ifconfig lo0 alias 127.0.0.2
# use stress to get some sample latency data
tools/bin/stress -d 127.0.0.1 -l 3 -n 1 -o insert
tools/bin/stress -d 127.0.0.1 -l 3 -n 1 -o read
bin/nodetool -h 127.0.0.1 -p 7100 predictconsistency 3 100 1
{code}
h3. What and Why
We've implemented [Probabilistically Bounded
Staleness|http://pbs.cs.berkeley.edu/#demo], a new technique for predicting
consistency-latency trade-offs within Cassandra. Our
[paper|http://arxiv.org/pdf/1204.6082.pdf] will appear in [VLDB
2012|http://www.vldb2012.org/], and, in it, we've used PBS to profile a range
of Dynamo-style data store deployments at places like LinkedIn and Yammer in
addition to profiling our own Cassandra deployments. In our experience,
prediction is both accurate and much more lightweight than profiling and
manually testing each possible replication configuration (especially in
production!).
This analysis is important for the many users we've talked to and heard about
who use partial quorum operation (e.g., non-{{QUORUM}}
{{ConsistencyLevel}}). Should they use CL={{ONE}}? CL={{TWO}}? It likely
depends on their runtime environment and, short of profiling in production,
there's no existing way to answer these questions. (Keep in mind, Cassandra
defaults to CL={{ONE}}, meaning users don't know how stale their data will
[jira] [Comment Edited] (CASSANDRA-4261) [patch] Support consistency-latency prediction in nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291541#comment-13291541
]
Peter Bailis edited comment on CASSANDRA-4261 at 6/8/12 4:07 AM:
-
I agree that JMX would work better. I'll work on changing this configuration
and will post performance numbers shortly. I should be able to have this done
in a week or so (latency due to my schedule, not due to task difficulty).
was (Author: pbailis):
I agree that JMX would better. I'll work on changing this configuration and
will post performance numbers shortly. I should be able to have this done in a
week or so (latency due to my schedule, not due to task difficulty).
[patch] Support consistency-latency prediction in nodetool
--
Key: CASSANDRA-4261
URL: https://issues.apache.org/jira/browse/CASSANDRA-4261
Project: Cassandra
Issue Type: New Feature
Components: Tools
Affects Versions: 1.2
Reporter: Peter Bailis
Attachments: demo-pbs-v2.sh, pbs-nodetool-v2.patch
h3. Introduction
Cassandra supports a variety of replication configurations: ReplicationFactor
is set per-ColumnFamily and ConsistencyLevel is set per-request. Setting
{{ConsistencyLevel}} to {{QUORUM}} for reads and writes ensures strong
consistency, but {{QUORUM}} is often slower than {{ONE}}, {{TWO}}, or
{{THREE}}. What should users choose?
This patch provides a latency-consistency analysis within {{nodetool}}. Users
can accurately predict Cassandra's behavior in their production environments
without interfering with performance.
What's the probability that we'll read a write t seconds after it completes?
What about reading one of the last k writes? This patch provides answers via
{{nodetool predictconsistency}}:
{{nodetool predictconsistency ReplicationFactor TimeAfterWrite Versions}}
\\ \\
{code:title=Example output|borderStyle=solid}
//N == ReplicationFactor
//R == read ConsistencyLevel
//W == write ConsistencyLevel
user@test:$ nodetool predictconsistency 3 100 1
Performing consistency prediction
100ms after a given write, with maximum version staleness of k=1
N=3, R=1, W=1
Probability of consistent reads: 0.678900
Average read latency: 5.377900ms (99.900th %ile 40ms)
Average write latency: 36.971298ms (99.900th %ile 294ms)
N=3, R=1, W=2
Probability of consistent reads: 0.791600
Average read latency: 5.372500ms (99.900th %ile 39ms)
Average write latency: 303.630890ms (99.900th %ile 357ms)
N=3, R=1, W=3
Probability of consistent reads: 1.00
Average read latency: 5.426600ms (99.900th %ile 42ms)
Average write latency: 1382.650879ms (99.900th %ile 629ms)
N=3, R=2, W=1
Probability of consistent reads: 0.915800
Average read latency: 11.091000ms (99.900th %ile 348ms)
Average write latency: 42.663101ms (99.900th %ile 284ms)
N=3, R=2, W=2
Probability of consistent reads: 1.00
Average read latency: 10.606800ms (99.900th %ile 263ms)
Average write latency: 310.117615ms (99.900th %ile 335ms)
N=3, R=3, W=1
Probability of consistent reads: 1.00
Average read latency: 52.657501ms (99.900th %ile 565ms)
Average write latency: 39.949799ms (99.900th %ile 237ms)
{code}
h3. Demo
Here's an example scenario you can run using
[ccm|https://github.com/pcmanus/ccm]. The prediction is fast:
{code:borderStyle=solid}
cd cassandra-source-dir with patch applied
ant
# turn on consistency logging
sed -i .bak 's/log_latencies_for_consistency_prediction:
false/log_latencies_for_consistency_prediction: true/' conf/cassandra.yaml
ccm create consistencytest --cassandra-dir=.
ccm populate -n 5
ccm start
# if start fails, you might need to initialize more loopback interfaces
# e.g., sudo ifconfig lo0 alias 127.0.0.2
# use stress to get some sample latency data
tools/bin/stress -d 127.0.0.1 -l 3 -n 1 -o insert
tools/bin/stress -d 127.0.0.1 -l 3 -n 1 -o read
bin/nodetool -h 127.0.0.1 -p 7100 predictconsistency 3 100 1
{code}
h3. What and Why
We've implemented [Probabilistically Bounded
Staleness|http://pbs.cs.berkeley.edu/#demo], a new technique for predicting
consistency-latency trade-offs within Cassandra. Our
[paper|http://arxiv.org/pdf/1204.6082.pdf] will appear in [VLDB
2012|http://www.vldb2012.org/], and, in it, we've used PBS to profile a range
of Dynamo-style data store deployments at places like LinkedIn and Yammer in
addition to profiling our own Cassandra deployments. In our experience,
prediction is both accurate and much more lightweight than profiling and
manually testing each possible replication configuration (especially in
production!).
This analysis is important for the many users we've talked to and heard about
who use partial quorum operation
