[jira] [Resolved] (CASSANDRA-6199) Improve Stress Tool
[
https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Yaskevich resolved CASSANDRA-6199.
Resolution: Fixed
Fix Version/s: 2.1
Merged, with following changes:
- sun.reflect.generics.reflectiveObjects.NotImplementedException changed to
UnsupportedOperationException in CqlIndexedRangeSlicer, SettingsTransport,
StressSettings, CqlIndexedRangeSlicer.
- CqlIndexedRangeSlicer, changed StringBuilder append methods to avoid string
concatenation inside.
- fix SettingsPort to use correct Thrift port by default, was using default
value of options.nativePort.value() for both Native and Thrift.
- fixed NPE in StressSettings.disconnect().
- changed 2s sleeping to Uninterruptibles.sleepUninterruptibly(2,
TimeUnit.SECONDS); instead of Thread.sleep with try {} catch
- optimized imports and their arrangement.
- minor code formatting changes.
> Improve Stress Tool
> ---
>
> Key: CASSANDRA-6199
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6199
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Benedict
>Assignee: Benedict
>Priority: Minor
> Fix For: 2.1
>
> Attachments: new.read.latency.svg, new.read.rate.distribution.svg,
> new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg,
> old.read.rate.distribution.svg, old.write.latency.svg,
> old.write.rate.distribution.svg, ops.read.svg, ops.write.svg, patch
>
>
> The stress tool could do with sprucing up. The following is a list of
> essential improvements and things that would be nice to have.
> Essential:
> - Reduce variability of results, especially start/end tails. Do not trash
> first/last 10% of readings
> - Reduce contention/overhead in stress to increase overall throughput
> - Short warm-up period, which is ignored for summary (or summarised
> separately), though prints progress as usual. Potentially automatic detection
> of rate levelling.
> - Better configurability and defaults for data generation - current column
> generation populates columns with the same value for every row, which is very
> easily compressible. Possibly introduce partial random data generator
> (possibly dictionary-based random data generator)
> Nice to have:
> - Calculate and print stdev and mean
> - Add batched sequential access mode (where a single thread performs
> batch-size sequential requests before selecting another random key) to test
> how key proximity affects performance
> - Auto-mode which attempts to establish the maximum throughput rate, by
> varying the thread count (or otherwise gating the number of parallel
> requests) for some period, then configures rate limit or thread count to test
> performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained.
> - Auto-mode could have a target variance ratio for mean throughput and/or
> latency, and completes a test once this target is hit for x intervals
> - Fix key representation so independent of number of keys (possibly switch to
> 10 digit hex), and don't use String.format().getBytes() to construct it
> (expensive)
> Also, remove the skip-key setting, as it is currently ignored. Unless
> somebody knows the reason for it.
> - Fix latency stats
> - Read/write mode, with configurable recency-of-reads distribution
> - Add new exponential/extreme value distribution for value size, column count
> and recency-of-reads
> - Support more than 2^31 keys
> - Supports multiple concurrent stress inserts via key-offset parameter or
> similar
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[1/6] Improve Stress Tool patch by Benedict; reviewed by Pavel Yaskevich for CASSANDRA-6199
Updated Branches:
refs/heads/trunk 34235ad7b -> 2e1e98ad0
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e1e98ad/tools/stress/src/org/apache/cassandra/stress/util/SmartThriftClient.java
--
diff --git
a/tools/stress/src/org/apache/cassandra/stress/util/SmartThriftClient.java
b/tools/stress/src/org/apache/cassandra/stress/util/SmartThriftClient.java
new file mode 100644
index 000..99fa452
--- /dev/null
+++ b/tools/stress/src/org/apache/cassandra/stress/util/SmartThriftClient.java
@@ -0,0 +1,235 @@
+package org.apache.cassandra.stress.util;
+
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.datastax.driver.core.Host;
+import com.datastax.driver.core.Metadata;
+import org.apache.cassandra.stress.settings.StressSettings;
+import org.apache.cassandra.thrift.*;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.thrift.TException;
+
+public class SmartThriftClient implements ThriftClient
+{
+
+final String keyspace;
+final Metadata metadata;
+final StressSettings settings;
+final ConcurrentHashMap> cache = new
ConcurrentHashMap<>();
+
+final AtomicInteger queryIdCounter = new AtomicInteger();
+final ConcurrentHashMap queryStrings = new
ConcurrentHashMap<>();
+final ConcurrentHashMap queryIds = new
ConcurrentHashMap<>();
+
+public SmartThriftClient(StressSettings settings, String keyspace,
Metadata metadata)
+{
+this.metadata = metadata;
+this.keyspace = keyspace;
+this.settings = settings;
+}
+
+private final AtomicInteger roundrobin = new AtomicInteger();
+
+private Integer getId(String query)
+{
+Integer r;
+if ((r = queryIds.get(query)) != null)
+return r;
+r = queryIdCounter.incrementAndGet();
+if (queryIds.putIfAbsent(query, r) == null)
+return r;
+queryStrings.put(r, query);
+return queryIds.get(query);
+}
+
+final class Client
+{
+final Cassandra.Client client;
+final Host host;
+final Map queryMap = new HashMap<>();
+
+Client(Cassandra.Client client, Host host)
+{
+this.client = client;
+this.host = host;
+}
+
+Integer get(Integer id, boolean cql3) throws TException
+{
+Integer serverId = queryMap.get(id);
+if (serverId != null)
+return serverId;
+prepare(id, cql3);
+return queryMap.get(id);
+}
+
+ void prepare(Integer id, boolean cql3) throws TException
+ {
+ String query;
+ while ( null == (query = queryStrings.get(id)) ) ;
+ if (cql3)
+ {
+ Integer serverId =
client.prepare_cql3_query(ByteBufferUtil.bytes(query), Compression.NONE).itemId;
+ queryMap.put(id, serverId);
+ }
+ else
+ {
+ Integer serverId =
client.prepare_cql_query(ByteBufferUtil.bytes(query), Compression.NONE).itemId;
+ queryMap.put(id, serverId);
+ }
+ }
+}
+
+private Client get(ByteBuffer pk)
+{
+Set hosts = metadata.getReplicas(keyspace, pk);
+int count = roundrobin.incrementAndGet() % hosts.size();
+if (count < 0)
+count = -count;
+Iterator iter = hosts.iterator();
+while (count > 0 && iter.hasNext())
+iter.next();
+Host host = iter.next();
+ConcurrentLinkedQueue q = cache.get(host);
+if (q == null)
+{
+ConcurrentLinkedQueue newQ = new
ConcurrentLinkedQueue();
+q = cache.putIfAbsent(host, newQ);
+if (q == null)
+q = newQ;
+}
+Client tclient = q.poll();
+if (tclient != null)
+return tclient;
+return new
Client(settings.getRawThriftClient(host.getAddress().getHostAddress()), host);
+}
+
+@Override
+public void batch_mutate(Map>>
record, ConsistencyLevel consistencyLevel) throws TException
+{
+for (Map.Entry>> e :
record.entrySet())
+{
+Client client = get(e.getKey());
+try
+{
+
client.client.batch_mutate(Collections.singletonMap(e.getKey(), e.getValue()),
consistencyLevel);
+} finally
+{
+cache.get(client.host).add(client);
+}
+}
+}
+
+@Override
+public List get_slice(ByteBuffer key, ColumnParent
parent, SlicePredicate predicate, ConsistencyLevel consistencyLevel) throws
InvalidRequestException, UnavailableException, TimedOutException, TException
+{
+Client client = get(key);
+try
+{
+return client.cli
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13856081#comment-13856081 ] Jonathan Ellis commented on CASSANDRA-4165: --- https://github.com/jbellis/cassandra/commits/4165-3 refactors and switches to an adler32 digest, WDYT? > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6199: Attachment: patch In case you wanted an actual patch file > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg, patch > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13856048#comment-13856048 ] Benedict commented on CASSANDRA-6199: - [https://github.com/belliottsmith/cassandra/tree/merge-6199] > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-4899) add gitignore
[ https://issues.apache.org/jira/browse/CASSANDRA-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-4899: -- Attachment: .gitignore Relatively complete .gitignore file which includes .gitignore itself :) > add gitignore > - > > Key: CASSANDRA-4899 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4899 > Project: Cassandra > Issue Type: Task >Reporter: Radim Kolar >Assignee: Radim Kolar >Priority: Trivial > Attachments: .gitignore, cass-gitignore.txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855979#comment-13855979 ] Pavel Yaskevich commented on CASSANDRA-6199: >From what you are saying it sounds like better option would be to use put as >you can ignore the previous value safely and keys are immutable, which raises >another question - if you know what you are going to get the same instance >back then why do you even need to do a get afterwards? Anyhow, can you attach >a squashed patch to the ticket so I can push it as an entity? > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855977#comment-13855977 ] Benedict commented on CASSANDRA-6199: - That is a very bad reason to consider it the only correct usage. There is a stronger reason to provide the return value, and that is linearizability. You cannot be sure in many use cases of CHM.putIfAbsent that get() will return the value you inserted, so some uses require it for actual correctness of function. Since each key is immutable in our use case, this is not true, and we can do it safely. Since there is a stronger reason to require the functionality, and because in highly contended use cases there is a performance benefit, it is not reasonable to assume that in situations where the stronger requirement is not present and the performance is irrelevant that the functionality must be used. Either way, I have agreed to the change, so I suggest we leave it there :-) > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855969#comment-13855969 ] Pavel Yaskevich commented on CASSANDRA-6199: What are even talking about there? Look at the javadoc for putIfAbsent method, this gives you an idea right away. Is original authors intended that method to be used differently, they wouldn't have made it to return previous value or even removed it as useless the way guava cache did. There is no point of ignoring return and doing get right after, no matter how small the collection is. > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855945#comment-13855945 ] Benedict commented on CASSANDRA-6199: - bq. you can do it in 2 lines too If you want: That is 3 LOC, you've just omitted you're initialisation of the CLQ from your snippet. bq. This is more about using map api correctly than about personal taste. I don't know where you got the idea there was only one correct way to use this API; generally correctness is determined by correctness, not style. However I do not feel strongly enough about it to further argue, so have uploaded a patch with your change. > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4369) Sortable counter type for 'top N' type query use cases
[ https://issues.apache.org/jira/browse/CASSANDRA-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855946#comment-13855946 ] Maheedhar Gunturu commented on CASSANDRA-4369: -- It will be really nice to have this issue fixed as we heavily rely on this use-case. [~jbellis]S - Can someone Please take this up. > Sortable counter type for 'top N' type query use cases > -- > > Key: CASSANDRA-4369 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4369 > Project: Cassandra > Issue Type: New Feature >Reporter: Ahmet AKYOL > Labels: ponies > > Current counter type is not sortable by values, AFAIK. Here's a related > question : > http://stackoverflow.com/questions/8428364/how-to-get-sorted-counters-from-cassandra > This feature can be very handy for 'top N' type query use cases like sorting > user points and finding top users in a reputation system. > Of course, support in CQL 3 is trivial. > IMHO, from implementation point of view, this looks unnatural to C* since C* > sorts column names not the values.Maybe a composite column solution like > (counter,char) is more proper but I can't imagine a way to make this work > right now, it doesn't make sense to me. On the other hand, it would be very > useful for us, the users. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855939#comment-13855939 ] Pavel Yaskevich commented on CASSANDRA-6199: I'm not sure how clarity is polluted, putIfAbsent is used correctly and value is swapped if needed, you can do it in 2 lines too If you want: if ((q = cache.putIfAbsent(...)) == null) q = newQ; This is more about using map api correctly than about personal taste. > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855927#comment-13855927 ] Benedict commented on CASSANDRA-6199: - bq. What I mentioned is a correct and preferred way to do element placement into concurrent map because putIfAbsent already does all the work for you. I know, and in highly contended cases I absolutely agree with you, but here it only pollutes clarity of the code without incurring a major benefit, since the map is small and updated only briefly, after which it is read only. > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[ https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855926#comment-13855926 ] Pavel Yaskevich commented on CASSANDRA-6199: bq. This was a conscious decision, as I prefer the brevity of the current expression (2 vs 5 LOC), and it's a cost incurred only ~once, but this is something I vary my position on, so I'm sanguine about changing it. What I mentioned is a correct and preferred way to do element placement into concurrent map because putIfAbsent already does all the work for you. > Improve Stress Tool > --- > > Key: CASSANDRA-6199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6199 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Benedict >Priority: Minor > Attachments: new.read.latency.svg, new.read.rate.distribution.svg, > new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg, > old.read.rate.distribution.svg, old.write.latency.svg, > old.write.rate.distribution.svg, ops.read.svg, ops.write.svg > > > The stress tool could do with sprucing up. The following is a list of > essential improvements and things that would be nice to have. > Essential: > - Reduce variability of results, especially start/end tails. Do not trash > first/last 10% of readings > - Reduce contention/overhead in stress to increase overall throughput > - Short warm-up period, which is ignored for summary (or summarised > separately), though prints progress as usual. Potentially automatic detection > of rate levelling. > - Better configurability and defaults for data generation - current column > generation populates columns with the same value for every row, which is very > easily compressible. Possibly introduce partial random data generator > (possibly dictionary-based random data generator) > Nice to have: > - Calculate and print stdev and mean > - Add batched sequential access mode (where a single thread performs > batch-size sequential requests before selecting another random key) to test > how key proximity affects performance > - Auto-mode which attempts to establish the maximum throughput rate, by > varying the thread count (or otherwise gating the number of parallel > requests) for some period, then configures rate limit or thread count to test > performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained. > - Auto-mode could have a target variance ratio for mean throughput and/or > latency, and completes a test once this target is hit for x intervals > - Fix key representation so independent of number of keys (possibly switch to > 10 digit hex), and don't use String.format().getBytes() to construct it > (expensive) > Also, remove the skip-key setting, as it is currently ignored. Unless > somebody knows the reason for it. > - Fix latency stats > - Read/write mode, with configurable recency-of-reads distribution > - Add new exponential/extreme value distribution for value size, column count > and recency-of-reads > - Support more than 2^31 keys > - Supports multiple concurrent stress inserts via key-offset parameter or > similar -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5220) Repair improvements when using vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855774#comment-13855774
]
Donald Smith edited comment on CASSANDRA-5220 at 12/23/13 8:49 PM:
---
We ran "nodetool repair" on a 3 node cassandra cluster with production-quality
hardware, using version 2.0.3. Each node had about 1TB of data. This is still
testing. After 5 days the repair job still hasn't finished. I can see it's
still running.
Here's the process:
{noformat}
root 30835 30774 0 Dec17 pts/000:03:53 /usr/bin/java -cp
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.3.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/thrift-server-0.3.2.jar
-Xmx32m -Dlog4j.configuration=log4j-tools.properties
-Dstorage-config=/etc/cassandra/conf org.apache.cassandra.tools.NodeCmd -p 7199
repair -pr as_reports
{noformat}
The log output has just:
{noformat}
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
[2013-12-17 23:26:48,144] Starting repair command #1, repairing 256 ranges for
keyspace as_reports
{noformat}
Here's the output of "nodetool tpstats":
{noformat}
cass3 /tmp> nodetool tpstats
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 1 0 38083403 0
0
RequestResponseStage 0 0 1951200451 0
0
MutationStage 0 0 2853354069 0
0
ReadRepairStage 0 03794926 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 04880147 0
0
AntiEntropyStage 1 3 9 0
0
MigrationStage0 0 30 0
0
MemoryMeter 0 0115 0
0
MemtablePostFlusher 0 0 75121 0
0
FlushWriter 0 0 49934 0
52
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 7 0
0
commitlog_archiver0 0 0 0
0
AntiEntropySessions 1 1 1 0
0
InternalResponseStage 0 0 9 0
0
HintedHandoff 0 0 1141 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 884
MUTATION 1407711
_TRACE 0
REQUEST_RESPONSE 0
{noformat}
The cluster has some write traffic to it. We decided to test it under load.
This is the busiest c
[jira] [Commented] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[ https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855920#comment-13855920 ] Jonathan Ellis commented on CASSANDRA-5351: --- Am I supposed to be looking for a different error than the incorrect new-size above? > Avoid repairing already-repaired data by default > > > Key: CASSANDRA-5351 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5351 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jonathan Ellis >Assignee: Lyuben Todorov > Labels: repair > Fix For: 2.1 > > Attachments: node1.log, node1_v2_full.log, node2.log, > node2_v2_full.log, node3.log, node3_v2_full.log > > > Repair has always built its merkle tree from all the data in a columnfamily, > which is guaranteed to work but is inefficient. > We can improve this by remembering which sstables have already been > successfully repaired, and only repairing sstables new since the last repair. > (This automatically makes CASSANDRA-3362 much less of a problem too.) > The tricky part is, compaction will (if not taught otherwise) mix repaired > data together with non-repaired. So we should segregate unrepaired sstables > from the repaired ones. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5220) Repair improvements when using vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855774#comment-13855774
]
Donald Smith edited comment on CASSANDRA-5220 at 12/23/13 8:30 PM:
---
We ran "nodetool repair" on a 3 node cassandra cluster with production-quality
hardware, using version 2.0.3. Each node had about 1TB of data. This is still
testing. After 5 days the repair job still hasn't finished. I can see it's
still running.
Here's the process:
{noformat}
root 30835 30774 0 Dec17 pts/000:03:53 /usr/bin/java -cp
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.3.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/thrift-server-0.3.2.jar
-Xmx32m -Dlog4j.configuration=log4j-tools.properties
-Dstorage-config=/etc/cassandra/conf org.apache.cassandra.tools.NodeCmd -p 7199
repair -pr as_reports
{noformat}
The log output has just:
{noformat}
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
[2013-12-17 23:26:48,144] Starting repair command #1, repairing 256 ranges for
keyspace as_reports
{noformat}
Here's the output of "nodetool tpstats":
{noformat}
cass3 /tmp> nodetool tpstats
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 1 0 38083403 0
0
RequestResponseStage 0 0 1951200451 0
0
MutationStage 0 0 2853354069 0
0
ReadRepairStage 0 03794926 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 04880147 0
0
AntiEntropyStage 1 3 9 0
0
MigrationStage0 0 30 0
0
MemoryMeter 0 0115 0
0
MemtablePostFlusher 0 0 75121 0
0
FlushWriter 0 0 49934 0
52
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 7 0
0
commitlog_archiver0 0 0 0
0
AntiEntropySessions 1 1 1 0
0
InternalResponseStage 0 0 9 0
0
HintedHandoff 0 0 1141 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 884
MUTATION 1407711
_TRACE 0
REQUEST_RESPONSE 0
{noformat}
The cluster has some write traffic to it. We decided to test it under load.
This is the busiest c
[jira] [Comment Edited] (CASSANDRA-5220) Repair improvements when using vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855774#comment-13855774
]
Donald Smith edited comment on CASSANDRA-5220 at 12/23/13 8:26 PM:
---
We ran "nodetool repair" on a 3 node cassandra cluster with production-quality
hardware, using version 2.0.3. Each node had about 1TB of data. This is still
testing. After 5 days the repair job still hasn't finished. I can see it's
still running.
Here's the process:
{noformat}
root 30835 30774 0 Dec17 pts/000:03:53 /usr/bin/java -cp
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.3.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/thrift-server-0.3.2.jar
-Xmx32m -Dlog4j.configuration=log4j-tools.properties
-Dstorage-config=/etc/cassandra/conf org.apache.cassandra.tools.NodeCmd -p 7199
repair -pr as_reports
{noformat}
The log output has just:
{noformat}
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
[2013-12-17 23:26:48,144] Starting repair command #1, repairing 256 ranges for
keyspace as_reports
{noformat}
Here's the output of "nodetool tpstats":
{noformat}
cass3 /tmp> nodetool tpstats
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 1 0 38083403 0
0
RequestResponseStage 0 0 1951200451 0
0
MutationStage 0 0 2853354069 0
0
ReadRepairStage 0 03794926 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 04880147 0
0
AntiEntropyStage 1 3 9 0
0
MigrationStage0 0 30 0
0
MemoryMeter 0 0115 0
0
MemtablePostFlusher 0 0 75121 0
0
FlushWriter 0 0 49934 0
52
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 7 0
0
commitlog_archiver0 0 0 0
0
AntiEntropySessions 1 1 1 0
0
InternalResponseStage 0 0 9 0
0
HintedHandoff 0 0 1141 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 884
MUTATION 1407711
_TRACE 0
REQUEST_RESPONSE 0
{noformat}
The cluster has some write traffic to it. We decided to test it under load.
This is the busiest c
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[
https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855907#comment-13855907
]
Benedict commented on CASSANDRA-6199:
-
Thanks for the review.
bq. Has a left over "public void run(Cassandra.Client client) throws
IOException" which should be removed.
Nice spot
bq. needs formatting of {}.
I copied the JavaDriverClient straight from an internal project, and forgot to
reformat, which was an oversight. Thanks
bq. Current logic for new cache per host could be replaced with following to
avoid double get on map.
This was a conscious decision, as I prefer the brevity of the current
expression (2 vs 5 LOC), and it's a cost incurred only ~once, but this is
something I vary my position on, so I'm sanguine about changing it.
bq. if cluster.shutdown() is a Future, you can uses
Futures.getUnchecked(cluster.shutdown()); instead of converting checked
exceptions into RE
Excellent point.
I've patched all but SmartThriftClient and updated the repository.
I've also quickly patched it to support configuring the native port, as I
discussed with Jason Brown a little while ago.
> Improve Stress Tool
> ---
>
> Key: CASSANDRA-6199
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6199
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Benedict
>Assignee: Benedict
>Priority: Minor
> Attachments: new.read.latency.svg, new.read.rate.distribution.svg,
> new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg,
> old.read.rate.distribution.svg, old.write.latency.svg,
> old.write.rate.distribution.svg, ops.read.svg, ops.write.svg
>
>
> The stress tool could do with sprucing up. The following is a list of
> essential improvements and things that would be nice to have.
> Essential:
> - Reduce variability of results, especially start/end tails. Do not trash
> first/last 10% of readings
> - Reduce contention/overhead in stress to increase overall throughput
> - Short warm-up period, which is ignored for summary (or summarised
> separately), though prints progress as usual. Potentially automatic detection
> of rate levelling.
> - Better configurability and defaults for data generation - current column
> generation populates columns with the same value for every row, which is very
> easily compressible. Possibly introduce partial random data generator
> (possibly dictionary-based random data generator)
> Nice to have:
> - Calculate and print stdev and mean
> - Add batched sequential access mode (where a single thread performs
> batch-size sequential requests before selecting another random key) to test
> how key proximity affects performance
> - Auto-mode which attempts to establish the maximum throughput rate, by
> varying the thread count (or otherwise gating the number of parallel
> requests) for some period, then configures rate limit or thread count to test
> performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained.
> - Auto-mode could have a target variance ratio for mean throughput and/or
> latency, and completes a test once this target is hit for x intervals
> - Fix key representation so independent of number of keys (possibly switch to
> 10 digit hex), and don't use String.format().getBytes() to construct it
> (expensive)
> Also, remove the skip-key setting, as it is currently ignored. Unless
> somebody knows the reason for it.
> - Fix latency stats
> - Read/write mode, with configurable recency-of-reads distribution
> - Add new exponential/extreme value distribution for value size, column count
> and recency-of-reads
> - Support more than 2^31 keys
> - Supports multiple concurrent stress inserts via key-offset parameter or
> similar
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5220) Repair improvements when using vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855774#comment-13855774
]
Donald Smith edited comment on CASSANDRA-5220 at 12/23/13 8:22 PM:
---
We ran "nodetool repair" on a 3 node cassandra cluster with production-quality
hardware, using version 2.0.3. Each node had about 1TB of data. This is still
testing. After 5 days the repair job still hasn't finished. I can see it's
still running.
Here's the process:
{noformat}
root 30835 30774 0 Dec17 pts/000:03:53 /usr/bin/java -cp
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.3.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/thrift-server-0.3.2.jar
-Xmx32m -Dlog4j.configuration=log4j-tools.properties
-Dstorage-config=/etc/cassandra/conf org.apache.cassandra.tools.NodeCmd -p 7199
repair -pr as_reports
{noformat}
The log output has just:
{noformat}
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
[2013-12-17 23:26:48,144] Starting repair command #1, repairing 256 ranges for
keyspace as_reports
{noformat}
Here's the output of "nodetool tpstats":
{noformat}
cass3 /tmp> nodetool tpstats
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 1 0 38083403 0
0
RequestResponseStage 0 0 1951200451 0
0
MutationStage 0 0 2853354069 0
0
ReadRepairStage 0 03794926 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 04880147 0
0
AntiEntropyStage 1 3 9 0
0
MigrationStage0 0 30 0
0
MemoryMeter 0 0115 0
0
MemtablePostFlusher 0 0 75121 0
0
FlushWriter 0 0 49934 0
52
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 7 0
0
commitlog_archiver0 0 0 0
0
AntiEntropySessions 1 1 1 0
0
InternalResponseStage 0 0 9 0
0
HintedHandoff 0 0 1141 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 884
MUTATION 1407711
_TRACE 0
REQUEST_RESPONSE 0
{noformat}
The cluster has some write traffic to it. We decided to test it under load.
This is the busiest c
[jira] [Updated] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[
https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lyuben Todorov updated CASSANDRA-5351:
--
Attachment: node3_v2_full.log
node1_v2_full.log
node2_v2_full.log
Attaching full logs for all 3 nodes with logging set to DEBUG. 1st repair was
on node 3, and 2nd repair was on node 2.
Nodetool output:
{noformat}
lyubent:bin lyubentodorov$ ./nodetool repair -p 7300
[2013-12-23 21:46:38,321] Nothing to repair for keyspace 'system'
[2013-12-23 21:46:38,326] Starting repair command #1, repairing 2 ranges for
keyspace test
[2013-12-23 21:46:39,536] Repair session f000e4b0-6c0a-11e3-acbf-975f903ccf5a
for range (-3074457345618258603,3074457345618258602] finished
[2013-12-23 21:46:39,541] Repair session f075c690-6c0a-11e3-acbf-975f903ccf5a
for range (-9223372036854775808,-3074457345618258603] finished
[2013-12-23 21:46:39,541] Repair command #1 finished
[2013-12-23 21:46:39,549] Starting repair command #2, repairing 2 ranges for
keyspace system_traces
[2013-12-23 21:46:39,811] Repair session f0babed0-6c0a-11e3-acbf-975f903ccf5a
for range (-3074457345618258603,3074457345618258602] finished
[2013-12-23 21:46:39,816] Repair session f0cebc00-6c0a-11e3-acbf-975f903ccf5a
for range (-9223372036854775808,-3074457345618258603] finished
[2013-12-23 21:46:39,816] Repair command #2 finished
lyubent:bin lyubentodorov$ ./nodetool repair -p 7200
[2013-12-23 21:47:08,115] Nothing to repair for keyspace 'system'
[2013-12-23 21:47:08,121] Starting repair command #1, repairing 2 ranges for
keyspace test
[2013-12-23 21:47:08,604] Repair session 01c2cab0-6c0b-11e3-8300-975f903ccf5a
for range (-9223372036854775808,-3074457345618258603] finished
{noformat}
> Avoid repairing already-repaired data by default
>
>
> Key: CASSANDRA-5351
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5351
> Project: Cassandra
> Issue Type: Task
> Components: Core
>Reporter: Jonathan Ellis
>Assignee: Lyuben Todorov
> Labels: repair
> Fix For: 2.1
>
> Attachments: node1.log, node1_v2_full.log, node2.log,
> node2_v2_full.log, node3.log, node3_v2_full.log
>
>
> Repair has always built its merkle tree from all the data in a columnfamily,
> which is guaranteed to work but is inefficient.
> We can improve this by remembering which sstables have already been
> successfully repaired, and only repairing sstables new since the last repair.
> (This automatically makes CASSANDRA-3362 much less of a problem too.)
> The tricky part is, compaction will (if not taught otherwise) mix repaired
> data together with non-repaired. So we should segregate unrepaired sstables
> from the repaired ones.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (CASSANDRA-6523) "Unable to contact any seeds!" with multi-DC cluster and listen != broadcast address
Chris Burroughs created CASSANDRA-6523: -- Summary: "Unable to contact any seeds!" with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with "Unable to contact any seeds!" * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6199) Improve Stress Tool
[
https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855871#comment-13855871
]
Pavel Yaskevich commented on CASSANDRA-6199:
I finally had time to look into this and everything looks good, here are my
comments:
*1a1140e625db409eb2f51f8fac10bb1bd9ee89fb*
_CqlMultiGetter.java_
Has a left over "public void run(Cassandra.Client client) throws IOException"
which should be removed.
_JavaDriverClient.java_
needs formatting of {}.
_SmartThriftClient.java_
- get(ByteBuffer pk)
Current logic for new cache per host could be replaced with following to avoid
double get on map.
if (q == null)
{
ConcurrentLinkedQueue newQ = new ConcurrentLinkedQueue();
q = cache.putIfAbsent(host, newQ);
if (q == null)
q = newQ;
}
*e1a0be5ce00fb67bc6a37c9c3b695b6d7b4473c6*
_JavaDriverClient.java_
if cluster.shutdown() is a Future, you can uses
Futures.getUnchecked(cluster.shutdown()); instead of converting checked
exceptions into RE
> Improve Stress Tool
> ---
>
> Key: CASSANDRA-6199
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6199
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Benedict
>Assignee: Benedict
>Priority: Minor
> Attachments: new.read.latency.svg, new.read.rate.distribution.svg,
> new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg,
> old.read.rate.distribution.svg, old.write.latency.svg,
> old.write.rate.distribution.svg, ops.read.svg, ops.write.svg
>
>
> The stress tool could do with sprucing up. The following is a list of
> essential improvements and things that would be nice to have.
> Essential:
> - Reduce variability of results, especially start/end tails. Do not trash
> first/last 10% of readings
> - Reduce contention/overhead in stress to increase overall throughput
> - Short warm-up period, which is ignored for summary (or summarised
> separately), though prints progress as usual. Potentially automatic detection
> of rate levelling.
> - Better configurability and defaults for data generation - current column
> generation populates columns with the same value for every row, which is very
> easily compressible. Possibly introduce partial random data generator
> (possibly dictionary-based random data generator)
> Nice to have:
> - Calculate and print stdev and mean
> - Add batched sequential access mode (where a single thread performs
> batch-size sequential requests before selecting another random key) to test
> how key proximity affects performance
> - Auto-mode which attempts to establish the maximum throughput rate, by
> varying the thread count (or otherwise gating the number of parallel
> requests) for some period, then configures rate limit or thread count to test
> performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained.
> - Auto-mode could have a target variance ratio for mean throughput and/or
> latency, and completes a test once this target is hit for x intervals
> - Fix key representation so independent of number of keys (possibly switch to
> 10 digit hex), and don't use String.format().getBytes() to construct it
> (expensive)
> Also, remove the skip-key setting, as it is currently ignored. Unless
> somebody knows the reason for it.
> - Fix latency stats
> - Read/write mode, with configurable recency-of-reads distribution
> - Add new exponential/extreme value distribution for value size, column count
> and recency-of-reads
> - Support more than 2^31 keys
> - Supports multiple concurrent stress inserts via key-offset parameter or
> similar
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-6199) Improve Stress Tool
[
https://issues.apache.org/jira/browse/CASSANDRA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855871#comment-13855871
]
Pavel Yaskevich edited comment on CASSANDRA-6199 at 12/23/13 7:28 PM:
--
I finally had time to look into this and everything looks good, here are my
comments:
*1a1140e625db409eb2f51f8fac10bb1bd9ee89fb*
_CqlMultiGetter.java_
Has a left over "public void run(Cassandra.Client client) throws IOException"
which should be removed.
_JavaDriverClient.java_
needs formatting of {}.
_SmartThriftClient.java_
- get(ByteBuffer pk)
Current logic for new cache per host could be replaced with following to avoid
double get on map.
{noformat}
if (q == null)
{
ConcurrentLinkedQueue newQ = new ConcurrentLinkedQueue();
q = cache.putIfAbsent(host, newQ);
if (q == null)
q = newQ;
}
{noformat}
*e1a0be5ce00fb67bc6a37c9c3b695b6d7b4473c6*
_JavaDriverClient.java_
if cluster.shutdown() is a Future, you can uses
Futures.getUnchecked(cluster.shutdown()); instead of converting checked
exceptions into RE
was (Author: xedin):
I finally had time to look into this and everything looks good, here are my
comments:
*1a1140e625db409eb2f51f8fac10bb1bd9ee89fb*
_CqlMultiGetter.java_
Has a left over "public void run(Cassandra.Client client) throws IOException"
which should be removed.
_JavaDriverClient.java_
needs formatting of {}.
_SmartThriftClient.java_
- get(ByteBuffer pk)
Current logic for new cache per host could be replaced with following to avoid
double get on map.
if (q == null)
{
ConcurrentLinkedQueue newQ = new ConcurrentLinkedQueue();
q = cache.putIfAbsent(host, newQ);
if (q == null)
q = newQ;
}
*e1a0be5ce00fb67bc6a37c9c3b695b6d7b4473c6*
_JavaDriverClient.java_
if cluster.shutdown() is a Future, you can uses
Futures.getUnchecked(cluster.shutdown()); instead of converting checked
exceptions into RE
> Improve Stress Tool
> ---
>
> Key: CASSANDRA-6199
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6199
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Benedict
>Assignee: Benedict
>Priority: Minor
> Attachments: new.read.latency.svg, new.read.rate.distribution.svg,
> new.write.latency.svg, new.write.rate.distribution.svg, old.read.latency.svg,
> old.read.rate.distribution.svg, old.write.latency.svg,
> old.write.rate.distribution.svg, ops.read.svg, ops.write.svg
>
>
> The stress tool could do with sprucing up. The following is a list of
> essential improvements and things that would be nice to have.
> Essential:
> - Reduce variability of results, especially start/end tails. Do not trash
> first/last 10% of readings
> - Reduce contention/overhead in stress to increase overall throughput
> - Short warm-up period, which is ignored for summary (or summarised
> separately), though prints progress as usual. Potentially automatic detection
> of rate levelling.
> - Better configurability and defaults for data generation - current column
> generation populates columns with the same value for every row, which is very
> easily compressible. Possibly introduce partial random data generator
> (possibly dictionary-based random data generator)
> Nice to have:
> - Calculate and print stdev and mean
> - Add batched sequential access mode (where a single thread performs
> batch-size sequential requests before selecting another random key) to test
> how key proximity affects performance
> - Auto-mode which attempts to establish the maximum throughput rate, by
> varying the thread count (or otherwise gating the number of parallel
> requests) for some period, then configures rate limit or thread count to test
> performance at e.g. 30%, 50%, 70%, 90%, 120%, 150% and unconstrained.
> - Auto-mode could have a target variance ratio for mean throughput and/or
> latency, and completes a test once this target is hit for x intervals
> - Fix key representation so independent of number of keys (possibly switch to
> 10 digit hex), and don't use String.format().getBytes() to construct it
> (expensive)
> Also, remove the skip-key setting, as it is currently ignored. Unless
> somebody knows the reason for it.
> - Fix latency stats
> - Read/write mode, with configurable recency-of-reads distribution
> - Add new exponential/extreme value distribution for value size, column count
> and recency-of-reads
> - Support more than 2^31 keys
> - Supports multiple concurrent stress inserts via key-offset parameter or
> similar
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[ https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855862#comment-13855862 ] Jonathan Ellis commented on CASSANDRA-5351: --- I don't see the error in node 2? We're probably going to need flush + compaction info as well as just the direct repair logging. > Avoid repairing already-repaired data by default > > > Key: CASSANDRA-5351 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5351 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jonathan Ellis >Assignee: Lyuben Todorov > Labels: repair > Fix For: 2.1 > > Attachments: node1.log, node2.log, node3.log > > > Repair has always built its merkle tree from all the data in a columnfamily, > which is guaranteed to work but is inefficient. > We can improve this by remembering which sstables have already been > successfully repaired, and only repairing sstables new since the last repair. > (This automatically makes CASSANDRA-3362 much less of a problem too.) > The tricky part is, compaction will (if not taught otherwise) mix repaired > data together with non-repaired. So we should segregate unrepaired sstables > from the repaired ones. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[ https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lyuben Todorov updated CASSANDRA-5351: -- Attachment: node3.log node2.log node1.log I've attached the logs from the 3 nodes (only the parts generated from repair) logs from node2 and 3 are the interesting ones (log1 is info only). The repair was first initiated on node 3 and completed sucessfully, but then when launched for node2 stalled after the first repair session completed at the assertion @ DataTracker#newSSTables > Avoid repairing already-repaired data by default > > > Key: CASSANDRA-5351 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5351 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jonathan Ellis >Assignee: Lyuben Todorov > Labels: repair > Fix For: 2.1 > > Attachments: node1.log, node2.log, node3.log > > > Repair has always built its merkle tree from all the data in a columnfamily, > which is guaranteed to work but is inefficient. > We can improve this by remembering which sstables have already been > successfully repaired, and only repairing sstables new since the last repair. > (This automatically makes CASSANDRA-3362 much less of a problem too.) > The tricky part is, compaction will (if not taught otherwise) mix repaired > data together with non-repaired. So we should segregate unrepaired sstables > from the repaired ones. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5220) Repair improvements when using vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855774#comment-13855774
]
Donald Smith edited comment on CASSANDRA-5220 at 12/23/13 6:21 PM:
---
We ran "nodetool repair" on a 3 node cassandra cluster with production-quality
hardware, using version 2.0.3. Each node had about 1TB of data. This is still
testing. After 5 days the repair job still hasn't finished. I can see it's
still running.
Here's the process:
{noformat}
root 30835 30774 0 Dec17 pts/000:03:53 /usr/bin/java -cp
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.3.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/thrift-server-0.3.2.jar
-Xmx32m -Dlog4j.configuration=log4j-tools.properties
-Dstorage-config=/etc/cassandra/conf org.apache.cassandra.tools.NodeCmd -p 7199
repair -pr as_reports
{noformat}
The log output has just:
{noformat}
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
[2013-12-17 23:26:48,144] Starting repair command #1, repairing 256 ranges for
keyspace as_reports
{noformat}
Here's the output of "nodetool tpstats":
{noformat}
cass3 /tmp> nodetool tpstats
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 1 0 38083403 0
0
RequestResponseStage 0 0 1951200451 0
0
MutationStage 0 0 2853354069 0
0
ReadRepairStage 0 03794926 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 04880147 0
0
AntiEntropyStage 1 3 9 0
0
MigrationStage0 0 30 0
0
MemoryMeter 0 0115 0
0
MemtablePostFlusher 0 0 75121 0
0
FlushWriter 0 0 49934 0
52
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 7 0
0
commitlog_archiver0 0 0 0
0
AntiEntropySessions 1 1 1 0
0
InternalResponseStage 0 0 9 0
0
HintedHandoff 0 0 1141 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 884
MUTATION 1407711
_TRACE 0
REQUEST_RESPONSE 0
{noformat}
The cluster has some write traffic to it. We decided to test it under load.
This is the busiest c
[jira] [Commented] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[ https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855778#comment-13855778 ] Donald Smith commented on CASSANDRA-5351: - As reported in https://issues.apache.org/jira/browse/CASSANDRA-5220, we ran "nodetool repair" on a 3-node cassandra v. 2.0.3 cluster using production hardware, on realistic test data. Each node has about 1TB of data. After over five days, the repair job is still running. > Avoid repairing already-repaired data by default > > > Key: CASSANDRA-5351 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5351 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jonathan Ellis >Assignee: Lyuben Todorov > Labels: repair > Fix For: 2.1 > > > Repair has always built its merkle tree from all the data in a columnfamily, > which is guaranteed to work but is inefficient. > We can improve this by remembering which sstables have already been > successfully repaired, and only repairing sstables new since the last repair. > (This automatically makes CASSANDRA-3362 much less of a problem too.) > The tricky part is, compaction will (if not taught otherwise) mix repaired > data together with non-repaired. So we should segregate unrepaired sstables > from the repaired ones. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5220) Repair improvements when using vnodes
[
https://issues.apache.org/jira/browse/CASSANDRA-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855774#comment-13855774
]
Donald Smith commented on CASSANDRA-5220:
-
We ran "nodetool repair" on a 3 node cassandra cluster with production-quality
hardware, using version 2.0.3. Each node had about 1TB of data. This is still
testing. After 5 days the repair job still hasn't finished. I can see it's
still running.
Here's the process:
{noformat}
root 30835 30774 0 Dec17 pts/000:03:53 /usr/bin/java -cp
/etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.3.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.3.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/thrift-server-0.3.2.jar
-Xmx32m -Dlog4j.configuration=log4j-tools.properties
-Dstorage-config=/etc/cassandra/conf org.apache.cassandra.tools.NodeCmd -p 7199
repair -pr as_reports
{noformat}
The log output has just:
{noformat}
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
[2013-12-17 23:26:48,144] Starting repair command #1, repairing 256 ranges for
keyspace as_reports
{noformat}
Here's the output of "nodetool tpstats":
{noformat}
cass3 /tmp> nodetool tpstats
xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 1 0 38083403 0
0
RequestResponseStage 0 0 1951200451 0
0
MutationStage 0 0 2853354069 0
0
ReadRepairStage 0 03794926 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 04880147 0
0
AntiEntropyStage 1 3 9 0
0
MigrationStage0 0 30 0
0
MemoryMeter 0 0115 0
0
MemtablePostFlusher 0 0 75121 0
0
FlushWriter 0 0 49934 0
52
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 7 0
0
commitlog_archiver0 0 0 0
0
AntiEntropySessions 1 1 1 0
0
InternalResponseStage 0 0 9 0
0
HintedHandoff 0 0 1141 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 884
MUTATION 1407711
_TRACE 0
REQUEST_RESPONSE 0
{noformat}
The cluster has some write traffic to it. We decided to test it under load.
This is the busiest column family, as reported by "nodetool cfstats":
{n
[jira] [Updated] (CASSANDRA-6210) Repair hangs when a new datacenter is added to a cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuki Morishita updated CASSANDRA-6210:
--
Fix Version/s: 2.0.4
> Repair hangs when a new datacenter is added to a cluster
>
>
> Key: CASSANDRA-6210
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6210
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Amazon Ec2
> 2 M1.large nodes
>Reporter: Russell Alexander Spitzer
>Assignee: Yuki Morishita
> Fix For: 2.0.4
>
> Attachments: RepairLogs.tar.gz
>
>
> Attempting to add a new datacenter to a cluster seems to cause repair
> operations to break. I've been reproducing this with 20~ node clusters but
> can get it to reliably occur on 2 node setups.
> {code}
> ##Basic Steps to reproduce
> #Node 1 is started using GossipingPropertyFileSnitch as dc1
> #Cassandra-stress is used to insert a minimal amount of data
> $CASSANDRA_STRESS -t 100 -R
> org.apache.cassandra.locator.NetworkTopologyStrategy --num-keys=1000
> --columns=10 --consistency-level=LOCAL_QUORUM --average-size-values -
> -compaction-strategy='LeveledCompactionStrategy' -O dc1:1
> --operation=COUNTER_ADD
> #Alter "Keyspace1"
> ALTER KEYSPACE "Keyspace1" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': 1 , 'dc2': 1 };
> #Add node 2 using GossipingPropertyFileSnitch as dc2
> run repair on node 1
> run repair on node 2
> {code}
> The repair task on node 1 never completes and while there are no exceptions
> in the logs of node1, netstat reports the following repair tasks
> {code}
> Mode: NORMAL
> Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab
> Repair 6c64ded0-36b4-11e3-bedc-1d1bb5c9abab
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive Pending Completed
> Commandsn/a 0 10239
> Responses n/a 0 3839
> {code}
> Checking on node 2 we see the following exceptions
> {code}
> ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:42:58,961 StreamSession.java
> (line 410) [Stream #4e71a250-36b4-11e3-bedc-1d1bb5c9abab] Streaming error
> occurred
> java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:724)
> ...
> ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:43:49,214 StreamSession.java
> (line 410) [Stream #6c64ded0-36b4-11e3-bedc-1d1bb5c9abab] Streaming error
> occurred
> java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:724)
> {code}
> Netstats on node 2 reports
> {code}
> automaton@ip-10-171-15-234:~$ nodetool netstats
> Mode: NORMAL
> Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive Pending Completed
> Commandsn/a 0 2562
> Responses n/a 0 4284
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6210) Repair hangs when a new datacenter is added to a cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuki Morishita updated CASSANDRA-6210:
--
Since Version: 2.0.0
> Repair hangs when a new datacenter is added to a cluster
>
>
> Key: CASSANDRA-6210
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6210
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Amazon Ec2
> 2 M1.large nodes
>Reporter: Russell Alexander Spitzer
>Assignee: Yuki Morishita
> Fix For: 2.0.4
>
> Attachments: RepairLogs.tar.gz
>
>
> Attempting to add a new datacenter to a cluster seems to cause repair
> operations to break. I've been reproducing this with 20~ node clusters but
> can get it to reliably occur on 2 node setups.
> {code}
> ##Basic Steps to reproduce
> #Node 1 is started using GossipingPropertyFileSnitch as dc1
> #Cassandra-stress is used to insert a minimal amount of data
> $CASSANDRA_STRESS -t 100 -R
> org.apache.cassandra.locator.NetworkTopologyStrategy --num-keys=1000
> --columns=10 --consistency-level=LOCAL_QUORUM --average-size-values -
> -compaction-strategy='LeveledCompactionStrategy' -O dc1:1
> --operation=COUNTER_ADD
> #Alter "Keyspace1"
> ALTER KEYSPACE "Keyspace1" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': 1 , 'dc2': 1 };
> #Add node 2 using GossipingPropertyFileSnitch as dc2
> run repair on node 1
> run repair on node 2
> {code}
> The repair task on node 1 never completes and while there are no exceptions
> in the logs of node1, netstat reports the following repair tasks
> {code}
> Mode: NORMAL
> Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab
> Repair 6c64ded0-36b4-11e3-bedc-1d1bb5c9abab
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive Pending Completed
> Commandsn/a 0 10239
> Responses n/a 0 3839
> {code}
> Checking on node 2 we see the following exceptions
> {code}
> ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:42:58,961 StreamSession.java
> (line 410) [Stream #4e71a250-36b4-11e3-bedc-1d1bb5c9abab] Streaming error
> occurred
> java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:724)
> ...
> ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:43:49,214 StreamSession.java
> (line 410) [Stream #6c64ded0-36b4-11e3-bedc-1d1bb5c9abab] Streaming error
> occurred
> java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:724)
> {code}
> Netstats on node 2 reports
> {code}
> automaton@ip-10-171-15-234:~$ nodetool netstats
> Mode: NORMAL
> Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive Pending Completed
> Commandsn/a 0 2562
> Responses n/a 0 4284
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6210) Repair hangs when a new datacenter is added to a cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855702#comment-13855702
]
Yuki Morishita commented on CASSANDRA-6210:
---
Thanks Russell.
>From the log, I can see streaming session on receiver side did not set up
>correctly, though sender thought we were ready. Sender wrote stream init
>message through SocketChannel API but the message was not sent/received.
I think there are two ways to fix this. 1) Sender waits for ACK from receiver
or 2) Receiver holds response to sender until ready.
I'm tend to fix this using 2) rather than add another message for ACK, since
the next message sender would send is only one stream PREPARE message.
Let me work on that patch as well as to fix AE I encountered above.
> Repair hangs when a new datacenter is added to a cluster
>
>
> Key: CASSANDRA-6210
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6210
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Amazon Ec2
> 2 M1.large nodes
>Reporter: Russell Alexander Spitzer
>Assignee: Yuki Morishita
> Attachments: RepairLogs.tar.gz
>
>
> Attempting to add a new datacenter to a cluster seems to cause repair
> operations to break. I've been reproducing this with 20~ node clusters but
> can get it to reliably occur on 2 node setups.
> {code}
> ##Basic Steps to reproduce
> #Node 1 is started using GossipingPropertyFileSnitch as dc1
> #Cassandra-stress is used to insert a minimal amount of data
> $CASSANDRA_STRESS -t 100 -R
> org.apache.cassandra.locator.NetworkTopologyStrategy --num-keys=1000
> --columns=10 --consistency-level=LOCAL_QUORUM --average-size-values -
> -compaction-strategy='LeveledCompactionStrategy' -O dc1:1
> --operation=COUNTER_ADD
> #Alter "Keyspace1"
> ALTER KEYSPACE "Keyspace1" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': 1 , 'dc2': 1 };
> #Add node 2 using GossipingPropertyFileSnitch as dc2
> run repair on node 1
> run repair on node 2
> {code}
> The repair task on node 1 never completes and while there are no exceptions
> in the logs of node1, netstat reports the following repair tasks
> {code}
> Mode: NORMAL
> Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab
> Repair 6c64ded0-36b4-11e3-bedc-1d1bb5c9abab
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive Pending Completed
> Commandsn/a 0 10239
> Responses n/a 0 3839
> {code}
> Checking on node 2 we see the following exceptions
> {code}
> ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:42:58,961 StreamSession.java
> (line 410) [Stream #4e71a250-36b4-11e3-bedc-1d1bb5c9abab] Streaming error
> occurred
> java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:724)
> ...
> ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:43:49,214 StreamSession.java
> (line 410) [Stream #6c64ded0-36b4-11e3-bedc-1d1bb5c9abab] Streaming error
> occurred
> java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174)
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293)
> at java.lang.Thread.run(Thread.java:724)
> {code}
> Netstats on node 2 reports
> {code}
> automaton@ip-10-171-15-234:~$ nodetool netstats
> Mode: NORMAL
> Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool NameActive Pending Completed
> Commandsn/a 0 2562
> Responses n/a 0 4284
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6421) Add bash completion to nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6421: -- Reviewer: Michael Shuler (was: Lyuben Todorov) > Add bash completion to nodetool > --- > > Key: CASSANDRA-6421 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6421 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Cyril Scetbon >Assignee: Cyril Scetbon >Priority: Trivial > Fix For: 2.0.5 > > > You can find the bash-completion file at > https://raw.github.com/cscetbon/cassandra/nodetool-completion/etc/bash_completion.d/nodetool > it uses cqlsh to get keyspaces and namespaces and could use an environment > variable (not implemented) to get access which cqlsh if authentification is > needed. But I think that's really a good start :) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5895) JDK7 u40 stack size fixes
[
https://issues.apache.org/jira/browse/CASSANDRA-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821089#comment-13821089
]
Cyril Scetbon edited comment on CASSANDRA-5895 at 12/23/13 3:30 PM:
It has been applied in cassandra-1.2 branch on conf/cassandra-env.sh but not on
test/cassandra.in.sh. You should also update comment in conf/cassandra-env.sh
from
{code}# u34 and greater need 180k{code}
to
{code}# u45 and greater need 256k{code}
was (Author: cscetbon):
It has been applied in cassandra-1.2 branch on conf/cassandra-env.sh but not on
test/cassandra.in.sh. You should also update comment in conf/cassandra-env.sh
from
\# u34 and greater need 180k
to
\# u45 and greater need 256k
> JDK7 u40 stack size fixes
> -
>
> Key: CASSANDRA-5895
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5895
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: cassandra, as launched by the default script by JDK
> 1.7.0_40 using CCM
>Reporter: John Sumsion
> Attachments:
> 0001-Increasing-stack-size-to-avoid-error-on-OpenJDK-7-u4.patch,
> 0002-Ignoring-files-produced-by-ant-build.patch
>
>
> I use Archlinux, and the latest OpenJDK, and run my cassandra via @pcmanus's
> ccm.
> When I tried the cassandra-2.0.0 branch via:
> {noformat}
> $ ccm create dev -v git:cassandra-2.0.0
> {noformat}
> Then when I say:
> {noformat}
> $ ccm populate -n3
> $ ccm node1 start -v
> {noformat}
> I get the following error:
> {noformat}
> xss = -ea -javaagent:/home/blah/.ccm/repository/1.2.2/lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1985M -Xmx1985M
> -Xmn496M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
> The stack size specified is too small, Specify at least 228k
> Error starting node node1
> Standard error output is:
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {noformat}
> I tracked it down to conf/cassandra-env.sh, and changed the -Xss180k =>
> -Xss228k and the node started.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5895) JDK7 u40 stack size fixes
[
https://issues.apache.org/jira/browse/CASSANDRA-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821089#comment-13821089
]
Cyril Scetbon edited comment on CASSANDRA-5895 at 12/23/13 3:29 PM:
It has been applied in cassandra-1.2 branch on conf/cassandra-env.sh but not on
test/cassandra.in.sh. You should also update comment in conf/cassandra-env.sh
from
\# u34 and greater need 180k
to
\# u45 and greater need 256k
was (Author: cscetbon):
It has been applied in cassandra-1.2 branch on conf/cassandra-env.sh but not on
test/cassandra.in.sh. You should also update comment in conf/cassandra-env.sh
from
# u34 and greater need 180k
to
# u45 and greater need 256k
> JDK7 u40 stack size fixes
> -
>
> Key: CASSANDRA-5895
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5895
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: cassandra, as launched by the default script by JDK
> 1.7.0_40 using CCM
>Reporter: John Sumsion
> Attachments:
> 0001-Increasing-stack-size-to-avoid-error-on-OpenJDK-7-u4.patch,
> 0002-Ignoring-files-produced-by-ant-build.patch
>
>
> I use Archlinux, and the latest OpenJDK, and run my cassandra via @pcmanus's
> ccm.
> When I tried the cassandra-2.0.0 branch via:
> {noformat}
> $ ccm create dev -v git:cassandra-2.0.0
> {noformat}
> Then when I say:
> {noformat}
> $ ccm populate -n3
> $ ccm node1 start -v
> {noformat}
> I get the following error:
> {noformat}
> xss = -ea -javaagent:/home/blah/.ccm/repository/1.2.2/lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1985M -Xmx1985M
> -Xmn496M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
> The stack size specified is too small, Specify at least 228k
> Error starting node node1
> Standard error output is:
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {noformat}
> I tracked it down to conf/cassandra-env.sh, and changed the -Xss180k =>
> -Xss228k and the node started.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5895) JDK7 u40 stack size fixes
[
https://issues.apache.org/jira/browse/CASSANDRA-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821089#comment-13821089
]
Cyril Scetbon edited comment on CASSANDRA-5895 at 12/23/13 3:29 PM:
It has been applied in cassandra-1.2 branch on conf/cassandra-env.sh but not on
test/cassandra.in.sh. You should also update comment in conf/cassandra-env.sh
from
# u34 and greater need 180k
to
# u45 and greater need 256k
was (Author: cscetbon):
Hi, can someone add it to the trunk of cassandra 1.2 ? it's really a minor but
needed patch.
> JDK7 u40 stack size fixes
> -
>
> Key: CASSANDRA-5895
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5895
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: cassandra, as launched by the default script by JDK
> 1.7.0_40 using CCM
>Reporter: John Sumsion
> Attachments:
> 0001-Increasing-stack-size-to-avoid-error-on-OpenJDK-7-u4.patch,
> 0002-Ignoring-files-produced-by-ant-build.patch
>
>
> I use Archlinux, and the latest OpenJDK, and run my cassandra via @pcmanus's
> ccm.
> When I tried the cassandra-2.0.0 branch via:
> {noformat}
> $ ccm create dev -v git:cassandra-2.0.0
> {noformat}
> Then when I say:
> {noformat}
> $ ccm populate -n3
> $ ccm node1 start -v
> {noformat}
> I get the following error:
> {noformat}
> xss = -ea -javaagent:/home/blah/.ccm/repository/1.2.2/lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1985M -Xmx1985M
> -Xmn496M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
> The stack size specified is too small, Specify at least 228k
> Error starting node node1
> Standard error output is:
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {noformat}
> I tracked it down to conf/cassandra-env.sh, and changed the -Xss180k =>
> -Xss228k and the node started.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6421) Add bash completion to nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855675#comment-13855675 ] Cyril Scetbon commented on CASSANDRA-6421: -- Now completion is provided for rebuild command and propose names of datacenters > Add bash completion to nodetool > --- > > Key: CASSANDRA-6421 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6421 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Cyril Scetbon >Assignee: Cyril Scetbon >Priority: Trivial > Fix For: 2.0.5 > > > You can find the bash-completion file at > https://raw.github.com/cscetbon/cassandra/nodetool-completion/etc/bash_completion.d/nodetool > it uses cqlsh to get keyspaces and namespaces and could use an environment > variable (not implemented) to get access which cqlsh if authentification is > needed. But I think that's really a good start :) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4914) Aggregate functions in CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855671#comment-13855671 ] Pablo Chacin commented on CASSANDRA-4914: - [~iamaleksey] Certainly, I was completely missing the need for reconciling potentially inconsistent replicas! > Aggregate functions in CQL > -- > > Key: CASSANDRA-4914 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4914 > Project: Cassandra > Issue Type: New Feature >Reporter: Vijay >Assignee: Vijay > Fix For: 2.1 > > > The requirement is to do aggregation of data in Cassandra (Wide row of column > values of int, double, float etc). > With some basic agree gate functions like AVG, SUM, Mean, Min, Max, etc (for > the columns within a row). > Example: > SELECT * FROM emp WHERE empID IN (130) ORDER BY deptID DESC; > > empid | deptid | first_name | last_name | salary > ---+++---+ >130 | 3 | joe| doe | 10.1 >130 | 2 | joe| doe |100 >130 | 1 | joe| doe | 1e+03 > > SELECT sum(salary), empid FROM emp WHERE empID IN (130); > > sum(salary) | empid > -+ >1110.1| 130 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6421) Add bash completion to nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyril Scetbon updated CASSANDRA-6421: - Description: You can find the bash-completion file at https://raw.github.com/cscetbon/cassandra/nodetool-completion/etc/bash_completion.d/nodetool it uses cqlsh to get keyspaces and namespaces and could use an environment variable (not implemented) to get access which cqlsh if authentification is needed. But I think that's really a good start :) was: You can find the patch from my commit here : https://github.com/cscetbon/cassandra/commit/07a10b99778f14362ac05c70269c108870555bf3.patch it uses cqlsh to get keyspaces and namespaces and could use an environment variable (not implemented) to get access which cqlsh if authentification is needed. But I think that's really a good start :) > Add bash completion to nodetool > --- > > Key: CASSANDRA-6421 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6421 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Cyril Scetbon >Assignee: Cyril Scetbon >Priority: Trivial > Fix For: 2.0.5 > > > You can find the bash-completion file at > https://raw.github.com/cscetbon/cassandra/nodetool-completion/etc/bash_completion.d/nodetool > it uses cqlsh to get keyspaces and namespaces and could use an environment > variable (not implemented) to get access which cqlsh if authentification is > needed. But I think that's really a good start :) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855628#comment-13855628 ] Radovan Zvoncek commented on CASSANDRA-4165: So I used Marcuse's patch as a basis and added checks preventing computation/writing of CRC components for compressed files. I'm still worried a bit about backwards compatibility though. I did some experiments and didn't encounter any problems. However, I can't fully comprehend the impact of this change and might have missed something. > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radovan Zvoncek updated CASSANDRA-4165: --- Attachment: 0002-dont-do-crc-and-add-digests-for-compressed-files.txt > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4914) Aggregate functions in CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855583#comment-13855583 ] Aleksey Yeschenko commented on CASSANDRA-4914: -- bq. In the case of Cassandra, however, one mayor concern would be that for partitionable functions like sum, count, min or max, each node could do its part of aggregation. For non-partitionable function like average or percentiles, all the aggregation must be done at the coordinator. This is not true at any RF > 1. The coordinator has to reconcile the partitions before doing any kind of aggregation on them. > Aggregate functions in CQL > -- > > Key: CASSANDRA-4914 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4914 > Project: Cassandra > Issue Type: New Feature >Reporter: Vijay >Assignee: Vijay > Fix For: 2.1 > > > The requirement is to do aggregation of data in Cassandra (Wide row of column > values of int, double, float etc). > With some basic agree gate functions like AVG, SUM, Mean, Min, Max, etc (for > the columns within a row). > Example: > SELECT * FROM emp WHERE empID IN (130) ORDER BY deptID DESC; > > empid | deptid | first_name | last_name | salary > ---+++---+ >130 | 3 | joe| doe | 10.1 >130 | 2 | joe| doe |100 >130 | 1 | joe| doe | 1e+03 > > SELECT sum(salary), empid FROM emp WHERE empID IN (130); > > sum(salary) | empid > -+ >1110.1| 130 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-4914) Aggregate functions in CQL
[
https://issues.apache.org/jira/browse/CASSANDRA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855523#comment-13855523
]
Pablo Chacin edited comment on CASSANDRA-4914 at 12/23/13 9:39 AM:
---
{quote} the aggregate function would be iteratively called on each grouping. Is
that accurate?
Exactly. Basically I'm suggesting that the grouping itself can be done
separately by the Selection object since it's completely generic, it doesn't
depend on which aggregate function you'll execute.
{quote}
[~slebresne] I completely agree with this approach. Actually, I used this
pattern, which is basically a visitor pattern, some time ago when implementing
a (small scale) time series plotting framework and it works nicely. You have a
lot of freedom on how you traverse data (e.g. grouping) with a generic set of
functions.
In the case of Cassandra, however, one mayor concern would be that for
partitionable functions like sum, count, min or max, each node could do its
part of aggregation. For non-partitionable function like average or
percentiles, all the aggregation must be done at the coordinator.
was (Author: pablochacin):
{quote} the aggregate function would be iteratively called on each grouping. Is
that accurate?
Exactly. Basically I'm suggesting that the grouping itself can be done
separately by the Selection object since it's completely generic, it doesn't
depend on which aggregate function you'll execute.
{quote}
[~Sylvain Lebresne] I completely agree with this approach. Actually, I used
this pattern, which is basically a visitor pattern, some time ago when
implementing a (small scale) time series plotting framework and it works
nicely. You have a lot of freedom on how you traverse data (e.g. grouping) with
a generic set of functions.
In the case of Cassandra, however, one mayor concern would be that for
partitionable functions like sum, count, min or max, each node could do its
part of aggregation. For non-partitionable function like average or
percentiles, all the aggregation must be done at the coordinator.
> Aggregate functions in CQL
> --
>
> Key: CASSANDRA-4914
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4914
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Vijay
>Assignee: Vijay
> Fix For: 2.1
>
>
> The requirement is to do aggregation of data in Cassandra (Wide row of column
> values of int, double, float etc).
> With some basic agree gate functions like AVG, SUM, Mean, Min, Max, etc (for
> the columns within a row).
> Example:
> SELECT * FROM emp WHERE empID IN (130) ORDER BY deptID DESC;
>
> empid | deptid | first_name | last_name | salary
> ---+++---+
>130 | 3 | joe| doe | 10.1
>130 | 2 | joe| doe |100
>130 | 1 | joe| doe | 1e+03
>
> SELECT sum(salary), empid FROM emp WHERE empID IN (130);
>
> sum(salary) | empid
> -+
>1110.1| 130
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4914) Aggregate functions in CQL
[
https://issues.apache.org/jira/browse/CASSANDRA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855523#comment-13855523
]
Pablo Chacin commented on CASSANDRA-4914:
-
{quote} the aggregate function would be iteratively called on each grouping. Is
that accurate?
Exactly. Basically I'm suggesting that the grouping itself can be done
separately by the Selection object since it's completely generic, it doesn't
depend on which aggregate function you'll execute.
{quote}
[~Sylvain Lebresne] I completely agree with this approach. Actually, I used
this pattern, which is basically a visitor pattern, some time ago when
implementing a (small scale) time series plotting framework and it works
nicely. You have a lot of freedom on how you traverse data (e.g. grouping) with
a generic set of functions.
In the case of Cassandra, however, one mayor concern would be that for
partitionable functions like sum, count, min or max, each node could do its
part of aggregation. For non-partitionable function like average or
percentiles, all the aggregation must be done at the coordinator.
> Aggregate functions in CQL
> --
>
> Key: CASSANDRA-4914
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4914
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Vijay
>Assignee: Vijay
> Fix For: 2.1
>
>
> The requirement is to do aggregation of data in Cassandra (Wide row of column
> values of int, double, float etc).
> With some basic agree gate functions like AVG, SUM, Mean, Min, Max, etc (for
> the columns within a row).
> Example:
> SELECT * FROM emp WHERE empID IN (130) ORDER BY deptID DESC;
>
> empid | deptid | first_name | last_name | salary
> ---+++---+
>130 | 3 | joe| doe | 10.1
>130 | 2 | joe| doe |100
>130 | 1 | joe| doe | 1e+03
>
> SELECT sum(salary), empid FROM emp WHERE empID IN (130);
>
> sum(salary) | empid
> -+
>1110.1| 130
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
