date:20150206

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309625#comment-14309625
]

Ryan Svihla commented on CASSANDRA-8754:

Also in regards to the ponying of this. It may help to understand the all too
common situation I have to help people with. Education of CL doesn't spread
across diverse dev teams accessing the same cluster, and worse turnover is
often high. So all new code has the potential to invalidate company policy. If
we have some artifact that is reviewable in Cassandra, I can be certain quickly
the issue isn't in them misunderstanding of what a Consistency Level is..or
some rogue users logging in and doing stuff by hand.

If we're just relying on code, I have to comb through all code that touches the
database sever, and I have to be able to read said code, trust I've seen said
code to be able to diagnose the data isn't consistent problems. This can take
an enormous amount of time, when reviewing say a schema file would take 2
minutes.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

Idea is to prevent a query based on a consistency level not being met. For
example we can specify that all queries should be at least CL LOCAL_QUORUM.
Lots of users struggle with getting all their dev teams on board with
consistency levels and all the ramifications. The normal solution for this
has traditionally to build a service in front of Cassandra that the entire
dev team accesses. However, this has proven challenging for some
organizations to do correctly, and I think an easier approach would be to
require a given consistency level as a matter of enforced policy in the
database.
I'm open for where this belongs. The most flexible approach is at a table
level, however I'm concerned this is potentially error prone and labor
intensive. It could be a table attribute similar to compaction strategy.
The simplest administratively is a cluster level, in say the cassandra.yaml
The middle ground is at they keyspace level, the only downside I could
foresee is keyspace explosion to fit involved minimum schemes. It could be a
keyspace attribute such as replication strategy.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309622#comment-14309622
]

Aleksey Yeschenko commented on CASSANDRA-8754:
--

Sounds reasonable to me. I'd go for a per-table blacklist, however, just for
implementation reasons.

Alternatively, we could use the capability-limitation framework that we've
discussed before, if that feels like a better fit (and right now it doesn't to
me).

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309623#comment-14309623
]

Brandon Williams commented on CASSANDRA-8754:
-

They can't be trusted to use consistency levels properly, but they can be
trusted with a super user?

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309652#comment-14309652
]

Ryan Svihla commented on CASSANDRA-8754:

Then their table will be changed..and I'll have a schema description to
indicate such

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309651#comment-14309651
]

Ryan Svihla commented on CASSANDRA-8754:

Then their table will be changed..and I'll have a schema description to
indicate such

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer

2015-02-06 Thread Akshat Aranya (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309717#comment-14309717
 ] 

Akshat Aranya commented on CASSANDRA-8723:
--

We have a Cassandra 2.1 cluster running where the Cassandra Java processes are 
running very high on VIRT memory size:

from top:
{code}
 20797 cass_ic_  18  -2  384g 9.2g  24m S 1947.9  7.3 455:06.82 java
{code}



 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
 /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
  -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof 
 -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log 
 org.apache.cassandra.service.CassandraDaemon
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309751#comment-14309751
]

Ryan Svihla commented on CASSANDRA-8754:

Auditing would help with diagnosis, but it'd have to be cheap enough to turn it
on. I still think there is a huge value for a number of users to have some
server side CL restriction, I've been asked personally more than a few times.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309641#comment-14309641
]

Brandon Williams commented on CASSANDRA-8754:
-

They can just use the super user to allow the CL then.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer

2015-02-06 Thread Alan Boudreault (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309685#comment-14309685
 ] 

Alan Boudreault commented on CASSANDRA-8723:


[~jeffl] Have you been able to reproduce it more than once or it can happen at 
any time after a node has been started ? Perhaps you can give us a detailed 
scenario of what you are doing.  All information is useful, distribution, java 
version etc. You are not the first user to report a similar issue and we would 
like very hard to reproduce it.

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
 /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
  -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof 
 -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log

[jira] [Updated] (CASSANDRA-8448) Comparison method violates its general contract in AbstractEndpointSnitch


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-8448:

Attachment: 8448-v3.txt

 Comparison method violates its general contract in AbstractEndpointSnitch
 ---

 Key: CASSANDRA-8448
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8448
 Project: Cassandra
  Issue Type: Bug
Reporter: J.B. Langston
Assignee: Brandon Williams
 Fix For: 2.1.3, 2.0.13

 Attachments: 8448-v2.txt, 8448-v3.txt, 8448.txt


 Seen in both 1.2 and 2.0.  The error is occurring here: 
 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java#L49
 {code}
 ERROR [Thrift:9] 2014-12-04 20:12:28,732 CustomTThreadPoolServer.java (line 
 219) Error occurred during processing of message.
 com.google.common.util.concurrent.UncheckedExecutionException: 
 java.lang.IllegalArgumentException: Comparison method violates its general 
 contract!
   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
   at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
   at 
 com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
   at 
 org.apache.cassandra.service.ClientState.authorize(ClientState.java:352)
   at 
 org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:224)
   at 
 org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:218)
   at 
 org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:202)
   at 
 org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:822)
   at 
 org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:954)
   at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:576)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3922)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3906)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.IllegalArgumentException: Comparison method violates its 
 general contract!
   at java.util.TimSort.mergeHi(TimSort.java:868)
   at java.util.TimSort.mergeAt(TimSort.java:485)
   at java.util.TimSort.mergeCollapse(TimSort.java:410)
   at java.util.TimSort.sort(TimSort.java:214)
   at java.util.TimSort.sort(TimSort.java:173)
   at java.util.Arrays.sort(Arrays.java:659)
   at java.util.Collections.sort(Collections.java:217)
   at 
 org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1408)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1402)
   at 
 org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:148)
   at 
 org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1223)
   at 
 org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1165)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:255)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:225)
   at org.apache.cassandra.auth.Auth.selectUser(Auth.java:243)
   at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84)
   at 
 org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
   at 
 org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:69)
   at org.apache.cassandra.service.ClientState$1.load(ClientState.java:338)
   at org.apache.cassandra.service.ClientState$1.load(ClientState.java:335)
   at

[jira] [Commented] (CASSANDRA-8754) Required consistency level

2015-02-06 Thread Jeremy Hanna (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309702#comment-14309702
]

Jeremy Hanna commented on CASSANDRA-8754:
-

I was thinking of auditing as well. You could definitively tell which CL has
been used for operations. Might be a lot of overhead though like tracing.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] cassandra git commit: Fix IllegalArgumentException in dynamic snitch

2015-02-06 Thread brandonwilliams

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.0 1467b9f78 - cf73de2dc
  refs/heads/cassandra-2.1 caba0a592 - 5e95684c9


Fix IllegalArgumentException in dynamic snitch

Patch by brandonwilliams, viewed by Benedict for CASSANDRA-8448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cf73de2d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cf73de2d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cf73de2d

Branch: refs/heads/cassandra-2.0
Commit: cf73de2dc99d4857cc4bac734cf48cf9a21aa9be
Parents: 1467b9f
Author: Brandon Williams brandonwilli...@apache.org
Authored: Fri Feb 6 13:23:05 2015 -0600
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Fri Feb 6 13:23:05 2015 -0600

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/locator/DynamicEndpointSnitch.java| 6 --
 2 files changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cf73de2d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index f4c96dc..fa9c77d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.13:
+ * Fix IllegalArgumentException in dynamic snitch (CASSANDRA-8448)
  * Add support for UPDATE ... IF EXISTS (CASSANDRA-8610)
  * Fix reversal of list prepends (CASSANDRA-8733)
  * Prevent non-zero default_time_to_live on tables with counters

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cf73de2d/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java 
b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 49442c8..3469847 100644
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@ -53,7 +53,7 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 private String mbeanName;
 private boolean registered = false;
 
-private final ConcurrentHashMapInetAddress, Double scores = new 
ConcurrentHashMapInetAddress, Double();
+private volatile HashMapInetAddress, Double scores = new 
HashMapInetAddress, Double();
 private final ConcurrentHashMapInetAddress, ExponentiallyDecayingSample 
samples = new ConcurrentHashMapInetAddress, ExponentiallyDecayingSample();
 
 public final IEndpointSnitch subsnitch;
@@ -243,6 +243,7 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 double maxLatency = 1;
 // We're going to weight the latency for each host against the worst 
one we see, to
 // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
+HashMapInetAddress, Double newScores = new HashMap();
 for (Map.EntryInetAddress, ExponentiallyDecayingSample entry : 
samples.entrySet())
 {
 double mean = entry.getValue().getSnapshot().getMedian();
@@ -257,8 +258,9 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 // Severity is basically a measure of compaction activity 
(CASSANDRA-3722).
 score += StorageService.instance.getSeverity(entry.getKey());
 // lowest score (least amount of badness) wins.
-scores.put(entry.getKey(), score);
+newScores.put(entry.getKey(), score);
 }
+scores = newScores;
 }

[3/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

2015-02-06 Thread brandonwilliams

Merge branch 'cassandra-2.0' into cassandra-2.1

Conflicts:
CHANGES.txt


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e95684c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e95684c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e95684c

Branch: refs/heads/cassandra-2.1
Commit: 5e95684c9c992c7eb3477972482abb1cd6216978
Parents: caba0a5 cf73de2
Author: Brandon Williams brandonwilli...@apache.org
Authored: Fri Feb 6 13:24:05 2015 -0600
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Fri Feb 6 13:24:05 2015 -0600

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/locator/DynamicEndpointSnitch.java| 6 --
 2 files changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e95684c/CHANGES.txt
--
diff --cc CHANGES.txt
index 959a2de,fa9c77d..a9e8c7c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,91 -1,5 +1,92 @@@
 -2.0.13:
 +2.1.3
 + * Write partition size estimates into a system table (CASSANDRA-7688)
 + * Upgrade libthrift to 0.9.2 (CASSANDRA-8685)
 + * Don't use the shared ref in sstableloader (CASSANDRA-8704)
 + * Purge internal prepared statements if related tables or
 +   keyspaces are dropped (CASSANDRA-8693)
 + * (cqlsh) Handle unicode BOM at start of files (CASSANDRA-8638)
 + * Stop compactions before exiting offline tools (CASSANDRA-8623)
 + * Update tools/stress/README.txt to match current behaviour (CASSANDRA-7933)
 + * Fix schema from Thrift conversion with empty metadata (CASSANDRA-8695)
 + * Safer Resource Management (CASSANDRA-7705)
 + * Make sure we compact highly overlapping cold sstables with
 +   STCS (CASSANDRA-8635)
 + * rpc_interface and listen_interface generate NPE on startup when specified
 +   interface doesn't exist (CASSANDRA-8677)
 + * Fix ArrayIndexOutOfBoundsException in nodetool cfhistograms 
(CASSANDRA-8514)
 + * Switch from yammer metrics for nodetool cf/proxy histograms 
(CASSANDRA-8662)
 + * Make sure we don't add tmplink files to the compaction
 +   strategy (CASSANDRA-8580)
 + * (cqlsh) Handle maps with blob keys (CASSANDRA-8372)
 + * (cqlsh) Handle DynamicCompositeType schemas correctly (CASSANDRA-8563)
 + * Duplicate rows returned when in clause has repeated values (CASSANDRA-6707)
 + * Add tooling to detect hot partitions (CASSANDRA-7974)
 + * Fix cassandra-stress user-mode truncation of partition generation 
(CASSANDRA-8608)
 + * Only stream from unrepaired sstables during inc repair (CASSANDRA-8267)
 + * Don't allow starting multiple inc repairs on the same sstables 
(CASSANDRA-8316)
 + * Invalidate prepared BATCH statements when related tables
 +   or keyspaces are dropped (CASSANDRA-8652)
 + * Fix missing results in secondary index queries on collections
 +   with ALLOW FILTERING (CASSANDRA-8421)
 + * Expose EstimatedHistogram metrics for range slices (CASSANDRA-8627)
 + * (cqlsh) Escape clqshrc passwords properly (CASSANDRA-8618)
 + * Fix NPE when passing wrong argument in ALTER TABLE statement 
(CASSANDRA-8355)
 + * Pig: Refactor and deprecate CqlStorage (CASSANDRA-8599)
 + * Don't reuse the same cleanup strategy for all sstables (CASSANDRA-8537)
 + * Fix case-sensitivity of index name on CREATE and DROP INDEX
 +   statements (CASSANDRA-8365)
 + * Better detection/logging for corruption in compressed sstables 
(CASSANDRA-8192)
 + * Use the correct repairedAt value when closing writer (CASSANDRA-8570)
 + * (cqlsh) Handle a schema mismatch being detected on startup (CASSANDRA-8512)
 + * Properly calculate expected write size during compaction (CASSANDRA-8532)
 + * Invalidate affected prepared statements when a table's columns
 +   are altered (CASSANDRA-7910)
 + * Stress - user defined writes should populate sequentally (CASSANDRA-8524)
 + * Fix regression in SSTableRewriter causing some rows to become unreadable 
 +   during compaction (CASSANDRA-8429)
 + * Run major compactions for repaired/unrepaired in parallel (CASSANDRA-8510)
 + * (cqlsh) Fix compression options in DESCRIBE TABLE output when compression
 +   is disabled (CASSANDRA-8288)
 + * (cqlsh) Fix DESCRIBE output after keyspaces are altered (CASSANDRA-7623)
 + * Make sure we set lastCompactedKey correctly (CASSANDRA-8463)
 + * (cqlsh) Fix output of CONSISTENCY command (CASSANDRA-8507)
 + * (cqlsh) Fixed the handling of LIST statements (CASSANDRA-8370)
 + * Make sstablescrub check leveled manifest again (CASSANDRA-8432)
 + * Check first/last keys in sstable when giving out positions (CASSANDRA-8458)
 + * Disable mmap on Windows (CASSANDRA-6993)
 + * Add missing ConsistencyLevels to cassandra-stress (CASSANDRA-8253)
 + * Add auth support to cassandra-stress (CASSANDRA-7985)
 + *

[2/3] cassandra git commit: Fix IllegalArgumentException in dynamic snitch

2015-02-06 Thread brandonwilliams

Fix IllegalArgumentException in dynamic snitch

Patch by brandonwilliams, viewed by Benedict for CASSANDRA-8448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cf73de2d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cf73de2d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cf73de2d

Branch: refs/heads/cassandra-2.1
Commit: cf73de2dc99d4857cc4bac734cf48cf9a21aa9be
Parents: 1467b9f
Author: Brandon Williams brandonwilli...@apache.org
Authored: Fri Feb 6 13:23:05 2015 -0600
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Fri Feb 6 13:23:05 2015 -0600

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/locator/DynamicEndpointSnitch.java| 6 --
 2 files changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cf73de2d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index f4c96dc..fa9c77d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.13:
+ * Fix IllegalArgumentException in dynamic snitch (CASSANDRA-8448)
  * Add support for UPDATE ... IF EXISTS (CASSANDRA-8610)
  * Fix reversal of list prepends (CASSANDRA-8733)
  * Prevent non-zero default_time_to_live on tables with counters

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cf73de2d/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java 
b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 49442c8..3469847 100644
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@ -53,7 +53,7 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 private String mbeanName;
 private boolean registered = false;
 
-private final ConcurrentHashMapInetAddress, Double scores = new 
ConcurrentHashMapInetAddress, Double();
+private volatile HashMapInetAddress, Double scores = new 
HashMapInetAddress, Double();
 private final ConcurrentHashMapInetAddress, ExponentiallyDecayingSample 
samples = new ConcurrentHashMapInetAddress, ExponentiallyDecayingSample();
 
 public final IEndpointSnitch subsnitch;
@@ -243,6 +243,7 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 double maxLatency = 1;
 // We're going to weight the latency for each host against the worst 
one we see, to
 // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
+HashMapInetAddress, Double newScores = new HashMap();
 for (Map.EntryInetAddress, ExponentiallyDecayingSample entry : 
samples.entrySet())
 {
 double mean = entry.getValue().getSnapshot().getMedian();
@@ -257,8 +258,9 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 // Severity is basically a measure of compaction activity 
(CASSANDRA-3722).
 score += StorageService.instance.getSeverity(entry.getKey());
 // lowest score (least amount of badness) wins.
-scores.put(entry.getKey(), score);
+newScores.put(entry.getKey(), score);
 }
+scores = newScores;
 }

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309627#comment-14309627
]

Ryan Svihla commented on CASSANDRA-8754:

People sneak it. Happened more than a few times. An artifact _helps_ immensily

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309806#comment-14309806
 ] 

Brandon Williams commented on CASSANDRA-8723:
-

That's normal.

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
 /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
  -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof 
 -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log 
 org.apache.cassandra.service.CassandraDaemon
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8730) Optimize UUIDType comparisons

2015-02-06 Thread J.B. Langston (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309901#comment-14309901
 ] 

J.B. Langston edited comment on CASSANDRA-8730 at 2/6/15 8:59 PM:
--

Looks like a bit of improvement: 12.63MB/sec vs 10.19MB/sec.  Looks like it 
threw away more data this time. I guess some tombstones passed gc grace since I 
last tested. Therefore, I'm not sure how apples-to-apples the comparison is, so 
I'm going to try again while setting my clock back to the date when I ran it 
before.

Before:

{code}
INFO 15:19:05 Compacted 4 sstables to 
[./../data/data/ocean/tbl_metric_data_dyn-0f578640a59211e4a5a2ef9f87394ca6/ocean-tbl_metric_data_dyn-ka-144263,].
 9,183,829,489 bytes to 9,180,536,394 (~99% of original) in 901,172ms = 
9.715395MB/s. 311,495 total partitions merged to 253,490. Partition merge 
counts were {1:195485, 2:58005, }
{code}

After:

{code}
INFO  20:47:24 Compacted 4 sstables to 
[./../data/data/ocean/tbl_metric_data_dyn-0f578640a59211e4a5a2ef9f87394ca6/ocean-tbl_metric_data_dyn-ka-144263,].
  8,152,562,772 bytes to 4,659,100,313 (~57% of original) in 615,577ms = 
7.218048MB/s.  311,495 total partitions merged to 80,012.  Partition merge 
counts were {1:195485, 2:58005, }
{code}


was (Author: jblangs...@datastax.com):
Looks like a bit of improvement: 12.63MB/sec vs 10.19MB/sec.  Looks like it 
threw away more data this time. I guess some tombstones passed gc grace since I 
last tested. Therefore, I'm not sure how apples-to-apples the comparison is, so 
I'm going to try again while setting my clock back to the date when I ran it 
before.

Before:

{code}
INFO 15:19:05 Compacted 4 sstables to 
[./../data/data/ocean/tbl_metric_data_dyn-0f578640a59211e4a5a2ef9f87394ca6/ocean-tbl_metric_data_dyn-ka-144263,].
 9,183,829,489 bytes to 9,180,536,394 (~99% of original) in 901,172ms = 
9.715395MB/s. 311,495 total partitions merged to 253,490. Partition merge 
counts were {1:195485, 2:58005, }
{code}

After:

{code}
INFO  20:47:24 Completed flushing 
/Users/jblangston/repos/cassandra/bin/./../data/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-48-Data.db
 (42 bytes) for commitlog position ReplayPosition(segmentId=1423254980101, 
position=758851)
INFO  20:47:24 Compacted 4 sstables to 
[./../data/data/ocean/tbl_metric_data_dyn-0f578640a59211e4a5a2ef9f87394ca6/ocean-tbl_metric_data_dyn-ka-144263,].
  8,152,562,772 bytes to 4,659,100,313 (~57% of original) in 615,577ms = 
7.218048MB/s.  311,495 total partitions merged to 80,012.  Partition merge 
counts were {1:195485, 2:58005, }
{code}

 Optimize UUIDType comparisons
 -

 Key: CASSANDRA-8730
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8730
 Project: Cassandra
  Issue Type: Improvement
Reporter: J.B. Langston
Assignee: Benedict
 Fix For: 2.1.4


 Compaction is slow on tables using compound keys containing UUIDs due to 
 being CPU bound by key comparison.  [~benedict] said he sees some easy 
 optimizations that could be made for UUID comparison.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY

2015-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-8535:
---
Attachment: 8535_v2.txt

v2 attached.  New patch closes out writers with FinishType.NORMAL in 
switchWriter and stores the resultant reader.  We then use that list of readers 
in finishAndMaybeThrow rather than finalizing and building a list at that time.

Both changes are Windows-only, obviously.

 java.lang.RuntimeException: Failed to rename XXX to YYY
 ---

 Key: CASSANDRA-8535
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8535
 Project: Cassandra
  Issue Type: Bug
 Environment: Windows 2008 X64
Reporter: Leonid Shalupov
Assignee: Joshua McKenzie
 Fix For: 2.1.3

 Attachments: 8535_v1.txt, 8535_v2.txt


 {code}
 java.lang.RuntimeException: Failed to rename 
 build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
  to 
 build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db
   at 
 org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170) 
 ~[main/:na]
   at 
 org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154) 
 ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569) 
 ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561) 
 ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535) 
 ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470) 
 ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349)
  ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324)
  ~[main/:na]
   at 
 org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304)
  ~[main/:na]
   at 
 org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
  ~[main/:na]
   at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
 ~[main/:na]
   at 
 org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
  ~[main/:na]
   at 
 org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
  ~[main/:na]
   at 
 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226)
  ~[main/:na]
   at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 ~[na:1.7.0_45]
   at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 ~[na:1.7.0_45]
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  ~[na:1.7.0_45]
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_45]
   at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
 Caused by: java.nio.file.FileSystemException: 
 build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
  - 
 build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db:
  The process cannot access the file because it is being used by another 
 process.
   at 
 sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) 
 ~[na:1.7.0_45]
   at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
 ~[na:1.7.0_45]
   at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) 
 ~[na:1.7.0_45]
   at 
 sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) 
 ~[na:1.7.0_45]
   at java.nio.file.Files.move(Files.java:1345) ~[na:1.7.0_45]
   at 
 org.apache.cassandra.io.util.FileUtils.atomicMoveWithFallback(FileUtils.java:184)
  ~[main/:na]
   at 
 org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:166) 
 ~[main/:na]
   ... 18 common frames omitted
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8448) Comparison method violates its general contract in AbstractEndpointSnitch


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309660#comment-14309660
 ] 

Benedict commented on CASSANDRA-8448:
-

The problem with this approach is that sorts will not be able to proceed in 
parallel, and can themselves be stopped by updates. We could prevent the first 
with a read/write lock, but this would not prevent the second (and in fact 
updates would win over sorters). I think a better solution might be to simply 
replace the map wholesale during update - it's only infrequent, and avoids any 
unnecessary costs on the common path. 

 Comparison method violates its general contract in AbstractEndpointSnitch
 ---

 Key: CASSANDRA-8448
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8448
 Project: Cassandra
  Issue Type: Bug
Reporter: J.B. Langston
Assignee: Brandon Williams
 Fix For: 2.1.3, 2.0.13

 Attachments: 8448-v2.txt, 8448.txt


 Seen in both 1.2 and 2.0.  The error is occurring here: 
 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java#L49
 {code}
 ERROR [Thrift:9] 2014-12-04 20:12:28,732 CustomTThreadPoolServer.java (line 
 219) Error occurred during processing of message.
 com.google.common.util.concurrent.UncheckedExecutionException: 
 java.lang.IllegalArgumentException: Comparison method violates its general 
 contract!
   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
   at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
   at 
 com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
   at 
 org.apache.cassandra.service.ClientState.authorize(ClientState.java:352)
   at 
 org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:224)
   at 
 org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:218)
   at 
 org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:202)
   at 
 org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:822)
   at 
 org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:954)
   at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:576)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3922)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3906)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.IllegalArgumentException: Comparison method violates its 
 general contract!
   at java.util.TimSort.mergeHi(TimSort.java:868)
   at java.util.TimSort.mergeAt(TimSort.java:485)
   at java.util.TimSort.mergeCollapse(TimSort.java:410)
   at java.util.TimSort.sort(TimSort.java:214)
   at java.util.TimSort.sort(TimSort.java:173)
   at java.util.Arrays.sort(Arrays.java:659)
   at java.util.Collections.sort(Collections.java:217)
   at 
 org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1408)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1402)
   at 
 org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:148)
   at 
 org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1223)
   at 
 org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1165)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:255)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:225)
   at org.apache.cassandra.auth.Auth.selectUser(Auth.java:243)

[jira] [Commented] (CASSANDRA-8448) Comparison method violates its general contract in AbstractEndpointSnitch


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309679#comment-14309679
 ] 

Brandon Williams commented on CASSANDRA-8448:
-

bq. sorts will not be able to proceed in parallel

Doh, good point.

bq. I think a better solution might be to simply replace the map wholesale 
during update

That's a good idea and even simpler.  v3 does this.

 Comparison method violates its general contract in AbstractEndpointSnitch
 ---

 Key: CASSANDRA-8448
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8448
 Project: Cassandra
  Issue Type: Bug
Reporter: J.B. Langston
Assignee: Brandon Williams
 Fix For: 2.1.3, 2.0.13

 Attachments: 8448-v2.txt, 8448-v3.txt, 8448.txt


 Seen in both 1.2 and 2.0.  The error is occurring here: 
 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java#L49
 {code}
 ERROR [Thrift:9] 2014-12-04 20:12:28,732 CustomTThreadPoolServer.java (line 
 219) Error occurred during processing of message.
 com.google.common.util.concurrent.UncheckedExecutionException: 
 java.lang.IllegalArgumentException: Comparison method violates its general 
 contract!
   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
   at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
   at 
 com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
   at 
 org.apache.cassandra.service.ClientState.authorize(ClientState.java:352)
   at 
 org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:224)
   at 
 org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:218)
   at 
 org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:202)
   at 
 org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:822)
   at 
 org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:954)
   at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:576)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3922)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3906)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.IllegalArgumentException: Comparison method violates its 
 general contract!
   at java.util.TimSort.mergeHi(TimSort.java:868)
   at java.util.TimSort.mergeAt(TimSort.java:485)
   at java.util.TimSort.mergeCollapse(TimSort.java:410)
   at java.util.TimSort.sort(TimSort.java:214)
   at java.util.TimSort.sort(TimSort.java:173)
   at java.util.Arrays.sort(Arrays.java:659)
   at java.util.Collections.sort(Collections.java:217)
   at 
 org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1408)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1402)
   at 
 org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:148)
   at 
 org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1223)
   at 
 org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1165)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:255)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:225)
   at org.apache.cassandra.auth.Auth.selectUser(Auth.java:243)
   at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84)
   at 
 org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
   at

[jira] [Commented] (CASSANDRA-8448) Comparison method violates its general contract in AbstractEndpointSnitch


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309693#comment-14309693
 ] 

Benedict commented on CASSANDRA-8448:
-

One nit: need to make the shared variable volatile. Otherwise +1

 Comparison method violates its general contract in AbstractEndpointSnitch
 ---

 Key: CASSANDRA-8448
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8448
 Project: Cassandra
  Issue Type: Bug
Reporter: J.B. Langston
Assignee: Brandon Williams
 Fix For: 2.1.3, 2.0.13

 Attachments: 8448-v2.txt, 8448-v3.txt, 8448.txt


 Seen in both 1.2 and 2.0.  The error is occurring here: 
 https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java#L49
 {code}
 ERROR [Thrift:9] 2014-12-04 20:12:28,732 CustomTThreadPoolServer.java (line 
 219) Error occurred during processing of message.
 com.google.common.util.concurrent.UncheckedExecutionException: 
 java.lang.IllegalArgumentException: Comparison method violates its general 
 contract!
   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
   at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
   at 
 com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
   at 
 org.apache.cassandra.service.ClientState.authorize(ClientState.java:352)
   at 
 org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:224)
   at 
 org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:218)
   at 
 org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:202)
   at 
 org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:822)
   at 
 org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:954)
   at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:576)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3922)
   at 
 org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3906)
   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
   at 
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.IllegalArgumentException: Comparison method violates its 
 general contract!
   at java.util.TimSort.mergeHi(TimSort.java:868)
   at java.util.TimSort.mergeAt(TimSort.java:485)
   at java.util.TimSort.mergeCollapse(TimSort.java:410)
   at java.util.TimSort.sort(TimSort.java:214)
   at java.util.TimSort.sort(TimSort.java:173)
   at java.util.Arrays.sort(Arrays.java:659)
   at java.util.Collections.sort(Collections.java:217)
   at 
 org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
   at 
 org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1408)
   at 
 org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1402)
   at 
 org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:148)
   at 
 org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1223)
   at 
 org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1165)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:255)
   at 
 org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:225)
   at org.apache.cassandra.auth.Auth.selectUser(Auth.java:243)
   at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84)
   at 
 org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
   at 
 org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:69)
   at org.apache.cassandra.service.ClientState$1.load(ClientState.java:338)
   at

[jira] [Commented] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309737#comment-14309737
]

Aleksey Yeschenko commented on CASSANDRA-8754:
--

I summon [~rlow]

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8714) row-cache: use preloaded jemalloc w/ Unsafe

2015-02-06 Thread Robert Stupp (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309882#comment-14309882
 ] 

Robert Stupp commented on CASSANDRA-8714:
-

I did some off-heap allocation microbenchmarking and published a blog post 
about it: 
http://hardcodejavadog.blogspot.de/2015/02/comparing-java-off-heap-memory.html

It contains measurements on 
* AMD CPU (6 cores, a bit old, but ok) / Linux
* Core i7 Ive Bridge (4 cores + HT), latest OS X
* m3.2xlarge (8 cores)
* c3.4xlarge (16 cores)
* c4.8xlarge (36 cores)

Some things to point out:
# jemalloc via JNA library can hit some busy deadlock that effectively leaves 
threads hanging (and consuming some CPU - feels like a busy spin lock) - filed 
[JNA bug #396|https://github.com/twall/jna/issues/396]
# jemalloc via JNA library is the slowest possible method on Linux and OS X
# jemalloc via JNA's own malloc() + free() methods with jemalloc preloaded is 
the fastest possible thing
# Unsafe with jemalloc preloaded is not bad, too
# That good old AMD CPU has a quite nice allocation throughput ;)

 row-cache: use preloaded jemalloc w/ Unsafe
 ---

 Key: CASSANDRA-8714
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8714
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Robert Stupp
Assignee: Robert Stupp
 Fix For: 3.0

 Attachments: 8714.txt


 Using jemalloc via Java's {{Unsafe}} is a better alternative on Linux than 
 using jemalloc via JNA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ryan Svihla updated CASSANDRA-8754:
---
Comment: was deleted

(was: Then their table will be changed..and I'll have a schema description to
indicate such)

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8730) Optimize UUIDType comparisons

2015-02-06 Thread J.B. Langston (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309901#comment-14309901
 ] 

J.B. Langston commented on CASSANDRA-8730:
--

Looks like a bit of improvement: 12.63MB/sec vs 10.19MB/sec.  Looks like it 
threw away more data this time. I guess some tombstones passed gc grace since I 
last tested. Therefore, I'm not sure how apples-to-apples the comparison is, so 
I'm going to try again while setting my clock back to the date when I ran it 
before.

Before:

{code}
INFO 15:19:05 Compacted 4 sstables to 
[./../data/data/ocean/tbl_metric_data_dyn-0f578640a59211e4a5a2ef9f87394ca6/ocean-tbl_metric_data_dyn-ka-144263,].
 9,183,829,489 bytes to 9,180,536,394 (~99% of original) in 901,172ms = 
9.715395MB/s. 311,495 total partitions merged to 253,490. Partition merge 
counts were {1:195485, 2:58005, }
{code}

After:

{code}
INFO  20:47:24 Completed flushing 
/Users/jblangston/repos/cassandra/bin/./../data/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-48-Data.db
 (42 bytes) for commitlog position ReplayPosition(segmentId=1423254980101, 
position=758851)
INFO  20:47:24 Compacted 4 sstables to 
[./../data/data/ocean/tbl_metric_data_dyn-0f578640a59211e4a5a2ef9f87394ca6/ocean-tbl_metric_data_dyn-ka-144263,].
  8,152,562,772 bytes to 4,659,100,313 (~57% of original) in 615,577ms = 
7.218048MB/s.  311,495 total partitions merged to 80,012.  Partition merge 
counts were {1:195485, 2:58005, }
{code}

 Optimize UUIDType comparisons
 -

 Key: CASSANDRA-8730
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8730
 Project: Cassandra
  Issue Type: Improvement
Reporter: J.B. Langston
Assignee: Benedict
 Fix For: 2.1.4


 Compaction is slow on tables using compound keys containing UUIDs due to 
 being CPU bound by key comparison.  [~benedict] said he sees some easy 
 optimizations that could be made for UUID comparison.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

2015-02-06 Thread Chris Lohfink (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309931#comment-14309931
]

Chris Lohfink commented on CASSANDRA-8754:
--

I could see:

# just create a Counter per CL type in CFMetrics that we can increment on
read/writes. An AtomicLong increment isn't much overhead.
# could throw it in the topK partitions profiler (then also rename nodetool
command from topkpartitions to profile). like CASSANDRA-7974. Would only add
overhead when people are interested then.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309943#comment-14309943
 ] 

Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:24 PM:
-

hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}


was (Author: jeffl):
hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp

[jira] [Commented] (CASSANDRA-5791) A nodetool command to validate all sstables in a node

2015-02-06 Thread Jeff Jirsa (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310025#comment-14310025
]

Jeff Jirsa commented on CASSANDRA-5791:
---

Duplicating my comment from 8703 here since its a dupe and prone to closure :

I've got a version at
https://github.com/jeffjirsa/cassandra/commits/cassandra-8703 that follows the
scrub read path and implements nodetool verify / sstableverify. This works, for
both compressed and uncompressed, but requires walking the entire sstable and
verifies each on disk atom. This works, it just isn't very fast (though it is
thorough).
The faster method will be checking against the Digest.sha1 file (which actually
contains an adler32 hash), and skipping the full iteration. I'll rebase and
work that in, using the 'walk all atoms' approach above as an optional extended
verify (-e) or similar, unless someone objects. Also going to rename the DIGEST
sstable component to Digest.adler32 since it's definitely not sha1 anymore.

A nodetool command to validate all sstables in a node
-

Key: CASSANDRA-5791
URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: sankalp kohli
Priority: Minor

CUrrently there is no nodetool command to validate all sstables on disk. The
only way to do this is to run a repair and see if it succeeds. But we cannot
repair the system keyspace.
Also we can run upgrade sstables but that re writes all the sstables.
This command should check the hash of all sstables and return whether all
data is readable all not. This should NOT care about consistency.
The compressed sstables do not have hash so not sure how it will work there.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-5791) A nodetool command to validate all sstables in a node

2015-02-06 Thread Jeff Jirsa (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reassigned CASSANDRA-5791:
-

Assignee: Jeff Jirsa

 A nodetool command to validate all sstables in a node
 -

 Key: CASSANDRA-5791
 URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: sankalp kohli
Assignee: Jeff Jirsa
Priority: Minor

 CUrrently there is no nodetool command to validate all sstables on disk. The 
 only way to do this is to run a repair and see if it succeeds. But we cannot 
 repair the system keyspace. 
 Also we can run upgrade sstables but that re writes all the sstables. 
 This command should check the hash of all sstables and return whether all 
 data is readable all not. This should NOT care about consistency. 
 The compressed sstables do not have hash so not sure how it will work there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer

2015-02-06 Thread Michael Shuler (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310061#comment-14310061
 ] 

Michael Shuler commented on CASSANDRA-8723:
---

c1.xlarge is 7G RAM, which is a relatively small amount for Cassandra. Do these 
servers have swap enabled and is it being used?

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
 /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
  -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof 
 -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log 
 org.apache.cassandra.service.CassandraDaemon
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8715) Possible Deadlock in Cqlsh in a Kerberos-enabled environment when using COPY ... FROM ...


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310164#comment-14310164
 ] 

Tyler Hobbs commented on CASSANDRA-8715:


It's not deadlock.  The problem is that a handful of the requests never 
complete (usually three or four, in my testing).  I verified this by dumping 
{{session.get_pool_state()}}, which shows the number of in-flight requests per 
connection.  These counts were consistent with the number of pending chains (on 
the countdown latch).  Since the python driver doesn't currently have a way to 
timeout execute_async() + callback requests, these hang indefinitely.

I also observed that with {{async_insert.CONCURRENCY}} set to 1, there were no 
problems.  This leads me to believe that there may be some sort of concurrency 
problem with kerberos, either in the driver, the python kerberos library, or in 
DSE/Cassandra.

It's worth noting that CASSANDRA-8225 will switch COPY FROM to a 
single-thread-per-process model, which might consequently avoid this bug.

Also, regarding the double-statement issue, I'm not sure what might be causing 
that in your environment, but I haven't been able to reproduce it (with or 
without kerberos).

 Possible Deadlock in Cqlsh in a Kerberos-enabled environment when using COPY 
 ... FROM ...
 ---

 Key: CASSANDRA-8715
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8715
 Project: Cassandra
  Issue Type: Bug
 Environment: Cassandra 2.1.2.160, cqlsh 5.0.1, Native protocol v3
Reporter: Eduard Tudenhoefner
Assignee: Tyler Hobbs
Priority: Minor
  Labels: cqlsh
 Fix For: 2.1.3


 When running a COPY ... FROM ... command in a Kerberos environment, I see the 
 number of rows processed, but eventually, Cqlsh never returns. I can verify, 
 that all the data was copied, but the progress bar shows me the last shown 
 info and cqlsh hangs there and never returns.
 Please note that this issue did *not* occur in the exact same environment 
 with *Cassandra 2.0.12.156*.
 With the help of Tyler Hobbs, I investigated the problem a little bit further 
 and added some debug statements at specific points. For example, in the 
 CountdownLatch class at 
 https://github.com/apache/cassandra/blob/a323a1a6d5f28ced1a51ba559055283f3eb356ff/pylib/cqlshlib/async_insert.py#L35-L36
  I can see that the counter always stays above zero and therefore never 
 returns (even when the data to be copied is already copied).
 I've also seen that somehow when I type in one cqlsh command, there will be 
 actually two commands. Let me give you an example:
 I added a debug statement just before 
 https://github.com/apache/cassandra/blob/d76450c7986202141f3a917b3623a4c3138c1094/bin/cqlsh#L920
 {code}
 cqlsh use libdata ; 
 2015-01-30 18:54:56,113 [DEBUG] root: STATEMENT: [('K_USE', 'use', (0, 3)), 
 ('identifier', 'libdata', (4, 11)), ('endtoken', ';', (12, 13))] 
 2015-01-30 18:54:56,113 [DEBUG] root: STATEMENT: [('K_USE', 'use', (0, 3)), 
 ('identifier', 'libdata', (4, 11)), ('endtoken', ';', (12, 13))]
 {code}
 and saw that all commands I enter, they end up being executed twice (same 
 goes for the COPY command).
 If I can provide any other input for debugging purposes, please let me know.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309943#comment-14309943
 ] 

Jeff Liu commented on CASSANDRA-8723:
-

hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
 /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:

[jira] [Commented] (CASSANDRA-8730) Optimize UUIDType comparisons


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309991#comment-14309991
 ] 

Aleksey Yeschenko commented on CASSANDRA-8730:
--

[~jblangs...@datastax.com] What's your schema? Does it use 'uuid' or 
'timeuuid'? b/c the patch in its current form does only optimize 'uuid'.

 Optimize UUIDType comparisons
 -

 Key: CASSANDRA-8730
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8730
 Project: Cassandra
  Issue Type: Improvement
Reporter: J.B. Langston
Assignee: Benedict
 Fix For: 2.1.4


 Compaction is slow on tables using compound keys containing UUIDs due to 
 being CPU bound by key comparison.  [~benedict] said he sees some easy 
 optimizations that could be made for UUID comparison.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310075#comment-14310075
 ] 

Jeff Liu commented on CASSANDRA-8723:
-

yes, we know c1.xlarge is relatively small in terms of the memory resource for 
cassandra. We haven't identified any other resource contention on this instance 
type with our application is the reason that we haven't turned to another high 
capacity instance type. Also c1.xlarge gave us a cheap solution with enough 
cheap storage. 

The swap is enabled and swappiness is set at 60%. 

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
 /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
  -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof

[jira] [Commented] (CASSANDRA-8730) Optimize UUIDType comparisons

2015-02-06 Thread J.B. Langston (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309953#comment-14309953
 ] 

J.B. Langston commented on CASSANDRA-8730:
--

Hmm, setting my clock back didn't help. I still got the same results as before. 
I'm not sure why it did not compact away almost half the data before.

 Optimize UUIDType comparisons
 -

 Key: CASSANDRA-8730
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8730
 Project: Cassandra
  Issue Type: Improvement
Reporter: J.B. Langston
Assignee: Benedict
 Fix For: 2.1.4


 Compaction is slow on tables using compound keys containing UUIDs due to 
 being CPU bound by key comparison.  [~benedict] said he sees some easy 
 optimizations that could be made for UUID comparison.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8703) incremental repair vs. bitrot

2015-02-06 Thread sankalp kohli (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309980#comment-14309980
 ] 

sankalp kohli edited comment on CASSANDRA-8703 at 2/6/15 9:44 PM:
--

This is a dupe of CASSANDRA-8169 which is a dupe of CASSANDRA-5791. 


was (Author: kohlisankalp):
This is a dupe of CASSANDRA-8169 which is intern is a dupe of CASSANDRA-5791. 

 incremental repair vs. bitrot
 -

 Key: CASSANDRA-8703
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8703
 Project: Cassandra
  Issue Type: Bug
Reporter: Robert Coli
Assignee: Jeff Jirsa

 Incremental repair is a great improvement in Cassandra, but it does not 
 contain a feature that non-incremental repair does : protection against 
 bitrot.
 Scenario :
 1) repair SSTable, marking it repaired
 2) cosmic ray hits hard drive, corrupting a record in SSTable
 3) range is actually unrepaired as of the time that SSTable was repaired, but 
 thinks it is repaired
 From my understanding, if bitrot is detected (via eg the CRC on the read 
 path) then all SSTables containing the corrupted range needs to be marked 
 unrepaired on all replicas. Per marcuse@IRC, the naive/simplest response 
 would be to just trigger a full repair in this case.
 I am concerned about incremental repair as an operational default while it 
 does not handle this case. As an aside, this would also seem to require a new 
 CRC on the uncompressed read path, as otherwise one cannot detect the 
 corruption without periodic checksumming of SSTables. Alternately, a 
 nodetool checksum function which verified table checksums, marking ranges 
 unrepaired on failure, and which could be run every gc_grace_seconds would 
 seem to meet the requirement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8703) incremental repair vs. bitrot

2015-02-06 Thread sankalp kohli (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309980#comment-14309980
 ] 

sankalp kohli commented on CASSANDRA-8703:
--

This is a dupe of CASSANDRA-8169 which is intern is a dupe of CASSANDRA-5791. 

 incremental repair vs. bitrot
 -

 Key: CASSANDRA-8703
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8703
 Project: Cassandra
  Issue Type: Bug
Reporter: Robert Coli
Assignee: Jeff Jirsa

 Incremental repair is a great improvement in Cassandra, but it does not 
 contain a feature that non-incremental repair does : protection against 
 bitrot.
 Scenario :
 1) repair SSTable, marking it repaired
 2) cosmic ray hits hard drive, corrupting a record in SSTable
 3) range is actually unrepaired as of the time that SSTable was repaired, but 
 thinks it is repaired
 From my understanding, if bitrot is detected (via eg the CRC on the read 
 path) then all SSTables containing the corrupted range needs to be marked 
 unrepaired on all replicas. Per marcuse@IRC, the naive/simplest response 
 would be to just trigger a full repair in this case.
 I am concerned about incremental repair as an operational default while it 
 does not handle this case. As an aside, this would also seem to require a new 
 CRC on the uncompressed read path, as otherwise one cannot detect the 
 corruption without periodic checksumming of SSTables. Alternately, a 
 nodetool checksum function which verified table checksums, marking ranges 
 unrepaired on failure, and which could be run every gc_grace_seconds would 
 seem to meet the requirement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8730) Optimize UUIDType comparisons

2015-02-06 Thread Jonathan Ellis (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-8730:
--
Reviewer: Ariel Weisberg

([~aweisberg] to review)

 Optimize UUIDType comparisons
 -

 Key: CASSANDRA-8730
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8730
 Project: Cassandra
  Issue Type: Improvement
Reporter: J.B. Langston
Assignee: Benedict
 Fix For: 2.1.4


 Compaction is slow on tables using compound keys containing UUIDs due to 
 being CPU bound by key comparison.  [~benedict] said he sees some easy 
 optimizations that could be made for UUID comparison.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8703) incremental repair vs. bitrot

2015-02-06 Thread Michael Shuler (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Shuler resolved CASSANDRA-8703.
---
Resolution: Duplicate

incremental repair vs. bitrot
-

Key: CASSANDRA-8703
URL: https://issues.apache.org/jira/browse/CASSANDRA-8703
Project: Cassandra
Issue Type: Bug
Reporter: Robert Coli
Assignee: Jeff Jirsa

Incremental repair is a great improvement in Cassandra, but it does not
contain a feature that non-incremental repair does : protection against
bitrot.
Scenario :
1) repair SSTable, marking it repaired
2) cosmic ray hits hard drive, corrupting a record in SSTable
3) range is actually unrepaired as of the time that SSTable was repaired, but
thinks it is repaired
From my understanding, if bitrot is detected (via eg the CRC on the read
path) then all SSTables containing the corrupted range needs to be marked
unrepaired on all replicas. Per marcuse@IRC, the naive/simplest response
would be to just trigger a full repair in this case.
I am concerned about incremental repair as an operational default while it
does not handle this case. As an aside, this would also seem to require a new
CRC on the uncompressed read path, as otherwise one cannot detect the
corruption without periodic checksumming of SSTables. Alternately, a
nodetool checksum function which verified table checksums, marking ranges
unrepaired on failure, and which could be run every gc_grace_seconds would
seem to meet the requirement.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/7] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

Merge branch 'cassandra-2.0' into cassandra-2.1

Conflicts:
CHANGES.txt


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e95684c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e95684c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e95684c

Branch: refs/heads/trunk
Commit: 5e95684c9c992c7eb3477972482abb1cd6216978
Parents: caba0a5 cf73de2
Author: Brandon Williams brandonwilli...@apache.org
Authored: Fri Feb 6 13:24:05 2015 -0600
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Fri Feb 6 13:24:05 2015 -0600

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/locator/DynamicEndpointSnitch.java| 6 --
 2 files changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e95684c/CHANGES.txt
--
diff --cc CHANGES.txt
index 959a2de,fa9c77d..a9e8c7c
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,91 -1,5 +1,92 @@@
 -2.0.13:
 +2.1.3
 + * Write partition size estimates into a system table (CASSANDRA-7688)
 + * Upgrade libthrift to 0.9.2 (CASSANDRA-8685)
 + * Don't use the shared ref in sstableloader (CASSANDRA-8704)
 + * Purge internal prepared statements if related tables or
 +   keyspaces are dropped (CASSANDRA-8693)
 + * (cqlsh) Handle unicode BOM at start of files (CASSANDRA-8638)
 + * Stop compactions before exiting offline tools (CASSANDRA-8623)
 + * Update tools/stress/README.txt to match current behaviour (CASSANDRA-7933)
 + * Fix schema from Thrift conversion with empty metadata (CASSANDRA-8695)
 + * Safer Resource Management (CASSANDRA-7705)
 + * Make sure we compact highly overlapping cold sstables with
 +   STCS (CASSANDRA-8635)
 + * rpc_interface and listen_interface generate NPE on startup when specified
 +   interface doesn't exist (CASSANDRA-8677)
 + * Fix ArrayIndexOutOfBoundsException in nodetool cfhistograms 
(CASSANDRA-8514)
 + * Switch from yammer metrics for nodetool cf/proxy histograms 
(CASSANDRA-8662)
 + * Make sure we don't add tmplink files to the compaction
 +   strategy (CASSANDRA-8580)
 + * (cqlsh) Handle maps with blob keys (CASSANDRA-8372)
 + * (cqlsh) Handle DynamicCompositeType schemas correctly (CASSANDRA-8563)
 + * Duplicate rows returned when in clause has repeated values (CASSANDRA-6707)
 + * Add tooling to detect hot partitions (CASSANDRA-7974)
 + * Fix cassandra-stress user-mode truncation of partition generation 
(CASSANDRA-8608)
 + * Only stream from unrepaired sstables during inc repair (CASSANDRA-8267)
 + * Don't allow starting multiple inc repairs on the same sstables 
(CASSANDRA-8316)
 + * Invalidate prepared BATCH statements when related tables
 +   or keyspaces are dropped (CASSANDRA-8652)
 + * Fix missing results in secondary index queries on collections
 +   with ALLOW FILTERING (CASSANDRA-8421)
 + * Expose EstimatedHistogram metrics for range slices (CASSANDRA-8627)
 + * (cqlsh) Escape clqshrc passwords properly (CASSANDRA-8618)
 + * Fix NPE when passing wrong argument in ALTER TABLE statement 
(CASSANDRA-8355)
 + * Pig: Refactor and deprecate CqlStorage (CASSANDRA-8599)
 + * Don't reuse the same cleanup strategy for all sstables (CASSANDRA-8537)
 + * Fix case-sensitivity of index name on CREATE and DROP INDEX
 +   statements (CASSANDRA-8365)
 + * Better detection/logging for corruption in compressed sstables 
(CASSANDRA-8192)
 + * Use the correct repairedAt value when closing writer (CASSANDRA-8570)
 + * (cqlsh) Handle a schema mismatch being detected on startup (CASSANDRA-8512)
 + * Properly calculate expected write size during compaction (CASSANDRA-8532)
 + * Invalidate affected prepared statements when a table's columns
 +   are altered (CASSANDRA-7910)
 + * Stress - user defined writes should populate sequentally (CASSANDRA-8524)
 + * Fix regression in SSTableRewriter causing some rows to become unreadable 
 +   during compaction (CASSANDRA-8429)
 + * Run major compactions for repaired/unrepaired in parallel (CASSANDRA-8510)
 + * (cqlsh) Fix compression options in DESCRIBE TABLE output when compression
 +   is disabled (CASSANDRA-8288)
 + * (cqlsh) Fix DESCRIBE output after keyspaces are altered (CASSANDRA-7623)
 + * Make sure we set lastCompactedKey correctly (CASSANDRA-8463)
 + * (cqlsh) Fix output of CONSISTENCY command (CASSANDRA-8507)
 + * (cqlsh) Fixed the handling of LIST statements (CASSANDRA-8370)
 + * Make sstablescrub check leveled manifest again (CASSANDRA-8432)
 + * Check first/last keys in sstable when giving out positions (CASSANDRA-8458)
 + * Disable mmap on Windows (CASSANDRA-6993)
 + * Add missing ConsistencyLevels to cassandra-stress (CASSANDRA-8253)
 + * Add auth support to cassandra-stress (CASSANDRA-7985)
 + * Fix

[7/7] cassandra git commit: Merge branch 'cassandra-2.1' into trunk

Merge branch 'cassandra-2.1' into trunk

Conflicts:
build.xml
src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/223d0e75
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/223d0e75
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/223d0e75

Branch: refs/heads/trunk
Commit: 223d0e755ee0480316f90621ac6389e942c23d97
Parents: ca41972 1512264
Author: Tyler Hobbs ty...@datastax.com
Authored: Fri Feb 6 16:47:00 2015 -0600
Committer: Tyler Hobbs ty...@datastax.com
Committed: Fri Feb 6 16:47:00 2015 -0600

--
 CHANGES.txt| 1 +
 NEWS.txt   | 2 ++
 doc/cql3/CQL.textile   | 2 +-
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java   | 3 ++-
 .../org/apache/cassandra/locator/DynamicEndpointSnitch.java| 6 --
 5 files changed, 10 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/223d0e75/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/223d0e75/NEWS.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/223d0e75/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/223d0e75/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --cc src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 9d71a67,aefce0d..b670b6a
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@@ -55,8 -54,8 +55,8 @@@ public class DynamicEndpointSnitch exte
  private String mbeanName;
  private boolean registered = false;
  
- private final ConcurrentHashMapInetAddress, Double scores = new 
ConcurrentHashMapInetAddress, Double();
 -private volatile HashMapInetAddress, Double scores = new 
HashMapInetAddress, Double();
 -private final ConcurrentHashMapInetAddress, ExponentiallyDecayingSample 
samples = new ConcurrentHashMapInetAddress, ExponentiallyDecayingSample();
++private volatile HashMapInetAddress, Double scores = new HashMap();
 +private final ConcurrentHashMapInetAddress, 
ExponentiallyDecayingReservoir samples = new ConcurrentHashMap();
  
  public final IEndpointSnitch subsnitch;
  
@@@ -245,7 -244,8 +245,8 @@@
  double maxLatency = 1;
  // We're going to weight the latency for each host against the worst 
one we see, to
  // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
+ HashMapInetAddress, Double newScores = new HashMap();
 -for (Map.EntryInetAddress, ExponentiallyDecayingSample entry : 
samples.entrySet())
 +for (Map.EntryInetAddress, ExponentiallyDecayingReservoir entry : 
samples.entrySet())
  {
  double mean = entry.getValue().getSnapshot().getMedian();
  if (mean  maxLatency)

[6/7] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

Merge branch 'cassandra-2.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15122647
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15122647
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15122647

Branch: refs/heads/trunk
Commit: 1512264703bc460baa804201a348f6ae82154157
Parents: 5e95684 08deff7
Author: Tyler Hobbs ty...@datastax.com
Authored: Fri Feb 6 16:38:06 2015 -0600
Committer: Tyler Hobbs ty...@datastax.com
Committed: Fri Feb 6 16:38:06 2015 -0600

--
 doc/cql3/CQL.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15122647/doc/cql3/CQL.textile
--

[1/7] cassandra git commit: Disable early open compaction

Repository: cassandra
Updated Branches:
  refs/heads/trunk ca41972ea - 223d0e755


Disable early open compaction


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98905809
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98905809
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98905809

Branch: refs/heads/trunk
Commit: 98905809c87cc8fef01621bdcfd069d6edc75324
Parents: 82a8c23
Author: T Jake Luciani j...@apache.org
Authored: Thu Feb 5 10:51:22 2015 -0500
Committer: T Jake Luciani j...@apache.org
Committed: Thu Feb 5 10:54:57 2015 -0500

--
 src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98905809/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 1dd1688..1ee74e9 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -1463,7 +1463,8 @@ public class DatabaseDescriptor
 
 public static int getSSTablePreempiveOpenIntervalInMB()
 {
-return conf.sstable_preemptive_open_interval_in_mb;
+//return conf.sstable_preemptive_open_interval_in_mb;
+return -1;
 }
 
 public static boolean getTrickleFsync()

[5/7] cassandra git commit: Clarify default timestamp in CQL docs

Clarify default timestamp in CQL docs

Patch by Tyler Hobbs for CASSANDRA-8538


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/08deff70
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/08deff70
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/08deff70

Branch: refs/heads/trunk
Commit: 08deff70a0f22e235b7c246c9ff6f4f7877c66ff
Parents: cf73de2
Author: Tyler Hobbs ty...@datastax.com
Authored: Fri Feb 6 16:36:55 2015 -0600
Committer: Tyler Hobbs ty...@datastax.com
Committed: Fri Feb 6 16:37:45 2015 -0600

--
 doc/cql3/CQL.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/08deff70/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index ab7da50..6085d00 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -545,7 +545,7 @@ The @id = id + collection-literal@ and @id[value1] = 
value2@ forms of @assign
 h4(#updateOptions). @options@
 
 The @UPDATE@ and @INSERT@ statements allows to specify the following options 
for the insertion:
-* @TIMESTAMP@: sets the timestamp for the operation. If not specified, the 
current time of the insertion (in microseconds) is used. This is usually a 
suitable default.
+* @TIMESTAMP@: sets the timestamp for the operation. If not specified, the 
coordinator will use the current time (in microseconds) at the start of 
statement execution as the timestamp. This is usually a suitable default.
 * @TTL@: allows to specify an optional Time To Live (in seconds) for the 
inserted values. If set, the inserted values are automatically removed from the 
database after the specified time. Note that the TTL concerns the inserted 
values, not the column themselves. This means that any subsequent update of the 
column will also reset the TTL (to whatever TTL is specified in that update). 
By default, values never expire. A TTL of 0 or a negative one is equivalent to 
no TTL.

[2/7] cassandra git commit: docs and bump version

docs and bump version


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/caba0a59
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/caba0a59
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/caba0a59

Branch: refs/heads/trunk
Commit: caba0a5920400f912a27dc8e73e887d09c450908
Parents: 9890580
Author: T Jake Luciani j...@apache.org
Authored: Thu Feb 5 11:11:14 2015 -0500
Committer: T Jake Luciani j...@apache.org
Committed: Thu Feb 5 11:11:14 2015 -0500

--
 NEWS.txt  | 2 ++
 build.xml | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/caba0a59/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index e344acc..602770c 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -21,6 +21,8 @@ Upgrading
 - Prepending a list to a list collection was erroneously resulting in
   the prepended list being reversed upon insertion.  If you were depending
   on this buggy behavior, note that it has been corrected.
+- Incremental replacement of compacted SSTables has been disabled for this
+  release.
 
 2.1.2
 =

http://git-wip-us.apache.org/repos/asf/cassandra/blob/caba0a59/build.xml
--
diff --git a/build.xml b/build.xml
index 0aa8c01..eaef534 100644
--- a/build.xml
+++ b/build.xml
@@ -25,7 +25,7 @@
 property name=debuglevel value=source,lines,vars/
 
 !-- default version and SCM information --
-property name=base.version value=2.1.2/
+property name=base.version value=2.1.3/
 property name=scm.connection 
value=scm:git://git.apache.org/cassandra.git/
 property name=scm.developerConnection 
value=scm:git://git.apache.org/cassandra.git/
 property name=scm.url 
value=http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree/

[3/7] cassandra git commit: Fix IllegalArgumentException in dynamic snitch

Fix IllegalArgumentException in dynamic snitch

Patch by brandonwilliams, viewed by Benedict for CASSANDRA-8448


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cf73de2d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cf73de2d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cf73de2d

Branch: refs/heads/trunk
Commit: cf73de2dc99d4857cc4bac734cf48cf9a21aa9be
Parents: 1467b9f
Author: Brandon Williams brandonwilli...@apache.org
Authored: Fri Feb 6 13:23:05 2015 -0600
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Fri Feb 6 13:23:05 2015 -0600

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/locator/DynamicEndpointSnitch.java| 6 --
 2 files changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cf73de2d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index f4c96dc..fa9c77d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.0.13:
+ * Fix IllegalArgumentException in dynamic snitch (CASSANDRA-8448)
  * Add support for UPDATE ... IF EXISTS (CASSANDRA-8610)
  * Fix reversal of list prepends (CASSANDRA-8733)
  * Prevent non-zero default_time_to_live on tables with counters

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cf73de2d/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
--
diff --git a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java 
b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
index 49442c8..3469847 100644
--- a/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
+++ b/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
@@ -53,7 +53,7 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 private String mbeanName;
 private boolean registered = false;
 
-private final ConcurrentHashMapInetAddress, Double scores = new 
ConcurrentHashMapInetAddress, Double();
+private volatile HashMapInetAddress, Double scores = new 
HashMapInetAddress, Double();
 private final ConcurrentHashMapInetAddress, ExponentiallyDecayingSample 
samples = new ConcurrentHashMapInetAddress, ExponentiallyDecayingSample();
 
 public final IEndpointSnitch subsnitch;
@@ -243,6 +243,7 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 double maxLatency = 1;
 // We're going to weight the latency for each host against the worst 
one we see, to
 // arrive at sort of a 'badness percentage' for them. First, find the 
worst for each:
+HashMapInetAddress, Double newScores = new HashMap();
 for (Map.EntryInetAddress, ExponentiallyDecayingSample entry : 
samples.entrySet())
 {
 double mean = entry.getValue().getSnapshot().getMedian();
@@ -257,8 +258,9 @@ public class DynamicEndpointSnitch extends 
AbstractEndpointSnitch implements ILa
 // Severity is basically a measure of compaction activity 
(CASSANDRA-3722).
 score += StorageService.instance.getSeverity(entry.getKey());
 // lowest score (least amount of badness) wins.
-scores.put(entry.getKey(), score);
+newScores.put(entry.getKey(), score);
 }
+scores = newScores;
 }

[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309943#comment-14309943
 ] 

Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:27 PM:
-

hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}
4. nodetool status show that the nodes are having over 40GB data/load:
{noformat}
--  Address Load   Tokens  OwnsHost ID  
 Rack
UN  10.98.xxx.5343.62 GB   256 ?   
1ab1fec7-383b-4801-b899-059ea0c18def  us-east-1d
UN  10.182.xxx.24   46.53 GB   256 ?   
07a9849b-73f2-4166-a8e9-bedd40a1f30e  us-east-1e
UN  10.32.xxx.222   62.36 GB   256 ?   
fb8c38fc-0eb1-41c2-96c3-fc597aeede10  us-east-1d
UN  10.66.xxx.48 45.03 GB   256 ?   
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2  us-east-1c
UN  10.232.xxx.154   50.57 GB   256 ?   
a3ccf3ba-2649-4d40-8992-41d38dcab493  us-east-1e
UN  10.98.xxx.31 43.71 GB   256 ?   
290a48fc-9efe-4e3d-9237-c363894c7b13  us-east-1d
UN  10.99.xxx.18739.76 GB   256 ?   
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b  us-east-1d
{noformat}



was (Author: jeffl):
hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}
4. nodetool status show that the nodes are having over 40GB data/load:
{noformat}
--  Address Load   Tokens  OwnsHost ID  
 Rack
UN  10.98.xxx.5343.62 GB   256 ?   
1ab1fec7-383b-4801-b899-059ea0c18def  us-east-1d
UN  10.182.xxx.24   46.53 GB   256 ?   
07a9849b-73f2-4166-a8e9-bedd40a1f30e  us-east-1e
UN  10.32.xxx.222   62.36 GB   256 ?   
fb8c38fc-0eb1-41c2-96c3-fc597aeede10  us-east-1d
UN  10.66.xxx.48 45.03 GB   256 ?   
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2  us-east-1c
UN  10.232.xxx.154   50.57 GB   256 ?   
a3ccf3ba-2649-4d40-8992-41d38dcab493  us-east-1e
UN  10.98.xxx.31 43.71 GB   256 ?   
290a48fc-9efe-4e3d-9237-c363894c7b13  us-east-1d
UN  10.99.79.18739.76 GB   256 ?   
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b  us-east-1d
{noformat}


 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199

[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309943#comment-14309943
 ] 

Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:26 PM:
-

hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}
4. nodetool status show that the nodes are having over 40GB data/load:
{noformat}
--  Address Load   Tokens  OwnsHost ID  
 Rack
UN  10.98.xxx.5343.62 GB   256 ?   
1ab1fec7-383b-4801-b899-059ea0c18def  us-east-1d
UN  10.182.xxx.24   46.53 GB   256 ?   
07a9849b-73f2-4166-a8e9-bedd40a1f30e  us-east-1e
UN  10.32.xxx.222   62.36 GB   256 ?   
fb8c38fc-0eb1-41c2-96c3-fc597aeede10  us-east-1d
UN  10.66.xxx.48 45.03 GB   256 ?   
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2  us-east-1c
UN  10.232.xxx.154   50.57 GB   256 ?   
a3ccf3ba-2649-4d40-8992-41d38dcab493  us-east-1e
UN  10.98.xxx.31 43.71 GB   256 ?   
290a48fc-9efe-4e3d-9237-c363894c7b13  us-east-1d
UN  10.99.79.18739.76 GB   256 ?   
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b  us-east-1d
{noformat}



was (Author: jeffl):
hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}
4. nodetool status show that the nodes are having 40GB data/load on average:
{noformat}
--  Address Load   Tokens  OwnsHost ID  
 Rack
UN  10.98.xxx.5343.62 GB   256 ?   
1ab1fec7-383b-4801-b899-059ea0c18def  us-east-1d
UN  10.182.xxx.24   46.53 GB   256 ?   
07a9849b-73f2-4166-a8e9-bedd40a1f30e  us-east-1e
UN  10.32.xxx.222   62.36 GB   256 ?   
fb8c38fc-0eb1-41c2-96c3-fc597aeede10  us-east-1d
UN  10.66.xxx.48 45.03 GB   256 ?   
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2  us-east-1c
UN  10.232.xxx.154   50.57 GB   256 ?   
a3ccf3ba-2649-4d40-8992-41d38dcab493  us-east-1e
UN  10.98.xxx.31 43.71 GB   256 ?   
290a48fc-9efe-4e3d-9237-c363894c7b13  us-east-1d
UN  10.99.79.18739.76 GB   256 ?   
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b  us-east-1d
{noformat}


 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199

[jira] [Updated] (CASSANDRA-8225) Production-capable COPY FROM


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-8225:
---
Labels: cqlsh  (was: )

 Production-capable COPY FROM
 

 Key: CASSANDRA-8225
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8225
 Project: Cassandra
  Issue Type: New Feature
  Components: Tools
Reporter: Jonathan Ellis
Assignee: Tyler Hobbs
  Labels: cqlsh
 Fix For: 2.1.4

 Attachments: 8225-2.1.txt


 Via [~schumacr],
 bq. I pulled down a sourceforge data generator and created a moc file of 
 500,000 rows that had an incrementing sequence number, date, and SSN. I then 
 used our COPY command and MySQL's LOAD DATA INFILE to load the file on my 
 Mac. Results were: 
 {noformat}
 mysql load data infile '/Users/robin/dev/datagen3.txt'  into table p_test  
 fields terminated by ',';
 Query OK, 50 rows affected (2.18 sec)
 {noformat}
 C* 2.1.0 (pre-CASSANDRA-7405)
 {noformat}
 cqlsh:dev copy p_test from '/Users/robin/dev/datagen3.txt' with 
 delimiter=',';
 50 rows imported in 16 minutes and 45.485 seconds.
 {noformat}
 Cassandra 2.1.1:
 {noformat}
 cqlsh:dev copy p_test from '/Users/robin/dev/datagen3.txt' with 
 delimiter=',';
 Processed 50 rows; Write: 4037.46 rows/s
 50 rows imported in 2 minutes and 3.058 seconds.
 {noformat}
 [jbellis] 7405 gets us almost an order of magnitude improvement.  
 Unfortunately we're still almost 2 orders slower than mysql.
 I don't think we can continue to tell people, use sstableloader instead.  
 The number of users sophisticated enough to use the sstable writers is small 
 and (relatively) decreasing as our user base expands.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8692) Coalesce intra-cluster network messages

2015-02-06 Thread Ariel Weisberg (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310079#comment-14310079
]

Ariel Weisberg commented on CASSANDRA-8692:
---

1-4 make sense. #5 you might be right I need to check. When I was doing the
unit tests I had a problem with a large sleep, but I think I fixed that after.

The timing of providing messages to the average doesn't matter since the
timestamp is generated by the submitter not the sending thread. It's already
only a vague stab at how often messages are arriving. Is assuming a uniform
distribution safe? You could be tricked into thinking coalescing makes sense
and then not do it effectively. Maybe with a smaller time horizon it would be
safer?

It seems like trying to do better would help bring down average latency by 100,
maybe 200, microseconds when latency is 2+ milliseconds. Average latency is
still competitive with no coalescing. Coalescing is turned off when there is
less load (and average latency is lower) so you don't feel the pinch there
either.

I am on the fence as to whether something smarter than a moving average counts
as scope creep. I see that as fodder for future iteration unless you want to
make the case that there is a scenario where the current implementation is not
a safe default and will give people a bad experience. Maybe that is something I
need to test for.

Coalesce intra-cluster network messages
---

Key: CASSANDRA-8692
URL: https://issues.apache.org/jira/browse/CASSANDRA-8692
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
Fix For: 2.1.4

Attachments: batching-benchmark.png

While researching CASSANDRA-8457 we found that it is effective and can be
done without introducing additional latency at low concurrency/throughput.
The patch from that was used and found to be useful in a real life scenario
so I propose we implement this in 2.1 in addition to 3.0.
The change set is a single file and is small enough to be reviewable.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8538) Clarify default write time timestamp for CQL insert and update


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310091#comment-14310091
 ] 

Tyler Hobbs commented on CASSANDRA-8538:


I've clarified the default timestamp behavior in the CQL3 docs as committ 
{{08deff70a0}}.

CASSANDRA-7123 already clarified the batch timestamp behavior.  Let me know if 
you need something else there.

bq. It would also be helpful if the tracing option was specific as to which 
time is the official write time for the insert or update.

I'm not sure what you mean.

 Clarify default write time timestamp for CQL insert and update
 --

 Key: CASSANDRA-8538
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8538
 Project: Cassandra
  Issue Type: Improvement
  Components: Documentation  website
Reporter: Jack Krupansky
Assignee: Tyler Hobbs
Priority: Minor

 The current CQL spec (and downstream doc) says that the default timestamp for 
 write time for CQL inserts and updates is the current time of the 
 insertion, but that is somewhat vague and non-specific. In particular, is 
 that the time when the coordinator node parses the CQL statement, or... when 
 the row is inserted or updated on the target nodes, or... something else?
 In particular, if the coordinator doesn't own the token of the primary key, 
 will the owning node set the write time or does the coordinator node do that?
 Obviously the application can set an explicit TIMESTAMP, but this issue is 
 concerned with the default if that explicit option is not used.
 Also, will all replicas of the insert or update share the precisely same 
 write time, or will they reflect the actual time when each particular replica 
 row is inserted or updated on each of the replica nodes?
 Finally, if a batch statement is used to insert or update multiple rows, will 
 they all share the same write time (e.g., the time the batch statement was 
 parsed) or when each replica row is actually inserted or updated on the 
 target (if the coordinator node does not own the token of the partition key) 
 or replica nodes?
 It would also be helpful if the tracing option was specific as to which time 
 is the official write time for the insert or update.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8752) invalid counter shard detected in Version 2.1.2

2015-02-06 Thread Kevin Ye (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310479#comment-14310479
 ] 

Kevin Ye edited comment on CASSANDRA-8752 at 2/7/15 2:36 AM:
-

Hi Aleksey. 

Switching to another version is not an easy decision for my project. So we hope 
to fix the issuse in our 2.1.2 code. 
It will be very helpful if you can tell the issuse number you mentioned.

At first we need to reproduce this problem. One possible cause(not sure) may 
like this:
1. +100 to the counter, counter become (1,+100)
2. Shutdown and reboot the node, which will read the data file asynchronously.
3. Before data file is load, rpc service is started and -33 to the counter 
comes.
4. Since the data file is not loaded, reading of old counter(1,+100) returns 
nothing. Then an (1, -33) is written.
5. (1,+100) and (1, -33) conflict occurs when compact happens.

What's your opinion about it?


was (Author: simsky):
Hi Aleksey. 

Switching to other version is not an easy decision for my project. So we hope 
to fix the issuse in our 2.1.2 code. 
It will be very helpful if you can tell the issuse number you mentioned.

At first we need to reproduce this problem. One possible cause(not sure) may 
like this:
1. +100 to the counter, counter become (1,+100)
2. Shutdown and reboot the node, which will read the data file asynchronously.
3. Before data file is load, rpc service is started and -33 to the counter 
comes.
4. Since the data file is not loaded, reading of old counter(1,+100) returns 
nothing. Then an (1, -33) is written.
5. (1,+100) and (1, -33) conflict occurs when compact happens.

What's your opinion about it?

 invalid counter shard detected in Version 2.1.2
 ---

 Key: CASSANDRA-8752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: SUSE11 SP1, Cassandra 2.1.2, java version 1.7.0_55.
 4 node cluster, vnode = 1, replication = 2
Reporter: Kevin Ye
Assignee: Aleksey Yeschenko

 I was doing counter test (first +100 several times, then -33) on a 4 nodes 
 cluster while below log appear at 2 nodes.There is no concurrent access to 
 same counter.
 WARN  [CompactionExecutor:757] 2015-02-02 13:02:33,375 
 CounterContext.java:431 - invalid global counter shard detected; 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, -33) and 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, 100) differ only in count; will 
 pick highest to self-heal on compaction
 Anyone has encounter this problem? I thought Cassandra 2.1.2 had solved this 
 counter problem, but it appeared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309943#comment-14309943
 ] 

Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:26 PM:
-

hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}
4. nodetool status show that the nodes are having 40GB data/load on average:
{noformat}
--  Address Load   Tokens  OwnsHost ID  
 Rack
UN  10.98.xxx.5343.62 GB   256 ?   
1ab1fec7-383b-4801-b899-059ea0c18def  us-east-1d
UN  10.182.xxx.24   46.53 GB   256 ?   
07a9849b-73f2-4166-a8e9-bedd40a1f30e  us-east-1e
UN  10.32.xxx.222   62.36 GB   256 ?   
fb8c38fc-0eb1-41c2-96c3-fc597aeede10  us-east-1d
UN  10.66.xxx.48 45.03 GB   256 ?   
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2  us-east-1c
UN  10.232.xxx.154   50.57 GB   256 ?   
a3ccf3ba-2649-4d40-8992-41d38dcab493  us-east-1e
UN  10.98.xxx.31 43.71 GB   256 ?   
290a48fc-9efe-4e3d-9237-c363894c7b13  us-east-1d
UN  10.99.79.18739.76 GB   256 ?   
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b  us-east-1d
{noformat}



was (Author: jeffl):
hi Alan,
  thanks for your relay and the answer is yes, we have been able to reproduce 
it several times. The first cassandra version we observed this issue was 2.1.0, 
and reproduced in 2.1.1 and 2.1.3.

1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
21279 cassandr  20   0 21.1g 3.7g  23m S  149 55.1  12001:58 java
{noformat}

 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp

[jira] [Updated] (CASSANDRA-8225) Production-capable COPY FROM


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-8225:
---
Attachment: 8225-2.1.txt

The attached patch (and 
[branch|https://github.com/thobbs/cassandra/tree/CASSANDRA-8225]) gives us 
roughly a 10x speedup on COPY FROM:

{noformat}
cqlsh:ks1 COPY foo FROM 'stuff.csv' ;
446736 rows imported in 16.667 seconds.rows/s
{noformat}

On my laptop, I see between 40k and 55k inserts per second until Cassandra 
starts to flush and garbage collect, which tends to slow things down (cqlsh and 
C* are competing for resources).  Even with those problems, it averages about 
30k rows per second.  I imagine that if C* were run on a separate machine, ~45k 
would be a more typical average.

 Production-capable COPY FROM
 

 Key: CASSANDRA-8225
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8225
 Project: Cassandra
  Issue Type: New Feature
  Components: Tools
Reporter: Jonathan Ellis
Assignee: Tyler Hobbs
 Fix For: 2.1.3

 Attachments: 8225-2.1.txt


 Via [~schumacr],
 bq. I pulled down a sourceforge data generator and created a moc file of 
 500,000 rows that had an incrementing sequence number, date, and SSN. I then 
 used our COPY command and MySQL's LOAD DATA INFILE to load the file on my 
 Mac. Results were: 
 {noformat}
 mysql load data infile '/Users/robin/dev/datagen3.txt'  into table p_test  
 fields terminated by ',';
 Query OK, 50 rows affected (2.18 sec)
 {noformat}
 C* 2.1.0 (pre-CASSANDRA-7405)
 {noformat}
 cqlsh:dev copy p_test from '/Users/robin/dev/datagen3.txt' with 
 delimiter=',';
 50 rows imported in 16 minutes and 45.485 seconds.
 {noformat}
 Cassandra 2.1.1:
 {noformat}
 cqlsh:dev copy p_test from '/Users/robin/dev/datagen3.txt' with 
 delimiter=',';
 Processed 50 rows; Write: 4037.46 rows/s
 50 rows imported in 2 minutes and 3.058 seconds.
 {noformat}
 [jbellis] 7405 gets us almost an order of magnitude improvement.  
 Unfortunately we're still almost 2 orders slower than mysql.
 I don't think we can continue to tell people, use sstableloader instead.  
 The number of users sophisticated enough to use the sstable writers is small 
 and (relatively) decreasing as our user base expands.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8714) row-cache: use preloaded jemalloc w/ Unsafe

2015-02-06 Thread Ariel Weisberg (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309983#comment-14309983
 ] 

Ariel Weisberg commented on CASSANDRA-8714:
---

Something doesn't sit right with me for the difference between JNA accessing 
jemalloc via JNI and Unsafe. Since you are benchmarking access path I think you 
only need to allocate 1-byte. Make sure to store a value to the location so the 
page is touched.

If Unsafe is slower I would be comfortable with using JNA to allocate.

You mention address translation (addr_to_java), but that is just a cast to get 
things to compile. I followed it through to os.cpp and I don't see any real 
extra work.

It's quite possible the results are accurate and JNI just ended up better 
optimized then the native function calls in Unsafe. JNI has received a lot of 
attention.

 row-cache: use preloaded jemalloc w/ Unsafe
 ---

 Key: CASSANDRA-8714
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8714
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Robert Stupp
Assignee: Robert Stupp
 Fix For: 3.0

 Attachments: 8714.txt


 Using jemalloc via Java's {{Unsafe}} is a better alternative on Linux than 
 using jemalloc via JNA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]

2015-02-06 Thread Jonathan Ellis (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-8689:
--
Reviewer: T Jake Luciani

[~tjake] to review

 Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
 ---

 Key: CASSANDRA-8689
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
 Fix For: 2.1.3


 After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the 
 following assertion error.
 {noformat}
 ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451 
 CassandraDaemon.java:153 - Exception in thread 
 Thread[IndexSummaryManager:1,1,main]
 java.lang.AssertionError: null
 at org.apache.cassandra.io.util.Memory.size(Memory.java:307) 
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 [na:1.7.0_45]
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
 [na:1.7.0_45]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  [na:1.7.0_45]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  [na:1.7.0_45]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  [na:1.7.0_45]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_45]
 at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
 {noformat}
 cassandra service is still running despite the issue. Node has total 8G 
 memory with 2G allocated to heap. We are basically running read queries to 
 retrieve data out of cassandra.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8750) Ensure SSTableReader.last corresponds exactly with the file end

2015-02-06 Thread Jonathan Ellis (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-8750:
--
Reviewer: Marcus Eriksson

 Ensure SSTableReader.last corresponds exactly with the file end
 ---

 Key: CASSANDRA-8750
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8750
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Benedict
Assignee: Benedict
Priority: Minor
 Fix For: 2.1.4


 Following on from CASSANDRA-8744, CASSANDRA-8749 and CASSANDRA-8747, this 
 patch attempts to make the whole opening early of compaction results more 
 robust and with more clearly understood behaviour. The improvements of 
 CASSANDRA-8747 permit is to easily align the last key with a summary 
 boundary, and an index and data file end position. This patch modifies 
 SegmentedFile to permit the provision of an explicit length, which is then 
 provided to any readers, which enforce it, ensuring no code may accidentally 
 see an end inconsistent with the one advertised. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8595) Emit timeouts per endpoint

2015-02-06 Thread sankalp kohli (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310022#comment-14310022
 ] 

sankalp kohli commented on CASSANDRA-8595:
--

[~pramod_nair] Can you take this?

 Emit timeouts per endpoint
 --

 Key: CASSANDRA-8595
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8595
 Project: Cassandra
  Issue Type: Improvement
Reporter: sankalp kohli
Priority: Minor

 We currently emit number of timeouts experienced by a co-ordinator while 
 doing reads and writes. This does not tell us which replica or endpoint is 
 responsible for the timeouts. 
 We can keep a map of endpoint to number of timeouts which could be emitted 
 via JMX.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1

Merge branch 'cassandra-2.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15122647
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15122647
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15122647

Branch: refs/heads/cassandra-2.1
Commit: 1512264703bc460baa804201a348f6ae82154157
Parents: 5e95684 08deff7
Author: Tyler Hobbs ty...@datastax.com
Authored: Fri Feb 6 16:38:06 2015 -0600
Committer: Tyler Hobbs ty...@datastax.com
Committed: Fri Feb 6 16:38:06 2015 -0600

--
 doc/cql3/CQL.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/15122647/doc/cql3/CQL.textile
--

cassandra git commit: Clarify default timestamp in CQL docs

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.0 cf73de2dc - 08deff70a


Clarify default timestamp in CQL docs

Patch by Tyler Hobbs for CASSANDRA-8538


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/08deff70
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/08deff70
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/08deff70

Branch: refs/heads/cassandra-2.0
Commit: 08deff70a0f22e235b7c246c9ff6f4f7877c66ff
Parents: cf73de2
Author: Tyler Hobbs ty...@datastax.com
Authored: Fri Feb 6 16:36:55 2015 -0600
Committer: Tyler Hobbs ty...@datastax.com
Committed: Fri Feb 6 16:37:45 2015 -0600

--
 doc/cql3/CQL.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/08deff70/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index ab7da50..6085d00 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -545,7 +545,7 @@ The @id = id + collection-literal@ and @id[value1] = 
value2@ forms of @assign
 h4(#updateOptions). @options@
 
 The @UPDATE@ and @INSERT@ statements allows to specify the following options 
for the insertion:
-* @TIMESTAMP@: sets the timestamp for the operation. If not specified, the 
current time of the insertion (in microseconds) is used. This is usually a 
suitable default.
+* @TIMESTAMP@: sets the timestamp for the operation. If not specified, the 
coordinator will use the current time (in microseconds) at the start of 
statement execution as the timestamp. This is usually a suitable default.
 * @TTL@: allows to specify an optional Time To Live (in seconds) for the 
inserted values. If set, the inserted values are automatically removed from the 
database after the specified time. Note that the TTL concerns the inserted 
values, not the column themselves. This means that any subsequent update of the 
column will also reset the TTL (to whatever TTL is specified in that update). 
By default, values never expire. A TTL of 0 or a negative one is equivalent to 
no TTL.

[1/2] cassandra git commit: Clarify default timestamp in CQL docs

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 5e95684c9 - 151226470


Clarify default timestamp in CQL docs

Patch by Tyler Hobbs for CASSANDRA-8538


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/08deff70
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/08deff70
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/08deff70

Branch: refs/heads/cassandra-2.1
Commit: 08deff70a0f22e235b7c246c9ff6f4f7877c66ff
Parents: cf73de2
Author: Tyler Hobbs ty...@datastax.com
Authored: Fri Feb 6 16:36:55 2015 -0600
Committer: Tyler Hobbs ty...@datastax.com
Committed: Fri Feb 6 16:37:45 2015 -0600

--
 doc/cql3/CQL.textile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/08deff70/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index ab7da50..6085d00 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -545,7 +545,7 @@ The @id = id + collection-literal@ and @id[value1] = 
value2@ forms of @assign
 h4(#updateOptions). @options@
 
 The @UPDATE@ and @INSERT@ statements allows to specify the following options 
for the insertion:
-* @TIMESTAMP@: sets the timestamp for the operation. If not specified, the 
current time of the insertion (in microseconds) is used. This is usually a 
suitable default.
+* @TIMESTAMP@: sets the timestamp for the operation. If not specified, the 
coordinator will use the current time (in microseconds) at the start of 
statement execution as the timestamp. This is usually a suitable default.
 * @TTL@: allows to specify an optional Time To Live (in seconds) for the 
inserted values. If set, the inserted values are automatically removed from the 
database after the specified time. Note that the TTL concerns the inserted 
values, not the column themselves. This means that any subsequent update of the 
column will also reset the TTL (to whatever TTL is specified in that update). 
By default, values never expire. A TTL of 0 or a negative one is equivalent to 
no TTL.

[jira] [Commented] (CASSANDRA-8692) Coalesce intra-cluster network messages

[
https://issues.apache.org/jira/browse/CASSANDRA-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310236#comment-14310236
]

Benedict commented on CASSANDRA-8692:
-

This is the kind of discussion it would really help to have over a whiteboard,
or perhaps over video chat secondarily. I'll do my best to explain my position,
but it's verbose and it seems I already failed to portray it the first time.

bq. The timing of providing messages to the average doesn't matter since the
timestamp is generated by the submitter not the sending thread

It matters because this moving average is over a fixed number of messages,
rather than a fixed time interval. So the order of provision wrt querying
matters a lot, because a (non-uniformly-distributed) close clustering of
messages will have smaller than average delta between them. A single batch you
process at once is likely to fit this bill. The converse is also true; the
first message in a batch is more likely to be abnormally distant from the prior
messages. So your implementation depends on the assumption that the message
arrival is uniform, just implicitly :)

The main thing is that the non-uniformity of message arrival is highly unlikely
to average out over 16 messages, whereas over a 1s time horizon it is more
likely, and so the assumption is perhaps likely to lead to more reasonable
calculations? With a 150K/s message arrival rates (estimated from your
numbers), that means the average is calculated over, on average, 100us, which
is a very short time horizon to extract any predictive power.

bq. I am on the fence as to whether something smarter than a moving average
counts as scope creep.

I'm still only suggesting a moving average, just one measured over a fixed time
horizon, instead of a fixed number of measurements. I'm also suggesting a
different strategy for utilising this calculation, one that uses more
information at our disposal. The two suggestions are somewhat orthogonal, and
you could implement each change independently. Right now your algorithm is:
average over most recent 16 messages, and wait as long as the average delta
between the last 16, which we hope gives us one more message to coalesce. Mine
is 1) to make the average calculation more robust to fluctuations in arrival
rates (but still decay rapidly); and 2) make a decision on whether to coalesce
based on the amount of potential win we will get; as the number of messages
we have to flush grows the benefit to waiting declines, and we also potentially
wait longer if there is the expectation we can flush a significant cluster of
messages at once. Thus it should (theoretically) lead to less delay when
unhelpful, and more when helpful.

bq. It seems like trying to do better would help bring down average latency by
100, maybe 200, microseconds when latency is 2+ milliseconds.

Well, my aim isn't so much to bring down latency further but to make the
algorithm understandable to me. It strikes me that its behaviour isn't
necessarily emerging from its design, but from other correlated factors. Take a
random data point in your results for illustration: 400K messages with a
coalesce window of 100us; guessing your RF=5 and CL=ALL from your other
comments, it looks like each node would have 150K/s traffic, which translates
to ~7us average latency between message arrival. Which means you should only be
waiting 7us to coalesce based on your algorithm, on average, but then why is it
so much faster than a maximum coalesce window of 6us, 12us and 25us? That means
it's regularly predicting an average message delay of 100us, which is not
correct, so it's not clear to me what's actually driving its results. My
concern, as a result, is that this might not actually translate into as much
advantage in the field as it does in the lab, in which we do produce a very
steady load.

My handwavy suspicion right now is that the non-uniformity of message arrivals
leads us to some kind of phased work completion, where the delta between the
phases is large enough to tip our calculation to delay, which steadily leads to
phased work provision from the client since they behave synchronously. This
might also explain why it takes a few minutes for the cluster to reach its
performance steady state - which doesn't otherwise make much sense, which you
also highlighted your confusion over.

Either way, fewer unecessary delays also translates into higher throughput, not
just reduced latency. However mostly I want a beast I can explain all of the
properties and behaviours of.

Coalesce intra-cluster network messages
---

Key: CASSANDRA-8692
URL: https://issues.apache.org/jira/browse/CASSANDRA-8692
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ariel Weisberg

[jira] [Commented] (CASSANDRA-8752) invalid counter shard detected in Version 2.1.2

2015-02-06 Thread Kevin Ye (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310479#comment-14310479
 ] 

Kevin Ye commented on CASSANDRA-8752:
-

Hi Aleksey. 

Switching to other version is not an easy decision for my project. So we hope 
to fix the issuse in our 2.1.2 code. 
It will be very helpful if you can tell the issuse number you mentioned.

At first we need to reproduce this problem. One possible cause(not sure) may 
like this:
1. +100 to the counter, counter become (1,+100)
2. Shutdown and reboot the node, which will read the data file asynchronously.
3. Before data file is load, rpc service is started and -33 to the counter 
comes.
4. Since the data file is not loaded, reading of old counter(1,+100) returns 
nothing. Then an (1, -33) is written.
5. (1,+100) and (1, -33) conflict occurs when compact happens.

What's your opinion about it?

 invalid counter shard detected in Version 2.1.2
 ---

 Key: CASSANDRA-8752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: SUSE11 SP1, Cassandra 2.1.2, java version 1.7.0_55.
 4 node cluster, vnode = 1, replication = 2
Reporter: Kevin Ye
Assignee: Aleksey Yeschenko

 I was doing counter test (first +100 several times, then -33) on a 4 nodes 
 cluster while below log appear at 2 nodes.There is no concurrent access to 
 same counter.
 WARN  [CompactionExecutor:757] 2015-02-02 13:02:33,375 
 CounterContext.java:431 - invalid global counter shard detected; 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, -33) and 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, 100) differ only in count; will 
 pick highest to self-heal on compaction
 Anyone has encounter this problem? I thought Cassandra 2.1.2 had solved this 
 counter problem, but it appeared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8752) invalid counter shard detected in Version 2.1.2

[
https://issues.apache.org/jira/browse/CASSANDRA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310485#comment-14310485
]

Aleksey Yeschenko commented on CASSANDRA-8752:
--

bq. Switching to another version is not an easy decision for my project. So we
hope to fix the issuse in our 2.1.2 code.

You *will* have to upgrade to 2.1.3 once it's out. Unfortunately 2.1.2 is
simply unstable enough for production use. There is no decision to make,
because there are no options.

So, as I said, please try cassandra-2.1 branch recent, or wait till next week,
upgrade to 2.1.3 when it's out, and tell me if you can still replicate. Given
that you see the issue during compaction time, this is most likely fixed by one
of the 2.1.3 sstable tickets.

As for your steps, 2) and 3) are impossible, there is no such thing as reading
the data file asynchronously. Cassandra will not start serving requests until
it replays the commitlog and loads the sstables. 4) and 5) are what's
happening, except it's not because 'the data file is not loaded', which is not
a thing.

invalid counter shard detected in Version 2.1.2
---

Key: CASSANDRA-8752
URL: https://issues.apache.org/jira/browse/CASSANDRA-8752
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: SUSE11 SP1, Cassandra 2.1.2, java version 1.7.0_55.
4 node cluster, vnode = 1, replication = 2
Reporter: Kevin Ye
Assignee: Aleksey Yeschenko

I was doing counter test (first +100 several times, then -33) on a 4 nodes
cluster while below log appear at 2 nodes.There is no concurrent access to
same counter.
WARN [CompactionExecutor:757] 2015-02-02 13:02:33,375
CounterContext.java:431 - invalid global counter shard detected;
(9cca9262-934a-4275-963b-66802471b0c2, 1, -33) and
(9cca9262-934a-4275-963b-66802471b0c2, 1, 100) differ only in count; will
pick highest to self-heal on compaction
Anyone has encounter this problem? I thought Cassandra 2.1.2 had solved this
counter problem, but it appeared.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8154) desc table output shows key-only index ambiguously


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310355#comment-14310355
 ] 

Tyler Hobbs commented on CASSANDRA-8154:


Relevant python driver ticket: 
https://datastax-oss.atlassian.net/browse/PYTHON-222

 desc table output shows key-only index ambiguously
 --

 Key: CASSANDRA-8154
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8154
 Project: Cassandra
  Issue Type: Bug
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Minor
  Labels: cqlsh

 When creating a secondary index on a map type, for keys, the DESC TABLE 
 output does not create correct DDL (it omits the keys part). So if someone 
 uses describe to recreate a schema they could end up with a values index 
 instead of a keys index.
 First, create a table and add an index:
 {noformat}
 CREATE TABLE test.foo (
 id1 text,
 id2 text,
 categories maptext, text,
 PRIMARY KEY (id1, id2));
 create index on foo(keys(categories));|
 {noformat}
 Now DESC TABLE and you'll see the incomplete index DDL:
 {noformat}
 CREATE TABLE test.foo (
 id1 text,
 id2 text,
 categories maptext, text,
 PRIMARY KEY (id1, id2)
 ) WITH CLUSTERING ORDER BY (id2 ASC)
 ...snip..
 CREATE INDEX foo_categories_idx ON test.foo (categories);
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

2015-02-06 Thread Wei Deng (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309634#comment-14309634
]

Wei Deng commented on CASSANDRA-8754:
-

+1 to Ryan. Especially when we are talking about reviewing code in all
languages the Cassandra drivers support.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

2015-02-06 Thread Chris Lohfink (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309670#comment-14309670
]

Chris Lohfink commented on CASSANDRA-8754:
--

Would an auditing tool be good then? Something like
{code}
Keyspace.CF Top Consistency Levels:
ONE 100
QUORUM 2
LOCAL_ONE 0

{code}
could give more context to how they are used as well.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla
Labels: ponies

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer

2015-02-06 Thread Akshat Aranya (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309717#comment-14309717
 ] 

Akshat Aranya edited comment on CASSANDRA-8723 at 2/6/15 7:31 PM:
--

We have a Cassandra 2.1.2 cluster running where the Cassandra Java processes 
are running very high on VIRT memory size:

from top:
{code}
 20797 cass_ic_  18  -2  384g 9.2g  24m S 1947.9  7.3 455:06.82 java
{code}




was (Author: aaranya):
We have a Cassandra 2.1 cluster running where the Cassandra Java processes are 
running very high on VIRT memory size:

from top:
{code}
 20797 cass_ic_  18  -2  384g 9.2g  24m S 1947.9  7.3 455:06.82 java
{code}



 Cassandra 2.1.2 Memory issue - java process memory usage continuously 
 increases until process is killed by OOM killer
 -

 Key: CASSANDRA-8723
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu
 Fix For: 2.1.3

 Attachments: cassandra.yaml


 Issue:
 We have an on-going issue with cassandra nodes running with continuously 
 increasing memory until killed by OOM.
 {noformat}
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
 process 13919 (java) score 911 or sacrifice child
 Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
 (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
 {noformat}
 System Profile:
 cassandra version 2.1.2
 system: aws c1.xlarge instance with 8 cores, 7.1G memory.
 cassandra jvm:
 -Xms1792M -Xmx1792M -Xmn400M -Xss256k
 {noformat}
 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
 -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
 -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
 -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
 -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
 -Dcom.sun.management.jmxremote.ssl=false 
 -Dcom.sun.management.jmxremote.authenticate=false 
 -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
  -Dlogback.configurationFile=logback.xml 
 -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
 -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp

[jira] [Commented] (CASSANDRA-7216) Restricted superuser account request

2015-02-06 Thread Sam Tunnicliffe (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308963#comment-14308963
 ] 

Sam Tunnicliffe commented on CASSANDRA-7216:


Updated following commit of CASSANDRA-8650 - new version is 
[here|https://github.com/beobal/cassandra/tree/7216]

 Restricted superuser account request
 

 Key: CASSANDRA-7216
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7216
 Project: Cassandra
  Issue Type: Improvement
Reporter: Oded Peer
Assignee: Sam Tunnicliffe
Priority: Minor
 Fix For: 3.0

 Attachments: 7216-8650.txt, 7216-POC.txt, 7216.txt


 I am developing a multi-tenant service.
 Every tenant has its own user, keyspace and can access only his keyspace.
 As new tenants are provisioned there is a need to create new users and 
 keyspaces.
 Only a superuser can issue CREATE USER requests, so we must have a super user 
 account in the system. On the other hand super users have access to all the 
 keyspaces, which poses a security risk.
 For tenant provisioning I would like to have a restricted account which can 
 only create new users, without read access to keyspaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-8752) invalid counter shard detected in Version 2.1.2


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams reassigned CASSANDRA-8752:
---

Assignee: Aleksey Yeschenko

 invalid counter shard detected in Version 2.1.2
 ---

 Key: CASSANDRA-8752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: SUSE11 SP1, Cassandra 2.1.2, java version 1.7.0_55.
 4 node cluster, vnode = 1, replication = 2
Reporter: Kevin Ye
Assignee: Aleksey Yeschenko

 I was doing counter test (first +100 several times, then -33) on a 4 nodes 
 cluster while below log appear at 2 nodes.There is no concurrent access to 
 same counter.
 WARN  [CompactionExecutor:757] 2015-02-02 13:02:33,375 
 CounterContext.java:431 - invalid global counter shard detected; 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, -33) and 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, 100) differ only in count; will 
 pick highest to self-heal on compaction
 Anyone has encounter this problem? I thought Cassandra 2.1.2 had solved this 
 counter problem, but it appeared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8752) invalid counter shard detected in Version 2.1.2


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309040#comment-14309040
 ] 

Aleksey Yeschenko commented on CASSANDRA-8752:
--

It did solve it, yes (although you'll still potentially see if if you lose an 
sstable, for example, which is why that warning exists in code in the first 
place).

You'll also see it if the read-before-write to update the counter returned 
nothing. So it could be one of our 2.1 sstable related issues.

I suggest you try it again with Cassandra 2.1.3 once it's released (soon), or 
build from cassandra-2.1 branch directly, and see if you can reproduce it.

 invalid counter shard detected in Version 2.1.2
 ---

 Key: CASSANDRA-8752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: SUSE11 SP1, Cassandra 2.1.2, java version 1.7.0_55.
 4 node cluster, vnode = 1, replication = 2
Reporter: Kevin Ye
Assignee: Aleksey Yeschenko

 I was doing counter test (first +100 several times, then -33) on a 4 nodes 
 cluster while below log appear at 2 nodes.There is no concurrent access to 
 same counter.
 WARN  [CompactionExecutor:757] 2015-02-02 13:02:33,375 
 CounterContext.java:431 - invalid global counter shard detected; 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, -33) and 
 (9cca9262-934a-4275-963b-66802471b0c2, 1, 100) differ only in count; will 
 pick highest to self-heal on compaction
 Anyone has encounter this problem? I thought Cassandra 2.1.2 had solved this 
 counter problem, but it appeared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-8752) invalid counter shard detected in Version 2.1.2

2015-02-06 Thread Kevin Ye (JIRA)

Kevin Ye created CASSANDRA-8752:
---

 Summary: invalid counter shard detected in Version 2.1.2
 Key: CASSANDRA-8752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: SUSE11 SP1, Cassandra 2.1.2, java version 1.7.0_55.
4 node cluster, vnode = 1, replication = 2
Reporter: Kevin Ye


I was doing counter test (first +100 several times, then -33) on a 4 nodes 
cluster while below log appear at 2 nodes.There is no concurrent access to same 
counter.

WARN  [CompactionExecutor:757] 2015-02-02 13:02:33,375 CounterContext.java:431 
- invalid global counter shard detected; (9cca9262-934a-4275-963b-66802471b0c2, 
1, -33) and (9cca9262-934a-4275-963b-66802471b0c2, 1, 100) differ only in 
count; will pick highest to self-heal on compaction

Anyone has encounter this problem? I thought Cassandra 2.1.2 had solved this 
counter problem, but it appeared.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

[
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308989#comment-14308989
]

Marcus Eriksson commented on CASSANDRA-8707:

I think the penny has finally dropped, the thing I'm missing now is more
why?-comments, for example, why do we need to copy CompressionMetadata in
CM.Writer#open(..) if finish type is final and not otherwise? In general, more
why?-comments than what? would be helpful, at least around the tricky
parts, I think future readers will appreciate that.

Also, I think we need some high level documentation with concrete examples
about the life cycle of sstables. I think this part of the code would be much
simpler to maintain with a description of why and when the different cases
occur.

So, I'm +1 once we have some documentation (perhaps as a big comment on top in
SSTableReader?), and a bit of a tweak to the current code comments to focus
more on why we do certain things.

Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

Key: CASSANDRA-8707
URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
Project: Cassandra
Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
Fix For: 2.1.3

There are still a few bugs with resource management, especially around
SSTableReader cleanup, esp. when intermixing with compaction. This migration
should help. We can simultaneously simplify the logic in SSTableReader to
not track the replacement chain, only to take a new reference to each of the
underlying resources.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-8753) Support UPDATE IF NOT EXIST OR compare checks in LWT transactions

2015-02-06 Thread Gerrit Jansen van Vuuren (JIRA)

Gerrit Jansen van Vuuren created CASSANDRA-8753:
---

 Summary: Support UPDATE IF NOT EXIST OR compare checks in LWT 
transactions
 Key: CASSANDRA-8753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8753
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Gerrit Jansen van Vuuren


I've got a column that I want to update e.g  min = x IF min  x
but if there is no value for min e.g min == null then the update will never 
succeed, which means I need to do a READ then INSERT and then run UPDATES which 
defies the purpose of using the LWT for this use case.
is it possible for LWT to support either IF min  x OR min == null ?  or 
otherwise take null as equal to zero for numbers ?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

[
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308989#comment-14308989
]

Marcus Eriksson edited comment on CASSANDRA-8707 at 2/6/15 11:21 AM:
-

I think the penny has finally dropped for me, the thing I'm missing now is more
why?-comments, for example, why do we need to copy CompressionMetadata in
CM.Writer#open(..) if finish type is final and not otherwise? In general, more
why?-comments than what? would be helpful, at least around the tricky
parts, I think future readers will appreciate that.

So, I'm +1 once we have some documentation (perhaps as a big comment on top in
SSTableReader?), and a bit of a tweak to the current code comments to focus
more on why we do certain things.

was (Author: krummas):
I think the penny has finally dropped, the thing I'm missing now is more
why?-comments, for example, why do we need to copy CompressionMetadata in
CM.Writer#open(..) if finish type is final and not otherwise? In general, more
why?-comments than what? would be helpful, at least around the tricky
parts, I think future readers will appreciate that.

So, I'm +1 once we have some documentation (perhaps as a big comment on top in
SSTableReader?), and a bit of a tweak to the current code comments to focus
more on why we do certain things.

Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

Key: CASSANDRA-8707
URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
Project: Cassandra
Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7032) Improve vnode allocation


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309190#comment-14309190
 ] 

Benedict commented on CASSANDRA-7032:
-

Before going ahead and integrating it, could we confirm that the algorithm 
performs well under heterogenous data centre / rack configurations, by which I 
mean different numbers of nodes in each data centre / rack (not differing 
numbers of vnodes), for varying numbers of data centres and racks? And that it, 
most importantly, evenly distributes load across the disks within any single 
node?

It would be great to perform a simulation of a wide range of state spaces, 
along each dimension of: 

# number of data centres
# size variability of data centres
# number of disks per node
# variability of number of disks (perhaps only varying between data centres)
# number of racks within a data centre

And graph the variability of ownership _by disk_. Getting balanced disks is the 
really hard thing to do, and is really essential for CASSANDRA-6696. Data 
centres may also behave unexpectedly for replication; you may own an even 
amount of the ring, but it may translate into a disproportionately contiguous 
range within the data centre, so you end up owning an uneven quantity within 
the data centre.

 Improve vnode allocation
 

 Key: CASSANDRA-7032
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7032
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Benedict
Assignee: Branimir Lambov
  Labels: performance, vnodes
 Fix For: 3.0

 Attachments: TestVNodeAllocation.java, TestVNodeAllocation.java, 
 TestVNodeAllocation.java, TestVNodeAllocation.java, TestVNodeAllocation.java, 
 TestVNodeAllocation.java


 It's been known for a little while that random vnode allocation causes 
 hotspots of ownership. It should be possible to improve dramatically on this 
 with deterministic allocation. I have quickly thrown together a simple greedy 
 algorithm that allocates vnodes efficiently, and will repair hotspots in a 
 randomly allocated cluster gradually as more nodes are added, and also 
 ensures that token ranges are fairly evenly spread between nodes (somewhat 
 tunably so). The allocation still permits slight discrepancies in ownership, 
 but it is bound by the inverse of the size of the cluster (as opposed to 
 random allocation, which strangely gets worse as the cluster size increases). 
 I'm sure there is a decent dynamic programming solution to this that would be 
 even better.
 If on joining the ring a new node were to CAS a shared table where a 
 canonical allocation of token ranges lives after running this (or a similar) 
 algorithm, we could then get guaranteed bounds on the ownership distribution 
 in a cluster. This will also help for CASSANDRA-6696.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

[
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309214#comment-14309214
]

Benedict commented on CASSANDRA-8707:
-

bq. I just want a small 'why' instead of a comment stating the obvious:

I'm honestly not meaning to be obtuse here, I'm just looking for help. For some
of this code I struggle(d) to see how I could make it clearer, so just dropped
in a few comments willynilly. In your latter example, I do my best to explain
the why at the top of each of the Tidy classes, and especially the
InstanceTidier, that outlines the hierarchy. Here I was just duplicating some
information so that the concept that was being related to was directly
mentioned. This is why I ask for guidance, because whilst it may _now_ be clear
to you it wasn't before, but conversely it is not at all clear to me what
_isn't/wasn't_ clear, and I really need some help knowing where to fill in. The
why comments seem to already be there for me in that second example, excepting
the class-level outline of the lifecycle.

Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

Key: CASSANDRA-8707
URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
Project: Cassandra
Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309217#comment-14309217
 ] 

Marcus Eriksson commented on CASSANDRA-8707:


As I said, I'll walk over the code properly and explain the parts I struggled 
with.

I think the problem is that once you spend the time to understand it, it is 
obvious

 Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted
 

 Key: CASSANDRA-8707
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
 Project: Cassandra
  Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.3


 There are still a few bugs with resource management, especially around 
 SSTableReader cleanup, esp. when intermixing with compaction. This migration 
 should help. We can simultaneously simplify the logic in SSTableReader to 
 not track the replacement chain, only to take a new reference to each of the 
 underlying resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309226#comment-14309226
 ] 

Benedict commented on CASSANDRA-8707:
-

bq. As I said, I'll walk over the code properly and explain the parts I 
struggled with.

Thanks. I do want the code to be clear, and would also like to contribute to 
that where I am able, so if there are specific things you think I can elaborate 
on better, please do point me to them and leave it to me. 

Like you say, the code is made up of a lot of individually obvious things, that 
aren't obvious as a whole, and I've been lost in the weeds long enough to not 
be able to tell the difference between the two.

 Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted
 

 Key: CASSANDRA-8707
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
 Project: Cassandra
  Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.3


 There are still a few bugs with resource management, especially around 
 SSTableReader cleanup, esp. when intermixing with compaction. This migration 
 should help. We can simultaneously simplify the logic in SSTableReader to 
 not track the replacement chain, only to take a new reference to each of the 
 underlying resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7032) Improve vnode allocation

2015-02-06 Thread Branimir Lambov (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Branimir Lambov updated CASSANDRA-7032:
---
Attachment: TestVNodeAllocation.java

Attached new version that does not depend on a fixed number of vnodes per
token. It distributes the load proportionally to the number of tokens allocated
to the node and aims to minimize the over/underutilization as a percentage of
the node's load.

The scheme is the same as the above, only in order to take varying loads into
account the load of nodes and tokens in the improvement calculation is
normalized via the inverse of the size of the node.

The results achieved in asymmetric node configurations are very similar to the
homogeneous scenario. The behaviour for fixed vnode counts is exactly the same
as the previous version.

I am done tinkering with the algorithm and have started work to attach it to
the code base.

Improve vnode allocation

Key: CASSANDRA-7032
URL: https://issues.apache.org/jira/browse/CASSANDRA-7032
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Benedict
Assignee: Branimir Lambov
Labels: performance, vnodes
Fix For: 3.0

Attachments: TestVNodeAllocation.java, TestVNodeAllocation.java,
TestVNodeAllocation.java, TestVNodeAllocation.java, TestVNodeAllocation.java,
TestVNodeAllocation.java

It's been known for a little while that random vnode allocation causes
hotspots of ownership. It should be possible to improve dramatically on this
with deterministic allocation. I have quickly thrown together a simple greedy
algorithm that allocates vnodes efficiently, and will repair hotspots in a
randomly allocated cluster gradually as more nodes are added, and also
ensures that token ranges are fairly evenly spread between nodes (somewhat
tunably so). The allocation still permits slight discrepancies in ownership,
but it is bound by the inverse of the size of the cluster (as opposed to
random allocation, which strangely gets worse as the cluster size increases).
I'm sure there is a decent dynamic programming solution to this that would be
even better.
If on joining the ring a new node were to CAS a shared table where a
canonical allocation of token ranges lives after running this (or a similar)
algorithm, we could then get guaranteed bounds on the ownership distribution
in a cluster. This will also help for CASSANDRA-6696.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309166#comment-14309166
 ] 

Marcus Eriksson commented on CASSANDRA-8707:


bq. It would be helpful if, once I've introdiced a high-level overview, the 
individuals familiar with the tool or special compaction lifecycles (or others) 
could amend the SSTableReader description
Absolutely, for now I think that a concrete explanation of what happens to an 
sstable that is early opened and one that is cloneWithNewStart(..):ed is enough

bq. Could you list the places you think need this service? Since the code 
outside of SSTableReader tidying is largely functionally unchanged
I don't think *I* need the service, it is mostly to avoid future confusion, and 
I figured now was a good time to clear up the confusing bits since we are 
touching those code parts (when would we do that otherwise?). I know most of 
the code was just moved around from other places and it was terrible before 
this, but that doesn't really stop us from making it easier for the future.

I just want a small 'why' instead of a comment stating the obvious:
{code}
// we don't want to alert deletions if not final
{code}
{code}
// get a new reference to the shared descriptor-type tidy
{code}

Those are just two examples, once the life cycle description is done, I can add 
the comments about the details I found confusing

 Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted
 

 Key: CASSANDRA-8707
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
 Project: Cassandra
  Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.3


 There are still a few bugs with resource management, especially around 
 SSTableReader cleanup, esp. when intermixing with compaction. This migration 
 should help. We can simultaneously simplify the logic in SSTableReader to 
 not track the replacement chain, only to take a new reference to each of the 
 underlying resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309166#comment-14309166
 ] 

Marcus Eriksson edited comment on CASSANDRA-8707 at 2/6/15 2:16 PM:


bq. It would be helpful if, once I've introdiced a high-level overview, the 
individuals familiar with the tool or special compaction lifecycles (or others) 
could amend the SSTableReader description
Absolutely, for now I think that a concrete explanation of what happens to an 
sstable that is early opened and one that is cloneWithNewStart(..):ed is enough

bq. Could you list the places you think need this service? Since the code 
outside of SSTableReader tidying is largely functionally unchanged
It is mostly to avoid future confusion, and I figured now was a good time to 
clear up the confusing bits since we are touching those code parts (when would 
we do that otherwise?). I know most of the code was just moved around from 
other places and it was terrible before this, but that doesn't really stop us 
from making it easier for the future.

I just want a small 'why' instead of a comment stating the obvious:
{code}
// we don't want to alert deletions if not final
{code}
{code}
// get a new reference to the shared descriptor-type tidy
{code}

Those are just two examples, once the life cycle description is done, I can add 
the comments about the details I found confusing


was (Author: krummas):
bq. It would be helpful if, once I've introdiced a high-level overview, the 
individuals familiar with the tool or special compaction lifecycles (or others) 
could amend the SSTableReader description
Absolutely, for now I think that a concrete explanation of what happens to an 
sstable that is early opened and one that is cloneWithNewStart(..):ed is enough

bq. Could you list the places you think need this service? Since the code 
outside of SSTableReader tidying is largely functionally unchanged
I don't think *I* need the service, it is mostly to avoid future confusion, and 
I figured now was a good time to clear up the confusing bits since we are 
touching those code parts (when would we do that otherwise?). I know most of 
the code was just moved around from other places and it was terrible before 
this, but that doesn't really stop us from making it easier for the future.

I just want a small 'why' instead of a comment stating the obvious:
{code}
// we don't want to alert deletions if not final
{code}
{code}
// get a new reference to the shared descriptor-type tidy
{code}

Those are just two examples, once the life cycle description is done, I can add 
the comments about the details I found confusing

 Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted
 

 Key: CASSANDRA-8707
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
 Project: Cassandra
  Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.3


 There are still a few bugs with resource management, especially around 
 SSTableReader cleanup, esp. when intermixing with compaction. This migration 
 should help. We can simultaneously simplify the logic in SSTableReader to 
 not track the replacement chain, only to take a new reference to each of the 
 underlying resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309060#comment-14309060
]

Aleksey Yeschenko commented on CASSANDRA-7688:
--

You are quoting the wrong code here, but how do you *not* background it? It's
not strictly about cost, it's about not having any other triggering mechanism.

When we add vtable support (cql tables backed by classes, not sstables) - then
we'll switch sizing (and several other system sstables) to that. Until then,
what other options do we have?

This is a simple temporary replacement for describe_splits_ex, its *only* goal
is to free Spark and others from having to maintain an extra Thrift connection
*now*. Hence the lack of metrics or configurability of the refresh interval.

I'm open to increasing/decreasing the hard-coded one, however, if you have
better options.

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Assignee: Aleksey Yeschenko
Fix For: 2.1.3

Attachments: 7688.txt

Currently you can't implement something similar to describe_splits_ex purely
from the a native protocol driver.
https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily
getting ownership information to a client in the java-driver. But you still
need the data sizing part to get splits of a given size. We should add the
sizing information to a system table so that native clients can get to it.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table

2015-02-06 Thread mck (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309077#comment-14309077
]

mck commented on CASSANDRA-7688:

{quote}You are quoting the wrong code here, but how do you not background it?
{quote}

yes i can see that it's not possible at the moment. (i didn't realise that at
first, but it really wasn't my train of thought either).

{quote}When we add vtable support (cql tables backed by classes, not sstables)
- then we'll switch sizing (and several other system sstables) to that.{quote}

niceto know. thanks.

{quote}This is a simple temporary replacement for describe_splits_ex, its only
goal is to free Spark and others from having to maintain an extra Thrift
connection now. Hence the lack of metrics or configurability of the refresh
interval.

I'm open to increasing/decreasing the hard-coded one, however, if you have
better options.{quote}

i have no suggestion.
i'm more concerned/curious as to why 5 minutes?
if there's no good answer then isn't metrics important?
and being able to configure it.

quick examples that come to mind:
- what if an installation has lots of jobs built upon each others data and for
them there's a strong benefit (if not a requirement) for more accurate sizes
(ie faster schedule rate),
- what if there bugs/load caused from this that can be avoided by configuring
it to zero (disabling), giving an immediate option to upgrading-to/waiting-for
next version.

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Assignee: Aleksey Yeschenko
Fix For: 2.1.3

Attachments: 7688.txt

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: remove dead code

2015-02-06 Thread dbrosius

Repository: cassandra
Updated Branches:
  refs/heads/trunk 217721ae9 - ca41972ea


remove dead code


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ca41972e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ca41972e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ca41972e

Branch: refs/heads/trunk
Commit: ca41972ea011ee9e082c02afb1c37e1151b8cd2d
Parents: 217721a
Author: Dave Brosius dbros...@mebigfatguy.com
Authored: Fri Feb 6 08:13:44 2015 -0500
Committer: Dave Brosius dbros...@mebigfatguy.com
Committed: Fri Feb 6 08:13:44 2015 -0500

--
 .../org/apache/cassandra/service/StorageServiceServerTest.java | 2 --
 1 file changed, 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ca41972e/test/unit/org/apache/cassandra/service/StorageServiceServerTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/service/StorageServiceServerTest.java 
b/test/unit/org/apache/cassandra/service/StorageServiceServerTest.java
index d1a4d85..a20fa6c 100644
--- a/test/unit/org/apache/cassandra/service/StorageServiceServerTest.java
+++ b/test/unit/org/apache/cassandra/service/StorageServiceServerTest.java
@@ -476,8 +476,6 @@ public class StorageServiceServerTest
 metadata.updateNormalToken(new LongToken(3000L), 
InetAddress.getByName(127.0.0.3));
 metadata.updateNormalToken(new LongToken(4000L), 
InetAddress.getByName(127.0.0.4));
 
-MapString, String configOptions = new HashMapString, String();
-configOptions.put(replication_factor, 3);
 CollectionRangeToken repairRangeFrom = 
StorageService.instance.createRepairRangeFrom(1500, 3700);
 assert repairRangeFrom.size() == 3;
 assert repairRangeFrom.contains(new RangeToken(new LongToken(1500L), 
new LongToken(2000L)));

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309081#comment-14309081
 ] 

Aleksey Yeschenko commented on CASSANDRA-7688:
--

All good points. I'll add a -D option to change the interval/disable it.

 Add data sizing to a system table
 -

 Key: CASSANDRA-7688
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
 Project: Cassandra
  Issue Type: New Feature
Reporter: Jeremiah Jordan
Assignee: Aleksey Yeschenko
 Fix For: 2.1.3

 Attachments: 7688.txt


 Currently you can't implement something similar to describe_splits_ex purely 
 from the a native protocol driver.  
 https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily 
 getting ownership information to a client in the java-driver.  But you still 
 need the data sizing part to get splits of a given size.  We should add the 
 sizing information to a system table so that native clients can get to it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8722) Auth MBean needs to be registered


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-8722:

Attachment: 8722.txt

Patch to rework both of these after CASSANDRA-8194, and register the mbean.

 Auth MBean needs to be registered
 -

 Key: CASSANDRA-8722
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8722
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Minor
 Fix For: 2.1.3, 2.0.13

 Attachments: 8722.txt


 In CASSANDRA-7968 we created this bean but forgot to register it :(  This 
 also makes CASSANDRA-7977 unusable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8744) Ensure SSTableReader.first/last are honoured universally

2015-02-06 Thread Alan Boudreault (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309384#comment-14309384
 ] 

Alan Boudreault commented on CASSANDRA-8744:


[~benedict] It doesn't look like an issue I can easily reproduce with a 
specific use case . Am I correct, or I sould be able to reproduce that in some 
way?

 Ensure SSTableReader.first/last are honoured universally
 

 Key: CASSANDRA-8744
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8744
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.4


 Split out from CASSANDRA-8683; we don't honour the first/last properties of 
 an sstablereader, and we tend to assume that we do. This can cause problems 
 in LCS validation compactions, for instance, where a scanner is assumed to 
 only cover the defined range, but may return data either side of that range. 
 In general it is only wasteful to not honour these ranges anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7032) Improve vnode allocation

2015-02-06 Thread Branimir Lambov (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309383#comment-14309383
]

Branimir Lambov commented on CASSANDRA-7032:

The answer relies on a couple of mappings:
- a Cassandra data centre is a separate ring to the algorithm. Data centres
have their own allocation of replicas, hence one can only balance load within
them, not across multiple DCs. The algorithm can simply ignore everything
outside the data centre where the new node is spawned.
- in current Cassandra terms, a node is a node to the algorithm.
- in future Cassandra terms, a disk is a node to the algorithm*.
- a Cassandra rack is a rack to the algorithm if there number within the data
centre is equal to or higher than the replication factor, otherwise the
Cassandra node is the rack of the algorithm. Implicitly all disks in a machine
belong to the same rack; the machine level can be completely ignored if racks
are defined.

The only fine point is starred. I know your preference is to split the token
space in a single range per disk. I see little benefit in this compared to
simply assigning a subset of the machine's vnodes to each disk. The latter is
more flexible, perhaps even easier to achieve when SSTables are per vnode.
(Perhaps what's not so easy to see is that each vnode is always responsible for
replicating a contiguous range of tokens and it is easy to convert between a
vnode and the token range that it serves (see
{{ReplicationStrategy.replicationStart}} in the attached).) Slower disks take a
small number of vnodes (4?), faster ones take more (8/16?). Bigger machines
have more disks hence more vnodes and automatically take a higher load.

The algorithm handles heterogeneous rack and node configuration without issues
as long as the number of distinct racks (in algorithm terms) is higher than the
replication factor.

Improve vnode allocation

Attachments: TestVNodeAllocation.java, TestVNodeAllocation.java,
TestVNodeAllocation.java, TestVNodeAllocation.java, TestVNodeAllocation.java,
TestVNodeAllocation.java

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8744) Ensure SSTableReader.first/last are honoured universally


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309389#comment-14309389
 ] 

Benedict commented on CASSANDRA-8744:
-

Without this, CASSANDRA-6863 is not fixed, and you should find at the very 
least LeveledCompactionStrategyTest.testValidationMultipleSSTablePerLevel fails 
periodically. What other implications there are, I'm not sure.

 Ensure SSTableReader.first/last are honoured universally
 

 Key: CASSANDRA-8744
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8744
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.4


 Split out from CASSANDRA-8683; we don't honour the first/last properties of 
 an sstablereader, and we tend to assume that we do. This can cause problems 
 in LCS validation compactions, for instance, where a scanner is assumed to 
 only cover the defined range, but may return data either side of that range. 
 In general it is only wasteful to not honour these ranges anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8744) Ensure SSTableReader.first/last are honoured universally

2015-02-06 Thread Alan Boudreault (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Boudreault updated CASSANDRA-8744:
---
Tester: Alan Boudreault

 Ensure SSTableReader.first/last are honoured universally
 

 Key: CASSANDRA-8744
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8744
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Benedict
Assignee: Benedict
Priority: Critical
 Fix For: 2.1.4


 Split out from CASSANDRA-8683; we don't honour the first/last properties of 
 an sstablereader, and we tend to assume that we do. This can cause problems 
 in LCS validation compactions, for instance, where a scanner is assumed to 
 only cover the defined range, but may return data either side of that range. 
 In general it is only wasteful to not honour these ranges anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8707) Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

[
https://issues.apache.org/jira/browse/CASSANDRA-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309098#comment-14309098
]

Benedict commented on CASSANDRA-8707:
-

bq. Also, I think we need some high level documentation with concrete examples
about the life cycle of sstables. ... (perhaps as a big comment on top in
SSTableReader?)

This seems like an excellent idea, and this ticket seems like a good
opportunity to introduce it, although it will have to be revised with each of
my follow ons. It didn't really occur to me to do this since that is all
unchanged here, but it has grown overtime to be very non-obvious. It would be
helpful if, once I've introdiced a high-level overview, the individuals
familiar with the _tool_ or special compaction lifecycles (or others) could
amend the SSTableReader description to make it clear how they are expected to
behave and treat sstables, and what assumptions they utilise, as that is often
subtly different and I am not at all an expert in those areas.

bq. I think the penny has finally dropped for me, the thing I'm missing now is
more why?-comments, for example, why do we need to copy CompressionMetadata
in CM.Writer#open(..) if finish type is final and not otherwise?

Could you list the places you think need this service? Since the code outside
of SSTableReader tidying is largely functionally unchanged, only swapping in a
new API, it's not clear which bits need this, outside of the high level
overview. This particular spot, for instance, had this behaviour previously
(and it's about resizing, rather than copying, since we now know the final size
of the data so can release it). One of the follow tickets also cleans this
particular spot up a little, by introducing an enum specific to it that
describes the reasons for opening (which should make that more obvious). I'm
hoping a small follow up I've yet to publish will also simplify that particular
bit further. I'm more than happy to flesh out comments, but would appreciate an
outline of which bits you think have been lacking historically.

Move SegmentedFile, IndexSummary and BloomFilter to utilising RefCounted

Key: CASSANDRA-8707
URL: https://issues.apache.org/jira/browse/CASSANDRA-8707
Project: Cassandra
Issue Type: Bug
Reporter: Benedict
Assignee: Benedict
Priority: Critical
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot

2015-02-06 Thread Sucwinder Bassi (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309046#comment-14309046
 ] 

Sucwinder Bassi commented on CASSANDRA-8696:


We've also seen this error appear in the system.log for a node running C* 
2.0.11.82.

 nodetool repair on cassandra 2.1.2 keyspaces return 
 java.lang.RuntimeException: Could not create snapshot
 -

 Key: CASSANDRA-8696
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
 Project: Cassandra
  Issue Type: Bug
Reporter: Jeff Liu

 When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra 
 throw java exceptions: cannot create snapshot. 
 the error log from system.log:
 {noformat}
 INFO  [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815 
 StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227 
 ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5 
 files(632105 bytes)
 INFO  [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 
 StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227] 
 Session with /10.97.9.110 is complete
 INFO  [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 
 StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227] 
 All sessions completed
 INFO  [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047 
 StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] 
 streaming task succeed, returning response to /10.98.194.68
 INFO  [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 - 
 [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for 
 Repair
 INFO  [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065 
 StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227] 
 Starting streaming to /10.66.187.201
 INFO  [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070 
 StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227, 
 ID#0] Beginning stream session with /10.66.187.201
 INFO  [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465 
 StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227 
 ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5 
 files(632105 bytes)
 INFO  [StreamReceiveTask:22] 2015-01-28 02:07:31,971 
 StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227] 
 Session with /10.66.187.201 is complete
 INFO  [StreamReceiveTask:22] 2015-01-28 02:07:31,972 
 StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227] 
 All sessions completed
 INFO  [StreamReceiveTask:22] 2015-01-28 02:07:31,972 
 StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] 
 streaming task succeed, returning response to /10.98.194.68
 ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error 
 occurred during snapshot phase
 java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
 at 
 org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347) 
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 ~[na:1.7.0_45]
 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 ~[na:1.7.0_45]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  [na:1.7.0_45]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_45]
 at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
 INFO  [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260 
 - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync 
 /10.98.194.68, /10.66.187.201, /10.226.218.135 on range 
 (12817179804668051873746972069086
 2638799,12863540308359254031520865977436165] for events.[bigint0text, 
 bigint0boolean, bigint0int, dataset_catalog, column_categories, 
 bigint0double, bigint0bigint]
 ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303 
 - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the 
 following error
 java.io.IOException: Failed during snapshot creation.
 at 
 org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
  ~[apache-cassandra-2.1.2.jar:2.1.2]
 at 
 org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128) 
 ~[apache-cassandra-2.1.2.jar:2.1.2]
 at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) 
 ~[guava-16.0.jar:na]
 at

[jira] [Comment Edited] (CASSANDRA-7688) Add data sizing to a system table

2015-02-06 Thread mck (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309077#comment-14309077
 ] 

mck edited comment on CASSANDRA-7688 at 2/6/15 12:59 PM:
-

{quote}You are quoting the wrong code here, but how do you not background it? 
{quote}

yes i can see that it's not possible at the moment. (i didn't realise that at 
first, but it really wasn't my train of thought either).

{quote}When we add vtable support (cql tables backed by classes, not sstables) 
- then we'll switch sizing (and several other system sstables) to that.{quote}

niceto know. thanks.

{quote}This is a simple temporary replacement for describe_splits_ex, its only 
goal is to free Spark and others from having to maintain an extra Thrift 
connection now. Hence the lack of metrics or configurability of the refresh 
interval.

I'm open to increasing/decreasing the hard-coded one, however, if you have 
better options.{quote}

i have no suggestion.
i'm more concerned/curious as to why 5 minutes?
 if there's no good answer then isn't metrics important?
 and being able to configure it.

quick examples that come to mind: 
 - what if an installation has lots of jobs built upon each others data and for 
them there's a strong benefit (if not a requirement) for more accurate sizes 
(ie faster schedule rate),
 - what if there's bugs/load caused from this that can be avoided (for an 
installation that doesn't ever use hadoop/spark) by configuring it to zero 
(disabling), giving an immediate option to upgrading-to/waiting-for next 
version.


was (Author: michaelsembwever):
{quote}You are quoting the wrong code here, but how do you not background it? 
{quote}

yes i can see that it's not possible at the moment. (i didn't realise that at 
first, but it really wasn't my train of thought either).

{quote}When we add vtable support (cql tables backed by classes, not sstables) 
- then we'll switch sizing (and several other system sstables) to that.{quote}

niceto know. thanks.

{quote}This is a simple temporary replacement for describe_splits_ex, its only 
goal is to free Spark and others from having to maintain an extra Thrift 
connection now. Hence the lack of metrics or configurability of the refresh 
interval.

I'm open to increasing/decreasing the hard-coded one, however, if you have 
better options.{quote}

i have no suggestion.
i'm more concerned/curious as to why 5 minutes?
 if there's no good answer then isn't metrics important?
 and being able to configure it.

quick examples that come to mind: 
 - what if an installation has lots of jobs built upon each others data and for 
them there's a strong benefit (if not a requirement) for more accurate sizes 
(ie faster schedule rate),
 - what if there bugs/load caused from this that can be avoided by configuring 
it to zero (disabling), giving an immediate option to upgrading-to/waiting-for 
next version.

 Add data sizing to a system table
 -

 Key: CASSANDRA-7688
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
 Project: Cassandra
  Issue Type: New Feature
Reporter: Jeremiah Jordan
Assignee: Aleksey Yeschenko
 Fix For: 2.1.3

 Attachments: 7688.txt


 Currently you can't implement something similar to describe_splits_ex purely 
 from the a native protocol driver.  
 https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily 
 getting ownership information to a client in the java-driver.  But you still 
 need the data sizing part to get splits of a given size.  We should add the 
 sizing information to a system table so that native clients can get to it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-8754) Required consistency level

Ryan Svihla created CASSANDRA-8754:
--

 Summary: Required consistency level
 Key: CASSANDRA-8754
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
 Project: Cassandra
  Issue Type: New Feature
Reporter: Ryan Svihla


Based on restrictions prevent a query based on a query level not being met.

Lots of customers struggle with getting all their dev teams on board with 
consistency levels and all the ramifications. The normal solution for this has 
traditionally to build a service in front of Cassandra that the entire dev team 
accesses. However, this has proven challenging for some organizations to do 
correctly, and I think an easier approach would be to require a given 
consistency level as a matter of enforced policy in the database. 

I'm open for where this belongs. The most flexible approach is at a table 
level, however I'm concerned this is potentially error prone and labor 
intensive. It could be a table attribute similar to compaction strategy.

The simplest administratively is a cluster level, in say the cassandra.yaml

The middle ground is at they keyspace level, the only downside I could foresee 
is keyspace explosion to fit involved minimum schemes. It could be a keyspace 
attribute such as replication strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8754) Required consistency level

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ryan Svihla updated CASSANDRA-8754:
---
Description:
Idea is to prevent a query based on a consistency level not being met.

Lots of customers struggle with getting all their dev teams on board with
consistency levels and all the ramifications. The normal solution for this has
traditionally to build a service in front of Cassandra that the entire dev team
accesses. However, this has proven challenging for some organizations to do
correctly, and I think an easier approach would be to require a given
consistency level as a matter of enforced policy in the database.

I'm open for where this belongs. The most flexible approach is at a table
level, however I'm concerned this is potentially error prone and labor
intensive. It could be a table attribute similar to compaction strategy.

The simplest administratively is a cluster level, in say the cassandra.yaml

The middle ground is at they keyspace level, the only downside I could foresee
is keyspace explosion to fit involved minimum schemes. It could be a keyspace
attribute such as replication strategy.

was:
Based on restrictions prevent a query based on a query level not being met.

The simplest administratively is a cluster level, in say the cassandra.yaml

The middle ground is at they keyspace level, the only downside I could foresee
is keyspace explosion to fit involved minimum schemes. It could be a keyspace
attribute such as replication strategy.

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla

Idea is to prevent a query based on a consistency level not being met.
Lots of customers struggle with getting all their dev teams on board with
consistency levels and all the ramifications. The normal solution for this
has traditionally to build a service in front of Cassandra that the entire
dev team accesses. However, this has proven challenging for some
organizations to do correctly, and I think an easier approach would be to
require a given consistency level as a matter of enforced policy in the
database.
I'm open for where this belongs. The most flexible approach is at a table
level, however I'm concerned this is potentially error prone and labor
intensive. It could be a table attribute similar to compaction strategy.
The simplest administratively is a cluster level, in say the cassandra.yaml
The middle ground is at they keyspace level, the only downside I could
foresee is keyspace explosion to fit involved minimum schemes. It could be a
keyspace attribute such as replication strategy.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8754) Required consistency level

2015-02-06 Thread Tupshin Harper (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309493#comment-14309493
]

Tupshin Harper commented on CASSANDRA-8754:
---

-1 on cluster level (too limiting, IMO), but big +1 for table level
restrictions of CL

Required consistency level
--

Key: CASSANDRA-8754
URL: https://issues.apache.org/jira/browse/CASSANDRA-8754
Project: Cassandra
Issue Type: New Feature
Reporter: Ryan Svihla

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8754) Required consistency level