[jira] [Updated] (CASSANDRA-8177) sequential repair is much more expensive than parallel repair
[ https://issues.apache.org/jira/browse/CASSANDRA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Jalkanen updated CASSANDRA-8177: -- Attachment: cassc-week.png sequential repair is much more expensive than parallel repair - Key: CASSANDRA-8177 URL: https://issues.apache.org/jira/browse/CASSANDRA-8177 Project: Cassandra Issue Type: Bug Reporter: Sean Bridges Assignee: Yuki Morishita Attachments: cassc-week.png, iostats.png This is with 2.0.10 The attached graph shows io read/write throughput (as measured with iostat) when doing repairs. The large hump on the left is a sequential repair of one node. The two much smaller peaks on the right are parallel repairs. This is a 3 node cluster using vnodes (I know vnodes on small clusters isn't recommended). Cassandra reports load of 40 gigs. We noticed a similar problem with a larger cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8177) sequential repair is much more expensive than parallel repair
[ https://issues.apache.org/jira/browse/CASSANDRA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183362#comment-14183362 ] Janne Jalkanen commented on CASSANDRA-8177: --- Kinda having the same problem - attaching the compactions graph from Munin. Started a serial compaction on 21st midnight, took something like 45 hrs. On 23rd and 24th we ran a repair with -par. It's a lot shorter and causes a lot less compaction traffic. This is with 2.0.10 from a production cluster with 9:1 read/write ratio, recently upgraded from 1.2.18. sequential repair is much more expensive than parallel repair - Key: CASSANDRA-8177 URL: https://issues.apache.org/jira/browse/CASSANDRA-8177 Project: Cassandra Issue Type: Bug Reporter: Sean Bridges Assignee: Yuki Morishita Attachments: cassc-week.png, iostats.png This is with 2.0.10 The attached graph shows io read/write throughput (as measured with iostat) when doing repairs. The large hump on the left is a sequential repair of one node. The two much smaller peaks on the right are parallel repairs. This is a 3 node cluster using vnodes (I know vnodes on small clusters isn't recommended). Cassandra reports load of 40 gigs. We noticed a similar problem with a larger cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4785) Secondary Index Sporadically Doesn't Return Rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835281#comment-13835281 ] Janne Jalkanen commented on CASSANDRA-4785: --- I can no longer repro the issue on 1.2.12 with a few minutes of testing. Looking good so far :) Secondary Index Sporadically Doesn't Return Rows Key: CASSANDRA-4785 URL: https://issues.apache.org/jira/browse/CASSANDRA-4785 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.5, 1.1.6 Environment: Ubuntu 10.04 Java 6 Sun Cassandra 1.1.5 upgraded from 1.1.2 - 1.1.3 - 1.1.5 Reporter: Arya Goudarzi Assignee: Sam Tunnicliffe Attachments: entity_aliases.txt, repro.py I have a ColumnFamily with caching = ALL. I have 2 secondary indexes on it. I have noticed if I query using the secondary index in the where clause, sometimes I get the results and sometimes I don't. Until 2 weeks ago, the caching option on this CF was set to NONE. So, I suspect something happened in secondary index caching scheme. Here are things I tried: 1. I rebuild indexes for that CF on all nodes; 2. I set the caching to KEYS_ONLY and rebuild the index again; 3. I set the caching to NONE and rebuild the index again; None of the above helped. I suppose the caching still exists as this behavior looks like cache mistmatch. I did a bit research, and found CASSANDRA-4197 that could be related. Please advice. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6102) CassandraStorage broken for bigints and ints
[
https://issues.apache.org/jira/browse/CASSANDRA-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13780719#comment-13780719
]
Janne Jalkanen commented on CASSANDRA-6102:
---
Confirmed working; result for the test case above is now (Pig 0.11.1 and
patched 1.2 latest)
grunt testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
grunt dump testc;
(foo,(ivalue,10),(svalue,bar),(value,65),{})
Thank you!
CassandraStorage broken for bigints and ints
Key: CASSANDRA-6102
URL: https://issues.apache.org/jira/browse/CASSANDRA-6102
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Environment: Cassandra 1.2.9 1.2.10, Pig 0.11.1, OSX 10.8.x
Reporter: Janne Jalkanen
Assignee: Alex Liu
Attachments: 6102-1.2-branch.txt
I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle
integer values.
Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1. Single node for
testing this.
First a table:
{noformat}
CREATE TABLE testc (
key text PRIMARY KEY,
ivalue int,
svalue text,
value bigint
) WITH COMPACT STORAGE;
insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
select * from testc;
key | ivalue | svalue | value
-+++---
foo | 10 |bar | 65
{noformat}
For my Pig setup, I then use libraries from different C* versions to actually
talk to my database (which stays on 1.2.10 all the time).
Cassandra 1.0.12 (using cassandra_storage.jar):
{noformat}
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
{noformat}
Cassandra 1.1.10:
{noformat}
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
{noformat}
Cassandra 1.2.10:
{noformat}
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
{noformat}
To me it appears that ints and bigints are interpreted as ascii values in
cass 1.2.10. Did something change for CassandraStorage, is there a
regression, or am I doing something wrong? Quick perusal of the JIRA didn't
reveal anything that I could directly pin on this.
Note that using compact storage does not seem to affect the issue, though it
obviously changes the resulting pig format.
In addition, trying to use Pygmalion
{noformat}
tf = foreach testc generate key,
flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
(ivalue:int,svalue:chararray,lvalue:long);
dump tf
(foo,
,bar,A)
{noformat}
So no help there. Explicitly casting the values to (long) or (int) just
results in a ClassCastException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-6102) CassandraStorage broken for bigints and ints
[
https://issues.apache.org/jira/browse/CASSANDRA-6102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13780708#comment-13780708
]
Janne Jalkanen commented on CASSANDRA-6102:
---
What is the patch against? Trying it on both trunk and cassandra-1.2 branches;
on cassandra-1.2 it fails because of
[javac]
/Users/jalkanen/Eclipse/cassandra/src/java/org/apache/cassandra/hadoop/pig/CqlStorage.java:57:
error: CqlStorage is not abstract and does not override abstract method
getColumnMetadata(Client) in AbstractCassandraStorage
CassandraStorage broken for bigints and ints
Key: CASSANDRA-6102
URL: https://issues.apache.org/jira/browse/CASSANDRA-6102
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Environment: Cassandra 1.2.9 1.2.10, Pig 0.11.1, OSX 10.8.x
Reporter: Janne Jalkanen
Assignee: Alex Liu
Attachments: 6102-1.2-branch.txt
I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle
integer values.
Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1. Single node for
testing this.
First a table:
{noformat}
CREATE TABLE testc (
key text PRIMARY KEY,
ivalue int,
svalue text,
value bigint
) WITH COMPACT STORAGE;
insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
select * from testc;
key | ivalue | svalue | value
-+++---
foo | 10 |bar | 65
{noformat}
For my Pig setup, I then use libraries from different C* versions to actually
talk to my database (which stays on 1.2.10 all the time).
Cassandra 1.0.12 (using cassandra_storage.jar):
{noformat}
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
{noformat}
Cassandra 1.1.10:
{noformat}
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
{noformat}
Cassandra 1.2.10:
{noformat}
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
{noformat}
To me it appears that ints and bigints are interpreted as ascii values in
cass 1.2.10. Did something change for CassandraStorage, is there a
regression, or am I doing something wrong? Quick perusal of the JIRA didn't
reveal anything that I could directly pin on this.
Note that using compact storage does not seem to affect the issue, though it
obviously changes the resulting pig format.
In addition, trying to use Pygmalion
{noformat}
tf = foreach testc generate key,
flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
(ivalue:int,svalue:chararray,lvalue:long);
dump tf
(foo,
,bar,A)
{noformat}
So no help there. Explicitly casting the values to (long) or (int) just
results in a ClassCastException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-6102) CassandraStorage broken for bigints and ints
Janne Jalkanen created CASSANDRA-6102:
-
Summary: CassandraStorage broken for bigints and ints
Key: CASSANDRA-6102
URL: https://issues.apache.org/jira/browse/CASSANDRA-6102
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Environment: Cassandra 1.2.9 1.2.10, Pig 0.11.1, OSX 10.8.x
Reporter: Janne Jalkanen
I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle
integer values.
Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1. Single node for
testing this.
First a table:
{noformat}
CREATE TABLE testc (
key text PRIMARY KEY,
ivalue int,
svalue text,
value bigint
) WITH COMPACT STORAGE;
insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
select * from testc;
key | ivalue | svalue | value
-+++---
foo | 10 |bar | 65
{noformat}
For my Pig setup, I then use libraries from different C* versions to actually
talk to my database (which stays on 1.2.10 all the time).
Cassandra 1.0.12 (using cassandra_storage.jar):
{noformat}
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
{noformat}
Cassandra 1.1.10:
{noformat}
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
{noformat}
Cassandra 1.2.10:
{noformat}
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
{noformat}
To me it appears that ints and bigints are interpreted as ascii values in cass
1.2.10. Did something change for CassandraStorage, is there a regression, or
am I doing something wrong? Quick perusal of the JIRA didn't reveal anything
that I could directly pin on this.
Note that using compact storage does not seem to affect the issue, though it
obviously changes the resulting pig format.
In addition, trying to use Pygmalion
{noformat}
tf = foreach testc generate key,
flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
(ivalue:int,svalue:chararray,lvalue:long);
dump tf
(foo,
,bar,A)
{noformat}
So no help there. Explicitly casting the values to (long) or (int) just results
in a ClassCastException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5968) Nodetool info throws NPE when connected to a booting instance
Janne Jalkanen created CASSANDRA-5968:
-
Summary: Nodetool info throws NPE when connected to a booting
instance
Key: CASSANDRA-5968
URL: https://issues.apache.org/jira/browse/CASSANDRA-5968
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 1.2.9, Ubuntu 12.04 LTS, Oracle JVM 7u25
Reporter: Janne Jalkanen
Priority: Minor
When an instance is newly added to the cluster and it's still streaming stuff,
trying to call nodetool info on it throws NPE. Stack trace below.
To replicate: add a new node to the cluster, run nodetool info before bootstrap
is complete.
Expected behaviour: is nice and just says RPC server is not running.
{noformat}
$ nodetool info
Token: (invoke with -T/--tokens to see all 0 tokens)
ID : cc7bcf48-4a54-48af-97f6-99c82bce76f2
Gossip active: true
Exception in thread main java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService.isRPCServerRunning(StorageService.java:330)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1464)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:657)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5732) Can not query secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13704633#comment-13704633 ] Janne Jalkanen commented on CASSANDRA-5732: --- Is this the same as CASSANDRA-4785? Can not query secondary index - Key: CASSANDRA-5732 URL: https://issues.apache.org/jira/browse/CASSANDRA-5732 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.5 Environment: Windows 8, Jre 1.6.0_45 32-bit Reporter: Tony Anecito Noticed after taking a column family that already existed and assigning to an IntegerType index_type:KEYS and the caching was already set to 'ALL' that the prepared statement do not return rows neither did it throw an exception. Here is the sequence. 1. Starting state query running with caching off for a Column Family with the query using the secondary index for te WHERE clause. 2, Set Column Family caching to ALL using Cassandra-CLI and update CQL. Cassandra-cli Describe shows column family caching set to ALL 3. Rerun query and it works. 4. Restart Cassandra and run query and no rows returned. Cassandra-cli Describe shows column family caching set to ALL 5. Set Column Family caching to NONE using Cassandra-cli and update CQL. Rerun query and no rows returned. Cassandra-cli Describe for column family shows caching set to NONE. 6. Restart Cassandra. Rerun query and it is working again. We are now back to the starting state. Best Regards, -Tony -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5732) Can not query secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13704633#comment-13704633 ] Janne Jalkanen edited comment on CASSANDRA-5732 at 7/10/13 3:05 PM: Is this the same as CASSANDRA-4785 and CASSANDRA-4973? was (Author: jalkanen): Is this the same as CASSANDRA-4785? Can not query secondary index - Key: CASSANDRA-5732 URL: https://issues.apache.org/jira/browse/CASSANDRA-5732 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.5 Environment: Windows 8, Jre 1.6.0_45 32-bit Reporter: Tony Anecito Noticed after taking a column family that already existed and assigning to an IntegerType index_type:KEYS and the caching was already set to 'ALL' that the prepared statement do not return rows neither did it throw an exception. Here is the sequence. 1. Starting state query running with caching off for a Column Family with the query using the secondary index for te WHERE clause. 2, Set Column Family caching to ALL using Cassandra-CLI and update CQL. Cassandra-cli Describe shows column family caching set to ALL 3. Rerun query and it works. 4. Restart Cassandra and run query and no rows returned. Cassandra-cli Describe shows column family caching set to ALL 5. Set Column Family caching to NONE using Cassandra-cli and update CQL. Rerun query and no rows returned. Cassandra-cli Describe for column family shows caching set to NONE. 6. Restart Cassandra. Rerun query and it is working again. We are now back to the starting state. Best Regards, -Tony -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4785) Secondary Index Sporadically Doesn't Return Rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611862#comment-13611862 ] Janne Jalkanen commented on CASSANDRA-4785: --- Same as CASSANDRA-4973? Secondary Index Sporadically Doesn't Return Rows Key: CASSANDRA-4785 URL: https://issues.apache.org/jira/browse/CASSANDRA-4785 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.5, 1.1.6 Environment: Ubuntu 10.04 Java 6 Sun Cassandra 1.1.5 upgraded from 1.1.2 - 1.1.3 - 1.1.5 Reporter: Arya Goudarzi I have a ColumnFamily with caching = ALL. I have 2 secondary indexes on it. I have noticed if I query using the secondary index in the where clause, sometimes I get the results and sometimes I don't. Until 2 weeks ago, the caching option on this CF was set to NONE. So, I suspect something happened in secondary index caching scheme. Here are things I tried: 1. I rebuild indexes for that CF on all nodes; 2. I set the caching to KEYS_ONLY and rebuild the index again; 3. I set the caching to NONE and rebuild the index again; None of the above helped. I suppose the caching still exists as this behavior looks like cache mistmatch. I did a bit research, and found CASSANDRA-4197 that could be related. Please advice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4785) Secondary Index Sporadically Doesn't Return Rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568890#comment-13568890 ] Janne Jalkanen commented on CASSANDRA-4785: --- I see this regularly too. Typically this occurs after recycling one node - that node will no longer have the secondary index and returns zero. Yes, the CF in question is Cache=ALL. The only way to fix it seems to be dropping the index in cqlsh and recreating it from scratch. Neither repair or index rebuild helps. This is on cass 1.1.8/Ubuntu 12.04.1 LTS/OpenJDK 7u9. Secondary Index Sporadically Doesn't Return Rows Key: CASSANDRA-4785 URL: https://issues.apache.org/jira/browse/CASSANDRA-4785 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.5, 1.1.6 Environment: Ubuntu 10.04 Java 6 Sun Cassandra 1.1.5 upgraded from 1.1.2 - 1.1.3 - 1.1.5 Reporter: Arya Goudarzi I have a ColumnFamily with caching = ALL. I have 2 secondary indexes on it. I have noticed if I query using the secondary index in the where clause, sometimes I get the results and sometimes I don't. Until 2 weeks ago, the caching option on this CF was set to NONE. So, I suspect something happened in secondary index caching scheme. Here are things I tried: 1. I rebuild indexes for that CF on all nodes; 2. I set the caching to KEYS_ONLY and rebuild the index again; 3. I set the caching to NONE and rebuild the index again; None of the above helped. I suppose the caching still exists as this behavior looks like cache mistmatch. I did a bit research, and found CASSANDRA-4197 that could be related. Please advice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4785) Secondary Index Sporadically Doesn't Return Rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568890#comment-13568890 ] Janne Jalkanen edited comment on CASSANDRA-4785 at 2/1/13 5:23 PM: --- I see this regularly too. Typically this occurs after recycling one node - that node will no longer have the secondary index and returns zero. Yes, the CF in question is Cache=ALL. The only way to fix it seems to be dropping the index in cqlsh and recreating it from scratch. Neither repair or index rebuild helps. This is on cass 1.1.8/Ubuntu 12.04.1 LTS/OpenJDK 7u9. N=3, RF=3. was (Author: jalkanen): I see this regularly too. Typically this occurs after recycling one node - that node will no longer have the secondary index and returns zero. Yes, the CF in question is Cache=ALL. The only way to fix it seems to be dropping the index in cqlsh and recreating it from scratch. Neither repair or index rebuild helps. This is on cass 1.1.8/Ubuntu 12.04.1 LTS/OpenJDK 7u9. Secondary Index Sporadically Doesn't Return Rows Key: CASSANDRA-4785 URL: https://issues.apache.org/jira/browse/CASSANDRA-4785 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.5, 1.1.6 Environment: Ubuntu 10.04 Java 6 Sun Cassandra 1.1.5 upgraded from 1.1.2 - 1.1.3 - 1.1.5 Reporter: Arya Goudarzi I have a ColumnFamily with caching = ALL. I have 2 secondary indexes on it. I have noticed if I query using the secondary index in the where clause, sometimes I get the results and sometimes I don't. Until 2 weeks ago, the caching option on this CF was set to NONE. So, I suspect something happened in secondary index caching scheme. Here are things I tried: 1. I rebuild indexes for that CF on all nodes; 2. I set the caching to KEYS_ONLY and rebuild the index again; 3. I set the caching to NONE and rebuild the index again; None of the above helped. I suppose the caching still exists as this behavior looks like cache mistmatch. I did a bit research, and found CASSANDRA-4197 that could be related. Please advice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5175) Unbounded (?) thread growth connecting to an removed node
Janne Jalkanen created CASSANDRA-5175:
-
Summary: Unbounded (?) thread growth connecting to an removed node
Key: CASSANDRA-5175
URL: https://issues.apache.org/jira/browse/CASSANDRA-5175
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.8
Environment: EC2, JDK 7u9, Ubuntu 12.04.1 LTS
Reporter: Janne Jalkanen
Priority: Minor
The following lines started repeating every minute in the log file
{noformat}
INFO [GossipStage:1] 2013-01-19 19:35:43,929 Gossiper.java (line 831)
InetAddress /10.238.x.y is now dead.
INFO [GossipStage:1] 2013-01-19 19:35:43,930 StorageService.java (line 1291)
Removing token 170141183460469231731687303715884105718 for /10.238.x.y
{noformat}
Also, I got about 3000 threads which all look like this:
{noformat}
Name: WRITE-/10.238.x.y
State: WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bb65c0f
Total blocked: 0 Total waited: 3
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:104)
{noformat}
A new thread seems to be created every minute, and they never go away.
The endpoint in question had been a part of the cluster weeks ago, and the node
exhibiting the thread growth was added yesterday.
Anyway, assassinating the endpoint in question stopped thread growth (but kept
the existing threads running), so this isn't a huge issue. But I don't think
the thread count is supposed to be increasing like this...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552492#comment-13552492 ] Janne Jalkanen commented on CASSANDRA-4417: --- Turns out that no amount of repair (I ran both repair -pr and full repair) allows the counter values to converge. One node had consistently wrong counts that would not be repaired no matter what. In the end I took out the node, removed all data and brought it back into the cluster and let it reinitialize itself. Now the values are converged. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549412#comment-13549412 ] Janne Jalkanen edited comment on CASSANDRA-4417 at 1/10/13 7:58 AM: I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. Environment is Ubuntu Linux 12.04 LTS, JVM is OpenJDK 7u9. Last repair picked 497 invalid counter shards, and we have approximately 8 million counters, of which about a hundred is incremented each second (and sometimes subtracted from if our read repair kicks in - we have our own in-app repair for certain low values). All the counter writes are batched with 100 increments/batch. So this is only affecting a really small subset, though it's rather annoying when it happens, as it means that you can never really trust the counters to be even in the ballpark :-/ was (Author: jalkanen): I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. Environment is Ubuntu Linux 12.04 LTS, JVM is OpenJDK 7u9. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549412#comment-13549412 ] Janne Jalkanen commented on CASSANDRA-4417: --- I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. I did disablegossip, disablethrift, drain, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549412#comment-13549412 ] Janne Jalkanen edited comment on CASSANDRA-4417 at 1/10/13 7:48 AM: I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. was (Author: jalkanen): I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. I did disablegossip, disablethrift, drain, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549412#comment-13549412 ] Janne Jalkanen edited comment on CASSANDRA-4417 at 1/10/13 7:48 AM: I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. was (Author: jalkanen): I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549412#comment-13549412 ] Janne Jalkanen edited comment on CASSANDRA-4417 at 1/10/13 7:49 AM: I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. Environment is Ubuntu Linux 12.04 LTS. was (Author: jalkanen): I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549412#comment-13549412 ] Janne Jalkanen edited comment on CASSANDRA-4417 at 1/10/13 7:50 AM: I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. Environment is Ubuntu Linux 12.04 LTS, JVM is OpenJDK 7u9. was (Author: jalkanen): I'm seeing this while running repair -pr. Three-cluster node, RF 3. Straight upgrade from 1.0.12 to 1.1.8; no topology changes. I see two invalid shard IDs, counts differ by more than one - sometimes even by 3000 or more. Seems random to my eyes. Our counters are in a composite column family, no TTLs in use. We *mostly* increment by one, but sometimes more. I did disablegossip, disablethrift, drain, shutdown, upgrade, restart on every node in a rolling fashion. Then I did upgradesstables and repair -pr on every node when the entire cluster had been upgraded. Environment is Ubuntu Linux 12.04 LTS. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Attachments: cassandra-mck.log.bz2, err.txt Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5110) NodeCmd misspells positives
Janne Jalkanen created CASSANDRA-5110: - Summary: NodeCmd misspells positives Key: CASSANDRA-5110 URL: https://issues.apache.org/jira/browse/CASSANDRA-5110 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.8 Environment: Any Reporter: Janne Jalkanen Priority: Trivial Running nodetool cfstats speaks of Bloom Filter False Postives. It annoys my OCD to see the misspelling all the time. :-) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Deleted] (CASSANDRA-5110) NodeCmd misspells positives
[ https://issues.apache.org/jira/browse/CASSANDRA-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Jalkanen updated CASSANDRA-5110: -- Comment: was deleted (was: Patch against cassandra-1.1.) NodeCmd misspells positives - Key: CASSANDRA-5110 URL: https://issues.apache.org/jira/browse/CASSANDRA-5110 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.8 Environment: Any Reporter: Janne Jalkanen Priority: Trivial Running nodetool cfstats speaks of Bloom Filter False Postives. It annoys my OCD to see the misspelling all the time. :-) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5110) NodeCmd misspells positives
[ https://issues.apache.org/jira/browse/CASSANDRA-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Jalkanen updated CASSANDRA-5110: -- Attachment: CASSANDRA-5110.patch Patch against cassandra-1.1 NodeCmd misspells positives - Key: CASSANDRA-5110 URL: https://issues.apache.org/jira/browse/CASSANDRA-5110 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.8 Environment: Any Reporter: Janne Jalkanen Priority: Trivial Attachments: CASSANDRA-5110.patch Running nodetool cfstats speaks of Bloom Filter False Postives. It annoys my OCD to see the misspelling all the time. :-) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
