Re: Mass deletion -- slowing down
i think what he means is...do you know what day the 'oldest' day is? eg if you have a rolling window of say 2 weeks, structure your query so that your slice range only goes back 2 weeks, rather than to the beginning of time. this would avoid iterating over all the tombstones from prior to the 2 week window. this wouldn't work if you are deleting arbitrary days in the middle of your date range. On 14/11/2011 02:02, Maxim Potekhin wrote: Thanks Peter, I'm not sure I entirely follow. By the oldest data, do you mean the primary key corresponding to the limit of the time horizon? Unfortunately, unique IDs and the timstamps do not correlate in the sense that chronologically newer entries might have a smaller sequential ID. That's because the timestamp corresponds to the last update that's stochastic in the sense that the jobs can take from seconds to days to complete. As I said I'm not sure I understood you correctly. Also, I note that queries on different dates (i.e. not contaminated with lots of tombstones) work just fine, which is consistent with the picture that emerged so far. Theoretically -- would compaction or cleanup help? Thanks Maxim On 11/13/2011 8:39 PM, Peter Schuller wrote: I do limit the number of rows I'm asking for in Pycassa. Queries on primary keys still work fine, Is it feasable in your situation to keep track of the oldest possible data (for example, if there is a single sequential writer that rotates old entries away it could keep a record of what the oldest might be) so that you can bound your index lookup= that value (and avoid the tombstones)?
Re: indexes from CassandraSF
ok great, thanks ed, that's really helpful. just wanted to make sure i wasn't missing something fundamental. On 13/11/2011 23:57, Ed Anuff wrote: Yes, correct, it's not going to clean itself. Using your example with a little more detail: 1 ) A(T1) reads previous location (T0,L0) from index_entries for user U0 2 ) B(T2) reads previous location (T0,L0) from index_entries for user U0 3 ) A(T1) deletes previous location (T0,L0) from index_entries for user U0 4 ) B(T2) deletes previous location (T0,L0) from index_entries for user U0 5 ) A(T1) deletes previous location (L0,T0,U0) for user U0 from index 6 ) B(T2) deletes previous location (L0,T0,U0) for user U0 from index 7 ) A(T1) inserts new location (T1,L1) into index_entries for user U0 8 ) B(T2) inserts new location (T2,L2) into index_entries for user U0 9 ) index_entries for user U0 now contains (T1,L1),(T2,L2) 10) A(T1) inserts new location (L1,T1,U0) for user U0 into index 11) B(T2) inserts new location (L2,T2,U0) for user U0 into index 12) A(T1) sets new location (L1) on user U0 13) B(T2) sets new location (L2) on user U0 14) C(T3) queries for users where location equals L1, gets back user U0 where current location is actually L2 So, you want to either verify on read by making sure the queried field is correct before returning it in your result set to the rest of your app, or you want to use locking (ex. lock on (U0,location) during updates). The key thing here is that although the index is not in the desired state at (14), the information is in the system to get to that state (the previous values in index_entries). This lets the cleanup happen on the next update of location for user U0: 15) D(T4) reads previous locations (T1,L1),(T2,L2) from index entries for user U0 16) D(T4) deletes previous locations (T1,L1),(T2,L2) from index entries for user U0 17) D(T4) deletes previous locations (L1,T1,U0),(L2,T2,U0) for user U0 from index 18) D(T4) inserts new location (T4,L3) into index entries for user U0 19) D(T4) inserts new location (L3,T4,U0) for user U0 into index 20) D(T4) sets new location (L3) on user U0 BTW, just to reiterate since this sometimes comes up, the timestamps being stored in these tuples are not longs, they're time UUIDs, so T1 and T2 are never equal. Ed On Sun, Nov 13, 2011 at 6:52 AM, Guy Incognitodnd1...@gmail.com wrote: [1] i'm not particularly worried about transient conditions so that's ok. i think there's still the possibility of a non-transient false positive...if 2 writes were to happen at exactly the same time (highly unlikely), eg 1) A reads previous location (L1) from index entries 2) B reads previous location (L1) from index entries 3) A deletes previous location (L1) from index entries 4) B deletes previous location (L1) from index entries 5) A deletes previous location (L1) from index 6) B deletes previous location (L1) from index 7) A enters new location (L2) into index entries 8) B enters new location (L3) into index entries 9 ) A enters new location (L2) into index 10) B enters new location (L3) into index 11) A sets new location (L2) on users 12) B sets new location (L2) on users after this, don't i end up with an incorrect L2 location in index entries and in the index, that won't be resolved until the next write of location for that user? [2] ah i see...so the client would continuously retry until the update works. that's fine provided the client doesn't bomb out with some other error, if that were to happen then i have potentially deleted the index entry columns without deleting the corresponding index columns. i can handle both of the above for my use case, i just want to clarify whether they are possible (however unlikely) scenarios. On 13/11/2011 02:41, Ed Anuff wrote: 1) The index updates should be eventually consistent. This does mean that you can get a transient false-positive on your search results. If this doesn't work for you, then you either need to use ZK or some other locking solution or do read repair by making sure that the row you retrieve contains the value you're searching for before passing it on to the rest of your applicaiton. 2) You should be able to reapply the batch updates til they succeed. The update is idempotent. One thing that's important that the slides don't make clear is that this requires using time-based uuids as your timestamp components. Take a look at the sample code. Hope this helps, Ed On Sat, Nov 12, 2011 at 3:59 PM, Guy Incognitodnd1...@gmail.comwrote: help? On 10/11/2011 19:34, Guy Incognito wrote: hi, i've been looking at the model below from Ed Anuff's presentation at Cassandra CF (http://www.slideshare.net/edanuff/indexing-in-cassandra). Couple of questions: 1) Isn't there still the chance that two concurrent updates may end up with the index containing two entries for the given user, only one of which would be match the actual value in the Users cf? 2) What happens if your batch fails partway through the update? If i understand
RE: Client Timeouts on incrementing counters
I have digged a bit more to try to find the root cause of the error, and I have some more information. It seems that all started after I upgraded Cassandra from 0.8.x to 1.0.0 When I do a incr on the CLI I also get a timeout. row_cache_save_period_in_seconds is set to 60sec. Could be a problem from the upgrade? I just did a rolling restart of all nodes one-by-one. From: Tyler Hobbs [mailto:ty...@datastax.com] Sent: vrijdag 11 november 2011 20:18 To: user@cassandra.apache.org Subject: Re: Client Timeouts on incrementing counters On Fri, Nov 11, 2011 at 7:17 AM, Carlos Rolo c.r...@ocom.commailto:c.r...@ocom.com wrote: Also Cassandra logs have lots (as in, several times per second) of this message now: INFO 14:15:25,740 Saved ClusterCassandra-CounterFamily-RowCache (52 items) in 1 ms What does the CLI say the row_cache_save_period_in_seconds for this CF is? -- Tyler Hobbs DataStaxhttp://datastax.com/
Re: Fast lookups for userId to username and vice versa
Check if Cassandra secondary index meets your requirement. Thank you, Jaydeep From: Aklin_81 asdk...@gmail.com To: user user@cassandra.apache.org Sent: Sunday, 13 November 2011 12:32 PM Subject: Fast lookups for userId to username and vice versa I need to create mapping from userId(s) to username(s) which need to provide for fast lookups service ? Also I need to provide a mapping from username to userId inorder to implement search functionality in my application. What could be a good strategy to implement this ? (I would welcome suggestions to use any new technologies if they are really worth for my case.)
Re: multi datacenter cluster, without fibre speeds
Thanks for your feedback Kolar. Well to be honest I was thinking of using that connection in production, not for a backup node. My Cassandra deployment works just like an expensive file caching and replication - I mean, all I use it for is to replicate some 5million files of 2M each across few nodes and intensively read/write. I used MySQL until around I had some 1.2Million entries, then sharding it when numbers grow larger. Not only the files themselves but I also need to attach some tags to each file (see them as key=value) so I though of Haadop but in the end settle for Cassandra because of better consistency, community support, no single point of failure and some! Thanks, Marco On Nov 11, 2011 7:09 PM, Radim Kolar h...@sendmail.cz wrote: Dne 11.11.2011 19:14, M Vieira napsal(a): Has anyone experimented running cassandra clusters in geographicly separated locations connected thru ordinary broadband? By ordinary broadband I mean 30Mbps or 50Mbps for backup purposes, like place 1 replica on remote location over WAN?? yes. it works good enough, only anti-entropy repairs are kinda slow. Just make sure that applications will not read from that remote node.
Re: multi datacenter cluster, without fibre speeds
Well to be honest I was thinking of using that connection in production, not for a backup node. For productions. there are several problems. Added network latency which is inconsistent and vary greatly during day, sometimes you will face network lags which will break cluster for a while (about 1-2 minutes). Also network bandwidth is problem especially during peak hours. It might not be problem if you dont have interactive workload - app can wait, human cant. Be sure to use connection pooling to different servers at client. Over WAN you can have about 4:1 ratio in available bw in peak hours/night hours. - You need to schedule antientropy repairs at nights. My Cassandra deployment works just like an expensive file caching and replication - I mean, all I use it for is to replicate some 5million files of 2M each across few nodes and intensively read/write. for mass replication of large files hadoop is really better then cassandra because there are no compactions. Not only the files themselves but I also need to attach some tags to each file (see them as key=value) so I though of Haadop but in the end settle for Cassandra because of better consistency, community support, no single point of failure and some! hadoop is far better then cassandra for batch processing if your batch processing changes majority of data set. SPOF is not problem, but it is way harder to write optimised applications for hadoop, its kinda low level.
Re: multi datacenter cluster, without fibre speeds
Broadband here is fairly stable, to be honest don't remember last time I had problems such as larger than expected latency or downtime - ISP Bethere /UK My application can cope fine with up to 10 min lag (data freshness), however taking your input into consideration I agree with you, so don't think I should trust the setup and believe my data will be sync'ed across the two clusters as failures will surely occur. Regarding Hadoop, it requires a lot more coding if compared to what I need to get my app working with Cassandra. Also, Hadoop's single point of failure weakness scares me because I don't have budget nor time to work on fail-safe kind of solution. Once again, thanks for your feedback Marco On 14 November 2011 09:43, Radim Kolar h...@sendmail.cz wrote: Well to be honest I was thinking of using that connection in production, not for a backup node. For productions. there are several problems. Added network latency which is inconsistent and vary greatly during day, sometimes you will face network lags which will break cluster for a while (about 1-2 minutes). Also network bandwidth is problem especially during peak hours. It might not be problem if you dont have interactive workload - app can wait, human cant. Be sure to use connection pooling to different servers at client. Over WAN you can have about 4:1 ratio in available bw in peak hours/night hours. - You need to schedule antientropy repairs at nights. My Cassandra deployment works just like an expensive file caching and replication - I mean, all I use it for is to replicate some 5million files of 2M each across few nodes and intensively read/write. for mass replication of large files hadoop is really better then cassandra because there are no compactions. Not only the files themselves but I also need to attach some tags to each file (see them as key=value) so I though of Haadop but in the end settle for Cassandra because of better consistency, community support, no single point of failure and some! hadoop is far better then cassandra for batch processing if your batch processing changes majority of data set. SPOF is not problem, but it is way harder to write optimised applications for hadoop, its kinda low level.
compaction on hintedhandoff is done too early
from log output it seems that during hintedhandoff delivery compaction is kicked too soon. There needs to be some delay for flusher to write sstable. INFO [GossipStage:1] 2011-11-14 13:16:03,933 Gossiper.java (line 745) InetAddress /***.99.40 is now UP INFO [HintedHandoff:1] 2011-11-14 13:16:15,064 HintedHandOffManager.java (line 268) Started hinted handoff for token: 99070591730234615865843651857942052864 with IP: /*.99.40 WARN [pool-1-thread-1] 2011-11-14 13:16:18,389 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.5454013335736169 INFO [pool-1-thread-1] 2011-11-14 13:16:18,389 Memtable.java (line 179) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 2.3651230775748844 (just-counted was 1.0). calculation took 10ms for 492 columns INFO [HintedHandoff:1] 2011-11-14 13:16:19,347 ColumnFamilyStore.java (line 688) Enqueuing flush of Memtable-HintsColumnFamily@258947641(693625/2050635 serialized/live bytes, 985 ops) INFO [FlushWriter:4] 2011-11-14 13:16:19,348 Memtable.java (line 239) Writing Memtable-HintsColumnFamily@258947641(693625/2050635 serialized/live bytes, 985 ops) INFO [CompactionExecutor:9] 2011-11-14 13:16:19,348 CompactionTask.java (line 238) Nothing to compact in HintsColumnFamily. Use forceUserDefinedCompaction if you wish to force compaction of single sstables (e.g. for tombstone collection) INFO [HintedHandoff:1] 2011-11-14 13:16:19,348 HintedHandOffManager.java (line 334) Finished hinted handoff of 492 rows to endpoint INFO [FlushWriter:4] 2011-11-14 13:16:19,365 Memtable.java (line 275) Completed flushing /usr/local/cassandra/data/system/HintsColumnFamily-hb-2-Data.db (711045 bytes)
Random access
I am new to cassandra. I search for random write examples in wiki (http://wiki.apache.org/cassandra/ClientOptions) and mailing list, but do not find similar one. My question is - does cassandra suppport random write access? Is there any code example that explains this? Or any doc that may provide some information needed? Thanks
Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 1:21 AM, Michael Vaknine micha...@citypath.com wrote: Hi, After configuring the encryption on Cassandra.yaml I get this error when upgrading from 1.0.0 to 1.0.2 Attached the log file with the errors. https://issues.apache.org/jira/browse/CASSANDRA-3466 -Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
Does this means that I have to wait to 1.0.3? -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Monday, November 14, 2011 3:51 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Mon, Nov 14, 2011 at 1:21 AM, Michael Vaknine micha...@citypath.com wrote: Hi, After configuring the encryption on Cassandra.yaml I get this error when upgrading from 1.0.0 to 1.0.2 Attached the log file with the errors. https://issues.apache.org/jira/browse/CASSANDRA-3466 -Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 7:53 AM, Michael Vaknine micha...@citypath.com wrote: Does this means that I have to wait to 1.0.3? In the meantime you can just delete the hints and rely on read repair or antientropy repair if you're concerned about the consistency of your replicas. -Brandon
Ec2Snitch and Ec2MultiRegionSnitch - does us-west-2 cause a bug with this snitch?
Hi,
As of Nov. 9, 2011 Amazon added us-west-2 (US West Oregon) region:
http://aws.typepad.com/aws/2011/11/now-open-us-west-portland-region.html
In looking at the EC2Snitch code (in the 0.8.x and 1.0.x branches), I see
it determining which data center (which I think is supposed to be
equivalent to region) it is in by this:
public Ec2Snitch() throws IOException, ConfigurationException
{
// Split us-east-1a or asia-1a into us-east/1a and asia/1a.
String[] splits = awsApiCall(ZONE_NAME_QUERY_URL).split(-);
ec2zone = splits[splits.length - 1];
ec2region = splits.length 3 ? splits[0] : splits[0] + - + splits[1];
logger.info(EC2Snitch using region: + ec2region + , zone:
+ ec2zone + .);
}
The Ec2MultiRegionSnitch then has this reconnect logic - which I think the
existence of a us-west-1 and us-west-2 regions and the way the Ec2Snitch is
assigning data centers will cause this to break:
private void reConnect(InetAddress endpoint, VersionedValue versionedValue)
{
if (!getDatacenter(endpoint).equals(getDatacenter(public_ip)))
return; // do nothing return back...
try
{
InetAddress remoteIP = InetAddress.getByName(versionedValue.value);
MessagingService.instance().getConnectionPool(endpoint).reset(remoteIP);
logger.debug(String.format(Intiated reconnect to an
Internal IP %s for the %s, remoteIP, endpoint));
} catch (UnknownHostException e)
{
logger.error(Error in getting the IP address resolved: , e);
}
}
Am I correct or do I not understand what was intended by these snitches?
Allen
RE: Upgrade Cassandra Cluster to 1.0.2
Well, I tried to delete the hints on the failed cluster but I could not start it I got other errors such as ERROR [MutationStage:34] 2011-11-14 15:37:43,813 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:34,5,main] java.lang.StackOverflowError at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216) at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.compareAndSet(AtomicReferenceFieldUpdater.java:227) at java.util.concurrent.ConcurrentSkipListMap.casHead(ConcurrentSkipListMap.jav a:359) at java.util.concurrent.ConcurrentSkipListMap.insertIndex(ConcurrentSkipListMap .java:968) at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java: 898) at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap .java:1893) at org.apache.cassandra.db.ThreadSafeSortedColumns.addColumn(ThreadSafeSortedCo lumns.java:88) at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCont ainer.java:122) at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCont ainer.java:117) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColum ns(SSTableNamesIterator.java:183) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableName sIterator.java:130) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNa mesIterator.java:72) at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(Nam esQueryFilter.java:60) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFil ter.java:78) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:113) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) 187179,2-995% Should I have deleted all the hints on all the clustr? Is 1.0.0 stable enough? I am checking this and don't want to upgrade again in a week or so. Thanks Michael -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Monday, November 14, 2011 4:01 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Mon, Nov 14, 2011 at 7:53 AM, Michael Vaknine micha...@citypath.com wrote: Does this means that I have to wait to 1.0.3? In the meantime you can just delete the hints and rely on read repair or antientropy repair if you're concerned about the consistency of your replicas. -Brandon
Re: Random access
I am new to cassandra. I search for random write examples you can access cassandra data at any node and keys can be accessed at random.
Re: questions on frequency and timing of async replication between DCs
It may be the case that your CL is the issue. You are writing it at ONE, which means that out of the 4 replicas of that key (two in each data center), you are only putting it on one of them. When you read at CL ONE, if only looks at a single replica to see if the data is there. In other words. If your read and your write do not select the same node to look at, there is no guarantee that any successfully written data is going to be there. The rule is, in order to be able to access data properly, CL.READ + CL.WRITE RF. Your RF is 4 while your CL are both 1. On Fri, Nov 11, 2011 at 18:35, Subrahmanya Harve subrahmanyaha...@gmail.com wrote: If i query by list command, i do see the data i am looking for. Interestingly, if do list on each of the nodes using cli, i get back a different number of rows! i did not want to query by ALL because of 2 reasons - 1. It might take too long to return. 2 it might not return me anything just because a node was temporarily down? any idea whats going on here? i am also wondering if anyone has gone to production with cross-dc using 0.8.7 and how its working out? lessons learnt, etc On Fri, Nov 11, 2011 at 2:55 PM, Jeremiah Jordan jeremiah.jor...@morningstar.com wrote: If you query with ALL do you get the data? If you query with a range slice do you get the data (list from the cli)? On 11/11/2011 04:10 PM, Subrahmanya Harve wrote: I have cross dc replication set up using 0.8.7 with 3 nodes on each DC by following the +1 rule for tokens. I am seeing an issue where the insert into a DC happened successfully but on querying from cli or through Hector, i am not seeing the data being returned. i used cli on every node of both DCs and every node returned blank. So basic question is where is my data? CL.WRITE=ONE, CL.READ=1. RF = DC:2, DC:2 Apart from checking the data directory size on each DC to verify that cross-dc replication has happened, what others steps can i take to verify that cross dc replication is happening successfully? What tuning params can i control with regard to cross-dc replication? (frequency? batch size?, etc) would greatly appreciate any help.
Re: Is a direct upgrade from .6 to 1.0 possible?
You should be able to do it as long as you shut down the whole cluster for it: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Upgrading-to-1-0-tp6954908p6955316.html On 11/13/2011 02:14 PM, Timothy Smith wrote: Due to some application dependencies I've been holding off on a Cassandra upgrade for a while. Now that my last application using the old thrift client is updated I have the green light to prep my upgrade. Since I'm on .6 the upgrade is obviously a bit trickier. Do the standard instructions for upgrading from .6 to .7 still apply or do I have to step from .6 - .7 - 1.0? Thanks, Tim
Re: Random access
you can access cassandra data at any node and keys can be accessed at random. Including individual columns in a row. -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Cassandra London - Monday 21/11
Hi all, Quick note about our Cassandra London 1st birthday party! We'll be looking at what's changed in Cassandra over the past year, with talks on feature improvements, performance and Hadoop integration. Please come along if you're UK-based! It's a great chance to meet other Cassandra users. http://www.meetup.com/Cassandra-London/events/36802872/ Dave
Proposed 1.0.3: StackOverflowError on repair
Hi
While testing the proposed 1.0.3 version I got the following exception
while running repair: (StackOverflowError)
http://pastebin.com/raw.php?i=35Rt7ryB
The affected column family is denfined like this:
create column family FileStore
with comparator=UTF8Type and key_validation_class = 'UTF8Type'
and column_metadata=[{column_name: namespace, validation_class:
UTF8Type, index_type: KEYS},
{column_name: version, validation_class:
IntegerType},
{column_name: chunk_size, validation_class:
IntegerType},
{column_name: num_chunks, validation_class:
IntegerType},
{column_name: size, validation_class: LongType}];
Is this a known issue?
/ Jonas
Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 8:06 AM, Michael Vaknine micha...@citypath.com wrote: Well, I tried to delete the hints on the failed cluster but I could not start it I got other errors such as ERROR [MutationStage:34] 2011-11-14 15:37:43,813 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:34,5,main] java.lang.StackOverflowError at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216) This is different an unrelated to hints. What jvm are you using? It looks like it just simply ran out of stack space, which is odd, but you can control that with the -Xss option if needed. -Brandon
Re: Off-heap caching through ByteBuffer.allocateDirect when JNA not available ?
Additionally, you have the barely-documented but nasty behavior of Hotspot forcing full GCs when allocateDirect reaches -XX:MaxDirectMemorySize. On Sun, Nov 13, 2011 at 2:09 PM, Peter Schuller peter.schul...@infidyne.com wrote: I would like to know it also - actually is should be similar, plus there are no dependencies to sun.misc packages. I don't remember the discussion, but I assume the reason is that allocateDirect() is not freeable except by waiting for soft ref counting. This is enforced by the API in order to enable safe use of allocated memory without it being possible to use to break out of the JVM sandbox. JNA or misc.unsafe allows explicit freeing (at the cost of application bugs maybe segfaulting the JVM or causing other side-effects; i.e., breaking out of the managed runtime sandbox). -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com) -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Cassandra as a LinkedIn Skill
Hi, Sorry for the intrusion. I was speaking to some of the LinkedIn engineers at ApacheCon last week about to see how to get Cassandra into the linkedin skills page [1]. They claim if more people add Cassandra as a skill in their profile then it will show up. So my request is if you use Cassandra please add it as a skill on your linked in profile. Here is mine as an example [2]. Thanks, -Jake [1] http://www.linkedin.com/skills [2] http://www.linkedin.com/in/tjake -- http://twitter.com/tjake
BulkLoader
Hello everyone, We're using the bulk loader to load data every day to Cassandra. The machines that use the bulkloader are diferent every day so their IP addresses change. When I do describe cluster i see all the unreachable nodes that keep piling up for the past few days. Is there a way to remove those IP addresses without terminating the whole cluster at the same time and restarting it? The unreachable nodes cause issues when we want to make schema changes to all the nodes or when we want to truncate a CF. Any suggestions? -Giannis
Re: Mass deletion -- slowing down
Thanks for the note. Ideally I would not like to keep track of what is the oldest indexed date, because this means that I'm already creating a bit of infrastructure on top of my database, with attendant referential integrity problems. But I suppose I'll be forced to do that. In addition, I'll have to wait until the grace period is over and compact, removing the tombstones and finally clearing the disk (which is what I need to do in the first place). Frankly, this whole situation for me illustrates a very real deficiency in Cassandra -- one would think that deleting less than one percent of data shouldn't really lead to complete failures in certain indexed queries. That's bad. Maxim On 11/14/2011 3:01 AM, Guy Incognito wrote: i think what he means is...do you know what day the 'oldest' day is? eg if you have a rolling window of say 2 weeks, structure your query so that your slice range only goes back 2 weeks, rather than to the beginning of time. this would avoid iterating over all the tombstones from prior to the 2 week window. this wouldn't work if you are deleting arbitrary days in the middle of your date range. On 14/11/2011 02:02, Maxim Potekhin wrote: Thanks Peter, I'm not sure I entirely follow. By the oldest data, do you mean the primary key corresponding to the limit of the time horizon? Unfortunately, unique IDs and the timstamps do not correlate in the sense that chronologically newer entries might have a smaller sequential ID. That's because the timestamp corresponds to the last update that's stochastic in the sense that the jobs can take from seconds to days to complete. As I said I'm not sure I understood you correctly. Also, I note that queries on different dates (i.e. not contaminated with lots of tombstones) work just fine, which is consistent with the picture that emerged so far. Theoretically -- would compaction or cleanup help? Thanks Maxim On 11/13/2011 8:39 PM, Peter Schuller wrote: I do limit the number of rows I'm asking for in Pycassa. Queries on primary keys still work fine, Is it feasable in your situation to keep track of the oldest possible data (for example, if there is a single sequential writer that rotates old entries away it could keep a record of what the oldest might be) so that you can bound your index lookup= that value (and avoid the tombstones)?
RE: BulkLoader
Hello Giannis, Can you share a little bit on how to use the bulk loader ? We're considering use bulk loader for a user case. Thanks, Mike From: Giannis Neokleous [mailto:gian...@generalsentiment.com] Sent: Monday, November 14, 2011 2:50 PM To: user@cassandra.apache.org Subject: BulkLoader Hello everyone, We're using the bulk loader to load data every day to Cassandra. The machines that use the bulkloader are diferent every day so their IP addresses change. When I do describe cluster i see all the unreachable nodes that keep piling up for the past few days. Is there a way to remove those IP addresses without terminating the whole cluster at the same time and restarting it? The unreachable nodes cause issues when we want to make schema changes to all the nodes or when we want to truncate a CF. Any suggestions? -Giannis This email was sent to you by Thomson Reuters, the global news and information company. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Thomson Reuters.
describe cluster error
Hi I'm getting this error when I try to run describe cluster: [] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: Error retrieving data: Internal error processing describe_schema_versions with this in the logs: ERROR [pool-2-thread-3] 2011-11-14 21:40:51,975 Cassandra.java (line 3540) Internal error processing describe_schema_versions java.lang.NoSuchMethodError: org.apache.commons.lang.StringUtils.join(Ljava/util/Collection;Ljava/lang/String;)Ljava/lang/String; at org.apache.cassandra.service.StorageProxy.describeSchemaVersions(StorageProxy.java:959) at org.apache.cassandra.thrift.CassandraServer.describe_schema_versions(CassandraServer.java:1151) at org.apache.cassandra.thrift.Cassandra$Processor$describe_schema_versions.process(Cassandra.java:3536) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Anyone have any insight? It's a 3node 1.0.2 cluster, recently upgraded from .87. Thank you
word count example using it's own job tracker, not the one I setup
I am trying to run the word count example and looking at the source, I don't see where it knows which job tracker to connect to. 1. Would I be correct in guessing that I have to run hadoop jar ? (just wondering why I don't see a port/hostname for job tracker in the WordCount.java file) This then brings up a second question if I am right about that which is 2. What is the best way to distribute the core thrift jars to tasktrackers? a. Put it in the tasktracker lib or something b. have it shipped with the job(makes it easier to vary the thrift or hector client I guess) thanks for any input on this, Dean
Re: Second Cassandra users survey
- It would be super cool if all of that counter work made it possible to support other atomic data types (sets? CAS? just pass a assoc/commun Function to apply). - Again with types, pluggable type specific compression. - Wishy washy wish: Simpler elasticity I would like to go from 6--8--7 nodes without each of those being an annoying fight with tokens. - Gossip as library. Gossip/failure detection is something C* seems to have gotten particularly right (or at least it's something that has not needed to change much). It would be cool to use Cassandra's gossip protocol as distributed systems building tool a la ZooKeeper. On 11/01/2011 06:59 PM, Jonathan Ellis wrote: Hi all, Two years ago I asked for Cassandra use cases and feature requests. [1] The results [2] have been extremely useful in setting and prioritizing goals for Cassandra development. But with the release of 1.0 we've accomplished basically everything from our original wish list. [3] I'd love to hear from modern Cassandra users again, especially if you're usually a quiet lurker. What does Cassandra do well? What are your pain points? What's your feature wish list? As before, if you're in stealth mode or don't want to say anything in public, feel free to reply to me privately and I will keep it off the record. [1] http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01148.html [2] http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html [3] http://www.mail-archive.com/dev@cassandra.apache.org/msg01524.html
Re: Second Cassandra users survey
Re Simpler elasticity: Latest opscenter will now rebalance cluster optimally http://www.datastax.com/dev/blog/whats-new-in-opscenter-1-3 /plug -Jake On Mon, Nov 14, 2011 at 7:27 PM, Chris Burroughs chris.burrou...@gmail.comwrote: - It would be super cool if all of that counter work made it possible to support other atomic data types (sets? CAS? just pass a assoc/commun Function to apply). - Again with types, pluggable type specific compression. - Wishy washy wish: Simpler elasticity I would like to go from 6--8--7 nodes without each of those being an annoying fight with tokens. - Gossip as library. Gossip/failure detection is something C* seems to have gotten particularly right (or at least it's something that has not needed to change much). It would be cool to use Cassandra's gossip protocol as distributed systems building tool a la ZooKeeper. On 11/01/2011 06:59 PM, Jonathan Ellis wrote: Hi all, Two years ago I asked for Cassandra use cases and feature requests. [1] The results [2] have been extremely useful in setting and prioritizing goals for Cassandra development. But with the release of 1.0 we've accomplished basically everything from our original wish list. [3] I'd love to hear from modern Cassandra users again, especially if you're usually a quiet lurker. What does Cassandra do well? What are your pain points? What's your feature wish list? As before, if you're in stealth mode or don't want to say anything in public, feel free to reply to me privately and I will keep it off the record. [1] http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01148.html [2] http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html [3] http://www.mail-archive.com/dev@cassandra.apache.org/msg01524.html -- http://twitter.com/tjake
Re: Second Cassandra users survey
On Mon, Nov 14, 2011 at 4:44 PM, Jake Luciani jak...@gmail.com wrote: Re Simpler elasticity: Latest opscenter will now rebalance cluster optimally http://www.datastax.com/dev/blog/whats-new-in-opscenter-1-3 /plug Does it cause any impact on reads and writes while re-balance is in progress? How is it handled on live cluster? -Jake On Mon, Nov 14, 2011 at 7:27 PM, Chris Burroughs chris.burrou...@gmail.com wrote: - It would be super cool if all of that counter work made it possible to support other atomic data types (sets? CAS? just pass a assoc/commun Function to apply). - Again with types, pluggable type specific compression. - Wishy washy wish: Simpler elasticity I would like to go from 6--8--7 nodes without each of those being an annoying fight with tokens. - Gossip as library. Gossip/failure detection is something C* seems to have gotten particularly right (or at least it's something that has not needed to change much). It would be cool to use Cassandra's gossip protocol as distributed systems building tool a la ZooKeeper. On 11/01/2011 06:59 PM, Jonathan Ellis wrote: Hi all, Two years ago I asked for Cassandra use cases and feature requests. [1] The results [2] have been extremely useful in setting and prioritizing goals for Cassandra development. But with the release of 1.0 we've accomplished basically everything from our original wish list. [3] I'd love to hear from modern Cassandra users again, especially if you're usually a quiet lurker. What does Cassandra do well? What are your pain points? What's your feature wish list? As before, if you're in stealth mode or don't want to say anything in public, feel free to reply to me privately and I will keep it off the record. [1] http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01148.html [2] http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html [3] http://www.mail-archive.com/dev@cassandra.apache.org/msg01524.html -- http://twitter.com/tjake
Re: Second Cassandra users survey
+1 on coprocessors On Mon, Nov 14, 2011 at 6:51 PM, Mohit Anchlia mohitanch...@gmail.comwrote: On Mon, Nov 14, 2011 at 4:44 PM, Jake Luciani jak...@gmail.com wrote: Re Simpler elasticity: Latest opscenter will now rebalance cluster optimally http://www.datastax.com/dev/blog/whats-new-in-opscenter-1-3 /plug Does it cause any impact on reads and writes while re-balance is in progress? How is it handled on live cluster? -Jake On Mon, Nov 14, 2011 at 7:27 PM, Chris Burroughs chris.burrou...@gmail.com wrote: - It would be super cool if all of that counter work made it possible to support other atomic data types (sets? CAS? just pass a assoc/commun Function to apply). - Again with types, pluggable type specific compression. - Wishy washy wish: Simpler elasticity I would like to go from 6--8--7 nodes without each of those being an annoying fight with tokens. - Gossip as library. Gossip/failure detection is something C* seems to have gotten particularly right (or at least it's something that has not needed to change much). It would be cool to use Cassandra's gossip protocol as distributed systems building tool a la ZooKeeper. On 11/01/2011 06:59 PM, Jonathan Ellis wrote: Hi all, Two years ago I asked for Cassandra use cases and feature requests. [1] The results [2] have been extremely useful in setting and prioritizing goals for Cassandra development. But with the release of 1.0 we've accomplished basically everything from our original wish list. [3] I'd love to hear from modern Cassandra users again, especially if you're usually a quiet lurker. What does Cassandra do well? What are your pain points? What's your feature wish list? As before, if you're in stealth mode or don't want to say anything in public, feel free to reply to me privately and I will keep it off the record. [1] http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01148.html [2] http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html [3] http://www.mail-archive.com/dev@cassandra.apache.org/msg01524.html -- http://twitter.com/tjake
wordcount example exception that I think should not occur
There are 4 jobs submitted by the wordcount cassandra example and the first one fails and the other 3 all pass and work with results. The first job I noticed is looking for column name text0 due to i being 0 in the loop. The exception is not going through the wordcount code at all though, but this text0 column does not exist in the cassandra database(is this an InputFormat bug in the cassandra code to wire in the map/reduce stuff or is there a way to fix this???) The exception... 2011-11-14 18:16:26,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_20141812_0002_m_00_1: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:300) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:183) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:139) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) thanks, Dean
This seems to fix it.....wordcount example exception that I think should not occur
Well, by edting src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java in version 1.0.2 cassandra src just before the totalRead++; KeySlice ks = rows.get(i++); SortedMapByteBuffer, IColumn map = new TreeMapByteBuffer, IColumn(comparator); I added the code if(i = rows.size()) return endOfData(); That seems to fix it AND this is only needed by the jobtracker so you don't need to redeploy cassandra for the fix which is niceit seems to work now with rows that don't contain that column. If anyone thinks this will have bad side affects, please let me know. later, Dean http://blog.alvazan.com On Mon, Nov 14, 2011 at 7:01 PM, Dean Hiller d...@alvazan.com wrote: There are 4 jobs submitted by the wordcount cassandra example and the first one fails and the other 3 all pass and work with results. The first job I noticed is looking for column name text0 due to i being 0 in the loop. The exception is not going through the wordcount code at all though, but this text0 column does not exist in the cassandra database(is this an InputFormat bug in the cassandra code to wire in the map/reduce stuff or is there a way to fix this???) The exception... 2011-11-14 18:16:26,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_20141812_0002_m_00_1: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:300) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:183) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:139) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) thanks, Dean
Re: Second Cassandra users survey
oh yeah, one more BIG one.in memory writes with asynch write-behind to disk like cassandra does for speed. So if you have atomic locking, it writes to the primary node(memory) and some other node(memory) and returns with success to the client. asynch then writes to disk later. This prove to be very fast and 2 machines make it pretty reliable and of course it is asynchronously writing to that third or fourth machine depending on replication factor. later, Dean On Mon, Nov 14, 2011 at 6:59 PM, Dean Hiller d...@alvazan.com wrote: +1 on coprocessors On Mon, Nov 14, 2011 at 6:51 PM, Mohit Anchlia mohitanch...@gmail.comwrote: On Mon, Nov 14, 2011 at 4:44 PM, Jake Luciani jak...@gmail.com wrote: Re Simpler elasticity: Latest opscenter will now rebalance cluster optimally http://www.datastax.com/dev/blog/whats-new-in-opscenter-1-3 /plug Does it cause any impact on reads and writes while re-balance is in progress? How is it handled on live cluster? -Jake On Mon, Nov 14, 2011 at 7:27 PM, Chris Burroughs chris.burrou...@gmail.com wrote: - It would be super cool if all of that counter work made it possible to support other atomic data types (sets? CAS? just pass a assoc/commun Function to apply). - Again with types, pluggable type specific compression. - Wishy washy wish: Simpler elasticity I would like to go from 6--8--7 nodes without each of those being an annoying fight with tokens. - Gossip as library. Gossip/failure detection is something C* seems to have gotten particularly right (or at least it's something that has not needed to change much). It would be cool to use Cassandra's gossip protocol as distributed systems building tool a la ZooKeeper. On 11/01/2011 06:59 PM, Jonathan Ellis wrote: Hi all, Two years ago I asked for Cassandra use cases and feature requests. [1] The results [2] have been extremely useful in setting and prioritizing goals for Cassandra development. But with the release of 1.0 we've accomplished basically everything from our original wish list. [3] I'd love to hear from modern Cassandra users again, especially if you're usually a quiet lurker. What does Cassandra do well? What are your pain points? What's your feature wish list? As before, if you're in stealth mode or don't want to say anything in public, feel free to reply to me privately and I will keep it off the record. [1] http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01148.html [2] http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html [3] http://www.mail-archive.com/dev@cassandra.apache.org/msg01524.html -- http://twitter.com/tjake
Re: Second Cassandra users survey
+1 on co-processors. Edward
Re: BulkLoader
Giannis, From here: http://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely Have you tried nodetool removetoken ? Ernie On Mon, Nov 14, 2011 at 4:20 PM, mike...@thomsonreuters.com wrote: Hello Giannis, ** ** Can you share a little bit on how to use the bulk loader ? We’re considering use bulk loader for a user case. ** ** Thanks, Mike ** ** *From:* Giannis Neokleous [mailto:gian...@generalsentiment.com] *Sent:* Monday, November 14, 2011 2:50 PM *To:* user@cassandra.apache.org *Subject:* BulkLoader ** ** Hello everyone, We're using the bulk loader to load data every day to Cassandra. The machines that use the bulkloader are diferent every day so their IP addresses change. When I do describe cluster i see all the unreachable nodes that keep piling up for the past few days. Is there a way to remove those IP addresses without terminating the whole cluster at the same time and restarting it? The unreachable nodes cause issues when we want to make schema changes to all the nodes or when we want to truncate a CF. Any suggestions? -Giannis This email was sent to you by Thomson Reuters, the global news and information company. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Thomson Reuters.
Re: describe cluster error
Can you describe how you did the upgrade on these machines? You may still have some old jars on the classpath. On Mon, Nov 14, 2011 at 4:03 PM, Silviu Matei silvma...@gmail.com wrote: Hi I'm getting this error when I try to run describe cluster: [] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: Error retrieving data: Internal error processing describe_schema_versions with this in the logs: ERROR [pool-2-thread-3] 2011-11-14 21:40:51,975 Cassandra.java (line 3540) Internal error processing describe_schema_versions java.lang.NoSuchMethodError: org.apache.commons.lang.StringUtils.join(Ljava/util/Collection;Ljava/lang/String;)Ljava/lang/String; at org.apache.cassandra.service.StorageProxy.describeSchemaVersions(StorageProxy.java:959) at org.apache.cassandra.thrift.CassandraServer.describe_schema_versions(CassandraServer.java:1151) at org.apache.cassandra.thrift.Cassandra$Processor$describe_schema_versions.process(Cassandra.java:3536) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Anyone have any insight? It's a 3node 1.0.2 cluster, recently upgraded from .87. Thank you
RE: Upgrade Cassandra Cluster to 1.0.2
I am running java version: java version 1.6.0_20 OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode) I have tried to run with -Xss96k or -Xss256k or -Xss320 but it still give me the error TheNode has 8GB memory and %GB allocated to JAVA ERROR [MutationStage:104] 2011-11-15 08:55:59,408 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:104,5,main] java.lang.StackOverflowError at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:209) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:74) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) at org.apache.cassandra.db.Table.apply(Table.java:444) at org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:260) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:159) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) at org.apache.cassandra.db.Table.apply(Table.java:444) at org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:260) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:159) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) at org.apache.cassandra.db.Table.apply(Table.java:444) I ma getting WARN as well after changing JAVA WARN [MutationStage:44] 2011-11-15 08:51:57,479 Memtable.java (line 142) MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Monday, November 14, 2011 7:53 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Mon, Nov 14, 2011 at 8:06 AM, Michael Vaknine micha...@citypath.com wrote: Well, I tried to delete the hints on the failed cluster but I could not start it I got other errors such as ERROR [MutationStage:34] 2011-11-14 15:37:43,813 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:34,5,main] java.lang.StackOverflowError at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216) This is different an unrelated to hints. What jvm are you using? It looks like it just simply ran out of stack space, which is odd, but you can control that with the -Xss option if needed. -Brandon
Re: questions on frequency and timing of async replication between DCs
It may be the case that your CL is the issue. You are writing it at ONE, which means that out of the 4 replicas of that key (two in each data center), you are only putting it on one of them. cassandra will always try to replicate key to all available replicas. Under normal conditions if you do write + wait about 1 sec + then read at CL.ONE you should get back correct value. Consistency in cassandra is not problem in real world, its way better then SQL databases with async replication. Make sure you have your clients clock synced via NTP,
