Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-09 Thread aaron morton
That was my initial thought, just wanted to see if there was anything else going on. Sounds like Henrik has a workaround so all is well. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 9 May 2011, at 18:10, Jonathan Ellis wrote:

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-08 Thread Jonathan Ellis
Strongly suspect that he has invalid unicode characters in his keys. 0.6 wasn't as good at validating those as 0.7. On Sun, May 8, 2011 at 8:51 PM, aaron morton wrote: > Out of interest i've done some more digging. Not sure how much more I've > contributed but here goes... > Ran this against an c

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-08 Thread aaron morton
Out of interest i've done some more digging. Not sure how much more I've contributed but here goes... Ran this against an clean v 0.6.12 and it works (I expected it to fail on the first read) client = pycassa.connect() standard1 = pycassa.ColumnFamily(client, 'Keyspace1', 'Standard1')

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-07 Thread Jonathan Ellis
Right, that's sort of a half-repair: it will repair differences in replies it got, but it won't doublecheck md5s on the rest in the background. So if you're doing CL.ONE reads this is a no-op. On Sat, May 7, 2011 at 4:25 PM, aaron morton wrote: > I remembered something like that so had a look at

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-07 Thread aaron morton
I remembered something like that so had a look at RangeSliceResponseResolver.resolve() in 0.6.12 and it looks like it schedules the repairs... protected Row getReduced() { ColumnFamily resolved = ReadResponseResolver.resolveSuperset(versions);

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-07 Thread Jake Luciani
I know thrift and python and Unicode don't mix. On May 7, 2011, at 4:21 PM, aaron morton wrote: > I've been able to reproduce the fault using python on my mac book see > https://github.com/amorton/cassandra-unicode-bug > > When we try to find the unicode key in the index in 0.7 it fails be

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-07 Thread aaron morton
I've been able to reproduce the fault using python on my mac book see https://github.com/amorton/cassandra-unicode-bug When we try to find the unicode key in the index in 0.7 it fails because the tokens are different. The readme in the github project has more info. Any thoughts? Will try to f

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-07 Thread Jonathan Ellis
range_slices respects consistencylevel, but only single-row reads and multiget do the *repair* part of RR. On Sat, May 7, 2011 at 1:44 AM, aaron morton wrote: > get_range_slices() does read repair if enabled (checked > DoConsistencyChecksBoolean in the config, it's on by default) so you should

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-07 Thread aaron morton
get_range_slices() does read repair if enabled (checked DoConsistencyChecksBoolean in the config, it's on by default) so you should be getting good reads. If you want belt-and-braces run nodetool repair first. Hope that helps. On 7 May 2011, at 11:46, Jeremy Hanna wrote: > Great! I just wan

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-06 Thread Jeremy Hanna
Great! I just wanted to make sure you were getting the information you needed. On May 6, 2011, at 6:42 PM, Henrik Schröder wrote: > Well, I already completed the migration program. Using get_range_slices I > could migrate a few thousand rows per second, which means that migrating all > of our

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-06 Thread Henrik Schröder
Well, I already completed the migration program. Using get_range_slices I could migrate a few thousand rows per second, which means that migrating all of our data would take a few minutes, and we'll end up with pristine datafiles for the new cluster. Problem solved! I'll see if I can create datafi

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-06 Thread Jeremy Hanna
If you're able, go into the #cassandra channel on freenode (IRC) and talk to driftx or jbellis or aaron_morton about your problem. It could be that you don't have to do all of this based on a conversation there. On May 6, 2011, at 5:04 AM, Henrik Schröder wrote: > I'll see if I can make some e

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-06 Thread Henrik Schröder
I'll see if I can make some example broken files this weekend. /Henrik Schröder On Fri, May 6, 2011 at 02:10, aaron morton wrote: > The difficulty is the different thrift clients between 0.6 and 0.7. > > If you want to roll your own solution I would consider: > - write an app to talk to 0.6 an

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-06 Thread Henrik Schröder
Uhm, having a program that can talk to 0.6 and 0.7 servers at the same time is not the hard problem, it took way less than five minutes to copy in both generated clients in the same project and rename the C# namespaces. Two apps and write to disk inbetween? Maven? That's crazy talk. :-D What I was

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-06 Thread Stephen Connolly
maven-shade-plugin could help with having two versions of thrift at the same time... but you'd need to build some stuff with maven, and some people don't like that idea - Stephen --- Sent from my Android phone, so random spelling mistakes, random nonsense words and other nonsense are a direct res

Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-05 Thread aaron morton
The difficulty is the different thrift clients between 0.6 and 0.7. If you want to roll your own solution I would consider: - write an app to talk to 0.6 and pull out the data using keys from the other system (so you know can check referential integrity while you are at it). Dump the data to fla

Migrating all rows from 0.6.13 to 0.7.5 over thrift?

2011-05-05 Thread Henrik Schröder
We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have rows stored that have unicode keys, and Cassandra 0.7.5 thinks those rows in the sstables are corrupt, and it seems impossible to clean it up without losing data. However, we can still read all rows perfectly via thrift so we are