Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
Right, that's sort of a half-repair: it will repair differences in
replies it got, but it won't doublecheck md5s on the rest in the
background. So if you're doing CL.ONE reads this is a no-op.
On Sat, May 7, 2011 at 4:25 PM, aaron morton wrote:
> I remembered something like that so had a look at
> RangeSliceResponseResolver.resolve() in 0.6.12 and it looks like it
> schedules the repairs...
>
> protected Row getReduced()
> {
> ColumnFamily resolved =
> ReadResponseResolver.resolveSuperset(versions);
> ReadResponseResolver.maybeScheduleRepairs(resolved, table,
> key, versions, versionSources);
> versions.clear();
> versionSources.clear();
> return new Row(key, resolved);
> }
>
>
> Is that right?
>
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8 May 2011, at 00:48, Jonathan Ellis wrote:
>
>> range_slices respects consistencylevel, but only single-row reads and
>> multiget do the *repair* part of RR.
>>
>> On Sat, May 7, 2011 at 1:44 AM, aaron morton wrote:
>>> get_range_slices() does read repair if enabled (checked
>>> DoConsistencyChecksBoolean in the config, it's on by default) so you should
>>> be getting good reads. If you want belt-and-braces run nodetool repair
>>> first.
>>>
>>> Hope that helps.
>>>
>>>
>>> On 7 May 2011, at 11:46, Jeremy Hanna wrote:
>>>
Great! I just wanted to make sure you were getting the information you
needed.
On May 6, 2011, at 6:42 PM, Henrik Schröder wrote:
> Well, I already completed the migration program. Using get_range_slices I
> could migrate a few thousand rows per second, which means that migrating
> all of our data would take a few minutes, and we'll end up with pristine
> datafiles for the new cluster. Problem solved!
>
> I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so
> that you all can repeat this and hopefully fix it.
>
>
> /Henrik Schröder
>
> On Sat, May 7, 2011 at 00:35, Jeremy Hanna
> wrote:
> If you're able, go into the #cassandra channel on freenode (IRC) and talk
> to driftx or jbellis or aaron_morton about your problem. It could be
> that you don't have to do all of this based on a conversation there.
>
> On May 6, 2011, at 5:04 AM, Henrik Schröder wrote:
>
>> I'll see if I can make some example broken files this weekend.
>>
>>
>> /Henrik Schröder
>>
>> On Fri, May 6, 2011 at 02:10, aaron morton
>> wrote:
>> The difficulty is the different thrift clients between 0.6 and 0.7.
>>
>> If you want to roll your own solution I would consider:
>> - write an app to talk to 0.6 and pull out the data using keys from the
>> other system (so you know can check referential integrity while you are
>> at it). Dump the data to flat file.
>> - write an app to talk to 0.7 to load the data back in.
>>
>> I've not given up digging on your migration problem, having to manually
>> dump and reload if you've done nothing wrong is not the best solution.
>> I'll try to find some time this weekend to test with:
>>
>> - 0.6 server, random paritioner, standard CF's, byte column
>> - load with python or the cli on osx or ubuntu (dont have a window
>> machine any more)
>> - migrate and see whats going on.
>>
>> If you can spare some sample data to load please send it over in the
>> user group or my email address.
>>
>> Cheers
>>
>> -
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 6 May 2011, at 05:52, Henrik Schröder wrote:
>>
>>> We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have
>>> rows stored that have unicode keys, and Cassandra 0.7.5 thinks those
>>> rows in the sstables are corrupt, and it seems impossible to clean it
>>> up without losing data.
>>>
>>> However, we can still read all rows perfectly via thrift so we are now
>>> looking at building a simple tool that will copy all rows from our
>>> 0.6.3 cluster to a parallell 0.7.5 cluster. Our question is now how to
>>> do that and ensure that we actually get all rows migrated? It's a
>>> pretty small cluster, 3 machines, a single keyspace, a singke
>>> columnfamily, ~2 million rows, a few GB of data, and a replication
>>> factor of 3.
>>>
>>> So what's the best way? Call get_range_slices and move through the
>>> entire token space? We also have all row keys in a secondary system,
>>> would it be better to use that and make calls to get_multi or
>>> get_multi_slices instead? Are we correct in assuming that if we use the
>>> consistencylevel ALL we'll get all rows?
>>>
>>
Re: Memory Usage During Read
The live:serialized size ratio depends on what your data looks like (small columns will be less efficient than large blobs) but using the rule of thumb of 10x, around 1G * (1 + memtable_flush_writers + memtable_flush_queue_size). So first thing I would do is drop writers and queue to 1 and 1. Then I would drop the max heap to 1G, memtable size to 8MB so the heap dump is easier to analyze. Then let it OOM and look at the dump with http://www.eclipse.org/mat/ On Sat, May 7, 2011 at 3:54 PM, Serediuk, Adam wrote: > How much memory should a single hot cf with a 128mb memtable take with row > and key caching disabled during read? > > Because I'm seeing heap go from 3.5gb skyrocketing straight to max > (regardless of the size, 8gb and 24gb both do the same) at which time the jvm > will do nothing but full gc and is unable to reclaim any meaningful amount of > memory. Cassandra then becomes unusable. > > I see the same behavior with smaller memtables, eg 64mb. > > This happens well into the read operation an only on a small number of nodes > in the cluster(1-4 out of a total of 60 nodes.) > > Sent from my iPhone > > On May 6, 2011, at 22:45, "Jonathan Ellis" wrote: > >> You don't GC storm without legitimately having a too-full heap. It's >> normal to see occasional full GCs from fragmentation, but that will >> actually compact the heap and everything goes back to normal IF you >> had space actually freed up. >> >> You say you've played w/ memtable size but that would still be my bet. >> Most people severely underestimate how much space this takes (10x in >> memory over serialized size), which will bite you when you have lots >> of CFs defined. >> >> Otherwise, force a heap dump after a full GC and take a look to see >> what's referencing all the memory. >> >> On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam >> wrote: >>> We're troubleshooting a memory usage problem during batch reads. We've >>> spent the last few days profiling and trying different GC settings. The >>> symptoms are that after a certain amount of time during reads one or more >>> nodes in the cluster will exhibit extreme memory pressure followed by a gc >>> storm. We've tried every possible JVM setting and different GC methods and >>> the issue persists. This is pointing towards something instantiating a lot >>> of objects and keeping references so that they can't be cleaned up. >>> >>> Typically nothing is ever logged other than the GC failures however just >>> now one of the nodes emitted logs we've never seen before: >>> >>> INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java (line >>> 2218) Unable to reduce heap usage since there are no dirty column families >>> >>> We have tried increasing the heap on these nodes to large values, eg 24GB >>> and still run into the same issue. We're running 8GB of heap normally and >>> only one or two nodes will ever exhibit this issue, randomly. We don't use >>> key/row caching and our memtable sizing is 64mb/0.3. Larger or smaller >>> memtables make no difference in avoiding the issue. We're on 0.7.5, mmap, >>> jna and jdk 1.6.0_24 >>> >>> We've somewhat hit the wall in troubleshooting and any advice is greatly >>> appreciated. >>> >>> -- >>> Adam >>> >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com >> > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
I remembered something like that so had a look at
RangeSliceResponseResolver.resolve() in 0.6.12 and it looks like it schedules
the repairs...
protected Row getReduced()
{
ColumnFamily resolved =
ReadResponseResolver.resolveSuperset(versions);
ReadResponseResolver.maybeScheduleRepairs(resolved, table, key,
versions, versionSources);
versions.clear();
versionSources.clear();
return new Row(key, resolved);
}
Is that right?
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 8 May 2011, at 00:48, Jonathan Ellis wrote:
> range_slices respects consistencylevel, but only single-row reads and
> multiget do the *repair* part of RR.
>
> On Sat, May 7, 2011 at 1:44 AM, aaron morton wrote:
>> get_range_slices() does read repair if enabled (checked
>> DoConsistencyChecksBoolean in the config, it's on by default) so you should
>> be getting good reads. If you want belt-and-braces run nodetool repair first.
>>
>> Hope that helps.
>>
>>
>> On 7 May 2011, at 11:46, Jeremy Hanna wrote:
>>
>>> Great! I just wanted to make sure you were getting the information you
>>> needed.
>>>
>>> On May 6, 2011, at 6:42 PM, Henrik Schröder wrote:
>>>
Well, I already completed the migration program. Using get_range_slices I
could migrate a few thousand rows per second, which means that migrating
all of our data would take a few minutes, and we'll end up with pristine
datafiles for the new cluster. Problem solved!
I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so
that you all can repeat this and hopefully fix it.
/Henrik Schröder
On Sat, May 7, 2011 at 00:35, Jeremy Hanna
wrote:
If you're able, go into the #cassandra channel on freenode (IRC) and talk
to driftx or jbellis or aaron_morton about your problem. It could be that
you don't have to do all of this based on a conversation there.
On May 6, 2011, at 5:04 AM, Henrik Schröder wrote:
> I'll see if I can make some example broken files this weekend.
>
>
> /Henrik Schröder
>
> On Fri, May 6, 2011 at 02:10, aaron morton
> wrote:
> The difficulty is the different thrift clients between 0.6 and 0.7.
>
> If you want to roll your own solution I would consider:
> - write an app to talk to 0.6 and pull out the data using keys from the
> other system (so you know can check referential integrity while you are
> at it). Dump the data to flat file.
> - write an app to talk to 0.7 to load the data back in.
>
> I've not given up digging on your migration problem, having to manually
> dump and reload if you've done nothing wrong is not the best solution.
> I'll try to find some time this weekend to test with:
>
> - 0.6 server, random paritioner, standard CF's, byte column
> - load with python or the cli on osx or ubuntu (dont have a window
> machine any more)
> - migrate and see whats going on.
>
> If you can spare some sample data to load please send it over in the user
> group or my email address.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6 May 2011, at 05:52, Henrik Schröder wrote:
>
>> We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have rows
>> stored that have unicode keys, and Cassandra 0.7.5 thinks those rows in
>> the sstables are corrupt, and it seems impossible to clean it up without
>> losing data.
>>
>> However, we can still read all rows perfectly via thrift so we are now
>> looking at building a simple tool that will copy all rows from our 0.6.3
>> cluster to a parallell 0.7.5 cluster. Our question is now how to do that
>> and ensure that we actually get all rows migrated? It's a pretty small
>> cluster, 3 machines, a single keyspace, a singke columnfamily, ~2
>> million rows, a few GB of data, and a replication factor of 3.
>>
>> So what's the best way? Call get_range_slices and move through the
>> entire token space? We also have all row keys in a secondary system,
>> would it be better to use that and make calls to get_multi or
>> get_multi_slices instead? Are we correct in assuming that if we use the
>> consistencylevel ALL we'll get all rows?
>>
>>
>> /Henrik Schröder
>
>
>>>
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: Memory Usage During Read
How much memory should a single hot cf with a 128mb memtable take with row and key caching disabled during read? Because I'm seeing heap go from 3.5gb skyrocketing straight to max (regardless of the size, 8gb and 24gb both do the same) at which time the jvm will do nothing but full gc and is unable to reclaim any meaningful amount of memory. Cassandra then becomes unusable. I see the same behavior with smaller memtables, eg 64mb. This happens well into the read operation an only on a small number of nodes in the cluster(1-4 out of a total of 60 nodes.) Sent from my iPhone On May 6, 2011, at 22:45, "Jonathan Ellis" wrote: > You don't GC storm without legitimately having a too-full heap. It's > normal to see occasional full GCs from fragmentation, but that will > actually compact the heap and everything goes back to normal IF you > had space actually freed up. > > You say you've played w/ memtable size but that would still be my bet. > Most people severely underestimate how much space this takes (10x in > memory over serialized size), which will bite you when you have lots > of CFs defined. > > Otherwise, force a heap dump after a full GC and take a look to see > what's referencing all the memory. > > On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam > wrote: >> We're troubleshooting a memory usage problem during batch reads. We've spent >> the last few days profiling and trying different GC settings. The symptoms >> are that after a certain amount of time during reads one or more nodes in >> the cluster will exhibit extreme memory pressure followed by a gc storm. >> We've tried every possible JVM setting and different GC methods and the >> issue persists. This is pointing towards something instantiating a lot of >> objects and keeping references so that they can't be cleaned up. >> >> Typically nothing is ever logged other than the GC failures however just now >> one of the nodes emitted logs we've never seen before: >> >> INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java (line >> 2218) Unable to reduce heap usage since there are no dirty column families >> >> We have tried increasing the heap on these nodes to large values, eg 24GB >> and still run into the same issue. We're running 8GB of heap normally and >> only one or two nodes will ever exhibit this issue, randomly. We don't use >> key/row caching and our memtable sizing is 64mb/0.3. Larger or smaller >> memtables make no difference in avoiding the issue. We're on 0.7.5, mmap, >> jna and jdk 1.6.0_24 >> >> We've somewhat hit the wall in troubleshooting and any advice is greatly >> appreciated. >> >> -- >> Adam >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
I know thrift and python and Unicode don't mix. On May 7, 2011, at 4:21 PM, aaron morton wrote: > I've been able to reproduce the fault using python on my mac book see > https://github.com/amorton/cassandra-unicode-bug > > When we try to find the unicode key in the index in 0.7 it fails because the > tokens are different. The readme in the github project has more info. > > Any thoughts? Will try to find some more time to keep digging. > > Aaron > > > On 7 May 2011, at 20:44, aaron morton wrote: > >> get_range_slices() does read repair if enabled (checked >> DoConsistencyChecksBoolean in the config, it's on by default) so you should >> be getting good reads. If you want belt-and-braces run nodetool repair >> first. >> >> Hope that helps. >> >> >> On 7 May 2011, at 11:46, Jeremy Hanna wrote: >> >>> Great! I just wanted to make sure you were getting the information you >>> needed. >>> >>> On May 6, 2011, at 6:42 PM, Henrik Schröder wrote: >>> Well, I already completed the migration program. Using get_range_slices I could migrate a few thousand rows per second, which means that migrating all of our data would take a few minutes, and we'll end up with pristine datafiles for the new cluster. Problem solved! I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so that you all can repeat this and hopefully fix it. /Henrik Schröder On Sat, May 7, 2011 at 00:35, Jeremy Hanna wrote: If you're able, go into the #cassandra channel on freenode (IRC) and talk to driftx or jbellis or aaron_morton about your problem. It could be that you don't have to do all of this based on a conversation there. On May 6, 2011, at 5:04 AM, Henrik Schröder wrote: > I'll see if I can make some example broken files this weekend. > > > /Henrik Schröder > > On Fri, May 6, 2011 at 02:10, aaron morton > wrote: > The difficulty is the different thrift clients between 0.6 and 0.7. > > If you want to roll your own solution I would consider: > - write an app to talk to 0.6 and pull out the data using keys from the > other system (so you know can check referential integrity while you are > at it). Dump the data to flat file. > - write an app to talk to 0.7 to load the data back in. > > I've not given up digging on your migration problem, having to manually > dump and reload if you've done nothing wrong is not the best solution. > I'll try to find some time this weekend to test with: > > - 0.6 server, random paritioner, standard CF's, byte column > - load with python or the cli on osx or ubuntu (dont have a window > machine any more) > - migrate and see whats going on. > > If you can spare some sample data to load please send it over in the user > group or my email address. > > Cheers > > - > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 6 May 2011, at 05:52, Henrik Schröder wrote: > >> We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have rows >> stored that have unicode keys, and Cassandra 0.7.5 thinks those rows in >> the sstables are corrupt, and it seems impossible to clean it up without >> losing data. >> >> However, we can still read all rows perfectly via thrift so we are now >> looking at building a simple tool that will copy all rows from our 0.6.3 >> cluster to a parallell 0.7.5 cluster. Our question is now how to do that >> and ensure that we actually get all rows migrated? It's a pretty small >> cluster, 3 machines, a single keyspace, a singke columnfamily, ~2 >> million rows, a few GB of data, and a replication factor of 3. >> >> So what's the best way? Call get_range_slices and move through the >> entire token space? We also have all row keys in a secondary system, >> would it be better to use that and make calls to get_multi or >> get_multi_slices instead? Are we correct in assuming that if we use the >> consistencylevel ALL we'll get all rows? >> >> >> /Henrik Schröder > > >>> >> >
Re: CFHistograms?
I believe the offset value of Writes and Reads are in *micro*seconds right? (that page talks about *milli*seconds) Also, are any timeouts or errors reflected in those times or just successful operations? if not, is there any JMX or other tool to keep track of them? Josep M. On Fri, May 6, 2011 at 9:09 AM, Jonathan Ellis wrote: > Those are updated at compaction time. > > On Thu, May 5, 2011 at 11:38 PM, Xaero S wrote: > > > > Can someone point me to a document that explains how to interpret > > CFHistograms output? i went through > > > http://narendrasharma.blogspot.com/2011/04/cassandra-07x-understanding-output-of.html > > which is a good beginning, but was wondering if there was anything more > > detailed. For e.g when i run CFHistograms, i see rowsize and columncount > > items in the table always 0 (which cant be right?) > > > > -Xaero > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
I've been able to reproduce the fault using python on my mac book see https://github.com/amorton/cassandra-unicode-bug When we try to find the unicode key in the index in 0.7 it fails because the tokens are different. The readme in the github project has more info. Any thoughts? Will try to find some more time to keep digging. Aaron On 7 May 2011, at 20:44, aaron morton wrote: > get_range_slices() does read repair if enabled (checked > DoConsistencyChecksBoolean in the config, it's on by default) so you should > be getting good reads. If you want belt-and-braces run nodetool repair first. > > Hope that helps. > > > On 7 May 2011, at 11:46, Jeremy Hanna wrote: > >> Great! I just wanted to make sure you were getting the information you >> needed. >> >> On May 6, 2011, at 6:42 PM, Henrik Schröder wrote: >> >>> Well, I already completed the migration program. Using get_range_slices I >>> could migrate a few thousand rows per second, which means that migrating >>> all of our data would take a few minutes, and we'll end up with pristine >>> datafiles for the new cluster. Problem solved! >>> >>> I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so >>> that you all can repeat this and hopefully fix it. >>> >>> >>> /Henrik Schröder >>> >>> On Sat, May 7, 2011 at 00:35, Jeremy Hanna >>> wrote: >>> If you're able, go into the #cassandra channel on freenode (IRC) and talk >>> to driftx or jbellis or aaron_morton about your problem. It could be that >>> you don't have to do all of this based on a conversation there. >>> >>> On May 6, 2011, at 5:04 AM, Henrik Schröder wrote: >>> I'll see if I can make some example broken files this weekend. /Henrik Schröder On Fri, May 6, 2011 at 02:10, aaron morton wrote: The difficulty is the different thrift clients between 0.6 and 0.7. If you want to roll your own solution I would consider: - write an app to talk to 0.6 and pull out the data using keys from the other system (so you know can check referential integrity while you are at it). Dump the data to flat file. - write an app to talk to 0.7 to load the data back in. I've not given up digging on your migration problem, having to manually dump and reload if you've done nothing wrong is not the best solution. I'll try to find some time this weekend to test with: - 0.6 server, random paritioner, standard CF's, byte column - load with python or the cli on osx or ubuntu (dont have a window machine any more) - migrate and see whats going on. If you can spare some sample data to load please send it over in the user group or my email address. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6 May 2011, at 05:52, Henrik Schröder wrote: > We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have rows > stored that have unicode keys, and Cassandra 0.7.5 thinks those rows in > the sstables are corrupt, and it seems impossible to clean it up without > losing data. > > However, we can still read all rows perfectly via thrift so we are now > looking at building a simple tool that will copy all rows from our 0.6.3 > cluster to a parallell 0.7.5 cluster. Our question is now how to do that > and ensure that we actually get all rows migrated? It's a pretty small > cluster, 3 machines, a single keyspace, a singke columnfamily, ~2 million > rows, a few GB of data, and a replication factor of 3. > > So what's the best way? Call get_range_slices and move through the entire > token space? We also have all row keys in a secondary system, would it be > better to use that and make calls to get_multi or get_multi_slices > instead? Are we correct in assuming that if we use the consistencylevel > ALL we'll get all rows? > > > /Henrik Schröder >>> >>> >> >
Re: GC for ParNew (cassandra under performance ) 0.6.3
Hi All some on suggested about linux scheduler cat /sys/block/sda/queue/scheduler there are quite few of them noop [anticipatory] deadline cfq.I have changed from cfq default to anticipatory,Now problem is less for me kswapd0 take 10-50% of cpu in very short bust less then milliseconds.Also i have changed /proc/sys/vm/pagecache from 100 to 10. On 05/07/2011 10:36 PM, Ali Ahsan wrote: This what free -m and vmstat show now my CPU is on 100% [root@cassandra2 ~]# free -m total used free sharedbuffers cached Mem: 16043 15966 76 0 4900 -/+ buffers/cache: 15061982 Swap:0 0 0 root@cassandra2 ~]# vmstat procs ---memory-- ---swap-- -io --system-- -cpu-- r b swpd free buff cache si sobibo in cs us sy id wa st 1 8 0 79312 4776 92180011373211 1 0 98 1 0 On 05/07/2011 05:47 PM, Jonathan Ellis wrote: kswapd0 100% -- S.Ali Ahsan Senior System Engineer e-Business (Pvt) Ltd 49-C Jail Road, Lahore, P.O. Box 676 Lahore 54000, Pakistan Tel: +92 (0)42 3758 7140 Ext. 128 Mobile: +92 (0)345 831 8769 Fax: +92 (0)42 3758 0027 Email: ali.ah...@panasiangroup.com www.ebusiness-pg.com www.panasiangroup.com Confidentiality: This e-mail and any attachments may be confidential and/or privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person use it for any purpose or store or copy the information in any medium. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. We do not accept liability for any errors or omissions.
Re: Performance thoughts on CQL
The only "query planning" Cassandra does is deciding which index to use when there are multiple options. That is handled the same way whether the request is from CQL or classic Thrift. On Sat, May 7, 2011 at 8:17 AM, Edward Capriolo wrote: > Having used mysql as key value store before I have noticed that when > doing low-latency queries the query planning time can actually end up > being larger then the execution time. Even if you use a prepared > statement mechanism with clients connecting and disconnecting this can > be a lot of overhead. Will CQL prepare statements on the client or the > server side? How does CQL perform against get_slice when a read comes > from pure row cache? > > Edward > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: compaction strategy
> It does not really make sense to me to go through all these minor merges > when a full compaction will do a much faster and better job. In a system heavily reliant on caching (platter drives, large data sizes, much larger than RAM) major compactions can be very detrimental to performance due to the effects of the temporary spike in data size and cache coldness. Sounds like it makes good sense in your situation though. -- / Peter Schuller
Re: GC for ParNew (cassandra under performance ) 0.6.3
This what free -m and vmstat show now my CPU is on 100% [root@cassandra2 ~]# free -m total used free sharedbuffers cached Mem: 16043 15966 76 0 4900 -/+ buffers/cache: 15061982 Swap:0 0 0 root@cassandra2 ~]# vmstat procs ---memory-- ---swap-- -io --system-- -cpu-- r b swpd free buff cache si sobibo in cs us sy id wa st 1 8 0 79312 4776 92180011373211 1 0 98 1 0 On 05/07/2011 05:47 PM, Jonathan Ellis wrote: kswapd0 100% -- S.Ali Ahsan Senior System Engineer e-Business (Pvt) Ltd 49-C Jail Road, Lahore, P.O. Box 676 Lahore 54000, Pakistan Tel: +92 (0)42 3758 7140 Ext. 128 Mobile: +92 (0)345 831 8769 Fax: +92 (0)42 3758 0027 Email: ali.ah...@panasiangroup.com www.ebusiness-pg.com www.panasiangroup.com Confidentiality: This e-mail and any attachments may be confidential and/or privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person use it for any purpose or store or copy the information in any medium. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. We do not accept liability for any errors or omissions.
Re: compaction strategy
This is an all ssd system. I have no problems with read/write performance due to I/O. I do have a potential with the crazy explosion you can get in terms of disk use if compaction cannot keep up. As things falls behind and you get many generations of data, yes, read performance gets a problem due to the number of sstables. As things start falling behind, you have a bunch of minor compactions trying to merge 20MB (sstables cassandra generally dumps with current config when under pressure) into 40 MB into 80MB into Anyone wants to do the math on how many times you are rewriting the data going this route? There is just no way this can keep up. It will just fall more and more behind. Only way to recover as I can see would be to trigger a full compaction? It does not really make sense to me to go through all these minor merges when a full compaction will do a much faster and better job. Terje On Sat, May 7, 2011 at 9:54 PM, Jonathan Ellis wrote: > On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen > wrote: > > 1. Would it make sense to make full compactions occur a bit more > aggressive. > > I'd rather reduce the performance impact of being behind, than do more > full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498 > > > 2. I > > would think the code should be smart enough to either trigger a full > > compaction and scrap the current queue, or at least merge some of those > > pending tasks into larger ones > > Not crazy but a queue-rewriter would be nontrivial. For now I'm okay > with saying "add capacity until compaction can mostly keep up." (Most > people's problem is making compaction LESS aggressive, hence > https://issues.apache.org/jira/browse/CASSANDRA-2156.) > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: compaction strategy
> If you are seeing 600 pending compaction tasks regularly you almost > definitely need more hardware. Note that pending compactions is pretty misleading and you can't really draw conclusions just based on the pending compactions number/graph. For example, standard behavior during e.g.a long repair may end up accumulating thousands of pending compactions that suddenly drop to zero once you're done and a bunch of tasks that don't actually need to do anything are "completed". With the concurrent compaction support I suppose this will be mitigated as long as you don't hit your concurrency limit. -- / Peter Schuller
Re: compaction strategy
On Sat, May 7, 2011 at 8:54 AM, Jonathan Ellis wrote: > On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen > wrote: >> 1. Would it make sense to make full compactions occur a bit more aggressive. > > I'd rather reduce the performance impact of being behind, than do more > full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498 > >> 2. I >> would think the code should be smart enough to either trigger a full >> compaction and scrap the current queue, or at least merge some of those >> pending tasks into larger ones > > Not crazy but a queue-rewriter would be nontrivial. For now I'm okay > with saying "add capacity until compaction can mostly keep up." (Most > people's problem is making compaction LESS aggressive, hence > https://issues.apache.org/jira/browse/CASSANDRA-2156.) > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > Adjusting compaction and memtable settings is a tuning thing. Tuning is not usually a game changer. Every once in a while you hit a something wonderful and get 20% or 30% enhancement. But normally you are in the gain 1%-3% range. If you are seeing 600 pending compaction tasks regularly you almost definitely need more hardware.
Re: Ant error in Eclipse when building Cassandra
Yes, it is, those last two lines repeat forever: at org.apache.tools.ant.PropertyHelper.getPropertyHook(PropertyHelper.java:189) at org.apache.maven.artifact.ant.POMPropertyHelper.getPropertyHook(POMPropertyHelper.java:50) It's trying to look up a property value somewhere and getting caught in a loop. It's odd that it only happens in Eclipse's auto-rebuild cycle though, not when it's run explicitly or from the command line, so somethings being set or not set by Eclipse for Ant. I'll try it on my PC and see if the same thing happens. Ed On Fri, May 6, 2011 at 10:49 PM, Jonathan Ellis wrote: > Default stack is huge, so maven-ant-tasks-retrieve-build is probably > recursing infinitely somewhere :( > > On Fri, May 6, 2011 at 2:42 PM, Ed Anuff wrote: >> I finally got around to getting Eclipse set up to build Cassandra >> following the directions on the wiki and it seems to be working, >> Eclipse isn't showing any errors except that when it fires off the >> automatic ant build I get the following error: >> >> maven-ant-tasks-retrieve-build: >> >> BUILD FAILED >> java.lang.StackOverflowError >> at >> org.apache.tools.ant.Project.executeSortedTargets(Project.java:1346) >> at org.apache.tools.ant.Project.executeTarget(Project.java:1306) >> at >> org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) >> at >> org.eclipse.ant.internal.core.ant.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:32) >> at org.apache.tools.ant.Project.executeTargets(Project.java:1189) >> at >> org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:662) >> at >> org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:495) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.eclipse.ant.core.AntRunner.run(AntRunner.java:378) >> at >> org.eclipse.ant.internal.launching.launchConfigurations.AntLaunchDelegate.runInSameVM(AntLaunchDelegate.java:321) >> at >> org.eclipse.ant.internal.launching.launchConfigurations.AntLaunchDelegate.launch(AntLaunchDelegate.java:274) >> at >> org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:853) >> at >> org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:702) >> at >> org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:695) >> at >> org.eclipse.core.externaltools.internal.model.ExternalToolBuilder.launchBuild(ExternalToolBuilder.java:181) >> at >> org.eclipse.core.externaltools.internal.model.ExternalToolBuilder.doBuildBasedOnScope(ExternalToolBuilder.java:169) >> at >> org.eclipse.core.externaltools.internal.model.ExternalToolBuilder.build(ExternalToolBuilder.java:88) >> at >> org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:629) >> at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) >> at >> org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:172) >> at >> org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:203) >> at >> org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:255) >> at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) >> at >> org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:258) >> at >> org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:311) >> at >> org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:343) >> at >> org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:144) >> at >> org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:242) >> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) >> Caused by: java.lang.StackOverflowError >> at >> org.apache.tools.ant.PropertyHelper.getPropertyHook(PropertyHelper.java:189) >> at >> org.apache.maven.artifact.ant.POMPropertyHelper.getPropertyHook(POMPropertyHelper.java:50) >> >> >> I never get this error when building from the command line and if I >> right click on build.xml in Eclipse and select Run As Ant Build it >> works fine as well. Any ideas? This is on a Mac. >> >> Ed >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Performance thoughts on CQL
Having used mysql as key value store before I have noticed that when doing low-latency queries the query planning time can actually end up being larger then the execution time. Even if you use a prepared statement mechanism with clients connecting and disconnecting this can be a lot of overhead. Will CQL prepare statements on the client or the server side? How does CQL perform against get_slice when a read comes from pure row cache? Edward
Re: compaction strategy
On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen wrote: > 1. Would it make sense to make full compactions occur a bit more aggressive. I'd rather reduce the performance impact of being behind, than do more full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498 > 2. I > would think the code should be smart enough to either trigger a full > compaction and scrap the current queue, or at least merge some of those > pending tasks into larger ones Not crazy but a queue-rewriter would be nontrivial. For now I'm okay with saying "add capacity until compaction can mostly keep up." (Most people's problem is making compaction LESS aggressive, hence https://issues.apache.org/jira/browse/CASSANDRA-2156.) -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
range_slices respects consistencylevel, but only single-row reads and multiget do the *repair* part of RR. On Sat, May 7, 2011 at 1:44 AM, aaron morton wrote: > get_range_slices() does read repair if enabled (checked > DoConsistencyChecksBoolean in the config, it's on by default) so you should > be getting good reads. If you want belt-and-braces run nodetool repair first. > > Hope that helps. > > > On 7 May 2011, at 11:46, Jeremy Hanna wrote: > >> Great! I just wanted to make sure you were getting the information you >> needed. >> >> On May 6, 2011, at 6:42 PM, Henrik Schröder wrote: >> >>> Well, I already completed the migration program. Using get_range_slices I >>> could migrate a few thousand rows per second, which means that migrating >>> all of our data would take a few minutes, and we'll end up with pristine >>> datafiles for the new cluster. Problem solved! >>> >>> I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so >>> that you all can repeat this and hopefully fix it. >>> >>> >>> /Henrik Schröder >>> >>> On Sat, May 7, 2011 at 00:35, Jeremy Hanna >>> wrote: >>> If you're able, go into the #cassandra channel on freenode (IRC) and talk >>> to driftx or jbellis or aaron_morton about your problem. It could be that >>> you don't have to do all of this based on a conversation there. >>> >>> On May 6, 2011, at 5:04 AM, Henrik Schröder wrote: >>> I'll see if I can make some example broken files this weekend. /Henrik Schröder On Fri, May 6, 2011 at 02:10, aaron morton wrote: The difficulty is the different thrift clients between 0.6 and 0.7. If you want to roll your own solution I would consider: - write an app to talk to 0.6 and pull out the data using keys from the other system (so you know can check referential integrity while you are at it). Dump the data to flat file. - write an app to talk to 0.7 to load the data back in. I've not given up digging on your migration problem, having to manually dump and reload if you've done nothing wrong is not the best solution. I'll try to find some time this weekend to test with: - 0.6 server, random paritioner, standard CF's, byte column - load with python or the cli on osx or ubuntu (dont have a window machine any more) - migrate and see whats going on. If you can spare some sample data to load please send it over in the user group or my email address. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6 May 2011, at 05:52, Henrik Schröder wrote: > We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have rows > stored that have unicode keys, and Cassandra 0.7.5 thinks those rows in > the sstables are corrupt, and it seems impossible to clean it up without > losing data. > > However, we can still read all rows perfectly via thrift so we are now > looking at building a simple tool that will copy all rows from our 0.6.3 > cluster to a parallell 0.7.5 cluster. Our question is now how to do that > and ensure that we actually get all rows migrated? It's a pretty small > cluster, 3 machines, a single keyspace, a singke columnfamily, ~2 million > rows, a few GB of data, and a replication factor of 3. > > So what's the best way? Call get_range_slices and move through the entire > token space? We also have all row keys in a secondary system, would it be > better to use that and make calls to get_multi or get_multi_slices > instead? Are we correct in assuming that if we use the consistencylevel > ALL we'll get all rows? > > > /Henrik Schröder >>> >>> >> > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: GC for ParNew (cassandra under performance ) 0.6.3
I'll quote the top google hit for [kswapd0 100%]: How much memory did you have in "cached" when you looked with top (and no swap enabled) ? If the amount of "cached" memory is very low, it could mean that your shared libraries are being pushed out of memory, instead of the kernel swapping out some page that belongs to only one process. On Sat, May 7, 2011 at 1:39 AM, Ali Ahsan wrote: > Hi That sloved my issue of GC,Now i am facing with new one > > i have no swap but this process kswapd0 take up 100% cpu when there is load > on cassandra any idea why this is happening > > On 05/05/2011 09:06 PM, Ali Ahsan wrote: >> >> Thanks replying,let me disable my swap memory. >> >> On 05/05/2011 09:01 PM, Jonathan Ellis wrote: >>> >>> 6s parnew is insane. you're probably swapping. Easiest fix is >>> disabling swap entirely. >>> >>> P.S. 0.6.3 is ancient. >>> >>> >> >> > > > -- > S.Ali Ahsan > > Senior System Engineer > > e-Business (Pvt) Ltd > > 49-C Jail Road, Lahore, P.O. Box 676 > Lahore 54000, Pakistan > > Tel: +92 (0)42 3758 7140 Ext. 128 > > Mobile: +92 (0)345 831 8769 > > Fax: +92 (0)42 3758 0027 > > Email: ali.ah...@panasiangroup.com > > > > www.ebusiness-pg.com > > www.panasiangroup.com > > Confidentiality: This e-mail and any attachments may be confidential > and/or privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person > use it for any purpose or store or copy the information in any medium. > Internet communications cannot be guaranteed to be timely, secure, error > or virus-free. We do not accept liability for any errors or omissions. > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
compaction strategy
Even with the current concurrent compactions, given a high speed datafeed, compactions will obviously start lagging at some stage, and once it does, things can turn bad in terms of disk usage and read performance. I have not read the compaction code well, but if http://wiki.apache.org/cassandra/MemtableSSTable is up to date, I am wondering if it: 1. Would it make sense to make full compactions occur a bit more aggressive. That is, regardless of sstables of matching sizes, if the total number of outstanding sstables gets above a certain datasize, would it make sense to just schedule a full compaction rather than go through all the hoops of gradually merging them in groups of matching sizes? 2. This is same topic, just another viewpoint. When you get to the stage that compactionstats shows you something crazy like "pending tasks: 600", I would think the code should be smart enough to either trigger a full compaction and scrap the current queue, or at least merge some of those pending tasks into larger ones instead of reading and writing the same data again and again gradually merging it into larger and larger sizes? The target behind both 1 and 2 would be to reduce the number of time data is re-read and re-written in compactions before you reach the full dataset size.
Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
get_range_slices() does read repair if enabled (checked DoConsistencyChecksBoolean in the config, it's on by default) so you should be getting good reads. If you want belt-and-braces run nodetool repair first. Hope that helps. On 7 May 2011, at 11:46, Jeremy Hanna wrote: > Great! I just wanted to make sure you were getting the information you > needed. > > On May 6, 2011, at 6:42 PM, Henrik Schröder wrote: > >> Well, I already completed the migration program. Using get_range_slices I >> could migrate a few thousand rows per second, which means that migrating all >> of our data would take a few minutes, and we'll end up with pristine >> datafiles for the new cluster. Problem solved! >> >> I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so >> that you all can repeat this and hopefully fix it. >> >> >> /Henrik Schröder >> >> On Sat, May 7, 2011 at 00:35, Jeremy Hanna >> wrote: >> If you're able, go into the #cassandra channel on freenode (IRC) and talk to >> driftx or jbellis or aaron_morton about your problem. It could be that you >> don't have to do all of this based on a conversation there. >> >> On May 6, 2011, at 5:04 AM, Henrik Schröder wrote: >> >>> I'll see if I can make some example broken files this weekend. >>> >>> >>> /Henrik Schröder >>> >>> On Fri, May 6, 2011 at 02:10, aaron morton wrote: >>> The difficulty is the different thrift clients between 0.6 and 0.7. >>> >>> If you want to roll your own solution I would consider: >>> - write an app to talk to 0.6 and pull out the data using keys from the >>> other system (so you know can check referential integrity while you are at >>> it). Dump the data to flat file. >>> - write an app to talk to 0.7 to load the data back in. >>> >>> I've not given up digging on your migration problem, having to manually >>> dump and reload if you've done nothing wrong is not the best solution. I'll >>> try to find some time this weekend to test with: >>> >>> - 0.6 server, random paritioner, standard CF's, byte column >>> - load with python or the cli on osx or ubuntu (dont have a window machine >>> any more) >>> - migrate and see whats going on. >>> >>> If you can spare some sample data to load please send it over in the user >>> group or my email address. >>> >>> Cheers >>> >>> - >>> Aaron Morton >>> Freelance Cassandra Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 6 May 2011, at 05:52, Henrik Schröder wrote: >>> We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have rows stored that have unicode keys, and Cassandra 0.7.5 thinks those rows in the sstables are corrupt, and it seems impossible to clean it up without losing data. However, we can still read all rows perfectly via thrift so we are now looking at building a simple tool that will copy all rows from our 0.6.3 cluster to a parallell 0.7.5 cluster. Our question is now how to do that and ensure that we actually get all rows migrated? It's a pretty small cluster, 3 machines, a single keyspace, a singke columnfamily, ~2 million rows, a few GB of data, and a replication factor of 3. So what's the best way? Call get_range_slices and move through the entire token space? We also have all row keys in a secondary system, would it be better to use that and make calls to get_multi or get_multi_slices instead? Are we correct in assuming that if we use the consistencylevel ALL we'll get all rows? /Henrik Schröder >>> >>> >> >> >
Re: GC for ParNew (cassandra under performance ) 0.6.3
Hi That sloved my issue of GC,Now i am facing with new one i have no swap but this process kswapd0 take up 100% cpu when there is load on cassandra any idea why this is happening On 05/05/2011 09:06 PM, Ali Ahsan wrote: Thanks replying,let me disable my swap memory. On 05/05/2011 09:01 PM, Jonathan Ellis wrote: 6s parnew is insane. you're probably swapping. Easiest fix is disabling swap entirely. P.S. 0.6.3 is ancient. -- S.Ali Ahsan Senior System Engineer e-Business (Pvt) Ltd 49-C Jail Road, Lahore, P.O. Box 676 Lahore 54000, Pakistan Tel: +92 (0)42 3758 7140 Ext. 128 Mobile: +92 (0)345 831 8769 Fax: +92 (0)42 3758 0027 Email: ali.ah...@panasiangroup.com www.ebusiness-pg.com www.panasiangroup.com Confidentiality: This e-mail and any attachments may be confidential and/or privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person use it for any purpose or store or copy the information in any medium. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. We do not accept liability for any errors or omissions.
Re: Running Cassandra across different Amazon EC2 regions
vpn on ubuntu should be easy if you ask your good friend google... you should not have to pay for it (but paying might get you a fancy GUI, or perhaps very optimized performance that could squeeze a few more %) - Stephen --- Sent from my Android phone, so random spelling mistakes, random nonsense words and other nonsense are a direct result of using swype to type on the screen On 6 May 2011 23:25, "Sameer Farooqui" wrote:
Re: Ant error in Eclipse when building Cassandra
if you can give me (an intellij user) enough details to reproduce on my MBP I'll try and fix it. things like, download this eclipse distro, add these update centers, set these env variables, then clicks through this horrible UI as follows... presto crash! - Stephen --- Sent from my Android phone, so random spelling mistakes, random nonsense words and other nonsense are a direct result of using swype to type on the screen On 7 May 2011 06:50, "Jonathan Ellis" wrote: > Default stack is huge, so maven-ant-tasks-retrieve-build is probably > recursing infinitely somewhere :( > > On Fri, May 6, 2011 at 2:42 PM, Ed Anuff wrote: >> I finally got around to getting Eclipse set up to build Cassandra >> following the directions on the wiki and it seems to be working, >> Eclipse isn't showing any errors except that when it fires off the >> automatic ant build I get the following error: >> >> maven-ant-tasks-retrieve-build: >> >> BUILD FAILED >> java.lang.StackOverflowError >>at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1346) >>at org.apache.tools.ant.Project.executeTarget(Project.java:1306) >>at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) >>at org.eclipse.ant.internal.core.ant.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:32) >>at org.apache.tools.ant.Project.executeTargets(Project.java:1189) >>at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:662) >>at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:495) >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>at java.lang.reflect.Method.invoke(Method.java:597) >>at org.eclipse.ant.core.AntRunner.run(AntRunner.java:378) >>at org.eclipse.ant.internal.launching.launchConfigurations.AntLaunchDelegate.runInSameVM(AntLaunchDelegate.java:321) >>at org.eclipse.ant.internal.launching.launchConfigurations.AntLaunchDelegate.launch(AntLaunchDelegate.java:274) >>at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:853) >>at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:702) >>at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:695) >>at org.eclipse.core.externaltools.internal.model.ExternalToolBuilder.launchBuild(ExternalToolBuilder.java:181) >>at org.eclipse.core.externaltools.internal.model.ExternalToolBuilder.doBuildBasedOnScope(ExternalToolBuilder.java:169) >>at org.eclipse.core.externaltools.internal.model.ExternalToolBuilder.build(ExternalToolBuilder.java:88) >>at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:629) >>at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) >>at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:172) >>at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:203) >>at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:255) >>at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) >>at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:258) >>at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:311) >>at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:343) >>at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:144) >>at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:242) >>at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) >> Caused by: java.lang.StackOverflowError >>at org.apache.tools.ant.PropertyHelper.getPropertyHook(PropertyHelper.java:189) >>at org.apache.maven.artifact.ant.POMPropertyHelper.getPropertyHook(POMPropertyHelper.java:50) >> >> >> I never get this error when building from the command line and if I >> right click on build.xml in Eclipse and select Run As Ant Build it >> works fine as well. Any ideas? This is on a Mac. >> >> Ed >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com
