Re: Unable to add columns to empty row in Column family: Cassandra

[chovatia jaydeep]

One small correction in my mail below. 
Second insertion time-stamp has to be greater than delete time-stamp in-order 
to retrieve the data.

Thank you,
Jaydeep


From: chovatia jaydeep 
To: "user@cassandra.apache.org" 
Sent: Monday, 2 May 2011 11:52 PM
Subject: Re: Unable to add columns to empty row in Column family: Cassandra


Hi Anuya,

> However, columns are not being inserted.


Do you mean to say that after insert operation you couldn't retrieve the same 
data? If so, then please check the time-stamp when you reinserted after delete 
operation. Your second insertion time-stamp has to be greater than the previous 
insertion.

Thank you,
Jaydeep


From: anuya joshi 
To: user@cassandra.apache.org
Sent: Monday, 2 May 2011 11:34 PM
Subject: Re: Unable to add columns to empty row in Column family: Cassandra


Hello,

I am using Cassandra for my application.My Cassandra client uses Thrift APIs 
directly. The problem I am facing currently is as follows:

1) I added a row and columns in it dynamically via Thrift API Client
2) Next, I used command line client to delete row which actually deleted all 
the columns in it, leaving empty row with original row id.
3) Now, I am trying to add columns dynamically using client program into this 
empty row with same row key
    However, columns are not being inserted.
    But, when tried from command line client, it worked correctly.

Any pointer on this would be of great use

Thanks in  advance,

Regards,
Anuya

Re: Unable to add columns to empty row in Column family: Cassandra

[chovatia jaydeep]

Hi Anuya,

> However, columns are not being inserted.


Do you mean to say that after insert operation you couldn't retrieve the same 
data? If so, then please check the time-stamp when you reinserted after delete 
operation. Your second insertion time-stamp has to be greater than the previous 
insertion.

Thank you,
Jaydeep


From: anuya joshi 
To: user@cassandra.apache.org
Sent: Monday, 2 May 2011 11:34 PM
Subject: Re: Unable to add columns to empty row in Column family: Cassandra


Hello,

I am using Cassandra for my application.My Cassandra client uses Thrift APIs 
directly. The problem I am facing currently is as follows:

1) I added a row and columns in it dynamically via Thrift API Client
2) Next, I used command line client to delete row which actually deleted all 
the columns in it, leaving empty row with original row id.
3) Now, I am trying to add columns dynamically using client program into this 
empty row with same row key
    However, columns are not being inserted.
    But, when tried from command line client, it worked correctly.

Any pointer on this would be of great use

Thanks in  advance,

Regards,
Anuya

Re: Building from source from behind firewall since Maven switch?

[Stephen Connolly]

-autoproxy worked for me when I write the original patch

but as I no longer work for the company where I wrote the patch, I don't
have a firewall to deal with

worst case you might have to create a ~/.m2/settings.xml with the proxy
details... if that is the case can you raise a jira in MANTTASKS (which is
at jira.codehaus.org for hysterical reasons)

- Stephen

---
Sent from my Android phone, so random spelling mistakes, random nonsense
words and other nonsense are a direct result of using swype to type on the
screen
On 3 May 2011 01:06, "Suan-Aik Yeo"  wrote:

Unable to add columns to empty row in Column family: Cassandra

[anuya joshi]

Hello,

I am using Cassandra for my application.My Cassandra client uses Thrift APIs
directly. The problem I am facing currently is as follows:

1) I added a row and columns in it dynamically via Thrift API Client
2) Next, I used command line client to delete row which actually deleted all
the columns in it, leaving empty row with original row id.
3) Now, I am trying to add columns dynamically using client program into
this empty row with same row key
However, columns are not being inserted.
But, when tried from command line client, it worked correctly.

Any pointer on this would be of great use

Thanks in  advance,

Regards,
Anuya

Building from source from behind firewall since Maven switch?

[Suan-Aik Yeo]

Has anybody successfully built-from-source from behind a firewall since 
0.7.1 when they switched from using Ivy to Maven? We were able to build 
from source in 0.7.0 by inserting "proxy.host" and "proxy.port" property 
names in build.xml and exporting -Dhttp.proxyHost and -Dhttp.proxyPort 
variables in ANT_OPTS.


However, in 0.7.5 we've tried all sorts of options and xml tweaking and 
still can't pull in the needed dependencies. The build process starts 
hiccuping on the "maven-ant-tasks-retrieve-build" target and we see a 
whole slew of:
[artifact:dependencies] Downloading: 
com/thoughtworks/paranamer/paranamer-ant/2.1/paranamer-ant-2.1.pom from 
repository central at http://repo1.maven.org/maven2

[artifact:dependencies] Error transferring file: Network is unreachable
[artifact:dependencies] [WARNING] Unable to get resource 
'com.thoughtworks.paranamer:paranamer-ant:pom:2.1' from repository 
central (http://repo1.maven.org/maven2): Error transferring file: 
Network is unreachable
[artifact:dependencies] Downloading: 
com/thoughtworks/paranamer/paranamer-ant/2.1/paranamer-ant-2.1.pom from 
repository apache at 
https://repository.apache.org/content/repositories/releases

...

I can provide more info on the things we've tried upon request, but if 
somebody's already got it working then that would be best.



Thanks,
Suan

Cassandra Meetup in DC

[Chris Burroughs]

http://www.meetup.com/Cassandra-DC-Meetup/


*What*: First Cassandra DC Meetup

*When*: Thursday, May 12, 2011 at 6:30 PM

*Where*: Northside Social Coffee & Wine - 3211 Wilson Blvd Arlington, VA


I'm pleased to announce the the first Cassandra DC Meetup
. Come have
a drink, meet your fellow members, talk about Apache Cassandra, discuss
Greek mythological prophets, and what you want out of the group.

Re: Replica data distributing between racks

[Eric tamme]

On Mon, May 2, 2011 at 5:59 PM, aaron morton  wrote:
> My bad, I missed the way TokenMetadata.ringIterator() and firstTokenIndex() 
> work.
>
> Eric, can you show the output from nodetool ring ?
>
>

Sorry if the previous paste was way to unformatted, here is a
pastie.org link with nicer formatting of nodetool ring output than
plain text email allows.

http://pastie.org/private/50khpakpffjhsmgf66oetg

Re: Replica data distributing between racks

[Eric tamme]

On Mon, May 2, 2011 at 5:59 PM, aaron morton  wrote:
> My bad, I missed the way TokenMetadata.ringIterator() and firstTokenIndex() 
> work.
>
> Eric, can you show the output from nodetool ring ?
>

Here is output from nodtool ring - ip addresses changed obviously.

Address Status State   LoadOwnsToken

127605887595351923798765477786913079296
:0::111:0:0:0: Up Normal  195.28 GB   25.00%  0
:0::111:0:0:0:aaab Up Normal  47.12 GB25.00%
42535295865117307932921825928971026432
:0::112:0:0:0: Up Normal  189.96 GB   25.00%
85070591730234615865843651857942052864
:0::112:0:0:0:aaab Up Normal  42.82 GB25.00%
127605887595351923798765477786913079296

Re: Replica data distributing between racks

[aaron morton]

My bad, I missed the way TokenMetadata.ringIterator() and firstTokenIndex() 
work. 

Eric, can you show the output from nodetool ring ?


Aaron

On 3 May 2011, at 07:30, Eric tamme wrote:

> On Mon, May 2, 2011 at 3:22 PM, Jonathan Ellis  wrote:
>> On Mon, May 2, 2011 at 2:18 PM, aaron morton  wrote:
>>> When the NTS selects replicas in a DC it orders the tokens available in  
>>> the DC, then (in the first pass) iterates through them placing a replica in 
>>> each unique rack.  e.g. if the RF in each DC was 2, the replicas would be 
>>> put on 2 unique racks if possible. So the lowest token in the DC will 
>>> *always* get a write.
>> 
>> It's supposed to start w/ the node closest to the token in each DC, so
>> that shouldn't be the case unless you are using BOP/OPP instead of RP.
>> 
> 
> I am using a RandomPartitioner as shown below:
> 
> Cluster Information:
>   Snitch: org.apache.cassandra.locator.PropertyFileSnitch
>   Partitioner: org.apache.cassandra.dht.RandomPartitioner
> 
> So as far as "closeness" .. how does that get factored in when using a
> PropertyFileSnitch?  Is one rack closer than the other?  In reality
> for each data center there are two nodes in the same rack on the same
> switch,  but I set the topology file up to have 2 racks per data
> center specifically so I would get distribution.
> 
> -Eric

Re: Replica data distributing between racks

[Eric tamme]

On Mon, May 2, 2011 at 3:22 PM, Jonathan Ellis  wrote:
> On Mon, May 2, 2011 at 2:18 PM, aaron morton  wrote:
>> When the NTS selects replicas in a DC it orders the tokens available in  the 
>> DC, then (in the first pass) iterates through them placing a replica in each 
>> unique rack.  e.g. if the RF in each DC was 2, the replicas would be put on 
>> 2 unique racks if possible. So the lowest token in the DC will *always* get 
>> a write.
>
> It's supposed to start w/ the node closest to the token in each DC, so
> that shouldn't be the case unless you are using BOP/OPP instead of RP.
>

I am using a RandomPartitioner as shown below:

Cluster Information:
   Snitch: org.apache.cassandra.locator.PropertyFileSnitch
   Partitioner: org.apache.cassandra.dht.RandomPartitioner

So as far as "closeness" .. how does that get factored in when using a
PropertyFileSnitch?  Is one rack closer than the other?  In reality
for each data center there are two nodes in the same rack on the same
switch,  but I set the topology file up to have 2 racks per data
center specifically so I would get distribution.

-Eric

Re: Replica data distributing between racks

[Jonathan Ellis]

On Mon, May 2, 2011 at 2:18 PM, aaron morton  wrote:
> When the NTS selects replicas in a DC it orders the tokens available in  the 
> DC, then (in the first pass) iterates through them placing a replica in each 
> unique rack.  e.g. if the RF in each DC was 2, the replicas would be put on 2 
> unique racks if possible. So the lowest token in the DC will *always* get a 
> write.

It's supposed to start w/ the node closest to the token in each DC, so
that shouldn't be the case unless you are using BOP/OPP instead of RP.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Replica data distributing between racks

[aaron morton]

That appears to be working correctly, but does not sound great. 

When the NTS selects replicas in a DC it orders the tokens available in  the 
DC, then (in the first pass) iterates through them placing a replica in each 
unique rack.  e.g. if the RF in each DC was 2, the replicas would be put on 2 
unique racks if possible. So the lowest token in the DC will *always* get a 
write.

It's not possible to load balance between the racks as there is no state shared 
between requests. A possible alternative would be to find the nearest token to 
the key and start allocating replicas from there. But as each DC contains only 
a part (say half) of the token range the likelihood is that half of the keys 
would match to either end of the DC's range so it would not be a great 
solution. 

I think what you are trying to achieve is not possible. Do you have the 
capacity to run RF 2 in each DC ? That would at least even things out.

Aaron
 

On 3 May 2011, at 06:40, Eric tamme wrote:

> I am experiencing an issue where replication is not being distributed
> between racks when using PropertyFileSnitch in conjunction with
> NetworkTopologyStrategy.
> 
> I am running 0.7.3 from a tar.gz on  cassandra.apache.org
> 
> I have 4 nodes, 2 data centers, and 2 racks in each data center.  Each
> rack has 1 node.
> 
> I have even token distribution so that each node gets 25%:
> 
> 0
> 425352958651173079329218259289
> 71026432
> 85070591730234615865843651857942052864
> 127605887595351923798765477786913079296
> 
> My cassandra-topology.properties is as follows:
> 
> # Cassandra Node IP=Data Center:Rack
> \:0\:\:\:\:fffe=NY1:RAC1
> \:0\:\:\:\:=NY1:RAC2
> 
> \:0\:\:\:\:fffe=LA1:RAC1
> \:0\:\:\:\:=LA1:RAC2
> 
> # default for unknown nodes
> default=NY1:RAC1
> 
> 
> My Keyspace replication strategy is as follows:
> Keyspace: SipTrace:
>  Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
>Options: [LA1:1,NY1:1]
> 
> So each data center should get 1 copy of the data, and this does
> happen.  The problem is that the replicated copies get pinned to the
> first host configured in the properties file, from what I can discern,
> and DO NOT distribute between racks.  So I have 2 nodes that have a 4
> to 1 ratio of data compared to the other 2 nodes.  This is a problem!
> 
> Can any one please tell me if I have misconfigured this?  Or how I can
> get replica data to distribute evenly between racks within a
> datacenter?  I was led to believe that cassandra will try to
> distribute between racks for replica data automatically under this
> setup.
> 
> Thank you for your help in advance!
> 
> -Eric

Replica data distributing between racks

[Eric tamme]

I am experiencing an issue where replication is not being distributed
between racks when using PropertyFileSnitch in conjunction with
NetworkTopologyStrategy.

I am running 0.7.3 from a tar.gz on  cassandra.apache.org

I have 4 nodes, 2 data centers, and 2 racks in each data center.  Each
rack has 1 node.

I have even token distribution so that each node gets 25%:

0
425352958651173079329218259289
71026432
85070591730234615865843651857942052864
127605887595351923798765477786913079296

My cassandra-topology.properties is as follows:

# Cassandra Node IP=Data Center:Rack
\:0\:\:\:\:fffe=NY1:RAC1
\:0\:\:\:\:=NY1:RAC2

\:0\:\:\:\:fffe=LA1:RAC1
\:0\:\:\:\:=LA1:RAC2

# default for unknown nodes
default=NY1:RAC1


My Keyspace replication strategy is as follows:
Keyspace: SipTrace:
  Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Options: [LA1:1,NY1:1]

So each data center should get 1 copy of the data, and this does
happen.  The problem is that the replicated copies get pinned to the
first host configured in the properties file, from what I can discern,
and DO NOT distribute between racks.  So I have 2 nodes that have a 4
to 1 ratio of data compared to the other 2 nodes.  This is a problem!

Can any one please tell me if I have misconfigured this?  Or how I can
get replica data to distribute evenly between racks within a
datacenter?  I was led to believe that cassandra will try to
distribute between racks for replica data automatically under this
setup.

Thank you for your help in advance!

-Eric

Re: Experiences with Map&Reduce Stress Tests

[Jeremy Hanna]

Udo,

One thing to get out of the way - you're running task trackers on all of your 
cassandra nodes, right?  That is the first and foremost way to get good 
performance.  Otherwise you don't have data locality, which is really the point 
of map/reduce, co-locating your data and your processes operating over that 
data.  You're probably already doing that, but I had forgotten to ask that 
before.

Besides that...

You might try messing with those values a bit more as well as the input split 
size - cassandra.input.split.size which defaults to ~65k.  So you might try rpc 
timeout of 30s just to see if that helps and try reducing the input split size 
significantly to see if that helps.

For your setup I don't see the range batch size as being meaningful at all with 
your narrow rows, so don't worry about that.

Also, the capacity of your nodes and the number of mappers/reducers you're 
trying to use will also have an effect on whether it has to timeout.  
Essentially it's getting overwhelmed for some reason.  You might lower the 
number of mappers and reducers you're hitting your cassandra cluster with to 
see if that helps.

Jeremy

On May 2, 2011, at 6:25 AM, Subscriber wrote:

> Hi Jeremy, 
> 
> thanks for the link.
> I doubled the rpc_timeout (20 seconds) and reduced the range-batch-size to 
> 2048, but I still get timeouts...
> 
> Udo
> 
> Am 29.04.2011 um 18:53 schrieb Jeremy Hanna:
> 
>> It sounds like there might be some tuning you can do to your jobs - take a 
>> look at the wiki's HadoopSupport page, specifically the Troubleshooting 
>> section:
>> http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
>> 
>> On Apr 29, 2011, at 11:45 AM, Subscriber wrote:
>> 
>>> Hi all, 
>>> 
>>> We want to share our experiences we got during our Cassandra plus Hadoop 
>>> Map/Reduce evaluation.
>>> Our question was whether Cassandra is suitable for massive distributed data 
>>> writes using Hadoop's Map/Reduce feature.
>>> 
>>> Our setup is described in the attached file 'cassandra_stress_setup.txt'.
>>> 
>>> 
>>> 
>>> The stress test uses 800 map-tasks to generate data and store it into 
>>> cassandra.
>>> Each map task writes 500.000 items (i.e. rows) resulting in totally 
>>> 400.000.000 items. 
>>> There are max. 8 map tasks in parallel on each node. An item contains 
>>> (beside the key) two long and two double values, 
>>> so that items are a few 100 bytes in size. This leads to a total data size 
>>> of approximately 120GB.
>>> 
>>> The Map-Tasks uses the Hector API. Hector is "feeded" with all three data 
>>> nodes. The data is written in chunks of 1000 items.
>>> The ConsitencyLevel is set to ONE.
>>> 
>>> We ran the stress tests in several runs with different configuration 
>>> settings (for example I started with cassandra's default configuration and 
>>> I used Pelops for another test).
>>> 
>>> Our observations are like this:
>>> 
>>> 1) Cassandra is really fast - we are really impressed about the huge write 
>>> throughput. A map task writing 500.000 items (appr. 200MB) usually finishes 
>>> under 5 minutes.
>>> 2) However - unfortunately all tests failed in the end
>>> 
>>> In the beginning there are no problems. The first 100 (in some tests the 
>>> first 300(!)) map tasks are looking fine. But then the trouble starts.
>>> 
>>> Hadoop's sample output after ~15 minutes:
>>> 
>>> Kind% Complete  Num Tasks   Pending Running Complete
>>> Killed  Failed/Killed Task Attempts
>>> map 14.99%  800 680 24  96  0   
>>> 0 / 0
>>> reduce  3.99%   1   0   1   0   
>>> 0   0 / 0
>>> 
>>> Some stats:
 top
>>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND  
>>> 
>>> 
>>> 
>>> 31159   20   0 2569m 2.2g 9.8m S  450 18.6  61:44.73 java   
>>> 
>>> 
>>> 
>>> 
 vmstat 1 5
>>> procs ---memory-- ---swap-- -io -system-- 
>>> cpu
>>> r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id wa
>>> 2  1  36832 353688 242820 683752000157332  3  0 96  >>> 0
>>> 11  1  36832 350992 242856 685213600  1024 20900 4508 11738 19  1 
>>> 74  6
>>> 8  0  36832 339728 242876 685982800 0  1068 45809 107008 69 10 
>>> 20  0
>>> 1  0  36832 330212 242884 686852000 080 42112 92930 71  8 
>>> 21  0
>>> 2  0  36832 311888 242908 688770800  1024 0 20277 46669 46  7 
>>> 47  0
>>> 
 cassandra/bin/nodetool -h tirdata1 -p 28080 ring
>>> Address Status State   LoadOwns  

Re: Specifying exact nodes for Consistency Levels

[Robert Jackson]

The only way I can think of would be to use NetworkTopologyStrategy and 
PropertyFileSnitch to define two data centers. Set node1-node3 in DC1 then set 
node4 in DC2. Then set the strategy_options=[{DC1:3, DC2:1}] for that keyspace. 

This would make your cluster conceptually similar to a split datacenter as 
described in the Brisk white paper. 

References: 

http://www.datastax.com/docs/0.7/operations/datacenter 
http://www.datastax.com/wp-content/uploads/2011/03/WP-Brisk.pdf 

Robert Jackson 

- Original Message -
From: "A J"  
To: user@cassandra.apache.org 
Sent: Monday, May 2, 2011 1:26:31 PM 
Subject: Specifying exact nodes for Consistency Levels 

Is it possible in some way to specify what specific nodes I want to 
include (or exclude) from the Consistency Level fulfillment ? 
Example, I have a cluster of 4 nodes (n1,n2,n3 and n4) and set N=4. I 
want to set W=3 and want to ensure that it is n1,n2 and n3 only that 
are used to satisfy w=3 (i.e. n4 is excluded for w=3 calculation). 

Thanks. 

Re: Specifying exact nodes for Consistency Levels

[Jonathan Ellis]

No.

On Mon, May 2, 2011 at 12:26 PM, A J  wrote:
> Is it possible in some way to specify what specific nodes I want to
> include (or exclude) from the Consistency Level fulfillment ?
> Example, I have a cluster of 4 nodes (n1,n2,n3 and n4) and set N=4. I
> want to set W=3 and want to ensure that it is n1,n2 and n3 only that
> are used to satisfy w=3 (i.e. n4 is excluded for w=3 calculation).
>
> Thanks.
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Terrible CQL idea: > and < aliases of >= and <=

[Jonathan Ellis]

Created https://issues.apache.org/jira/browse/CASSANDRA-2592 to address this.

On Mon, May 2, 2011 at 10:46 AM, Eric Evans  wrote:
> On Mon, 2011-05-02 at 10:41 -0500, Jonathan Ellis wrote:
>> Where is that happening then?  RelationType.forString is not lossy,
>> and neither is the RelationType -> IndexExpression conversion in
>> getIndexedSlices.
>
> WhereClause.and(Relation) where it assigns start/end keys.
>
> --
> Eric Evans
> eev...@rackspace.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Specifying exact nodes for Consistency Levels

[A J]

Is it possible in some way to specify what specific nodes I want to
include (or exclude) from the Consistency Level fulfillment ?
Example, I have a cluster of 4 nodes (n1,n2,n3 and n4) and set N=4. I
want to set W=3 and want to ensure that it is n1,n2 and n3 only that
are used to satisfy w=3 (i.e. n4 is excluded for w=3 calculation).

Thanks.

Re: Combining all CFs into one big one

[Tyler Hobbs]

On Mon, May 2, 2011 at 12:06 PM, David Boxenhorn  wrote:

> I guess I'm still feeling fuzzy on this because my actual use-case isn't so
> black-and-white. I don't have any CFs that are accessed purely, or even
> mostly, in once-through batch mode. What I have is CFs with more and less
> data, and CFs that are accessed more and less frequently.
>

I figured that was the case; the black-and-white CFs just make easier
examples.  Back at the beginning of the thread, I recommend merging CFs that
have similar data "shapes" and highly correlated access patterns.  For
example, if you had two CFs, "user profile" and "group profile", that both
had fixed-length rows and tended to both be read for some type of
operations, those would be a good candidate for merging.  Just take the
ideas from the black-and-white examples and apply them in a more nuanced
way.

-- 
Tyler Hobbs
Software Engineer, DataStax 
Maintainer of the pycassa  Cassandra
Python client library

Re: Combining all CFs into one big one

[David Boxenhorn]

I guess I'm still feeling fuzzy on this because my actual use-case isn't so
black-and-white. I don't have any CFs that are accessed purely, or even
mostly, in once-through batch mode. What I have is CFs with more and less
data, and CFs that are accessed more and less frequently.


On Mon, May 2, 2011 at 7:52 PM, Tyler Hobbs  wrote:

> On Mon, May 2, 2011 at 5:05 AM, David Boxenhorn  wrote:
>
>> Wouldn't it be the case that the once-used rows in your batch process
>> would quickly be traded out of the cache, and replaced by frequently-used
>> rows?
>>
>
> Yes, and you'll pay a cache miss penalty for each of the replacements.
>
>
>> This would be the case even if your batch process goes on for a long time,
>> since caching is done on a row-by-row basis. In effect, it would mean that
>> part of your cache is taken up by the batch process, much as if you
>> dedicated a permanent cache to the batch - except that it isn't permanent,
>> so it's better!
>>
>
> Right, but we didn't want to cache any of the batch CF in the first place,
> because caching that CF is worth very little.  With separate CFs, we could
> explicitly give it no cache.  Now we have no control over how much of the
> cache it evicts.
>
>

Re: Combining all CFs into one big one

[Tyler Hobbs]

On Mon, May 2, 2011 at 5:05 AM, David Boxenhorn  wrote:

> Wouldn't it be the case that the once-used rows in your batch process would
> quickly be traded out of the cache, and replaced by frequently-used rows?
>

Yes, and you'll pay a cache miss penalty for each of the replacements.


> This would be the case even if your batch process goes on for a long time,
> since caching is done on a row-by-row basis. In effect, it would mean that
> part of your cache is taken up by the batch process, much as if you
> dedicated a permanent cache to the batch - except that it isn't permanent,
> so it's better!
>

Right, but we didn't want to cache any of the batch CF in the first place,
because caching that CF is worth very little.  With separate CFs, we could
explicitly give it no cache.  Now we have no control over how much of the
cache it evicts.

Re: Terrible CQL idea: > and < aliases of >= and <=

[Eric Evans]

On Mon, 2011-05-02 at 10:41 -0500, Jonathan Ellis wrote:
> Where is that happening then?  RelationType.forString is not lossy,
> and neither is the RelationType -> IndexExpression conversion in
> getIndexedSlices.

WhereClause.and(Relation) where it assigns start/end keys.

-- 
Eric Evans
eev...@rackspace.com

Re: Terrible CQL idea: > and < aliases of >= and <=

[Jonathan Ellis]

Where is that happening then?  RelationType.forString is not lossy,
and neither is the RelationType -> IndexExpression conversion in
getIndexedSlices.

On Mon, May 2, 2011 at 10:30 AM, Eric Evans  wrote:
> On Mon, 2011-05-02 at 09:29 -0500, Jonathan Ellis wrote:
>> 80% sure that is an obsolete comment.
>>
>> Eric, can you verify?
>
> No, it's true.
>
> --
> Eric Evans
> eev...@rackspace.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Terrible CQL idea: > and < aliases of >= and <=

[Eric Evans]

On Mon, 2011-05-02 at 09:29 -0500, Jonathan Ellis wrote:
> 80% sure that is an obsolete comment.
> 
> Eric, can you verify? 

No, it's true.

-- 
Eric Evans
eev...@rackspace.com

Re: Terrible CQL idea: > and < aliases of >= and <=

[Jonathan Ellis]

80% sure that is an obsolete comment.

Eric, can you verify?

On Mon, May 2, 2011 at 5:22 AM, David Boxenhorn  wrote:
> Is this still true?
>
> Note: The greater-than and less-than operators (> and <) result in key
> ranges that are inclusive of the terms. There is no supported notion of
> “strictly” greater-than or less-than; these operators are merely supported
> as aliases to >= and <=.
>
> I think that making > and < aliases of >= and <= is a terrible idea!
>
> First of all, it is very misleading. Second, what will happen to old code
> when > and < are really supported? (Some day they will be supported!)
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Strange corrupt sstable

[Jonathan Ellis]

Thanks for tracking that down!

0.7 OPP adds additional checks (and if you're starting from scratch
you should use BOP instead) that keys are valid UTF8, so it shouldn't
be an issue there.

On Mon, May 2, 2011 at 7:39 AM, Daniel Doubleday
 wrote:
> Just for the record:
>
> The problem had nothing to do with bad memory. After some more digging it 
> turned out that due to a bug we wrote invalid utf-8 sequences as row keys. In 
> 0.6 the key tokens are constructed from string decoded bytes. This does not 
> happen anymore in 0.7 files. So what apparently happened during compaction was
>
> 1. read sst and generate string based order rows
> 2. write the new file based on that order
> 3. read the compacted file based on raw bytes order -> crash
>
> That bug never made it to production so we are fine.
>
> On Apr 29, 2011, at 10:32 AM, Daniel Doubleday wrote:
>
>> Bad == Broken
>>
>> That means you cannot rely on 1 == 1. In such a scenario everything can 
>> happen including data loss.
>> That's why you want ECC mem on production servers. Our cheapo dev boxes dont.
>>
>> On Apr 28, 2011, at 7:46 PM, mcasandra wrote:
>>
>>> What do you mean by Bad memory? Is it less heap size, OOM issues or 
>>> something
>>> else? What happens in such scenario, is there a data loss?
>>>
>>> Sorry for many questions just trying to understand since data is critical
>>> afterall :)
>>>
>>> --
>>> View this message in context: 
>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Strange-corrupt-sstable-tp6314052p6314218.html
>>> Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
>>> Nabble.com.
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: CQL where clause and "OR"

[Jonathan Ellis]

OR will not be supported for a while yet, however IN support is in
trunk and will be in 0.8.1 (but not 0.8.0).

On Mon, May 2, 2011 at 5:10 AM, Miguel Auso  wrote:
> hi!,
> It`s posible
> for exemple:
> select * from aw_advancednewsletter_subscriptions where id='1' OR id='3';
>
>
> Un Saludo
> Miguel Ángel Ausó
>
>
> 2011/5/2 Sébastien Druon 
>>
>> Hi!
>> Is it possible to use an "OR" operator in the "WHERE" clause of a "SELECT"
>> statement? I do not find any documentation on that
>> i.e.
>> SELECT name1 FROM cf1 WHERE name1=value1 OR name1=value2
>> Thanks a lot in advance
>> Sébastien Druon
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Strange corrupt sstable

[Daniel Doubleday]

Just for the record:

The problem had nothing to do with bad memory. After some more digging it 
turned out that due to a bug we wrote invalid utf-8 sequences as row keys. In 
0.6 the key tokens are constructed from string decoded bytes. This does not 
happen anymore in 0.7 files. So what apparently happened during compaction was 

1. read sst and generate string based order rows
2. write the new file based on that order
3. read the compacted file based on raw bytes order -> crash

That bug never made it to production so we are fine.
 
On Apr 29, 2011, at 10:32 AM, Daniel Doubleday wrote:

> Bad == Broken
> 
> That means you cannot rely on 1 == 1. In such a scenario everything can 
> happen including data loss. 
> That's why you want ECC mem on production servers. Our cheapo dev boxes dont.
> 
> On Apr 28, 2011, at 7:46 PM, mcasandra wrote:
> 
>> What do you mean by Bad memory? Is it less heap size, OOM issues or something
>> else? What happens in such scenario, is there a data loss?
>> 
>> Sorry for many questions just trying to understand since data is critical
>> afterall :)
>> 
>> --
>> View this message in context: 
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Strange-corrupt-sstable-tp6314052p6314218.html
>> Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
>> Nabble.com.
> 

Re: Experiences with Map&Reduce Stress Tests

[Subscriber]

Hi Jeremy, 

thanks for the link.
I doubled the rpc_timeout (20 seconds) and reduced the range-batch-size to 
2048, but I still get timeouts...

Udo

Am 29.04.2011 um 18:53 schrieb Jeremy Hanna:

> It sounds like there might be some tuning you can do to your jobs - take a 
> look at the wiki's HadoopSupport page, specifically the Troubleshooting 
> section:
> http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
> 
> On Apr 29, 2011, at 11:45 AM, Subscriber wrote:
> 
>> Hi all, 
>> 
>> We want to share our experiences we got during our Cassandra plus Hadoop 
>> Map/Reduce evaluation.
>> Our question was whether Cassandra is suitable for massive distributed data 
>> writes using Hadoop's Map/Reduce feature.
>> 
>> Our setup is described in the attached file 'cassandra_stress_setup.txt'.
>> 
>> 
>> 
>> The stress test uses 800 map-tasks to generate data and store it into 
>> cassandra.
>> Each map task writes 500.000 items (i.e. rows) resulting in totally 
>> 400.000.000 items. 
>> There are max. 8 map tasks in parallel on each node. An item contains 
>> (beside the key) two long and two double values, 
>> so that items are a few 100 bytes in size. This leads to a total data size 
>> of approximately 120GB.
>> 
>> The Map-Tasks uses the Hector API. Hector is "feeded" with all three data 
>> nodes. The data is written in chunks of 1000 items.
>> The ConsitencyLevel is set to ONE.
>> 
>> We ran the stress tests in several runs with different configuration 
>> settings (for example I started with cassandra's default configuration and I 
>> used Pelops for another test).
>> 
>> Our observations are like this:
>> 
>> 1) Cassandra is really fast - we are really impressed about the huge write 
>> throughput. A map task writing 500.000 items (appr. 200MB) usually finishes 
>> under 5 minutes.
>> 2) However - unfortunately all tests failed in the end
>> 
>> In the beginning there are no problems. The first 100 (in some tests the 
>> first 300(!)) map tasks are looking fine. But then the trouble starts.
>> 
>> Hadoop's sample output after ~15 minutes:
>> 
>> Kind % Complete  Num Tasks   Pending Running CompleteKilled  
>> Failed/Killed Task Attempts
>> map  14.99%  800 680 24  96  0   
>> 0 / 0
>> reduce   3.99%   1   0   1   0   
>> 0   0 / 0
>> 
>> Some stats:
>>> top
>>  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND  
>>  
>>  
>>   
>> 31159   20   0 2569m 2.2g 9.8m S  450 18.6  61:44.73 java
>>  
>>  
>>  
>> 
>>> vmstat 1 5
>> procs ---memory-- ---swap-- -io -system-- cpu
>> r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id wa
>> 2  1  36832 353688 242820 683752000157332  3  0 96  0
>> 11  1  36832 350992 242856 685213600  1024 20900 4508 11738 19  1 74 
>>  6
>> 8  0  36832 339728 242876 685982800 0  1068 45809 107008 69 10 
>> 20  0
>> 1  0  36832 330212 242884 686852000 080 42112 92930 71  8 21 
>>  0
>> 2  0  36832 311888 242908 688770800  1024 0 20277 46669 46  7 47 
>>  0
>> 
>>> cassandra/bin/nodetool -h tirdata1 -p 28080 ring
>> Address Status State   LoadOwnsToken 
>>  
>>   
>> 113427455640312821154458202477256070484 
>> 192.168.11.198  Up Normal  6.72 GB 33.33%  0 
>>  
>> 192.168.11.199  Up Normal  6.72 GB 33.33%  
>> 56713727820156410577229101238628035242  
>> 192.168.11.202  Up Normal  6.68 GB 33.33%  
>> 113427455640312821154458202477256070484 
>> 
>> 
>> Hadoop's sample output after ~20 minutes:
>> 
>> Kind % Complete  Num Tasks   Pending Running CompleteKilled  
>> Failed/Killed Task Attempts
>> map  15.49%  800 673 24  103 0   
>> 6 / 0
>> reduce   4.16%   1   0   1   0   
>> 0   0 / 0
>> 
>> What went wrong? It's always the same. The clients cannot reach the nodes 
>> anymore. 
>> 
>> java.lang.RuntimeException: work failed
>>  at 
>> com.zfabrik.hadoop.impl.HadoopProcessRunner.work(HadoopProcessRunner.java:109)
>>  at 
>> com.zfabrik.hadoop.impl.DelegatingMapper.run(DelegatingMapper.java:40)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:625)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>  at org.

Terrible CQL idea: > and < aliases of >= and <=

[David Boxenhorn]

Is this still true?

*Note: The greater-than and less-than operators (> and <) result in key
ranges that are inclusive of the terms. There is no supported notion of
“strictly” greater-than or less-than; these operators are merely supported
as aliases to >= and <=.

*
I think that making > and < aliases of >= and <= is a terrible idea!

First of all, it is very misleading. Second, what will happen to old code
when > and < are really supported? (*Some* day they will be supported!)

Re: CQL where clause and "OR"

[Miguel Auso]

hi!,

It`s posible

for exemple:

select * from aw_advancednewsletter_subscriptions where id='1' OR id='3';



Un Saludo

Miguel Ángel Ausó



2011/5/2 Sébastien Druon 

> Hi!
>
> Is it possible to use an "OR" operator in the "WHERE" clause of a "SELECT"
> statement? I do not find any documentation on that
>
> i.e.
> SELECT name1 FROM cf1 WHERE name1=value1 OR name1=value2
>
> Thanks a lot in advance
>
> Sébastien Druon
>

Re: Combining all CFs into one big one

[David Boxenhorn]

Wouldn't it be the case that the once-used rows in your batch process would
quickly be traded out of the cache, and replaced by frequently-used rows?
This would be the case even if your batch process goes on for a long time,
since caching is done on a row-by-row basis. In effect, it would mean that
part of your cache is taken up by the batch process, much as if you
dedicated a permanent cache to the batch - except that it isn't permanent,
so it's better!


On Mon, May 2, 2011 at 7:50 AM, Tyler Hobbs  wrote:

> If you had one big cache, wouldn't it be the case that it's mostly
>> populated with frequently accessed rows, and less populated with rarely
>> accessed rows?
>>
>
> Yes.
>
> In fact, wouldn't one big cache dynamically and automatically give you
>> exactly what you want? If you try to partition the same amount of memory
>> manually, by guesswork, among many tables, aren't you always going to do a
>> worse job?
>>
>
> Suppose you have one CF that's used constantly through interaction by
> users.  Suppose you have another CF that's only used periodically by a batch
> process, you tend to access most or all of the rows during the batch
> process, and it's too large to cache all of the rows.  Normally, you would
> dedicate cache space to the first CF as anything with human interaction
> tends to have good temporal locality and you want to keep latencies there
> low.  On the other hand, caching the second CF provides little to no real
> benefit.  When you combine these two CFs, every time your batch process
> runs, rows from the second CF will populate the cache and will cause
> eviction of rows from the first CF, even though having those rows in the
> cache provides little benefit to you.
>
> As another example, if you mix a CF with wide rows and a CF with small
> rows, you no longer have the option of using a row cache, even if it makes
> great sense for the small-row CF data.
>
> Knowledge of data and access patterns gives you a very good advantage when
> it comes to caching your data effectively.
>
>
> --
> Tyler Hobbs
> Software Engineer, DataStax 
> Maintainer of the pycassa  Cassandra
> Python client library
>
>

CQL where clause and "OR"

[Sébastien Druon]

Hi!

Is it possible to use an "OR" operator in the "WHERE" clause of a "SELECT"
statement? I do not find any documentation on that

i.e.
SELECT name1 FROM cf1 WHERE name1=value1 OR name1=value2

Thanks a lot in advance

Sébastien Druon