On Tue, Apr 12, 2011 at 12:26 AM, aaron morton aa...@thelastpickle.comwrote:
The reset interval clears the latency tracked for each node so a bad node
will be read from again. The scores for each node are then updated every
100ms (default) using the last 100 responses from a node.
How long
2011/4/12 aaron morton aa...@thelastpickle.com
In JConsole go to o.a.c.db.HintedHandoffManager and try the
deleteHintsForEndpopints operation.
This is also called as when a token is removed from the ring, or when a
node is decomissioned.
What process did you use to reconfigure the cluster?
Hello Mark,
Disable verbose mode (-w or $VERBOSE) of ruby.
Or, you can cleanup ruby thrift library by yourself.
2011/4/12 Mark Lilback mlilb...@stat.wvu.edu:
I'm trying to connect to Cassandra from a Ruby script. I'm using rvm, and
made a clean install of Ruby 1.9.2 and then did gem install
I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
[root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p 8090
ring
Address Status State LoadOwnsToken
109028275973926493413574716008500203721
192.168.1.25Up Normal 157.25 MB
This is normal when you just add single nodes. When no token is assigned,
the new node takes a portion of the ring from the most heavily loaded node.
As a consequence of this, the nodes will be out of balance.
In other words, when you double the amount nodes you would not have this
The 3 nodes were added to the cluster at the same time, so I'm not sure whey
the data vary.
I calculate the tokens and get:
node 0: 0
node 1: 56713727820156410577229101238628035242
node 2: 113427455640312821154458202477256070485
So I should set these tokens to the three nodes?
And during the
After the nodetool move, I got this:
[root@server3 apache-cassandra-0.7.4]# bin/nodetool -h 10.18.101.213 ring
Address Status State LoadOwnsToken
113427455640312821154458202477256070485
10.18.101.211 ? Normal 82.31 MB33.33% 0
10.18.101.212 ?
On Apr 12, 2011 5:01 AM, Dikang Gu dikan...@gmail.com wrote:
I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
[root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p
8090
ring
Address Status State Load Owns Token
109028275973926493413574716008500203721
Couple of hits here, one from jonathan and some previous discussions on the
user list http://www.google.co.nz/search?q=cassandra+iostat
Same here for cfhistograms
http://www.google.co.nz/search?q=cassandra+cfhistograms
cfhistograms includes information on the number of sstables read during
when you do a move, the node is decommissioned and bootstrapped. During the
autobootstrap process the node will not receive reads until bootstrapping is
complete. I assume during the decommission phase the node will also be
unavailable, someone correct me if I'm wrong.
the ring distribution
Hi All
I have migrated my server to centos 5.5.Every thing is up but facing a
little issue i have two cassandra nodes.
10.0.0.4 cassandra2
10.0.0.3 cassandra1
I am using open jdk with cassandra,We are faing following error when
using nodetool.Only on one server that is cassandra2.Hosts
There are a few other threads related to problems with the nodetool repair in
0.7.4. However I'm not seeing any errors, just never getting a message that
the repair completed successfully.
In my production and test cluster (with just a few MB data) the repair
nodetool prompt never returns
Something feels odd.
From Peters nice write up of the dynamic snitch
http://www.mail-archive.com/user@cassandra.apache.org/msg12092.html The
RackInferringSnitch (and the PropertyFileSnitch) derive from the
AbstractNetworkTopologySnitch and should...
In the case of the
On 12/04/2011 13:31, Jonathan Colby wrote:
There are a few other threads related to problems with the nodetool repair in
0.7.4. However I'm not seeing any errors, just never getting a message that
the repair completed successfully.
In my production and test cluster (with just a few MB data)
Bug in the CLI, created / fixed
https://issues.apache.org/jira/browse/CASSANDRA-2458
use 70 for now.
Thanks
Aaron
On 12 Apr 2011, at 20:46, Héctor Izquierdo Seliva wrote:
Hi everyone.
I've changed the read repair chance of one of my column families from
cassandra-cli with the following
If you are seeing a different views of the ring from different nodes you may
have some sickness
http://www.datastax.com/docs/0.7/troubleshooting/index#view-of-ring-differs-between-some-nodes
The ? in the ring output happens when one node does not know if the other is
alice or dead. This could
There is no Repair session message either. It just starts with a message
like:
INFO [manual-repair-2af33a51-f46a-4ba2-b1fb-ead5159dc723] 2011-04-10
14:00:59,051 AntiEntropyService.java (line 770) Waiting for repair requests:
[#TreeRequest manual-repair-2af33a51-f46a-4ba2-b1fb-ead5159dc723,
Thanks Aaron!
El mar, 12-04-2011 a las 23:52 +1200, aaron morton escribió:
Bug in the CLI, created /
fixed https://issues.apache.org/jira/browse/CASSANDRA-2458
use 70 for now.
Thanks
Aaron
On 12 Apr 2011, at 20:46, Héctor Izquierdo Seliva wrote:
Hi everyone.
I've
Hi everyone.
Looking for ways to monitor cassandra with zabbix I could not found
anything that was really usable, till I found mention of a nice class by
smeet. I have based my modification upon his work and now I give it back
to the community.
Here's the project url:
does a repair just compare the existing data from sstables on the node being
repaired, or will it figure out which data this node should have and copy it
in?
I'm trying to refresh all the data for a given node (without reassigning the
token) starting with an emptied out data directory.
I
I think I answered the question myself. The data is streaming in from other
replicas even though the node's data dir was emptied out (system dir was left
alone).
I'm not sure if this is the kosher way to rebuild the sstable data, but it
seemed to work.
/var/lib/cassandra/data #
Hi experts,
We are planning to deploy Cassandra in 2 datacenters. Let assume there
are 3 nodes, RF=3, 2 nodes in 1 DC and 1 node in 2nd DC. Under normal
operations, we would read and write at QUORUM. What we want to do though is
if we lose a datacenter which has 2 nodes, DC1 in this case, we
When the down data center comes back up, the Quorum reads will result in a
read-repair, so you will get valid data. Besides that, hinted handoff will
take care of getting data replicated to a previously down node.
You're example is a little unrealistic because you could theoretically have a
Please any one can
On 04/12/2011 04:07 PM, Ali Ahsan wrote:
Hi All
I have migrated my server to centos 5.5.Every thing is up but facing a
little issue i have two cassandra nodes.
10.0.0.4 cassandra2
10.0.0.3 cassandra1
I am using open jdk with cassandra,We are faing following error when
Thanks for sharing this info,I am getting following error,Can please be
more specific how can i run this
java -cp
/home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar
127.0.0.1 ks1:cf1,ks1:cf2
Exception in thread main java.lang.NoClassDefFoundError: 127/0/0/1
Caused by:
Interesting issue this morning.
My apps started throwing a bunch of pycassa timeouts all of a sudden.
The ring looked perfect. No load issues anywhere, and no errors in the
logs.
The site was basically down, so I got desperate and whacked a random
node in the ring. As soon as gossip saw it go
Here is what cfhistograms look like. Don't really understand what this means,
will try to read. I also %util in iostat continuously 90%. Not sure if this
is caused by extra reads by cassandra. It seems unusual.
[root@dsdb4 ~]# nodetool -h `hostname` cfhistograms StressKeyspace
StressStandard
[I wrote this Apr 10, 2011 at 12:09 but my message seems to have gotten lost
along the way.]
I use Pelops (the 1.0-0.7.x build from the Github Maven repo) and have
occasionally seen this message (under load or during GC). I have a test app
running in two separate single-threaded processes doing a
El mar, 12-04-2011 a las 21:24 +0500, Ali Ahsan escribió:
Thanks for sharing this info,I am getting following error,Can please be
more specific how can i run this
java -cp
/home/ali/apache-cassandra-0.6.3/lib/simple-cassandra-monitoring-1.0.jar
127.0.0.1 ks1:cf1,ks1:cf2
Exception in
hi, just deployed a new keyspace on 0.7.4 and added the following column family:
create column family applications with comparator=UTF8Type and column_metadata=[
{column_name: app_name, validation_class: UTF8Type},
{column_name: app_uri, validation_class: UTF8Type,index_type: KEYS},
On 04/12/2011 10:42 PM, Héctor Izquierdo Seliva wrote:
I forgot to mention it has been coded against 0.7.x, and I'm not sure it
will work on 0.6.x. I'll try to add support for both 0.6.x and the new
0.8.x version as soon as possible.
I think these error is because of 0.6.3 ?
xception in
Hi All
I have two cassandra node's,If Boot strapped nodes goes down my service
remains alive,But if my non Bootstrap (master) node goes down my live
site goes down as well,I am using cassandra 0.6.3 can any elaborate on
this problem.
I'm not sure. Are you runing it in the same host as the cassandra node?
El mar, 12-04-2011 a las 22:54 +0500, Ali Ahsan escribió:
On 04/12/2011 10:42 PM, Héctor Izquierdo Seliva wrote:
I forgot to mention it has been coded against 0.7.x, and I'm not sure it
will work on 0.6.x. I'll try to
Hi,
I'm getting closer to commiting to cassandra, and now I'm in system/IT
issues and questions. I'm in the amazon EC2 cloud. I previously used this
forum to discover the best practice for disk layouts (large instance + the
two ephemeral disks in RAID0 for data + root volume for everything
Yes same host,I will test this with my developer team and let you know
more on it.
On 04/12/2011 11:14 PM, Héctor Izquierdo Seliva wrote:
I'm not sure. Are you runing it in the same host as the cassandra node?
--
S.Ali Ahsan
Senior System Engineer
e-Business (Pvt) Ltd
49-C Jail Road,
Can someone please help?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Lot-of-pending-tasks-for-writes-tp6263462p6266213.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
I am using cassandra 7.4 and getting these messages.
Heap is 0.7802529021498031 full. You may need to reduce memtable and/or
cache sizes Cassandra will now flush up to the two largest memtables to free
up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if
you don't want
NTS is overkill in the sense that it doesn't really benefit you in a
single DC, but if you think you may expand to another DC in the future
it's much simpler if you were already using NTS, than first migrating
to NTS (changing strategy is painful).
I can't think of any downsides to using NTS in a
Excellent to know! (and yes, I figure I'll expand someday, so I'm glad I
found this out before digging a hole).
The other issue I've been pondering is a normal column family of encoded
objects (in my case JSON) vs. a super column. Based on my use case, things
I've read, etc... right now I'm
http://wiki.apache.org/cassandra/FAQ#unsubscribe
http://wiki.apache.org/cassandra/FAQ#unsubscribeIs this what you're
looking for?
Joaquin Casares
DataStax
Software Engineer/Support
On Tue, Apr 12, 2011 at 2:03 PM, Denis Kirpichenkov
den.doki.kirpichen...@gmail.com wrote:
how long as it been in Leaving status? Is the cluster under stress test load
while you are doing the decommission?
On Apr 12, 2011, at 6:53 PM, Baskar Duraikannu wrote:
I have setup a 4 node cluster for testing. When I setup the cluster, I have
setup initial tokens in such a way that each
your jvm heap has reached 78% so cassandra automatically flushes its memtables.
you need to explain more about your configuration. 32 or 64 bit OS, what is
max heap, how much ram installed?
If this happens under stress test conditions its probably understandable. you
should look into
Hi,
I am trying to run json2sstable with the following command but am receiving the
below error.
json2sstable -K testks -c testcf output.json
/var/lib/cassandra/data/testks/testcf-f-1-Data.db
Importing 321 keys...
java.lang.NullPointerException
at
No. I stopped the stress test before issuing decommission command. So, it was
not under ANY load.
I waited for over an hour and nothing changed.
Then , I turned on DEBUG in the log4j-server.properties and then restarted the
Cassandra process .
As soon as I restarted, the decommissioned node
64 bit 12 core 96 GB RAM
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266400.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
I think this is reasonable assuming you have enough backhaul to perform
reads across DC if read requests hit DC2 (with one copy of data) or one
replica from DC1 is down.
Moreover, since you clearly stated that you would prefer availability over
consistency, you should be prepared for stale reads
Hi,
What operations will be executed (and what is the associated overhead)
when the Keyspace replication factor is changed online, in a
multi-datacenter setup with NetworkTopologyStrategy?
I checked the wiki and the archive of the mailing list and find this,
but it is not very complete.
Hi All,
I am getting the following errors when I am trying to start
cassandra .
Error occurred during initialization of VM
Could not reserve enough space for object heap
I am using cassandra 0.7.3
uname -a
Linux hostname 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010
x86_64
I have two cassandra node's,If Boot strapped nodes goes down my service
remains alive,But if my non Bootstrap (master) node goes down my live site
goes down as well,I am using cassandra 0.6.3 can any elaborate on this
problem.
Assuming your RF is 2 (not 1), and that you are reading at
I was able to resolve this by changing the heap size
Thanks
Anurag
On Tue, Apr 12, 2011 at 1:38 PM, Anurag Gujral anurag.guj...@gmail.comwrote:
Hi All,
I am getting the following errors when I am trying to start
cassandra .
Error occurred during initialization of VM
Could not
I am just running simple test in 6 node cassandra 4 GB heap, 96 GB RAM and
12 core per host. I am inserting 1M rows with avg col size of 250k. I keep
getting Dropped mutation messages in logs. Not sure how to troubleshoot or
tune it.
Average col size of 250k - that sounds to me like you're
Heap is 0.7802529021498031 full. You may need to reduce memtable and/or
cache sizes Cassandra will now flush up to the two largest memtables to free
up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if
you don't want Cassandra to do this automatically
How do I verify
When the down data center comes back up, the Quorum reads will result in a
read-repair, so you will get valid data. Besides that, hinted handoff will
take care of getting data replicated to a previously down node.
*Eventually* though, but yes. I.e., there would be no expectation to
I was able to resolve this by changing the heap size
And that is the preferred solution. While adjusting stuff like the
kernel overcommit settings might allow the JVM to start, there is no
reason ever to have a heap size larger than what physical memory on
the server can actually sustain. So
Yes
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266726.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
It does appear that I am IO bound. Disks show about 90% util.
Well, also pay attention to the average queue size column. If there
are constantly more requests waiting to be serviced than you have
platters, you're almost certainly I/O bound. The utilization number
can be a bit flaky sometimes,
Yes
Without checking I don't know the details of the memtable threshold
calculations enough to be sure whether large columns are somehow
causing the size estimations to be ineffective (off hand I would
expect the reverse since the overhead of the Java object structures
become much less
I'm running into the same issue with 0.7.4. You don't need to specify
lexicaluuid, seems any valid key type will work- it just needs to fit
with your data (ascii, bytes, etc).
On Sun, Apr 10, 2011 at 7:13 PM, Patrick Julien pjul...@gmail.com wrote:
put in an assumption first, so from
Yes for interactive == real time queries. Hadoop based techniques are non
time critical queries, but they do have greater analytical capabilities.
particle_pairs:
1) Yes and no and sort of. Under the hood the get_slice api call will be used
by your client library to pull back chunks of
To now answer my own question, the critical points that are different
from what I said earlier are: that CL.ONE does prefer *one* node (which
one depending on snitch) and that RR uses digests (which are not
mentioned on the wiki page [1]) instead of comparing raw requests.
I updated it to
On 04/12/2011 06:27 PM, Peter Schuller wrote:
So to increase pinny-ness I'll further reduce RR chance and set a
badness threshold. Thanks all.
Just be aware that, assuming I am not missing something, while this
will indeed give you better cache locality under normal circumstances
- once
On 04/12/2011 11:11 AM, Jonathan Colby wrote:
I'm not sure if this is the kosher way to rebuild the sstable data, but it
seemed to work.
http://wiki.apache.org/cassandra/Operations#Handling_failure
Option #3.
One thing I am noticing is that cache hit rate is very low even though my
cache key size is 1M and I have less than 1M rows. Not sure why so many
cache miss?
Keyspace: StressKeyspace
Read Count: 162506
Read Latency: 45.22479006928975 ms.
Write Count: 247180
Write
Is there anybody else that might see a problem with just using delete
mutations instead of remove calls?
I'm thinking about changing a Cassandra client to always use delete
mutations when removing objects, that way the delete/remove call
interface can be kept the same:
1- the delete/remove client
Hello,
I've been running a single node cluster (0.7.4 built from the SVN
tag, running on JDK 1.6.0_21 on Ubuntu 10.10) for testing purposes.
After running fine for a couple of weeks, I got the error below on
startup. It sounded like the error which is supposed to be fixed by
the nodetool scrub
Ah, unreadable rows and in the validation compaction no less. Makes a little
more sense now.
Anyone help with the EOF when deserializing columns ? Is the fix to run scrub
or drop the sstable ?
Here's a a theory, AES is trying to...
1) Create TreeRequest 's that specify a range we want to
Can you connect from the local machine using 127.0.0.1 ?
Are you running any sort of fire wall?
Check you can connect from the node to the JMX port (8080 by default) using
telnet
Aaron
On 13 Apr 2011, at 04:25, Ali Ahsan wrote:
Any one can guide me on this issue ?
On 04/12/2011 04:07
Built indexes are there for me
[default@unknown] describe keyspace Keyspace1;
Keyspace: Keyspace1:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Replication Factor: 1
Column Families:
ColumnFamily: Indexed1
default_validation_class:
If you can use standard + encoded I would go with that.
Aaron
On 13 Apr 2011, at 07:07, William Oberman wrote:
Excellent to know! (and yes, I figure I'll expand someday, so I'm glad I
found this out before digging a hole).
The other issue I've been pondering is a normal column family of
Reading the code looks like it could not find a subColumns item for the row in
the json file.
The target CF is a super CF, is the data from a super CF ?
Aaron
On 13 Apr 2011, at 07:24, Steven Teo wrote:
Hi,
I am trying to run json2sstable with the following command but am receiving
Are you changing the replication factor or moving nodes ?
To change the RF you need to repair and then once all repairing is done run
cleanup to remove the hold data.
You can move whole nodes by moving all their data with them, assigning a new
ip, and updating the topology file if used.
One thing I am noticing is that cache hit rate is very low even though my
cache key size is 1M and I have less than 1M rows. Not sure why so many
cache miss?
The key cache should be strictly LRU for read-only workloads. For
write/read workloads it may not be strictly LRU because compaction
Aaron,
Thank you so much for your help. It is greatly appreciated!
Looking at the design of the particle pairs:
- key: expriement_id.time_interval
- column name: pair_id
- column value: distance, angle, other data packed together as JSON or
some other format
You wrote that retrieving
Is there a limit to the size that can be stored in one 'cell' (by 'cell' I
mean the intersection between a *key* and a *data column*)? is there a
limit to the size of data of one *key*? one *data column*?
http://wiki.apache.org/cassandra/CassandraLimitations
This is a problem reading the commitlog, which is not something scrub
can help with.
Looks like there is bad data in
/home/paul/apps/cassandra/node1/commitlog/CommitLog-1302567818267.log.
Somehow it's corrupt in a way that the checksum is ok. (Which sounds
like
Steven,
Thank you.
You wrote: The data of cassandra are partitioned by the row key;
therefore, if you want to put all pairs into the same row, you should
consider the disk size
Can you please explain why the disk size is / might be a problem?
Thanks,
Shalom.
-Original Message-
From:
the data is a custom json, seems like i may have got the structure wrong.
how should the import json be like?
Steven Teo
On 13-Apr-2011, at 10:43 AM, aaron morton wrote:
Reading the code looks like it could not find a subColumns item for the row
in the json file.
The target CF is a
Does it really matter how long cassandra has been running? I thought it will
keep keys of 1M at least.
Regarding your previous question about queue size in iostat I see it ranging
from 114-300.
--
View this message in context:
Does it really matter how long cassandra has been running? I thought it will
keep keys of 1M at least.
It will keep up to the limit, and it will save caches periodically and
reload them on start. But the cache needs to be populated by traffic
first. If you wrote a bunch of data, enabled the row
Hi,
I am getting the following exception while starting Cassandra trunk build,
am I missing any configuration options, please help ?
Thanks,
Shariq.
Stack track
~/work/cassandra-trunk$ ./bin/cassandra -f
INFO 11:04:07,864 Logging initialized
INFO 11:04:07,877 Heap size:
cool! and I thought I made that one up myself : )
On Apr 13, 2011, at 2:13 AM, Chris Burroughs wrote:
On 04/12/2011 11:11 AM, Jonathan Colby wrote:
I'm not sure if this is the kosher way to rebuild the sstable data, but it
seemed to work.
great tips. I will investigate further with your suggestions in mind.
Hopefully the problem has gone away since I pulled in fresh data on the node
with problems.
On Apr 13, 2011, at 3:54 AM, aaron morton wrote:
Ah, unreadable rows and in the validation compaction no less. Makes a little
84 matches
Mail list logo
