Hi everyone. One of the nodes in my 6 node cluster died with disk
failures. I have replaced the disks, and it's clean. It has the same
configuration (same ip, same token).
When I try to restart the node it starts to throw mmap underflow
exceptions till it closes again.
I tried setting io to
what version are you running and what's the recent upgrade history ?
Cheers
Aaron
On 3 May 2011, at 23:09, Héctor Izquierdo Seliva wrote:
Hi everyone. One of the nodes in my 6 node cluster died with disk
failures. I have replaced the disks, and it's clean. It has the same
configuration
http://www.thelastpickle.com
On 4 May 2011, at 17:54, Héctor Izquierdo Seliva wrote:
Hi Aaron
It has no data files whatsoever. The upgrade path is 0.7.4 - 0.7.5. It
turns out the initial problem was the sw raid failing silently because
of another faulty disk.
Now
El mié, 04-05-2011 a las 21:02 +1200, aaron morton escribió:
Certainly sounds a bit sick.
The first error looks like it happens when the index file points to the
wrong place in the data file for the SSTable. The second one happens when
the index file is corrupted. The should be
Hi everyone.
I have a few sstables with around 500 million keys, and memory usage has
grown a lot, I suppose because of the indexes. This sstables are
comprised of skinny rows, but a lot of them. Would tuning index interval
make the memory usage go down? And what would the performance hit be?
I
El lun, 09-05-2011 a las 17:58 +0200, Peter Schuller escribió:
I have a few sstables with around 500 million keys, and memory usage has
grown a lot, I suppose because of the indexes. This sstables are
comprised of skinny rows, but a lot of them. Would tuning index interval
make the memory
El mié, 11-05-2011 a las 14:24 +1200, aaron morton escribió:
What version and what were the values for RecentBloomFilterFalsePositives and
BloomFilterFalsePositives ?
The bloom filter metrics are updated in SSTableReader.getPosition() the only
slightly odd thing I can see is that we do not
for RecentBloomFilterFalsePositives and
BloomFilterFalsePositives the non ratio ones ?
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 11 May 2011, at 19:53, Héctor Izquierdo Seliva wrote:
El mié, 11-05-2011 a las 14:24 +1200, aaron morton escribió:
What
Hi everyone. I see in the logs that Concurrent Mark Sweep is taking 12
seconds to do its stuff. Is this normal? There is no stop-the-world GC,
it just takes 12 seconds.
Configuration: 0.7.5 , 8GB Heap, 16GB machines. 7 * 64 MB memtables.
Hi,
Is there a way to control what sstables go to what data directory? I
have a fast but space limited ssd, and a way slower raid, and i'd like
to put latency sensitive data into the ssd and leave the other data in
the raid. Is this possible? If not, how well does cassandra play with
symlinks?
El mié, 08-06-2011 a las 08:42 -0500, Jonathan Ellis escribió:
No. https://issues.apache.org/jira/browse/CASSANDRA-2749 is open to
track this but nobody is working on it to my knowledge.
Cassandra is fine with symlinks at the data directory level but I
don't think that helps you, since you
Hi,
I have an index I use to translate ids. I usually only read a column at
a time, and it's becoming a bottleneck. I could rewrite the application
to read a bunch at a time but it would make the application logic much
harder, as it would involve buffering incoming data.
As far as I know, to
El jue, 09-06-2011 a las 13:28 +0200, Richard Low escribió:
Remember also that partitioning is done by rows, not columns. So
large rows are stored on a single host. This means they can't be load
balanced and also all requests to that row will hit one host. Having
separate rows will allow
I'm actually using it in a couple of nodes, but is slower than directly
accesing the data in a ssd.
El jue, 09-06-2011 a las 11:10 -0400, Chris Burroughs escribió:
On 06/08/2011 05:54 AM, Héctor Izquierdo Seliva wrote:
Is there a way to control what sstables go to what data directory? I
have
I think I will follow the advice of better balancing and I will split
the index into several pieces. Thanks everybody for your input!
Hi Terje,
There are 12 SSTables, so I don't think that's the problem. I will try
anyway and see what happens.
El vie, 10-06-2011 a las 20:21 +0900, Terje Marthinussen escribió:
bug in the 0.8.0 release version.
Cassandra splits the sstables depending on size and tries to find (by
El vie, 10-06-2011 a las 20:21 +0900, Terje Marthinussen escribió:
bug in the 0.8.0 release version.
Cassandra splits the sstables depending on size and tries to find (by
default) at least 4 files of similar size.
If it cannot find 4 files of similar size, it logs that message in
El vie, 10-06-2011 a las 23:40 +0900, Terje Marthinussen escribió:
Yes, which is perfectly fine for a short time if all you want is to
compact to one file for some reason.
I run min_compaction_threshold = 2 on one system here with SSD. No
problems with the more aggressive disk utilization
Hi All. I found a way to be able to compact. I have to call scrub on
the column family. Then scrub gets stuck forever. I restart the node,
and voila! I can compact again without any message about not having
enough space. This looks like a bug to me. What info would be needed to
fill a report?
I was already way over the minimum. There were 12 sstables. Also, is
there any reason why scrub got stuck? I did not see anything in the
logs. Via jmx I saw that the scrubbed bytes were equal to one of the
sstables size, and it stuck there for a couple hours .
El lun, 13-06-2011 a las 22:55
This is what I use:
http://code.google.com/p/simple-cassandra-monitoring/
Disclaimer: I did it myself, don't expect too much :P
El jue, 16-06-2011 a las 19:35 +0300, Viktor Jevdokimov escribió:
There's possibility to use command line JMX client with standard
Zabbix agent to request JMX
this is not normal behaviour.
El lun, 13-06-2011 a las 10:32 -0500, Jonathan Ellis escribió:
As Terje already said in this thread, the threshold is per bucket
(group of similarly sized sstables) not per CF.
2011/6/13 Héctor Izquierdo Seliva izquie...@strands.com:
I was already way over
http://www.thelastpickle.com
On 23 Jun 2011, at 02:04, Héctor Izquierdo Seliva wrote:
Hi All. I set the compaction threshold at minimum 2, maximum 2 and try
to run compact, but it's not doing anything. There are over 69 sstables
now, read performance is horrible, and it's taking
Hi all,
I have upgraded all my cluster to 0.8.1. Today one of the disks in one
of the nodes died. After replacing the disk I tried running repair, but
this message appears:
INFO [manual-repair-bdb4055a-d370-4d2a-a1dd-70a7e4fa60cf] 2011-06-30
20:36:25,085 AntiEntropyService.java (line 179)
Hi All, sorry for taking so long to answer. I was away from the
internet.
Héctor, when you say I have upgraded all my cluster to 0.8.1, from
which version was
that: 0.7.something or 0.8.0 ?
0.7.6-2 to 0.8.1
This is the same behavior I reported in 2768 as Aaron referenced ...
What was
(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
El mié, 06-07-2011 a las 11:22 +0200, Sylvain Lebresne escribió:
2011/7/6 Héctor Izquierdo Seliva izquie...@strands.com:
Hi, i've been struggling to repair my failed node for the past few days,
and I've seen
Hi all,
I don't seem to be able to complete a full repair on one of the nodes.
Memory consuptiom keeps growing till it starts complaining about not
having enough heap. I had to disable the automatic memtable flush, as it
was generating thousands of almost empty memtables.
My guess is that the
Forcing a full gc doesn't help either. Now the node is stuck in an
endless loop of full gcs that don't free any memory.
Hi everyone,
I'm having thousands of these errors:
WARN [CompactionExecutor:1] 2011-07-08 16:36:45,705
CompactionManager.java (line 737) Non-fatal error reading row
(stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size
6292724931198053
at
://www.thelastpickle.com
On 8 Jul 2011, at 09:38, Héctor Izquierdo Seliva wrote:
Hi everyone,
I'm having thousands of these errors:
WARN [CompactionExecutor:1] 2011-07-08 16:36:45,705
CompactionManager.java (line 737) Non-fatal error reading row
(stacktrace follows)
java.io.IOError
Hi Peter.
I have a problem with repair, and it's that it always brings the node
doing the repairs down. I've tried setting index_interval to 5000, and
it still dies with OutOfMemory errors, or even worse, it generates
thousands of tiny sstables before dying.
I've tried like 20 repairs during
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 9 Jul 2011, at 16:45, Héctor Izquierdo Seliva wrote:
Hi Peter.
I have a problem with repair, and it's that it always brings the node
doing the repairs down. I've tried setting index_interval to 5000
I'm also having problems with removetoken. Maybe I'm doing it wrong, but
I was under the impression that I just had to call once removetoken.
When I take a look at the nodes ring, the dead node keeps popping up.
What's even more incredible is that in some of them it says UP
At the end I had to restart the whole cluster. This is the second time
I've had to do this. Would it be possible to add a command that forces
all nodes to remove all the ring data and start it fresh? I'd rather
have a few seconds of errors in the clients that the two to five minutes
that takes a
I've been using flashcache for a while in production. It improves read
performance and latency was halved by a good chunk, though I don't
remember the exact numbers.
Problems: compactions will trash your cache, and so will memtable
flushes. Right now there's no way to avoid that.
If you want, I
Hector, some before/after numbers would be great if you can find them.
Thanks!
I'll try and get some for you :)
What happens when your cache gets trashed? Do compactions and flushes
go slower?
If you use flashcache-wt flushed and compacted sstables will go to the
cache.
All
Interesting. So, there is no segregation between read and write cache
space? A compaction or flush can evict blocks in the read cache if it
needs the space for write buffering?
There are two versions, the -wt (write through) that will cache also
what is written, and the normal version
If using the version that has both rt and wt caches, is it just the wt
cache that's polluted for compactions/flushes? If not, why does the rt
cache also get polluted?
As I said, all reads go through flashcache, so if you read three 10 GB
sstables for a compaction you will get those 30
Of course. I wasn't thinking clearly.
So, back to a previous point you brought up, I will have heavy reads and
even heavier writes. How would you rate the benefits of flashcache in
such a scenario? Is it still an overall performance boost worth the
expense?
We have also heavy reads
Hi all,
Maybe I'm doing something wrong, but calling ./nodetool -h host repair
mykeyspace mycolumnfamily should only repair mycolumnfamily right?
Everytime I try a repair it repairs the whole key space instead of just
one column family. I'm on cassandra 0.8.1
Are there any plans to backport this to 0.8?
El mar, 19-07-2011 a las 11:43 -0500, Jonathan Ellis escribió:
https://issues.apache.org/jira/browse/CASSANDRA-2280
2011/7/19 Héctor Izquierdo Seliva izquie...@strands.com:
Hi all,
Maybe I'm doing something wrong, but calling ./nodetool -h
El sáb, 20-08-2011 a las 01:22 +0200, Peter Schuller escribió:
Is there any chance that the entire file from source node got streamed to
destination node even though only small amount of data in hte file from
source node is supposed to be streamed destination node?
Yes, but the thing
Hi everyone.
I'm sure this question or similar has come up before, but I can't find a
clear answer. I have to store a unknown number of items in cassandra,
which can vary from a few hundreds to a few millions per customer.
I read that in cassandra wide rows are better than a lot of rows, but
El lun, 11-10-2010 a las 11:08 -0400, Edward Capriolo escribió:
Inlined:
2010/10/11 Héctor Izquierdo Seliva izquie...@strands.com:
Hi everyone.
I'm sure this question or similar has come up before, but I can't find a
clear answer. I have to store a unknown number of items in cassandra
That happened to me too. Try with a ; at the end of the line.
El jue, 25-11-2010 a las 17:22 +, Marcin escribió:
Hi guys,
I am having weird problem, cassandra is working but can't get
cassandra-cli to work.
When I run command - any command like even help and hit error I am not
Try ending the lines with ;
Regards
El vie, 26-11-2010 a las 21:25 +1100, jasonmp...@gmail.com escribió:
Hi,
So I had this working perfectly with beta 3 and now it fails.
Basically what I do is follows:
1) Extract new rc1 tarball.
2) Prepare location based on instructions in Readme.txt:
- well I certainly feel stupid! Is this new, it worked without
it on beta 3?
2010/11/26 Héctor Izquierdo Seliva izquie...@strands.com:
Try ending the lines with ;
Regards
El vie, 26-11-2010 a las 21:25 +1100, jasonmp...@gmail.com escribió:
Hi,
So I had this working perfectly
Hi everyone.
I have a question about data modeling in my application. I have to store
items of a customer, and I can do it in one fat row per customer where
the column name is the id and the value a json serialized object, or one
entry per item with the same layout. This data is updated almost
Hi!
Can some body give me some hints about how to configure a keyspace with
NetworkTopologyStrategy via cassandra-cli? Or what is the preferred
method to do so?
Thanks!
Thanks! I totally overlooked that.
El lun, 21-02-2011 a las 08:14 +1300, Aaron Morton escribió:
The best examples I know of are in the internal cli help, and
conf/casandra.yaml
Aaron
On 19/02/2011, at 12:51 AM, Héctor Izquierdo Seliva izquie...@strands.com
wrote:
Hi!
Can some
Hi all.
Is there a way (besides changing the code) to replicate data from a Data
center 1 to a Data center 2, but not the other way around? I need to
have a preproduction environment with production data, and ideally with
only a fraction of the data (for example, by key preffixes). I have
poked
Hi Everyone.
I'm testing performance differences of millions of columns in a row vs
millions of rows. So far it seems wide rows perform better in terms of
reads, but there can be potentially hundreds of millions of columns in a
row. Is this going to be a problem? Should I go with individual rows?
}] and the same keyspace in DC2 with options [{DC2:1,
DC1:0}]. Is that correct?
Aaron
On 22/02/2011, at 3:10 AM, Héctor Izquierdo Seliva izquie...@strands.com
wrote:
Hi all.
Is there a way (besides changing the code) to replicate data from a Data
center 1 to a Data center 2, but not the other
to the position of the columns within the row.
How much space does the key cache uses per row? This would make the
number of rows increase by a big factor.
On 22/02/2011, at 3:56 AM, Héctor Izquierdo Seliva izquie...@strands.com
wrote:
Hi Everyone.
I'm testing performance differences of millions
Hi everyone. I'm having trouble while inserting big amounts of data into
cassandra. I'm getting this exception:
batch_mutate failed: out of sequence response
I'm gessing is due to very big mutates. I have made the batch mutates
smaller and it seems to be behaving. Can somebody shed some light?
Izquierdo Seliva [mailto:izquie...@strands.com]
Sent: April-05-11 8:30
To: user@cassandra.apache.org
Subject: batch_mutate failed: out of sequence response
Hi everyone. I'm having trouble while inserting big amounts of data into
cassandra. I'm getting this exception:
batch_mutate failed
: Héctor Izquierdo Seliva [mailto:izquie...@strands.com]
Sent: April-05-11 8:30
To: user@cassandra.apache.org
Subject: batch_mutate failed: out of sequence response
Hi everyone. I'm having trouble while inserting big amounts of data into
cassandra. I'm getting this exception
are taking
around 5 seconds. I'm running cassandra with a heap of 8 GB. Should I tune this
somehow?
Is any of this wrong?
-Original Message-
From: Héctor Izquierdo Seliva [mailto:izquie...@strands.com]
Sent: April-05-11 8:30
To: user@cassandra.apache.org
Subject
I took a look at vmstats, and there was no swap. Also, our monitoring
tools showed no swap being used at all. It's running with mlockall and
all that. 8GB heap on a 16GB machine
El mar, 05-04-2011 a las 21:24 +0200, Peter Schuller escribió:
Would you recommend to disable system swap as a rule?
digitalpigeon.com
On Tuesday, 5 April 2011 at 11:43 PM, Héctor Izquierdo Seliva wrote:
El mar, 05-04-2011 a las 09:35 -0400, Dan Hendry escribió:
I too have seen the out of sequence response problem. My solution
has just been to retry and it seems to work. None of my mutations
are THAT large
El mié, 06-04-2011 a las 09:18 +0200, Héctor Izquierdo Seliva escribió:
I took a look at vmstats, and there was no swap. Also, our monitoring
tools showed no swap being used at all. It's running with mlockall and
all that. 8GB heap on a 16GB machine
I tried disabling swap completely
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió:
out of sequence response is thrift's way of saying I got a response
for request Y when I expected request X.
my money is on using a single connection from multiple threads. don't do
that.
I'm not using thrift directly, and
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió:
out of sequence response is thrift's way of saying I got a response
for request Y when I expected request X.
my money is on using a single connection from multiple threads. don't do
that.
I'm not using thrift directly, and
Thanks Aaron!
El mar, 12-04-2011 a las 23:52 +1200, aaron morton escribió:
Bug in the CLI, created /
fixed https://issues.apache.org/jira/browse/CASSANDRA-2458
use 70 for now.
Thanks
Aaron
On 12 Apr 2011, at 20:46, Héctor Izquierdo Seliva wrote:
Hi everyone.
I've
Hi everyone.
Looking for ways to monitor cassandra with zabbix I could not found
anything that was really usable, till I found mention of a nice class by
smeet. I have based my modification upon his work and now I give it back
to the community.
Here's the project url:
and the new
0.8.x version as soon as possible.
On 04/12/2011 07:26 PM, Héctor Izquierdo Seliva wrote:
Hi everyone.
Looking for ways to monitor cassandra with zabbix I could not found
anything that was really usable, till I found mention of a nice class by
smeet. I have based my
I'm not sure. Are you runing it in the same host as the cassandra node?
El mar, 12-04-2011 a las 22:54 +0500, Ali Ahsan escribió:
On 04/12/2011 10:42 PM, Héctor Izquierdo Seliva wrote:
I forgot to mention it has been coded against 0.7.x, and I'm not sure it
will work on 0.6.x. I'll try
Hi everyone, is there any recommended procedure to warm up a node before
bringing it up?
Thanks!
How difficult do you think this could be? I would be interested into
developing this if it's feasible.
El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió:
Hi everyone, is there any recommended procedure to warm up a node before
bringing it up?
Currently the only out-of-the-box
about retrying I'd assume not...
--
Dan Washusen
On Thursday, 7 April 2011 at 7:39 PM, Héctor Izquierdo Seliva
wrote:
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió:
out of sequence response is thrift's way of saying I got a
response
for request Y
Shouldn't the dynamic snitch take into account response times and ask a
slow node for less requests? It seems that at node startup, only a
handfull of requests arrive to the node and it keeps up well, but
there's moment where there's more than it can handle with a cold cache
and starts droping
Hi everyone. I've configured in one of my column families
memtable_operations = 0.02 and started deleting keys. I have already
deleted 54k, but there hasn't been any flush of the memtable. Memory
keeps pilling up and eventually nodes start to do stop-the-world GCs. Is
this the way this is supposed
in this case? Lower the major compaction threshold and
memtable_operations to some very low number?
Thanks
El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo Seliva escribió:
Hi everyone. I've configured in one of my column families
memtable_operations = 0.02 and started deleting keys. I have
on shutdown and re read those parts of the
files on startup. Could it be done in a similar way to the work that's
being done on page migrations? What do you think?
Thanks for your time!
On 20 Apr 2011, at 00:41, Héctor Izquierdo Seliva wrote:
Shouldn't the dynamic snitch take into account response
the
delete ?
Thanks
Aaron
I'm using 0.7.4. I have a file with all the row keys I have to delete
(around 100 million) and I just go through the file and issue deletes
through pelops.
Should I manually issue flushes with a cron every x time?
On 20 Apr 2011, at 04:21, Héctor Izquierdo Seliva wrote
El mar, 19-04-2011 a las 23:33 +0300, shimi escribió:
You can use memtable_flush_after_mins instead of the cron
Shimi
Good point! I'll try that.
Wouldn't it be better to count a delete as a one column operation so it
contributes to flush by operations?
2011/4/19 Héctor Izquierdo Seliva
I poste it a couple of messages back, but here it is again:
I'm using 0.7.4. I have a file with all the row keys I have to delete
(around 100 million) and I just go through the file and issue deletes
through pelops. Should I manually issue flushes with a cron every x
time?
El mié, 20-04-2011 a las 23:00 +1200, aaron morton escribió:
Looks like a bug, I've added a patch
here https://issues.apache.org/jira/browse/CASSANDRA-2519
Aaron
That was fast! Thanks Aaron
78 matches
Mail list logo
