I would think that compaction_throughput_kb_per_sec does have indirect impact
on disk IO. High number means or setting it to 0 means there is no
throttling on how much IO is being performed. Wouldn't it impact normal
reads from disk during the time when disk IO or util is high which
compaction is
and updates could be scattered all over
before compaction?
No, updates to a given row will be still be in a single sstable.
Can you please explain little more? You mean that if Level 1 file contains
range from 1-100 all the updates would still go in that file?
The link on leveldb
This is a great new! Is it possible to do a write-up of main changes like
Leveldb and explain it a little bit. I get lost reading JIRA and sometimes
is difficult to follow the thread. It looks like there are some major
changes in this release.
--
View this message in context:
Thanks for the update
Jeremy Hanna wrote:
It appears though that when choosing the non-local replicas, it looks for
the next token in the ring of the same rack and the next token of a
different rack (depending on which it is looking for).
Can you please explain this little more?
--
View
Ruby Stevenson wrote:
hi Sasha -
Yes indeed. this solution was in the second part of my original
question - it just seems out of norm on what people usually use
Cassandra for, I guess I am looking for some reassurance before I roll
up the sleeve of trying it.
Thanks
Ruby
It
Are you seeing lot of these errors? Can you try XX:-OmitStackTraceInFastThrow
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/no-stack-trace-tp6654590p6657485.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Check things like netstats, disk space etc to see why it's in Leaving state.
Anything in the logs that shows Leaving?
--
View this message in context:
What happens when DC is in different time zone so 9:00 pacific vs 11:00
Central
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Question-about-eventually-consistent-in-Cassandra-tp6646430p6649490.html
Sent from the
First run nodetool move and then you can run nodetool repair. Before you run
nodetool move you will need to determine tokens that each node will be
responsible for. Then use that token to perform move.
--
View this message in context:
springrider wrote:
is that okay to do nodetool move before a completely repair?
using this equation?
def tokens(nodes):
- for x in xrange(nodes):
- print 2 ** 127 / nodes * x
Yes use that logic to get the tokens. I think it's safe to run move first
and reair later. You are
CASSANDRA learner wrote:
Hi,
Can we store images , java objects, files in cassandra, if so , how
Please let me know this as i need it urgently...
Look at http://goo.gl/S2E3C http://goo.gl/S2E3C
It really depends on your workload. With heavy workloads cassandra is not
the right
Trying to understand the overhead when multiple columns are spread accross
ssTables. For eg: Key K1 column b and c are in ssTable 1 and column a in
ssTable 2. As I understand columns in a given row are sorted at the time
it's stored. So does it mean that when a goes to ssTable 2 it also fetches
Thanks! Then does it mean that before compaction if read call comes for that
key sort is done at the read time since column b, c and a are in different
ssTables.
--
View this message in context:
Peter Schuller wrote:
Recently upgraded to 0.8.1 and noticed what seems to be missing data
after a
commitlog replay on a single-node cluster. I start the node, insert a
bunch
of stuff (~600MB), stop it, and restart it. There are log messages
If you stop by a kill, make sure you use
try nohup
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-Capistrano-recipes-tp6556591p6556636.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
aaron morton wrote:
Not sure what the intended purpose is, but we've mostly used it as an
emergency disk-capacity-increase option
Thats what I've used it for.
Cheers
How does compaction work in terms of utilizing multiple data dirs? Also, is
there a reference on wiki somewhere that
I thought there is an option to give multiple data dirs in cassandra.yaml.
What's the purpose of that?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/RAID-or-no-RAID-tp6522904p6523523.html
Sent from the cassandra-u...@incubator.apache.org
How should one go about creating a data model from RDBMS ER into Big Table
Data model? For eg: RDBMS has many indexes required for queries and I think
this is the most important aspect when desiging the data model in Big Table.
I was initially planning to denormalize into one CF and use
Speaking purely from my personal experience, I haven't found cassandra
optimal for storing big fat rows. Even if it is only 100s of KB I didn't
find cassandra suitable for it. In my case I am looking at 400 writes + 400
reads per sec and grow 20%-30% every ear with file sizes from 70k-300k. What
I
In my opinion 5 9s don't matter. It's the number of impacted customers. You
might be down during peak for 5 mts causing 1000s of customer turn aways
while you might be down during night causing only few customer turn aways.
There is no magic bullet. It's all about learning and improving. You will
Start with reading comments on cassandra.yaml and
http://wiki.apache.org/cassandra/Operations
http://wiki.apache.org/cassandra/Operations
As far as I know there is no comprehensive list for performance tuning. More
specifically common setting applicable to everyone. For most part issues
revolve
Les Hazlewood wrote:
I have architected, built and been responsible for systems that support
4-5
9s for years.
So have most of us. But probably by now it should be clear that no
technology can provide concrete recommendations. They can only provide what
might be helpful which varies from
LOCAL_QUORUM gurantees consistency in the local data center only. Other
replica nodes in the same DC and other DC not part of the QUORUM will be
eventually consistent. If you want to ensure consistency accross DCs you can
use EACH_QUORUM but keep in mind the latency involved assuming DCs are not
Well it depends on the requirements. If you use any combination of CL with
EACH_QUORUM it means you are accepting the fact that you are ok if one of
the DC is down. And in your scenario you care more about DCs being
consistent even if writes were to fail. Also you are ok with network
latency.
I
Please give more detailed info about what exactly you are worried about or
trying to solve.
Please take a step back and look at cassandra's architecture again and what
it's trying to solve. It's a distributed database so if you do what you are
describing there is a potential of getting hotspots.
Khanh Nguyen wrote:
Is there a way to tell where a piece of data is stored in a cluster?
For example, can I tell if LastNameColumn['A'] is stored at node 1 in
the ring?
I have not used it but you can see getNaturalEndpoints in jmx. It will tell
you which nodes are responsible for a given
Fredrik Stigbäck wrote:
Does reading quorum mean only waiting for quorum respones or does it mean
quorum respones with same latest timestamp?
Regards
/Fredrik
Well it depends on how your CL is for writes. If you write with QUORUM and
then read with QUORUM then yes you will get at least
Can you post the output of netstat -anp|grep LISTEN|grep java from all
the 3 nodes?
Also compare seconds nodes yaml with new nodes yaml and see what diff. you
find, if any.
Another thing try telnet tests from seed node to the new node.
--
View this message in context:
Whenever I hear someone say data is corrupted I panic :) I have seen few
people have reported that but have not seen the real reason for it. Is it a
manual error, config error, bug etc. It will be good to identify why these
things happen so that it can be fixed before it happens in PROD :(
--
What's your avg column size and row size? Your read latency in most case will
directly be related to how much you are trying to read. In my experience you
will see high read latency if you have big column size.
--
View this message in context:
Coherence is similar to memcachd (free). It's in memory cache layer on top of
the DB. You as a user need to keep that cache in sync with the DB.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-Vs-Oracle-Coherence-tp6375561p6386847.html
Brandon Williams wrote:
There was a bug, it is fixed. It's just a cache, chill.
There is no time to chill when fighting it in production :) It's good to
know it's fixed.
Another question, when this happens are we able to restore data from replica
nodes?
--
View this message in context:
In this case, yes. I was asking for the cases where commit log corruption was
reported.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Re-Exception-when-starting-tp6383464p6387101.html
Sent from the cassandra-u...@incubator.apache.org mailing
I am wondering if running nodetool repair will help in anyway
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Inconsistent-results-using-secondary-indexes-between-two-DC-tp632p6382819.html
Sent from the cassandra-u...@incubator.apache.org
Do you see anything in log files?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Commitlog-Disk-Full-tp6356797p6374234.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
Those messages are ok to ignore. It's basically deleting the files that are
already flused as SSTables.
Which version are you running?
Have you tried restarting the node?
Pick one node and send ls -ltr output also the complete log files since
your last restart from the same node. I looked at
You can try to update column family using cassandra-cli. Try to set
memtable_throughput to 32 first.
[default@unknown] help update column family;
update column family Bar;
update column family Bar with att1=value1;
update column family Bar with att1=value1 and att2=value2...;
Update a column
Is there a way to look at the actual size of memtable? Would that help?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Commitlog-Disk-Full-tp6356797p6360001.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
5G in one hour is actually very low. Something else is wrong. Peter pointed
to something related to memtable size could be causing this problem, can you
turn down memtable_throughput and see if that helps.
--
View this message in context:
What do you mean by Bad memory? Is it less heap size, OOM issues or something
else? What happens in such scenario, is there a data loss?
Sorry for many questions just trying to understand since data is critical
afterall :)
--
View this message in context:
What would be the procedure in this case? Run drain on the node that is
disagreeing? But is it enough to run just drain or you suggest drain + rm
system files?
--
View this message in context:
In my case all hosts were reachable and I ran nodetool ring before running
the schema update. I don't think it was because of node being down. I tihnk
for some reason it just took over 10 secs because I was reducing key_cache
from 1M to 1000. I think it might be taking long to trim the keys hence
I don't think I got correct answer to my original post. Can someone please
help?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Two-versions-of-schema-tp6277365p6280070.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive
How to intepret Key cache hit rate? What does this no mean?
Keyspace: StressKeyspace
Read Count: 87579
Read Latency: 11.792417360326105 ms.
Write Count: 179749
Write Latency: 0.009272318622078566 ms.
Pending Tasks: 0
Column Family:
Is there a problem?
[default@StressKeyspace] update column family StressStandard with
keys_cached=100;
854ee0a0-6792-11e0-81f9-93d987913479
Waiting for schema agreement...
The schema has not settled in 10 seconds; further migrations are ill-advised
until it does.
Versions are
I ran stress test to read 50K rows and since then I am getting below error
even though ring show all nodes are up:
ERROR 12:40:29,999 Exception:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked
down. Retry burden pushed out to client.
at
Peter Schuller wrote:
Saturated.
But read latency is still something like 30ms which I would think would be
much higher if it's saturated.
--
View this message in context:
One correction qu size in iostat ranges between 6-120. But still this doesn't
explain why read latency is low in cfstats.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6269875.html
Sent from
I still don't understand. You would expect read latency to increase
drastically when it's fully saturated and lot of READ drop messages also,
correct? I don't see that in cfstats or system.log which I don't really
understand why.
--
View this message in context:
Actually when I run 2 stress clients in parallel I see Read Latency stay the
same. I wonder if cassandra is reporting accurate nos.
I understand your analogy but for some reason I don't see that happening
with the results I am seeing with multiple stress clients running. So I am
just confused
Here is what cfhistograms look like. Don't really understand what this means,
will try to read. I also %util in iostat continuously 90%. Not sure if this
is caused by extra reads by cassandra. It seems unusual.
[root@dsdb4 ~]# nodetool -h `hostname` cfhistograms StressKeyspace
StressStandard
Can someone please help?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Lot-of-pending-tasks-for-writes-tp6263462p6266213.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
I am using cassandra 7.4 and getting these messages.
Heap is 0.7802529021498031 full. You may need to reduce memtable and/or
cache sizes Cassandra will now flush up to the two largest memtables to free
up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if
you don't want
64 bit 12 core 96 GB RAM
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266400.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
Yes
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266726.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
One thing I am noticing is that cache hit rate is very low even though my
cache key size is 1M and I have less than 1M rows. Not sure why so many
cache miss?
Keyspace: StressKeyspace
Read Count: 162506
Read Latency: 45.22479006928975 ms.
Write Count: 247180
Write
Does it really matter how long cassandra has been running? I thought it will
keep keys of 1M at least.
Regarding your previous question about queue size in iostat I see it ranging
from 114-300.
--
View this message in context:
I am running stress test using hector. In the client logs I see:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at
I see this occurring often when all cassandra nodes all of a sudden show CPU
spike. All reads fail for about 2 mts. GC.log and system.log doesn't reveal
much.
Only think I notice is that when I restart nodes there are tons of files
that gets deleted. cfstats from one of the nodes looks like this:
It looks like hector did retry on all the nodes and failed. Does this then
mean cassandra is down for clients in this scenario? That would be bad.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Timeout-during-stress-test-tp6262430p6263270.html
I am running stress test and on one of the nodes I see:
[root@dsdb5 ~]# nodetool -h `hostname` tpstats
Pool NameActive Pending Completed
ReadStage 0 0 2495
RequestResponseStage 0 0 242202
But I don't understand the reason for oveload. It was doing simple read of 12
threads and reasing 5 rows. Avg CPU only 20%, No GC issues that I see. I
would expect cassandra to be able to process more with 6 nodes, 12 core, 96
GB RAM and 4 GB heap.
--
View this message in context:
aaron morton wrote:
You'll need to provide more information, from the TP stats the read stage
could not keep up. If the node is not CPU bound then it is probably IO
bound.
What sort of read?
How many columns was it asking for ?
How many columns do the rows have ?
Was the test
What's the difference between a row index and sstable index?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Columns-values-integer-need-frequent-updates-increments-tp6251464p6259882.html
Sent from the cassandra-u...@incubator.apache.org mailing
What is a storage proxy latency?
By query latency you mean the one in cfstats and cfhistorgrams?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/What-need-to-be-monitored-while-running-stress-test-tp6255765p6257932.html
Sent from the
If there are multiple updates to same columns and scattered accross multiple
sstables then how does cassandra know which sstable has the most recent
value.
--
View this message in context:
Thanks for the info!
Does this also happens if initial_token is set?
Also, I am unable to understand the last line in that JIRA
A potential complication was that seed nodes were moved without using the
correct procedure of de-seeding them first. This was clearly wrong
What is de-seeding
That I understand but my basic quesiton was how does it know that there are
multiple updates that have occurred on the same column? and how does it
efficiently knows which sstable have these updates?
--
View this message in context:
in yaml:
# Set to true to make new [non-seed] nodes automatically migrate data
# to themselves from the pre-existing nodes in the cluster.
Why only non-seed nodes? What if seed nodes need to bootstrap?
--
View this message in context:
I am starting a stress test using hector on 6 node machine 4GB heap and 12
core. In hectore readme this is what I got by default:
create keyspace StressKeyspace
with replication_factor = 3
and placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy';
use StressKeyspace;
drop
What are the key things to monitor while running a stress test? There is tons
of details in nodetoll tpstats/netstats/cfstats. What in particular should I
be looking at?
Also, I've been looking at iostat and await really goes high but cfstats
shows low latency in microsecs. Is latency in cfstats
I see this error in the logs posted. Is this normal?
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
I think best is to have global load balancer in front of web servers/app
servers. And leave app servers to handle requests at local quoram. If data
center goes down then load balancer will simply hand out only one DCs ips.
--
View this message in context:
Where can I read more about CQL? I am assuming it's similar to SQL and
drivers like JDBC can be written on top of it. Is that right?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ditching-Cassandra-tp6221436p6231654.html
Sent from the
Is there a way to monitor the compactions using nodetools? I don't see it in
tpstats.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Endless-minor-compactions-after-heavy-inserts-tp6229633p6231672.html
Sent from the
I can't find it on wiki. Do you have a link where it can give detail help?
Also, is the latency in micro sec. or millisec?
How about latency in cfstats? Is it micro or mill? It says ms which is gen.
millisec.
--
View this message in context:
If I am not wrong node repair need to be run on all the nodes in staggerred
manner. It is required to take care of tombstones. Please correct me team if
I am wrong :)
See Distributed Deletes:
http://wiki.apache.org/cassandra/Operations
--
View this message in context:
Cassandra 7.4:
nodetool -h `hostname` cfhistograms system schema
Exception in thread main java.lang.reflect.UndeclaredThrowableException
at $Proxy5.getRecentReadLatencyHistogramMicros(Unknown Source)
at
org.apache.cassandra.tools.NodeCmd.printCfHistograms(NodeCmd.java:452)
It looks like if I use system schema it fails. Is it because of
LocalPartitioner?
I ran with other keyspace and got following output.
Offset SSTables Write Latency Read Latency Row Size Column Count
1 0 0 0 0 0
2 0 0 0 0 0
179 0 0 0 320 320
Can someone please help me understand the output in
Is there a way to monitor and tell if one of the node require repair? For eg:
Node was down and came back up but in the meantime HH were dropped. Now
unless we are really careful in all the scenarios we wouldn't have any
problems :) but in general when things are going awry you might forget about
Yes but that doesn't really provide the monitoring that will really be
helpful. If I don't realize it until 2 days then we potentially could be
returning inconsistent results or not have data sync for 2 days until repair
is run. It will be best to be able to monitor these things so that it can be
I think what I feel is that there is a need to know if repair is required
flag in order for team to manage the cluster.
Atleast at minimum, Is there a flag somewhere that tells if repair was run
within GCGracePeriod?
--
View this message in context:
I am also interested in knowing when 8 will be released. Also, is there
someplace where we can read about features that will be relased in 8? Looks
like some major changes are going to come out.
--
View this message in context:
Thanks everyone this gives me a good head start.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-monitoring-of-Cassandra-cluster-tp6205275p6208331.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
Can someone share if they have centralized monitoring for all cassandra
servers. With many nodes it becomes difficult to monitor them individually
unless we can look at data in one place. I am looking at solutions where
this can be done. Looking at Cacti currently but not sure how to integrate
it
I think what I am trying to ask is this:
what happens if it's RF=3 with network toplogy (RackInferringSnitch) and 2
copies are stored in Site A and 1 copy in Site B data center. Now client for
some reason is directed to Site B data center and does a write/update on
existing column, now would Site
CL is just a way to satisfy consistency but you still want majority of your
reads (preferrably) occurring in the same DC.
I don't think that answers my question at all. I understand the CL but I
think I have more basic and important question about active/active data
center and the replicas in
When in active/active data center how to decide right replication factor?
Client may connect and request for the information from either data center
so if locally it's RF=3 then in multiple data center should it be RF=6 in
active/active?
Or what happens if it's RF=3 with network toplogy and 2
Also when it comes to RAID controller there are other options like write
policy, read policy, cache io/direct io. Is there any preference on which
policies should be chosen?
In our case:
http://support.dell.com/support/edocs/software/svradmin/1.9/en/stormgmt/cntrls.html
--
View this message in
That is from the wiki http://wiki.apache.org/cassandra/StorageConfiguration
http://wiki.apache.org/cassandra/StorageConfiguration
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Seed-tp6162837p6174450.html
Sent from the
I am assuming it is the seed node that tells who are the other member in the
cluster. And then does the new node joining the cluster send join message
(something like that) to other nodes or is there a master coordinator (like
jboss cluster) that tells other nodes that new node has joined?
--
Tyler Hobbs-2 wrote:
Seeds:
Never use a node's own address as a seed if you are bootstrapping it by
setting autobootstrap to true!
I came accross this on the wiki. Can someone please help me understand this
with some example?
--
View this message in context:
Currently, in cassandra.yaml disk_access_mode is set to auto but the
recommendation seems to be to use 'mmap_index_only'.
If we use HugePages then do we still need to worry about setting
disk_access_mode to mmap? I am planning to enable HugePages and use
-XX:+UseLargePages option in JVM. I had a
Jonathan Ellis-3 wrote:
Wrong. The recommendation is to leave it on auto.
this is where I see mmap recommended for index.
http://wiki.apache.org/cassandra/StorageConfiguration
http://wiki.apache.org/cassandra/StorageConfiguration
Jonathan Ellis-3 wrote:
HugePages has nothing to do
Thanks! I think it still is a good idea to enable HiugePages and use
UseLargePageSize option in JVM. What do you think?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Linux-HugePages-and-mmap-tp6170193p6171008.html
Sent from the
Is there a way to monitor how far behind the sync is? In case of hinted hand
off or when node is down for extended period of time it will probably be
helpful to know.
--
View this message in context:
Yes I understand that piece but my thought is that if node is down and came
up but at that point we want to know how long the sync will take in case
there were another node to fail in the replica set. It also is good data
point to see how long it takes to sync.
It's always good to have this data
I've read in some posts before and on the wiki that all the nodes in the
cluster should have same seed list and 2 is the no. of seeds recommended. My
question is it advisable to have node seed itself. Say for eg Node A, Node B
and Node C in a cluster have a seed list of A and B. Now according to
mcasandra wrote:
aaron morton wrote:
The issue I think you and Patrik are seeing occurs when you *remove*
nodes from the ring. The ring does not know if they are up or down. E.g.
you have a ring of 3 nodes, and add a keyspace with RF 3. Then for
whatever reason 2 nodes are removed
I am completely confused. I repeated same test after turning on
auto_bootstrap to true and it worked this time. I did it exactly same way
where I killed 2 nodes and this time it started with no issues.
Could it be because once auto_bootstrap is off it's off forever?
I am using hector and
1 - 100 of 159 matches
Mail list logo
