Re: Possible problem with disk latency
Hi, 8GB Heap is a good value already - going above 8GB will often result in noticeable gc pause times in java, but you can give 12G a try just to see if that helps (and turn it back down again). You can add a Heap Used graph in opscenter to get a quick overview of your heap state. Best regards, Roland
Re: Possible problem with disk latency
Hi Piotrek, your disks are mostly idle as far as I can see (the one with 17% busy isn't that high on load). One thing came up to my mind did you look on the sizes of your sstables? I did this with something like find /var/lib/cassandra/data -type f -size -1k -name *Data.db | wc find /var/lib/cassandra/data -type f -size -10k -name *Data.db | wc find /var/lib/cassandra/data -type f -size -100k -name *Data.db | wc ... find /var/lib/cassandra/data -type f -size -100k -name *Data.db | wc Your count is growing from opscenter - and if there are many really small tables I would guess you are running out of heap. If memory pressure is high it is likely that there will be much flushes of memtables to disk with many small files - had this once. You can increase heap in cassandra-env.sh, but be careful. Best regards, Roland
Re: Many pending compactions
Hi, 2.1.3 is now the official latest release - I checked this morning and got this good surprise. Now it's update time - thanks to all guys involved, if I meet anyone one beer from me :-) The changelist is rather long: https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-2.1.3 Hopefully that will solve many of those oddities and not invent to much new ones :-) Cheers, Roland
Re: can't delete tmp file
Hi, try 2.1.3 - with 2.1.2 this is normal. From the changelog: * Make sure we don't add tmplink files to the compaction strategy (CASSANDRA-8580) * Remove tmplink files for offline compactions (CASSANDRA-8321) In most cases they are safe to delete, I did this when the node was down. Cheers, Roland
Re: Data tiered compaction and data model question
Hi Cass, just a hint from the off - if I got it right you have: Table 1: PRIMARY KEY ( (event_day,event_hr),event_time) Table 2: PRIMARY KEY (event_day,event_time) Assuming your events to write come in by wall clock time, the first table design will have a hotspot on a specific node getting all writes for a single hour as (event_day,event_hr) is the partioning key. The second table design will put this hotspot on a specific node per day as event_day is the partitoning key. So please be careful if you have a write intensive workload. I have designed my logging tables with a non datetime key in my partioning key to distribute writes to all nodes at a specific point in time. I have for example PRIMARY KEY ((sensor_id,measure_date)) and the timestamp-value pairs in the rows. They are quite wide as I have about 1 measurements per sensor and id, but analytics and cleanup jobs run daily. Of course as a not so long time cassandra user I can be wrong, please feel free to correct me. Cheers, Roland
Re: Many pending compactions
Hi, 1) Actual Cassandra 2.1.3, it was upgraded from 2.1.0 (suggested by Al Tobey from DataStax) 7) minimal reads (usually none, sometimes few) those two points keep me repeating an anwser I got. First where did you get 2.1.3 from? Maybe I missed it, I will have a look. But if it is 2.1.2 whis is the latest released version, that version has many bugs - most of them I got kicked by while testing 2.1.2. I got many problems with compactions not beeing triggred on column families not beeing read, compactions and repairs not beeing completed. See https://www.mail-archive.com/search?l=user@cassandra.apache.orgq=subject:%22Re%3A+Compaction+failing+to+trigger%22o=newestf=1 https://www.mail-archive.com/user%40cassandra.apache.org/msg40768.html Apart from that, how are those both datacenters connected? Maybe there is a bottleneck. Also do you have ntp up and running on all nodes to keep all clocks in thight sync? Note: I'm no expert (yet) - just sharing my 2 cents. Cheers, Roland
Re: How to deal with too many sstables
Hi,
maybe you are running into an issue that I also had on my test cluster.
Since there were almost no reads on it cassandra did not run any minor
compactions at all. Solution for me (in this case) was:
ALTER TABLE tablename WITH compaction = {'class':
'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold':
'32', 'cold_reads_to_omit': 0.0};
where cold_reads_to_omit is the trick.
Anyway as Eric and Marcus among others suggest, do not run 2.1.2 for
production as it has many issues. I'm looking forward to test 2.1.3 when
it arrives.
Cheers,
Roland
Am 03.02.2015 um 03:05 schrieb 曹志富:
Hi,all:
I have 18 nodes C* cluster with cassandra2.1.2.Some nodes have aboud
40,000+ sstables.
my compaction strategy is STCS.
Could someone give me some solution to deal with this situation.
Thanks.
--
曹志富
手机：18611121927
邮箱：caozf.zh...@gmail.com mailto:caozf.zh...@gmail.com
微博：http://weibo.com/boliza/
Re: incremential repairs - again
Hi, the automatically meant this reply earlier: If you are on 2.1.2+ (or using STCS) you don't those steps (should probably update the blog post). Now we keep separate levelings for the repaired/unrepaired data and move the sstables over after the first incremental repair My understanding was, that there are no steps necessary to migrate to incremental repairs (disableautocompaction and so on) for 2.1.2+. I'm looking forward to the next release - and I'll give a test run. Thanks so far :-)
incremential repairs - again
Hi, a short question about the new incremental repairs again. I am running 2.1.2 (for testing). Marcus pointed me that 2.1.2 should do incremental repairs automatically, so I rolled back all steps taken. I expect that routine repair times will decrease when I do not put many new data on the cluster. But they dont - they are constant at about 1000 minutes per node, so I extracted all Repaired at with sstablemetadata and I cant see any recent date. I put several GB of data into the cluster in 2015 and I run nodetool repair -pr on every node regularly. Am I still missing something? Or is this one of the issues with 2.1.2 (CASSANDRA-8316)? Thanks for hints, Jan
Re: SStables can't compat automaticly
Hi,
are you running 2.1.2 evenutally? I had this problem recently and there
were two topics here about this. Problem was, that my test cluster had
almost no reads and did not compact sstables.
Reason for me was that those minor compactions did not get triggered
since there were almost no reads on that tables. Setting
'cold_reads_to_omit' to 0 did the job for me:
ALTER TABLE tablename WITH compaction = {'class':
'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold':
'32', 'cold_reads_to_omit': 0.0};
Credits to Tyler and Eric for the pointers.
Maybe that could help.
Cheers,
Roland
Am 26.01.2015 um 09:56 schrieb 曹志富:
No,to confirm this I have run the command all my nodes:bin/nodetool
enableautocompaction
--
曹志富
手机：18611121927
邮箱：caozf.zh...@gmail.com mailto:caozf.zh...@gmail.com
微博：http://weibo.com/boliza/
2015-01-26 16:49 GMT+08:00 Jason Wee peich...@gmail.com
mailto:peich...@gmail.com:
Did you disable auto compaction through nodetool?
disableautocompactionDisable autocompaction for the given
keyspace and column family
Jason
On Mon, Jan 26, 2015 at 11:34 AM, 曹志富 cao.zh...@gmail.com
mailto:cao.zh...@gmail.com wrote:
Hi everybody:
I have 18 nodes using cassandra2.1.2.Every node has 4 core, 32
GB RAM, 2T hard disk,OS is CentOS release 6.2 (Final).
I have follow the Recommended production settings to config
my system.such as disable SWAP,unlimited mem lock...
My heap size is:
MAX_HEAP_SIZE=8G
MIN_HEAP_SIZE=8G
HEAP_NEWSIZE=2G
I use STCS,other config using default,using Datastax Java
Driver 2.1.2. BatchStatment 100key commit per time.
When I run my cluster and insert data from kafka (1
keys/s) after 2 days,every node can't compact some there too
many sstables.
I try to use major compact to compact the sstables , it cost a
long long time .Also the new sstables can't compat automatic.
I trace the log , the CMS GC too often,almost 30 minute onetime.
Could someone help me to solve this problem.
--
曹志富
手 机：18611121927
邮箱：caozf.zh...@gmail.com mailto:caozf.zh...@gmail.com
微博：http://weibo.com/boliza/
Re: Many really small SSTables
Hi, just as a short follow up, it worked - all nodes now have 20-30 sstables instead of thousands. Cheers, Roland
Re: Compaction failing to trigger
Hi Flavien,
I hit some problem with minor compations recently (just some days ago) -
but with many more tables. In my case compactions got not triggered, you
can check this with nodetool compactionstats.
Reason for me was that those minor compactions did not get triggered
since there were almost no reads on that tables. Setting
'cold_reads_to_omit' to 0 did the job for me:
ALTER TABLE tablename WITH compaction = {'class':
'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold': '32',
'cold_reads_to_omit': 0.0};
Credits to Tyler and Eric for the pointers.
Cheers,
Roland
Many really small SSTables
Hi, I'm testing around with cassandra fair a bit, using 2.1.2 which I know has some major issues,but it is a test environment. After some bulk loading, testing with incremental repairs and running out of heap once I found that now I have a quit large number of sstables which are really small: 1k 0 0,0% 10k 2780 76,8% 100k 3392 93,7% 1000k3461 95,6% 1k 3471 95,9% 10k 3517 97,1% 100k 3596 99,3% all 3621100,0% 76,8% of all sstables in this particular column familiy are smaller that 10kB, 93.7% are smaller then 100kB. Just for my understanding - does that impact performance? And is there any way to reduce the number of sstables? A full run of nodetool compact is running for a really long time (more than 1day). Thanks for any input, Roland
incremental repairs
Hi, I am currently trying to migrate my test cluster to incremental repairs. These are the steps I'm doing on every node: - touch marker - nodetool disableautocompation - nodetool repair - cassandra stop - find all *Data*.db files older then marker - invoke sstablerepairedset on those - cassandra start This is essentially what http://www.datastax.com/dev/blog/anticompaction-in-cassandra-2-1 says. After all nodes migrated this way, I think I need to run my regular repairs more often and they should be faster afterwards. But do I need to run nodetool repair or is nodetool repair -pr sufficient? And do I need to reenable autocompation? Oder do I need to compact myself? Thanks for any input, Roland
Re: incremental repairs
Hi Marcus, thanks for that quick reply. I did also look at: http://www.datastax.com/documentation/cassandra/2.1/cassandra/operations/ops_repair_nodes_c.html which describes the same process, it's 2.1.x, so I see that 2.1.2+ is not covered there. I did upgrade my testcluster to 2.1.2 and with your hint I take a look at sstablemetadata from a non migrated node and there are indeed Repaired at entries on some sstables already. So if I got this right, in 2.1.2+ there is nothing to do to switch to incremental repairs (apart from running the repairs themself). But one thing I see during testing is that there are many sstables, with small size: - in total there are 5521 sstables on one node - 115 sstables are bigger than 1MB - 4949 sstables are smaller than 10kB I don't know where they came from - I found one piece of information where this happend when cassandra was low on heap which happend to me while running tests (the suggested solution is to trigger compaction via JMX). Question for me: I did disable autocompaction on some nodes of our test cluster as the blog and docs said. Should/can I reenable autocompaction again with incremental repairs? Cheers, Roland
Re: incremental repairs
Hi Marcus, thanks a lot for those pointers. Now further testing can begin - and I'll wait for 2.1.3. Right now on production repair times are really painful, maybe that will become better. At least I hope so :-)
