RE: Compacted_at timestamp
I created a small script recently converting this timestamp into a human
readable string and sort all entries ascending.
nodetool compactionhistory |awk '{timestamp = strftime(%a %b %e %H:%M:%S %Z
%Y,$4 / 1000);in_m=$5/1024/1024;out_m=$6/1024/1024;
printf(%s\t%s\t%s\t%s\t%dM\t%dM\n,$4,timestamp,$2,$3,in_m,out_m)}' |sort -n
Regards
Andi
From: Mark Reddy [mark.l.re...@gmail.com]
Sent: 08 February 2015 21:55
To: user@cassandra.apache.org
Subject: Re: Compacted_at timestamp
Hi Santo,
If you are seeing the compacted_at value as timestamp and want to convert it to
a human readable date, this is not possible via nodetool. You will always write
a script to make the compactionhistory call and then convert the output (the
fourth column - compacted_at) to a readable date.
If you are seeing something other than an expected timestamp value, can you
post an example of what you are getting?
Regards,
Mark
On 8 February 2015 at 13:20, Havere Shanmukhappa, Santhosh
santhosh_havereshanmukha...@intuit.commailto:santhosh_havereshanmukha...@intuit.com
wrote:
When I run nodetool compactionhistory command, it displays ‘compacted_at’
timestamp in non-readable format. Any way to read that column in readable
format? I am using c*2.0.11 version.
Thanks,
Santo
Writing the same column frequently - anti pattern?
Hi, we are currently writing the same column within a row multiple times (up to 10 times a second). I am familiar with the concept of tombstones in SSTables. My question is: I assume that in our case in most cases when a column gets overwritten it still resides in the memtable. So I assume for that particular case no tombstone is set but the column is replaced in memory and then the 'newest' version is flushed to disk. Is this assumption correct? Or Is writing the same column an an anti-pattern? I am thankful for any input. Regards Andi
RE: Problems with user defined types (cql) and Datastax Java Driver
Hi Alex,
I did so. Thanks for that hint.
Andi
From: Alex Popescu [al...@datastax.com]
Sent: 05 February 2015 18:14
To: user
Subject: Re: Problems with user defined types (cql) and Datastax Java Driver
Andreas,
Can you please post your question to the Java driver ml
https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user
as you'll have better chances to get an answer there.
thanks
On Thu, Feb 5, 2015 at 9:10 AM, Andreas Finke
andreas.fi...@solvians.commailto:andreas.fi...@solvians.com wrote:
Hi,
I encountered the problem that in Java the Session does not create a valid
UserType for my corresponding CQL user defined type.
CQL_SCHEMA:
create keyspace if not exists quotes
WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TYPE IF NOT EXISTS quotes.price (
value double,
size bigint,
timestamp bigint,
delay int
);
JAVA
UserType priceType =
session.getCluster().getMetadata().getKeyspace(quotes).getUserType(price);
Assert.assertNotNull(priceType); // true
Assert.assertEquals(price, priceType.getTypeName()); // true
Assert.assertEquals(4, priceType.getFieldNames().size()); //
AssertionFailedError: expected:4 but was:0
I am testing with Cassandra v.2.1.2 on Windows using Datastax Java Driver 2.1.2.
I am thankful for any suggestions.
Regards
Andi
--
[:-a)
Alex Popescu
Sen. Product Manager @ DataStax
@al3xandru
Problems with user defined types (cql) and Datastax Java Driver
Hi,
I encountered the problem that in Java the Session does not create a valid
UserType for my corresponding CQL user defined type.
CQL_SCHEMA:
create keyspace if not exists quotes
WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TYPE IF NOT EXISTS quotes.price (
value double,
size bigint,
timestamp bigint,
delay int
);
JAVA
UserType priceType =
session.getCluster().getMetadata().getKeyspace(quotes).getUserType(price);
Assert.assertNotNull(priceType); // true
Assert.assertEquals(price, priceType.getTypeName()); // true
Assert.assertEquals(4, priceType.getFieldNames().size()); //
AssertionFailedError: expected:4 but was:0
I am testing with Cassandra v.2.1.2 on Windows using Datastax Java Driver 2.1.2.
I am thankful for any suggestions.
Regards
Andi
RE: How do replica become out of sync
Hi, right, QUORUM means that data is written to all replicas but the coordinator waits for QUORUM responses before returning back to client. If a replica is out of sync due to network or internal issue than consistency is ensured through: - HintedHandoff (Automatically http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_hh_c.html) - ReadRepair (Automatically http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dmlClientRequestsRead.html) - nodetool repair (Manually http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html) Regards Andi From: Flavien Charlon [flavien.char...@gmail.com] Sent: 19 January 2015 22:50 To: user@cassandra.apache.org Subject: How do replica become out of sync Hi, When writing to Cassandra using CL = Quorum (or anything less than ALL), is it correct to say that Cassandra tries to write to all the replica, but only waits for Quorum? If so, what can cause some replica to become out of sync when they're all online? Thanks Flavien
RE: how dump a query result into csv file
I think this might be what you are looking for http://www.datastax.com/documentation/cql/3.1/cql/cql_reference/copy_r.html Andi From: Rahul Bhardwaj [rahul.bhard...@indiamart.com] Sent: 12 January 2015 09:22 To: user Subject: how dump a query result into csv file Hi All, I want to dump a query result into a csv file with custom column delimiter. Please help. Regards: Rahul Bhardwaj Follow IndiaMART.comhttp://www.indiamart.com for latest updates on this and more: [http://www.indiamart.com/newsletters/mailer/images/google-plus-icon.jpg] https://plus.google.com/+indiamart [http://www.indiamart.com/newsletters/mailer/images/facebk-icon.jpg] https://www.facebook.com/IndiaMART [http://www.indiamart.com/newsletters/mailer/images/twitter-icon.jpg] https://twitter.com/IndiaMART Mobile Channel: [http://www.indiamart.com/newsletters/mailer/images/apple.png] https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641mt=8 [http://www.indiamart.com/newsletters/mailer/images/gplay.gif] https://play.google.com/store/apps/details?id=com.indiamart.m [http://www.indiamart.com/newsletters/mailer/images/mapp.gif] http://m.indiamart.com/ https://www.youtube.com/watch?v=DzORNbeSXN8list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1index=2 Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Haihttps://www.youtube.com/watch?v=hmS4Afl2bNU!!!
RE: Why read row is so slower than read column.
Hi, I would recommend to turn tracing on in CQL. Using this you can find out that part of the query results in high latency. http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/tracing_r.html Regards Andi From: yhq...@sina.com [yhq...@sina.com] Sent: 26 December 2014 14:01 To: user Subject: Why read row is so slower than read column. Hi, all: In my cf, each row has two column, one column is the timestamp(64bit), another column is data which may be 500k about. I read row, the qps is about 30. I read that data column, the qps is about 500. Why read performance is so slow where add a so small column in read?? Thanks.
RE: mysql based columnar DB to Cassandra DB - Migration
Hi Akshay, this heavily depends on your data model. There is no general way to do it. It includes several steps: 1) Migration of applications using Mysql to Cassandra 2) Migration of the Mysql Database to Cassandra itself Keep in mind that there are no such things like relations or joins in Cassandra. In general Actually Datastax published some documents that could be of interest for you: http://www.datastax.com/2012/03/how-to-move-data-from-relational-databases-to-datastax-enterprise-cassandra-using-sqoop http://www.datastax.com/resources/whitepapers/mysql-to-cassandra Also I found this from EbayTech: http://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-practices-part-1/#.VHV2BlXF_Eg Regards Andi From: Akshay Ballarpure [akshay.ballarp...@tcs.com] Sent: 26 November 2014 07:15 To: user@cassandra.apache.org Subject: Re: mysql based columnar DB to Cassandra DB - Migration Hello Folks, I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.comhttp://www.tcs.com/ Experience certainty.IT Services Business Solutions Consulting From:Akshay Ballarpure/HYD/TCS To:user@cassandra.apache.org Date:11/18/2014 09:00 PM Subject:mysql based columnar DB to Cassandra DB - Migration I have one mysql based columnar DB, i want to migrate it to Cassandra. How its possible ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.comhttp://www.tcs.com/ Experience certainty. IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Upgrade: C* 2.0.8 - C* 2.1.1 - Ten thousands of sstable files
Hi, we upgraded a 6 node Cluster from Cassandra 2.0.7 to 2.1.1 recently sticking to this guide: http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html After upgrade cluster was less responsive than before. One node did not came up at all. When checking the data directory, we discovered a huge amount of SSTABLES: # ls |wc -l 654057 # ls |cut -d'-' -f5 |sort|uniq -c |sort 1 snapshots 81757 CompressionInfo.db 81757 Data.db 81757 Digest.sha1 81757 Filter.db 81757 Index.db 81757 Statistics.db 81757 Summary.db 81757 TOC.txt Did anyone upgraded yet experiencing this kind of problem? Thanks and regards Andi
RE: Upgrade: C* 2.0.8 - C* 2.1.1 - Ten thousands of sstable files
Hi Robert, thanks for your fast reply. Luckily this was a upgrade in QA environment ;) We are running SizeTierCompaction. I wonder whether all those files were created during running sstables command. Unfortunately we didn’t check at that time. I will try to reproduce again and if this is the case open an issue. However thanks for the link. This is really useful. Thanks! Andi From: Robert Coli [mailto:rc...@eventbrite.com] Sent: Thursday, November 20, 2014 5:14 PM To: user@cassandra.apache.org Subject: Re: Upgrade: C* 2.0.8 - C* 2.1.1 - Ten thousands of sstable files On Thu, Nov 20, 2014 at 7:00 AM, Andreas Finke andreas.fi...@solvians.commailto:andreas.fi...@solvians.com wrote: we upgraded a 6 node Cluster from Cassandra 2.0.7 to 2.1.1 recently sticking to this guide After upgrade cluster was less responsive than before. One node did not came up at all. For the record : https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
RE: Upgrade: C* 2.0.8 - C* 2.1.1 - Ten thousands of sstable files
Hi Tyler, thanks fort that hint. I think we may postpone the C* upgrade for now and will give it another try with a higher version. There is no pressure on that. Regards Andi From: Tyler Hobbs [mailto:ty...@datastax.com] Sent: Thursday, November 20, 2014 6:09 PM To: user@cassandra.apache.org Subject: Re: Upgrade: C* 2.0.8 - C* 2.1.1 - Ten thousands of sstable files This could be due to https://issues.apache.org/jira/browse/CASSANDRA-8239, which is fixed in 2.1.2. However, it sounds like the SSTables were present immediately after upgrading, which would rule that out. On Thu, Nov 20, 2014 at 10:25 AM, Andreas Finke andreas.fi...@solvians.commailto:andreas.fi...@solvians.com wrote: Hi Robert, thanks for your fast reply. Luckily this was a upgrade in QA environment ;) We are running SizeTierCompaction. I wonder whether all those files were created during running sstables command. Unfortunately we didn’t check at that time. I will try to reproduce again and if this is the case open an issue. However thanks for the link. This is really useful. Thanks! Andi From: Robert Coli [mailto:rc...@eventbrite.commailto:rc...@eventbrite.com] Sent: Thursday, November 20, 2014 5:14 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: Upgrade: C* 2.0.8 - C* 2.1.1 - Ten thousands of sstable files On Thu, Nov 20, 2014 at 7:00 AM, Andreas Finke andreas.fi...@solvians.commailto:andreas.fi...@solvians.com wrote: we upgraded a 6 node Cluster from Cassandra 2.0.7 to 2.1.1 recently sticking to this guide After upgrade cluster was less responsive than before. One node did not came up at all. For the record : https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob -- Tyler Hobbs DataStaxhttp://datastax.com/
RE: Can SSTables overlap with SizeTieredCompactionStrategy?
Hi Phil, I found an interesting blog entry that may address your problem. http://www.datastax.com/dev/blog/optimizations-around-cold-sstables It seems that compaction is skipped for stables which so mit satisfy a certain read rate. Please check. Kind regards Andreas Finke Java Developer Solvians IT-Solutions GmbH Phil Luckhurst wrote Definitely no TTL and records are only written once with no deletions. Phil DuyHai Doan wrote Are you sure there is no TTL set on your data? It might explain the shrink in sstable size after compaction. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Can-SSTables-overlap-with-SizeTieredCompactionStrategy-tp7594574p7594644.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
RE: Can SSTables overlap with SizeTieredCompactionStrategy?
Hi Phil, there is no dump question ;) What is your size estimation based on e.g. what size is a column in your calculation? From: Phil Luckhurst [phil.luckhu...@powerassure.com] Sent: Wednesday, May 21, 2014 5:42 PM To: cassandra-u...@incubator.apache.org Subject: Re: Can SSTables overlap with SizeTieredCompactionStrategy? I'm wondering if the lack of response to this means it was a dumb question however I've searched the documentation again but I still can't find an answer :-( Phil -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Can-SSTables-overlap-with-SizeTieredCompactionStrategy-tp7594574p7594627.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
RE: Tombstones
Hi Dimetrio, From the wiki: Since 0.6.8, minor compactions also GC tombstones Regards Andi Dimetrio wrote Does cassandra delete tombstones during simple LCS compaction or I should use node tool repair? Thanks. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Tombstones-tp7594467.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
RE: Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
Hi Aaron, I assume you mean seed_provider setting in cassandra.yaml by seed list. The current setting for vm1-vm6 is: seed_provider = vm1,vm2,vm3,vm4 This setting also applied when the vm5 and vm6 were added. I checked the read repair metrics and it is about mean 20/s on vm5 and vm6. I tried to investigate the real distribution of tokens again and did on vm1: 1. nodetool describering marketdata /tmp/ring.txt 2. for node in vm1 vm2 vm3 vm4 vm5 vm6 ; do cat /tmp/ring.txt |grep ip_of($node) | wc -l; done This prints the number of times when a node was listed as endpoint: vm1: 303 vm2: 312 vm3: 332 vm4: 311 vm5: 901 vm6: 913 So this shows that we are really unbalanced. 1. Is there any way how we can fix that on a running production cluster? 2. Our backup plan is to snapshot all data, raise a complete fresh 6 node cluster and stream the data using sstable loader. Are there any objections about that plan from your point of view? Thanks in advance! Andi From: Aaron Morton [aa...@thelastpickle.com] Sent: Wednesday, December 18, 2013 3:14 AM To: Cassandra User Subject: Re: Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes Node: 4 CPU, 6 GB RAM, virtual appliance Cassandra: 3 GB Heap, vnodes 256 FWIW that’s a very low powered node. Maybe we forgot necessary actions during or after cluster expanding process. We are open for every idea. Where the nodes in the seed list when they joined the cluster? If so they do not bootstrap. The extra writes in nodes 5 and 6 could be from Read Repair writing to them. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/12/2013, at 11:49 pm, Andreas Finke andreas.fi...@solvians.com wrote: Hi, after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing high load on both new nodes. After doing some investigation we found out the following: - High cpu load on vm5+6 - Higher data load on vm5+6 - Write requests are evenly distributed to all 6 nodes by our client application (opscenter - metrics - WriteRequests) - Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s) - Nodetool output: UN vm1 9.51 GB256 20,7% 13fa7bb7-19cb-44f5-af83-71a72e04993a X1 UN vm2 9.41 GB256 20,0% b71c2d3d-4721-4dde-a418-802f1af4b7a1 D1 UN vm3 9.37 GB256 18,9% 8ce4c419-d79c-4ef1-b3fd-8936bff3e44f X1 UN vm4 9.23 GB256 19,5% 17974f20-5756-4eba-a377-52feed3a1b10 D1 UN vm5 15.95 GB 256 10,7% 0c6db9ea-4c60-43f6-a12e-51a7d76f8e80 X1 UN vm6 14.86 GB 256 10,2% f64d1909-dd96-442b-b602-efee29eee0a0 D1 Although the ownership is lower on vm5-6 (which already is not right) the data load is way higher. Some cluster facts: Node: 4 CPU, 6 GB RAM, virtual appliance Cassandra: 3 GB Heap, vnodes 256 Schema: Replication strategy network, RF:2 Has anyone an idea what could be the cause for the unbalancing. Maybe we forgot necessary actions during or after cluster expanding process. We are open for every idea. Regards Andi
RE: Data File Mechanism
Hi Bonnet, regarding http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/cql_storage_options_c.html#concept_ds_xnr_4mw_xj__moreCompaction there is the setting sstable_size_in_mb // The target size for SSTables that use the leveled compaction strategy. for LeveledCompactionStrategy Regards Andi From: Bonnet Jonathan. [jonathan.bon...@externe.bnpparibas.com] Sent: Friday, December 20, 2013 11:07 AM To: user@cassandra.apache.org Subject: Data File Mechanism Hello, If possible, i need to know if it's possible to chose how a file *.db can grow, it's limit and the general mechanism about the data files. There is only one file *.db by column familly for one node (except index-filter-statistics-summary) ?, or when the CF grow cassandra add it one more ? Regards, Bonnet Jonathan.
Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
Hi, after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing high load on both new nodes. After doing some investigation we found out the following: - High cpu load on vm5+6 - Higher data load on vm5+6 - Write requests are evenly distributed to all 6 nodes by our client application (opscenter - metrics - WriteRequests) - Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s) - Nodetool output: UN vm1 9.51 GB256 20,7% 13fa7bb7-19cb-44f5-af83-71a72e04993a X1 UN vm2 9.41 GB256 20,0% b71c2d3d-4721-4dde-a418-802f1af4b7a1 D1 UN vm3 9.37 GB256 18,9% 8ce4c419-d79c-4ef1-b3fd-8936bff3e44f X1 UN vm4 9.23 GB256 19,5% 17974f20-5756-4eba-a377-52feed3a1b10 D1 UN vm5 15.95 GB 256 10,7% 0c6db9ea-4c60-43f6-a12e-51a7d76f8e80 X1 UN vm6 14.86 GB 256 10,2% f64d1909-dd96-442b-b602-efee29eee0a0 D1 Although the ownership is lower on vm5-6 (which already is not right) the data load is way higher. Some cluster facts: Node: 4 CPU, 6 GB RAM, virtual appliance Cassandra: 3 GB Heap, vnodes 256 Schema: Replication strategy network, RF:2 Has anyone an idea what could be the cause for the unbalancing. Maybe we forgot necessary actions during or after cluster expanding process. We are open for every idea. Regards Andi
