Re: Data.db too large and after sstableloader still large
Tks,Rob. We use spark-cassandra-connector to read data from table, then do repartition action. If some nodes with large file bring out running this tasktoo slow, maybe serveral hours which is unacceptable. But those nodes with small file running finished quickly. So I think if sstableloader can split to small size, and can balance to all nodes, thus our spark job can running quickly. Tks,qihuang.zheng 原始邮件 发件人:Robert colirc...@eventbrite.com 收件人:user@cassandra.apache.orgu...@cassandra.apache.org 发送时间:2015年11月13日(周五) 04:04 主题:Re: Data.db too large and after sstableloader still large On Thu, Nov 12, 2015 at 6:44 AM, qihuang.zheng qihuang.zh...@fraudmetrix.cn wrote: question is : why sstableloader can’t balance data file size? Because it streams ranges from the source SStable to a distributed set of ranges, especially if you are using vnodes. It is a general property of Cassandra's streaming that it results in SStables that are likely different in size than those that result from flush. Why are you preoccupied with the filesizes of files sized in the hundreds of megabytes? Why do you care about this amount of variance in file sized? =Rob
Data.db too large and after sstableloader still large
We do snapshot, and found some Data.db too large: [qihuang.zheng@spark047219 5]$ find . -type f -size +800M -print0 | xargs -0 ls -lh -rw-r--r--. 2 qihuang.zheng users 1.5G 10月 28 14:49 ./forseti/velocity/forseti-velocity-jb-103631-Data.db And sstableloader to new cluster, one node has this large file: [qihuang.zheng@spark047243 velocity]$ ll -rth | grep Data -rw-r--r--. 1 admin admin 46M 11月 12 18:22 forseti-velocity-jb-21-Data.db -rw-r--r--. 1 admin admin 156M 11月 12 18:22 forseti-velocity-jb-22-Data.db -rw-r--r--. 1 admin admin 2.6M 11月 12 18:22 forseti-velocity-jb-23-Data.db -rw-r--r--. 1 admin admin 162M 11月 12 18:22 forseti-velocity-jb-24-Data.db -rw-r--r--. 1 admin admin 1.5G 11月 12 18:22 forseti-velocity-jb-25-Data.db -BigFile Still here Seems sstableloader don’t split file very well. Why sstableloader can’t split to small filter to new cluster? I tried usesstablesplit at snapshot before sstableloader, but this progress is too slow. Tks,qihuang.zheng
回复：Data.db too large and after sstableloader still large
Original snapshot files: [qihuang.zheng@spark047219 226_1105]$ ll 2/forseti/velocity/ -h | grep Data -rw-r--r--. 1 qihuang.zheng users 158M 10月 28 15:03 forseti-velocity-jb-102486-Data.db -rw-r--r--. 1 qihuang.zheng users 161M 10月 28 16:28 forseti-velocity-jb-103911-Data.db -rw-r--r--. 1 qihuang.zheng users 161M 10月 28 14:23 forseti-velocity-jb-103920-Data.db -rw-r--r--. 1 qihuang.zheng users 370M 10月 28 14:10 forseti-velocity-jb-105829-Data.db ⬅️ A Big File ① -rw-r--r--. 1 qihuang.zheng users 161M 10月 28 14:07 forseti-velocity-jb-107113-Data.db -rw-r--r--. 1 qihuang.zheng users 160M 10月 28 15:53 forseti-velocity-jb-73122-Data.db -rw-r--r--. 1 qihuang.zheng users 161M 10月 28 14:46 forseti-velocity-jb-85829-Data.db -rw-r--r--. 1 qihuang.zheng users 161M 10月 28 15:29 forseti-velocity-jb-87661-Data.db -rw-r--r--. 1 qihuang.zheng users 161M 10月 28 15:05 forseti-velocity-jb-93091-Data.db sstable to new cluster [qihuang.zheng@cass047202 ~]$ ./psshA.sh ip_spark.txt 'ls /home/admin/cassandra/data/forseti/velocity -hl |grep Data' Warning: do not enter your password if anyone else has superuser privileges or access to your account. Password: [1] 22:29:43 [SUCCESS] 192.168.47.208 -rw-r--r--. 1 admin admin 365K 11月 12 22:10 forseti-velocity-jb-20-Data.db -rw-r--r--. 1 admin admin 370M 11月 12 22:10 forseti-velocity-jb-21-Data.db ⬅️ File Still Large! and same size as ① -rw-r--r--. 1 admin admin 11M 11月 12 22:10 forseti-velocity-jb-22-Data.db [2] 22:29:43 [SUCCESS] 192.168.47.212 -rw-r--r--. 1 admin admin 146M 11月 12 22:09 forseti-velocity-jb-22-Data.db -rw-r--r--. 1 admin admin 3.7M 11月 12 22:09 forseti-velocity-jb-23-Data.db [3] 22:29:43 [SUCCESS] 192.168.47.215 -rw-r--r--. 1 admin admin 916K 11月 12 22:09 forseti-velocity-jb-14-Data.db [4] 22:29:43 [SUCCESS] 192.168.47.242 ⬅️ Almost Go To This Node! -rw-r--r--. 1 admin admin 106M 11月 12 22:10 forseti-velocity-jb-24-Data.db -rw-r--r--. 1 admin admin 160M 11月 12 22:10 forseti-velocity-jb-25-Data.db -rw-r--r--. 1 admin admin 158M 11月 12 22:10 forseti-velocity-jb-26-Data.db -rw-r--r--. 1 admin admin 160M 11月 12 22:10 forseti-velocity-jb-27-Data.db [5] 22:29:43 [FAILURE] 192.168.47.223 Exited with error code 1 ⬅️ This Node has None Files! [6] 22:29:43 [SUCCESS] 192.168.47.244 -rw-r--r--. 1 admin admin 111M 11月 12 22:09 forseti-velocity-jb-18-Data.db [7] 22:29:43 [SUCCESS] 192.168.47.245 -rw-r--r--. 1 admin admin 50M 11月 12 22:09 forseti-velocity-jb-22-Data.db -rw-r--r--. 1 admin admin 170K 11月 12 22:09 forseti-velocity-jb-23-Data.db [8] 22:29:43 [SUCCESS] 192.168.47.241 -rw-r--r--. 1 admin admin 7.5M 11月 12 22:09 forseti-velocity-jb-30-Data.db [9] 22:29:43 [FAILURE] 192.168.47.218 Exited with error code 1 ⬅️ No Files [10] 22:29:43 [SUCCESS] 192.168.47.243 -rw-r--r--. 1 admin admin 15M 11月 12 22:09 forseti-velocity-jb-29-Data.db [11] 22:29:43 [SUCCESS] 192.168.47.219 -rw-r--r--. 1 admin admin 160M 11月 12 22:09 forseti-velocity-jb-23-Data.db [12] 22:29:43 [SUCCESS] 192.168.47.217 -rw-r--r--. 1 admin admin 30M 11月 12 22:09 forseti-velocity-jb-22-Data.db [13] 22:29:44 [SUCCESS] 192.168.47.216 -rw-r--r--. 1 admin admin 3.5M 11月 12 22:09 forseti-velocity-jb-20-Data.db -rw-r--r--. 1 admin admin 161M 11月 12 22:09 forseti-velocity-jb-21-Data.db We use spark-case-connecot to read table and repartition. Spark repartition job below indicate: If nodes has none data.db like first two nodes, InputSize is 0.0B,and nodes with large files like the last one running too long! My question is : why sstableloader can’t balance data file size? Tks,qihuang.zheng 原始邮件 发件人:qihuang.zhengqihuang.zh...@fraudmetrix.cn 收件人:useru...@cassandra.apache.org 发送时间:2015年11月12日(周四) 21:20 主题:Data.db too large and after sstableloader still large We do snapshot, and found some Data.db too large: [qihuang.zheng@spark047219 5]$ find . -type f -size +800M -print0 | xargs -0 ls -lh -rw-r--r--. 2 qihuang.zheng users 1.5G 10月 28 14:49 ./forseti/velocity/forseti-velocity-jb-103631-Data.db And sstableloader to new cluster, one node has this large file: [qihuang.zheng@spark047243 velocity]$ ll -rth | grep Data -rw-r--r--. 1 admin admin 46M 11月 12 18:22 forseti-velocity-jb-21-Data.db -rw-r--r--. 1 admin admin 156M 11月 12 18:22 forseti-velocity-jb-22-Data.db -rw-r--r--. 1 admin admin 2.6M 11月 12 18:22 forseti-velocity-jb-23-Data.db -rw-r--r--. 1 admin admin 162M 11月 12 18:22 forseti-velocity-jb-24-Data.db -rw-r--r--. 1 admin admin 1.5G 11月 12 18:22 forseti-velocity-jb-25-Data.db -BigFile Still here Seems sstableloader don’t split file very well. Why sstableloader can’t split to small filter to new cluster? I tried usesstablesplit at snapshot before sstableloader, but this progress is too slow. Tks,qihuang.zheng
回复：Can't save Opscenter Dashboard
We have this problem with version 5.2.0. so we decide to update to 5.2.2
But this problem seems remain. We solve this by totally delele relate agent
file and process and restart. just like first time install.
sudo kill -9 `ps -ef|grep datastax_agent_monitor | head -1 |awk '{print $2}'` \
sudo kill -9 `cat /var/run/datastax-agent/datastax-agent.pid` \
sudo rm -rf /var/lib/datastax-agent \
sudo rm -rf /usr/share/datastax-agent
qihuang.zheng
原始邮件
发件人:Kai wangdep...@gmail.com
收件人:useru...@cassandra.apache.org
发送时间:2015年11月5日(周四) 04:39
主题:Can't save Opscenter Dashboard
Hi,
Today after one of the nodes is rebooted, OpsCenter dashboard doesn't save
anymore. It starts with an empty dashboard with no widget or graph. If I add
some graph/widget, they are being updated fine. But if I refresh the browser,
the dashboard became empty again.
Also there's no "DEFAULT" tab on the dashboard as the user guide shows. I am
not sure if it was there before.
Two DC of same cluster migrate to Two different Cluster
Few weeks ago, we move a ks to another DC but in same cluster.
Original: cluster_1: DC1,ks1+ks2
After: cluster_1: DC1,ks1; DC2,ks2
by reference http://www.planetcassandra.org/blog/cassandra-migration-to-ec2,
our steps :
1.in all new Node(DC2) :
$ vi /usr/install/cassandra/conf/cassandra.yaml:
cluster_name: cluster_1
endpoint_snitch: GossipingPropertyFileSnitch
- seeds: "DC1 Nodes,DC2 Node"
auto_bootstrap: false
$ sudo -u admin vi /usr/install/cassandra/conf/cassandra-rackdc.properties
dc=DC2
rack=RAC1
2.add DC2 to ks2
ALTER KEYSPACE ks2 WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy',
'DC2' : 3, 'DC1' : 3};
3.migration data from original DC:
/usr/install/cassandra/bin/nodetool rebuild DC1
4.ALTER KEYSPACE ks2 WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy',
'DC2' : 3};
After that, I think It's OK now. because ks1 and ks2 now had different DC to
separate the data come from.
But recently we had some issue: node in DC1 still invoke DC2.
use iftop command, I can see DC1 node flow to DC2.
I think it's the problem of seed in DC2 still use DC1 node at step1.
Our leader want to totaly seperate ks1 and ks2. so my program is modify DC2
nodes to use another cluster_name.
But just reboot DC2 nodes can’t success start node:
org.apache.cassandra.exceptions.ConfigurationException: Saved cluster name
cluster_1 != configured name cluster_2
and ref from
here:http://stackoverflow.com/questions/22006887/cassandra-saved-cluster-name-test-cluster-configured-name
can start node but has warning:
WARN 16:41:35,824 ClusterName mismatch from /192.168.47.216 cluster_1!=cluster2
in this way also has some problem of nodetool status:
nodetool status at DC1 nodes:
[qihuang.zheng@spark047219 ~]$ /usr/install/cassandra/bin/nodetool status
Datacenter: DC1
===
-- Address LoadTokens Owns Host IDRack
UN 192.168.47.219 62.61 GB 256 7.2% 953dda8c-3908-401f-9adb-aa59c4cb92d1
RAC1
Datacenter: DC2
===
-- Address LoadTokens Owns Host IDRack
DN 192.168.47.223 49.49 GB 256 7.5% a4b91faf-3e1f-46df-a1cc-39bb267bc683
RAC1
nodetool status at DC2 nodes after change cluster_name:
[qihuang.zheng@spark047223 ~]$ /usr/install/cassandra/bin/nodetool status
Datacenter: DC1
===
-- Address LoadTokens Owns Host IDRack
DN 192.168.47.219 ? 256 7.2% 953dda8c-3908-401f-9adb-aa59c4cb92d1 r1
DN 192.168.47.218 ? 256 7.7% 42b8bfae-a6ee-439b-b101-4a0963c9aaa0 r1
...
Datacenter: DC2
===
-- Address LoadTokens Owns Host IDRack
UN 192.168.47.223 49.49 GB 256 7.5% a4b91faf-3e1f-46df-a1cc-39bb267bc683
RAC1
As u can see, DC1 think DC2 is DOWN, and DC2 think DC1 is DN and load is ?. but
actualy all nodes are UP.
So, Here is the problem:
1. If I delete seeds of DC1 node but still in same cluster_name. will DC1 node
iftop still flow to DC2?
2. If I want to change DC2 nodes to new cluster_name, what should I do next?
I thinks first way is easy but may be not fit our leader’s opinion.
TKS,qihuang.zheng
nodetool status Load not same with disk used
We have some nodes Load too large, but some are normal. [qihuang.zheng@cass047221 forseti]$ /usr/install/cassandra/bin/nodetool status -- AddressLoadTokens Owns Host ID Rack UN 192.168.47.221 2.66 TB 256 8.7% 87e100ed-85c4-44cb-9d9f-2d602d016038 RAC1 UN 192.168.47.204 614.58 GB 256 8.2% 91ad3d42-4207-46fe-8188-34c3f0b2dbd2 RAC1 I check the node with df command, and find disk used only 715G. [qihuang.zheng@cass047221 forseti]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda220G 8.6G 11G 47% / tmpfs 16G 0 16G 0% /dev/shm /dev/sda1190M 58M 123M 32% /boot /dev/sda43.5T 715G 2.6T 22% /home and this is a normal node’s disk used: [qihuang.zheng@cass047204 ~]$ df -h Filesystem Size Used Avail Use% Mounted on tmpfs 16G 0 16G 0% /dev/shm /dev/sda1 485M 57M 403M 13% /boot /dev/mapper/VolGroup-lv_home 3.4T 659G 2.6T 21% /home How does nodetool status Load come from? should't It based on sstable file size which also based on disk used? Tks, qihuang.zheng
Re: C* Table Changed and Data Migration with new primary key
I try spark-cassandra-connector. As our data from src table has ttl, and
saveToCassandra default does’t insert ttl .
fortunately we have a timestamp field indicate insert time. but
TTLOption.perRow should based on a column,
not support query this column then do calculation to setup ttl. so before
saveToCassandra,
I map to newly case class which had a ttl field(last one) so that can directory
used in TTLOption.perRow
case class Velocity(attribute: String,partner_code:String,
app_name: String,attr_type:String, timestamp: Long,ttl: Int)
def localTest(tbl : String): Unit = {
val velocitySrcTbl = sc.cassandraTable(“ks", "velocity").filter(row =
(row.getLong("timestamp"): java.lang.Long) != null)
val nowlong = System.currentTimeMillis()
val now = (nowlong/1000).toInt
val velocityRDD = velocitySrcTbl.map(row={
val ts = (row.getLong("timestamp")/1000).toInt
Velocity(
row.getString("attribute"),
row.getString("partner_code"),
row.getString("app_name"),
row.getString("type"),
row.getLong("timestamp"),
90*86400-(now-ts) //calculation ttl and directly used as parameter in
TTLOption.perRow()
)
})
velocityRDD.saveToCassandra("forseti", tbl,
SomeColumns("attribute", "partner_code", "app_name", "type" as "attr_type",
"timestamp"),
writeConf = WriteConf(ttl = TTLOption.perRow("ttl")))
}
But there are something wrong here:
WARN scheduler.TaskSetManager: Lost task 1.3 in stage 16.0 (TID 87,
192.168.6.53): java.lang.NullPointerException: Unexpected null value of column
5. Use get[Option[...]] to receive null values.
I alreay filter column5: timestamp filed not null. But why this exception
happen.
I also try use : getLongOption, but this exception still happen.
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md
at first I want to ask issue on spark-case-connector project, but there are no
issues there, so I ask here.
Tks, qihuang.zheng
原始邮件
发件人:DuyHai doandoanduy...@gmail.com
收件人:useru...@cassandra.apache.org
发送时间:2015年10月22日(周四) 19:50
主题:Re: C* Table Changed and Data Migration with new primary key
Use Spark to distribute the job of copying data all over the cluster and help
accelerating the migration. The Spark connector does auto paging in the
background with the Java Driver
Le 22 oct. 2015 11:03, "qihuang.zheng" qihuang.zh...@fraudmetrix.cn a écrit :
I tried using java driver with auto paging query: setFetchSize instead of token
function. as Cass has this feature already.
ref from
here:http://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0
But I tried in test envrionment with only 1million data read then insert 3
tables, It’s too slow.
After running 20 min, Exception likeNoHostAvailableException happen, offcourse
data did’t sync completed.
And our product env has nearly 25 billion data. which is unacceptble for this
case. It’s there other ways?
Thanks Regards，
qihuang.zheng
原始邮件
发件人:Jeff jirsajeff.ji...@crowdstrike.com
收件人:user@cassandra.apache.orgu...@cassandra.apache.org
发送时间:2015年10月22日(周四) 13:52
主题:Re: C* Table Changed and Data Migration with new primary key
Because the data format has changed, you’ll need to read it out and write it
back in again.
This means using either a driver (java, python, c++, etc), or something like
spark.
In either case, split up the token range so you can parallelize it for
significant speed improvements.
From: "qihuang.zheng"
Reply-To: "user@cassandra.apache.org"
Date: Wednesday, October 21, 2015 at 6:18 PM
To: user
Subject: C* Table Changed and Data Migration with new primary key
Hi All:
We have a table defined only one partition key and some cluster key.
CREATE TABLE test1 (
attribute text,
partner text,
app text,
"timestamp" bigint,
event text,
PRIMARY KEY ((attribute), partner, app, "timestamp")
)
And now we want to split original test1 table to 3 tables like this:
test_global : PRIMARY KEY ((attribute),“timestamp")
test_partner: PRIMARY KEY ((attribute, partner), "timestamp”)
test_app:PRIMARY KEY ((attribute, partner, app), “timestamp”)
Why we split original table because when queryglobal databy timestamp desc like
this:
select * from test1 where attribute=? order by timestamp desc
is not support in Cass. As class order by support should use all clustering key.
But sql like this:
select * from test1 where attribute=? order by partner desc,app desc, timestamp
desc
can’t query the right global data by ts desc.
After Split table we could do globa data query right: select * from test_global
where attribute=? order by timestamp desc.
Now we have a problem ofdata migration.
As I Know,sstableloaderis the most easy way,but could’t deal with different
table name. (Am I right?)
Andcpcmd in cqlsh can’t fit
Find partition row of Compacted partition maximum bytes
I use nodetool cfstats to see table’s status, and findCompacted partition maximum bytes: 190G. Is there anyway to find this largest wide partition row? [qihuang.zheng@cass047202 cassandra]$ nodetool cfstats forseti.velocity Keyspace: forseti Read Count: 10470099 Read Latency: 1.3186399419909973 ms. Write Count: 146970362 Write Latency: 0.06062576270989929 ms. Pending Tasks: 0 Table: velocity SSTable count: 2144 SSTables in each level: [1, 10, 96, 723, 1314, 0, 0, 0, 0] Space used (live), bytes: 509031385679 Space used (total), bytes: 523815500936 Off heap memory used (total), bytes: 558210701 SSTable Compression Ratio: 0.23635049381008288 Number of keys (estimate): 269787648 Memtable cell count: 271431 Memtable data size, bytes: 141953019 Memtable switch count: 1713 Local read count: 10470099 Local read latency: 1.266 ms Local write count: 146970371 Local write latency: 0.053 ms Pending tasks: 0 Bloom filter false positives: 534721 Bloom filter false ratio: 0.13542 Bloom filter space used, bytes: 180529808 Bloom filter off heap memory used, bytes: 180512656 Index summary off heap memory used, bytes: 118613037 Compression metadata off heap memory used, bytes: 259085008 Compacted partition minimum bytes: 104 Compacted partition maximum bytes: 190420296972 Compacted partition mean bytes: 8656 Average live cells per slice (last five minutes): 0.0 Average tombstones per slice (last five minutes): 0.0 qihuang.zheng
decommission too slow
Recently we want to delete some c* nodes for data migration. C* verision:2.0.15 we use nodetooldecommission with nohup: nohup nodetool decommission -h xxx After execute 3 days already, seems this process did’t finished yet! This decommissioning Node data isnearly 400G. 1. I check jps -lm, and NodeCmd decommission process still there. 2. and nohup.out file is empty always. 3. I check opscenter, the node status is Leaving: 1 running task. 4. the running task is Compaction, I use nodetool stop compaction, but after sometime later compaction happend again. 5. thenodetool netstats show it’s leaving: Mode: LEAVING Unbootstrap 928e6be0-7950-11e5-9cfb-910d8a1425c3 …. Read Repair Statistics: Attempted: 4746100 Mismatch (Blocking): 950 Mismatch (Background): 100746 Pool Name Active Pending Completed Commandsn/a 0 1275402208 Responsesn/a 0 1034430957 I don’t know when decommission will finished. Or does something wrong inside? just 400G data takes 3 days(and still unfinished) seems abnormal. Tks, qihuang.zheng
Re: C* Table Changed and Data Migration with new primary key
Tks Doan! we would try spark. as our online already has spark1.4.1. but our C* version is 2.0.15, As spark1.4 need 2.1.5, when I query size get error “Failed to fetch size estimates for system.size_estimates”. but that’s not a serious problem. I tried use spark1.4.1 and cass2.01.5 on test env, It’s really fast than just use java driver api. But we may meet some problem on producet env, as our spark node deploy totally different with cassandra nodes. qihuang.zheng 原始邮件 发件人:DuyHai doandoanduy...@gmail.com 收件人:useru...@cassandra.apache.org 发送时间:2015年10月22日(周四) 19:50 主题:Re: C* Table Changed and Data Migration with new primary key Use Spark to distribute the job of copying data all over the cluster and help accelerating the migration. The Spark connector does auto paging in the background with the Java Driver Le 22 oct. 2015 11:03, "qihuang.zheng" qihuang.zh...@fraudmetrix.cn a écrit : I tried using java driver with auto paging query: setFetchSize instead of token function. as Cass has this feature already. ref from here:http://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0 But I tried in test envrionment with only 1million data read then insert 3 tables, It’s too slow. After running 20 min, Exception likeNoHostAvailableException happen, offcourse data did’t sync completed. And our product env has nearly 25 billion data. which is unacceptble for this case. It’s there other ways? Thanks Regards， qihuang.zheng 原始邮件 发件人:Jeff jirsajeff.ji...@crowdstrike.com 收件人:user@cassandra.apache.orgu...@cassandra.apache.org 发送时间:2015年10月22日(周四) 13:52 主题:Re: C* Table Changed and Data Migration with new primary key Because the data format has changed, you’ll need to read it out and write it back in again. This means using either a driver (java, python, c++, etc), or something like spark. In either case, split up the token range so you can parallelize it for significant speed improvements. From: "qihuang.zheng" Reply-To: "user@cassandra.apache.org" Date: Wednesday, October 21, 2015 at 6:18 PM To: user Subject: C* Table Changed and Data Migration with new primary key Hi All: We have a table defined only one partition key and some cluster key. CREATE TABLE test1 ( attribute text, partner text, app text, "timestamp" bigint, event text, PRIMARY KEY ((attribute), partner, app, "timestamp") ) And now we want to split original test1 table to 3 tables like this: test_global : PRIMARY KEY ((attribute),“timestamp") test_partner: PRIMARY KEY ((attribute, partner), "timestamp”) test_app:PRIMARY KEY ((attribute, partner, app), “timestamp”) Why we split original table because when queryglobal databy timestamp desc like this: select * from test1 where attribute=? order by timestamp desc is not support in Cass. As class order by support should use all clustering key. But sql like this: select * from test1 where attribute=? order by partner desc,app desc, timestamp desc can’t query the right global data by ts desc. After Split table we could do globa data query right: select * from test_global where attribute=? order by timestamp desc. Now we have a problem ofdata migration. As I Know,sstableloaderis the most easy way,but could’t deal with different table name. (Am I right?) Andcpcmd in cqlsh can’t fit our situation because our data is two large. (10Nodes, one nodes has 400G data) I alos try JavaAPI by query the origin table and then insert into 3 different splited table.But seems too slow Any Solution aboult quick data migration? TKS!! PS: Cass version: 2.0.15 Thanks Regards， qihuang.zheng
回复：Can consistency-levels be different for "read" and "write" inDatastax Java-Driver?
Using java driver, both read and write will transfer to Statement. We use
PrepareStatement like this:
public PreparedStatement getPrepareSTMT(String cql){
PreparedStatement statement = session.prepare(cql);
statement.setConsistencyLevel(ConsistencyLevel.ONE).setRetryPolicy(FallthroughRetryPolicy.INSTANCE);
return statement;
}
So that you can set ConsistencyLevel differently for read and write.
Tks, qihuang.zheng
原始邮件
发件人:Ajay gargajaygargn...@gmail.com
收件人:useru...@cassandra.apache.org
发送时间:2015年10月27日(周二) 02:17
主题:Can consistency-levels be different for "read" and "write" inDatastax
Java-Driver?
Right now, I have setup "LOCAL QUORUM" as the consistency level in the driver,
but it seems that "SERIAL" is being used during writes, and I consistently get
this error of type ::
Cassandra timeout during write query at consistency SERIAL (3 replica were
required but only 0 acknowledged the write)
Am I missing something?
--
Regards,
Ajay
回复：StatusLogger output help
I just want to know when StatusLogger will happen, find this question: http://qnalist.com/questions/4783598/help-on-statuslogger-output but no one reply it. seems this question ask Mar 2014. no one notice then. I just pull it out, hope someone answer this. TKS. qihuang.zheng 原始邮件 发件人:qihuang.zhengqihuang.zh...@fraudmetrix.cn 收件人:useru...@cassandra.apache.org 发送时间:2015年10月25日(周日) 15:19 主题:StatusLogger output help After dig system.log. I notice Every GC for ParNew or GC for CMS will produce StatusLogger. But I’m not really sure is GC cause it. Is anyone summary this appearance? INFO [ScheduledTasks:1] 2015-10-21 20:57:54,568 GCInspector.java (line 116) GC for ParNew: 1165 ms for 1 collections, 8289178440 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 20:57:54,621 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 20:57:54,622 StatusLogger.java (line 70) ReadStage 521 613058 0 0 …. INFO [ScheduledTasks:1] 2015-10-21 21:03:19,142 GCInspector.java (line 116) GC for ParNew: 267 ms for 1 collections, 9969463696 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:03:56,802 GCInspector.java (line 116) GC for ParNew: 1549 ms for 1 collections, 10679738240 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:03:56,803 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 21:03:56,804 StatusLogger.java (line 70) ReadStage 3 9 628954 0 0 …. INFO [ScheduledTasks:1] 2015-10-21 21:09:42,724 GCInspector.java (line 116) GC for ParNew: 1420 ms for 1 collections, 5274947408 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:09:42,725 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 21:09:42,725 StatusLogger.java (line 70) ReadStage 7 7 644911 0 0 qihuang.zheng
回复：StatusLogger output help
for ensure after GC , there will print StatusLogger, I found this code in GCInspector.java: if (durationPerCollection MIN_DURATION_TPSTATS) StatusLogger.log(); the defaultMIN_DURATION_TPSTATS=1second. so if GC duration more than 1s, there will be StatusLogger! But I check StatusLogger, I could’t figure out what happen inside in C at that moment. which status msg should I care on StatusLogger’s print msg?? Does Pending count important? or should I care aboutMemtable ops,data qihuang.zheng 原始邮件 发件人:qihuang.zhengqihuang.zh...@fraudmetrix.cn 收件人:useru...@cassandra.apache.org 发送时间:2015年10月25日(周日) 15:27 主题:回复：StatusLogger output help I just want to know when StatusLogger will happen, find this question: http://qnalist.com/questions/4783598/help-on-statuslogger-output but no one reply it. seems this question ask Mar 2014. no one notice then. I just pull it out, hope someone answer this. TKS. qihuang.zheng 原始邮件 发件人:qihuang.zhengqihuang.zh...@fraudmetrix.cn 收件人:useru...@cassandra.apache.org 发送时间:2015年10月25日(周日) 15:19 主题:StatusLogger output help After dig system.log. I notice Every GC for ParNew or GC for CMS will produce StatusLogger. But I’m not really sure is GC cause it. Is anyone summary this appearance? INFO [ScheduledTasks:1] 2015-10-21 20:57:54,568 GCInspector.java (line 116) GC for ParNew: 1165 ms for 1 collections, 8289178440 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 20:57:54,621 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 20:57:54,622 StatusLogger.java (line 70) ReadStage 521 613058 0 0 …. INFO [ScheduledTasks:1] 2015-10-21 21:03:19,142 GCInspector.java (line 116) GC for ParNew: 267 ms for 1 collections, 9969463696 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:03:56,802 GCInspector.java (line 116) GC for ParNew: 1549 ms for 1 collections, 10679738240 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:03:56,803 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 21:03:56,804 StatusLogger.java (line 70) ReadStage 3 9 628954 0 0 …. INFO [ScheduledTasks:1] 2015-10-21 21:09:42,724 GCInspector.java (line 116) GC for ParNew: 1420 ms for 1 collections, 5274947408 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:09:42,725 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 21:09:42,725 StatusLogger.java (line 70) ReadStage 7 7 644911 0 0 qihuang.zheng
StatusLogger output help
After dig system.log. I notice Every GC for ParNew or GC for CMS will produce StatusLogger. But I’m not really sure is GC cause it. Is anyone summary this appearance? INFO [ScheduledTasks:1] 2015-10-21 20:57:54,568 GCInspector.java (line 116) GC for ParNew: 1165 ms for 1 collections, 8289178440 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 20:57:54,621 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 20:57:54,622 StatusLogger.java (line 70) ReadStage 521 613058 0 0 …. INFO [ScheduledTasks:1] 2015-10-21 21:03:19,142 GCInspector.java (line 116) GC for ParNew: 267 ms for 1 collections, 9969463696 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:03:56,802 GCInspector.java (line 116) GC for ParNew: 1549 ms for 1 collections, 10679738240 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:03:56,803 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 21:03:56,804 StatusLogger.java (line 70) ReadStage 3 9 628954 0 0 …. INFO [ScheduledTasks:1] 2015-10-21 21:09:42,724 GCInspector.java (line 116) GC for ParNew: 1420 ms for 1 collections, 5274947408 used; max is 16750411776 INFO [ScheduledTasks:1] 2015-10-21 21:09:42,725 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked INFO [ScheduledTasks:1] 2015-10-21 21:09:42,725 StatusLogger.java (line 70) ReadStage 7 7 644911 0 0 qihuang.zheng
Understanding C* -XX:MaxTenuringThreshold=1
As Cassandra default-XX:MaxTenuringThreshold=1, which means: first time YGC, Eden's live object copy to Survivor(S0), those survivor object's age counter=1. Then next time YGC, Eden live object and S0 will copy to S1. and those still live object from S0 age counter=2 which large than MaxTenuringThreshold=1, then they will promot to Old Gen. from this, I get this conclusion: 1. After YGC, Eden live object copy toto survivor(age=1),from survivor(age=2)first copy toto survivorthen transfer to old gen. 2. Old Gen's increment size after this YGC will not large than latestfrom survivor's size. 3. Survivor's object age counter aways be 1. as those counter=2 promot to Old, then disappear from survivor. Plz tell me it’s right. TKS. qihuang.zheng
Re: C* Table Changed and Data Migration with new primary key
I tried using java driver with auto paging query: setFetchSize instead of token function. as Cass has this feature already. ref from here:http://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0 But I tried in test envrionment with only 1million data read then insert 3 tables, It’s too slow. After running 20 min, Exception likeNoHostAvailableException happen, offcourse data did’t sync completed. And our product env has nearly 25 billion data. which is unacceptble for this case. It’s there other ways? Thanks Regards， qihuang.zheng 原始邮件 发件人:Jeff jirsajeff.ji...@crowdstrike.com 收件人:user@cassandra.apache.orgu...@cassandra.apache.org 发送时间:2015年10月22日(周四) 13:52 主题:Re: C* Table Changed and Data Migration with new primary key Because the data format has changed, you’ll need to read it out and write it back in again. This means using either a driver (java, python, c++, etc), or something like spark. In either case, split up the token range so you can parallelize it for significant speed improvements. From: "qihuang.zheng" Reply-To: "user@cassandra.apache.org" Date: Wednesday, October 21, 2015 at 6:18 PM To: user Subject: C* Table Changed and Data Migration with new primary key Hi All: We have a table defined only one partition key and some cluster key. CREATE TABLE test1 ( attribute text, partner text, app text, "timestamp" bigint, event text, PRIMARY KEY ((attribute), partner, app, "timestamp") ) And now we want to split original test1 table to 3 tables like this: test_global : PRIMARY KEY ((attribute),“timestamp") test_partner: PRIMARY KEY ((attribute, partner), "timestamp”) test_app:PRIMARY KEY ((attribute, partner, app), “timestamp”) Why we split original table because when queryglobal databy timestamp desc like this: select * from test1 where attribute=? order by timestamp desc is not support in Cass. As class order by support should use all clustering key. But sql like this: select * from test1 where attribute=? order by partner desc,app desc, timestamp desc can’t query the right global data by ts desc. After Split table we could do globa data query right: select * from test_global where attribute=? order by timestamp desc. Now we have a problem ofdata migration. As I Know,sstableloaderis the most easy way,but could’t deal with different table name. (Am I right?) Andcpcmd in cqlsh can’t fit our situation because our data is two large. (10Nodes, one nodes has 400G data) I alos try JavaAPI by query the origin table and then insert into 3 different splited table.But seems too slow Any Solution aboult quick data migration? TKS!! PS: Cass version: 2.0.15 Thanks Regards， qihuang.zheng
C* Table Changed and Data Migration with new primary key
Hi All: We have a table defined only one partition key and some cluster key. CREATE TABLE test1 ( attribute text, partner text, app text, "timestamp" bigint, event text, PRIMARY KEY ((attribute), partner, app, "timestamp") ) And now we want to split original test1 table to 3 tables like this: test_global : PRIMARY KEY ((attribute),“timestamp") test_partner: PRIMARY KEY ((attribute, partner), "timestamp”) test_app:PRIMARY KEY ((attribute, partner, app), “timestamp”) Why we split original table because when queryglobal databy timestamp desc like this: select * from test1 where attribute=? order by timestamp desc is not support in Cass. As class order by support should use all clustering key. But sql like this: select * from test1 where attribute=? order by partner desc,app desc, timestamp desc can’t query the right global data by ts desc. After Split table we could do globa data query right: select * from test_global where attribute=? order by timestamp desc. Now we have a problem ofdata migration. As I Know,sstableloaderis the most easy way,but could’t deal with different table name. (Am I right?) Andcpcmd in cqlsh can’t fit our situation because our data is two large. (10Nodes, one nodes has 400G data) I alos try JavaAPI by query the origin table and then insert into 3 different splited table.But seems too slow Any Solution aboult quick data migration? TKS!! PS: Cass version: 2.0.15 Thanks Regards， qihuang.zheng
