Re: setting PIG_INPUT_INITIAL_ADDRESS environment . variable in Oozie for cassandra ...¿?
¡Eureka! At last !!! the trick?? As well as putting the jar libraries dependecies in the sharelib folder in hdfs ... I had define this and the other environment variables in the bash_profile and works fine if I launch pig scripts from command line shell. I have to define these variables: PIG_INITIAL_ADDRESS, PIG_CONF_DIR, PIG_RPC_PORT, PIG_PARTITIONER . in the hadoop_env.sh and works fine. The others exceptions were like null exceptions in cassandra's class, etc Now, It is working with pig 0.10 ... Run pig script using PigRunner.run() for Pig version 0.8+ Apache Pig version 0.10.0 (r1328203) ... Although I configue to use pig 0.11 and put this pig jar version in the oozie shrelib, pom , classpath, etc and , I can not run pig script by shell actions in oozie, but this another song!!! Regards 2013/12/12 Aaron Morton aa...@thelastpickle.com Caused by: java.io.IOException: PIG_INPUT_INITIAL_ADDRESS or PIG_INITIAL_ADDRESS environment variable not set at org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(CassandraStorage.java:314) at org.apache.cassandra.hadoop.pig.CassandraStorage.getSchema(CassandraStorage.java:358) at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151) ... 35 more Have you checked these are set ? Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 11/12/2013, at 4:00 am, Miguel Angel Martin junquera mianmarjun.mailingl...@gmail.com wrote: Hi, I have an error with pig action in oozie 4.0.0 using cassandraStorage. (cassandra 1.2.10) I can run pig scripts right with cassandra. but whe I try to use cassandraStorage to load data I have this error: Run pig script using PigRunner.run() for Pig version 0.8+ Apache Pig version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25 Run pig script using PigRunner.run() for Pig version 0.8+ 2013-12-10 12:24:39,084 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25 2013-12-10 12:24:39,084 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25 2013-12-10 12:24:39,095 [main] INFO org.apache.pig.Main - Logging error messages to: /tmp/hadoop-ec2-user/mapred/local/taskTracker/ec2-user/jobcache/job_201312100858_0007/attempt_201312100858_0007_m_00_0/work/pig-job_201312100858_0007.log 2013-12-10 12:24:39,095 [main] INFO org.apache.pig.Main - Logging error messages to: /tmp/hadoop-ec2-user/mapred/local/taskTracker/ec2-user/jobcache/job_201312100858_0007/attempt_201312100858_0007_m_00_0/work/pig-job_201312100858_0007.log 2013-12-10 12:24:39,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.228.243.18:9000 2013-12-10 12:24:39,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.228.243.18:9000 2013-12-10 12:24:39,510 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.228.243.18:9001 2013-12-10 12:24:39,510 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.228.243.18:9001 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2245: file testCassandra.pig, line 7, column 7 Cannot get schema from loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2245: file testCassandra.pig, line 7, column 7 Cannot get schema from loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage 2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245: file testCassandra.pig, line 7, column 7 Cannot get schema from loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:155) at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110) at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68) at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60) at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77) at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) at org.apache.pig.PigServer$Graph.compile(PigServer.java:1617) at
One big table/cf or many small ones?
Hello, I have two questions regarding modelling a schema optimized for performance for Cassandra. My service does have users who can add a message to a list. The list of message is sorted by date and displayed. When a user changes the message the date is changed and the message moves to the top of the list. At least that is how it suppose to be ;-) With cql i need to define 'date' column as clustered key to be able to sort on date. But by doing this i can't update the value in the 'date' column with a UPDATE query, cause partition and cluster keys can't be updated with cql. A possible solution is to remove row and insert it again, but i suspect this might be not the best solution. Is there a alternative solution? My second question is regarding the number of tables/column families in a keyspace. I can create a table which contains all messages from all users. But i can also create one table for every user which has a name like: messages_[userid], where [userid] is the id of the user. Or i can shard: messages_a (contains messages from user where name starts with a), messages_b (contains messages from user where name starts with b) My users count is around 100.000. And the messages per user are approx around 20.000. What would be the choice: put everything in 1 big table or go with the many small tables option. Thank you for offering the helping hand. Kind regards, Tinus
Unable to create collection inside collection
Hi, I am not able to create collection inside another collection in Cassandra. Please find screenshot below [cid:image001.png@01CEF750.4A9B5020] In the above screenshot, I am trying to create column named feeschedule with type Map and Map have values which is of type List. Could anybody suggest me how do I achieve it in Cassandra. My Cassandra version details are given below: cqlsh version- cqlsh 4.1.0 Cassandra version - 2.0.2 Thanks in advance, Regards Santosh Shet Software Engineer | VistaOne Solutions Direct India : +91 80 30273829 | Mobile India : +91 8105720582 Skype : santushet inline: image001.png
Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
Hi, after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing high load on both new nodes. After doing some investigation we found out the following: - High cpu load on vm5+6 - Higher data load on vm5+6 - Write requests are evenly distributed to all 6 nodes by our client application (opscenter - metrics - WriteRequests) - Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s) - Nodetool output: UN vm1 9.51 GB256 20,7% 13fa7bb7-19cb-44f5-af83-71a72e04993a X1 UN vm2 9.41 GB256 20,0% b71c2d3d-4721-4dde-a418-802f1af4b7a1 D1 UN vm3 9.37 GB256 18,9% 8ce4c419-d79c-4ef1-b3fd-8936bff3e44f X1 UN vm4 9.23 GB256 19,5% 17974f20-5756-4eba-a377-52feed3a1b10 D1 UN vm5 15.95 GB 256 10,7% 0c6db9ea-4c60-43f6-a12e-51a7d76f8e80 X1 UN vm6 14.86 GB 256 10,2% f64d1909-dd96-442b-b602-efee29eee0a0 D1 Although the ownership is lower on vm5-6 (which already is not right) the data load is way higher. Some cluster facts: Node: 4 CPU, 6 GB RAM, virtual appliance Cassandra: 3 GB Heap, vnodes 256 Schema: Replication strategy network, RF:2 Has anyone an idea what could be the cause for the unbalancing. Maybe we forgot necessary actions during or after cluster expanding process. We are open for every idea. Regards Andi
Re: user / password authentication advice
You could use CassandraAuthorizer and PaaswordAuthenticator which ships
with Cassandra. See this article[1] for a good overview.
[1]
http://www.datastax.com/dev/blog/a-quick-tour-of-internal-authentication-and-authorization-security-in-datastax-enterprise-and-apache-cassandra
On Thursday, December 12, 2013, onlinespending wrote:
OK, thanks for getting me going in the right direction. I imagine most
people would store password and tokenized authentication information in a
single table, using the username (e.g. email address) as the key?
On Dec 11, 2013, at 10:44 PM, Janne Jalkanen
janne.jalka...@ecyrd.comjavascript:_e({}, 'cvml',
'janne.jalka...@ecyrd.com');
wrote:
Hi!
You're right, this isn't really Cassandra-specific. Most languages/web
frameworks have their own way of doing user authentication, and then you
just typically write a plugin that just stores whatever data the system
needs in Cassandra.
For example, if you're using Java (or Scala or Groovy or anything else
JVM-based), Apache Shiro is a good way of doing user authentication and
authorization. http://shiro.apache.org/. Just implement a custom Realm
for Cassandra and you should be set.
/Janne
On Dec 12, 2013, at 05:31 , onlinespending
onlinespend...@gmail.comjavascript:_e({}, 'cvml',
'onlinespend...@gmail.com');
wrote:
Hi,
I’m using Cassandra in an environment where many users can login to use an
application I’m developing. I’m curious if anyone has any advice or links
to documentation / blogs where it discusses common implementations or best
practices for user and password authentication. My cursory search online
didn’t bring much up on the subject. I suppose the information needn’t even
be specific to Cassandra.
I imagine a few basic steps will be as follows:
- user types in username (e.g. email address) and password
- this is verified against a table storing username and passwords
(encrypted in some way)
- a token is return to the app / web browser to allow further
transactions using secure token (e.g. cookie)
Obviously I’m only scratching the surface and it’s the detail and best
practices of implementing this user / password authentication that I’m
curious about.
Thank you,
Ben
--
- John
Re: Write performance with 1.2.12
On Wed, Dec 11, 2013 at 10:49 PM, Aaron Morton aa...@thelastpickle.comwrote: It is the write latency, read latency is ok. Interestingly the latency is low when there is one node. When I join other nodes the latency drops about 1/3. To be specific, when I start sending traffic to the other nodes the latency for all the nodes increases, if I stop traffic to other nodes the latency drops again, I checked, this is not node specific it happens to any node. Is this the local write latency or the cluster wide write request latency ? This is a cluster wide write latency. What sort of numbers are you seeing ? I have a custom application that writes data to the cassandra node, so the numbers might be different than the standard stress test but it should be good enough for comparison. With the previous release 1.0.12 I was getting around 10K requests/ sec and with 1.2.12 I am getting around 6K requests/ sec. Everything else is the same. This is a three node cluster. With a single node I get 3K for cassandra 1.0.12 and 1.2.12. So I suspect there is some network chatter. I have started looking at the sources, hoping to find something. -sandeep Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/12/2013, at 3:39 pm, srmore comom...@gmail.com wrote: Thanks Aaron On Wed, Dec 11, 2013 at 8:15 PM, Aaron Morton aa...@thelastpickle.comwrote: Changed memtable_total_space_in_mb to 1024 still no luck. Reducing memtable_total_space_in_mb will increase the frequency of flushing to disk, which will create more for compaction to do and result in increased IO. You should return it to the default. You are right, had to revert it back to default. when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. What are you measuring, request latency or local read/write latency ? If it’s write latency it’s probably GC, if it’s read is probably IO or data model. It is the write latency, read latency is ok. Interestingly the latency is low when there is one node. When I join other nodes the latency drops about 1/3. To be specific, when I start sending traffic to the other nodes the latency for all the nodes increases, if I stop traffic to other nodes the latency drops again, I checked, this is not node specific it happens to any node. I don't see any GC activity in logs. Tried to control the compaction by reducing the number of threads, did not help much. Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 7/12/2013, at 8:05 am, srmore comom...@gmail.com wrote: Changed memtable_total_space_in_mb to 1024 still no luck. On Fri, Dec 6, 2013 at 11:05 AM, Vicky Kak vicky@gmail.com wrote: Can you set the memtable_total_space_in_mb value, it is defaulting to 1/3 which is 8/3 ~ 2.6 gb in capacity http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management The flushing of 2.6 gb to the disk might slow the performance if frequently called, may be you have lots of write operations going on. On Fri, Dec 6, 2013 at 10:06 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.comwrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with
Re: Raid Issue on EC2 Datastax ami, 1.2.11
Hi Aaron, As you can see in the picture, there is not much steal on iostat. That's the same with top. https://imageshack.com/i/0jm4jyp Philippe 2013/12/10 Aaron Morton aa...@thelastpickle.com Thanks for the update Philip, other people have reported high await on a single volume previously but I don’t think it’s been blamed on noisy neighbours. It’s interesting that you can have noisy neighbours for IO only. Out of interest was there much steal reported in top or iostat ? Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 6/12/2013, at 4:42 am, Philippe Dupont pdup...@teads.tv wrote: Hi again, I have much more in formations on this case : We did further investigations on the nodes affected and did find some await problems on one of the 4 disk in raid: http://imageshack.com/a/img824/2391/s7q3.jpg Here was the iostat of the node : http://imageshack.us/a/img7/7282/qq3w.pnghttp://www.google.com/url?q=http%3A%2F%2Fimageshack.us%2Fa%2Fimg7%2F7282%2Fqq3w.pngsa=Dsntz=1usg=AFQjCNGTu2l8P6sedK0Wc9lhoI6_3O3ixw You can see that the write and read throughput are exactly the same on the 4 disks of the instance. So the raid0 looks good enough. Yet, the global await, r_await and w_await are 3 to 5 times bigger on xvde disk than in other disks. We reported this to amazon support, and there is their answer : Hello, I deeply apologize for any inconvenience this has been causing you and thank you for the additional information and screenshots. Using the instance you based your iostat on (i-), I have looked into the underlying hardware it is currently using and I can see it appears to have a noisy neighbor leading to the higher await time on that particular device. Since most AWS services are multi-tenant, situations can arise where one customer's resource has the potential to impact the performance of a different customer's resource that reside on the same underlying hardware (a noisy neighbor). While these occurrences are rare, they are nonetheless inconvenient and I am very sorry for any impact it has created. I have also looked into the initial instance referred to when the case was created (i-xxx) and cannot see any existing issues (neighboring or otherwise) as to any I/O performance impacts; however, at the time the case was created, evidence on our end suggests there was a noisy neighbor then as well. Can you verify if you are still experiencing above average await times on this instance? If you would like to mitigate the impact of encountering noisy neighbors, you can look into our Dedicated Instance option; Dedicated Instances launch on hardware dedicated to only a single customer (though this can feasibly lead to a situation where a customer is their own noisy neighbor). However, this is an option available only to instances that are being launched into a VPC and may require modification of the architecture of your use-case. I understand the instances belonging to your cluster in question have been launched into EC2-Classic, I just wanted to bring this your attention as a possible solution. You can read more about Dedicated Instances here: http://aws.amazon.com/dedicated-instances/ Again, I am very sorry for the performance impact you have been experiencing due to having noisy neighbors. We understand the frustration and are always actively working to increase capacity so the effects of noisy neighbors is lessened. I hope this information has been useful and if you have any additional questions whatsoever, please do not hesitate to ask! To conclude, the only other solution to avoid VPC and Reserved Instance is to replace this instance by a new one, hoping to not having other Noisy neighbors... I hope that will help someone. Philippe 2013/11/28 Philippe DUPONT pdup...@teads.tv Hi, We have a Cassandra cluster of 28 nodes. Each one is an EC2 m1.xLarge based on datastax AMI with 4 storage in raid0 mode. Here is the ticket we opened with amazon support : This raid is created using the datastax public AMI : ami-b2212dc6. Sources are also available here : https://github.com/riptano/ComboAMI As you can see in the screenshot attached ( http://imageshack.com/a/img854/4592/xbqc.jpg) randomly but frequently one of the storage get fully used (100%) but 3 others are standing in low use. Because of this, the node becomes slow and the whole cassandra cluster is impacted. We are losing data due to writes fails and availability for our customers. it was in this state for one hour, and we decided to restart it. We already removed 3 other instances because of this same issue. (see other screenshots) http://imageshack.com/a/img824/2391/s7q3.jpg http://imageshack.com/a/img10/556/zzk8.jpg Amazon support took a close look at the instance as well as it's underlying hardware for any potential health issues and both
Re: 2 nodes cassandra cluster raid10 or JBOD
Thanks! On Wed, Dec 11, 2013 at 9:37 PM, Aaron Morton aa...@thelastpickle.comwrote: If you have two nodes, and RF 2, you will only be able to use eventual consistency. If you want to have stronger consistency and some redundancy 3 nodes is the minimum requirement. In the current setup, with only 2 nodes, I would use RAID 10 as it requires less operator intervention and there is a chance of data loss with RF 2. If a write (including the hints) is only committed to one node and the disk on that node fails the write will be lost. Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 11/12/2013, at 9:33 pm, Veysel Taşçıoğlu veysel.tascio...@gmail.com wrote: Hi, What about using JBOD and replication factor 2? Regards. On 11 Dec 2013 02:03, cem cayiro...@gmail.com wrote: Hi all, I need to setup 2 nodes Cassandra cluster. I know that Datastax recommends using JBOD as a disk configuration and have replication for the redundancy. I was planning to use RAID 10 but using JBOD can save 50% disk space and increase the performance . But I am not sure I should use JBOD with 2 nodes cluster since there is a higher chance to lose 50% of our cluster compare to a larger cluster. I may prefer to have stronger nodes if I have limited number of nodes. What do you think about that? Is there anyone who has 2 nodes cluster? Best Regards, Cem
Cassandra pytho pagination
Hey Folks,
I need some ideas about support implementing of pagination on the browser,
from the backend. So python code (backend) gets request from frontend with
page=1,2,3,4 and so on and count_per_page=50.
I am trying to use xget with column_count and buffer_size parameters. Can
someone explain me, how does it work? From doc, my understanding is that, I
can do something like,
total_cols is total columns for that key.
count is what user sends me.
.*xget*('Twitter_search', hh, column_count=total_cols, buffer_size=count):
Is my understanding correct? because its not working for page 2 and so on?
Please enlighten me with suggestions.
Thanks.
Re: Write performance with 1.2.12
Why did you switch to RandomPartitioner away from Murmur3Partitioner? Have you tried with Murmur3? 1. # partitioner: org.apache.cassandra.dht.Murmur3Partitioner 2. partitioner: org.apache.cassandra.dht.RandomPartitioner On Fri, Dec 6, 2013 at 10:36 AM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Re: Write performance with 1.2.12
On Thu, Dec 12, 2013 at 11:15 AM, J. Ryan Earl o...@jryanearl.us wrote: Why did you switch to RandomPartitioner away from Murmur3Partitioner? Have you tried with Murmur3? 1. # partitioner: org.apache.cassandra.dht.Murmur3Partitioner 2. partitioner: org.apache.cassandra.dht.RandomPartitioner Since I am comparing between the two versions I am keeping all the settings same. I see Murmur3Partitioner has some performance improvement but then switching back to RandomPartitioner should not cause performance to tank, right ? or am I missing something ? Also, is there an easier way to update the data from RandomPartitioner to Murmur3 ? (upgradesstable ?) On Fri, Dec 6, 2013 at 10:36 AM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
Migrate data to new cluster using datacenters?
Hello, We are in the process of isolating multiple applications currently running in one large cassandra cluster to individual smaller clusters. Each application runs in its own keyspace. In order to reduce/eliminate downtime for a migration, I was curious if anyone had attempted the following process to migrate data to a new cluster: 1) Add new cluster nodes as a new datacenter to existing cluster 2) Set RF for specific keyspace to non-zero for new cluster, use nodetool rebuild on new nodes to stream data 3) Change application node connections to point to new cluster 4) Set RF to 0 for original cluster (stop new writes from going to original cluster) 5) Break connection between nodes so new nodes become a standalone cluster??? - Is this possible? what would be the high level steps? If this is an extremely bad or misinformed idea, I would like to know that as well! I am aware of other tools available including sstableloader, etc, but this seemed like a more elegant solution, leveraging cassandra's active-active features. Thanks, -Andrew NISC
Re: Migrate data to new cluster using datacenters?
Hi, We did it once and it worked well. Those two links should help (this is more or less what we've done) : http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/operations/ops_add_dc_to_cluster_t.html http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/operations/ops_decomission_dc_t.html 2013/12/12 Andrew Cooper andrew.coo...@nisc.coop Hello, We are in the process of isolating multiple applications currently running in one large cassandra cluster to individual smaller clusters. Each application runs in its own keyspace. In order to reduce/eliminate downtime for a migration, I was curious if anyone had attempted the following process to migrate data to a new cluster: 1) Add new cluster nodes as a new datacenter to existing cluster 2) Set RF for specific keyspace to non-zero for new cluster, use nodetool rebuild on new nodes to stream data 3) Change application node connections to point to new cluster 4) Set RF to 0 for original cluster (stop new writes from going to original cluster) 5) Break connection between nodes so new nodes become a standalone cluster??? - Is this possible? what would be the high level steps? If this is an extremely bad or misinformed idea, I would like to know that as well! I am aware of other tools available including sstableloader, etc, but this seemed like a more elegant solution, leveraging cassandra's active-active features. Thanks, -Andrew NISC -- Fabien Rousseau aur...@yakaz.comwww.yakaz.com
Cassandra data update for a row
Hey Folks,
I have a row like this. 'twitter_row_key' is the row key and
411186035495010304 is column. Rest is values for 411186035495010304 column.
See below.
'twitter_row_key': OrderedDict([('411186035495010304', u'{score: 0,
tid: 411186035495010304, created_at: Thu Dec 12 17:29:24 + 2013,
favorite: 0, retweet: 0, approved: true}'),])
How can I set approved to 'false' ??
When I try insert for row key 'twitter_row_key' and column
411186035495010304, it overwrites the whole data and new row becomes like
this
'twitter_row_key': OrderedDict([('411186035495010304', u'{approved:
true}'),])
Any thoughts guys?
Re:
Thanks Aaron.
On Wed, Dec 11, 2013 at 10:45 PM, Aaron Morton aa...@thelastpickle.comwrote:
SYSTEM_MANAGER.create_column_family('Narrative','Twitter_search_test',
comparator_type='CompositeType', default_validation_class='UTF8Type',
key_validation_class='UTF8Type', column_validation_classes=validators)
CompositeType is a type composed of other types, see
http://pycassa.github.io/pycassa/assorted/composite_types.html?highlight=compositetype
Cheers
-
Aaron Morton
New Zealand
@aaronmorton
Co-Founder Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 12/12/2013, at 6:15 am, Kumar Ranjan winnerd...@gmail.com wrote:
Hey Folks,
So I am creating, column family using pycassaShell. See below:
validators = {
'approved': 'BooleanType',
'text': 'UTF8Type',
'favorite_count':'IntegerType',
'retweet_count': 'IntegerType',
'expanded_url': 'UTF8Type',
'tuid': 'LongType',
'screen_name': 'UTF8Type',
'profile_image': 'UTF8Type',
'embedly_data': 'CompositeType',
'created_at':'UTF8Type',
}
SYSTEM_MANAGER.create_column_family('Narrative','Twitter_search_test',
comparator_type='CompositeType', default_validation_class='UTF8Type',
key_validation_class='UTF8Type', column_validation_classes=validators)
I am getting this error:
InvalidRequestException: InvalidRequestException(why='Invalid definition
for comparator org.apache.cassandra.db.marshal.CompositeType.'
My data will look like this:
'row_key' : { 'tid' :
{
'expanded_url': u'http://instagram.com/p/hwDj2BJeBy/',
'text': '#snowinginNYC Makes me so
happy\xe2\x9d\x840brittles0 \xe2\x9b\x84 @ Grumman Studios
http://t.co/rlOvaYSfKa',
'profile_image': u'
https://pbs.twimg.com/profile_images/3262070059/1e82f895559b904945d28cd3ab3947e5_normal.jpeg
',
'tuid': 339322611,
'approved': 'true',
'favorite_count': 0,
'screen_name': u'LonaVigi',
'created_at': u'Wed Dec 11 01:10:05 + 2013',
'embedly_data': {u'provider_url': u'http://instagram.com/',
u'description': ulonavigi's photo on Instagram, u'title':
u'#snwinginNYC Makes me so happy\u2744@0brittles0 \u26c4', u'url': u'
http://distilleryimage7.ak.instagram.com/5b880dec61c711e3a50b129314edd3b_8.jpg',
u'thumbnail_width': 640, u'height': 640, u'width': 640, u'thumbnail_url': u'
http://distilleryimage7.ak.instagram.com/b880dec61c711e3a50b1293d14edd3b_8.jpg',
u'author_name': u'lonavigi', u'version': u'1.0', u'provider_name':
u'Instagram', u'type': u'poto', u'thumbnail_height': 640, u'author_url': u'
http://instagram.com/lonavigi'},
'tid': 410577192746500096,
'retweet_count': 0
}
}
Re: Nodetool repair exceptions in Cassandra 2.0.2
Thank you for the reply Aaron. Unfortunately, I could not seem to find any additional info in the logs. However, upgrading from 2.0.2 to 2.0.3 seems to have done the trick! Best regards, -David Laube On Dec 11, 2013, at 6:51 PM, Aaron Morton aa...@thelastpickle.com wrote: [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 the 10.x.x.48 node sent a tree response (merkle tree) to this node that did not contain the tree. This node then killed the repair session. Look for log messages on 10.x.x.48 that correlate with the repair session ID above. They may look like logger.error(Failed creating a merkle tree for + desc + , + initiator + (see log for details)”); or logger.info(String.format([repair #%s] Sending completed merkle tree to %s for %s/%s, desc.sessionId, initiator, desc.keyspace, desc.columnFamily)); Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 10/12/2013, at 12:57 pm, Laing, Michael michael.la...@nytimes.com wrote: My experience is that you must upgrade to 2.0.3 ASAP to fix this. Michael On Mon, Dec 9, 2013 at 6:39 PM, David Laube d...@stormpath.com wrote: Hi All, We are running Cassandra 2.0.2 and have recently stumbled upon an issue with nodetool repair. Upon running nodetool repair on each of the 5 nodes in the ring (one at a time) we observe the following exceptions returned to standard out; [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 [2013-12-08 11:04:02,063] Repair session 284c8b40-5ff8-11e3-97c0-5973cc397f8f for range (-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException: [repair #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,070] Repair session 399e7160-5ff8-11e3-97c0-5973cc397f8f for range (8901153810410866970,8915879751739915956] failed with error org.apache.cassandra.exceptions.RepairException: [repair #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (8901153810410866970,8915879751739915956]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,072] Repair session 3ea73340-5ff8-11e3-97c0-5973cc397f8f for range (1149084504576970235,1190026362216198862] failed with error org.apache.cassandra.exceptions.RepairException: [repair #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1149084504576970235,1190026362216198862]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,091] Repair session 6f0da460-5ff8-11e3-97c0-5973cc397f8f for range (-5407189524618266750,-5389231566389960750] failed with error org.apache.cassandra.exceptions.RepairException: [repair #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (-5407189524618266750,-5389231566389960750]] Validation failed in /10.x.x.103 [2013-12-09 23:16:36,962] Repair session 7efc2740-6127-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 [2013-12-09 23:16:36,986] Repair session a8c44260-6127-11e3-97c0-5973cc397f8f for range (-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException: [repair #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.210 The /var/log/cassandra/system.log shows similar info as above with no real explanation as to the root cause behind the exception(s). There also does not appear to be any additional info in /var/log/cassandra/cassandra.log. We have tried restoring a recent snapshot of the keyespace in question to a separate staging ring and the repair runs successfully and without exception there. This is even after we tried insert/delete on the keyspace in the separate staging ring. Has anyone seen this behavior before and what can we do to resolve this? Any assistance would be greatly appreciated. Best regards, -Dave
Get all the data for x number of seconds from CQL?
Below is my CQL table - CREATE TABLE test1 ( id text, record_name text, record_value blob, PRIMARY KEY (id, record_name) ) here id column will have data like this - timestamp.count And here timestamp is in milliseconds but rounded up to nearest seconds. So as an example, data in `id column` will be like this - 138688293.1 And a single row in the above table will be like this - 138688293.1 | event_name | hello-world Now my question is - Is it possible to get all the data for last 5 seconds or 10 seconds or 30 seconds by using the id column? I am running Cassandra 1.2.9
Re: Bulkoutputformat
Thanks Aaron, I was able to generate sstables and load using sstableloader. But after loading the tables when I do a select query I get this, the table has only one record. Is there anything I am missing or any logs I can look at. Request did not complete within rpc_timeout. On Wed, Dec 11, 2013 at 7:58 PM, Aaron Morton aa...@thelastpickle.comwrote: If you don’t need to use Hadoop then try the SSTableSimpleWriter and sstableloader , this post is a little old but still relevant http://www.datastax.com/dev/blog/bulk-loading Otherwise AFAIK BulkOutputFormat is what you want from hadoop http://www.datastax.com/docs/1.1/cluster_architecture/hadoop_integration Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/12/2013, at 11:27 am, varun allampalli vshoori.off...@gmail.com wrote: Hi All, I want to bulk insert data into cassandra. I was wondering of using BulkOutputformat in hadoop. Is it the best way or using driver and doing batch insert is the better way. Are there any disandvantages of using bulkoutputformat. Thanks for helping Varun
Re: Bulkoutputformat
Hi Aaron, It seems like you answered the question here. https://groups.google.com/forum/#!topic/nosql-databases/vjZA5vdycWA Can you give me the link to the blog which you mentioned http://thelastpickle.com/2013/01/11/primary-keys-in-cql/ Thanks in advance Varun On Thu, Dec 12, 2013 at 3:36 PM, varun allampalli vshoori.off...@gmail.comwrote: Thanks Aaron, I was able to generate sstables and load using sstableloader. But after loading the tables when I do a select query I get this, the table has only one record. Is there anything I am missing or any logs I can look at. Request did not complete within rpc_timeout. On Wed, Dec 11, 2013 at 7:58 PM, Aaron Morton aa...@thelastpickle.comwrote: If you don’t need to use Hadoop then try the SSTableSimpleWriter and sstableloader , this post is a little old but still relevant http://www.datastax.com/dev/blog/bulk-loading Otherwise AFAIK BulkOutputFormat is what you want from hadoop http://www.datastax.com/docs/1.1/cluster_architecture/hadoop_integration Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/12/2013, at 11:27 am, varun allampalli vshoori.off...@gmail.com wrote: Hi All, I want to bulk insert data into cassandra. I was wondering of using BulkOutputformat in hadoop. Is it the best way or using driver and doing batch insert is the better way. Are there any disandvantages of using bulkoutputformat. Thanks for helping Varun
Re: Write performance with 1.2.12
Quote from http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 *Murmur3Partitioner is NOT compatible with RandomPartitioner, so if you’re upgrading and using the new cassandra.yaml file, be sure to change the partitioner back to RandomPartitioner* On Thu, Dec 12, 2013 at 10:57 PM, srmore comom...@gmail.com wrote: On Thu, Dec 12, 2013 at 11:15 AM, J. Ryan Earl o...@jryanearl.us wrote: Why did you switch to RandomPartitioner away from Murmur3Partitioner? Have you tried with Murmur3? 1. # partitioner: org.apache.cassandra.dht.Murmur3Partitioner 2. partitioner: org.apache.cassandra.dht.RandomPartitioner Since I am comparing between the two versions I am keeping all the settings same. I see Murmur3Partitioner has some performance improvement but then switching back to RandomPartitioner should not cause performance to tank, right ? or am I missing something ? Also, is there an easier way to update the data from RandomPartitioner to Murmur3 ? (upgradesstable ?) On Fri, Dec 6, 2013 at 10:36 AM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak vicky@gmail.com wrote: You have passed the JVM configurations and not the cassandra configurations which is in cassandra.yaml. Apologies, was tuning JVM and that's what was in my mind. Here are the cassandra settings http://pastebin.com/uN42GgYT The spikes are not that significant in our case and we are running the cluster with 1.7 gb heap. Are these spikes causing any issue at your end? There are no big spikes, the overall performance seems to be about 40% low. On Fri, Dec 6, 2013 at 9:10 PM, srmore comom...@gmail.com wrote: On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak vicky@gmail.com wrote: Hard to say much without knowing about the cassandra configurations. The cassandra configuration is -Xms8G -Xmx8G -Xmn800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=2 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup. Were you able to get around it ? -VK On Fri, Dec 6, 2013 at 7:40 PM, srmore comom...@gmail.com wrote: We have a 3 node cluster running cassandra 1.2.12, they are pretty big machines 64G ram with 16 cores, cassandra heap is 8G. The interesting observation is that, when I send traffic to one node its performance is 2x more than when I send traffic to all the nodes. We ran 1.0.11 on the same box and we observed a slight dip but not half as seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to ONE make a slight improvement but not much. The read_Repair_chance is 0.1. We see some compactions running. following is my iostat -x output, sda is the ssd (for commit log) and sdb is the spinner. avg-cpu: %user %nice %system %iowait %steal %idle 66.460.008.950.010.00 24.58 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda2 0.0027.60 0.00 4.40 0.00 256.00 58.18 0.012.55 1.32 0.58 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 0.005.33 2.67 0.16 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.249.80 0.13 0.32 dm-4 0.00 0.00 0.00 6.60 0.0052.80 8.00 0.011.36 0.55 0.36 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 0.29 11.60 0.13 0.32 I can see I am cpu bound here but couldn't figure out exactly what is causing it, is this caused by GC or Compaction ? I am thinking it is compaction, I see a lot of context switches and interrupts in my vmstat output. I don't see GC activity in the logs but see some compaction activity. Has anyone seen this ? or know what can be done to free up the CPU. Thanks, Sandeep
