Re: Integration Error between Cassandra and Eclipse
I'm also a beginner, Really not sure if the source code is really necessary for eclipse IDE integration. If you want to get it from GIT - get a git client, and rest all is kinda similar to SVN. Thanks Kuldeep singh CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it, are confidential and may be privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any review, disclosure, copying, distribution, retransmission, dissemination or other use of any of the information contained in, or attached to, this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender. Please destroy the original transmission and its attachments without reading or saving in any manner. Thank you. Opera Solutions LLC From: dir dir [mailto:sikerasa...@gmail.com] Sent: Sunday, January 08, 2012 12:16 AM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: Integration Error between Cassandra and Eclipse I am also beginner user in cassandra. Honestly I wonder, if we can download binary installer from http://cassandra.apache.org/download/ but why we have to check out from: svn http://svn.apache.org/repos/asf/cassandra/trunk/ cassandra-trunk ?? What is the deference between them? Can we perform cassandra installation process by using binary installer in http://cassandra.apache.org/download/ without have to check out from svn?? Can we integrate cassandra and eclipse IDE without have to perform ant generate-eclipse-files ?? If cassandra project have already moved from svn to git, would you tell me how to check out cassandra project from git, please?? Thank you. On Fri, Jan 6, 2012 at 11:03 AM, Yuki Morishita mor.y...@gmail.commailto:mor.y...@gmail.com wrote: Also note that Cassandra project switched to git from svn. See Source control section of http://cassandra.apache.org/download/ . Regards, Yuki -- Yuki Morishita On Thursday, January 5, 2012 at 7:59 PM, Maki Watanabe wrote: Sorry, ignore my reply. I had same result with import. ( 1 error in unit test code many warnings ) 2012/1/6 Maki Watanabe watanabe.m...@gmail.commailto:watanabe.m...@gmail.com: How about to use File-Import... rather than File-New Java Project? After extracting the source, ant build, and ant generate-eclipse-files: 1. File-Import... 2. Choose Existing Project into workspace... 3. Choose your source directory as root directory and then push Finish 2012/1/6 bobby saputra zaibat...@gmail.commailto:zaibat...@gmail.com: Hi There, I am a beginner user in Cassandra. I hear from many people said Cassandra is a powerful database software which is used by Facebook, Twitter, Digg, etc. So I feel interesting to study more about Cassandra. When I performed integration process between Cassandra with Eclipse IDE (in this case I use Java as computer language), I get trouble and have many problem. I have already followed all instruction from http://wiki.apache.org/cassandra/RunningCassandraInEclipse, but this tutorial was not working properly. I got a lot of errors and warnings while creating Java project in eclipse. These are the errors and warnings: Error(X) (1 item): Description Resource Location The method rangeSet(RangeT...) in the type Range is not applicable for the arguments (Range[]) RangeTest.java line 178 Warnings(!) (100 of 2916 items): Description Resource Location AbstractType is a raw type. References to generic type AbstractTypeT should be parameterized AbstractColumnContainer.java line 72 (and many same warnings) These are what i've done: 1. I checked out cassandra-trunk from given link using SlikSvn as svn client. 2. I moved to cassandra-trunk folder, and build with ant using ant build command. 3. I generate eclipse files with ant using ant generate-eclipse-files command. 4. I create new java project on eclipse, insert project name with cassandra-trunk, browse the location into cassandra-trunk folder. Do I perform any mistakes? Or there are something wrong with the tutorial in http://wiki.apache.org/cassandra/RunningCassandraInEclipse ?? I have already googling to find the solution to solve this problem, but unfortunately I found no results. Would you want to help me by giving me a guide how to solve this problem? Please Thank you very much for your help. Best Regards, Wira Saputra -- w3m -- w3m
Re: Minor version upgrade and SSTable compatibilty
1. Will I need to upgrade all nodes before performing streaming/repair? AFAIK if they are all at least 1.0.3 you can, as compatibility and stream capability is based on the major version. 2. Will it be possible to downgrade a node from 1.0.6 to 1.0.3 and keep newly created SSTables? Or will I then have to revert to a pre-upgrade snapshot? AFAIK you can. *NOTE:* This is from a quick look through the code, anyone else confirm this ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/01/2012, at 4:13 AM, Jonas Borgström wrote: Hi, I have some questions about SSTable compatibility when doing a minor version upgrade. For example when upgrading from 1.0.3 (Uses version hb) to 1.0.6 (uses version hc). 1. Will I need to upgrade all nodes before performing streaming/repair? 2. Will it be possible to downgrade a node from 1.0.6 to 1.0.3 and keep newly created SSTables? Or will I then have to revert to a pre-upgrade snapshot? / Jonas
Re: Is this correct way to create a Composite Type
The reason I asked was because I could not understand
CFFollower_For_Users
RowKey#{userID,n}:followerID
I found about Composite Types and I am trying to solve a situation like using
composite. Also I read that Cassandra columns can save upto 2 billion values.
But in one of the presentations , I read that if you are saving billion
values in columns, there is possibly a flaw in design.
Not sure about a design floor. Having more data take longer to process in
situations where all the data must be processed. Repair and compaction will
take longer and use more IO.
Design rows to hold the data you need to read. If you are storing time series
data consider partitioning it into different rows for something like each day
or month.
One more thing I am not very clear is that the secondary indexing on columns
with say billion columns is a good performer.
See above, billions of columns may not be a great idea. see
http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/01/2012, at 7:47 AM, investtr wrote:
On 01/07/2012 02:38 AM, aaron morton wrote:
Can you provide some context ? e.g. what client are you using ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/01/2012, at 7:29 AM, investtr wrote:
I am trying to understand the composite type.
Is this a right way to create a Composite Data ?
CFFollower_For_Users
RowKey#{userID,n}:followerID
for simplicity I have replaced userID by followerID.
regards,
Ramesh
Thanks Aaron for your replies.
I will be using Hector.
I planned to use SuperColumns and it made sense before. But I got to a point
where
the need was like.
Multiple Products - Each Product has Sub-Product or Sub-Products - Each
Sub-Product have many Sub-Categories.
I found about Composite Types and I am trying to solve a situation like using
composite. Also I read that Cassandra columns can save upto 2 billion values.
But in one of the presentations , I read that if you are saving billion
values in columns, there is possibly a flaw in design.
One more thing I am not very clear is that the secondary indexing on columns
with say billion columns is a good performer.
Thank you
Regards,
Ramesh
Re: Lots and Lots of CompactionReducer Threads
How many threads ? Any errors in the server logs ? What does noodtool tpstats and nodetool compactionstats say ? Did you change compaction_strategy for the CF's ? By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml Have you set the JVM heap settings ? What does nodetool info show ? Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote: Hi Everybody, JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup… Cassandra 1.0.5 on CentOS 6.0 4 GB of RAM 50 GB SSD HD Memtable flush threshold = 128 MB compaction throughput limit = 16 MB/sec Multithreaded compaction = true It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks. Thanks, Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com
Re: Lots and Lots of CompactionReducer Threads
With the exception of a few little warnings on start-up about the Memtable live ratio, there is nothing at WARN or above in the logs. Just before the JVM terminates, there are about 10,000 threads in Reducer executor pools that look like this in JConsole … Name: CompactionReducer:1 State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea Total blocked: 0 Total waited: 1 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) The results from tpstats don't look too interesting… Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 03455159 0 0 RequestResponseStage 0 0 10133276 0 0 MutationStage 0 05898833 0 0 ReadRepairStage 0 02078449 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 236388 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemtablePostFlusher 0 0231 0 0 StreamStage 0 0 0 0 0 FlushWriter 0 0231 0 0 MiscStage 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 35 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 REQUEST_RESPONSE 0 The results from info seem unremarkable as well… Token: 15312706500 Gossip active: true Load : 5.6 GB Generation No: 1325995515 Uptime (seconds) : 67199 Heap Memory (MB) : 970.32 / 1968.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on the number of concurrent compactors. What is interesting to me here is that Cassandra creates an executor for every single compaction in ParallelCompactionIterable. Why couldn't we just create a pool with Runtime.availableProcessors() Threads and be done with it? Let me know if I left any info out. Thanks! Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com From: aaron morton aa...@thelastpickle.commailto:aa...@thelastpickle.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Sun, 8 Jan 2012 16:51:50 -0500 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: Lots and Lots of CompactionReducer Threads How many threads ? Any errors in the server logs ? What does noodtool tpstats and nodetool compactionstats say ? Did you change compaction_strategy for the CF's ? By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml Have you set the JVM heap settings ? What does nodetool info show ? Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote: Hi Everybody, JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup… Cassandra 1.0.5 on CentOS 6.0 4 GB of RAM 50 GB SSD HD Memtable flush threshold = 128 MB compaction throughput limit = 16 MB/sec Multithreaded compaction = true It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable
Re: Lots and Lots of CompactionReducer Threads
After some searching, I think I may have found something in the code itself, and so I've filed a big report - https://issues.apache.org/jira/browse/CASSANDRA-3711 Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.com From: Caleb Rackliffe ca...@steelhouse.commailto:ca...@steelhouse.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Sun, 8 Jan 2012 17:48:59 -0500 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: aa...@thelastpickle.commailto:aa...@thelastpickle.com aa...@thelastpickle.commailto:aa...@thelastpickle.com Subject: Re: Lots and Lots of CompactionReducer Threads With the exception of a few little warnings on start-up about the Memtable live ratio, there is nothing at WARN or above in the logs. Just before the JVM terminates, there are about 10,000 threads in Reducer executor pools that look like this in JConsole … Name: CompactionReducer:1 State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea Total blocked: 0 Total waited: 1 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) The results from tpstats don't look too interesting… Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 03455159 0 0 RequestResponseStage 0 0 10133276 0 0 MutationStage 0 05898833 0 0 ReadRepairStage 0 02078449 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 236388 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemtablePostFlusher 0 0231 0 0 StreamStage 0 0 0 0 0 FlushWriter 0 0231 0 0 MiscStage 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 35 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 REQUEST_RESPONSE 0 The results from info seem unremarkable as well… Token: 15312706500 Gossip active: true Load : 5.6 GB Generation No: 1325995515 Uptime (seconds) : 67199 Heap Memory (MB) : 970.32 / 1968.00 Data Center : datacenter1 Rack : rack1 Exceptions : 0 I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on the number of concurrent compactors. What is interesting to me here is that Cassandra creates an executor for every single compaction in ParallelCompactionIterable. Why couldn't we just create a pool with Runtime.availableProcessors() Threads and be done with it? Let me know if I left any info out. Thanks! Caleb Rackliffe | Software Developer M 949.981.0159 | ca...@steelhouse.commailto:ca...@steelhouse.com From: aaron morton aa...@thelastpickle.commailto:aa...@thelastpickle.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Sun, 8 Jan 2012 16:51:50 -0500 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: Lots and Lots of CompactionReducer Threads How many threads ? Any errors in the server logs ? What does noodtool tpstats and nodetool compactionstats say ? Did you change compaction_strategy for the CF's ? By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml Have
Can I use BulkOutputFormat from 1.1 to load data to older Cassandra versions?
Hi! Can the new BulkOutputFormat (https://issues.apache.org/jira/browse/CASSANDRA-3045) be used to load data to servers running cassandra 0.8.7 and/or Cassandra 1.0.6? I'm thinking of using jar files from the development version to load data onto a production cluster which I want to keep on a production version of Cassandra. Can I do that, or does BulkOutputFormat require an API level that is only in the development version of Cassandra? Thanks, \EF
[0.8.x] Node join stuck with all network transfers done
Hi ! I have problem with 0.8.7 node joining cluster of two 0.8.9s (RF=2). Seems all transfers ware done but joining node(.17) does not change it's state [3]. Strange is Nothing streaming from /192.168.3.8 netstats result [2] and still raising number of pending tasks [1], while .8 not transferring anything [4]. I tried to restart each of nodes, it didn't help except joining process started again, transferring all data again, stuck in same moment but having doubled load on joining node. Any hints ? Thanks in advance, regards. [1]: @192.168.3.17 ~]$ nodetool -h localhost compactionstats pending tasks: 836 [2]: @192.168.3.17 ~]$ nodetool -h localhost netstats Mode: Bootstrapping Not sending any streams. Nothing streaming from /192.168.3.8 Pool NameActive Pending Completed Commandsn/a 0171 Responses n/a 09387965 [3]: @192.168.3.17 ~]$ nodetool -h localhost ring Address DC RackStatus State LoadOwns Token 113427455640312821154458202477256070485 192.168.3.8 datacenter1 rack1 Up Normal 128.51 GB 33.33% 0 192.168.3.7 datacenter1 rack1 Up Normal 137.65 GB 50.00% 85070591730234615865843651857942052864 192.168.3.17datacenter1 rack1 Up Joining 127.02 GB 16.67% 113427455640312821154458202477256070485 [4]: @192.168.3.8 ~]$ nodetool -h localhost netstats Mode: Normal Not sending any streams. Not receiving any streams. Pool NameActive Pending Completed Commandsn/a 05261062 Responses n/a 02963742 -- Mateusz Korniak
