Cassandra and bug track type number sequencing.
Hi Guys, As per the subject, is there any way at all to easily associate small numbers in systems where users traditionally associate “bug/request” tickets with short numbers? In this use case I imagine the requirements would be as follows: The numbers don’t necessary need to be sequential, just need to be short enough for a user to read out loud. The numbers must be unique. It doesn’t need to scale, i.e. a typical “request” system is not getting hundreds of requests per second. In an ideal world, we could do away with associating “requests” with numbers, but its so ubiquitous I’m not sure you can sell doing away with short number codes. I am toying with the idea of a Cassandra table that makes available short “blocks” of numbers that an app server can hold “reservations” on. i.e. create table request_id_block( start int, end int, uuid uuid, reserved_by int, reserved_until bigint, primary key(start,end)); Will having an app server mark a block as reserved (QUOROM) and then reading it back (QUOROM) be enough to for an app server to know it owns that block of numbers? Best regards, Jacob
Re: Restore with archive commitlog
Andrey Ilinykh ailinykh at gmail.com writes: As someone told you this feature was added by Netflix to work with Priam (cassandra management tool). Priam itself uses it for several months only, so I doubt if anybody uses this feature in production. Any way, you can ping guys working on Priam. This is your best bet. https://github.com/Netflix/Priam Let us know if you can figure out how to use it. Thank you, Andrey Hello and thanks for answear, The link is not allowed by my company where i Work, so nobody never manage to make work a restore with archive commit log ? Strange to me to let this option in all new releases of cassandra if it doesn't work. Regards, Bonnet Jonathan.
Re: Try to configure commitlog_archiving.properties
Artur Kronenberg artur.kronenberg at openmarket.com writes: It's been a while since I tried that but here are some things I can think of: * the .log.out seems wrong. Unless your cassandra commitlogs don't end in .log.out. I tried this locally with your script and my commitlogs get extracted to .log files for me. * I never tried the restore procedure on a cluster with multiple nodes. I imagine if you have a quorum defined the replayed commitlog may be ignored because the commitlog write operation is older then the deletion in which case the latter will be returned (nothing in your case) Thanks to take time to answear Artur, When you talkin' about quorum, you think about replication factor or tuneable transaction quorum for consistency ? I work on a multiple node cluster, but for this restore i use a keyspace with no replication on the other node. Regards, Bonnet Jonathan.
Cassandra 1.2 : OutOfMemoryError: unable to create new native thread
Hi guys! I beleive my limits settings are correct. Here is the output of ulimits -a: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1547135 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 10 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited However, I just had a couple of cassandra nodes go down over the weekend for no apparent reason with the following error: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Any input is greatly appreciated. -- Regards, Oleg Dulin http://www.olegdulin.com
Re: Cassandra and bug track type number sequencing.
If you want sequential numbers, you can't trust distributed counters from Cassandra. However, you could use Redis for this. Additionally, you can also use a random UUID and only show the customer first 6 characters -- it is unique enough... Oleg On 2013-12-16 09:33:39 +, Jacob Rhoden said: Hi Guys, As per the subject, is there any way at all to easily associate small numbers in systems where users traditionally associate “bug/request” tickets with short numbers? In this use case I imagine the requirements would be as follows: • The numbers don’t necessary need to be sequential, just need to be short enough for a user to read out loud. • The numbers must be unique. • It doesn’t need to scale, i.e. a typical “request” system is not getting hundreds of requests per second. In an ideal world, we could do away with associating “requests” with numbers, but its so ubiquitous I’m not sure you can sell doing away with short number codes. I am toying with the idea of a Cassandra table that makes available short “blocks” of numbers that an app server can hold “reservations” on. i.e. create table request_id_block( start int, end int, uuid uuid, reserved_by int, reserved_until bigint, primary key(start,end)); Will having an app server mark a block as reserved (QUOROM) and then reading it back (QUOROM) be enough to for an app server to know it owns that block of numbers? Best regards, Jacob -- Regards, Oleg Dulin http://www.olegdulin.com
Re: Cassandra 1.2 : OutOfMemoryError: unable to create new native thread
the cassandra-env.sh has option JVM_OPTS=$JVM_OPTS -Xss180k it will give this error if you start cassandra with java 7. So increase the value, or remove option. Regards, Maciej On Mon, Dec 16, 2013 at 2:37 PM, srmore comom...@gmail.com wrote: What is your thread stack size (xss) ? try increasing that, that could help. Sometimes the limitation is imposed by the host provider (e.g. amazon ec2 etc.) Thanks, Sandeep On Mon, Dec 16, 2013 at 6:53 AM, Oleg Dulin oleg.du...@gmail.com wrote: Hi guys! I beleive my limits settings are correct. Here is the output of ulimits -a: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1547135 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 10 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited However, I just had a couple of cassandra nodes go down over the weekend for no apparent reason with the following error: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Any input is greatly appreciated. -- Regards, Oleg Dulin http://www.olegdulin.com
Re: Conference or training recommendations
Datstax recently started offering free virtual training. You may want to try that first: http://www.datastax.com/what-we-offer/products-services/training/virtual-training There are also many Cassandra meetups around the world: http://cassandra.meetup.com/ Datstax also offers classroom training, but it is not cheap: http://www.datastax.com/what-we-offer/products-services/training As for conferences, this year's Cassandra Summits were in San Francisco in June and London in October. I have not seen an announcement of next year's summit(s). -Ike On Sun, Dec 15, 2013 at 12:07 PM, Robert Wille rwi...@fold3.com wrote: I’d like to attend a conference or some form of training to become more proficient and knowledgable about Cassandra. Any suggestions?
RE: Cassandra 1.1.6 - Disk usage and Load displayed in ring doesn't match
Do you have any snapshots on the nodes where you are seeing this issue? Snapshots will link to sstables which will cause them not be deleted. -Arindam From: Narendra Sharma [mailto:narendra.sha...@gmail.com] Sent: Sunday, December 15, 2013 1:15 PM To: user@cassandra.apache.org Subject: Cassandra 1.1.6 - Disk usage and Load displayed in ring doesn't match We have 8 node cluster. Replication factor is 3. For some of the nodes the Disk usage (du -ksh .) in the data directory for CF doesn't match the Load reported in nodetool ring command. When we expanded the cluster from 4 node to 8 nodes (4 weeks back), everything was okay. Over period of last 2-3 weeks the disk usage has gone up. We increased the RF from 2 to 3 2 weeks ago. I am not sure if increasing the RF is causing this issue. For one of the nodes that I analyzed: 1. nodetool ring reported load as 575.38 GB 2. nodetool cfstats for the CF reported: SSTable count: 28 Space used (live): 572671381955 Space used (total): 572671381955 3. 'ls -1 *Data* | wc -l' in the data folder for CF returned 46 4. 'du -ksh .' in the data folder for CF returned 720G The above numbers indicate that there are some sstables that are obsolete and are still occupying space on disk. What could be wrong? Will restarting the node help? The cassandra process is running for last 45 days with no downtime. However, because the disk usage is high, we are not able to run full compaction. Also, I can't find reference to each of the sstables on disk in the system.log file. For eg I have one data file on disk as (ls -lth): 86G Nov 20 06:14 I have system.log file with first line: INFO [main] 2013-11-18 09:41:56,120 AbstractCassandraDaemon.java (line 101) Logging initialized The 86G file must be a result of some compaction. I see no reference of data file in system.log file between 11/18 to 11/25. What could be the reason for that? The only reference is dated 11/29 when the file was being streamed to another node (new node). How can I identify the obsolete files and remove them? I am thinking about following. Let me know if it make sense. 1. Restart the node and check the state. 2. Move the oldest data files to another location (to another mount point) 3. Restart the node again 4. Run repair on the node so that it can get the missing data from its peers. I compared the numbers of a healthy node for the same CF: 1. nodetool ring reported load as 662.95 GB 2. nodetool cfstats for the CF reported: SSTable count: 16 Space used (live): 670524321067 Space used (total): 670524321067 3. 'ls -1 *Data* | wc -l' in the data folder for CF returned 16 4. 'du -ksh .' in the data folder for CF returned 625G -Naren -- Narendra Sharma Software Engineer http://www.aeris.com http://narendrasharma.blogspot.com/
Re: Cassandra 1.1.6 - Disk usage and Load displayed in ring doesn't match
No snapshots. I restarted the node and now the Load in ring is in sync with the disk usage. Not sure what caused it to go out of sync. However, the Live SStable count doesn't match exactly with the number of data files on disk. I am going through the Cassandra code to understand what could be the reason for the mismatch in the sstable count and also why there is no reference of some of the data files in system.log. On Mon, Dec 16, 2013 at 2:45 PM, Arindam Barua aba...@247-inc.com wrote: Do you have any snapshots on the nodes where you are seeing this issue? Snapshots will link to sstables which will cause them not be deleted. -Arindam *From:* Narendra Sharma [mailto:narendra.sha...@gmail.com] *Sent:* Sunday, December 15, 2013 1:15 PM *To:* user@cassandra.apache.org *Subject:* Cassandra 1.1.6 - Disk usage and Load displayed in ring doesn't match We have 8 node cluster. Replication factor is 3. For some of the nodes the Disk usage (du -ksh .) in the data directory for CF doesn't match the Load reported in nodetool ring command. When we expanded the cluster from 4 node to 8 nodes (4 weeks back), everything was okay. Over period of last 2-3 weeks the disk usage has gone up. We increased the RF from 2 to 3 2 weeks ago. I am not sure if increasing the RF is causing this issue. For one of the nodes that I analyzed: 1. nodetool ring reported load as 575.38 GB 2. nodetool cfstats for the CF reported: SSTable count: 28 Space used (live): 572671381955 Space used (total): 572671381955 3. 'ls -1 *Data* | wc -l' in the data folder for CF returned 46 4. 'du -ksh .' in the data folder for CF returned 720G The above numbers indicate that there are some sstables that are obsolete and are still occupying space on disk. What could be wrong? Will restarting the node help? The cassandra process is running for last 45 days with no downtime. However, because the disk usage is high, we are not able to run full compaction. Also, I can't find reference to each of the sstables on disk in the system.log file. For eg I have one data file on disk as (ls -lth): 86G Nov 20 06:14 I have system.log file with first line: INFO [main] 2013-11-18 09:41:56,120 AbstractCassandraDaemon.java (line 101) Logging initialized The 86G file must be a result of some compaction. I see no reference of data file in system.log file between 11/18 to 11/25. What could be the reason for that? The only reference is dated 11/29 when the file was being streamed to another node (new node). How can I identify the obsolete files and remove them? I am thinking about following. Let me know if it make sense. 1. Restart the node and check the state. 2. Move the oldest data files to another location (to another mount point) 3. Restart the node again 4. Run repair on the node so that it can get the missing data from its peers. I compared the numbers of a healthy node for the same CF: 1. nodetool ring reported load as 662.95 GB 2. nodetool cfstats for the CF reported: SSTable count: 16 Space used (live): 670524321067 Space used (total): 670524321067 3. 'ls -1 *Data* | wc -l' in the data folder for CF returned 16 4. 'du -ksh .' in the data folder for CF returned 625G -Naren -- Narendra Sharma Software Engineer *http://www.aeris.com http://www.aeris.com* *http://narendrasharma.blogspot.com/ http://narendrasharma.blogspot.com/* -- Narendra Sharma Software Engineer *http://www.aeris.com http://www.aeris.com* *http://narendrasharma.blogspot.com/ http://narendrasharma.blogspot.com/*
