[jira] [Updated] (CASSANDRA-2691) If an upload to central repository fails for an artifact, retry a couple of times before killing the build
[ https://issues.apache.org/jira/browse/CASSANDRA-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Connolly updated CASSANDRA-2691: Attachment: CASSANDRA-2691.patch Will now retry deployment up to 10 times before giving up. If an upload to central repository fails for an artifact, retry a couple of times before killing the build -- Key: CASSANDRA-2691 URL: https://issues.apache.org/jira/browse/CASSANDRA-2691 Project: Cassandra Issue Type: Improvement Affects Versions: 0.7.7, 0.8.0 Reporter: Stephen Connolly Attachments: CASSANDRA-2691.patch Depends on MGPG-35 http://jira.codehaus.org/browse/MGPG-35 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2868) Native Memory Leak
[ https://issues.apache.org/jira/browse/CASSANDRA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063770#comment-13063770 ] Daniel Doubleday commented on CASSANDRA-2868: - Next: [anon]: 3675224 (+47616KB in 1 day) bq. Is your data size constant? If not you are probably seeing growth in the index samples and bloom filters. Well no - the data size is increasing. But I thought that index and bf is good old plain java heap no? JVM heap stats are really relaxed. Yet I think that doesn't really matter because what we are seeing is an ever increasing rss mem consumption even though we have -Xmx3G and -Xms3G and mlockall (pmap shows these 3G as one block). So something seems to be constantly allocating native mem that has nothing to do with java heap. Native Memory Leak -- Key: CASSANDRA-2868 URL: https://issues.apache.org/jira/browse/CASSANDRA-2868 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.6 Reporter: Daniel Doubleday Priority: Minor We have memory issues with long running servers. These have been confirmed by several users in the user list. That's why I report. The memory consumption of the cassandra java process increases steadily until it's killed by the os because of oom (with no swap) Our server is started with -Xmx3000M and running for around 23 days. pmap -x shows Total SST: 1961616 (mem mapped data and index files) Anon RSS: 6499640 Total RSS: 8478376 This shows that 3G are 'overallocated'. We will use BRAF on one of our less important nodes to check wether it is related to mmap and report back. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2886) cli command update keyspace fails to change replication_factor
cli command update keyspace fails to change replication_factor -- Key: CASSANDRA-2886 URL: https://issues.apache.org/jira/browse/CASSANDRA-2886 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.8.1 Environment: Windows or FreeBSD Reporter: Radim Kolar Priority: Minor Fix For: 0.8.2 In wiki http://wiki.apache.org/cassandra/Operations#Replication its documented that increasing replication on live cluster can be done with following command: [default@unknown] update keyspace test with replication_factor = 2 ; No enum const class org.apache.cassandra.cli.CliClient$AddKeyspaceArgument.REPLICATION_FACTOR this command do not works. Wiki should be updated or cassandra-cli fixed More info: [default@unknown] describe keyspace test; Keyspace: test: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: sipdb phone calls routing information Key Validation Class: org.apache.cassandra.db.marshal.IntegerType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.AsciiType Row cache size / save period in seconds: 0.0/0 Key cache size / save period in seconds: 0.0/0 Memtable thresholds: 0.0328125/7/1440 (millions of ops/MB/minutes) GC grace seconds: 0 Compaction min/max thresholds: 4/32 Read repair chance: 0.0 Replicate on write: false Built indexes: [] Column Metadata: Column Name: kam Validation Class: org.apache.cassandra.db.marshal.AsciiType Index Name: kam_idx [default@unknown] -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063788#comment-13063788 ] Terje Marthinussen commented on CASSANDRA-47: - Tried to import some data with json2sstable to compare sizes, but seems like json2sstable is not entirely working yet? SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063790#comment-13063790 ] Pavel Yaskevich commented on CASSANDRA-47: -- Tests are passing. Does it crush on you? As I wrote before - currently to check real size of the file (tested only on linux because OS X FS saves empty blocks to the disk for some reason) you need to get a block count using 'ls -alhs', current patch reserves an empty space for each chunk because we need to do seeks while we write data using SSTableWriter. I'm currently working to avoid that (CASSANDRA-2879 is a step in that direction). SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2771) Remove commitlog_rotation_threshold_in_mb
[ https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063797#comment-13063797 ] Radim Kolar commented on CASSANDRA-2771: I used that feature to change commit log to 32MB segments Remove commitlog_rotation_threshold_in_mb -- Key: CASSANDRA-2771 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771 Project: Cassandra Issue Type: Improvement Reporter: Patricio Echague Assignee: Patricio Echague Priority: Minor Labels: commitlog Fix For: 1.0 Attachments: CASSANDRA-2771-2-trunk.txt, CASSANDRA-2771-3-trunk.txt Remove the commitlog segment size config setting, nobody has ever changed it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2582) sstable2json: make TTL optional, possibility of incremental dumps
[ https://issues.apache.org/jira/browse/CASSANDRA-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063800#comment-13063800 ] Timo Nentwig commented on CASSANDRA-2582: - I didn't say that there's no alternative to loaded the files into RAM :) Anyway, actually modifying the JSON files was my first idea because patching distro code requires me to patch it in every new version again. I did it anyway because it's a really trivial patch and it save a lot of IO and time. sstable2json: make TTL optional, possibility of incremental dumps - Key: CASSANDRA-2582 URL: https://issues.apache.org/jira/browse/CASSANDRA-2582 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Timo Nentwig Priority: Trivial Fix For: 0.8.2 Attachments: nottl-and-lastmodified-0.7.5.patch, nottl-and-lastmodified.patch Data on live cluster is supposed to expire automatically but exported to an archive cluster beforehand. In order to prevent expiration on the archive cluster add an option to sstable2json to ignore TTL data. Furthermore add an lastModified option that ignores older columns for incremental archiving. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2887) decommision fails if pathseparators are different
decommision fails if pathseparators are different - Key: CASSANDRA-2887 URL: https://issues.apache.org/jira/browse/CASSANDRA-2887 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.1 Environment: FreeBSD 8 vs Windows XP Reporter: Radim Kolar I have 2 nodes in cluster. One is running on Windows XP and second is running on FreeBSD 8.2. If i try to decommission freebsd note windows node throws and exception during receiving streamed data: java.lang.AssertionError: Filename must include parent directory. at org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.ja va:122) at org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.dese rialize(PendingFile.java:130) at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.de serialize(StreamHeader.java:90) at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.de serialize(StreamHeader.java:72) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnect ion.java:95) this problem was discussed here http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/0-7-problem-on-cygwin-td5679932.html with patch for cassandra 0.7 included -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2771) Remove commitlog_rotation_threshold_in_mb
[ https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063828#comment-13063828 ] Jonathan Ellis commented on CASSANDRA-2771: --- Why? Remove commitlog_rotation_threshold_in_mb -- Key: CASSANDRA-2771 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771 Project: Cassandra Issue Type: Improvement Reporter: Patricio Echague Assignee: Patricio Echague Priority: Minor Labels: commitlog Fix For: 1.0 Attachments: CASSANDRA-2771-2-trunk.txt, CASSANDRA-2771-3-trunk.txt Remove the commitlog segment size config setting, nobody has ever changed it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2887) decommision fails if pathseparators are different
[ https://issues.apache.org/jira/browse/CASSANDRA-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2887. --- Resolution: Invalid Mixed windows/unix clusters are not supported for this reason. (Best practice is to have homogenous clusters anyway, to reduce the surface for environment errors.) decommision fails if pathseparators are different - Key: CASSANDRA-2887 URL: https://issues.apache.org/jira/browse/CASSANDRA-2887 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.1 Environment: FreeBSD 8 vs Windows XP Reporter: Radim Kolar I have 2 nodes in cluster. One is running on Windows XP and second is running on FreeBSD 8.2. If i try to decommission freebsd note windows node throws and exception during receiving streamed data: java.lang.AssertionError: Filename must include parent directory. at org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.ja va:122) at org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.dese rialize(PendingFile.java:130) at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.de serialize(StreamHeader.java:90) at org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.de serialize(StreamHeader.java:72) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnect ion.java:95) this problem was discussed here http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/0-7-problem-on-cygwin-td5679932.html with patch for cassandra 0.7 included -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2888) CQL support for JDBC DatabaseMetaData
CQL support for JDBC DatabaseMetaData - Key: CASSANDRA-2888 URL: https://issues.apache.org/jira/browse/CASSANDRA-2888 Project: Cassandra Issue Type: Improvement Components: Drivers Affects Versions: 0.8.1 Environment: J2SE 1.6.0_22 x64 on Fedora 15 Reporter: Dave Carlson Priority: Minor In order to increase the drop-in capability of CQL to existing JDBC app bases, CQL must be updated to include at least semi-valid responses to the JDBC metadata portion. without enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) Resource has error: java.lang.UnsupportedOperationException: method not supported ... with enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) org.apache.cassandra.cql.jdbc.CassandraConnection@1915470e -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063990#comment-13063990 ] Terje Marthinussen commented on CASSANDRA-47: - It starts making the file, but then it crashes. java.lang.IndexOutOfBoundsException at java.io.RandomAccessFile.readBytes(Native Method) at java.io.RandomAccessFile.read(RandomAccessFile.java:338) at org.apache.cassandra.io.compress.AbstractCompressedFile.readCompressedChunk(AbstractCompressedFile.java:169) at org.apache.cassandra.io.compress.AbstractCompressedFile.reBuffer(AbstractCompressedFile.java:85) at org.apache.cassandra.io.compress.CompressedDataFile$Output.writeAtMost(CompressedDataFile.java:420) at org.apache.cassandra.io.compress.CompressedDataFile$Output.write(CompressedDataFile.java:404) at org.apache.cassandra.db.ColumnIndexer.writeBloomFilter(ColumnIndexer.java:144) at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:114) at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50) at org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:107) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:156) at org.apache.cassandra.tools.SSTableImport.importUnsorted(SSTableImport.java:290) at org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:252) at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:476) ERROR: null I added a bit debug printing to the read/writeChunkLength logger.info(TNM: readChunkLength + len + in + this.path + at: + super.getFilePointer()); INFO 08:13:40,652 TNM: writeChunkLength 28511 in /var/folders/8d/8drNFwr6FEe7uEdizf5pAE+++TI/-Tmp-/Test-h-22-Data.db2879008826574155982compressing at: 1245194 INFO 08:13:40,655 TNM: writeChunkLength 65538 in /var/folders/8d/8drNFwr6FEe7uEdizf5pAE+++TI/-Tmp-/Test-h-22-Data.db2879008826574155982compressing at: 1310730 INFO 08:13:40,656 TNM: readChunkLength -1313507164 in /var/folders/8d/8drNFwr6FEe7uEdizf5pAE+++TI/-Tmp-/Test-h-22-Data.db2879008826574155982compressing at: 1376270 Importing 787 keys... Currently imported 1 keys. [Some lines deleted] INFO 15:10:18,219 TNM: writeChunkLength 14870 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1114122 INFO 15:10:18,219 TNM: readChunkLength 21106 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1179662 INFO 15:10:18,229 TNM: writeChunkLength 21104 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1179658 INFO 15:10:18,258 TNM: readChunkLength 26901 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1245198 INFO 15:10:18,260 TNM: writeChunkLength 26901 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1245194 INFO 15:10:18,260 TNM: readChunkLength 21104 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1179662 INFO 15:10:18,261 TNM: writeChunkLength 21106 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1179658 INFO 15:10:18,261 TNM: readChunkLength 26901 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1245198 INFO 15:20:18,348 TNM: writeChunkLength 26901 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1245194 INFO 15:20:18,370 TNM: readChunkLength 65545 in /Users/terje/work/cassandra-0.8/data/JP/Test-h-22-Data.db at: 1310734 And then the above exception Note, this is on OSX and there seems to be another report of problems with OSX above? SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2869) CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues
[ https://issues.apache.org/jira/browse/CASSANDRA-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13063993#comment-13063993 ] Brandon Williams commented on CASSANDRA-2869: - Looks like we can remove UDFCONTEXT_SCHEMA_KEY_PREFIX now too, no? CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues Key: CASSANDRA-2869 URL: https://issues.apache.org/jira/browse/CASSANDRA-2869 Project: Cassandra Issue Type: Bug Components: Contrib Affects Versions: 0.7.2 Reporter: Grant Ingersoll Assignee: Jeremy Hanna Attachments: 2869.txt CassandraStorage appears to have threading issues along the lines of those described at http://pig.markmail.org/message/oz7oz2x2dwp66eoz due to the sharing of the UDFContext. I believe the fix lies in implementing {code} public void setStoreFuncUDFContextSignature(String signature) { } {code} and then using that signature when getting the UDFContext. From the Pig manual: {quote} setStoreFunc!UDFContextSignature(): This method will be called by Pig both in the front end and back end to pass a unique signature to the Storer. The signature can be used to store into the UDFContext any information which the Storer needs to store between various method invocations in the front end and back end. The default implementation in StoreFunc has an empty body. This method will be called before other methods. {quote} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2869) CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues
[ https://issues.apache.org/jira/browse/CASSANDRA-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-2869: Reviewer: brandon.williams CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues Key: CASSANDRA-2869 URL: https://issues.apache.org/jira/browse/CASSANDRA-2869 Project: Cassandra Issue Type: Bug Components: Contrib Affects Versions: 0.7.2 Reporter: Grant Ingersoll Assignee: Jeremy Hanna Attachments: 2869.txt CassandraStorage appears to have threading issues along the lines of those described at http://pig.markmail.org/message/oz7oz2x2dwp66eoz due to the sharing of the UDFContext. I believe the fix lies in implementing {code} public void setStoreFuncUDFContextSignature(String signature) { } {code} and then using that signature when getting the UDFContext. From the Pig manual: {quote} setStoreFunc!UDFContextSignature(): This method will be called by Pig both in the front end and back end to pass a unique signature to the Storer. The signature can be used to store into the UDFContext any information which the Storer needs to store between various method invocations in the front end and back end. The default implementation in StoreFunc has an empty body. This method will be called before other methods. {quote} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064005#comment-13064005 ] Pavel Yaskevich edited comment on CASSANDRA-47 at 7/12/11 5:09 PM: --- Thanks for your report! This will be fixed in next patch. was (Author: xedin): Thanks for you report! This will be fixed in next patch. SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064005#comment-13064005 ] Pavel Yaskevich commented on CASSANDRA-47: -- Thanks for you report! This will be fixed in next patch. SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2888) CQL support for JDBC DatabaseMetaData
[ https://issues.apache.org/jira/browse/CASSANDRA-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-2888: - Assignee: Rick Shaw I think this is already done in the latest driver source. CQL support for JDBC DatabaseMetaData - Key: CASSANDRA-2888 URL: https://issues.apache.org/jira/browse/CASSANDRA-2888 Project: Cassandra Issue Type: Improvement Components: Drivers Affects Versions: 0.8.1 Environment: J2SE 1.6.0_22 x64 on Fedora 15 Reporter: Dave Carlson Assignee: Rick Shaw Priority: Minor Labels: cql, newbie Original Estimate: 96h Remaining Estimate: 96h In order to increase the drop-in capability of CQL to existing JDBC app bases, CQL must be updated to include at least semi-valid responses to the JDBC metadata portion. without enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) Resource has error: java.lang.UnsupportedOperationException: method not supported ... with enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) org.apache.cassandra.cql.jdbc.CassandraConnection@1915470e -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1145689 - /cassandra/branches/cassandra-0.8/CHANGES.txt
Author: jbellis Date: Tue Jul 12 17:32:43 2011 New Revision: 1145689 URL: http://svn.apache.org/viewvc?rev=1145689view=rev Log: belatedly add #1969 to CHANGES Modified: cassandra/branches/cassandra-0.8/CHANGES.txt Modified: cassandra/branches/cassandra-0.8/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1145689r1=1145688r2=1145689view=diff == --- cassandra/branches/cassandra-0.8/CHANGES.txt (original) +++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jul 12 17:32:43 2011 @@ -212,6 +212,7 @@ * add a server-wide cap on measured memtable memory usage and aggressively flush to keep under that threshold (CASSANDRA-2006) * add unified UUIDType (CASSANDRA-2233) + * add off-heap row cache support (CASSANDRA-1969) 0.7.5
[jira] [Resolved] (CASSANDRA-2856) log unavailableexception details at debug level
[ https://issues.apache.org/jira/browse/CASSANDRA-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2856. --- Resolution: Fixed Reviewer: slebresne committed log unavailableexception details at debug level --- Key: CASSANDRA-2856 URL: https://issues.apache.org/jira/browse/CASSANDRA-2856 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Jonathan Ellis Priority: Trivial Fix For: 0.7.8 Attachments: 2856.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2845) Cassandra uses 100% system CPU on Ubuntu Natty (11.04)
[ https://issues.apache.org/jira/browse/CASSANDRA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2845: -- Priority: Critical (was: Major) Affects Version/s: (was: 0.8.1) Fix Version/s: 0.8.2 Cassandra uses 100% system CPU on Ubuntu Natty (11.04) -- Key: CASSANDRA-2845 URL: https://issues.apache.org/jira/browse/CASSANDRA-2845 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0 Environment: Default install of Ubuntu 11.04 Reporter: Steve Corona Assignee: paul cannon Priority: Critical Fix For: 0.8.2 Step 1. Boot up a brand new, default Ubuntu 11.04 Server install Step 2. Install Cassandra from Apache APT Respository (deb http://www.apache.org/dist/cassandra/debian 08x main) Step 3. apt-get install cassandra, as soon as it cassandra starts it will freeze the machine What's happening is that as soon as cassandra starts up it immediately sucks up 100% of CPU and starves the machine. This effectively bricks the box until you boot into single user mode and disable the cassandra init.d script. Under htop, the CPU usage shows up as system cpu, not user. The machine I'm testing this on is a Quad-Core Sandy Bridge w/ 16GB of Memory, so it's not a system resource issue. I've also tested this on completely different hardware (Dual 64-Bit Xeons AMD X4) and it has the same effect. Ubuntu 10.10 does not exhibit the same issue. I have only tested 0.8 and 0.8.1. root@cassandra01:/# java -version java version 1.6.0_22 OpenJDK Runtime Environment (IcedTea6 1.10.2) (6b22-1.10.2-0ubuntu1~11.04.1) OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) root@cassandra:/# uname -a Linux cassandra01 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux /proc/cpu Intel(R) Xeon(R) CPU E31270 @ 3.40GHz /proc/meminfo MemTotal: 16459776 kB MemFree:14190708 kB -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064048#comment-13064048 ] Jonathan Ellis commented on CASSANDRA-2879: --- where is the -1 coming from in BF.serializedSize? Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064052#comment-13064052 ] Pavel Yaskevich commented on CASSANDRA-2879: I've double checked CF/columns and ColumnIndexer serializedSize() methods before adding -1 to BF, because size of RowIndex.serializedSize() + cf.serializedSize() was always generating one byte bigger size than actual data size computed by endPosition - (sizePosition + 8). Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2888) CQL support for JDBC DatabaseMetaData
[ https://issues.apache.org/jira/browse/CASSANDRA-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064061#comment-13064061 ] Rick Shaw commented on CASSANDRA-2888: -- The title of this JIRA and the messages you provide seem to be mismatched? What exactly are you trying to do? What version of the Driver are you using? Are you using some sort of call that expects a {{DataSource}}? This is not supported yet in even the most current available version in the trunk. Support for a driver provided {{DataSource}} has been proposed as part of the 1.1 roadmap. But the JIRA seems to be looking for support of {{DatabaseMetaData}}. This too is not currently supported, but it is optional and we reply with the proper {{SQLException}} in the latest version of the driver in the trunk (build from source). Do you think this support is required? What would you use it for? It is currently thought of as not required but a nice-to-have. Please look at CASSANDRA-2876 and put it in perspective with the current roadmap. The roadmap is not cast in stone. But I can tell you for the little I have looked at it, that it is a big job to support any substantive features in this data structure. CQL support for JDBC DatabaseMetaData - Key: CASSANDRA-2888 URL: https://issues.apache.org/jira/browse/CASSANDRA-2888 Project: Cassandra Issue Type: Improvement Components: Drivers Affects Versions: 0.8.1 Environment: J2SE 1.6.0_22 x64 on Fedora 15 Reporter: Dave Carlson Assignee: Rick Shaw Priority: Minor Labels: cql, newbie Original Estimate: 96h Remaining Estimate: 96h In order to increase the drop-in capability of CQL to existing JDBC app bases, CQL must be updated to include at least semi-valid responses to the JDBC metadata portion. without enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) Resource has error: java.lang.UnsupportedOperationException: method not supported ... with enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) org.apache.cassandra.cql.jdbc.CassandraConnection@1915470e -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2888) CQL support for JDBC DatabaseMetaData
[ https://issues.apache.org/jira/browse/CASSANDRA-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064069#comment-13064069 ] Dave Carlson commented on CASSANDRA-2888: - I'm using the 1.0.3 version right now but I'll test it against the latest. I don't believe this fits the roadmap (this is still in DriverManager land). The enhancement in my mind would provide compatibility - I wouldn't expect any complicated features. I'm currently evaluating the use of Cassandra as a drop-in from what is currently an Oracle installation - heavily tied to the use of JDBC with calls to Connection.getMetaData(). In the 1.0.3 source that raised an exception which barred CQL from use as a drop-in. To that end, I did write some code that does allow it to be dropped in to that situation successfully, but made a lot of cowboy assumptions along the way. Basically, I created a CassandraMetaData class that CassandraConnection.getMetaData creates that supports getURL() and hardcoded get{DatabaseProduct,Driver}{Name,Version}. I walked through the methods to support by popping it into DbVisualizer and progressing through the exceptions - of course nowhere complete for DbVisualizer functionality but good enough for the drop-in to the installation I support. CQL support for JDBC DatabaseMetaData - Key: CASSANDRA-2888 URL: https://issues.apache.org/jira/browse/CASSANDRA-2888 Project: Cassandra Issue Type: Improvement Components: Drivers Affects Versions: 0.8.1 Environment: J2SE 1.6.0_22 x64 on Fedora 15 Reporter: Dave Carlson Assignee: Rick Shaw Priority: Minor Labels: cql, newbie Original Estimate: 96h Remaining Estimate: 96h In order to increase the drop-in capability of CQL to existing JDBC app bases, CQL must be updated to include at least semi-valid responses to the JDBC metadata portion. without enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) Resource has error: java.lang.UnsupportedOperationException: method not supported ... with enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) org.apache.cassandra.cql.jdbc.CassandraConnection@1915470e -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064078#comment-13064078 ] Jonathan Ellis commented on CASSANDRA-2879: --- It sounds like there's a bug in either RI or CF size computation. We need to figure out where that's coming from and address it there. Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
Avoids having replicate on write tasks stacking up at CL.ONE Key: CASSANDRA-2889 URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Reporter: Sylvain Lebresne Assignee: Sylvain Lebresne The counter design involves a read on the first replica during a write. At CL.ONE, this read is not involved in the latency of the operation (the write is acknowledged before). This means it is fairly easy to insert too quickly at CL.ONE and have the replicate on write tasks falling behind. The goal of this ticket is to protect against that. An option could be to bound the replicate on write task queue so that write start to block once we have too much of those in the queue. Another option could be to drop the oldest tasks when they are too old, but it's probably a more unsafe option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1145728 - /cassandra/branches/cassandra-0.8/conf/cassandra.yaml
Author: jbellis Date: Tue Jul 12 19:20:14 2011 New Revision: 1145728 URL: http://svn.apache.org/viewvc?rev=1145728view=rev Log: add example of commitlog_sync_batch_window_in_ms to .yaml patch by Wojciech Meler for CASSANDRA-2280 Modified: cassandra/branches/cassandra-0.8/conf/cassandra.yaml Modified: cassandra/branches/cassandra-0.8/conf/cassandra.yaml URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/conf/cassandra.yaml?rev=1145728r1=1145727r2=1145728view=diff == --- cassandra/branches/cassandra-0.8/conf/cassandra.yaml (original) +++ cassandra/branches/cassandra-0.8/conf/cassandra.yaml Tue Jul 12 19:20:14 2011 @@ -83,13 +83,16 @@ commitlog_rotation_threshold_in_mb: 128 # commitlog_sync may be either periodic or batch. # When in batch mode, Cassandra won't ack writes until the commit log # has been fsynced to disk. It will wait up to -# CommitLogSyncBatchWindowInMS milliseconds for other writes, before +# commitlog_sync_batch_window_in_ms milliseconds for other writes, before # performing the sync. -commitlog_sync: periodic - +# +# commitlog_sync: batch +# commitlog_sync_batch_window_in_ms: 50 +# # the other option is periodic where writes may be acked immediately # and the CommitLog is simply synced every commitlog_sync_period_in_ms # milliseconds. +commitlog_sync: periodic commitlog_sync_period_in_ms: 1 # any class that implements the SeedProvider interface and has a constructor that takes a MapString, String of
svn commit: r1145729 - /cassandra/branches/cassandra-0.8/examples/client_only/conf/cassandra.yaml
Author: jbellis Date: Tue Jul 12 19:21:48 2011 New Revision: 1145729 URL: http://svn.apache.org/viewvc?rev=1145729view=rev Log: r/m commitlog_rotation_threshold_in_mb from examples/client_only Modified: cassandra/branches/cassandra-0.8/examples/client_only/conf/cassandra.yaml Modified: cassandra/branches/cassandra-0.8/examples/client_only/conf/cassandra.yaml URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/examples/client_only/conf/cassandra.yaml?rev=1145729r1=1145728r2=1145729view=diff == --- cassandra/branches/cassandra-0.8/examples/client_only/conf/cassandra.yaml (original) +++ cassandra/branches/cassandra-0.8/examples/client_only/conf/cassandra.yaml Tue Jul 12 19:21:48 2011 @@ -77,9 +77,6 @@ commitlog_directory: /var/lib/cassandra/ # saved caches saved_caches_directory: /var/lib/cassandra/saved_caches -# Size to allow commitlog to grow to before creating a new segment -commitlog_rotation_threshold_in_mb: 128 - # commitlog_sync may be either periodic or batch. # When in batch mode, Cassandra won't ack writes until the commit log # has been fsynced to disk. It will wait up to
[jira] [Commented] (CASSANDRA-2771) Remove commitlog_rotation_threshold_in_mb
[ https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064086#comment-13064086 ] Jonathan Ellis commented on CASSANDRA-2771: --- Kirk: removed the offending lines, thanks! Remove commitlog_rotation_threshold_in_mb -- Key: CASSANDRA-2771 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771 Project: Cassandra Issue Type: Improvement Reporter: Patricio Echague Assignee: Patricio Echague Priority: Minor Labels: commitlog Fix For: 1.0 Attachments: CASSANDRA-2771-2-trunk.txt, CASSANDRA-2771-3-trunk.txt Remove the commitlog segment size config setting, nobody has ever changed it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1145731 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java src/java/org/apache/cassandra/hadoop/ConfigHelper.java
Author: jbellis Date: Tue Jul 12 19:26:11 2011 New Revision: 1145731 URL: http://svn.apache.org/viewvc?rev=1145731view=rev Log: add KeyRangeoption to Hadoop inputformat patch by Mck SembWever; reviewed by jbellis for CASSANDRA-1125 Modified: cassandra/branches/cassandra-0.8/CHANGES.txt cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ConfigHelper.java Modified: cassandra/branches/cassandra-0.8/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1145731r1=1145730r2=1145731view=diff == --- cassandra/branches/cassandra-0.8/CHANGES.txt (original) +++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jul 12 19:26:11 2011 @@ -24,6 +24,7 @@ * add option to specify limit for get_slice in the CLI (CASSANDRA-2646) * decrease HH page size (CASSANDRA-2832) * reset cli keyspace after dropping the current one (CASSANDRA-2763) + * add KeyRange option to Hadoop inputformat (CASSANDRA-1125) 0.8.1 Modified: cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java?rev=1145731r1=1145730r2=1145731view=diff == --- cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java (original) +++ cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java Tue Jul 12 19:26:11 2011 @@ -35,8 +35,11 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.cassandra.db.IColumn; +import org.apache.cassandra.dht.IPartitioner; +import org.apache.cassandra.dht.Range; import org.apache.cassandra.thrift.Cassandra; import org.apache.cassandra.thrift.InvalidRequestException; +import org.apache.cassandra.thrift.KeyRange; import org.apache.cassandra.thrift.TokenRange; import org.apache.cassandra.thrift.TBinaryProtocol; import org.apache.hadoop.conf.Configuration; @@ -102,10 +105,44 @@ public class ColumnFamilyInputFormat ext try { ListFutureListInputSplit splitfutures = new ArrayListFutureListInputSplit(); +KeyRange jobKeyRange = ConfigHelper.getInputKeyRange(conf); +IPartitioner partitioner = null; +Range jobRange = null; +if (jobKeyRange != null) +{ +partitioner = ConfigHelper.getPartitioner(context.getConfiguration()); +assert partitioner.preservesOrder() : ConfigHelper.setInputKeyRange(..) can only be used with a order preserving paritioner; +assert jobKeyRange.start_key == null : only start_token supported; +assert jobKeyRange.end_key == null : only end_token supported; +jobRange = new Range(partitioner.getTokenFactory().fromString(jobKeyRange.start_token), + partitioner.getTokenFactory().fromString(jobKeyRange.end_token), + partitioner); +} + for (TokenRange range : masterRangeNodes) { +if (jobRange == null) +{ // for each range, pick a live owner and ask it to compute bite-sized splits splitfutures.add(executor.submit(new SplitCallable(range, conf))); +} +else +{ +Range dhtRange = new Range(partitioner.getTokenFactory().fromString(range.start_token), + partitioner.getTokenFactory().fromString(range.end_token), + partitioner); + +if (dhtRange.intersects(jobRange)) +{ +SetRange intersections = dhtRange.intersectionWith(jobRange); +assert intersections.size() == 1 : wrapping ranges not yet supported; +Range intersection = intersections.iterator().next(); +range.start_token = partitioner.getTokenFactory().toString(intersection.left); +range.end_token = partitioner.getTokenFactory().toString(intersection.right); +// for each range, pick a live owner and ask it to compute bite-sized splits +splitfutures.add(executor.submit(new SplitCallable(range, conf))); +} +} } // wait until we have all the results back Modified: cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ConfigHelper.java URL:
[jira] [Created] (CASSANDRA-2890) Randomize (to some extend) the choice of the first replica for counter increment
Randomize (to some extend) the choice of the first replica for counter increment Key: CASSANDRA-2890 URL: https://issues.apache.org/jira/browse/CASSANDRA-2890 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Reporter: Sylvain Lebresne Assignee: Sylvain Lebresne Priority: Minor Right now, we choose the first replica for a counter increments based solely on what the snitch returns. If the clients requests are well balanced over the cluster and the snitch not ill configured, this should not be a problem, but this is probably too strong an assumption to make. The goal of this ticket is to change this to choose a random replica in the current data center instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064101#comment-13064101 ] Sylvain Lebresne commented on CASSANDRA-47: --- bq. As I wrote before - currently to check real size of the file (tested only on linux because OS X FS saves empty blocks to the disk for some reason) you need to get a block count using 'ls -alhs', current patch reserves an empty space for each chunk because we need to do seeks while we write data using SSTableWriter. Yeah, I really think we shouldn't do that (i.e, have empty space between the compressed chunks). I'm happy to learn that linux (or at least whatever file system you are using, I haven't tried the patch on linux yet) is smart enough to avoid allocating empty blocks but we shouldn't rely on this. I bet not all file system do that (osx seems to prove that and I'm not sure all linux FS does this) and anyway if you transfer the sstables or tar them or anything, it'll still be more inefficient than necessary (because the file still *is* of the size of the uncompressed data). We're also losing some space even on linux depending on what the actual FS block size is (not a big deal, but this can add up). So I think we really need to change the index (and key cache) to store the offset in compressed data. Imho, the simplest way would be to instead of having in the index the key followed by the offset, to have for compressed file, the key, then the position of the chunk in the compressed file, then the offset in the uncompressed chunk. Another thing is that we will need that to be optional (if only because we cannot expect people to trust this from day one). Don't get me wrong, it's nice to have a first prototype to have an idea of what we're talking about, but I just wanted to mention this because it's probably easier to take that into account sooner than later (I also suspect we may be able to factor out some of the code of BRAF and CDF, but I haven't look too closely so maybe not). SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064106#comment-13064106 ] Pavel Yaskevich commented on CASSANDRA-47: -- Totally agree with you, Sylvain, I'm working in that direction. SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2280) Request specific column families using StreamIn
[ https://issues.apache.org/jira/browse/CASSANDRA-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064119#comment-13064119 ] Hudson commented on CASSANDRA-2280: --- Integrated in Cassandra-0.8 #214 (See [https://builds.apache.org/job/Cassandra-0.8/214/]) add example of commitlog_sync_batch_window_in_ms to .yaml patch by Wojciech Meler for CASSANDRA-2280 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1145728 Files : * /cassandra/branches/cassandra-0.8/conf/cassandra.yaml Request specific column families using StreamIn --- Key: CASSANDRA-2280 URL: https://issues.apache.org/jira/browse/CASSANDRA-2280 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Jonathan Ellis Fix For: 1.0 Attachments: 0001-Allow-specific-column-families-to-be-requested-for-str.txt, 0001-Allow-specific-column-families-to-be-requested-for-str.txt, 2280-v3.txt, 2280-v4.txt, 2280-v5.txt StreamIn.requestRanges only specifies a keyspace, meaning that requesting a range will request it for all column families: if you have a large number of CFs, this can cause quite a headache. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1125) Filter out ColumnFamily rows that aren't part of the query (using a KeyRange)
[ https://issues.apache.org/jira/browse/CASSANDRA-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064118#comment-13064118 ] Hudson commented on CASSANDRA-1125: --- Integrated in Cassandra-0.8 #214 (See [https://builds.apache.org/job/Cassandra-0.8/214/]) add KeyRangeoption to Hadoop inputformat patch by Mck SembWever; reviewed by jbellis for CASSANDRA-1125 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1145731 Files : * /cassandra/branches/cassandra-0.8/CHANGES.txt * /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ConfigHelper.java * /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java Filter out ColumnFamily rows that aren't part of the query (using a KeyRange) - Key: CASSANDRA-1125 URL: https://issues.apache.org/jira/browse/CASSANDRA-1125 Project: Cassandra Issue Type: New Feature Components: Hadoop Reporter: Jeremy Hanna Assignee: Mck SembWever Priority: Minor Fix For: 0.8.2 Attachments: 1125-formatted.txt, 1125-v3.txt, CASSANDRA-1125.patch, CASSANDRA-1125.patch Currently, when running a MapReduce job against data in a Cassandra data store, it reads through all the data for a particular ColumnFamily. This could be optimized to only read through those rows that have to do with the query. It's a small change but wanted to put it in Jira so that it didn't fall through the cracks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2868) Native Memory Leak
[ https://issues.apache.org/jira/browse/CASSANDRA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064128#comment-13064128 ] Jonathan Ellis commented on CASSANDRA-2868: --- We call getLastGcInfo several times a second. http://twitter.com/#!/kimchy/status/90861039930970113 You could try turning GCInspector methods into a no-op and see if that makes it go away. Native Memory Leak -- Key: CASSANDRA-2868 URL: https://issues.apache.org/jira/browse/CASSANDRA-2868 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.6 Reporter: Daniel Doubleday Priority: Minor We have memory issues with long running servers. These have been confirmed by several users in the user list. That's why I report. The memory consumption of the cassandra java process increases steadily until it's killed by the os because of oom (with no swap) Our server is started with -Xmx3000M and running for around 23 days. pmap -x shows Total SST: 1961616 (mem mapped data and index files) Anon RSS: 6499640 Total RSS: 8478376 This shows that 3G are 'overallocated'. We will use BRAF on one of our less important nodes to check wether it is related to mmap and report back. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2891) ColumnFamilySerializer is a mess and a potential source of bugs and inefficiency
ColumnFamilySerializer is a mess and a potential source of bugs and inefficiency Key: CASSANDRA-2891 URL: https://issues.apache.org/jira/browse/CASSANDRA-2891 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Jonathan Ellis Priority: Minor We use serializeForSSTable for on-disk storage, but serialize for network messages. This is confusing and it's easy to do the wrong thing. Example: SerializingCache calls serialize, which is wasteful since the on-disk version is smaller. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-2879: --- Attachment: CASSANDRA-2879-v2.patch I have overlooked that append(DecoratedKey, ColumnFamily) does not call serialize where boolean is added to the header of the CF, so now I've removed -1 from BFS and added CF.serializedSizeForSSTable() Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: CASSANDRA-2879-v2.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2869) CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues
[ https://issues.apache.org/jira/browse/CASSANDRA-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064149#comment-13064149 ] Jeremy Hanna commented on CASSANDRA-2869: - Yes. I was about to post an updated patch last night but got sidetracked. Do you mind removing that if it's otherwise good to go? Otherwise I can do that later today. CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues Key: CASSANDRA-2869 URL: https://issues.apache.org/jira/browse/CASSANDRA-2869 Project: Cassandra Issue Type: Bug Components: Contrib Affects Versions: 0.7.2 Reporter: Grant Ingersoll Assignee: Jeremy Hanna Attachments: 2869.txt CassandraStorage appears to have threading issues along the lines of those described at http://pig.markmail.org/message/oz7oz2x2dwp66eoz due to the sharing of the UDFContext. I believe the fix lies in implementing {code} public void setStoreFuncUDFContextSignature(String signature) { } {code} and then using that signature when getting the UDFContext. From the Pig manual: {quote} setStoreFunc!UDFContextSignature(): This method will be called by Pig both in the front end and back end to pass a unique signature to the Storer. The signature can be used to store into the UDFContext any information which the Storer needs to store between various method invocations in the front end and back end. The default implementation in StoreFunc has an empty body. This method will be called before other methods. {quote} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-2879: --- Attachment: CASSANDRA-2879-DBConstants-names-refactoring.patch removes underscores from DBConstants names, apply after v2. Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: CASSANDRA-2879-DBConstants-names-refactoring.patch, CASSANDRA-2879-v2.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2879: -- Attachment: 2879-v3.txt v3 attached that cleans up some special cases. I also fixed serializedSizeForSSTable to not include the 4 bytes for the CF id, which is not written to disk. But, this change breaks the tests spectacularly, so clearly it is now undercounting for some reason I do not understand. Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: 2879-v3.txt, CASSANDRA-2879-DBConstants-names-refactoring.patch, CASSANDRA-2879-v2.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-2879: --- Attachment: (was: CASSANDRA-2879-DBConstants-names-refactoring.patch) Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: 2879-v3.txt, CASSANDRA-2879-v2.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2892) Don't replicate_on_write with RF=1
Don't replicate_on_write with RF=1 Key: CASSANDRA-2892 URL: https://issues.apache.org/jira/browse/CASSANDRA-2892 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Reporter: Sylvain Lebresne Assignee: Sylvain Lebresne Priority: Trivial Fix For: 0.8.2 For counters with RF=1, we still do a read to replicate, even though there is nothing to replicate it too. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064184#comment-13064184 ] Terje Marthinussen commented on CASSANDRA-47: - Sparse files is indeed a filesystem feature. It is standard in extfs, xfs and probably most of the common filesystems on linux (not considering FAT :)) It is not standard in NTFS, but applications can enable sparse files on a file by file basis there. But my bet, without checking, is that this is not really supported by java... For OSX, it is not supported on HFS, but is supposedly supported on UFS (but I have not tested on OSX, so only read this). Anyway, I agree (from experience) stay away from using sparseness in files if possible. It's a portability headache and it tends to cause major issues when it fails. SSTable compression --- Key: CASSANDRA-47 URL: https://issues.apache.org/jira/browse/CASSANDRA-47 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Labels: compression Fix For: 1.0 Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar We should be able to do SSTable compression which would trade CPU for I/O (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2753) Capture the max client timestamp for an SSTable
[ https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Doubleday reopened CASSANDRA-2753: - While looking into CASSANDRA-2498 I think I found a problem in SuperColumn.maxTimestamp. One of the tests is writing a single deleted SC in a SST and the maxTimestamp is negativ which broke my superseding code. See attached patch Capture the max client timestamp for an SSTable --- Key: CASSANDRA-2753 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Alan Liang Assignee: Alan Liang Priority: Minor Fix For: 1.0 Attachments: 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V2.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V3.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to.patch, 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch, supercolumn.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2753) Capture the max client timestamp for an SSTable
[ https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Doubleday updated CASSANDRA-2753: Attachment: supercolumn.patch Capture the max client timestamp for an SSTable --- Key: CASSANDRA-2753 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Alan Liang Assignee: Alan Liang Priority: Minor Fix For: 1.0 Attachments: 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V2.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V3.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to.patch, 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch, supercolumn.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2888) CQL support for JDBC DatabaseMetaData
[ https://issues.apache.org/jira/browse/CASSANDRA-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064216#comment-13064216 ] Rick Shaw commented on CASSANDRA-2888: -- Right now (1.0.4-SNAPSHOT) we just return NULL for this call. Please be specific as to what you think is REQUIRED to work with your toolset (What is it BTW?) and we can look into providing a simplistic sparse implementation of static items. I believe the dynamic stuff is way beyond the scope of what can be done in the midst of the other items in the roadmap for 1.1. We can tackle the dynamic stuff when CQL decides how it will format the data in a structured ({{CQLResult}}) response. CQL support for JDBC DatabaseMetaData - Key: CASSANDRA-2888 URL: https://issues.apache.org/jira/browse/CASSANDRA-2888 Project: Cassandra Issue Type: Improvement Components: Drivers Affects Versions: 0.8.1 Environment: J2SE 1.6.0_22 x64 on Fedora 15 Reporter: Dave Carlson Assignee: Rick Shaw Priority: Minor Labels: cql, newbie Original Estimate: 96h Remaining Estimate: 96h In order to increase the drop-in capability of CQL to existing JDBC app bases, CQL must be updated to include at least semi-valid responses to the JDBC metadata portion. without enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) Resource has error: java.lang.UnsupportedOperationException: method not supported ... with enhancement: com.largecompany.JDBCManager.getConnection(vague Cassandra JNDI pointer) org.apache.cassandra.cql.jdbc.CassandraConnection@1915470e -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2893) Add row-level isolation
Add row-level isolation --- Key: CASSANDRA-2893 URL: https://issues.apache.org/jira/browse/CASSANDRA-2893 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Priority: Minor This could be done using something an AtomicReference and something like http://code.google.com/p/pcollections/ but with a SortedMap (for the columnfamily.columns collection). So we probably need to write our own. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable
[ https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064230#comment-13064230 ] Alan Liang commented on CASSANDRA-2753: --- Daniel, Which test does this break? Can you elaborate? Capture the max client timestamp for an SSTable --- Key: CASSANDRA-2753 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Alan Liang Assignee: Alan Liang Priority: Minor Fix For: 1.0 Attachments: 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V2.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V3.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to.patch, 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch, supercolumn.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1145818 - in /cassandra/trunk: ./ src/java/org/apache/cassandra/db/ src/java/org/apache/cassandra/io/sstable/ src/java/org/apache/cassandra/utils/
Author: jbellis Date: Tue Jul 12 23:33:37 2011 New Revision: 1145818 URL: http://svn.apache.org/viewvc?rev=1145818view=rev Log: optimize away seek when compacting wide rows patch by Pavel Yaskevich and jbellis for CASSANDRA-2879 Modified: cassandra/trunk/CHANGES.txt cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java cassandra/trunk/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java cassandra/trunk/src/java/org/apache/cassandra/utils/BloomFilter.java cassandra/trunk/src/java/org/apache/cassandra/utils/BloomFilterSerializer.java Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1145818r1=1145817r2=1145818view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Tue Jul 12 23:33:37 2011 @@ -11,6 +11,7 @@ * restrict repair streaming to specific columnfamilies (CASSANDRA-2280) * don't bother persisting columns shadowed by a row tombstone (CASSANDRA-2589) * reset CF and SC deletion times after gc_grace (CASSANDRA-2317) + * optimize away seek when compacting wide rows (CASSANDRA-2879) 0.8.2 Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java?rev=1145818r1=1145817r2=1145818view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java Tue Jul 12 23:33:37 2011 @@ -306,10 +306,15 @@ public class ColumnFamily extends Abstra public long serializedSize() { -int size = boolSize_ // bool - + intSize_ // id - + intSize_ // local deletion time - + longSize_ // client deltion time +return boolSize_ // nullness bool + + intSize_ // id + + serializedSizeForSSTable(); +} + +public long serializedSizeForSSTable() +{ +int size = intSize_ // local deletion time + + longSize_ // client deletion time + intSize_; // column count for (IColumn column : columns.values()) size += column.serializedSize(); Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java?rev=1145818r1=1145817r2=1145818view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java Tue Jul 12 23:33:37 2011 @@ -102,9 +102,9 @@ public class ColumnFamilySerializer impl dos.writeLong(columnFamily.getMarkedForDeleteAt()); } -public int serializeWithIndexes(ColumnFamily columnFamily, DataOutput dos) +public int serializeWithIndexes(ColumnFamily columnFamily, ColumnIndexer.RowHeader index, DataOutput dos) { -ColumnIndexer.serialize(columnFamily, dos); +ColumnIndexer.serialize(index, dos); return serializeForSSTable(columnFamily, dos); } Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java?rev=1145818r1=1145817r2=1145818view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java Tue Jul 12 23:33:37 2011 @@ -22,6 +22,7 @@ import java.io.DataOutput; import java.io.IOError; import java.io.IOException; import java.util.ArrayList; +import java.util.Collections; import java.util.List; import org.apache.cassandra.config.DatabaseDescriptor; @@ -39,15 +40,15 @@ public class ColumnIndexer /** * Given a column family this, function creates an in-memory structure that represents the * column index for the column family, and subsequently writes it to disk. + * * @param columns Column family to create index for * @param dos data output stream -* @throws IOException */ public static void serialize(IIterableColumns columns, DataOutput dos) { try { -serializeInternal(columns, dos); +writeIndex(serialize(columns), dos); } catch (IOException e) { @@ -55,24 +56,41 @@ public class
svn commit: r1145819 - /cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java
Author: jbellis Date: Tue Jul 12 23:37:01 2011 New Revision: 1145819 URL: http://svn.apache.org/viewvc?rev=1145819view=rev Log: fix supercolumn maxtimestamp bug patch by Daniel Doubleday; reviewed by jbellis for CASSANDRA-2753 Modified: cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java Modified: cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java?rev=1145819r1=1145818r2=1145819view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java Tue Jul 12 23:37:01 2011 @@ -138,7 +138,8 @@ public class SuperColumn extends Abstrac long maxTimestamp = Long.MIN_VALUE; for (IColumn subColumn : getSubColumns()) maxTimestamp = Math.max(maxTimestamp, subColumn.maxTimestamp()); -return maxTimestamp; + +return Math.max(maxTimestamp, getMarkedForDeleteAt()); } public long mostRecentLiveChangeAt()
[jira] [Resolved] (CASSANDRA-2753) Capture the max client timestamp for an SSTable
[ https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2753. --- Resolution: Fixed good find, Daniel. committed. as Alan implies it would be nice to have a test to catch this too. Capture the max client timestamp for an SSTable --- Key: CASSANDRA-2753 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Alan Liang Assignee: Alan Liang Priority: Minor Fix For: 1.0 Attachments: 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V2.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V3.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to.patch, 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch, supercolumn.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating an index, incremental snapshotting can hang
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064236#comment-13064236 ] Sylvain Lebresne commented on CASSANDRA-2872: - One option could be to have some kind of index generation that we would persist in the system tables. What I mean here is that if you create a index on say column 'birthdate' for some CF test, the index would start being called test.birthdate.1 (or just test.birthdate for compatibility sake), then if you drop it and recreate it, it would be called test.birthdate.2. That way, we would avoid the name clash during the incremental backup. While dropping and recreating an index, incremental snapshotting can hang -- Key: CASSANDRA-2872 URL: https://issues.apache.org/jira/browse/CASSANDRA-2872 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Reporter: Sylvain Lebresne Priority: Minor When creating a hard link (at list with JNA), link() hang if the target of the link already exists. In theory though, we should not hit that situation because we use a new directory for each manual snapshot and the generation number of the sstables should prevent this from hapenning with increment snapshot. However, when you drop, then recreate a secondary index, if the sstables are deleted after the drop and before we recreate the index, the recreated index sstables will start with a generation to 0. Thus, when we start backuping them incrementally, it will conflict with the sstables of the previously dropped index. First, we should check for the target existance because calling link() to at least avoid hanging. But then we must make sure that when we drop, then recreate an index, we will either not name the sstables the same way or the incremental snapshot use a different directory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating an index, incremental snapshotting can hang
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064240#comment-13064240 ] Jonathan Ellis commented on CASSANDRA-2872: --- I really don't like messing with the name. That feels like the implementation is leaking too much into something user-visible. While dropping and recreating an index, incremental snapshotting can hang -- Key: CASSANDRA-2872 URL: https://issues.apache.org/jira/browse/CASSANDRA-2872 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Reporter: Sylvain Lebresne Priority: Minor When creating a hard link (at list with JNA), link() hang if the target of the link already exists. In theory though, we should not hit that situation because we use a new directory for each manual snapshot and the generation number of the sstables should prevent this from hapenning with increment snapshot. However, when you drop, then recreate a secondary index, if the sstables are deleted after the drop and before we recreate the index, the recreated index sstables will start with a generation to 0. Thus, when we start backuping them incrementally, it will conflict with the sstables of the previously dropped index. First, we should check for the target existance because calling link() to at least avoid hanging. But then we must make sure that when we drop, then recreate an index, we will either not name the sstables the same way or the incremental snapshot use a different directory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2829) always flush memtables
[ https://issues.apache.org/jira/browse/CASSANDRA-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2829: -- Priority: Major (was: Minor) Affects Version/s: (was: 0.7.6) 0.7.0 Fix Version/s: 0.7.8 always flush memtables -- Key: CASSANDRA-2829 URL: https://issues.apache.org/jira/browse/CASSANDRA-2829 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.0 Reporter: Aaron Morton Assignee: Aaron Morton Fix For: 0.7.8 Attachments: 0001-2829-unit-test.patch, 0002-2829.patch Only dirty Memtables are flushed, and so only dirty memtables are used to discard obsolete commit log segments. This can result it log segments not been deleted even though the data has been flushed. Was using a 3 node 0.7.6-2 AWS cluster (DataStax AMI's) with pre 0.7 data loaded and a running application working against the cluster. Did a rolling restart and then kicked off a repair, one node filled up the commit log volume with 7GB+ of log data, there was about 20 hours of log files. {noformat} $ sudo ls -lah commitlog/ total 6.9G drwx-- 2 cassandra cassandra 12K 2011-06-24 20:38 . drwxr-xr-x 3 cassandra cassandra 4.0K 2011-06-25 01:47 .. -rw--- 1 cassandra cassandra 129M 2011-06-24 01:08 CommitLog-1308876643288.log -rw--- 1 cassandra cassandra 28 2011-06-24 20:47 CommitLog-1308876643288.log.header -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 01:36 CommitLog-1308877711517.log -rw-r--r-- 1 cassandra cassandra 28 2011-06-24 20:47 CommitLog-1308877711517.log.header -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 02:20 CommitLog-1308879395824.log -rw-r--r-- 1 cassandra cassandra 28 2011-06-24 20:47 CommitLog-1308879395824.log.header ... -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 20:38 CommitLog-1308946745380.log -rw-r--r-- 1 cassandra cassandra 36 2011-06-24 20:47 CommitLog-1308946745380.log.header -rw-r--r-- 1 cassandra cassandra 112M 2011-06-24 20:54 CommitLog-1308947888397.log -rw-r--r-- 1 cassandra cassandra 44 2011-06-24 20:47 CommitLog-1308947888397.log.header {noformat} The user KS has 2 CF's with 60 minute flush times. System KS had the default settings which is 24 hours. Will create another ticket see if these can be reduced or if it's something users should do, in this case it would not have mattered. I grabbed the log headers and used the tool in CASSANDRA-2828 and most of the segments had the system CF's marked as dirty. {noformat} $ bin/logtool dirty /tmp/logs/commitlog/ Not connected to a server, Keyspace and Column Family names are not available. /tmp/logs/commitlog/CommitLog-1308876643288.log.header Keyspace Unknown: Cf id 0: 444 /tmp/logs/commitlog/CommitLog-1308877711517.log.header Keyspace Unknown: Cf id 1: 68848763 ... /tmp/logs/commitlog/CommitLog-1308944451460.log.header Keyspace Unknown: Cf id 1: 61074 /tmp/logs/commitlog/CommitLog-1308945597471.log.header Keyspace Unknown: Cf id 1000: 43175492 Cf id 1: 108483 /tmp/logs/commitlog/CommitLog-1308946745380.log.header Keyspace Unknown: Cf id 1000: 239223 Cf id 1: 172211 /tmp/logs/commitlog/CommitLog-1308947888397.log.header Keyspace Unknown: Cf id 1001: 57595560 Cf id 1: 816960 Cf id 1000: 0 {noformat} CF 0 is the Status / LocationInfo CF and 1 is the HintedHandof CF. I dont have it now, but IIRC CFStats showed the LocationInfo CF with dirty ops. I was able to repo a case where flushing the CF's did not mark the log segments as obsolete (attached unit-test patch). Steps are: 1. Write to cf1 and flush. 2. Current log segment is marked as dirty at the CL position when the flush started, CommitLog.discardCompletedSegmentsInternal() 3. Do not write to cf1 again. 4. Roll the log, my test does this manually. 5. Write to CF2 and flush. 6. Only CF2 is flushed because it is the only dirty CF. cfs.maybeSwitchMemtable() is not called for cf1 and so log segment 1 is still marked as dirty from cf1. Step 5 is not essential, just matched what I thought was happening. I thought SystemTable.updateToken() was called which does not flush, and this was the last thing that happened. The expired memtable thread created by Table uses the same cfs.forceFlush() which is a no-op if the cf or it's secondary indexes are clean. I think the same problem would exist in 0.8. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-2879: --- Attachment: CASSANDRA-2879-DBConstants-name-refactoring.patch DBConstants name refactoring (underscores removed) Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: 2879-v3.txt, CASSANDRA-2879-DBConstants-name-refactoring.patch, CASSANDRA-2879-v2.patch, CASSANDRA-2879-v4.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1145822 - in /cassandra/trunk/src/java/org/apache/cassandra: db/ db/context/ utils/
Author: jbellis Date: Tue Jul 12 23:58:07 2011 New Revision: 1145822 URL: http://svn.apache.org/viewvc?rev=1145822view=rev Log: r/m underscores Modified: cassandra/trunk/src/java/org/apache/cassandra/db/Column.java cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java cassandra/trunk/src/java/org/apache/cassandra/db/CounterColumn.java cassandra/trunk/src/java/org/apache/cassandra/db/DBConstants.java cassandra/trunk/src/java/org/apache/cassandra/db/ExpiringColumn.java cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java cassandra/trunk/src/java/org/apache/cassandra/db/context/CounterContext.java cassandra/trunk/src/java/org/apache/cassandra/utils/BloomFilterSerializer.java Modified: cassandra/trunk/src/java/org/apache/cassandra/db/Column.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/Column.java?rev=1145822r1=1145821r2=1145822view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/Column.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/Column.java Tue Jul 12 23:58:07 2011 @@ -128,7 +128,7 @@ public class Column implements IColumn * + 4 bytes which basically indicates the size of the byte array * + entire byte array. */ -return DBConstants.shortSize_ + name.remaining() + 1 + DBConstants.tsSize_ + DBConstants.intSize_ + value.remaining(); +return DBConstants.shortSize + name.remaining() + 1 + DBConstants.tsSize + DBConstants.intSize + value.remaining(); } /* Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java?rev=1145822r1=1145821r2=1145822view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java Tue Jul 12 23:58:07 2011 @@ -23,7 +23,6 @@ import static org.apache.cassandra.db.DB import java.nio.ByteBuffer; import java.security.MessageDigest; import java.util.Map; -import java.util.SortedSet; import java.util.concurrent.ConcurrentSkipListMap; import org.apache.cassandra.config.CFMetaData; @@ -32,7 +31,6 @@ import org.apache.cassandra.db.filter.Qu import org.apache.cassandra.db.marshal.AbstractType; import org.apache.cassandra.db.marshal.MarshalException; import org.apache.cassandra.io.IColumnSerializer; -import org.apache.cassandra.io.util.IIterableColumns; import org.apache.cassandra.utils.FBUtilities; public class ColumnFamily extends AbstractColumnContainer @@ -306,16 +304,16 @@ public class ColumnFamily extends Abstra public long serializedSize() { -return boolSize_ // nullness bool - + intSize_ // id +return boolSize // nullness bool + + intSize // id + serializedSizeForSSTable(); } public long serializedSizeForSSTable() { -int size = intSize_ // local deletion time - + longSize_ // client deletion time - + intSize_; // column count +int size = intSize // local deletion time + + longSize // client deletion time + + intSize; // column count for (IColumn column : columns.values()) size += column.serializedSize(); return size; Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java?rev=1145822r1=1145821r2=1145822view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java Tue Jul 12 23:58:07 2011 @@ -188,9 +188,9 @@ public class ColumnIndexer public long serializedSize() { -return DBConstants.intSize_ // length of Bloom Filter +return DBConstants.intSize // length of Bloom Filter + bloomFilter.serializedSize() // BF data - + DBConstants.intSize_ // length of index block + + DBConstants.intSize // length of index block + entriesSize; // index block } } Modified: cassandra/trunk/src/java/org/apache/cassandra/db/CounterColumn.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/CounterColumn.java?rev=1145822r1=1145821r2=1145822view=diff == ---
[jira] [Commented] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064243#comment-13064243 ] Jonathan Ellis commented on CASSANDRA-2879: --- also committed Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: 2879-v3.txt, CASSANDRA-2879-DBConstants-name-refactoring.patch, CASSANDRA-2879-v2.patch, CASSANDRA-2879-v4.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2868) Native Memory Leak
[ https://issues.apache.org/jira/browse/CASSANDRA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064247#comment-13064247 ] Chris Burroughs commented on CASSANDRA-2868: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7066129 will be the id when bugs.sun.com gets around to doing it's thing. I confirmed that -XX:MaxDirectMemorySize does not protect you from this (ie it's a native native leak, not some DirectByteBuffer thing). I'll be able to test this but not until the end of this week at the earliest (and it will then take at least another week to be sure). Native Memory Leak -- Key: CASSANDRA-2868 URL: https://issues.apache.org/jira/browse/CASSANDRA-2868 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.6 Reporter: Daniel Doubleday Priority: Minor We have memory issues with long running servers. These have been confirmed by several users in the user list. That's why I report. The memory consumption of the cassandra java process increases steadily until it's killed by the os because of oom (with no swap) Our server is started with -Xmx3000M and running for around 23 days. pmap -x shows Total SST: 1961616 (mem mapped data and index files) Anon RSS: 6499640 Total RSS: 8478376 This shows that 3G are 'overallocated'. We will use BRAF on one of our less important nodes to check wether it is related to mmap and report back. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2895) add java classpath to cassandra startup logging
add java classpath to cassandra startup logging --- Key: CASSANDRA-2895 URL: https://issues.apache.org/jira/browse/CASSANDRA-2895 Project: Cassandra Issue Type: Improvement Reporter: Jackson Chung Priority: Minor this is helpful to determine/verify if the Cassandra is started with the expected classpath it's a simple 1-liner addon that are useful... will submit a patch later. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2879) Make SSTableWriter.append(...) methods seekless.
[ https://issues.apache.org/jira/browse/CASSANDRA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064251#comment-13064251 ] Hudson commented on CASSANDRA-2879: --- Integrated in Cassandra #955 (See [https://builds.apache.org/job/Cassandra/955/]) optimize away seek when compacting wide rows patch by Pavel Yaskevich and jbellis for CASSANDRA-2879 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1145818 Files : * /cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamily.java * /cassandra/trunk/CHANGES.txt * /cassandra/trunk/src/java/org/apache/cassandra/db/ColumnIndexer.java * /cassandra/trunk/src/java/org/apache/cassandra/utils/BloomFilterSerializer.java * /cassandra/trunk/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java * /cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java * /cassandra/trunk/src/java/org/apache/cassandra/utils/BloomFilter.java Make SSTableWriter.append(...) methods seekless. Key: CASSANDRA-2879 URL: https://issues.apache.org/jira/browse/CASSANDRA-2879 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Fix For: 1.0 Attachments: 2879-v3.txt, CASSANDRA-2879-DBConstants-name-refactoring.patch, CASSANDRA-2879-v2.patch, CASSANDRA-2879-v4.patch, CASSANDRA-2879.patch as we already have a CF.serializedSize() method we don't need to reserve a place to store data size when we write data to SSTable. Compaction should be seekless too because we calculate data size before we write actual content. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable
[ https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064252#comment-13064252 ] Hudson commented on CASSANDRA-2753: --- Integrated in Cassandra #955 (See [https://builds.apache.org/job/Cassandra/955/]) fix supercolumn maxtimestamp bug patch by Daniel Doubleday; reviewed by jbellis for CASSANDRA-2753 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1145819 Files : * /cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java Capture the max client timestamp for an SSTable --- Key: CASSANDRA-2753 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Alan Liang Assignee: Alan Liang Priority: Minor Fix For: 1.0 Attachments: 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V2.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to-V3.patch, 0001-capture-max-timestamp-and-created-SSTableMetadata-to.patch, 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch, supercolumn.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2868) Native Memory Leak
[ https://issues.apache.org/jira/browse/CASSANDRA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Burroughs updated CASSANDRA-2868: --- Attachment: 2868-v1.txt In case it is useful to anyone else this is what I intend to test with. Native Memory Leak -- Key: CASSANDRA-2868 URL: https://issues.apache.org/jira/browse/CASSANDRA-2868 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.6 Reporter: Daniel Doubleday Priority: Minor Attachments: 2868-v1.txt We have memory issues with long running servers. These have been confirmed by several users in the user list. That's why I report. The memory consumption of the cassandra java process increases steadily until it's killed by the os because of oom (with no swap) Our server is started with -Xmx3000M and running for around 23 days. pmap -x shows Total SST: 1961616 (mem mapped data and index files) Anon RSS: 6499640 Total RSS: 8478376 This shows that 3G are 'overallocated'. We will use BRAF on one of our less important nodes to check wether it is related to mmap and report back. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2850) Converting bytes to hex string is unnecessarily slow
[ https://issues.apache.org/jira/browse/CASSANDRA-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064310#comment-13064310 ] Jonathan Ellis commented on CASSANDRA-2850: --- David, bytesToHex4 looks good to me. Can you post a version of v2, with your improvements incorporated? Converting bytes to hex string is unnecessarily slow Key: CASSANDRA-2850 URL: https://issues.apache.org/jira/browse/CASSANDRA-2850 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.6, 0.8.1 Reporter: David Allsopp Priority: Minor Fix For: 0.8.2 Attachments: 2850-v2.patch, BytesToHexBenchmark.java, BytesToHexBenchmark2.java, BytesToHexBenchmark3.java, cassandra-2850a.diff ByteBufferUtil.bytesToHex() is unnecessarily slow - it doesn't pre-size the StringBuilder (so several re-sizes will be needed behind the scenes) and it makes quite a few method calls per byte. (OK, this may be a premature optimisation, but I couldn't resist, and it's a small change) Will attach patch shortly that speeds it up by about x3, plus benchmarking test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1145849 - in /cassandra/branches/cassandra-0.7: CHANGES.txt src/java/org/apache/cassandra/config/DatabaseDescriptor.java src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java
Author: jbellis Date: Wed Jul 13 03:18:27 2011 New Revision: 1145849 URL: http://svn.apache.org/viewvc?rev=1145849view=rev Log: support spaces in path to log4j configuration patch by David Allsopp; reviewed by jbellis for CASSANDRA-2383 Modified: cassandra/branches/cassandra-0.7/CHANGES.txt cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/config/DatabaseDescriptor.java cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java Modified: cassandra/branches/cassandra-0.7/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1145849r1=1145848r2=1145849view=diff == --- cassandra/branches/cassandra-0.7/CHANGES.txt (original) +++ cassandra/branches/cassandra-0.7/CHANGES.txt Wed Jul 13 03:18:27 2011 @@ -1,6 +1,7 @@ 0.7.8 * fix possibility of spurious UnavailableException for LOCAL_QUORUM reads with dynamic snitch + read repair disabled (CASSANDRA-2870) + * support spaces in path to log4j configuration (CASSANDRA-2383) 0.7.7 Modified: cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/config/DatabaseDescriptor.java URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/config/DatabaseDescriptor.java?rev=1145849r1=1145848r2=1145849view=diff == --- cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/config/DatabaseDescriptor.java (original) +++ cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/config/DatabaseDescriptor.java Wed Jul 13 03:18:27 2011 @@ -100,7 +100,7 @@ public classDatabaseDescriptor try { url = new URL(configUrl); -url.openStream(); // catches well-formed but bogus URLs +url.openStream().close(); // catches well-formed but bogus URLs } catch (Exception e) { Modified: cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java?rev=1145849r1=1145848r2=1145849view=diff == --- cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java (original) +++ cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java Wed Jul 13 03:18:27 2011 @@ -70,12 +70,30 @@ public abstract class AbstractCassandraD } catch (MalformedURLException ex) { -// load from the classpath. +// then try loading from the classpath. configLocation = AbstractCassandraDaemon.class.getClassLoader().getResource(config); -if (configLocation == null) -throw new RuntimeException(Couldn't figure out log4j configuration.); } -PropertyConfigurator.configureAndWatch(configLocation.getFile(), 1); + +if (configLocation == null) +throw new RuntimeException(Couldn't figure out log4j configuration: +config); + +// Now convert URL to a filename +String configFileName = null; +try +{ +// first try URL.getFile() which works for opaque URLs (file:foo) and paths without spaces +configFileName = configLocation.getFile(); +File configFile = new File(configFileName); +// then try alternative approach which works for all hierarchical URLs with or without spaces +if (!configFile.exists()) +configFileName = new File(configLocation.toURI()).getCanonicalPath(); +} +catch (Exception e) +{ +throw new RuntimeException(Couldn't convert log4j configuration location to a valid file, e); +} + +PropertyConfigurator.configureAndWatch(configFileName, 1); org.apache.log4j.Logger.getLogger(AbstractCassandraDaemon.class).info(Logging initialized); }
[jira] [Commented] (CASSANDRA-2383) log4j unable to load properties file from classpath
[ https://issues.apache.org/jira/browse/CASSANDRA-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064324#comment-13064324 ] Hudson commented on CASSANDRA-2383: --- Integrated in Cassandra-0.7 #527 (See [https://builds.apache.org/job/Cassandra-0.7/527/]) support spaces in path to log4j configuration patch by David Allsopp; reviewed by jbellis for CASSANDRA-2383 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1145849 Files : * /cassandra/branches/cassandra-0.7/CHANGES.txt * /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/config/DatabaseDescriptor.java * /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java log4j unable to load properties file from classpath --- Key: CASSANDRA-2383 URL: https://issues.apache.org/jira/browse/CASSANDRA-2383 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.6 Environment: OS : windows java : 1.6.0.23 Reporter: david lee Assignee: David Allsopp Priority: Minor Fix For: 0.7.8 Attachments: cassandra-0.7.6-2-2383.diff when cassandra home folder is placed inside a folder which has space characters in its name, log4j settings are not properly loaded and warning messages are shown. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of Committers by JonathanEllis
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The Committers page has been changed by JonathanEllis: http://wiki.apache.org/cassandra/Committers?action=diffrev1=13rev2=14 Comment: Sylvain added to PMC ||Jaakko Laine||Dec 2009||?|| || ||Brandon Williams||Jun 2010||Datastax||PMC member|| ||Jake Luciani||Jan 2011||Datastax||Also a [[http://thrift.apache.org/|Thrift]] committer|| - ||Sylvain Lebresne||Mar 2011||Datastax + ||Sylvain Lebresne||Mar 2011||Datastax||PMC member||
[jira] [Assigned] (CASSANDRA-1391) Allow Concurrent Schema Migrations
[ https://issues.apache.org/jira/browse/CASSANDRA-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-1391: - Assignee: Gary Dusbabek Does this sound sane at all to you, Gary? Allow Concurrent Schema Migrations -- Key: CASSANDRA-1391 URL: https://issues.apache.org/jira/browse/CASSANDRA-1391 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Gary Dusbabek CASSANDRA-1292 fixed multiple migrations started from the same node to properly queue themselves, but it is still possible for migrations initiated on different nodes to conflict and leave the cluster in a bad state. Since the system_add/drop/rename methods are accessible directly from the client API, they should be completely safe for concurrent use. It should be possible to allow for most types of concurrent migrations by converting the UUID schema ID into a VersionVectorClock (as provided by CASSANDRA-580). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1391) Allow Concurrent Schema Migrations
[ https://issues.apache.org/jira/browse/CASSANDRA-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064363#comment-13064363 ] Gary Dusbabek commented on CASSANDRA-1391: -- Not with the current state of things. Allow Concurrent Schema Migrations -- Key: CASSANDRA-1391 URL: https://issues.apache.org/jira/browse/CASSANDRA-1391 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Gary Dusbabek CASSANDRA-1292 fixed multiple migrations started from the same node to properly queue themselves, but it is still possible for migrations initiated on different nodes to conflict and leave the cluster in a bad state. Since the system_add/drop/rename methods are accessible directly from the client API, they should be completely safe for concurrent use. It should be possible to allow for most types of concurrent migrations by converting the UUID schema ID into a VersionVectorClock (as provided by CASSANDRA-580). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of FAQ_JP by MakiWatanabe
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The FAQ_JP page has been changed by MakiWatanabe: http://wiki.apache.org/cassandra/FAQ_JP?action=diffrev1=86rev2=87 Comment: Fix typo at #jna Anchor(jna) == Keyspaceを更新したり、スナップショットをとろうとしたときに java.io.IOException: Cannot run program ln が発生します。 == - Keyspaceを更新する場合、最初にスナップショットが取られます。これには既存のSSTあbれへのはーどりんくを生成することも含まれますが、Javaにはハードリンクをネイティブで生成する機能がないため、lnをforkする必要があります。forkを実行する際には、例え子プロセスがそのすべてを使用しないとしても親プロセスと同じ大きさのフリーメモリが必要です。Javaは巨大なプロセスですので、これは難しい問題です。解決策は[[http://jna.java.net/|Java Native Access]]をインストールし、ハードリンクの生成をネイティブで実行することです。 + Keyspaceを更新する場合、最初にスナップショットが取られます。これには既存のSSTableへのハードリンクを生成することを意味しますが、Javaにはハードリンクをネイティブで生成する機能がないため、lnをfork/execする必要があります。forkを実行する際には、例え子プロセスがそのすべてを使用しないとしても親プロセスと同じ大きさのフリーメモリが必要です。Javaは巨大なプロセスですので、これは難しい問題です。解決策は[[http://jna.java.net/|Java Native Access]]をインストールし、ハードリンクの生成をネイティブで実行することです。 Anchor(replicaplacement)