[jira] [Commented] (CASSANDRA-2820) Re-introduce FastByteArrayInputStream (and Output equivalent)
[ https://issues.apache.org/jira/browse/CASSANDRA-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061126#comment-13061126 ] Paul Loy commented on CASSANDRA-2820: - I'm actually not too much of a fan of extending ByteArray*Stream but redeclaring all the fields, giving fake argument to the super constructor. It's a bit ugly and not too optimal in memory size... Haha, me too! it may be simpler/cleaner to make FastByteArray*Stream extends Input/OutputStream directly I prefer this too. my earlier comment was more about avoiding to use FastByteArray*Stream in more places than strictly necessary Sure. I guess this was lost in translation :S I'll update the patch. Re-introduce FastByteArrayInputStream (and Output equivalent) - Key: CASSANDRA-2820 URL: https://issues.apache.org/jira/browse/CASSANDRA-2820 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Environment: n/a Reporter: Paul Loy Priority: Minor Labels: bytearrayinputstream, bytearrayoutputstream, license, synchronized Fix For: 1.0 Attachments: fast_bytearray_iostreams_harmony-patch-2.txt, fast_bytearray_iostreams_harmony-patch-3.txt In https://issues.apache.org/jira/browse/CASSANDRA-37 FastByteArrayInputStream and FastByteArrayOutputStream were removed due to being code copied from the JDK and then subsequently modified. The JDK license is incompatible with Apache 2 license so the code had to go. I have since had a look at the performance of the JDK ByteArrayInputStream and a FastByteArrayInputStream (i.e. one with synchronized methods made un-synchronized) and seen the difference is significant. After a warmup-period of 1 loops I get the following for 1 loops through a 128000 byte array: bais : 3513ms fbais: 72ms This varies depending on the OS, machine and Java version, but it's always in favour of the FastByteArrayInputStream as you might expect. Then, at Jonathan Ellis' suggestion, I tried this using a modified Apache Harmony ByteArrayInputStream - i.e. one whose license is compatible - and the results were the same. A significant boost. I will attach a patch with changes for the 0.8.0 tag. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2843) better performance on long row read
[ https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061134#comment-13061134 ] Yang Yang commented on CASSANDRA-2843: -- i just got down the patch and transfered it to a computer to read Sylvain's approach to compare the last element is quite clean i see no problems the only problem was due to me: the bin search high=mid-1 should be changed to high=mid also with this error fixed , you dont need to special case 1 2 in the end of binsearch https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] already sorted order it. Which doesn't mean it cannot optimize for it. If you look at the version I attached, at least as far a addColumn is concerned, it does the exact same thing as your version, with the only difference that I first check if adding at the tail is legit and fail back to a binary search if that is not the case. That is, as long as the input is in sorted order, it will be as fast as your implementation (there is one more bytebuffer comparison but I'm willing to bet that it has no noticeable impact on performance). But it won't create unsorted CF if the input is not in sorted order. stick w/ TreeMap for simplicity? the TreeMap implementation so that people can look for themselves. The test simply creates a CF, add columns to it (in sorted order) and do a simple iteration at the end. I've also add a delete at the end because at least in the case of super columns, we do call removeDeleted so the goal was to see if this has a significant impact (the deletes are made at the beginning of the CF, which is the worst case for the ArrayBacked solution). The test also allow to have some column overwrap (to exercise reconciliation). Not that when that happens, the input is not in strict sorted order anymore, but it's mostly at the disadvantage of the ArrayBack implementation there too. Playing with the parameters (number of columns added, number that overlaps, number of deletes) the results seems to always be the same. The ArrayBacked is consistently faster than the TreeMap one that is itself consistently faster than the CSLM one. Now what I meant is that the difference between ArrayBacked and TreeMap is generally not as big as the one with CSLM, but it is still often very noticeable. insertion in sorted order: the insertion is then O(1) and with a small constant factor because we're using ArrayList. TreeMap can't beat that. Given this, and given that ColumnFamily is one of our core data structure, I think we should choose the more efficient implementation for each use case. And truth is, the ArrayBacked implementation is really not very complicated, that's basic stuff. CSLM, and that's what we do on reads. does show that we're much much faster even without reconciliation happening. https://issues.apache.org/jira/browse/CASSANDRA-2843 microBenchmark.patch considerably slow (my test of and 40 bytes in value, is about 16ms. org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(ColumnFamily, Iterator, int) org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(IColumnContainer, Iterator, int) concurrentSkipListMap() that maps column names to values. it needs to maintain a more complex structure of map. output to be ListColumnOrSuperColumn so it does not make sense to use a luxury map data structure in the interium and finally convert it to a list. on the synchronization side, since the return CF is never going to be shared/modified by other threads, we know the access is always single thread, so no synchronization is needed. particularly write. so we can provide a different ColumnFamily to CFS.getTopLevelColumnFamily(), so getTopLevelColumnFamily no longer always creates the standard ColumnFamily, but take a provided returnCF, whose cost is much cheaper. agree on the general direction. provided. the main work is to let the FastColumnFamily use an array for internal storage. at first I used binary search to insert new columns in addColumn(), but later I found that even this is not necessary, since all calling scenarios of ColumnFamily.addColumn() has an invariant that the inserted columns come in sorted order (I still have an issue to resolve descending or ascending now, but ascending works). so the current logic is simply to compare the new column against the end column in the array, if names not equal, append, if equal, reconcile. flavors of the method, one accepting a returnCF. but we could definitely think about what is the better way to provide this returnCF. my application, and the performance improvement is
[jira] [Updated] (CASSANDRA-2843) better performance on long row read
[ https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yang updated CASSANDRA-2843: - Comment: was deleted (was: Thanks Sylvain Its great to see this taking shape Im on vacation with a pbone so very cursory comments this implementation should not expect the input always come in already sorted order then probably it would be difficult to achieve a lot of per gain easily. Mainly there are two areas of attack. Synch and cheaper data structure. Actually during my tests I found that sync is not the main issue: going from CSLM to treemap is actually slower. Your tests show different so we'd better confirm. If I were correct in tests. Then that means not assuming special circumstances would be like finding a generic silver bullet which is very difficult https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] ColumnFamily during is a good idea. It is clear that avoiding synchronization will be faster, and given the type of operations we do during reads (insertion in sorted order and iteration), an ArrayList backed solution is sure to be faster too. I will also be much gentle on the GC that the linked list ConcurrentSkipListMap uses. I think that all those will help even with relatively small reads. So let's focus on that for this ticket and let other potential improvement to other ticket, especially if it is unclear they bear any noticeable speedup. is quite frankly ugly and will be a maintenance nightmare (you'll have to check you did overwrite every function that touch the map (which is not the case in the patch) and every update to ColumnFamily have to be aware that it should update FastColumnFamily as well). functionnal ColumnFamily implementation (albeit not synchronized). That is, we can't assume that addition will always be in strict increasing order, otherwise again this will be too hard to use. Granted, I don't think it is used in the read path, but I think that the new ColumnFamily implementation could advantageously be used during compaction (by preCompactedRow typically, and possibly other places where concurrent access is not an issue) where this would matter. the remarks above. The patch is against trunk (not 0.8 branch), because it build on the recently committed refactor of ColumnFamily. It refactors ColumnFamily (AbstractColumnContainer actually) to allow for a pluggable backing column map. The ConcurrentSkipListMap implemn is name ThreadSafeColumnMap and the new one is called ArrayBackedColumnMap (which I prefer to FastSomething since it's not a very helpful name). getTopLevelColumns, I pass along a factory (that each backing implementation provides). The main goal was to avoid creating a columnFamily when it's useless (if row cache is enabled on the CF -- btw, this ticket only improve on read for column family with no cache). (addition of column + iteration), the ArrayBacked implementation is faster than the ConcurrentSkipListMap based one. Interestingly though, this is mainly true when some reconciliation of columns happens. That is, if you only add columns with different names, the ArrayBacked implementation is faster, but not dramatically so. If you start adding column that have to be resolved, the ArrayBacked implementation becomes much faster, even with a reasonably small number of columns (inserting 100 columns with only 10 unique column names, the ArrayBacked is already 30% faster). And this mostly due to the overhead of synchronization (of replace()): a TreeMap based implementation is slightly slower than the ArrayBacked one but not by a lot and thus is much faster than the ConcurrentSkipListMap implementation. use a few unit test for the new ArrayBacked implementation). considerably slow (my test of and 40 bytes in value, is about 16ms. org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(ColumnFamily, Iterator, int) org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(IColumnContainer, Iterator, int) concurrentSkipListMap() that maps column names to values. it needs to maintain a more complex structure of map. output to be ListColumnOrSuperColumn so it does not make sense to use a luxury map data structure in the interium and finally convert it to a list. on the synchronization side, since the return CF is never going to be shared/modified by other threads, we know the access is always single thread, so no synchronization is needed. particularly write. so we can provide a different ColumnFamily to CFS.getTopLevelColumnFamily(), so getTopLevelColumnFamily no longer always creates the standard ColumnFamily,
[jira] [Updated] (CASSANDRA-2843) better performance on long row read
[ https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yang updated CASSANDRA-2843: - Comment: was deleted (was: My comment before last was not quite correct The array implementation with binary search also works without special assumptions. Just that with assumption alot futher gains https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] ColumnFamily during is a good idea. It is clear that avoiding synchronization will be faster, and given the type of operations we do during reads (insertion in sorted order and iteration), an ArrayList backed solution is sure to be faster too. I will also be much gentle on the GC that the linked list ConcurrentSkipListMap uses. I think that all those will help even with relatively small reads. So let's focus on that for this ticket and let other potential improvement to other ticket, especially if it is unclear they bear any noticeable speedup. is quite frankly ugly and will be a maintenance nightmare (you'll have to check you did overwrite every function that touch the map (which is not the case in the patch) and every update to ColumnFamily have to be aware that it should update FastColumnFamily as well). functionnal ColumnFamily implementation (albeit not synchronized). That is, we can't assume that addition will always be in strict increasing order, otherwise again this will be too hard to use. Granted, I don't think it is used in the read path, but I think that the new ColumnFamily implementation could advantageously be used during compaction (by preCompactedRow typically, and possibly other places where concurrent access is not an issue) where this would matter. the remarks above. The patch is against trunk (not 0.8 branch), because it build on the recently committed refactor of ColumnFamily. It refactors ColumnFamily (AbstractColumnContainer actually) to allow for a pluggable backing column map. The ConcurrentSkipListMap implemn is name ThreadSafeColumnMap and the new one is called ArrayBackedColumnMap (which I prefer to FastSomething since it's not a very helpful name). getTopLevelColumns, I pass along a factory (that each backing implementation provides). The main goal was to avoid creating a columnFamily when it's useless (if row cache is enabled on the CF -- btw, this ticket only improve on read for column family with no cache). (addition of column + iteration), the ArrayBacked implementation is faster than the ConcurrentSkipListMap based one. Interestingly though, this is mainly true when some reconciliation of columns happens. That is, if you only add columns with different names, the ArrayBacked implementation is faster, but not dramatically so. If you start adding column that have to be resolved, the ArrayBacked implementation becomes much faster, even with a reasonably small number of columns (inserting 100 columns with only 10 unique column names, the ArrayBacked is already 30% faster). And this mostly due to the overhead of synchronization (of replace()): a TreeMap based implementation is slightly slower than the ArrayBacked one but not by a lot and thus is much faster than the ConcurrentSkipListMap implementation. use a few unit test for the new ArrayBacked implementation). considerably slow (my test of and 40 bytes in value, is about 16ms. org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(ColumnFamily, Iterator, int) org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(IColumnContainer, Iterator, int) concurrentSkipListMap() that maps column names to values. it needs to maintain a more complex structure of map. output to be ListColumnOrSuperColumn so it does not make sense to use a luxury map data structure in the interium and finally convert it to a list. on the synchronization side, since the return CF is never going to be shared/modified by other threads, we know the access is always single thread, so no synchronization is needed. particularly write. so we can provide a different ColumnFamily to CFS.getTopLevelColumnFamily(), so getTopLevelColumnFamily no longer always creates the standard ColumnFamily, but take a provided returnCF, whose cost is much cheaper. agree on the general direction. provided. the main work is to let the FastColumnFamily use an array for internal storage. at first I used binary search to insert new columns in addColumn(), but later I found that even this is not necessary, since all calling scenarios of ColumnFamily.addColumn() has an invariant that the inserted columns come in sorted order (I still have an issue to resolve descending or ascending
[jira] [Issue Comment Edited] (CASSANDRA-2843) better performance on long row read
[ https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061134#comment-13061134 ] Yang Yang edited comment on CASSANDRA-2843 at 7/7/11 8:34 AM: -- i just got down the patch and transfered it to a computer to read Sylvain's approach to compare the last element is quite clean i see no problems the only problem was due to me: the bin search high=mid-1 should be changed to high=mid also with this error fixed , you dont need to special case 1 2 in the end of binsearch https://issues.apache.org/jira/browse/CASSANDRA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] already sorted order it. Which doesn't mean it cannot optimize for it. If you look at the version I attached, at least as far a addColumn is concerned, it does the exact same thing as your version, with the only difference that I first check if adding at the tail is legit and fail back to a binary search if that is not the case. That is, as long as the input is in sorted order, it will be as fast as your implementation (there is one more bytebuffer comparison but I'm willing to bet that it has no noticeable impact on performance). But it won't create unsorted CF if the input is not in sorted order. stick w/ TreeMap for simplicity? the TreeMap implementation so that people can look for themselves. The test simply creates a CF, add columns to it (in sorted order) and do a simple iteration at the end. I've also add a delete at the end because at least in the case of super columns, we do call removeDeleted so the goal was to see if this has a significant impact (the deletes are made at the beginning of the CF, which is the worst case for the ArrayBacked solution). The test also allow to have some column overwrap (to exercise reconciliation). Not that when that happens, the input is not in strict sorted order anymore, but it's mostly at the disadvantage of the ArrayBack implementation there too. Playing with the parameters (number of columns added, number that overlaps, number of deletes) the results seems to always be the same. The ArrayBacked is consistently faster than the TreeMap one that is itself consistently faster than the CSLM one. Now what I meant is that the difference between ArrayBacked and TreeMap is generally not as big as the one with CSLM, but it is still often very noticeable. insertion in sorted order: the insertion is then O(1) and with a small constant factor because we're using ArrayList. TreeMap can't beat that. Given this, and given that ColumnFamily is one of our core data structure, I think we should choose the more efficient implementation for each use case. And truth is, the ArrayBacked implementation is really not very complicated, that's basic stuff. CSLM, and that's what we do on reads. does show that we're much much faster even without reconciliation happening. https://issues.apache.org/jira/browse/CASSANDRA-2843 microBenchmark.patch considerably slow (my test of and 40 bytes in value, is about 16ms. org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(QueryFilter, int, ColumnFamily) org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(ColumnFamily, Iterator, int) org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(IColumnContainer, Iterator, int) concurrentSkipListMap() that maps column names to values. it needs to maintain a more complex structure of map. output to be ListColumnOrSuperColumn so it does not make sense to use a luxury map data structure in the interium and finally convert it to a list. on the synchronization side, since the return CF is never going to be shared/modified by other threads, we know the access is always single thread, so no synchronization is needed. particularly write. so we can provide a different ColumnFamily to CFS.getTopLevelColumnFamily(), so getTopLevelColumnFamily no longer always creates the standard ColumnFamily, but take a provided returnCF, whose cost is much cheaper. agree on the general direction. provided. the main work is to let the FastColumnFamily use an array for internal storage. at first I used binary search to insert new columns in addColumn(), but later I found that even this is not necessary, since all calling scenarios of ColumnFamily.addColumn() has an invariant that the inserted columns come in sorted order (I still have an issue to resolve descending or ascending now, but ascending works). so the current logic is simply to compare the new column against the end column in the array, if names not equal, append, if equal, reconcile. flavors of the method, one accepting a returnCF. but we could definitely think about what is the better way to provide this returnCF.
[jira] [Created] (CASSANDRA-2867) Starting 0.8.1 after upgrade from 0.7.6-2 fails
Starting 0.8.1 after upgrade from 0.7.6-2 fails --- Key: CASSANDRA-2867 URL: https://issues.apache.org/jira/browse/CASSANDRA-2867 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.6 Environment: CentOS 5.6 Reporter: Yaniv Kunda After upgrading the binaries to 0.8.1 I get an exception when starting cassandra: [root@bserv2 local]# INFO 12:51:04,512 Logging initialized INFO 12:51:04,523 Heap size: 8329887744/8329887744 INFO 12:51:04,524 JNA not found. Native methods will be disabled. INFO 12:51:04,531 Loading settings from file:/usr/local/apache-cassandra-0.8.1/conf/cassandra.yaml INFO 12:51:04,621 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 12:51:04,707 Global memtable threshold is enabled at 2648MB INFO 12:51:04,708 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 12:51:04,713 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 12:51:04,714 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 12:51:04,716 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 12:51:04,717 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 12:51:04,719 Removing compacted SSTable files (see http://wiki.apache.org/cassandra/MemtableSSTable) INFO 12:51:04,770 reading saved cache /vm1/cassandraDB/saved_caches/system-IndexInfo-KeyCache INFO 12:51:04,776 Opening /vm1/cassandraDB/data/system/IndexInfo-f-9 INFO 12:51:04,792 reading saved cache /vm1/cassandraDB/saved_caches/system-Schema-KeyCache INFO 12:51:04,794 Opening /vm1/cassandraDB/data/system/Schema-f-194 INFO 12:51:04,797 Opening /vm1/cassandraDB/data/system/Schema-f-195 INFO 12:51:04,802 Opening /vm1/cassandraDB/data/system/Schema-f-193 INFO 12:51:04,811 Opening /vm1/cassandraDB/data/system/Migrations-f-193 INFO 12:51:04,814 reading saved cache /vm1/cassandraDB/saved_caches/system-LocationInfo-KeyCache INFO 12:51:04,815 Opening /vm1/cassandraDB/data/system/LocationInfo-f-292 INFO 12:51:04,843 Loading schema version 586e70fd-a332-11e0-828e-34b74a661156 ERROR 12:51:04,996 Exception encountered during startup. org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes: 15 at org.apache.cassandra.db.marshal.LongType.getString(LongType.java:72) at org.apache.cassandra.config.CFMetaData.getDefaultIndexName(CFMetaData.java:971) at org.apache.cassandra.config.CFMetaData.inflate(CFMetaData.java:381) at org.apache.cassandra.config.KSMetaData.inflate(KSMetaData.java:172) at org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99) at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:479) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:139) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) Exception encountered during startup. org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes: 15 at org.apache.cassandra.db.marshal.LongType.getString(LongType.java:72) at org.apache.cassandra.config.CFMetaData.getDefaultIndexName(CFMetaData.java:971) at org.apache.cassandra.config.CFMetaData.inflate(CFMetaData.java:381) at org.apache.cassandra.config.KSMetaData.inflate(KSMetaData.java:172) at org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99) at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:479) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:139) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80) It seems this has something to do with indexes, and I do have a CF with an index on it, but it is not used. I can try and remove the index with 0.7.x binaries, but I will wait a bit to see if anyone needs it to reproduce the bug. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2867) Starting 0.8.1 after upgrade from 0.7.6-2 fails
[
https://issues.apache.org/jira/browse/CASSANDRA-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaniv Kunda updated CASSANDRA-2867:
---
Description:
After upgrading the binaries to 0.8.1 I get an exception when starting
cassandra:
{noformat}
[root@bserv2 local]# INFO 12:51:04,512 Logging initialized
INFO 12:51:04,523 Heap size: 8329887744/8329887744
INFO 12:51:04,524 JNA not found. Native methods will be disabled.
INFO 12:51:04,531 Loading settings from
file:/usr/local/apache-cassandra-0.8.1/conf/cassandra.yaml
INFO 12:51:04,621 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 12:51:04,707 Global memtable threshold is enabled at 2648MB
INFO 12:51:04,708 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,713 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,714 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,716 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,717 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,719 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,770 reading saved cache
/vm1/cassandraDB/saved_caches/system-IndexInfo-KeyCache
INFO 12:51:04,776 Opening /vm1/cassandraDB/data/system/IndexInfo-f-9
INFO 12:51:04,792 reading saved cache
/vm1/cassandraDB/saved_caches/system-Schema-KeyCache
INFO 12:51:04,794 Opening /vm1/cassandraDB/data/system/Schema-f-194
INFO 12:51:04,797 Opening /vm1/cassandraDB/data/system/Schema-f-195
INFO 12:51:04,802 Opening /vm1/cassandraDB/data/system/Schema-f-193
INFO 12:51:04,811 Opening /vm1/cassandraDB/data/system/Migrations-f-193
INFO 12:51:04,814 reading saved cache
/vm1/cassandraDB/saved_caches/system-LocationInfo-KeyCache
INFO 12:51:04,815 Opening /vm1/cassandraDB/data/system/LocationInfo-f-292
INFO 12:51:04,843 Loading schema version 586e70fd-a332-11e0-828e-34b74a661156
ERROR 12:51:04,996 Exception encountered during startup.
org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes: 15
at org.apache.cassandra.db.marshal.LongType.getString(LongType.java:72)
at
org.apache.cassandra.config.CFMetaData.getDefaultIndexName(CFMetaData.java:971)
at org.apache.cassandra.config.CFMetaData.inflate(CFMetaData.java:381)
at org.apache.cassandra.config.KSMetaData.inflate(KSMetaData.java:172)
at org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99)
at
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:479)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:139)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Exception encountered during startup.
org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes: 15
at org.apache.cassandra.db.marshal.LongType.getString(LongType.java:72)
at
org.apache.cassandra.config.CFMetaData.getDefaultIndexName(CFMetaData.java:971)
at org.apache.cassandra.config.CFMetaData.inflate(CFMetaData.java:381)
at org.apache.cassandra.config.KSMetaData.inflate(KSMetaData.java:172)
at org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99)
at
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:479)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:139)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
{noformat}
It seems this has something to do with indexes, and I do have a CF with an
index on it, but it is not used.
I can try and remove the index with 0.7.x binaries, but I will wait a bit to
see if anyone needs it to reproduce the bug.
was:
After upgrading the binaries to 0.8.1 I get an exception when starting
cassandra:
[root@bserv2 local]# INFO 12:51:04,512 Logging initialized
INFO 12:51:04,523 Heap size: 8329887744/8329887744
INFO 12:51:04,524 JNA not found. Native methods will be disabled.
INFO 12:51:04,531 Loading settings from
file:/usr/local/apache-cassandra-0.8.1/conf/cassandra.yaml
INFO 12:51:04,621 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 12:51:04,707 Global memtable threshold is enabled at 2648MB
INFO 12:51:04,708 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,713 Removing compacted SSTable files
[jira] [Updated] (CASSANDRA-2867) Starting 0.8.1 after upgrade from 0.7.6-2 fails
[
https://issues.apache.org/jira/browse/CASSANDRA-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaniv Kunda updated CASSANDRA-2867:
---
Affects Version/s: (was: 0.7.6)
0.8.1
Starting 0.8.1 after upgrade from 0.7.6-2 fails
---
Key: CASSANDRA-2867
URL: https://issues.apache.org/jira/browse/CASSANDRA-2867
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.1
Environment: CentOS 5.6
Reporter: Yaniv Kunda
Labels: exception, index, starting
After upgrading the binaries to 0.8.1 I get an exception when starting
cassandra:
{noformat}
[root@bserv2 local]# INFO 12:51:04,512 Logging initialized
INFO 12:51:04,523 Heap size: 8329887744/8329887744
INFO 12:51:04,524 JNA not found. Native methods will be disabled.
INFO 12:51:04,531 Loading settings from
file:/usr/local/apache-cassandra-0.8.1/conf/cassandra.yaml
INFO 12:51:04,621 DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
INFO 12:51:04,707 Global memtable threshold is enabled at 2648MB
INFO 12:51:04,708 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,713 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,714 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,716 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,717 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,719 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
INFO 12:51:04,770 reading saved cache
/vm1/cassandraDB/saved_caches/system-IndexInfo-KeyCache
INFO 12:51:04,776 Opening /vm1/cassandraDB/data/system/IndexInfo-f-9
INFO 12:51:04,792 reading saved cache
/vm1/cassandraDB/saved_caches/system-Schema-KeyCache
INFO 12:51:04,794 Opening /vm1/cassandraDB/data/system/Schema-f-194
INFO 12:51:04,797 Opening /vm1/cassandraDB/data/system/Schema-f-195
INFO 12:51:04,802 Opening /vm1/cassandraDB/data/system/Schema-f-193
INFO 12:51:04,811 Opening /vm1/cassandraDB/data/system/Migrations-f-193
INFO 12:51:04,814 reading saved cache
/vm1/cassandraDB/saved_caches/system-LocationInfo-KeyCache
INFO 12:51:04,815 Opening /vm1/cassandraDB/data/system/LocationInfo-f-292
INFO 12:51:04,843 Loading schema version 586e70fd-a332-11e0-828e-34b74a661156
ERROR 12:51:04,996 Exception encountered during startup.
org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes:
15
at
org.apache.cassandra.db.marshal.LongType.getString(LongType.java:72)
at
org.apache.cassandra.config.CFMetaData.getDefaultIndexName(CFMetaData.java:971)
at org.apache.cassandra.config.CFMetaData.inflate(CFMetaData.java:381)
at org.apache.cassandra.config.KSMetaData.inflate(KSMetaData.java:172)
at
org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99)
at
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:479)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:139)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Exception encountered during startup.
org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes:
15
at
org.apache.cassandra.db.marshal.LongType.getString(LongType.java:72)
at
org.apache.cassandra.config.CFMetaData.getDefaultIndexName(CFMetaData.java:971)
at org.apache.cassandra.config.CFMetaData.inflate(CFMetaData.java:381)
at org.apache.cassandra.config.KSMetaData.inflate(KSMetaData.java:172)
at
org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:99)
at
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:479)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:139)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
{noformat}
It seems this has something to do with indexes, and I do have a CF with an
index on it, but it is not used.
I can try and remove the index with 0.7.x binaries, but I will wait a bit to
see if anyone needs it to reproduce the bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see:
[jira] [Updated] (CASSANDRA-2861) Building RPMs should be easy
[ https://issues.apache.org/jira/browse/CASSANDRA-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niels Basjes updated CASSANDRA-2861: Attachment: CASSANDRA-ANT-RPM-2011-07-07.patch Minor update to fix missing thrift library in RPM. Building RPMs should be easy Key: CASSANDRA-2861 URL: https://issues.apache.org/jira/browse/CASSANDRA-2861 Project: Cassandra Issue Type: Improvement Components: Packaging Affects Versions: 0.8.1 Reporter: Niels Basjes Priority: Minor Labels: ant, packaging, rpm Fix For: 0.8.2 Attachments: CASSANDRA-ANT-RPM-2011-07-05.patch, CASSANDRA-ANT-RPM-2011-07-07.patch I think that building the rpm for cassandra should be no brainer. The currently available .spec file needs manual tweaking(mainly the version number) before you can built the rpm. Goal is to make it as easy as doing ant rpm or ant -Drelease=true rpm -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1143816 - /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/cli/CliUserHelp.java
Author: slebresne
Date: Thu Jul 7 13:37:28 2011
New Revision: 1143816
URL: http://svn.apache.org/viewvc?rev=1143816view=rev
Log:
Minor update to CLI help
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/cli/CliUserHelp.java
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/cli/CliUserHelp.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/cli/CliUserHelp.java?rev=1143816r1=1143815r2=1143816view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/cli/CliUserHelp.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/cli/CliUserHelp.java
Thu Jul 7 13:37:28 2011
@@ -143,9 +143,9 @@ public class CliUserHelp {
state.out.println(Applies to Simple
and OldNT strategies but NOT NTS.);
state.out.println(placement_strategy: the fully qualified
class used to place replicas in);
state.out.println(this keyspace.
Valid values are);
-state.out.println(
org.apache.cassandra.locator.SimpleStrategy,);
-state.out.println(
org.apache.cassandra.locator.NetworkTopologyStrategy,);
-state.out.println(and
org.apache.cassandra.locator.OldNetworkTopologyStrategy);
+state.out.println(
'org.apache.cassandra.locator.SimpleStrategy',);
+state.out.println(
'org.apache.cassandra.locator.NetworkTopologyStrategy',);
+state.out.println(and
'org.apache.cassandra.locator.OldNetworkTopologyStrategy');
state.out.println( strategy_options: additional options
for placement_strategy.);
state.out.println(Applies only to
NetworkTopologyStrategy.);
state.out.println(\nexamples:);
@@ -168,9 +168,9 @@ public class CliUserHelp {
state.out.println(Applies to Simple
and OldNT strategies but NOT NTS.);
state.out.println(placement_strategy: the fully qualified
class used to place replicas in);
state.out.println(this keyspace.
Valid values are);
-state.out.println(
org.apache.cassandra.locator.SimpleStrategy,);
-state.out.println(
org.apache.cassandra.locator.NetworkTopologyStrategy,);
-state.out.println(and
org.apache.cassandra.locator.OldNetworkTopologyStrategy);
+state.out.println(
'org.apache.cassandra.locator.SimpleStrategy',);
+state.out.println(
'org.apache.cassandra.locator.NetworkTopologyStrategy',);
+state.out.println(and
'org.apache.cassandra.locator.OldNetworkTopologyStrategy');
state.out.println( strategy_options: additional options
for placement_strategy.);
state.out.println(Applies only to
NetworkTopologyStrategy.);
state.out.println(\nexamples:);
[jira] [Commented] (CASSANDRA-2864) Alternative Row Cache Implementation
[ https://issues.apache.org/jira/browse/CASSANDRA-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061303#comment-13061303 ] Daniel Doubleday commented on CASSANDRA-2864: - bq. However, I'm reluctant to add more special cases to the read path Well I was more thinking of replacing the old row cache :-) In terms of throughput over latency this might be a winner. It seems read latencies increase only moderately combined with lowered mem usage ... Also I think that implementing a variation of CASSANDRA-1956 will be pretty easy since we can work with the standard filters now. So instead of putting toplevel columns back in the cache one could just cache the filtered columns. Plus a little logic that decides wether the cache can handle the request. But I understand that this is quite a change and the patch is easy to maintain so we can always patch. bq. It looks like the CASSANDRA-2498 + CASSANDRA-2503 approach might offer similar benefits These look promising but it seems that they dont help for slicing (CASSANDRA-2503 might make the slicing case even worse) and FWIW we do slice a lot even in skinny cached rows. Looks like we have the worst case scenario there: lots of random updates (in terms of ranges) so even if there were cached range meta infos for sstables somewhere I doubt that it would really work for us. Anyways I will look at CASSANDRA-2498 this weekend and check if I think that I can come up with a patch. Or else report that I cant. Alternative Row Cache Implementation Key: CASSANDRA-2864 URL: https://issues.apache.org/jira/browse/CASSANDRA-2864 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.1 Reporter: Daniel Doubleday Priority: Minor Attachments: rowcache.patch we have been working on an alternative implementation to the existing row cache(s) We have 2 main goals: - Decrease memory - get more rows in the cache without suffering a huge performance penalty - Reduce gc pressure This sounds a lot like we should be using the new serializing cache in 0.8. Unfortunately our workload consists of loads of updates which would invalidate the cache all the time. The second unfortunate thing is that the idea we came up with doesn't fit the new cache provider api... It looks like this: Like the serializing cache we basically only cache the serialized byte buffer. we don't serialize the bloom filter and try to do some other minor compression tricks (var ints etc not done yet). The main difference is that we don't deserialize but use the normal sstable iterators and filters as in the regular uncached case. So the read path looks like this: return filter.collectCollatedColumns(memtable iter, cached row iter) The write path is not affected. It does not update the cache During flush we merge all memtable updates with the cached rows. The attached patch is based on 0.8 branch r1143352 It does not replace the existing row cache but sits aside it. Theres environment switch to choose the implementation. This way it is easy to benchmark performance differences. -DuseSSTableCache=true enables the alternative cache. It shares its configuration with the standard row cache. So the cache capacity is shared. We have duplicated a fair amount of code. First we actually refactored the existing sstable filter / reader but than decided to minimize dependencies. Also this way it is easy to customize serialization for in memory sstable rows. We have also experimented a little with compaction but since this task at this stage is mainly to kick off discussion we wanted to keep things simple. But there is certainly room for optimizations. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2864) Alternative Row Cache Implementation
[ https://issues.apache.org/jira/browse/CASSANDRA-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Doubleday updated CASSANDRA-2864: Description: we have been working on an alternative implementation to the existing row cache(s) We have 2 main goals: - Decrease memory - get more rows in the cache without suffering a huge performance penalty - Reduce gc pressure This sounds a lot like we should be using the new serializing cache in 0.8. Unfortunately our workload consists of loads of updates which would invalidate the cache all the time. The second unfortunate thing is that the idea we came up with doesn't fit the new cache provider api... It looks like this: Like the serializing cache we basically only cache the serialized byte buffer. we don't serialize the bloom filter and try to do some other minor compression tricks (var ints etc not done yet). The main difference is that we don't deserialize but use the normal sstable iterators and filters as in the regular uncached case. So the read path looks like this: return filter.collectCollatedColumns(memtable iter, cached row iter) The write path is not affected. It does not update the cache During flush we merge all memtable updates with the cached rows. The attached patch is based on 0.8 branch r1143352 It does not replace the existing row cache but sits aside it. Theres environment switch to choose the implementation. This way it is easy to benchmark performance differences. -DuseSSTableCache=true enables the alternative cache. It shares its configuration with the standard row cache. So the cache capacity is shared. We have duplicated a fair amount of code. First we actually refactored the existing sstable filter / reader but than decided to minimize dependencies. Also this way it is easy to customize serialization for in memory sstable rows. We have also experimented a little with compression but since this task at this stage is mainly to kick off discussion we wanted to keep things simple. But there is certainly room for optimizations. was: we have been working on an alternative implementation to the existing row cache(s) We have 2 main goals: - Decrease memory - get more rows in the cache without suffering a huge performance penalty - Reduce gc pressure This sounds a lot like we should be using the new serializing cache in 0.8. Unfortunately our workload consists of loads of updates which would invalidate the cache all the time. The second unfortunate thing is that the idea we came up with doesn't fit the new cache provider api... It looks like this: Like the serializing cache we basically only cache the serialized byte buffer. we don't serialize the bloom filter and try to do some other minor compression tricks (var ints etc not done yet). The main difference is that we don't deserialize but use the normal sstable iterators and filters as in the regular uncached case. So the read path looks like this: return filter.collectCollatedColumns(memtable iter, cached row iter) The write path is not affected. It does not update the cache During flush we merge all memtable updates with the cached rows. The attached patch is based on 0.8 branch r1143352 It does not replace the existing row cache but sits aside it. Theres environment switch to choose the implementation. This way it is easy to benchmark performance differences. -DuseSSTableCache=true enables the alternative cache. It shares its configuration with the standard row cache. So the cache capacity is shared. We have duplicated a fair amount of code. First we actually refactored the existing sstable filter / reader but than decided to minimize dependencies. Also this way it is easy to customize serialization for in memory sstable rows. We have also experimented a little with compaction but since this task at this stage is mainly to kick off discussion we wanted to keep things simple. But there is certainly room for optimizations. Alternative Row Cache Implementation Key: CASSANDRA-2864 URL: https://issues.apache.org/jira/browse/CASSANDRA-2864 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.1 Reporter: Daniel Doubleday Priority: Minor Attachments: rowcache.patch we have been working on an alternative implementation to the existing row cache(s) We have 2 main goals: - Decrease memory - get more rows in the cache without suffering a huge performance penalty - Reduce gc pressure This sounds a lot like we should be using the new serializing cache in 0.8. Unfortunately our workload consists of loads of updates which would invalidate the cache all the time. The second unfortunate thing is that the idea we came up with doesn't fit the new cache provider api... It
[jira] [Created] (CASSANDRA-2868) Native Memory Leak
Native Memory Leak -- Key: CASSANDRA-2868 URL: https://issues.apache.org/jira/browse/CASSANDRA-2868 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.6 Reporter: Daniel Doubleday Priority: Minor We have memory issues with long running servers. These have been confirmed by several users in the user list. That's why I report. The memory consumption of the cassandra java process increases steadily until it's killed by the os because of oom (with no swap) Our server is started with -Xmx3000M and running for around 23 days. pmap -x shows Total SST: 1961616 (mem mapped data and index files) Anon RSS: 6499640 Total RSS: 8478376 This shows that 3G are 'overallocated'. We will use BRAF on one of our less important nodes to check wether it is related to mmap and report back. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1143847 - in /cassandra/branches/cassandra-0.7: CHANGES.txt NEWS.txt build.xml debian/changelog test/unit/org/apache/cassandra/locator/EC2SnitchTest.java
Author: slebresne
Date: Thu Jul 7 14:35:03 2011
New Revision: 1143847
URL: http://svn.apache.org/viewvc?rev=1143847view=rev
Log:
Updates for 0.7.7 release
Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt
cassandra/branches/cassandra-0.7/NEWS.txt
cassandra/branches/cassandra-0.7/build.xml
cassandra/branches/cassandra-0.7/debian/changelog
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/locator/EC2SnitchTest.java
Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1143847r1=1143846r2=1143847view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Thu Jul 7 14:35:03 2011
@@ -34,6 +34,9 @@
* improve cli treatment of multiline comments (CASSANDRA-2852)
* fix describeOwnership for OPP (CASSANDRA-2800)
* ensure that string tokens do not contain commas (CASSANDRA-2762)
+ * add ant-optional as dependence for the debian package (CASSANDRA-2164)
+ * add option to specify limit for get_slice in the CLI (CASSANDRA-2646)
+ * decrease HH page size (CASSANDRA-2832)
0.7.6
Modified: cassandra/branches/cassandra-0.7/NEWS.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/NEWS.txt?rev=1143847r1=1143846r2=1143847view=diff
==
--- cassandra/branches/cassandra-0.7/NEWS.txt (original)
+++ cassandra/branches/cassandra-0.7/NEWS.txt Thu Jul 7 14:35:03 2011
@@ -1,3 +1,12 @@
+0.7.7
+=
+
+Upgrading
+-
+- Nothing specific to 0.7.7, but see 0.7.3 Upgrading if upgrading
+ from earlier than 0.7.1.
+
+
0.7.6
=
Modified: cassandra/branches/cassandra-0.7/build.xml
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/build.xml?rev=1143847r1=1143846r2=1143847view=diff
==
--- cassandra/branches/cassandra-0.7/build.xml (original)
+++ cassandra/branches/cassandra-0.7/build.xml Thu Jul 7 14:35:03 2011
@@ -24,7 +24,7 @@
property name=debuglevel value=source,lines,vars/
!-- default version and SCM information (we need the default SCM info as
people may checkout with git-svn) --
-property name=base.version value=0.7.6/
+property name=base.version value=0.7.7/
property name=scm.default.path
value=cassandra/branches/cassandra-0.7/
property name=scm.default.connection
value=scm:svn:http://svn.apache.org/repos/asf/${scm.default.path}/
property name=scm.default.developerConnection
value=scm:svn:https://svn.apache.org/repos/asf/${scm.default.path}/
Modified: cassandra/branches/cassandra-0.7/debian/changelog
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/debian/changelog?rev=1143847r1=1143846r2=1143847view=diff
==
--- cassandra/branches/cassandra-0.7/debian/changelog (original)
+++ cassandra/branches/cassandra-0.7/debian/changelog Thu Jul 7 14:35:03 2011
@@ -1,3 +1,9 @@
+cassandra (0.7.7) unstable; urgency=low
+
+ * New stable point release
+
+ -- Sylvain Lebresne slebre...@apache.org Thu, 07 Jul 2011 16:32:35 +0200
+
cassandra (0.7.6) unstable; urgency=low
* New stable point release.
Modified:
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/locator/EC2SnitchTest.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/locator/EC2SnitchTest.java?rev=1143847r1=1143846r2=1143847view=diff
==
---
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/locator/EC2SnitchTest.java
(original)
+++
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/locator/EC2SnitchTest.java
Thu Jul 7 14:35:03 2011
@@ -1,4 +1,25 @@
package org.apache.cassandra.locator;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
import static org.junit.Assert.assertEquals;
[jira] [Commented] (CASSANDRA-2861) Building RPMs should be easy
[ https://issues.apache.org/jira/browse/CASSANDRA-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061365#comment-13061365 ] Niels Basjes commented on CASSANDRA-2861: - I've just successfully deployed the RPM built using this patch on a test cluster of 12 (desktop) machines running CentOS 5.6 32-bit (needs EPEL to install). Building RPMs should be easy Key: CASSANDRA-2861 URL: https://issues.apache.org/jira/browse/CASSANDRA-2861 Project: Cassandra Issue Type: Improvement Components: Packaging Affects Versions: 0.8.1 Reporter: Niels Basjes Priority: Minor Labels: ant, packaging, rpm Fix For: 0.8.2 Attachments: CASSANDRA-ANT-RPM-2011-07-05.patch, CASSANDRA-ANT-RPM-2011-07-07.patch I think that building the rpm for cassandra should be no brainer. The currently available .spec file needs manual tweaking(mainly the version number) before you can built the rpm. Goal is to make it as easy as doing ant rpm or ant -Drelease=true rpm -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2805) Clean up mbeans that return Internal Cassandra types
[ https://issues.apache.org/jira/browse/CASSANDRA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061413#comment-13061413 ] Nick Bailey commented on CASSANDRA-2805: Ed, I'm not sure I see the advantage of a JMX Composite type as opposed to using something like a MapString, Int approach in any place that needs 'complex' types. Clean up mbeans that return Internal Cassandra types Key: CASSANDRA-2805 URL: https://issues.apache.org/jira/browse/CASSANDRA-2805 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Nick Bailey Assignee: Gaurav Sharma Priority: Minor Labels: lhf Fix For: 0.8.2 We need to clean up wherever we return internal cassandra objects over jmx. Namely CompactionInfo objects as well as Tokens. There may be a few other examples. This is bad for two reasons 1. You have to load the cassandra jar when querying these mbeans, which sucks. 2. Stuff breaks between versions when things are moved. For example, CASSANDRA-1610 moves the compaction related classes around. Any code querying those jmx mbeans in 0.8.0 is now broken in 0.8.2. (assuming those moves stay in the 0.8 branch) For things like CompactionInfo we should just expose more mbean methods or serialize to something standard like json. I'd like to target this for 0.8.2. Since we've already broken compatibility between 0.8.0 and 0.8.1, I'd say just fix this everywhere now. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2820) Re-introduce FastByteArrayInputStream (and Output equivalent)
[ https://issues.apache.org/jira/browse/CASSANDRA-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Loy updated CASSANDRA-2820: Attachment: fast_bytearray_iostreams_harmony-patch-4.txt More like my original: * extends Input/OutputStream * change uses of ByteArrayInputStream to use InputStream * uses of ByteArrayOutputStream have to now know they are using FastByteArrayOutputStream as that has the #toArray() method. Re-introduce FastByteArrayInputStream (and Output equivalent) - Key: CASSANDRA-2820 URL: https://issues.apache.org/jira/browse/CASSANDRA-2820 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Environment: n/a Reporter: Paul Loy Priority: Minor Labels: bytearrayinputstream, bytearrayoutputstream, license, synchronized Fix For: 1.0 Attachments: fast_bytearray_iostreams_harmony-patch-2.txt, fast_bytearray_iostreams_harmony-patch-3.txt, fast_bytearray_iostreams_harmony-patch-4.txt In https://issues.apache.org/jira/browse/CASSANDRA-37 FastByteArrayInputStream and FastByteArrayOutputStream were removed due to being code copied from the JDK and then subsequently modified. The JDK license is incompatible with Apache 2 license so the code had to go. I have since had a look at the performance of the JDK ByteArrayInputStream and a FastByteArrayInputStream (i.e. one with synchronized methods made un-synchronized) and seen the difference is significant. After a warmup-period of 1 loops I get the following for 1 loops through a 128000 byte array: bais : 3513ms fbais: 72ms This varies depending on the OS, machine and Java version, but it's always in favour of the FastByteArrayInputStream as you might expect. Then, at Jonathan Ellis' suggestion, I tried this using a modified Apache Harmony ByteArrayInputStream - i.e. one whose license is compatible - and the results were the same. A significant boost. I will attach a patch with changes for the 0.8.0 tag. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2869) CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues
CassandraStorage does not function properly when used multiple times in a
single pig script due to UDFContext sharing issues
Key: CASSANDRA-2869
URL: https://issues.apache.org/jira/browse/CASSANDRA-2869
Project: Cassandra
Issue Type: Bug
Components: Contrib
Reporter: Grant Ingersoll
CassandraStorage appears to have threading issues along the lines of those
described at http://pig.markmail.org/message/oz7oz2x2dwp66eoz due to the
sharing of the UDFContext.
I believe the fix lies in implementing
{code}
public void setStoreFuncUDFContextSignature(String signature)
{
}
{code}
and then using that signature when getting the UDFContext.
From the Pig manual:
{quote}
setStoreFunc!UDFContextSignature(): This method will be called by Pig both in
the front end and back end to pass a unique signature to the Storer. The
signature can be used to store into the UDFContext any information which the
Storer needs to store between various method invocations in the front end and
back end. The default implementation in StoreFunc has an empty body. This
method will be called before other methods.
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2869) CassandraStorage does not function properly when used multiple times in a single pig script due to UDFContext sharing issues
[
https://issues.apache.org/jira/browse/CASSANDRA-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Ingersoll updated CASSANDRA-2869:
---
Affects Version/s: 0.7.2
CassandraStorage does not function properly when used multiple times in a
single pig script due to UDFContext sharing issues
Key: CASSANDRA-2869
URL: https://issues.apache.org/jira/browse/CASSANDRA-2869
Project: Cassandra
Issue Type: Bug
Components: Contrib
Affects Versions: 0.7.2
Reporter: Grant Ingersoll
CassandraStorage appears to have threading issues along the lines of those
described at http://pig.markmail.org/message/oz7oz2x2dwp66eoz due to the
sharing of the UDFContext.
I believe the fix lies in implementing
{code}
public void setStoreFuncUDFContextSignature(String signature)
{
}
{code}
and then using that signature when getting the UDFContext.
From the Pig manual:
{quote}
setStoreFunc!UDFContextSignature(): This method will be called by Pig both in
the front end and back end to pass a unique signature to the Storer. The
signature can be used to store into the UDFContext any information which the
Storer needs to store between various method invocations in the front end and
back end. The default implementation in StoreFunc has an empty body. This
method will be called before other methods.
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2870) dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return
spurious UnavailableException
-
Key: CASSANDRA-2870
URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 0.7.8
When Read Repair is off, we want to avoid doing requests to more nodes than
necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
{code}
this.endpoints = repair || resolver instanceof RowRepairResolver
? endpoints
: endpoints.subList(0, Math.min(endpoints.size(),
blockfor)); // min so as to not throw exception until assureSufficient is called
{code}
You can see that it is assuming that the endpoints list is sorted in order of
preferred-ness for the read.
Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have
enough nodes to do the read:
{code}
int localEndpoints = 0;
for (InetAddress endpoint : endpoints)
{
if (localdc.equals(snitch.getDatacenter(endpoint)))
localEndpoints++;
}
if (localEndpoints blockfor)
throw new UnavailableException();
{code}
So if repair is off (so we truncate our endpoints list) AND dynamic snitch has
decided that nodes in another DC are to be preferred over local ones, we'll
throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2870) dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return spurious UnavailableException
[
https://issues.apache.org/jira/browse/CASSANDRA-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-2870:
--
Attachment: 2870.txt
patch extracts the give me the minimum set of endpoints necessary code into
preferredEndpoints(), and makes it DC-aware for LOCAL_QUORUM reads.
dynamic snitch + read repair off can cause LOCAL_QUORUM reads to return
spurious UnavailableException
-
Key: CASSANDRA-2870
URL: https://issues.apache.org/jira/browse/CASSANDRA-2870
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 0.7.8, 0.8.2
Attachments: 2870.txt
When Read Repair is off, we want to avoid doing requests to more nodes than
necessary to satisfy the ConsistencyLevel. ReadCallback does this here:
{code}
this.endpoints = repair || resolver instanceof RowRepairResolver
? endpoints
: endpoints.subList(0, Math.min(endpoints.size(),
blockfor)); // min so as to not throw exception until assureSufficient is
called
{code}
You can see that it is assuming that the endpoints list is sorted in order
of preferred-ness for the read.
Then the LOCAL_QUORUM code in DatacenterReadCallback checks to see if we have
enough nodes to do the read:
{code}
int localEndpoints = 0;
for (InetAddress endpoint : endpoints)
{
if (localdc.equals(snitch.getDatacenter(endpoint)))
localEndpoints++;
}
if (localEndpoints blockfor)
throw new UnavailableException();
{code}
So if repair is off (so we truncate our endpoints list) AND dynamic snitch
has decided that nodes in another DC are to be preferred over local ones,
we'll throw UE even if all the replicas are healthy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2475) Prepared statements
[
https://issues.apache.org/jira/browse/CASSANDRA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061743#comment-13061743
]
Rick Shaw commented on CASSANDRA-2475:
--
Yes.
The confusing part would be the use of ? in a {{Statement}} rather than a
{{PreparedStatement}}. I guess it will be allowed syntactically so it will
generate the prescribed token stream but it will be a semantic error when the
token string is (immediately) executed. The fetch the next Term parameter
token will be encountered but the next parameter will not exist so that would
signal an error.
Prepared statements
---
Key: CASSANDRA-2475
URL: https://issues.apache.org/jira/browse/CASSANDRA-2475
Project: Cassandra
Issue Type: Sub-task
Components: API, Core
Reporter: Eric Evans
Labels: cql
Fix For: 1.0
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2521) Move away from Phantom References for Compaction/Memtable
[ https://issues.apache.org/jira/browse/CASSANDRA-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13061761#comment-13061761 ] Jeff Kesselman commented on CASSANDRA-2521: --- Just a note: while relying on phantom references or finalizer for your cleanup is incorrect, it is reasonable to use them to do a verification that your cleanup has occurred on those objects that do get collected. This is with the caveat however that PhantomReferences always add something to your gc costs so you might want to make it switchable. Move away from Phantom References for Compaction/Memtable - Key: CASSANDRA-2521 URL: https://issues.apache.org/jira/browse/CASSANDRA-2521 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Chris Goffinet Assignee: Sylvain Lebresne Fix For: 1.0 Attachments: 0001-Use-reference-counting-to-decide-when-a-sstable-can-.patch, 0001-Use-reference-counting-to-decide-when-a-sstable-can-v2.patch, 0002-Force-unmapping-files-before-deletion-v2.patch, 2521-v3.txt, 2521-v4.txt http://wiki.apache.org/cassandra/MemtableSSTable Let's move to using reference counting instead of relying on GC to be called in StorageService. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
