[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data
[ https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433951#comment-13433951 ] Tommy Cheng commented on CASSANDRA-4538: Yes, consistently reproducible. Funny thing is that another machine is okay. I tried format the OS, health check the ram/harddisk, and all test pass. What extra thing should i provide? It is very important to find out the problem before really use cassandra for production. > Strange CorruptedBlockException when massive insert binary data > --- > > Key: CASSANDRA-4538 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4538 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.3 > Environment: Debian sequeeze 32bit >Reporter: Tommy Cheng >Priority: Critical > Labels: CorruptedBlockException, binary, insert > Attachments: cassandra-stresstest.zip > > > After inserting ~ 1 records, here is the error log > INFO 10:53:33,543 Compacted to > [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,]. > 407,681 to 409,133 (~100% of original) bytes for 9,250 keys at > 0.715926MB/s. Time: 545ms. > ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main] > java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:99) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377) > at > org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144) > at > org.apache.cas
[jira] [Created] (CASSANDRA-4539) potential backwards incompatibility in native protocol
paul cannon created CASSANDRA-4539: -- Summary: potential backwards incompatibility in native protocol Key: CASSANDRA-4539 URL: https://issues.apache.org/jira/browse/CASSANDRA-4539 Project: Cassandra Issue Type: Improvement Components: API Affects Versions: 1.2 Reporter: paul cannon Assignee: Sylvain Lebresne Priority: Minor Fix For: 1.2 In the text of the native_protocol.spec document, it explains the format for a notation called {{[option]}}, which should represent "{{a pair of }}". In doing a first-draft implementation of the protocol for the python driver, though, I found that I had a misunderstanding. I read that section as saying that {{}} was a {{[value]}}, and that it could have a length of 0 (i.e., the {{[int]}} on the front of the {{[value]}} could be 0). However, it turns out that {{}} might not be there at all, or might be *two* {{[value]}}'s, depending on the option id and message context. I'm not a fan of this, since * A protocol parsing library can't simply implement a single function to read in {{[option]}}'s, since the length of the value part is dependent on the message context * If we add a new native data type (a new option id which could be used inside a {{}} in a RESULT message), then older clients will not know how to read past the value part. Of course they won't know how to interpret the data or deserialize later rows of that unknown type - that's not the problem - the problem is that the older client should be able just to mark that column as unparseable and still handle the rest of the columns. Can we make {{}} be a {{[value]}}, the contents of which can be re-interpreted as a {{[string]}}, an {{[option]}}, two {{[option]}}'s, or whatever? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4540) nodetool clearsnapshot broken: gives java.io.IOException when trying to delete snapshot folder
Christopher Lörken created CASSANDRA-4540: - Summary: nodetool clearsnapshot broken: gives java.io.IOException when trying to delete snapshot folder Key: CASSANDRA-4540 URL: https://issues.apache.org/jira/browse/CASSANDRA-4540 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.2 Environment: Debian 6 Reporter: Christopher Lörken Priority: Minor nodetool clearsnapshots failes to delete snapshot directories and exits prematurely causing the exception at the bottom. The actual snapshot files _within_ the directory are correctly deleted but the folder itself is not deleted. I've chmodded all files and folders in the snapshots directory to 777 and rund the command as root to exclude file permissions as a cause. I also restarted cassandra which has no effect on the command. --- root@server:/var/lib/cassandra/data/MyKeyspace/MyCf/snapshots# nodetool clearsnapshot MyKeyspace Requested snapshot for: MyKeyspace Exception in thread "main" java.io.IOException: Failed to delete /var/lib/cassandra/data/MyKeyspace/MyCf/snapshots/1344875270796 at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:220) at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:216) at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:371) at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1560) at org.apache.cassandra.db.Table.clearSnapshot(Table.java:268) at org.apache.cassandra.service.StorageService.clearSnapshot(StorageService.java:1866) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434048#comment-13434048 ] Nicolas Favre-Felix commented on CASSANDRA-4482: Jonathan, I wrote the blog post linked in this ticket; the incremental repair process we've implemented is not doing any random I/O on insert as you suggest. Instead, we maintain a Merkle Tree (MT) in memory and update it with every single column insert in ColumnFamilyStore.apply(). We use column.updateDigest(digest) on all the changes in order to create a hash per column update and then XOR this hash with the existing one in the Merkle Tree bucket for the corresponding row. This Merkle Tree is created with the column family (one per range), initialized with zeros, and persisted to disk with regular snapshots. The commutative properties of XOR make it possible to update the MT incrementally without having to read on write. When an incremental repair session starts, the CFS swap out their existing MTs for new empty ones that will receive subsequent updates. There are a few downsides to this approach: * It is possible for the incremental MTs to miss a few inserts that happen when the replicas involved swap out their MTs for new ones. An insert will be in the previous MT for node "A" but in the fresh one for node "B", for instance. This leads to either a very small amount of extra streaming or some unrepaired changes. For this reason, we still recommend that users run either a full repair or a "tombstone-only repair" at least once every GCGraceSeconds. * There is some overhead to keeping these MTs in memory. We actually maintain only the leaves as a single ByteBuffer instead of creating all the intermediate nodes like the MerkleTree class does. To avoid using too much RAM, we allocate a fixed amount of memory per CF and divide it into a number of smaller buffers (one per range) in order to give the same guarantees regardless of the number of ranges per CF. * There is a small cost in insert, about half of which is due to the hash function (MD5). We are looking into making our patch available to the community and would welcome suggestions to solve or improve on these limitations. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4540) nodetool clearsnapshot broken: gives java.io.IOException when trying to delete snapshot folder
[ https://issues.apache.org/jira/browse/CASSANDRA-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4540: -- Priority: Trivial (was: Minor) Fix Version/s: 1.1.5 Labels: lhf (was: ) > nodetool clearsnapshot broken: gives java.io.IOException when trying to > delete snapshot folder > -- > > Key: CASSANDRA-4540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4540 > Project: Cassandra > Issue Type: Bug > Components: Tools >Affects Versions: 1.1.2 > Environment: Debian 6 >Reporter: Christopher Lörken >Priority: Trivial > Labels: lhf > Fix For: 1.1.5 > > > nodetool clearsnapshots failes to delete snapshot directories and exits > prematurely causing the exception at the bottom. > The actual snapshot files _within_ the directory are correctly deleted but > the folder itself is not deleted. > I've chmodded all files and folders in the snapshots directory to 777 and > rund the command as root to exclude file permissions as a cause. I also > restarted cassandra which has no effect on the command. > --- > root@server:/var/lib/cassandra/data/MyKeyspace/MyCf/snapshots# nodetool > clearsnapshot MyKeyspace > Requested snapshot for: MyKeyspace > Exception in thread "main" java.io.IOException: Failed to delete > /var/lib/cassandra/data/MyKeyspace/MyCf/snapshots/1344875270796 > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) > at > org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:220) > at > org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:216) > at > org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:371) > at > org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1560) > at org.apache.cassandra.db.Table.clearSnapshot(Table.java:268) > at > org.apache.cassandra.service.StorageService.clearSnapshot(StorageService.java:1866) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown > Source) > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown > Source) > at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) > at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) > at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown > Source) > at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown > Source) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown > Source) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown > Source) > at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown > Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) > at sun.rmi.transport.Transport$1.run(Unknown Source) > at sun.rmi.transport.Transport$1.run(Unknown Source) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Unknown Source) > at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown > Source) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434091#comment-13434091 ] Jonathan Ellis commented on CASSANDRA-4482: --- bq. The commutative properties of XOR make it possible to update the MT incrementally without having to read on write. Thanks for the clarification, Nicolas. That sounds like a reasonable approach. bq. To avoid using too much RAM, we allocate a fixed amount of memory per CF and divide it into a number of smaller buffers (one per range) in order to give the same guarantees regardless of the number of ranges per CF Meaning, you give each CF less than 64k ranges * 16 bytes / range? > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434099#comment-13434099 ] T Jake Luciani commented on CASSANDRA-4482: --- Is there a startup cost associated with the approach? i.e. How to you know the initial hash? > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434105#comment-13434105 ] Nicolas Favre-Felix commented on CASSANDRA-4482: bq. Meaning, you give each CF less than 64k ranges * 16 bytes / range? Right, that would be too much. At the moment, we give each CF 256 KB to be split into all of its ranges. For num_tokens=256, that's 1 KB per range on average - we do not yet scale this number according to the range size. A node with num_tokens = 1 owning a single range would allocate 256 KB in a single direct ByteBuffer. Moving to num_tokens = 256 gives the ColumnFamilyStore 256 ranges, and allocates a 1 KB ByteBuffer per range. In both cases the keys in any given range are covered by as many "leaf bytes" on average, regardless of the number of ranges. bq. Is there a startup cost associated with the approach? i.e. How to you know the initial hash? We do have to reload $num_tokens ByteBuffers when creating the ColumnFamilyStore, for a total of 256KB per CF with our current defaults. This is not something we've measured but I suspect that the cost is fairly small, as it is now for the cache snapshots: it is O(number of CFs), not O(N) like the old cache preloads. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3609) Assertion in ExpiringColumn
[ https://issues.apache.org/jira/browse/CASSANDRA-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar resolved CASSANDRA-3609. Resolution: Cannot Reproduce > Assertion in ExpiringColumn > -- > > Key: CASSANDRA-3609 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3609 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.5 >Reporter: Radim Kolar >Priority: Minor > > ERROR [MutationStage:1773] 2011-12-10 16:36:10,526 > AbstractCassandraDaemon.java (line 133) Fatal exception in thread > Thread[MutationStage:1773,5,main] > java.lang.AssertionError: -823952279 > at > org.apache.cassandra.db.ExpiringColumn.(ExpiringColumn.java:58) > at > org.apache.cassandra.db.ExpiringColumn.(ExpiringColumn.java:51) > at > org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:154) > at org.apache.cassandra.db.RowMutation.add(RowMutation.java:199) > at org.apache.cassandra.db.RowMutation.hintFor(RowMutation.java:140) > at > org.apache.cassandra.service.StorageProxy$5.runMayThrow(StorageProxy.java:364) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434150#comment-13434150 ] Jonathan Ellis commented on CASSANDRA-4482: --- bq. The commutative properties of XOR make it possible to update the MT incrementally without having to read on write Hang on, let's flesh this out. I have an md5 hash (or part of one, see below) per row in a MerkleTree TreeRange. I xor all these together to get my initial state, S. To update row A to row A', I need to take S xor hash(A) xor hash(A'). So I still need to read-on-write to compute hash(A), I just don't have to rehash everything else in the same TreeRange. (I can imagine breaking this down into xoring individual columns, which would mean we would only need to read modified columns and not the entire row, but the principle is the same.) bq. For num_tokens=256, that's 1 KB per range on average I see, you mean vnode ranges. What I meant was MT TreeRanges... a MT can have 64k TR. Ideally you will have 16 bytes (md5 size) per TR. You can throw away some bytes at the cost of false negatives, i.e., with a single byte per TR, two replicas will think they have the same data even when they do not 1/256 of the time. But if you have 64k 1-byte treeranges, how do you fit that into 1KB? Do you reduce the TR granularity further? 64k already feels too low... although this is mitigated somewhat by vnodes. bq. do have to reload $num_tokens ByteBuffers when creating the ColumnFamilyStore And sync the BB saving with CF flushes so CL replay matches up, I imagine. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4541) Replace Throttle with Guava's RateLimiter
Michaël Figuière created CASSANDRA-4541: --- Summary: Replace Throttle with Guava's RateLimiter Key: CASSANDRA-4541 URL: https://issues.apache.org/jira/browse/CASSANDRA-4541 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.1.3 Reporter: Michaël Figuière Guava 13 introduced {{RateLimiter}} [1] which should be a good replacement for {{o.a.c.utils.Throttle}} that is used in Compaction and Streaming as a throughput limiter. [1] [http://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/util/concurrent/RateLimiter.java] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4541) Replace Throttle with Guava's RateLimiter
[ https://issues.apache.org/jira/browse/CASSANDRA-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michaël Figuière updated CASSANDRA-4541: Priority: Minor (was: Major) > Replace Throttle with Guava's RateLimiter > - > > Key: CASSANDRA-4541 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4541 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1.3 >Reporter: Michaël Figuière >Priority: Minor > > Guava 13 introduced {{RateLimiter}} [1] which should be a good replacement > for {{o.a.c.utils.Throttle}} that is used in Compaction and Streaming as a > throughput limiter. > [1] > [http://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/util/concurrent/RateLimiter.java] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434171#comment-13434171 ] Nicolas Favre-Felix commented on CASSANDRA-4482: Indeed, I've used "range" as Range, the range of tokens owned by a node; I should have made this clearer. We are not using the MerkleTree class or its TreeRange objects, but updating a single ByteBuffer directly instead of creating the whole tree with its hundreds of internal objects. This is equivalent to updating the leaves alone, without propagating the hash upwards in the tree. Yes, that means comparing two trees is O(leaf count). bq. I xor all these together to get my initial state, S. To update row A to row A', I need to take S xor hash(A) xor hash(A'). If you've lready xor'd all these together, S does include the hash of your existing row A. Updating A to A' hashes A' and returns S' = S xor hash(A'), which is hash(A') xor hash(A). In practice, this is how it works step by step: # Load existing buffers when the ColumnFamilyStore is created: per Range, load an existing buffer or create a new one initialized with zeros. # ColumnFamilyStore.apply() is called with columns X and Y in row A. For instance, row A could have token 0x10, falling in the range (0x00, 0x20]. The incremental repair ByteBuffer for this range is 1 KB in size. # Create a new digest and run Column.updateDigest() on X and Y sucessively. We end up with H = hash(X) xor hash(Y); H is 16 bytes long. # Calculate O, the offset in the ByteBuffer that corresponds to H: in this case, it's around 512 since 0x10 is close to the middle of the range (0x00, 0x20]. # For each byte i of H, we set buffer[O+i] = buffer[O+i] xor H[i]. During the repair session, the replicas send out their existing ByteBuffers for the range being repaired and replace them with empty ones that will receive subsequent inserts. bq. And sync the BB saving with CF flushes so CL replay matches up, I imagine. Yes. If you terminate Cassandra at this stage, the ByteBuffer is written to disk and will contains [0,0 a few bytes of hash(X) xor hash(Y) around the middle ... 0,0,0,0]. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434175#comment-13434175 ] Jonathan Ellis commented on CASSANDRA-4482: --- bq. If you've lready xor'd all these together, S does include the hash of your existing row A. Updating A to A' hashes A' Right, I'm saying that given S-including-hash-of-A, what you want is to take A *out* when you add A'. Otherwise, if you have a node that (correctly) has A', but doesn't know about A (maybe it was compacted out before it got to build S), then it won't agree on the same state S even though it has the same data. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4541) Replace Throttle with Guava's RateLimiter
[ https://issues.apache.org/jira/browse/CASSANDRA-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-4541: - Assignee: Yuki Morishita > Replace Throttle with Guava's RateLimiter > - > > Key: CASSANDRA-4541 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4541 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michaël Figuière >Assignee: Yuki Morishita >Priority: Minor > > Guava 13 introduced {{RateLimiter}} [1] which should be a good replacement > for {{o.a.c.utils.Throttle}} that is used in Compaction and Streaming as a > throughput limiter. > [1] > [http://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/util/concurrent/RateLimiter.java] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4541) Replace Throttle with Guava's RateLimiter
[ https://issues.apache.org/jira/browse/CASSANDRA-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4541: -- Reviewer: xedin Affects Version/s: (was: 1.1.3) > Replace Throttle with Guava's RateLimiter > - > > Key: CASSANDRA-4541 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4541 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michaël Figuière >Assignee: Yuki Morishita >Priority: Minor > > Guava 13 introduced {{RateLimiter}} [1] which should be a good replacement > for {{o.a.c.utils.Throttle}} that is used in Compaction and Streaming as a > throughput limiter. > [1] > [http://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/util/concurrent/RateLimiter.java] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4285) Atomic, eventually-consistent batches
[ https://issues.apache.org/jira/browse/CASSANDRA-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434222#comment-13434222 ] Jonathan Ellis commented on CASSANDRA-4285: --- I think the above overcomplicates things a bit. The idea with sharding was to # mitigate wide batchlog rows # spread batchlog load better across the cluster # avoid having to fail a batch write if *the* batchlog replica for a given coordinator is down If we make the batchlog *replica* in charge of replay, these all go away or get simpler. # replica can examine batchlog entries and convert to hints if they are older than 2x write timeout. We expect very few of these so the inefficiency (over having coordinator do it only after restart) is not a Big Deal. This also simplifies replay a great deal (now coordinator no longer has poll for the BL replicas to come up) # since we never have to do a non-local *read* of the batchlog, we can use a WriteAnywhereStrategy that just picks a random node in the local DC # can be mitigated by making our hypothetical WAS FD-aware, or simply by going to RF=2 (and doing CL.ONE writes). I like having replay be local and automatic a great deal, over having the coordinator do it (which implies having some manual? failover mechanism when the coordinator is down for good). Note that we'd want to define batchlog with LocalStrategy (in system ks), we'd manually invoke WriteAnywhereStrategy from StorageProxy. Thinking about it, we probably wouldn't want an actual Strategy, just similar code, since we don't actually depend on the row key to pick replicas. > Atomic, eventually-consistent batches > - > > Key: CASSANDRA-4285 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4285 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis > > I discussed this in the context of triggers (CASSANDRA-1311) but it's useful > as a standalone feature as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4285) Atomic, eventually-consistent batches
[ https://issues.apache.org/jira/browse/CASSANDRA-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434224#comment-13434224 ] Jonathan Ellis commented on CASSANDRA-4285: --- Rolling our own "replication strategy" in storage proxy also allows us to pick min(2, nodes in cluster) as our replication factor out of the box. > Atomic, eventually-consistent batches > - > > Key: CASSANDRA-4285 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4285 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis > > I discussed this in the context of triggers (CASSANDRA-1311) but it's useful > as a standalone feature as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4285) Atomic, eventually-consistent batches
[ https://issues.apache.org/jira/browse/CASSANDRA-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4285: -- Reviewer: jbellis Assignee: Aleksey Yeschenko (was: Jonathan Ellis) > Atomic, eventually-consistent batches > - > > Key: CASSANDRA-4285 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4285 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Aleksey Yeschenko > > I discussed this in the context of triggers (CASSANDRA-1311) but it's useful > as a standalone feature as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4542) add atomic_batch_mutate method
Jonathan Ellis created CASSANDRA-4542: - Summary: add atomic_batch_mutate method Key: CASSANDRA-4542 URL: https://issues.apache.org/jira/browse/CASSANDRA-4542 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Fix For: 1.2 atomic_batch_mutate will have the same parameters as batch_mutate, but will write to the batchlog before attempting distribution to the batch rows' replicas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4543) batchlog replay
Jonathan Ellis created CASSANDRA-4543: - Summary: batchlog replay Key: CASSANDRA-4543 URL: https://issues.apache.org/jira/browse/CASSANDRA-4543 Project: Cassandra Issue Type: Sub-task Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4542) add atomic_batch_mutate method
[ https://issues.apache.org/jira/browse/CASSANDRA-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434248#comment-13434248 ] Jonathan Ellis commented on CASSANDRA-4542: --- Summary of discussion from CASSANDRA-4285: {code} CREATE TABLE batchlog ( coordinator inet, id uuid, datablob, PRIMARY KEY ((coordinator, shard)) ) WITH gc_grace_seconds=0; {code} abm adds extra steps before and after writing a batch. Before writing the batch (but *after* doing availability check against the FD for the row replicas): write the entire batch to a batchlog elsewhere in the cluster. After: remove the batchlog entry (after writing hints for the batch rows, if necessary). The batchlog itself should be defined with LocalStrategy. Replay will be handled locally (CASSANDRA-4543). Thus, we can't use the defined ReplicationStrategy when writing to the BL from StorageProxy; we should pick replicas manually: - replicas should be in the local datacenter - replicas should be alive according to the failure detector - write to min(2, number of qualifying candiates above) batchlog nodes Other notes: - need to add an optimization to flush that if the memtable is older than gc_g_s, we should do an extra removeDeleted pass to avoid writing tombstones that are already obsolete. this is crucial to keeping batchlog from becoming a new source of compaction pain. - need acknowledged_by_batchlog boolean for TimedOutException; if we timeout during the BL write (which is always effective CL.ONE) then we should return acknowledged_by_batchlog=false, acknowledged_by=0; if we timeout *after* the BL write, we should return acknowledged_by_batchlog=true, acknowledged_by=-1 > add atomic_batch_mutate method > -- > > Key: CASSANDRA-4542 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4542 > Project: Cassandra > Issue Type: Sub-task > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Aleksey Yeschenko > Fix For: 1.2 > > > atomic_batch_mutate will have the same parameters as batch_mutate, but will > write to the batchlog before attempting distribution to the batch rows' > replicas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4544) fix acknowledge_by for batch_mutate
Jonathan Ellis created CASSANDRA-4544: - Summary: fix acknowledge_by for batch_mutate Key: CASSANDRA-4544 URL: https://issues.apache.org/jira/browse/CASSANDRA-4544 Project: Cassandra Issue Type: Sub-task Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Priority: Minor CASSANDRA-4414 added TimedOutException.acknowledged_by, but for a batch write will send back the acknowledged_by for a random row, which usually does not reflect the status for the batch as a whole. Should override this to -1 in that case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4543) batchlog replay
[ https://issues.apache.org/jira/browse/CASSANDRA-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434257#comment-13434257 ] Jonathan Ellis commented on CASSANDRA-4543: --- After a batchlog entry (CASSANDRA-4542) is written, if it isn't removed within 2*rpc_write_timeout, we should be fairly safe in assuming that the coordinator failed and the batch will need to be replayed. (If in fact the coordinator is still alive but degraded, replaying it will be idempotent and harmless.) Suggest leveraging the existing hint mechanism for replay: turn the batch entry into its component mutations, hint those, and let HHOM replay when the target is alive. > batchlog replay > --- > > Key: CASSANDRA-4543 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4543 > Project: Cassandra > Issue Type: Sub-task >Reporter: Jonathan Ellis >Assignee: Aleksey Yeschenko > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4545) add cql support for batchlog
Jonathan Ellis created CASSANDRA-4545: - Summary: add cql support for batchlog Key: CASSANDRA-4545 URL: https://issues.apache.org/jira/browse/CASSANDRA-4545 Project: Cassandra Issue Type: Sub-task Reporter: Jonathan Ellis Need to expose the equivalent of atomic_batch_mutate (CASSANDRA-4542) to CQL3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4285) Atomic, eventually-consistent batches
[ https://issues.apache.org/jira/browse/CASSANDRA-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434264#comment-13434264 ] Jonathan Ellis commented on CASSANDRA-4285: --- Split implementation out into four subtasks. > Atomic, eventually-consistent batches > - > > Key: CASSANDRA-4285 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4285 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Aleksey Yeschenko > > I discussed this in the context of triggers (CASSANDRA-1311) but it's useful > as a standalone feature as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4545) add cql support for batchlog
[ https://issues.apache.org/jira/browse/CASSANDRA-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434263#comment-13434263 ] Jonathan Ellis commented on CASSANDRA-4545: --- Sylvain proposed making this the default, which I'm not convinced of -- I suspect the performance impact will not be negligible, so this could be an unpleasant surprise for upgraders. Either way though we need syntax to request batchlog either on or off, depending on which we do make the default. > add cql support for batchlog > > > Key: CASSANDRA-4545 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4545 > Project: Cassandra > Issue Type: Sub-task >Reporter: Jonathan Ellis > > Need to expose the equivalent of atomic_batch_mutate (CASSANDRA-4542) to CQL3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434267#comment-13434267 ] Nicolas Favre-Felix commented on CASSANDRA-4482: A node wouldn't have missed A because of a compaction but because A was not inserted there, since S is not built from the existing data on disk but incrementally with each change: S really represents the _combined history of all the changes_ performed on the token range since the last repair session. So nodes don't have to scan their data to build S, they simply start with S=0 when incremental repair is first enabled regardless of their initial differences and start again with S=0 after each incremental repair session. But it is indeed possible for two replicas to have the same data but differing values for S, for instance if a replica gets A and A' whereas another misses A but gets A': this would lead to some unnecessary streaming even though they both have the latest value A'. This could be avoided by removing A from S as you suggest, but the cost of doing random I/O after each write is too expensive, as you pointed out earlier. We are open to suggestions on how to improve this process and get this feature upstreamed with these issues addressed or understood as inherent limitations. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data
[ https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434333#comment-13434333 ] Cathy Daw commented on CASSANDRA-4538: -- I tried to reproduce on a 32-bit debian squeeze medium instance on EC2 and could not get the error. I wonder if you are dealing with a permanently corrupted as the result of a intermittent bug. Can you drop this column family and keyspace, recreate them, and then re-run the test? Can you also paste the DDL to create the column family? > Strange CorruptedBlockException when massive insert binary data > --- > > Key: CASSANDRA-4538 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4538 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.3 > Environment: Debian sequeeze 32bit >Reporter: Tommy Cheng >Priority: Critical > Labels: CorruptedBlockException, binary, insert > Attachments: cassandra-stresstest.zip > > > After inserting ~ 1 records, here is the error log > INFO 10:53:33,543 Compacted to > [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,]. > 407,681 to 409,133 (~100% of original) bytes for 9,250 keys at > 0.715926MB/s. Time: 545ms. > ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main] > java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:99) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377) > at > org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns
[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair
[ https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434406#comment-13434406 ] Jonathan Ellis commented on CASSANDRA-4482: --- bq. we still recommend that users run either a full repair or a "tombstone-only repair" at least once every GCGraceSeconds What is a tombstone-only repair? If we still need repair every gcgs, I'm not sure how much win there is here, given that with "durable HH" (CASSANDRA-2034) you only need AES repair when nodes die (or lose disks) permanently. Could be interesting to replace TreeRange with your optimized ByteBuffer (or BigLongArray -- CASSANDRA-3432) though, with or without full "incremental mode" later. I'd be glad to review a patch along those lines as a first step. > In-memory merkle trees for repair > - > > Key: CASSANDRA-4482 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4482 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson > > this sounds cool, we should reimplement it in the open source cassandra; > http://www.acunu.com/2/post/2012/07/incremental-repair.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4533) Multithreaded cache saving can skip caches
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4533: -- Reviewer: jbellis Description: Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time. However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1. Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true. I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either. was: Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time. However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1. Due to the effect of above atomic flag, only a cache can be written out to disk. Other writer are cancelled when the flag is true. I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either. Priority: Trivial (was: Major) Affects Version/s: (was: 1.1.3) (was: 1.0.11) 0.8.0 Fix Version/s: 1.1.5 Assignee: Yuki Morishita Summary: Multithreaded cache saving can skip caches (was: Cache saving does not work) Looks like we should switch to a ConcurrentSet like we did in Memtable.meteringInProgress. > Multithreaded cache saving can skip caches > -- > > Key: CASSANDRA-4533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4533 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Zhu Han >Assignee: Yuki Morishita >Priority: Trivial > Fix For: 1.1.5 > > > Cassandra flushes the key and row cache to disk periodically. It also uses a > atomic flag in flushInProgress to enforce single cache writer at any time. > However, the cache saving task could be submitted to CompactionManager > concurrently, as long as the number of worker thread in CompactionManager is > larger than 1. > Due to the effect of above atomic flag, only one cache will be written out to > disk. Other writer are cancelled when the flag is true. > I observe the situation in Cassandra 1.0. If nothing is changed, the problem > should remain in Cassandra 1.1, either. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2710) Get multiple column ranges
[ https://issues.apache.org/jira/browse/CASSANDRA-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2710: -- Reviewer: xedin Component/s: Core API Fix Version/s: 1.2 > Get multiple column ranges > -- > > Key: CASSANDRA-2710 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2710 > Project: Cassandra > Issue Type: Sub-task > Components: API, Core >Reporter: David Boxenhorn >Assignee: Vijay > Labels: compositeColumns, cql > Fix For: 1.2 > > Attachments: 0001-2710-multiple-column-ranges-cql.patch, > 0001-2710-multiple-column-ranges-thrift.patch > > > I have replaced all my super column families with regular column families > using composite columns. I have easily been able to support all previous > functionality (I don't need range delete) except for one thing: getting > multiple super columns with a single access. For this, I would need to get > multiple ranges. (I can get multiple columns, or a single range, but not > multiple ranges.) > For example, I used to have > [,],[,] > and I could get superColumnName1, superColumnName2 > Now I have > [0..0],[0..0] > and I need to get superColumnName1..superColumnName1+, > superColumnName2..superColumnName2+ > to get the same functionality > I would like the clients to support this functionality, e.g. Hector to have > .setRages parallel to .setColumnNames > and for CQL to support a syntax like > SELECT [FIRST N] [REVERSED] name1..nameN1, name2..nameN2... FROM ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2710) Get multiple column ranges
[ https://issues.apache.org/jira/browse/CASSANDRA-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2710: -- Reviewer: jbellis (was: xedin) Fix Version/s: (was: 1.2) We should probably leave the old cql package (cql2) alone. We can express this in CQL3 but allowing the parser to recognize the construction of such a query seems fragile to me: {{SELECT * FROM Standard1 WHERE key = 100 AND ((column1 >= 0 AND column1 <= 2) OR (column1 >= 3))}} Leaning towards the position that we don't want to actually expose this directly, we just want to support it internally (so we can use it to provide compatibility w/ the old supercolumn APIs). > Get multiple column ranges > -- > > Key: CASSANDRA-2710 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2710 > Project: Cassandra > Issue Type: Sub-task > Components: API, Core >Reporter: David Boxenhorn >Assignee: Vijay > Labels: compositeColumns, cql > Attachments: 0001-2710-multiple-column-ranges-cql.patch, > 0001-2710-multiple-column-ranges-thrift.patch > > > I have replaced all my super column families with regular column families > using composite columns. I have easily been able to support all previous > functionality (I don't need range delete) except for one thing: getting > multiple super columns with a single access. For this, I would need to get > multiple ranges. (I can get multiple columns, or a single range, but not > multiple ranges.) > For example, I used to have > [,],[,] > and I could get superColumnName1, superColumnName2 > Now I have > [0..0],[0..0] > and I need to get superColumnName1..superColumnName1+, > superColumnName2..superColumnName2+ > to get the same functionality > I would like the clients to support this functionality, e.g. Hector to have > .setRages parallel to .setColumnNames > and for CQL to support a syntax like > SELECT [FIRST N] [REVERSED] name1..nameN1, name2..nameN2... FROM ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3974) Per-CF TTL
[ https://issues.apache.org/jira/browse/CASSANDRA-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434497#comment-13434497 ] Jonathan Ellis commented on CASSANDRA-3974: --- I think Sylvain is right that that makes more sense... sorry about the wild goose chase! > Per-CF TTL > -- > > Key: CASSANDRA-3974 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3974 > Project: Cassandra > Issue Type: New Feature >Affects Versions: 1.2 >Reporter: Jonathan Ellis >Assignee: Kirk True >Priority: Minor > Fix For: 1.2 > > Attachments: trunk-3974.txt, trunk-3974v2.txt, trunk-3974v3.txt, > trunk-3974v4.txt > > > Per-CF TTL would allow compaction optimizations ("drop an entire sstable's > worth of expired data") that we can't do with per-column. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Git Push Summary
Updated Tags: refs/tags/1.1.4-tentative [deleted] 7cefd7411
[jira] [Commented] (CASSANDRA-4453) Better support of collections in the binary protocol
[ https://issues.apache.org/jira/browse/CASSANDRA-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434498#comment-13434498 ] Jonathan Ellis commented on CASSANDRA-4453: --- Sounds reasonable to me. Waiting for Paul's review of the rest. > Better support of collections in the binary protocol > > > Key: CASSANDRA-4453 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4453 > Project: Cassandra > Issue Type: Improvement >Affects Versions: 1.2 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 1.2 > > Attachments: 0001-Adds-generics-to-collection-types.txt, > 0002-Support-collections-natively-in-the-binary-protocol.txt, > 0003-Use-binary-format-for-thrift.txt > > > Currently, collections results are serialized to json string and send that > way. This doesn't feel right at all for the binary protocol and we should use > a simple binary serialization of the collection instead. > For the thrift protocol, we might want to keep the json serialization or use > the same binary serialization. I don't really have much opinion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[1/2] git commit: add #4494 to CHANGES
Updated Branches: refs/heads/cassandra-1.0 8b6ce324b -> 197511f0b refs/heads/cassandra-1.1 9dc560812 -> a201422b2 add #4494 to CHANGES Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a201422b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a201422b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a201422b Branch: refs/heads/cassandra-1.1 Commit: a201422b24e1fc4457d09b082f4a04f6b89cf49b Parents: 9dc5608 Author: Jonathan Ellis Authored: Mon Aug 13 18:06:32 2012 -0500 Committer: Jonathan Ellis Committed: Mon Aug 13 18:06:32 2012 -0500 -- CHANGES.txt |2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a201422b/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index aa40626..dcc8a4e 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,5 +1,7 @@ 1.1.4 * fix offline scrub to catch >= out of order rows (CASSANDRA-4411) + * fix cassandra-env.sh on RHEL and other non-dash-based systems + (CASSANDRA-4494) 1.1.3
[2/2] git commit: fix setting key length for old-style mapred api patch by jbellis; reviewed by brandonwilliams for CASSANDRA-4534
fix setting key length for old-style mapred api patch by jbellis; reviewed by brandonwilliams for CASSANDRA-4534 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/197511f0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/197511f0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/197511f0 Branch: refs/heads/cassandra-1.0 Commit: 197511f0b1b10b67fdd4942fe6f9c88adcad6f57 Parents: 8b6ce32 Author: Jonathan Ellis Authored: Mon Aug 13 14:22:36 2012 -0500 Committer: Jonathan Ellis Committed: Mon Aug 13 14:22:36 2012 -0500 -- CHANGES.txt|1 + .../cassandra/hadoop/ColumnFamilyRecordReader.java |2 +- 2 files changed, 2 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/197511f0/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 222081b..d025bef 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 1.0.12 + * (Hadoop) fix setting key length for old-style mapred api (CASSANDRA-4534) * (Hadoop) fix iterating through a resultset consisting entirely of tombstoned rows (CASSANDRA-4466) http://git-wip-us.apache.org/repos/asf/cassandra/blob/197511f0/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index 20d6068..aac61ad 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -416,7 +416,7 @@ public class ColumnFamilyRecordReader extends RecordReader
Git Push Summary
Updated Tags: refs/tags/1.1.4-tentative [created] 22a97da66
Git Push Summary
Updated Tags: refs/tags/1.1.4-tentative [deleted] 22a97da66
[3/14] git commit: merge from 1.0
merge from 1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17c1787e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17c1787e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17c1787e Branch: refs/heads/trunk Commit: 17c1787e94f2bcc9ed81851139e3a0e1bd7b28cb Parents: a201422 197511f Author: Jonathan Ellis Authored: Tue Aug 14 16:01:17 2012 -0500 Committer: Jonathan Ellis Committed: Tue Aug 14 16:01:17 2012 -0500 -- CHANGES.txt|4 build.xml |2 +- .../cassandra/hadoop/ColumnFamilyRecordReader.java |2 +- 3 files changed, 6 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/17c1787e/CHANGES.txt -- diff --cc CHANGES.txt index dcc8a4e,d025bef..dfb77d6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,157 -1,12 +1,161 @@@ -1.0.12 +1.1.4 + * fix offline scrub to catch >= out of order rows (CASSANDRA-4411) + * fix cassandra-env.sh on RHEL and other non-dash-based systems + (CASSANDRA-4494) ++Merged from 1.0: + * (Hadoop) fix setting key length for old-style mapred api (CASSANDRA-4534) - * (Hadoop) fix iterating through a resultset consisting entirely - of tombstoned rows (CASSANDRA-4466) -1.0.11 +1.1.3 + * munmap commitlog segments before rename (CASSANDRA-4337) + * (JMX) rename getRangeKeySample to sampleKeyRange to avoid returning + multi-MB results as an attribute (CASSANDRA-4452) + * flush based on data size, not throughput; overwritten columns no + longer artificially inflate liveRatio (CASSANDRA-4399) + * update default commitlog segment size to 32MB and total commitlog + size to 32/1024 MB for 32/64 bit JVMs, respectively (CASSANDRA-4422) + * avoid using global partitioner to estimate ranges in index sstables + (CASSANDRA-4403) + * restore pre-CASSANDRA-3862 approach to removing expired tombstones + from row cache during compaction (CASSANDRA-4364) + * (stress) support for CQL prepared statements (CASSANDRA-3633) + * Correctly catch exception when Snappy cannot be loaded (CASSANDRA-4400) + * (cql3) Support ORDER BY when IN condition is given in WHERE clause (CASSANDRA-4327) + * (cql3) delete "component_index" column on DROP TABLE call (CASSANDRA-4420) + * change nanoTime() to currentTimeInMillis() in schema related code (CASSANDRA-4432) + * add a token generation tool (CASSANDRA-3709) + * Fix LCS bug with sstable containing only 1 row (CASSANDRA-4411) + * fix "Can't Modify Index Name" problem on CF update (CASSANDRA-4439) + * Fix assertion error in getOverlappingSSTables during repair (CASSANDRA-4456) + * fix nodetool's setcompactionthreshold command (CASSANDRA-4455) + * Ensure compacted files are never used, to avoid counter overcount (CASSANDRA-4436) +Merged from 1.0: + * Push the validation of secondary index values to the SecondaryIndexManager (CASSANDRA-4240) + * (Hadoop) fix iterating through a resultset consisting entirely + of tombstoned rows (CASSANDRA-4466) + * allow dropping columns shadowed by not-yet-expired supercolumn or row +tombstones in PrecompactedRow (CASSANDRA-4396) + + +1.1.2 + * Fix cleanup not deleting index entries (CASSANDRA-4379) + * Use correct partitioner when saving + loading caches (CASSANDRA-4331) + * Check schema before trying to export sstable (CASSANDRA-2760) + * Raise a meaningful exception instead of NPE when PFS encounters + an unconfigured node + no default (CASSANDRA-4349) + * fix bug in sstable blacklisting with LCS (CASSANDRA-4343) + * LCS no longer promotes tiny sstables out of L0 (CASSANDRA-4341) + * skip tombstones during hint replay (CASSANDRA-4320) + * fix NPE in compactionstats (CASSANDRA-4318) + * enforce 1m min keycache for auto (CASSANDRA-4306) + * Have DeletedColumn.isMFD always return true (CASSANDRA-4307) + * (cql3) exeption message for ORDER BY constraints said primary filter can be +an IN clause, which is misleading (CASSANDRA-4319) + * (cql3) Reject (not yet supported) creation of 2ndardy indexes on tables with + composite primary keys (CASSANDRA-4328) + * Set JVM stack size to 160k for java 7 (CASSANDRA-4275) + * cqlsh: add COPY command to load data from CSV flat files (CASSANDRA-4012) + * CFMetaData.fromThrift to throw ConfigurationException upon error (CASSANDRA-4353) + * Use CF comparator to sort indexed columns in SecondaryIndexManager + (CASSANDRA-4365) + * add strategy_options to the KSMetaData.toString() output (CASSANDRA-4248) + * (cql3) fix range queries containing unqueried results (CASSANDRA-4372) + * (cql3) allow updating column_alias types (CASSANDRA-4041) + * (cql3) Fix deletion bug (CASSANDRA
[2/14] git commit: merge from 1.0
merge from 1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17c1787e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17c1787e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17c1787e Branch: refs/heads/cassandra-1.1 Commit: 17c1787e94f2bcc9ed81851139e3a0e1bd7b28cb Parents: a201422 197511f Author: Jonathan Ellis Authored: Tue Aug 14 16:01:17 2012 -0500 Committer: Jonathan Ellis Committed: Tue Aug 14 16:01:17 2012 -0500 -- CHANGES.txt|4 build.xml |2 +- .../cassandra/hadoop/ColumnFamilyRecordReader.java |2 +- 3 files changed, 6 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/17c1787e/CHANGES.txt -- diff --cc CHANGES.txt index dcc8a4e,d025bef..dfb77d6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,157 -1,12 +1,161 @@@ -1.0.12 +1.1.4 + * fix offline scrub to catch >= out of order rows (CASSANDRA-4411) + * fix cassandra-env.sh on RHEL and other non-dash-based systems + (CASSANDRA-4494) ++Merged from 1.0: + * (Hadoop) fix setting key length for old-style mapred api (CASSANDRA-4534) - * (Hadoop) fix iterating through a resultset consisting entirely - of tombstoned rows (CASSANDRA-4466) -1.0.11 +1.1.3 + * munmap commitlog segments before rename (CASSANDRA-4337) + * (JMX) rename getRangeKeySample to sampleKeyRange to avoid returning + multi-MB results as an attribute (CASSANDRA-4452) + * flush based on data size, not throughput; overwritten columns no + longer artificially inflate liveRatio (CASSANDRA-4399) + * update default commitlog segment size to 32MB and total commitlog + size to 32/1024 MB for 32/64 bit JVMs, respectively (CASSANDRA-4422) + * avoid using global partitioner to estimate ranges in index sstables + (CASSANDRA-4403) + * restore pre-CASSANDRA-3862 approach to removing expired tombstones + from row cache during compaction (CASSANDRA-4364) + * (stress) support for CQL prepared statements (CASSANDRA-3633) + * Correctly catch exception when Snappy cannot be loaded (CASSANDRA-4400) + * (cql3) Support ORDER BY when IN condition is given in WHERE clause (CASSANDRA-4327) + * (cql3) delete "component_index" column on DROP TABLE call (CASSANDRA-4420) + * change nanoTime() to currentTimeInMillis() in schema related code (CASSANDRA-4432) + * add a token generation tool (CASSANDRA-3709) + * Fix LCS bug with sstable containing only 1 row (CASSANDRA-4411) + * fix "Can't Modify Index Name" problem on CF update (CASSANDRA-4439) + * Fix assertion error in getOverlappingSSTables during repair (CASSANDRA-4456) + * fix nodetool's setcompactionthreshold command (CASSANDRA-4455) + * Ensure compacted files are never used, to avoid counter overcount (CASSANDRA-4436) +Merged from 1.0: + * Push the validation of secondary index values to the SecondaryIndexManager (CASSANDRA-4240) + * (Hadoop) fix iterating through a resultset consisting entirely + of tombstoned rows (CASSANDRA-4466) + * allow dropping columns shadowed by not-yet-expired supercolumn or row +tombstones in PrecompactedRow (CASSANDRA-4396) + + +1.1.2 + * Fix cleanup not deleting index entries (CASSANDRA-4379) + * Use correct partitioner when saving + loading caches (CASSANDRA-4331) + * Check schema before trying to export sstable (CASSANDRA-2760) + * Raise a meaningful exception instead of NPE when PFS encounters + an unconfigured node + no default (CASSANDRA-4349) + * fix bug in sstable blacklisting with LCS (CASSANDRA-4343) + * LCS no longer promotes tiny sstables out of L0 (CASSANDRA-4341) + * skip tombstones during hint replay (CASSANDRA-4320) + * fix NPE in compactionstats (CASSANDRA-4318) + * enforce 1m min keycache for auto (CASSANDRA-4306) + * Have DeletedColumn.isMFD always return true (CASSANDRA-4307) + * (cql3) exeption message for ORDER BY constraints said primary filter can be +an IN clause, which is misleading (CASSANDRA-4319) + * (cql3) Reject (not yet supported) creation of 2ndardy indexes on tables with + composite primary keys (CASSANDRA-4328) + * Set JVM stack size to 160k for java 7 (CASSANDRA-4275) + * cqlsh: add COPY command to load data from CSV flat files (CASSANDRA-4012) + * CFMetaData.fromThrift to throw ConfigurationException upon error (CASSANDRA-4353) + * Use CF comparator to sort indexed columns in SecondaryIndexManager + (CASSANDRA-4365) + * add strategy_options to the KSMetaData.toString() output (CASSANDRA-4248) + * (cql3) fix range queries containing unqueried results (CASSANDRA-4372) + * (cql3) allow updating column_alias types (CASSANDRA-4041) + * (cql3) Fix deletion bug (C
[1/14] git commit: merge from 1.1
Updated Branches: refs/heads/cassandra-1.1 a201422b2 -> 17c1787e9 refs/heads/trunk e243db435 -> 1a44fa7ea merge from 1.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a44fa7e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a44fa7e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a44fa7e Branch: refs/heads/trunk Commit: 1a44fa7ea20a0be92cc0cb0f87c270a7dd2f6334 Parents: e243db4 17c1787 Author: Jonathan Ellis Authored: Tue Aug 14 16:01:41 2012 -0500 Committer: Jonathan Ellis Committed: Tue Aug 14 16:01:41 2012 -0500 -- CHANGES.txt|6 ++ build.xml |2 +- debian/changelog |6 ++ .../cassandra/hadoop/ColumnFamilyRecordReader.java |2 +- 4 files changed, 14 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a44fa7e/CHANGES.txt -- diff --cc CHANGES.txt index f823937,dfb77d6..5d878bc --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,46 -1,12 +1,50 @@@ +1.2-dev + * clean up ioexceptions (CASSANDRA-2116) + * Introduce new json format with row level deletion (CASSANDRA-4054) + * remove redundant "name" column from schema_keyspaces (CASSANDRA-4433) + * improve "nodetool ring" handling of multi-dc clusters (CASSANDRA-3047) + * update NTS calculateNaturalEndpoints to be O(N log N) (CASSANDRA-3881) + * add UseCondCardMark XX jvm settings on jdk 1.7 (CASSANDRA-4366) + * split up rpc timeout by operation type (CASSANDRA-2819) + * rewrite key cache save/load to use only sequential i/o (CASSANDRA-3762) + * update MS protocol with a version handshake + broadcast address id + (CASSANDRA-4311) + * multithreaded hint replay (CASSANDRA-4189) + * add inter-node message compression (CASSANDRA-3127) + * remove COPP (CASSANDRA-2479) + * Track tombstone expiration and compact when tombstone content is + higher than a configurable threshold, default 20% (CASSANDRA-3442, 4234) + * update MurmurHash to version 3 (CASSANDRA-2975) + * (CLI) track elapsed time for `delete' operation (CASSANDRA-4060) + * (CLI) jline version is bumped to 1.0 to properly support + 'delete' key function (CASSANDRA-4132) + * Save IndexSummary into new SSTable 'Summary' component (CASSANDRA-2392, 4289) + * Add support for range tombstones (CASSANDRA-3708) + * Improve MessagingService efficiency (CASSANDRA-3617) + * Avoid ID conflicts from concurrent schema changes (CASSANDRA-3794) + * Set thrift HSHA server thread limit to unlimited by default (CASSANDRA-4277) + * Avoids double serialization of CF id in RowMutation messages + (CASSANDRA-4293) + * stream compressed sstables directly with java nio (CASSANDRA-4297) + * Support multiple ranges in SliceQueryFilter (CASSANDRA-3885) + * Add column metadata to system column families (CASSANDRA-4018) + * (cql3) Always use composite types by default (CASSANDRA-4329) + * (cql3) Add support for set, map and list (CASSANDRA-3647) + * Validate date type correctly (CASSANDRA-4441) + * (cql3) Allow definitions with only a PK (CASSANDRA-4361) + * (cql3) Add support for row key composites (CASSANDRA-4179) + * improve DynamicEndpointSnitch by using reservoir sampling (CASSANDRA-4038) + * (cql3) Add support for 2ndary indexes (CASSANDRA-3680) + + 1.1.4 * fix offline scrub to catch >= out of order rows (CASSANDRA-4411) + * fix cassandra-env.sh on RHEL and other non-dash-based systems +(CASSANDRA-4494) + Merged from 1.0: + * (Hadoop) fix setting key length for old-style mapred api (CASSANDRA-4534) -1.1.3 * munmap commitlog segments before rename (CASSANDRA-4337) * (JMX) rename getRangeKeySample to sampleKeyRange to avoid returning multi-MB results as an attribute (CASSANDRA-4452) http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a44fa7e/build.xml -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a44fa7e/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java --
[4/14] git commit: add #4494 to CHANGES
add #4494 to CHANGES Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a201422b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a201422b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a201422b Branch: refs/heads/trunk Commit: a201422b24e1fc4457d09b082f4a04f6b89cf49b Parents: 9dc5608 Author: Jonathan Ellis Authored: Mon Aug 13 18:06:32 2012 -0500 Committer: Jonathan Ellis Committed: Mon Aug 13 18:06:32 2012 -0500 -- CHANGES.txt |2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a201422b/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index aa40626..dcc8a4e 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,5 +1,7 @@ 1.1.4 * fix offline scrub to catch >= out of order rows (CASSANDRA-4411) + * fix cassandra-env.sh on RHEL and other non-dash-based systems + (CASSANDRA-4494) 1.1.3
[8/14] git commit: a startswith func that works for dash and bash
a startswith func that works for dash and bash Patch by eevans for CASSANDRA-4494 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/510689e3 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/510689e3 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/510689e3 Branch: refs/heads/trunk Commit: 510689e3a44addcabc4e505e0ebed88edcee244b Parents: 9fb63a2 Author: Eric Evans Authored: Fri Aug 10 10:33:12 2012 -0500 Committer: Eric Evans Committed: Mon Aug 13 09:28:48 2012 -0500 -- conf/cassandra-env.sh |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/510689e3/conf/cassandra-env.sh -- diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh index 2928018..6ae28a0 100644 --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@ -177,7 +177,7 @@ if [ "x$CASSANDRA_HEAPDUMP_DIR" != "x" ]; then fi -startswith () [ "${1#$2}" != "$1" ] +startswith() { [ "${1#$2}" != "$1" ]; } if [ "`uname`" = "Linux" ] ; then # reduce the per-thread stack size to minimize the impact of Thrift
[5/14] git commit: updated for 1.1.4 release
updated for 1.1.4 release Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9dc56081 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9dc56081 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9dc56081 Branch: refs/heads/trunk Commit: 9dc560812ea23a4462411a30be2263b803e39fd9 Parents: 510689e Author: Eric Evans Authored: Mon Aug 13 15:16:02 2012 -0500 Committer: Eric Evans Committed: Mon Aug 13 15:16:02 2012 -0500 -- build.xml|2 +- debian/changelog |6 ++ 2 files changed, 7 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9dc56081/build.xml -- diff --git a/build.xml b/build.xml index e23483c..10dbecc 100644 --- a/build.xml +++ b/build.xml @@ -25,7 +25,7 @@ - + http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/> http://git-wip-us.apache.org/repos/asf/cassandra/blob/9dc56081/debian/changelog -- diff --git a/debian/changelog b/debian/changelog index 946498c..9c6bfdc 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +cassandra (1.1.4) unstable; urgency=low + + * New release + + -- Eric Evans Mon, 13 Aug 2012 15:13:20 -0500 + cassandra (1.1.3) unstable; urgency=low * New release
[7/14] git commit: fix setting key length for old-style mapred api patch by jbellis; reviewed by brandonwilliams for CASSANDRA-4534
fix setting key length for old-style mapred api patch by jbellis; reviewed by brandonwilliams for CASSANDRA-4534 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/197511f0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/197511f0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/197511f0 Branch: refs/heads/cassandra-1.1 Commit: 197511f0b1b10b67fdd4942fe6f9c88adcad6f57 Parents: 8b6ce32 Author: Jonathan Ellis Authored: Mon Aug 13 14:22:36 2012 -0500 Committer: Jonathan Ellis Committed: Mon Aug 13 14:22:36 2012 -0500 -- CHANGES.txt|1 + .../cassandra/hadoop/ColumnFamilyRecordReader.java |2 +- 2 files changed, 2 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/197511f0/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 222081b..d025bef 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 1.0.12 + * (Hadoop) fix setting key length for old-style mapred api (CASSANDRA-4534) * (Hadoop) fix iterating through a resultset consisting entirely of tombstoned rows (CASSANDRA-4466) http://git-wip-us.apache.org/repos/asf/cassandra/blob/197511f0/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index 20d6068..aac61ad 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -416,7 +416,7 @@ public class ColumnFamilyRecordReader extends RecordReader
[9/14] git commit: fix CFRR iterating through resultset consisting entirely of tombstones patch by jbellis; tested by Niel Drummand and reviewed by Brandon Williams for CASSANDRA-4466
fix CFRR iterating through resultset consisting entirely of tombstones patch by jbellis; tested by Niel Drummand and reviewed by Brandon Williams for CASSANDRA-4466 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8b6ce324 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8b6ce324 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8b6ce324 Branch: refs/heads/cassandra-1.1 Commit: 8b6ce324bf35210196e0f0ec0665ba87b0f3991f Parents: 813553b Author: Jonathan Ellis Authored: Tue Jul 31 10:41:26 2012 -0500 Committer: Jonathan Ellis Committed: Tue Jul 31 10:47:14 2012 -0500 -- CHANGES.txt|5 + .../cassandra/hadoop/ColumnFamilyRecordReader.java |9 ++--- 2 files changed, 11 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b6ce324/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index c708aea..222081b 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,3 +1,8 @@ +1.0.12 + * (Hadoop) fix iterating through a resultset consisting entirely + of tombstoned rows (CASSANDRA-4466) + + 1.0.11 * allow dropping columns shadowed by not-yet-expired supercolumn or row tombstones in PrecompactedRow (CASSANDRA-4396) http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b6ce324/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index 5d0ac72..20d6068 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -307,18 +307,21 @@ public class ColumnFamilyRecordReader extends RecordReader it = rows.iterator(); -while (it.hasNext()) +KeySlice ks; +do { -KeySlice ks = it.next(); +ks = it.next(); if (ks.getColumnsSize() == 0) { it.remove(); } -} +} while (it.hasNext()); // all ghosts, spooky if (rows.isEmpty()) { +// maybeInit assumes it can get the start-with key from the rows collection, so add back the last +rows.add(ks); maybeInit(); return; }
[6/14] git commit: fix setting key length for old-style mapred api patch by jbellis; reviewed by brandonwilliams for CASSANDRA-4534
fix setting key length for old-style mapred api patch by jbellis; reviewed by brandonwilliams for CASSANDRA-4534 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/197511f0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/197511f0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/197511f0 Branch: refs/heads/trunk Commit: 197511f0b1b10b67fdd4942fe6f9c88adcad6f57 Parents: 8b6ce32 Author: Jonathan Ellis Authored: Mon Aug 13 14:22:36 2012 -0500 Committer: Jonathan Ellis Committed: Mon Aug 13 14:22:36 2012 -0500 -- CHANGES.txt|1 + .../cassandra/hadoop/ColumnFamilyRecordReader.java |2 +- 2 files changed, 2 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/197511f0/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 222081b..d025bef 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 1.0.12 + * (Hadoop) fix setting key length for old-style mapred api (CASSANDRA-4534) * (Hadoop) fix iterating through a resultset consisting entirely of tombstoned rows (CASSANDRA-4466) http://git-wip-us.apache.org/repos/asf/cassandra/blob/197511f0/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index 20d6068..aac61ad 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -416,7 +416,7 @@ public class ColumnFamilyRecordReader extends RecordReader
[12/14] git commit: formatting
formatting Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/813553be Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/813553be Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/813553be Branch: refs/heads/trunk Commit: 813553bea9d1145cca671d61737bc9673315b8f0 Parents: 4f0237a Author: Jonathan Ellis Authored: Tue Jul 31 10:44:13 2012 -0500 Committer: Jonathan Ellis Committed: Tue Jul 31 10:44:13 2012 -0500 -- .../cassandra/hadoop/ColumnFamilyRecordReader.java | 12 +--- 1 files changed, 5 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/813553be/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index b84eb85..5d0ac72 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -302,22 +302,20 @@ public class ColumnFamilyRecordReader extends RecordReader it = rows.iterator(); - -while(it.hasNext()) +while (it.hasNext()) { KeySlice ks = it.next(); - if (ks.getColumnsSize() == 0) { - it.remove(); +it.remove(); } } - + // all ghosts, spooky if (rows.isEmpty()) { @@ -325,7 +323,7 @@ public class ColumnFamilyRecordReader extends RecordReader
[13/14] git commit: Update versions, news and changes for 1.0.11 release
Update versions, news and changes for 1.0.11 release Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4f0237ac Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4f0237ac Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4f0237ac Branch: refs/heads/trunk Commit: 4f0237acd5ee8097f90732ac416622588e4d7552 Parents: 7ff8e3c Author: Sylvain Lebresne Authored: Fri Jul 27 17:14:32 2012 +0200 Committer: Sylvain Lebresne Committed: Fri Jul 27 17:14:32 2012 +0200 -- CHANGES.txt |4 NEWS.txt |8 build.xml|2 +- debian/changelog |6 ++ 4 files changed, 19 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index b3fa1a8..c708aea 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -13,6 +13,10 @@ * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419) * Push the validation of secondary index values to the SecondaryIndexManager (CASSANDRA-4240) * Don't purge columns during upgradesstables (CASSANDRA-4462) + * Make cqlsh work with piping (CASSANDRA-4113) + * Validate arguments for nodetool decommission (CASSANDRA-4061) + * Report thrift status in nodetool info (CASSANDRA-4010) + 1.0.10 * fix maxTimestamp to include row tombstones (CASSANDRA-4116) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index 42bea7c..1654b44 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -8,6 +8,14 @@ upgrade, just in case you need to roll back to the previous version. (Cassandra version X + 1 will always be able to read data files created by version X, but the inverse is not necessarily the case.) +1.0.11 +== + +Upgrading +- +- Nothing specific to 1.0.10 + + 1.0.10 == http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/build.xml -- diff --git a/build.xml b/build.xml index 261691d..705db5e 100644 --- a/build.xml +++ b/build.xml @@ -25,7 +25,7 @@ - + http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/> http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/debian/changelog -- diff --git a/debian/changelog b/debian/changelog index c34d7a0..7631284 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +cassandra (1.0.11) unstable; urgency=low + + * New release + + -- Sylvain Lebresne Fri, 27 Jul 2012 17:04:14 +0200 + cassandra (1.0.10) unstable; urgency=low * New release
[14/14] git commit: Update versions, news and changes for 1.0.11 release
Update versions, news and changes for 1.0.11 release Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4f0237ac Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4f0237ac Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4f0237ac Branch: refs/heads/cassandra-1.1 Commit: 4f0237acd5ee8097f90732ac416622588e4d7552 Parents: 7ff8e3c Author: Sylvain Lebresne Authored: Fri Jul 27 17:14:32 2012 +0200 Committer: Sylvain Lebresne Committed: Fri Jul 27 17:14:32 2012 +0200 -- CHANGES.txt |4 NEWS.txt |8 build.xml|2 +- debian/changelog |6 ++ 4 files changed, 19 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index b3fa1a8..c708aea 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -13,6 +13,10 @@ * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419) * Push the validation of secondary index values to the SecondaryIndexManager (CASSANDRA-4240) * Don't purge columns during upgradesstables (CASSANDRA-4462) + * Make cqlsh work with piping (CASSANDRA-4113) + * Validate arguments for nodetool decommission (CASSANDRA-4061) + * Report thrift status in nodetool info (CASSANDRA-4010) + 1.0.10 * fix maxTimestamp to include row tombstones (CASSANDRA-4116) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index 42bea7c..1654b44 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -8,6 +8,14 @@ upgrade, just in case you need to roll back to the previous version. (Cassandra version X + 1 will always be able to read data files created by version X, but the inverse is not necessarily the case.) +1.0.11 +== + +Upgrading +- +- Nothing specific to 1.0.10 + + 1.0.10 == http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/build.xml -- diff --git a/build.xml b/build.xml index 261691d..705db5e 100644 --- a/build.xml +++ b/build.xml @@ -25,7 +25,7 @@ - + http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/> http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f0237ac/debian/changelog -- diff --git a/debian/changelog b/debian/changelog index c34d7a0..7631284 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +cassandra (1.0.11) unstable; urgency=low + + * New release + + -- Sylvain Lebresne Fri, 27 Jul 2012 17:04:14 +0200 + cassandra (1.0.10) unstable; urgency=low * New release
[11/14] git commit: formatting
formatting Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/813553be Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/813553be Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/813553be Branch: refs/heads/cassandra-1.1 Commit: 813553bea9d1145cca671d61737bc9673315b8f0 Parents: 4f0237a Author: Jonathan Ellis Authored: Tue Jul 31 10:44:13 2012 -0500 Committer: Jonathan Ellis Committed: Tue Jul 31 10:44:13 2012 -0500 -- .../cassandra/hadoop/ColumnFamilyRecordReader.java | 12 +--- 1 files changed, 5 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/813553be/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index b84eb85..5d0ac72 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -302,22 +302,20 @@ public class ColumnFamilyRecordReader extends RecordReader it = rows.iterator(); - -while(it.hasNext()) +while (it.hasNext()) { KeySlice ks = it.next(); - if (ks.getColumnsSize() == 0) { - it.remove(); +it.remove(); } } - + // all ghosts, spooky if (rows.isEmpty()) { @@ -325,7 +323,7 @@ public class ColumnFamilyRecordReader extends RecordReader
[10/14] git commit: fix CFRR iterating through resultset consisting entirely of tombstones patch by jbellis; tested by Niel Drummand and reviewed by Brandon Williams for CASSANDRA-4466
fix CFRR iterating through resultset consisting entirely of tombstones patch by jbellis; tested by Niel Drummand and reviewed by Brandon Williams for CASSANDRA-4466 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8b6ce324 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8b6ce324 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8b6ce324 Branch: refs/heads/trunk Commit: 8b6ce324bf35210196e0f0ec0665ba87b0f3991f Parents: 813553b Author: Jonathan Ellis Authored: Tue Jul 31 10:41:26 2012 -0500 Committer: Jonathan Ellis Committed: Tue Jul 31 10:47:14 2012 -0500 -- CHANGES.txt|5 + .../cassandra/hadoop/ColumnFamilyRecordReader.java |9 ++--- 2 files changed, 11 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b6ce324/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index c708aea..222081b 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,3 +1,8 @@ +1.0.12 + * (Hadoop) fix iterating through a resultset consisting entirely + of tombstoned rows (CASSANDRA-4466) + + 1.0.11 * allow dropping columns shadowed by not-yet-expired supercolumn or row tombstones in PrecompactedRow (CASSANDRA-4396) http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b6ce324/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java -- diff --git a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java index 5d0ac72..20d6068 100644 --- a/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java +++ b/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java @@ -307,18 +307,21 @@ public class ColumnFamilyRecordReader extends RecordReader it = rows.iterator(); -while (it.hasNext()) +KeySlice ks; +do { -KeySlice ks = it.next(); +ks = it.next(); if (ks.getColumnsSize() == 0) { it.remove(); } -} +} while (it.hasNext()); // all ghosts, spooky if (rows.isEmpty()) { +// maybeInit assumes it can get the start-with key from the rows collection, so add back the last +rows.add(ks); maybeInit(); return; }
git commit: remove spurious character
Updated Branches: refs/heads/cassandra-1.1 17c1787e9 -> 8b1336f00 remove spurious character Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8b1336f0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8b1336f0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8b1336f0 Branch: refs/heads/cassandra-1.1 Commit: 8b1336f0044b31dad0b1b27f76620f952a06662e Parents: 17c1787 Author: Eric Evans Authored: Tue Aug 14 16:06:09 2012 -0500 Committer: Eric Evans Committed: Tue Aug 14 16:06:09 2012 -0500 -- build.xml |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b1336f0/build.xml -- diff --git a/build.xml b/build.xml index 6785aed..10dbecc 100644 --- a/build.xml +++ b/build.xml @@ -1,4 +1,4 @@ -v +
Git Push Summary
Updated Tags: refs/tags/1.1.4-tentative [created] 94e46ff95
[jira] [Commented] (CASSANDRA-4453) Better support of collections in the binary protocol
[ https://issues.apache.org/jira/browse/CASSANDRA-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434538#comment-13434538 ] paul cannon commented on CASSANDRA-4453: Only a tiny nit here: might be worth removing the `o.a.c.cql3.ResultSet.Metadata.dataTypeCodec` static instance declaration, since it's unused now and could be a little confusing with `DataType.codec` serving the same purpose. Also- this is only somewhat related, but I came across it while testing and reviewing, and it might be a good place to shoehorn in an extra tweaks: The native_protocol.spec doc doesn't ever explicitly mention endianness. It's big-endian, network byte order, as probably everyone would expect, but it's still nice to make that clear. But yeah, +1. > Better support of collections in the binary protocol > > > Key: CASSANDRA-4453 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4453 > Project: Cassandra > Issue Type: Improvement >Affects Versions: 1.2 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 1.2 > > Attachments: 0001-Adds-generics-to-collection-types.txt, > 0002-Support-collections-natively-in-the-binary-protocol.txt, > 0003-Use-binary-format-for-thrift.txt > > > Currently, collections results are serialized to json string and send that > way. This doesn't feel right at all for the binary protocol and we should use > a simple binary serialization of the collection instead. > For the thrift protocol, we might want to keep the json serialization or use > the same binary serialization. I don't really have much opinion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[2/3] git commit: Support collections natively in the binary protocol patch by slebresne; reviewed by Paul Cannon for CASSANDRA-4453
Support collections natively in the binary protocol patch by slebresne; reviewed by Paul Cannon for CASSANDRA-4453 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ba231f4e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ba231f4e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ba231f4e Branch: refs/heads/trunk Commit: ba231f4eb53f01ceb297f85d52592600f14a9bbb Parents: 5e5fbc6 Author: Jonathan Ellis Authored: Tue Aug 14 16:50:39 2012 -0500 Committer: Jonathan Ellis Committed: Tue Aug 14 16:51:08 2012 -0500 -- doc/native_protocol.spec |8 ++ src/java/org/apache/cassandra/cql3/ResultSet.java |5 +- .../org/apache/cassandra/transport/DataType.java | 78 ++- .../apache/cassandra/transport/OptionCodec.java| 10 ++- 4 files changed, 91 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ba231f4e/doc/native_protocol.spec -- diff --git a/doc/native_protocol.spec b/doc/native_protocol.spec index ae3c1fd..0092f5d 100644 --- a/doc/native_protocol.spec +++ b/doc/native_protocol.spec @@ -49,6 +49,8 @@ Table of Contents . . + + The protocol is big-endian (network byte order). + Each frame contains a fixed size header (8 bytes) followed by a variable size body. The header is described in Section 2. The content of the body depends on the header opcode value (the body can in particular be empty for some @@ -355,6 +357,12 @@ Table of Contents 0x000DVarchar 0x000EVarint 0x000FTimeuuid +0x0020List: the value is an [option], representing the type +of the elements of the list. +0x0021Map: the value is two [option], representing the types of the + keys and values of the map +0x0022Set: the value is an [option], representing the type +of the elements of the set - is an [int] representing the number of rows present in this result. Those rows are serialized in the part. - is composed of ... where m is . http://git-wip-us.apache.org/repos/asf/cassandra/blob/ba231f4e/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index cb5e89f..152edb9 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -204,7 +204,6 @@ public class ResultSet public static class Metadata { -private static OptionCodec dataTypeCodec = new OptionCodec(DataType.class); public static final CBCodec codec = new Codec(); public final EnumSet flags; @@ -277,7 +276,7 @@ public class ResultSet String ksName = globalTablesSpec ? globalKsName : CBUtil.readString(body); String cfName = globalTablesSpec ? globalCfName : CBUtil.readString(body); ColumnIdentifier colName = new ColumnIdentifier(CBUtil.readString(body), true); -AbstractType type = DataType.toType(dataTypeCodec.decodeOne(body)); +AbstractType type = DataType.toType(DataType.codec.decodeOne(body)); names.add(new ColumnSpecification(ksName, cfName, colName, type)); } return new Metadata(flags, names); @@ -309,7 +308,7 @@ public class ResultSet builder.addString(name.cfName); } builder.addString(name.toString()); - builder.add(dataTypeCodec.encodeOne(DataType.fromType(name.type))); + builder.add(DataType.codec.encodeOne(DataType.fromType(name.type))); } return builder.build(); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/ba231f4e/src/java/org/apache/cassandra/transport/DataType.java -- diff --git a/src/java/org/apache/cassandra/transport/DataType.java b/src/java/org/apache/cassandra/transport/DataType.java index 5254945..9a8c2f0 100644 --- a/src/java/org/apache/cassandra/transport/DataType.java +++ b/src/java/org/apache/cassandra/transport/DataType.java @@ -17,8 +17,11 @@ */ package org.apache.cassandra.transport; +import java.util.ArrayList; +import java.util.Arrays; import java.util.HashMap; import java.util.Map; +import java.ut
[1/3] git commit: Use binary collection encoding for Thrift resultsets patch by slebresne; reviewed by Paul Cannon for CASSANDRA-4453
Updated Branches: refs/heads/trunk 1a44fa7ea -> 6ce498f5a Use binary collection encoding for Thrift resultsets patch by slebresne; reviewed by Paul Cannon for CASSANDRA-4453 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6ce498f5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6ce498f5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6ce498f5 Branch: refs/heads/trunk Commit: 6ce498f5ad5879fdc58dc0c92af6cc52002fd198 Parents: ba231f4 Author: Jonathan Ellis Authored: Tue Aug 14 16:52:02 2012 -0500 Committer: Jonathan Ellis Committed: Tue Aug 14 16:52:02 2012 -0500 -- src/java/org/apache/cassandra/cql3/ResultSet.java |7 ++- 1 files changed, 2 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6ce498f5/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index 152edb9..c6a93e6 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -112,7 +112,7 @@ public class ResultSet { ByteBuffer colName = ByteBufferUtil.bytes(name.toString()); schema.name_types.put(colName, UTF8); -schema.value_types.put(colName, TypeParser.getShortName(name.type)); +schema.value_types.put(colName, name.type.toString()); } List cqlRows = new ArrayList(rows.size()); @@ -122,10 +122,7 @@ public class ResultSet for (int i = 0; i < metadata.names.size(); i++) { Column col = new Column(ByteBufferUtil.bytes(metadata.names.get(i).toString())); -if (row.get(i) != null && metadata.names.get(i).type.isCollection()) - col.setValue(ByteBufferUtil.bytes(FBUtilities.json(metadata.names.get(i).type.compose(row.get(i); -else -col.setValue(row.get(i)); +col.setValue(row.get(i)); thriftCols.add(col); } // The key of CqlRow shoudn't be needed in CQL3
[3/3] git commit: generify collection types patch by slebresnse; reviewed by Paul Cannon for CASSANDRA-4453
generify collection types patch by slebresnse; reviewed by Paul Cannon for CASSANDRA-4453 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e5fbc68 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e5fbc68 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e5fbc68 Branch: refs/heads/trunk Commit: 5e5fbc6853468e47abe5d25817be1220e90c980f Parents: 1a44fa7 Author: Jonathan Ellis Authored: Tue Aug 14 16:48:19 2012 -0500 Committer: Jonathan Ellis Committed: Tue Aug 14 16:48:19 2012 -0500 -- src/java/org/apache/cassandra/cql3/ResultSet.java |6 +- .../cassandra/cql3/statements/SelectStatement.java |2 +- .../cassandra/db/marshal/CollectionType.java | 27 +++-- .../org/apache/cassandra/db/marshal/ListType.java | 61 +-- .../org/apache/cassandra/db/marshal/MapType.java | 81 --- .../org/apache/cassandra/db/marshal/SetType.java | 66 +--- 6 files changed, 185 insertions(+), 58 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e5fbc68/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index 568e3ee..cb5e89f 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -33,6 +33,7 @@ import org.apache.cassandra.thrift.CqlResult; import org.apache.cassandra.thrift.CqlResultType; import org.apache.cassandra.thrift.CqlRow; import org.apache.cassandra.utils.ByteBufferUtil; +import org.apache.cassandra.utils.FBUtilities; public class ResultSet { @@ -121,7 +122,10 @@ public class ResultSet for (int i = 0; i < metadata.names.size(); i++) { Column col = new Column(ByteBufferUtil.bytes(metadata.names.get(i).toString())); -col.setValue(row.get(i)); +if (row.get(i) != null && metadata.names.get(i).type.isCollection()) + col.setValue(ByteBufferUtil.bytes(FBUtilities.json(metadata.names.get(i).type.compose(row.get(i); +else +col.setValue(row.get(i)); thriftCols.add(col); } // The key of CqlRow shoudn't be needed in CQL3 http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e5fbc68/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java index 95a1fee..37dd205 100644 --- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java @@ -863,7 +863,7 @@ public class SelectStatement implements CQLStatement if (collection == null) cqlRows.addColumnValue(null); else - cqlRows.addColumnValue(((CollectionType)name.type).serializeForThrift(collection)); + cqlRows.addColumnValue(((CollectionType)name.type).serialize(collection)); break; } IColumn c = columns.getSimple(name.name.key); http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e5fbc68/src/java/org/apache/cassandra/db/marshal/CollectionType.java -- diff --git a/src/java/org/apache/cassandra/db/marshal/CollectionType.java b/src/java/org/apache/cassandra/db/marshal/CollectionType.java index 21e96d3..a19912b 100644 --- a/src/java/org/apache/cassandra/db/marshal/CollectionType.java +++ b/src/java/org/apache/cassandra/db/marshal/CollectionType.java @@ -30,7 +30,7 @@ import org.apache.cassandra.utils.Pair; * Please note that this comparator shouldn't be used "manually" (through thrift for instance). * */ -public abstract class CollectionType extends AbstractType +public abstract class CollectionType extends AbstractType { public enum Kind { @@ -49,7 +49,7 @@ public abstract class CollectionType extends AbstractType protected abstract void appendToStringBuilder(StringBuilder sb); -public abstract ByteBuffer serializeForThrift(List> columns); +public abstract ByteBuffer serialize(List> columns); @Override public String toString() @@ -64,16 +64,6 @@ public abstract class CollectionType extends AbstractType throw new UnsupportedOperationException("CollectionType should not be use directly as a comparator"); } -
[jira] [Commented] (CASSANDRA-4496) NPE on creating secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434566#comment-13434566 ] David Semeria commented on CASSANDRA-4496: -- Ok, false alarm. AbstractType changed underneath me thus breaking my cutsom comparator. Might I suggest that on the creation of a new column family (an infrequent event) core executes a suite of simple tests on the supplied comparator(s) just to make sure they perform to spec. This would make catching similar situations in the future much easier. Keep up the excellent work! > NPE on creating secondary index > --- > > Key: CASSANDRA-4496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4496 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.2, 1.1.3 > Environment: Ubuntu 12.4 > Linux 3.2.0-27-generic #43-Ubuntu SMP Fri Jul 6 14:25:57 UTC 2012 x86_64 > x86_64 x86_64 GNU/Linux > java version "1.7.0" > Java(TM) SE Runtime Environment (build 1.7.0-b147) > Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode) >Reporter: David Semeria >Assignee: Pavel Yaskevich > Attachments: hector-core-1.0-3.jar, jellyfish.jar, > JNameComparator.java, JSerializer.java, JValueComparator.java > > > The following code has been working fine up to and including 1.0.x > public static String createIndexedColumnFamily(String cf){ > > Cluster cluster = HectorConfig.cluster; > ComparatorType ctName = > ComparatorType.getByClassName(JNameComparator.class.getName()); > > try{ > cluster.dropColumnFamily(HectorConfig.dfltKeyspaceName, cf ); > } catch (Exception e){} > > List cdL = new ArrayList(); > BasicColumnDefinition cd; > > cd = new BasicColumnDefinition(); > cd.setName(ss.toByteBuffer("id")); > cd.setIndexName("id"); > cd.setIndexType(ColumnIndexType.KEYS); > cd.setValidationClass(JValueComparator.class.getName()); > cdL.add(cd); > ThriftCfDef cfd= new ThriftCfDef(HectorConfig.dfltKeyspaceName, cf, > ctName, cdL); > cfd.setKeyValidationClass(ComparatorType.UTF8TYPE.getClassName()); > cfd.setDefaultValidationClass(JValueComparator.class.getName()); > cluster.addColumnFamily(cfd); > > return "created: " + cf; > } > } > I'm inclined to exclude the presence of the custom comparator since: > (1) there is no issue using it if the cf doesn't have a secondary index > (2) the stack trace (see below) doesn't include the comparator > The above code throws the following error in Cassandra 1.1.2 and 1.1.3 > david@vlap1:~/opt/cassandra$ sudo bin/cassandra -f > xss = -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar > -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms128M -Xmx128M > -Xmn32M -XX:+HeapDumpOnOutOfMemoryError -Xss160k > INFO 23:15:31,333 Logging initialized > INFO 23:15:31,337 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0 > INFO 23:15:31,338 Heap size: 130875392/130875392 > INFO 23:15:31,338 Classpath: > /etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/hector-core-1.0-3.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jellyfish.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO 23:15:31,340 JNA not found. Native methods will be disabled. > INFO 23:15:31,351 Loading settings from file:/etc/cassandra/cassandra.yaml > INFO 23:15:31,566 DiskAccessMode 'auto' determined to be mmap, > indexAccessMode is mmap > INFO 23:15:31,875 Global memtable threshold is enabled at 41MB > INFO 23:15:32,380 In
[jira] [Updated] (CASSANDRA-2118) Provide failure modes if issues with the underlying filesystem of a node
[ https://issues.apache.org/jira/browse/CASSANDRA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2118: -- Attachment: 2118-tweaked.txt Looks reasonable. Tweaked version attached w/ some minor cleanup. Other things worth addressing: - Is there a reason for the FSError.Op enum? Looks like we don't need it if we just use instanceof instead in handleFSError. - Instead of trying to catch all the places we iterate sstables, what about either (1) removing unreadable sstables in DataTracker.get[Uncompacting]SSTables or (2) ripping them out of DataTracker when we handle the error? Either of those seems more foolproof to me. - Would be nice to persist the blacklisted sstables somehow. Maybe write a copy to each (other) data directory, so we don't try to read sstables that we've blacklisted, after a restart? - May be worth adding another option: best_effort_with_repair, where when we detect an unreadable disk we kick off a repair to rebuild that data automatically. > Provide failure modes if issues with the underlying filesystem of a node > > > Key: CASSANDRA-2118 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2118 > Project: Cassandra > Issue Type: Sub-task > Components: Core >Reporter: Chris Goffinet >Assignee: Aleksey Yeschenko > Fix For: 1.2 > > Attachments: > 0001-Provide-failure-modes-if-issues-with-the-underlying-.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v2.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v3.patch, > 2118-tweaked.txt, CASSANDRA-2118-part1.patch, CASSANDRA-2118-v1.patch > > > CASSANDRA-2116 introduces the ability to detect FS errors. Let's provide a > mode in cassandra.yaml so operators can decide that in the event of failure > what to do: > 1) standard - means continue on all errors (default) > 2) read - means only stop gossip/rpc server if 'reads' fail from drive, > writes can fail but not kill gossip/rpc server > 3) readwrite - means stop gossip/rpc server if any read or write errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2118) Provide failure modes if issues with the underlying filesystem of a node
[ https://issues.apache.org/jira/browse/CASSANDRA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434595#comment-13434595 ] Aleksey Yeschenko commented on CASSANDRA-2118: -- The enum is there for logging purposes only. There used to be two places that logged the error and it was cleaner this way. And since there was the enum already, I used it in the comparison instead of instanceof. If I don't use it in any new places after everything else is done, I'll get rid of the enum. I like your point 2. Not sure about the persistence part. What if there is no longer an issue (say, the directory is again available for writes)? > Provide failure modes if issues with the underlying filesystem of a node > > > Key: CASSANDRA-2118 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2118 > Project: Cassandra > Issue Type: Sub-task > Components: Core >Reporter: Chris Goffinet >Assignee: Aleksey Yeschenko > Fix For: 1.2 > > Attachments: > 0001-Provide-failure-modes-if-issues-with-the-underlying-.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v2.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v3.patch, > 2118-tweaked.txt, CASSANDRA-2118-part1.patch, CASSANDRA-2118-v1.patch > > > CASSANDRA-2116 introduces the ability to detect FS errors. Let's provide a > mode in cassandra.yaml so operators can decide that in the event of failure > what to do: > 1) standard - means continue on all errors (default) > 2) read - means only stop gossip/rpc server if 'reads' fail from drive, > writes can fail but not kill gossip/rpc server > 3) readwrite - means stop gossip/rpc server if any read or write errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4539) potential backwards incompatibility in native protocol
[ https://issues.apache.org/jira/browse/CASSANDRA-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434599#comment-13434599 ] paul cannon commented on CASSANDRA-4539: Another potential problem is if a client program wants to use a STARTUP option not yet understood by the driver. If an option's value part is always a {{[value]}}, then the driver doesn't have to guess how to send that argument to the server. (It might also be worthwhile to make the values in STARTUP optionlists always be {{[string]}} s, so that apps don't have to bother with potentially encoding binary values themselves when the driver doesn't recognize those options.) > potential backwards incompatibility in native protocol > -- > > Key: CASSANDRA-4539 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4539 > Project: Cassandra > Issue Type: Improvement > Components: API >Affects Versions: 1.2 >Reporter: paul cannon >Assignee: Sylvain Lebresne >Priority: Minor > Labels: cql, native_protocol > Fix For: 1.2 > > > In the text of the native_protocol.spec document, it explains the format for > a notation called {{[option]}}, which should represent "{{a pair of > }}". > In doing a first-draft implementation of the protocol for the python driver, > though, I found that I had a misunderstanding. I read that section as saying > that {{}} was a {{[value]}}, and that it could have a length of 0 > (i.e., the {{[int]}} on the front of the {{[value]}} could be 0). However, it > turns out that {{}} might not be there at all, or might be *two* > {{[value]}}'s, depending on the option id and message context. > I'm not a fan of this, since > * A protocol parsing library can't simply implement a single function to > read in {{[option]}}'s, since the length of the value part is dependent on > the message context > * If we add a new native data type (a new option id which could be used > inside a {{}} in a RESULT message), then older clients will not > know how to read past the value part. Of course they won't know how to > interpret the data or deserialize later rows of that unknown type - that's > not the problem - the problem is that the older client should be able just to > mark that column as unparseable and still handle the rest of the columns. > Can we make {{}} be a {{[value]}}, the contents of which can be > re-interpreted as a {{[string]}}, an {{[option]}}, two {{[option]}}'s, or > whatever? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2118) Provide failure modes if issues with the underlying filesystem of a node
[ https://issues.apache.org/jira/browse/CASSANDRA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434609#comment-13434609 ] Jonathan Ellis commented on CASSANDRA-2118: --- bq. The enum is there for logging purposes only I'd say let's just log the exception object and give it a decent toString. bq. What if there is no longer an issue Having the operator clear out the blacklist files on restart isn't unreasonable if that's what he wants. > Provide failure modes if issues with the underlying filesystem of a node > > > Key: CASSANDRA-2118 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2118 > Project: Cassandra > Issue Type: Sub-task > Components: Core >Reporter: Chris Goffinet >Assignee: Aleksey Yeschenko > Fix For: 1.2 > > Attachments: > 0001-Provide-failure-modes-if-issues-with-the-underlying-.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v2.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v3.patch, > 2118-tweaked.txt, CASSANDRA-2118-part1.patch, CASSANDRA-2118-v1.patch > > > CASSANDRA-2116 introduces the ability to detect FS errors. Let's provide a > mode in cassandra.yaml so operators can decide that in the event of failure > what to do: > 1) standard - means continue on all errors (default) > 2) read - means only stop gossip/rpc server if 'reads' fail from drive, > writes can fail but not kill gossip/rpc server > 3) readwrite - means stop gossip/rpc server if any read or write errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2118) Provide failure modes if issues with the underlying filesystem of a node
[ https://issues.apache.org/jira/browse/CASSANDRA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434613#comment-13434613 ] Jonathan Ellis commented on CASSANDRA-2118: --- bq. May be worth adding another option: best_effort_with_repair Let's save this for a followup after we see how well the blacklisting actually works in production. :) > Provide failure modes if issues with the underlying filesystem of a node > > > Key: CASSANDRA-2118 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2118 > Project: Cassandra > Issue Type: Sub-task > Components: Core >Reporter: Chris Goffinet >Assignee: Aleksey Yeschenko > Fix For: 1.2 > > Attachments: > 0001-Provide-failure-modes-if-issues-with-the-underlying-.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v2.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v3.patch, > 2118-tweaked.txt, CASSANDRA-2118-part1.patch, CASSANDRA-2118-v1.patch > > > CASSANDRA-2116 introduces the ability to detect FS errors. Let's provide a > mode in cassandra.yaml so operators can decide that in the event of failure > what to do: > 1) standard - means continue on all errors (default) > 2) read - means only stop gossip/rpc server if 'reads' fail from drive, > writes can fail but not kill gossip/rpc server > 3) readwrite - means stop gossip/rpc server if any read or write errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-2118) Provide failure modes if issues with the underlying filesystem of a node
[ https://issues.apache.org/jira/browse/CASSANDRA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434613#comment-13434613 ] Jonathan Ellis edited comment on CASSANDRA-2118 at 8/15/12 10:00 AM: - bq. May be worth adding another option: best_effort_with_repair Let's save this for a followup after we see how well the blacklisting actually works in production. :) And since this is the main reason we'd want to persist the blacklist, I'm okay with punting that down the road too. was (Author: jbellis): bq. May be worth adding another option: best_effort_with_repair Let's save this for a followup after we see how well the blacklisting actually works in production. :) > Provide failure modes if issues with the underlying filesystem of a node > > > Key: CASSANDRA-2118 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2118 > Project: Cassandra > Issue Type: Sub-task > Components: Core >Reporter: Chris Goffinet >Assignee: Aleksey Yeschenko > Fix For: 1.2 > > Attachments: > 0001-Provide-failure-modes-if-issues-with-the-underlying-.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v2.patch, > 0001-Provide-failure-modes-if-issues-with-the-underlying-v3.patch, > 2118-tweaked.txt, CASSANDRA-2118-part1.patch, CASSANDRA-2118-v1.patch > > > CASSANDRA-2116 introduces the ability to detect FS errors. Let's provide a > mode in cassandra.yaml so operators can decide that in the event of failure > what to do: > 1) standard - means continue on all errors (default) > 2) read - means only stop gossip/rpc server if 'reads' fail from drive, > writes can fail but not kill gossip/rpc server > 3) readwrite - means stop gossip/rpc server if any read or write errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: remove spurious "v"
Updated Branches: refs/heads/trunk 6ce498f5a -> afbbe1abc remove spurious "v" Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/afbbe1ab Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/afbbe1ab Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/afbbe1ab Branch: refs/heads/trunk Commit: afbbe1abcae7a4f808cabde291d60103c53eea6a Parents: 6ce498f Author: Eric Evans Authored: Tue Aug 14 18:19:25 2012 -0500 Committer: Eric Evans Committed: Tue Aug 14 18:19:25 2012 -0500 -- build.xml |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/afbbe1ab/build.xml -- diff --git a/build.xml b/build.xml index 5e720e5..a58ded5 100644 --- a/build.xml +++ b/build.xml @@ -1,4 +1,4 @@ -v +
[jira] [Created] (CASSANDRA-4546) cqlsh: handle when full cassandra type class names are given
paul cannon created CASSANDRA-4546: -- Summary: cqlsh: handle when full cassandra type class names are given Key: CASSANDRA-4546 URL: https://issues.apache.org/jira/browse/CASSANDRA-4546 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.2 Reporter: paul cannon Assignee: paul cannon Fix For: 1.2 When a builtin Cassandra type was being used for data in previous versions of Cassandra, only the short name was sent: "UTF8Type", "TimeUUIDType", etc. Starting with 1.2, as of CASSANDRA-4453, the full class names are sent. Cqlsh doesn't know how to handle this, and is currently treating all data as if it were an unknown type. This goes as far as to cause an exception when the type is actually a number, because the driver deserializes it right, and then cqlsh tries to use it as a string. Here for googlage: {noformat} AttributeError: 'int' object has no attribute 'replace' {noformat} Fixeries are in order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4546) cqlsh: handle when full cassandra type class names are given
[ https://issues.apache.org/jira/browse/CASSANDRA-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] paul cannon updated CASSANDRA-4546: --- Attachment: 4546.patch.txt Fix attached. Also available on the 4546 branch on my github- this version tagged pending/4546: https://github.com/thepaul/cassandra/tree/4546 > cqlsh: handle when full cassandra type class names are given > > > Key: CASSANDRA-4546 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4546 > Project: Cassandra > Issue Type: Bug > Components: Tools >Affects Versions: 1.2 >Reporter: paul cannon >Assignee: paul cannon > Labels: cqlsh > Fix For: 1.2 > > Attachments: 4546.patch.txt > > > When a builtin Cassandra type was being used for data in previous versions of > Cassandra, only the short name was sent: "UTF8Type", "TimeUUIDType", etc. > Starting with 1.2, as of CASSANDRA-4453, the full class names are sent. > Cqlsh doesn't know how to handle this, and is currently treating all data as > if it were an unknown type. This goes as far as to cause an exception when > the type is actually a number, because the driver deserializes it right, and > then cqlsh tries to use it as a string. > Here for googlage: > {noformat} > AttributeError: 'int' object has no attribute 'replace' > {noformat} > Fixeries are in order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4324) Implement Lucene FST in for key index
[ https://issues.apache.org/jira/browse/CASSANDRA-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434727#comment-13434727 ] Jason Rutherglen commented on CASSANDRA-4324: - Jonathan, do you mean there is no need for 'array index' lookups into the 'index' keys? > Implement Lucene FST in for key index > - > > Key: CASSANDRA-4324 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4324 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Rutherglen >Assignee: Jason Rutherglen >Priority: Minor > Fix For: 1.2 > > Attachments: CASSANDRA-4324.patch, CASSANDRA-4324.patch, > CASSANDRA-4324.patch, lucene-core-4.0-SNAPSHOT.jar > > > The Lucene FST data structure offers a compact and fast system for indexing > Cassandra keys. More keys may be loaded which in turn should seeks faster. > * Update the IndexSummary class to make use of the Lucene FST, overriding the > serialization mechanism. > * Alter SSTableReader to make use of the FST seek mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data
[ https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434770#comment-13434770 ] Tommy Cheng commented on CASSANDRA-4538: Yes, i tried recreate keyspace, column family already. I also tried delete /var/lib/cassandra, format debian (to make sure it is the same setting with another PC). The problem is still here, so i think may be my particular hardware cause the problem. The DDL is included in cassandra-stresstest\schema\schema-stresstest.txt > Strange CorruptedBlockException when massive insert binary data > --- > > Key: CASSANDRA-4538 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4538 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.3 > Environment: Debian sequeeze 32bit >Reporter: Tommy Cheng >Priority: Critical > Labels: CorruptedBlockException, binary, insert > Attachments: cassandra-stresstest.zip > > > After inserting ~ 1 records, here is the error log > INFO 10:53:33,543 Compacted to > [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,]. > 407,681 to 409,133 (~100% of original) bytes for 9,250 keys at > 0.715926MB/s. Time: 545ms. > ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main] > java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:99) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377) > at > org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamil
[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data
[ https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434772#comment-13434772 ] Tommy Cheng commented on CASSANDRA-4538: You may try to run the test in eclipse, it is a eclipse project. > Strange CorruptedBlockException when massive insert binary data > --- > > Key: CASSANDRA-4538 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4538 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.3 > Environment: Debian sequeeze 32bit >Reporter: Tommy Cheng >Priority: Critical > Labels: CorruptedBlockException, binary, insert > Attachments: cassandra-stresstest.zip > > > After inserting ~ 1 records, here is the error log > INFO 10:53:33,543 Compacted to > [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,]. > 407,681 to 409,133 (~100% of original) bytes for 9,250 keys at > 0.715926MB/s. Time: 545ms. > ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main] > java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:99) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: > (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption > detected, chunk at 7530128 of length 19575. > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377) > at > org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:112) >
[jira] [Commented] (CASSANDRA-4324) Implement Lucene FST in for key index
[ https://issues.apache.org/jira/browse/CASSANDRA-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434782#comment-13434782 ] Jonathan Ellis commented on CASSANDRA-4324: --- Not from what I saw skimming the uses. > Implement Lucene FST in for key index > - > > Key: CASSANDRA-4324 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4324 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Rutherglen >Assignee: Jason Rutherglen >Priority: Minor > Fix For: 1.2 > > Attachments: CASSANDRA-4324.patch, CASSANDRA-4324.patch, > CASSANDRA-4324.patch, lucene-core-4.0-SNAPSHOT.jar > > > The Lucene FST data structure offers a compact and fast system for indexing > Cassandra keys. More keys may be loaded which in turn should seeks faster. > * Update the IndexSummary class to make use of the Lucene FST, overriding the > serialization mechanism. > * Alter SSTableReader to make use of the FST seek mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2710) Get multiple column ranges
[ https://issues.apache.org/jira/browse/CASSANDRA-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434802#comment-13434802 ] Vijay commented on CASSANDRA-2710: -- It has a lot of potential (Provides flexibility) while modeling wide rows with composite columns. Should we at-least support it via thrift? > Get multiple column ranges > -- > > Key: CASSANDRA-2710 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2710 > Project: Cassandra > Issue Type: Sub-task > Components: API, Core >Reporter: David Boxenhorn >Assignee: Vijay > Labels: compositeColumns, cql > Attachments: 0001-2710-multiple-column-ranges-cql.patch, > 0001-2710-multiple-column-ranges-thrift.patch > > > I have replaced all my super column families with regular column families > using composite columns. I have easily been able to support all previous > functionality (I don't need range delete) except for one thing: getting > multiple super columns with a single access. For this, I would need to get > multiple ranges. (I can get multiple columns, or a single range, but not > multiple ranges.) > For example, I used to have > [,],[,] > and I could get superColumnName1, superColumnName2 > Now I have > [0..0],[0..0] > and I need to get superColumnName1..superColumnName1+, > superColumnName2..superColumnName2+ > to get the same functionality > I would like the clients to support this functionality, e.g. Hector to have > .setRages parallel to .setColumnNames > and for CQL to support a syntax like > SELECT [FIRST N] [REVERSED] name1..nameN1, name2..nameN2... FROM ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-2710) Get multiple column ranges
[ https://issues.apache.org/jira/browse/CASSANDRA-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434802#comment-13434802 ] Vijay edited comment on CASSANDRA-2710 at 8/15/12 3:25 PM: --- Hi Jonathan, It has a lot of potential (Provides flexibility) while modeling wide rows with composite columns. Should we at-least support it via thrift? was (Author: vijay2...@yahoo.com): It has a lot of potential (Provides flexibility) while modeling wide rows with composite columns. Should we at-least support it via thrift? > Get multiple column ranges > -- > > Key: CASSANDRA-2710 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2710 > Project: Cassandra > Issue Type: Sub-task > Components: API, Core >Reporter: David Boxenhorn >Assignee: Vijay > Labels: compositeColumns, cql > Attachments: 0001-2710-multiple-column-ranges-cql.patch, > 0001-2710-multiple-column-ranges-thrift.patch > > > I have replaced all my super column families with regular column families > using composite columns. I have easily been able to support all previous > functionality (I don't need range delete) except for one thing: getting > multiple super columns with a single access. For this, I would need to get > multiple ranges. (I can get multiple columns, or a single range, but not > multiple ranges.) > For example, I used to have > [,],[,] > and I could get superColumnName1, superColumnName2 > Now I have > [0..0],[0..0] > and I need to get superColumnName1..superColumnName1+, > superColumnName2..superColumnName2+ > to get the same functionality > I would like the clients to support this functionality, e.g. Hector to have > .setRages parallel to .setColumnNames > and for CQL to support a syntax like > SELECT [FIRST N] [REVERSED] name1..nameN1, name2..nameN2... FROM ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can skip caches
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434830#comment-13434830 ] Zhu Han commented on CASSANDRA-4533: Is it possible that it is fixed also in 1.0? > Multithreaded cache saving can skip caches > -- > > Key: CASSANDRA-4533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4533 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Zhu Han >Assignee: Yuki Morishita >Priority: Trivial > Fix For: 1.1.5 > > > Cassandra flushes the key and row cache to disk periodically. It also uses a > atomic flag in flushInProgress to enforce single cache writer at any time. > However, the cache saving task could be submitted to CompactionManager > concurrently, as long as the number of worker thread in CompactionManager is > larger than 1. > Due to the effect of above atomic flag, only one cache will be written out to > disk. Other writer are cancelled when the flag is true. > I observe the situation in Cassandra 1.0. If nothing is changed, the problem > should remain in Cassandra 1.1, either. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4533) Multithreaded cache saving can skip caches
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434830#comment-13434830 ] Zhu Han edited comment on CASSANDRA-4533 at 8/15/12 5:15 PM: - Is it possible to fix it also in 1.0? was (Author: hanzhu): Is it possible that it is fixed also in 1.0? > Multithreaded cache saving can skip caches > -- > > Key: CASSANDRA-4533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4533 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Zhu Han >Assignee: Yuki Morishita >Priority: Trivial > Fix For: 1.1.5 > > > Cassandra flushes the key and row cache to disk periodically. It also uses a > atomic flag in flushInProgress to enforce single cache writer at any time. > However, the cache saving task could be submitted to CompactionManager > concurrently, as long as the number of worker thread in CompactionManager is > larger than 1. > Due to the effect of above atomic flag, only one cache will be written out to > disk. Other writer are cancelled when the flag is true. > I observe the situation in Cassandra 1.0. If nothing is changed, the problem > should remain in Cassandra 1.1, either. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira