[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data
[ https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444765#comment-13444765 ] Christian Schnidrig commented on CASSANDRA-4538: I'm affraid, I ran into the same bug with version 1.1.4: INFO [CompactionExecutor:1137] 2012-08-29 16:24:14,005 CompactionTask.java (line 109) Compacting [SSTableReader(path='/mnt/md0/cassandra/data/content/oneChunkFileData/content-oneChunkFileData-he-6698-Data.db'), SSTableReader(path='/mnt/md0/cassandra/data/content/oneChun kFileData/content-oneChunkFileData-he-6697-Data.db'), SSTableReader(path='/mnt/md0/cassandra/data/content/oneChunkFileData/content-oneChunkFileData-he-6696-Data.db'), SSTableReader(path='/mnt/md0/cassandra/data/content/oneChunkFileData/content-oneChunkFileData-he-6889-Da ta.db'), SSTableReader(path='/mnt/md0/cassandra/data/content/oneChunkFileData/content-oneChunkFileData-he-7053-Data.db')] ERROR [CompactionExecutor:1137] 2012-08-29 16:24:14,712 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:1137,1,main] java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: (/mnt/md0/cassandra/data/content/oneChunkFileData/content-oneChunkFileData-he-6889-Data.db): corruption detected, chunk at 262155 of length 65545. at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:99) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/mnt/md0/cassandra/data/content/oneChunkFileData/content-oneChunkFileData-he-6889-Data.db): corruption detected, chunk at 262155 of length 65545. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394) at org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:112) ... 21 more - This happend on a CF with binary
[jira] [Created] (CASSANDRA-4587) StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext
Christian Schnidrig created CASSANDRA-4587: -- Summary: StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext Key: CASSANDRA-4587 URL: https://issues.apache.org/jira/browse/CASSANDRA-4587 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.4 Environment: debian OpenJDK 64-Bit Server VM/1.6.0_18 Heap size: 8341422080/8342470656 Reporter: Christian Schnidrig while running nodetool repair, the following was logged in system.log: ERROR [ValidationExecutor:2] 2012-08-30 10:58:19,490 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ValidationExecutor:2,1,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76) at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258) at java.lang.StringCoding.encode(StringCoding.java:290) at java.lang.String.getBytes(String.java:954) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(RandomAccessFile.java:233) at org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:67) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:64) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1007) at org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:56) at org.apache.cassandra.io.sstable.SSTableBoundedScanner.init(SSTableBoundedScanner.java:41) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:869) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:247) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) . (about 900 lines deleted) . at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:202) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147) at org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:90) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:47) at org.apache.cassandra.db.compaction.CompactionIterable.iterator(CompactionIterable.java:60) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:703) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:69) at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:442) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at
[Cassandra Wiki] Trivial Update of ZLeoma by ZLeoma
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The ZLeoma page has been changed by ZLeoma: http://wiki.apache.org/cassandra/ZLeoma New page: Nothing to write about myself at all.BRYes! Im a member of apache.org.BRI just hope Im useful at allBRBRLook at my blog [[http://www.bootland.nl/tercoo-perago-roterende-straler.html|Informatie]]
[jira] [Created] (CASSANDRA-4588) CQL COPY ... FROM command is slow
Piotr Kołaczkowski created CASSANDRA-4588: - Summary: CQL COPY ... FROM command is slow Key: CASSANDRA-4588 URL: https://issues.apache.org/jira/browse/CASSANDRA-4588 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.4 Environment: Ubuntu Linux 12.04, kernel 3.4.0 Reporter: Piotr Kołaczkowski 1. created a csv file with 10,000,000 rows with two integer columns; saved it to an SSD disk, it took a few seconds, the file is 184 MB large. 2. started a single local cassandra node from fresh empty data and commit log dirs 3. created a keyspace with simple strategy and RF=1 4. loading the file with COPY ... FROM command - it is over 15 minutes now and still loading top reports about 50% CPU usage for java (cassandra) and 50% for python. I/O is almost idle, iowait 0.1%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4588) CQL COPY ... FROM command is slow
[ https://issues.apache.org/jira/browse/CASSANDRA-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-4588. - Resolution: Won't Fix Wontfixing, since performance was never a goal as outlined in CASSANDRA-4012 CQL COPY ... FROM command is slow - Key: CASSANDRA-4588 URL: https://issues.apache.org/jira/browse/CASSANDRA-4588 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.4 Environment: Ubuntu Linux 12.04, kernel 3.4.0 Reporter: Piotr Kołaczkowski 1. created a csv file with 10,000,000 rows with two integer columns; saved it to an SSD disk, it took a few seconds, the file is 184 MB large. 2. started a single local cassandra node from fresh empty data and commit log dirs 3. created a keyspace with simple strategy and RF=1 4. loading the file with COPY ... FROM command - it is over 15 minutes now and still loading top reports about 50% CPU usage for java (cassandra) and 50% for python. I/O is almost idle, iowait 0.1%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[3/3] git commit: Fix writing sstables to wrong directory when compacting
Fix writing sstables to wrong directory when compacting
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e6167da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e6167da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e6167da
Branch: refs/heads/trunk
Commit: 4e6167da57915e803946f35f039d7a33680f4693
Parents: 0525ae2
Author: Yuki Morishita yu...@apache.org
Authored: Wed Aug 29 10:20:11 2012 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Thu Aug 30 07:19:01 2012 -0500
--
.../cassandra/db/compaction/CompactionTask.java|4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e6167da/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index 4d2a90f..ff08b61 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@ -155,7 +155,7 @@ public class CompactionTask extends AbstractCompactionTask
return;
}
-SSTableWriter writer = cfs.createCompactionWriter(keysPerSSTable,
dataDirectory, toCompact);
+SSTableWriter writer = cfs.createCompactionWriter(keysPerSSTable,
cfs.directories.getLocationForDisk(dataDirectory), toCompact);
writers.add(writer);
while (nni.hasNext())
{
@@ -187,7 +187,7 @@ public class CompactionTask extends AbstractCompactionTask
sstables.add(toIndex);
if (nni.hasNext())
{
-writer = cfs.createCompactionWriter(keysPerSSTable,
dataDirectory, toCompact);
+writer = cfs.createCompactionWriter(keysPerSSTable,
cfs.directories.getLocationForDisk(dataDirectory), toCompact);
writers.add(writer);
cachedKeys = new HashMapDecoratedKey,
RowIndexEntry();
}
[jira] [Resolved] (CASSANDRA-4292) Improve JBOD loadbalancing and reduce contention
[ https://issues.apache.org/jira/browse/CASSANDRA-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuki Morishita resolved CASSANDRA-4292. --- Resolution: Fixed Committed. Improve JBOD loadbalancing and reduce contention Key: CASSANDRA-4292 URL: https://issues.apache.org/jira/browse/CASSANDRA-4292 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Yuki Morishita Fix For: 1.2.0 beta 1 Attachments: 0001-Fix-writing-sstables-to-wrong-directory-when-compact.patch, 4292.txt, 4292-v2.txt, 4292-v3.txt, 4292-v4.txt As noted in CASSANDRA-809, we have a certain amount of flush (and compaction) threads, which mix and match disk volumes indiscriminately. It may be worth creating a tight thread - disk affinity, to prevent unnecessary conflict at that level. OTOH as SSDs become more prevalent this becomes a non-issue. Unclear how much pain this actually causes in practice in the meantime. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: (was: 0002-Fix-tests.txt) Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: (was: 0003-Add-tokens-and-status-atomically.txt) Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: (was: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt) Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: 0002-Fix-tests.txt 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: (was: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt) Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4383) Binary encoding of vnode tokens
[
https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444935#comment-13444935
]
Brandon Williams commented on CASSANDRA-4383:
-
bq. The approach taken here continues to make me a little hesitant simply
because, I think, it introduces for the first time a need for proper ordering
of STATE transmission/reception. I don't have a clear-enough understanding of
how the underlying messaging works to know if we can firmly rely on that or not
In this case, it's ok, because SS only reacts to STATUS, so we can change any
other gossip state before changing that one and, regardless of how many gossip
events fire in the meantime, be guaranteed that other hosts react to the STATUS
update after all other gossip state changes are received.
bq. What I had in mind was (at least) something like a
sendState{Normal,Bootstrap,...}(Collectiontoken tokens) to encapsulate those
operations sensitive to ordering. Gossiper.addLocalApplicationStates(...) still
makes it too easy to do the wrong thing.
That's basically syntactic sugar over
{{Gossiper.addLocalApplicationStates(...)}} which we'd still need as a building
block. I've given up on my previous attempt which actually did not provide
atomicity, and it looks like doing that would require rewriting EndpointState
to use the Reference and SnapTree approach similar to AtomicSortedColumns, and
that is way, way out of scope for this ticket (especially since we don't
actually need it)
bq. And in addition to getHostId, usesHostId also seems better suited to the
Gossiper.
You're right, updated patch moves those methods to Gossiper. I kept the name
{{getHostId}} there though since, by asking the gossiper, you should know
you're getting the hostId as it is *in gossip*, which may be a necessary thing
instead of tMD for non-ring members (or old removed members, or whatever.)
bq. But, mostly I meant that it doesn't read as well as the old code that
clearly did one thing when the version was X, and another when it was = Y.
That makes sense, I brought the old NET_VERSION checks back, but relocated to
the gossiper.
Binary encoding of vnode tokens
---
Key: CASSANDRA-4383
URL: https://issues.apache.org/jira/browse/CASSANDRA-4383
Project: Cassandra
Issue Type: Sub-task
Reporter: Brandon Williams
Assignee: Brandon Williams
Fix For: 1.2.0 beta 1
Attachments:
0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt,
0002-Fix-tests.txt
Since after CASSANDRA-4317 we can know which version a remote node is using
(that is, whether it is vnode-aware or not) this a good opportunity to change
the token encoding to binary, since with a default of 256 tokens per node
even a fixed-length 16 byte encoding per token provides a great deal of
savings in gossip traffic over a text representation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: (was: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt) Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: 0002-Fix-tests.txt 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4383: Attachment: (was: 0002-Fix-tests.txt) Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4383) Binary encoding of vnode tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444944#comment-13444944 ] Brandon Williams commented on CASSANDRA-4383: - Quoting myself: bq. The last wrinkle is the case of bootstrapping a node without using vnodes, and without specifying a token. This technically still works, but produces a harmless NPE on the existing nodes when they first see the state and look for TOKENS. I'm hesitant to allow this to pass through though since it could paper over a truly serious bug, and bootstrapping in this fashion is already deprecated by vnodes. I fixed that in this patch too, by simple confirming that TOKENS is actually present for the host in handleStateBootstrap, and if not, just handle things the old way since it's a legacy-style bootstrap. Binary encoding of vnode tokens --- Key: CASSANDRA-4383 URL: https://issues.apache.org/jira/browse/CASSANDRA-4383 Project: Cassandra Issue Type: Sub-task Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 1.2.0 beta 1 Attachments: 0001-Add-HOST_ID-and-TOKENS-app-states-binary-serialization.txt, 0002-Fix-tests.txt Since after CASSANDRA-4317 we can know which version a remote node is using (that is, whether it is vnode-aware or not) this a good opportunity to change the token encoding to binary, since with a default of 256 tokens per node even a fixed-length 16 byte encoding per token provides a great deal of savings in gossip traffic over a text representation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2897) Secondary indexes without read-before-write
[ https://issues.apache.org/jira/browse/CASSANDRA-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444955#comment-13444955 ] Sam Tunnicliffe commented on CASSANDRA-2897: Looks like 0525ae25 introduced that test failure. Secondary indexes without read-before-write --- Key: CASSANDRA-2897 URL: https://issues.apache.org/jira/browse/CASSANDRA-2897 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.0 Reporter: Sylvain Lebresne Assignee: Sam Tunnicliffe Priority: Minor Labels: secondary_index Fix For: 1.2.0 beta 1 Attachments: 0001-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt, 0002-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt, 0003-CASSANDRA-2897.txt, 2897-apply-cleanup.txt, 2897-v4.txt, 41ec9fc-2897.txt Currently, secondary index updates require a read-before-write to maintain the index consistency. Keeping the index consistent at all time is not necessary however. We could let the (secondary) index get inconsistent on writes and repair those on reads. This would be easy because on reads, we make sure to request the indexed columns anyway, so we can just skip the row that are not needed and repair the index at the same time. This does trade work on writes for work on reads. However, read-before-write is sufficiently costly that it will likely be a win overall. There is (at least) two small technical difficulties here though: # If we repair on read, this will be racy with writes, so we'll probably have to synchronize there. # We probably shouldn't only rely on read to repair and we should also have a task to repair the index for things that are rarely read. It's unclear how to make that low impact though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-4567) Error in log related to Murmur3Partitioner
[
https://issues.apache.org/jira/browse/CASSANDRA-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reopened CASSANDRA-4567:
---
looks like this broke
SSTableReaderTest.testPersistantStatisticsWithSecondaryIndex
Error in log related to Murmur3Partitioner
--
Key: CASSANDRA-4567
URL: https://issues.apache.org/jira/browse/CASSANDRA-4567
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 1
Environment: Using ccm on ubuntu
Reporter: Tyler Patterson
Assignee: Vijay
Fix For: 1.2.0 beta 1
Attachments: 0001-CASSANDRA-4567.patch, 0001-CASSANDRA-4567-v2.patch,
0001-CASSANDRA-4567-v3.patch
Start a 2-node cluster on cassandra-1.1. Bring down one node, upgrade it to
trunk, start it back up. The following error shows up in the log:
{code}
...
INFO [main] 2012-08-22 10:44:40,012 CacheService.java (line 170) Scheduling
row cache save to each 0 seconds (going to save all keys).
INFO [SSTableBatchOpen:1] 2012-08-22 10:44:40,106 SSTableReader.java (line
164) Opening
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-2
(148 bytes)
INFO [SSTableBatchOpen:2] 2012-08-22 10:44:40,106 SSTableReader.java (line
164) Opening
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-1
(226 bytes)
INFO [SSTableBatchOpen:3] 2012-08-22 10:44:40,106 SSTableReader.java (line
164) Opening
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-3
(89 bytes)
ERROR [SSTableBatchOpen:3] 2012-08-22 10:44:40,114 CassandraDaemon.java (line
131) Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.RuntimeException: Cannot open
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-3
because partitioner does not match
org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [SSTableBatchOpen:2] 2012-08-22 10:44:40,114 CassandraDaemon.java (line
131) Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.RuntimeException: Cannot open
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-1
because partitioner does not match
org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [SSTableBatchOpen:1] 2012-08-22 10:44:40,114 CassandraDaemon.java (line
131) Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: Cannot open
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-2
because partitioner does not match
org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at
[jira] [Comment Edited] (CASSANDRA-2897) Secondary indexes without read-before-write
[
https://issues.apache.org/jira/browse/CASSANDRA-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444965#comment-13444965
]
Jonathan Ellis edited comment on CASSANDRA-2897 at 8/31/12 12:59 AM:
-
bq. the test was definitely failing until I got dummyColumn figured out
FTR, the problem was
{{IColumn dummyColumn = new Column(liveColumn.name(), column.value(),
column.timestamp());}}
Here, {{column}} is the column from the index row, so column.value is always
empty. To actually delete the old index entry we need to use the column value
that was indexed:
{{IColumn dummyColumn = new Column(baseColumnName, indexedValue,
column.timestamp());}}
There was a corresponding bug for the non-composite case as well.
(Edit: liveColumn.name() == baseColumnName, just thought the latter was a bit
more clear.)
was (Author: jbellis):
bq. the test was definitely failing until I got dummyColumn figured out
FTR, the problem was
{{IColumn dummyColumn = new Column(liveColumn.name(), column.value(),
column.timestamp());}}
Here, {{column}} is the column from the index row, so column.value is always
empty. To actually delete the old index entry we need to use the column value
that was indexed:
{{IColumn dummyColumn = new Column(baseColumnName, indexedValue,
column.timestamp());}}
There was a corresponding bug for the non-composite case as well.
Secondary indexes without read-before-write
---
Key: CASSANDRA-2897
URL: https://issues.apache.org/jira/browse/CASSANDRA-2897
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Assignee: Sam Tunnicliffe
Priority: Minor
Labels: secondary_index
Fix For: 1.2.0 beta 1
Attachments:
0001-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt,
0002-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt,
0003-CASSANDRA-2897.txt, 2897-apply-cleanup.txt, 2897-v4.txt, 41ec9fc-2897.txt
Currently, secondary index updates require a read-before-write to maintain
the index consistency. Keeping the index consistent at all time is not
necessary however. We could let the (secondary) index get inconsistent on
writes and repair those on reads. This would be easy because on reads, we
make sure to request the indexed columns anyway, so we can just skip the row
that are not needed and repair the index at the same time.
This does trade work on writes for work on reads. However, read-before-write
is sufficiently costly that it will likely be a win overall.
There is (at least) two small technical difficulties here though:
# If we repair on read, this will be racy with writes, so we'll probably have
to synchronize there.
# We probably shouldn't only rely on read to repair and we should also have a
task to repair the index for things that are rarely read. It's unclear how to
make that low impact though.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2897) Secondary indexes without read-before-write
[
https://issues.apache.org/jira/browse/CASSANDRA-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444965#comment-13444965
]
Jonathan Ellis commented on CASSANDRA-2897:
---
bq. the test was definitely failing until I got dummyColumn figured out
FTR, the problem was
{{IColumn dummyColumn = new Column(liveColumn.name(), column.value(),
column.timestamp());}}
Here, {{column}} is the column from the index row, so column.value is always
empty. To actually delete the old index entry we need to use the column value
that was indexed:
{{IColumn dummyColumn = new Column(baseColumnName, indexedValue,
column.timestamp());}}
There was a corresponding bug for the non-composite case as well.
Secondary indexes without read-before-write
---
Key: CASSANDRA-2897
URL: https://issues.apache.org/jira/browse/CASSANDRA-2897
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Assignee: Sam Tunnicliffe
Priority: Minor
Labels: secondary_index
Fix For: 1.2.0 beta 1
Attachments:
0001-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt,
0002-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt,
0003-CASSANDRA-2897.txt, 2897-apply-cleanup.txt, 2897-v4.txt, 41ec9fc-2897.txt
Currently, secondary index updates require a read-before-write to maintain
the index consistency. Keeping the index consistent at all time is not
necessary however. We could let the (secondary) index get inconsistent on
writes and repair those on reads. This would be easy because on reads, we
make sure to request the indexed columns anyway, so we can just skip the row
that are not needed and repair the index at the same time.
This does trade work on writes for work on reads. However, read-before-write
is sufficiently costly that it will likely be a win overall.
There is (at least) two small technical difficulties here though:
# If we repair on read, this will be racy with writes, so we'll probably have
to synchronize there.
# We probably shouldn't only rely on read to repair and we should also have a
task to repair the index for things that are rarely read. It's unclear how to
make that low impact though.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2897) Secondary indexes without read-before-write
[ https://issues.apache.org/jira/browse/CASSANDRA-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444970#comment-13444970 ] Jonathan Ellis commented on CASSANDRA-2897: --- bq. failing test in SSTableReaderTest reopened CASSANDRA-4567 for that. fixed javadoc typo and committed. Secondary indexes without read-before-write --- Key: CASSANDRA-2897 URL: https://issues.apache.org/jira/browse/CASSANDRA-2897 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.0 Reporter: Sylvain Lebresne Assignee: Sam Tunnicliffe Priority: Minor Labels: secondary_index Fix For: 1.2.0 beta 1 Attachments: 0001-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt, 0002-CASSANDRA-2897-Secondary-indexes-without-read-before-w.txt, 0003-CASSANDRA-2897.txt, 2897-apply-cleanup.txt, 2897-v4.txt, 41ec9fc-2897.txt Currently, secondary index updates require a read-before-write to maintain the index consistency. Keeping the index consistent at all time is not necessary however. We could let the (secondary) index get inconsistent on writes and repair those on reads. This would be easy because on reads, we make sure to request the indexed columns anyway, so we can just skip the row that are not needed and repair the index at the same time. This does trade work on writes for work on reads. However, read-before-write is sufficiently costly that it will likely be a win overall. There is (at least) two small technical difficulties here though: # If we repair on read, this will be racy with writes, so we'll probably have to synchronize there. # We probably shouldn't only rely on read to repair and we should also have a task to repair the index for things that are rarely read. It's unclear how to make that low impact though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[2/2] redesign KEYS indexes to avoid read-before-write patch by Sam Tunnicliffe, jbellis, and Philip Jenvey for CASSANDRA-2897
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a1b93d7/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 8fd5807..090aad3 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -60,6 +60,7 @@ import org.apache.cassandra.config.ConfigurationException;
import org.apache.cassandra.db.columniterator.IdentityQueryFilter;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.index.SecondaryIndex;
+import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.db.marshal.LexicalUUIDType;
import org.apache.cassandra.db.marshal.LongType;
import org.apache.cassandra.dht.Bounds;
@@ -424,6 +425,143 @@ public class ColumnFamilyStoreTest extends SchemaLoader
assert k1.equals( key );
}
+
+@Test
+public void testDeleteOfInconsistentValuesInKeysIndex() throws Exception
+{
+String keySpace = Keyspace2;
+String cfName = Indexed1;
+
+Table table = Table.open(keySpace);
+ColumnFamilyStore cfs = table.getColumnFamilyStore(cfName);
+cfs.truncate().get();
+
+ByteBuffer rowKey = ByteBufferUtil.bytes(k1);
+ByteBuffer colName = ByteBufferUtil.bytes(birthdate);
+ByteBuffer val1 = ByteBufferUtil.bytes(1L);
+ByteBuffer val2 = ByteBufferUtil.bytes(2L);
+
+// create a row and update the birthdate value, test that the index
query fetches this version
+RowMutation rm;
+rm = new RowMutation(keySpace, rowKey);
+rm.add(new QueryPath(cfName, null, colName), val1, 0);
+rm.apply();
+IndexExpression expr = new IndexExpression(colName, IndexOperator.EQ,
val1);
+ListIndexExpression clause = Arrays.asList(expr);
+IFilter filter = new IdentityQueryFilter();
+RangeRowPosition range = Util.range(, );
+ListRow rows = table.getColumnFamilyStore(cfName).search(clause,
range, 100, filter);
+assertEquals(1, rows.size());
+
+// force a flush, so our index isn't being read from a memtable
+table.getColumnFamilyStore(cfName).forceBlockingFlush();
+
+// now apply another update, but force the index update to be skipped
+rm = new RowMutation(keySpace, rowKey);
+rm.add(new QueryPath(cfName, null, colName), val2, 1);
+table.apply(rm, true, false);
+
+// Now searching the index for either the old or new value should
return 0 rows
+// because the new value was not indexed and the old value should be
ignored
+// (and in fact purged from the index cf).
+// first check for the old value
+rows = table.getColumnFamilyStore(cfName).search(clause, range, 100,
filter);
+assertEquals(0, rows.size());
+// now check for the updated value
+expr = new IndexExpression(colName, IndexOperator.EQ, val2);
+clause = Arrays.asList(expr);
+filter = new IdentityQueryFilter();
+range = Util.range(, );
+rows = table.getColumnFamilyStore(cfName).search(clause, range, 100,
filter);
+assertEquals(0, rows.size());
+
+// now, reset back to the original value, still skipping the index
update, to
+// make sure the value was expunged from the index when it was
discovered to be inconsistent
+rm = new RowMutation(keySpace, rowKey);
+rm.add(new QueryPath(cfName, null, colName), ByteBufferUtil.bytes(1L),
3);
+table.apply(rm, true, false);
+
+expr = new IndexExpression(colName, IndexOperator.EQ,
ByteBufferUtil.bytes(1L));
+clause = Arrays.asList(expr);
+filter = new IdentityQueryFilter();
+range = Util.range(, );
+rows = table.getColumnFamilyStore(cfName).search(clause, range, 100,
filter);
+assertEquals(0, rows.size());
+}
+
+@Test
+public void testDeleteOfInconsistentValuesFromCompositeIndex() throws
Exception
+{
+String keySpace = Keyspace2;
+String cfName = Indexed2;
+
+Table table = Table.open(keySpace);
+ColumnFamilyStore cfs = table.getColumnFamilyStore(cfName);
+cfs.truncate().get();
+
+ByteBuffer rowKey = ByteBufferUtil.bytes(k1);
+ByteBuffer clusterKey = ByteBufferUtil.bytes(ck1);
+ByteBuffer colName = ByteBufferUtil.bytes(col1);
+CompositeType baseComparator = (CompositeType)cfs.getComparator();
+CompositeType.Builder builder = baseComparator.builder();
+builder.add(clusterKey);
+builder.add(colName);
+ByteBuffer compositeName = builder.build();
+
+ByteBuffer val1 = ByteBufferUtil.bytes(v1);
+ByteBuffer val2 = ByteBufferUtil.bytes(v2);
+
+// create a
[jira] [Commented] (CASSANDRA-4542) add atomic_batch_mutate method
[ https://issues.apache.org/jira/browse/CASSANDRA-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445002#comment-13445002 ] Aleksey Yeschenko commented on CASSANDRA-4542: -- Is it safe to replay counter mutations? If not, should they even be allowed as part of the batch? I currently don't serialize them to the batchlog, only saving RowMutations. Am I right about this? add atomic_batch_mutate method -- Key: CASSANDRA-4542 URL: https://issues.apache.org/jira/browse/CASSANDRA-4542 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Fix For: 1.2.1 atomic_batch_mutate will have the same parameters as batch_mutate, but will write to the batchlog before attempting distribution to the batch rows' replicas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4542) add atomic_batch_mutate method
[ https://issues.apache.org/jira/browse/CASSANDRA-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445012#comment-13445012 ] Jonathan Ellis commented on CASSANDRA-4542: --- It is not safe, so they should not be allowed in the batch. add atomic_batch_mutate method -- Key: CASSANDRA-4542 URL: https://issues.apache.org/jira/browse/CASSANDRA-4542 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Fix For: 1.2.1 atomic_batch_mutate will have the same parameters as batch_mutate, but will write to the batchlog before attempting distribution to the batch rows' replicas. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of Metrics by yukim
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Cassandra Wiki for
change notification.
The Metrics page has been changed by yukim:
http://wiki.apache.org/cassandra/Metrics
New page:
= Cassandra Metrics =
This page describes new
metrics([[https://issues.apache.org/jira/browse/CASSANDRA-4009|CASSANDRA-4009]])
planned in upcoming version 1.2.
== Overview ==
Apache Cassandra version 1.1 introduced metrics using
[[https://github.com/codahale/metrics|Codahale's Metrics library]]. The library
enables easier exposure of metrics and integration with other systems.
What you can get from metrics are basically the same with 1.1 but reimplemented
and put them in order using Metrics library. You can still query using old JMX
paths,
but they are deprecated and may be removed in future version.
As of version 1.2, Cassandra exposes following group of metrics.
* [[#cache|Cache]]
* [[#clientrequest|Client Request]]
* [[#columnfamily|ColumnFamily]]
* [[#commitlog|Commit Log]]
* [[#compaction|Compaction]]
* [[#connection|Connection]]
* [[#droppedmessage|Dropped Message]]
* [[#streaming|Streaming]]
* [[#storage|Storage]]
* [[#threadpool|Thread Pool]]
Anchor(cache)
== Cache (5 metrics/cache) ==
Cache metrics are created per cache type (key cache, row cache).
=== Codahale Metric Name ===
This section shows defined !MetricName properties.
|| group || org.apache.cassandra.metrics ||
|| type || Cache||
|| scope || !KeyCache or !RowCache ||
|| name || name of metric ||
=== JMX Object Name ===
This section shows JMX !ObjectName for easy category.
{{{ org.apache.cassandra.metrics:type=Cache,scope=(Cache type),name=(Metric
name) }}}
=== Metrics ===
!CapacityInBytes
: Cache capacity in bytes.
Hits
: Cache hit count.
!HitRate
: Cache hit rate.
Requests
: Cache request count.
Size
: Cache size in bytes.
Anchor(clientrequest)
== Client Request (4 metrics/request type) ==
Metrics for read/range slice/write client request.
=== Codahale Metric Name ===
|| group || org.apache.cassandra.metrics ||
|| type || !ClientRequest ||
|| scope || Read or Write or !RangeSlice ||
|| name || name of metric ||
=== JMX Object Name ===
{{{
org.apache.cassandra.metrics:type=ClientRequest,scope=(Read|Write|RangeSlice),name=(Metric
name) }}}
=== Metrics ===
Latency
: Latency statistics.
!TotalLatency
: Total latecy in micro seconds.
Timeouts
: Total number of timeout requests. More precisely, total number of
!TimeoutException thrown.
Unavailables
: Total number of unavailable requests. More precisely, total number of
!UnavailableException thrown.
== ColumnFamily (23 metrics/ColumnFamily) ==
!ColumnFamily metrics are created per !ColumnFamily.
=== Codahale Metric Name ===
|| group || org.apache.cassandra.metrics ||
|| type || !ColumnFamily or !IndexColumnFamily ||
|| scope || (Keyspace name).(!ColumnFamily name) ||
|| name || name of metric ||
If !ColumnFamily is for secondary index, then ''type'' will be
''IndexColumnFamily''.
=== JMX Object Name ===
{{{
org.apache.cassandra.metrics:type=(ColumnFamily|IndexColumnFamily),keyspace=(Keyspace
name),scope=(ColumnFamily Name),name=(Metric name) }}}
=== Metrics ===
!BloomFilterDiskSpaceUsed
: Disk space used by bloom filter.
!BloomFilterFalsePositives
: Number of false positives for bloom filter.
!BloomFilterFalseRatio
: False positive ratio of bloom filter.
!CompressionRatio
: Current compression ratio for all SSTables.
!EstimatedRowSizeHistogram
: Histogram of estimated row size (in bytes).
!EstimatedColumnCountHistogram
: Histogram of estimated number of columns.
!LiveDiskSpaceUsed
: Disk space used by 'live' SSTables.
LiveSSTableCount
: Number of 'livw' SSTables.
!MaxRowSize
: Size of the largest compacted row.
!MeanRowSize
: Mean size of compacted rows.
!MemtableColumnsCount
: Total number of columns present in memtable.
!MemtableDataSize
: Total amount of data stored in memtable, including column related overhead.
!MemtableSwitchCount
: Number of times flushing has resulted in memtable being switched out.
!MinRowSize
: Size of the smallest compacted row.
!PendingTasks
: Estimated number of tasks pending.
!ReadLatency
: Read latency statistics.
!ReadTotalLatency
: Total latecy in micro seconds for reads.
!RecentBloomFilterFalsePositives
: Number of false positives since last check.
!RecentBloomFilterFalseRatio
: False positive ratio since last check.
SSTablesPerReadHistogram
: Histogram of the number of SSTables accessed per read.
!TotalDiskSpaceUsed
: Total disk space used by SSTables including obsolete ones waiting to be GC'd.
!WriteLatency
: Write latency statistics.
!WriteTotalLatency
: Total latecy in micro seconds for writes.
Anchor(commitlog)
== Commit Log (3 metrics) ==
=== Codahale Metric Name ===
|| group || org.apache.cassandra.metrics ||
|| type || !CommitLog ||
[jira] [Updated] (CASSANDRA-4497) Update CQL pseudo-maps to real maps
[
https://issues.apache.org/jira/browse/CASSANDRA-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Yaskevich updated CASSANDRA-4497:
---
Attachment: CASSANDRA-4497.patch
{replication, compression, compaction}_parameters are made to be set using
key = { k : v, ... } syntax.
Update CQL pseudo-maps to real maps
---
Key: CASSANDRA-4497
URL: https://issues.apache.org/jira/browse/CASSANDRA-4497
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.0 beta 1
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
Fix For: 1.2.0 beta 1
Attachments: CASSANDRA-4497.patch
- compression_parameters
- replication_parameters (combine strategy + options like we did compression)
- anything else?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4587) StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext
[ https://issues.apache.org/jira/browse/CASSANDRA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4587: -- Attachment: 4587.txt patch to simplify LeveledScanner.computeNext and avoid recursion StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext -- Key: CASSANDRA-4587 URL: https://issues.apache.org/jira/browse/CASSANDRA-4587 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.4 Environment: debian OpenJDK 64-Bit Server VM/1.6.0_18 Heap size: 8341422080/8342470656 Reporter: Christian Schnidrig Assignee: Jonathan Ellis Priority: Minor Labels: compaction Fix For: 1.1.5 Attachments: 4587.txt while running nodetool repair, the following was logged in system.log: ERROR [ValidationExecutor:2] 2012-08-30 10:58:19,490 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ValidationExecutor:2,1,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76) at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258) at java.lang.StringCoding.encode(StringCoding.java:290) at java.lang.String.getBytes(String.java:954) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(RandomAccessFile.java:233) at org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:67) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:64) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1007) at org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:56) at org.apache.cassandra.io.sstable.SSTableBoundedScanner.init(SSTableBoundedScanner.java:41) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:869) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:247) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) . (about 900 lines deleted) . at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:202) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147) at org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:90) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:47) at org.apache.cassandra.db.compaction.CompactionIterable.iterator(CompactionIterable.java:60) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:703) at
[jira] [Updated] (CASSANDRA-4587) StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext
[ https://issues.apache.org/jira/browse/CASSANDRA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4587: -- Reviewer: yukim Priority: Minor (was: Major) Fix Version/s: 1.1.5 Assignee: Jonathan Ellis Labels: compaction (was: ) StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext -- Key: CASSANDRA-4587 URL: https://issues.apache.org/jira/browse/CASSANDRA-4587 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.4 Environment: debian OpenJDK 64-Bit Server VM/1.6.0_18 Heap size: 8341422080/8342470656 Reporter: Christian Schnidrig Assignee: Jonathan Ellis Priority: Minor Labels: compaction Fix For: 1.1.5 Attachments: 4587.txt while running nodetool repair, the following was logged in system.log: ERROR [ValidationExecutor:2] 2012-08-30 10:58:19,490 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ValidationExecutor:2,1,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76) at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258) at java.lang.StringCoding.encode(StringCoding.java:290) at java.lang.String.getBytes(String.java:954) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(RandomAccessFile.java:233) at org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:67) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:64) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1007) at org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:56) at org.apache.cassandra.io.sstable.SSTableBoundedScanner.init(SSTableBoundedScanner.java:41) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:869) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:247) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) . (about 900 lines deleted) . at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:202) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147) at org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:90) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:47) at org.apache.cassandra.db.compaction.CompactionIterable.iterator(CompactionIterable.java:60) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:703) at
[jira] [Created] (CASSANDRA-4589) writes are allowed when RFN
Brandon Williams created CASSANDRA-4589:
---
Summary: writes are allowed when RFN
Key: CASSANDRA-4589
URL: https://issues.apache.org/jira/browse/CASSANDRA-4589
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.2
Reporter: Brandon Williams
Fix For: 1.2.0
Easily repro'd by running stress with a ridiculous rf:
{{ # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 }}
We're supposed to allow creation of a ks where the rf exceeds the amount of
nodes, but we shouldn't be able to write to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4589) writes are allowed when RFN
[
https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-4589:
Description:
Easily repro'd by running stress with a ridiculous rf:
# tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456
We're supposed to allow creation of a ks where the rf exceeds the amount of
nodes, but we shouldn't be able to write to it.
was:
Easily repro'd by running stress with a ridiculous rf:
{{ # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 }}
We're supposed to allow creation of a ks where the rf exceeds the amount of
nodes, but we shouldn't be able to write to it.
writes are allowed when RFN
Key: CASSANDRA-4589
URL: https://issues.apache.org/jira/browse/CASSANDRA-4589
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.2
Reporter: Brandon Williams
Fix For: 1.2.0
Easily repro'd by running stress with a ridiculous rf:
# tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456
We're supposed to allow creation of a ks where the rf exceeds the amount of
nodes, but we shouldn't be able to write to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4567) Error in log related to Murmur3Partitioner
[
https://issues.apache.org/jira/browse/CASSANDRA-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay resolved CASSANDRA-4567.
--
Resolution: Fixed
Fixed in 9cd53fba648ae5a30a181f8a06786f33db95a0fe
Error in log related to Murmur3Partitioner
--
Key: CASSANDRA-4567
URL: https://issues.apache.org/jira/browse/CASSANDRA-4567
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 1
Environment: Using ccm on ubuntu
Reporter: Tyler Patterson
Assignee: Vijay
Fix For: 1.2.0 beta 1
Attachments: 0001-CASSANDRA-4567.patch, 0001-CASSANDRA-4567-v2.patch,
0001-CASSANDRA-4567-v3.patch
Start a 2-node cluster on cassandra-1.1. Bring down one node, upgrade it to
trunk, start it back up. The following error shows up in the log:
{code}
...
INFO [main] 2012-08-22 10:44:40,012 CacheService.java (line 170) Scheduling
row cache save to each 0 seconds (going to save all keys).
INFO [SSTableBatchOpen:1] 2012-08-22 10:44:40,106 SSTableReader.java (line
164) Opening
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-2
(148 bytes)
INFO [SSTableBatchOpen:2] 2012-08-22 10:44:40,106 SSTableReader.java (line
164) Opening
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-1
(226 bytes)
INFO [SSTableBatchOpen:3] 2012-08-22 10:44:40,106 SSTableReader.java (line
164) Opening
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-3
(89 bytes)
ERROR [SSTableBatchOpen:3] 2012-08-22 10:44:40,114 CassandraDaemon.java (line
131) Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.RuntimeException: Cannot open
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-3
because partitioner does not match
org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [SSTableBatchOpen:2] 2012-08-22 10:44:40,114 CassandraDaemon.java (line
131) Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.RuntimeException: Cannot open
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-1
because partitioner does not match
org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [SSTableBatchOpen:1] 2012-08-22 10:44:40,114 CassandraDaemon.java (line
131) Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: Cannot open
/tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-2
because partitioner does not match
org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO
[jira] [Commented] (CASSANDRA-4572) lost+found directory in the data dir causes problems again
[
https://issues.apache.org/jira/browse/CASSANDRA-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445129#comment-13445129
]
Jonathan Ellis commented on CASSANDRA-4572:
---
+1
lost+found directory in the data dir causes problems again
--
Key: CASSANDRA-4572
URL: https://issues.apache.org/jira/browse/CASSANDRA-4572
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Reporter: Brandon Williams
Assignee: Yuki Morishita
Fix For: 1.1.5
Attachments: 4572-1.1.txt
Looks like we've regressed from CASSANDRA-1547 and mounting a fs directly on
the data dir is a problem again.
{noformat}
INFO [main] 2012-08-22 23:30:03,710 Directories.java (line 475) Upgrade from
pre-1.1 version detected: migrating sstables to new directory layout ERROR
[main] 2012-08-22 23:30:03,712 AbstractCassandraDaemon.java (line 370)
Exception encountered during startup
java.lang.NullPointerException at
org.apache.cassandra.db.Directories.migrateSSTables(Directories.java:487)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445165#comment-13445165 ] Vijay commented on CASSANDRA-4589: -- I think it might be efficient to check this when we create a KS and remove a node? Else we need to check this for every write/read. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445169#comment-13445169 ] Brandon Williams commented on CASSANDRA-4589: - It shouldn't be any different than checking if there are enough live nodes on every read/write. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4587) StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext
[ https://issues.apache.org/jira/browse/CASSANDRA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445173#comment-13445173 ] Yuki Morishita commented on CASSANDRA-4587: --- +1 StackOverflowError in LeveledCompactionStrategy$LeveledScanner.computeNext -- Key: CASSANDRA-4587 URL: https://issues.apache.org/jira/browse/CASSANDRA-4587 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.4 Environment: debian OpenJDK 64-Bit Server VM/1.6.0_18 Heap size: 8341422080/8342470656 Reporter: Christian Schnidrig Assignee: Jonathan Ellis Priority: Minor Labels: compaction Fix For: 1.1.5 Attachments: 4587.txt while running nodetool repair, the following was logged in system.log: ERROR [ValidationExecutor:2] 2012-08-30 10:58:19,490 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ValidationExecutor:2,1,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76) at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258) at java.lang.StringCoding.encode(StringCoding.java:290) at java.lang.String.getBytes(String.java:954) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.init(RandomAccessFile.java:233) at org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:67) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:64) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1007) at org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:56) at org.apache.cassandra.io.sstable.SSTableBoundedScanner.init(SSTableBoundedScanner.java:41) at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:869) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:247) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) . (about 900 lines deleted) . at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:240) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:248) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy$LeveledScanner.computeNext(LeveledCompactionStrategy.java:202) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147) at org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:90) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:47) at org.apache.cassandra.db.compaction.CompactionIterable.iterator(CompactionIterable.java:60) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:703) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:69)
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445176#comment-13445176 ] Brandon Williams commented on CASSANDRA-4589: - This is a regression from CASSANDRA-2129 writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[1/3] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/cassandra-1.1 de039598f - 60dadc5dc refs/heads/trunk 9cd53fba6 - f471b927b Merge branch 'cassandra-1.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f471b927 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f471b927 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f471b927 Branch: refs/heads/trunk Commit: f471b927bbe1f2e7b30ea005b3331be951449c4d Parents: 9cd53fb 60dadc5 Author: Jonathan Ellis jbel...@apache.org Authored: Thu Aug 30 13:43:24 2012 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Thu Aug 30 13:43:24 2012 -0500 -- conf/cassandra-env.sh | 10 +- 1 files changed, 5 insertions(+), 5 deletions(-) --
[2/3] git commit: heap defaults are pretty good now, change language to may wish to adjust
heap defaults are pretty good now, change language to may wish to adjust Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/60dadc5d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/60dadc5d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/60dadc5d Branch: refs/heads/trunk Commit: 60dadc5dcbe470804f2dad153d3e1da64cac7540 Parents: de03959 Author: Jonathan Ellis jbel...@apache.org Authored: Thu Aug 30 13:40:22 2012 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Thu Aug 30 13:40:22 2012 -0500 -- conf/cassandra-env.sh | 10 +- 1 files changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/60dadc5d/conf/cassandra-env.sh -- diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh index ff3fc86..c2a1078 100644 --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@ -110,11 +110,11 @@ esac # Override these to set the amount of memory to allocate to the JVM at -# start-up. For production use you almost certainly want to adjust -# this for your environment. MAX_HEAP_SIZE is the total amount of -# memory dedicated to the Java heap; HEAP_NEWSIZE refers to the size -# of the young generation. Both MAX_HEAP_SIZE and HEAP_NEWSIZE should -# be either set or not (if you set one, set the other). +# start-up. For production use you may wish to adjust this for your +# environment. MAX_HEAP_SIZE is the total amount of memory dedicated +# to the Java heap; HEAP_NEWSIZE refers to the size of the young +# generation. Both MAX_HEAP_SIZE and HEAP_NEWSIZE should be either set +# or not (if you set one, set the other). # # The main trade-off for the young generation is that the larger it # is, the longer GC pause times will be. The shorter it is, the more
[3/3] git commit: heap defaults are pretty good now, change language to may wish to adjust
heap defaults are pretty good now, change language to may wish to adjust Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/60dadc5d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/60dadc5d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/60dadc5d Branch: refs/heads/cassandra-1.1 Commit: 60dadc5dcbe470804f2dad153d3e1da64cac7540 Parents: de03959 Author: Jonathan Ellis jbel...@apache.org Authored: Thu Aug 30 13:40:22 2012 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Thu Aug 30 13:40:22 2012 -0500 -- conf/cassandra-env.sh | 10 +- 1 files changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/60dadc5d/conf/cassandra-env.sh -- diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh index ff3fc86..c2a1078 100644 --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@ -110,11 +110,11 @@ esac # Override these to set the amount of memory to allocate to the JVM at -# start-up. For production use you almost certainly want to adjust -# this for your environment. MAX_HEAP_SIZE is the total amount of -# memory dedicated to the Java heap; HEAP_NEWSIZE refers to the size -# of the young generation. Both MAX_HEAP_SIZE and HEAP_NEWSIZE should -# be either set or not (if you set one, set the other). +# start-up. For production use you may wish to adjust this for your +# environment. MAX_HEAP_SIZE is the total amount of memory dedicated +# to the Java heap; HEAP_NEWSIZE refers to the size of the young +# generation. Both MAX_HEAP_SIZE and HEAP_NEWSIZE should be either set +# or not (if you set one, set the other). # # The main trade-off for the young generation is that the larger it # is, the longer GC pause times will be. The shorter it is, the more
[jira] [Created] (CASSANDRA-4590) The system cannot find the path specified when creating hard link on Windows
Allen Servedio created CASSANDRA-4590: - Summary: The system cannot find the path specified when creating hard link on Windows Key: CASSANDRA-4590 URL: https://issues.apache.org/jira/browse/CASSANDRA-4590 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.3 Environment: Windows 7 - 64 bit Reporter: Allen Servedio Priority: Minor When upgrading from Cassandra 1.0.5 to 1.1.3, we have a test case (uses embedded Cassandra) that started failing as shown below. Other than the upgrade, no changes were made to the code or config. I believe this MAY be related to the change made in CASSANDRA-3101. We verified that the file it is trying to create the hard link to does exist - so it is purely the creation of the link that is failing. Here is the basic failure: # [11:31:00.307] [ERROR] [o.a.c.u.CLibrary] [createHardLinkWithExec] [Unable to create hard link] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\XYZ\source_code\s7-t1\crs-inventory\inventory-core\target\test\cassandra\data\RevKeyspace\PropertyProductDefaultInventoryCounts\snapshots\1346340659980-PropertyProductDefaultInventoryCounts\RevKeyspace-PropertyProductDefaultInventoryCounts-he-1-CompressionInfo.db C:\XYZ\source_code\s7-t1\crs-inventory\inventory-core\target\test\cassandra\data\RevKeyspace\PropertyProductDefaultInventoryCounts\RevKeyspace-PropertyProductDefaultInventoryCounts-he-1-CompressionInfo.db, command error Code: 1, command output: The system cannot find the path specified. Here is a more complete log output: # [11:30:59.975] [DEBUG] [o.a.c.d.CollationController] [collectAllData] [collectAllData] # [11:30:59.976] [DEBUG] [o.a.c.i.u.FileUtils] [deleteWithConfirm] [Deleting system-schema_columnfamilies-he-4-Digest.sha1] # [11:30:59.977] [DEBUG] [o.a.c.i.u.FileUtils] [deleteWithConfirm] [Deleting system-schema_columnfamilies-he-4-Index.db] # [11:30:59.978] [DEBUG] [o.a.c.i.u.FileUtils] [deleteWithConfirm] [Deleting system-schema_columnfamilies-he-4-Filter.db] # [11:30:59.978] [DEBUG] [o.a.c.d.CollationController] [collectAllData] [collectAllData] # [11:30:59.979] [DEBUG] [o.a.c.d.CollationController] [collectAllData] [collectAllData] # [11:30:59.979] [DEBUG] [o.a.c.i.u.FileUtils] [deleteWithConfirm] [Deleting system-schema_columnfamilies-he-4-Statistics.db] # [11:30:59.979] [DEBUG] [o.a.c.d.CollationController] [collectAllData] [collectAllData] # [11:30:59.980] [DEBUG] [o.a.c.d.CollationController] [collectAllData] [collectAllData] # [11:30:59.980] [DEBUG] [o.a.c.i.s.SSTable] [delete] [Deleted target\test\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-he-4] # [11:30:59.981] [INFO ] [o.a.c.d.ColumnFamilyStore] [maybeSwitchMemtable] [Enqueuing flush of Memtable-PropertyProductDefaultInventoryCounts@2002512083(74/92 serialized/live bytes, 1 ops)] # [11:30:59.981] [INFO ] [o.a.c.d.Memtable] [writeSortedContents] [Writing Memtable-PropertyProductDefaultInventoryCounts@2002512083(74/92 serialized/live bytes, 1 ops)] # [11:30:59.992] [DEBUG] [o.a.c.d.Directories] [getLocationWithMaximumAvailableSpace] [expected data files size is 134; largest free partition (target\test\cassandra\data\RevKeyspace\PropertyProductDefaultInventoryCounts) has 82645161984 bytes free] # [11:31:00.012] [INFO ] [o.a.c.d.Memtable] [writeSortedContents] [Completed flushing target\test\cassandra\data\RevKeyspace\PropertyProductDefaultInventoryCounts\RevKeyspace-PropertyProductDefaultInventoryCounts-he-1-Data.db (123 bytes) for commitlog position ReplayPosition(segmentId=592725621297887, position=6701)] # [11:31:00.013] [DEBUG] [o.a.c.u.I.IntervalNode] [init] [Creating IntervalNode from [Interval(DecoratedKey(70791399548943621833439300945136455431, 50726f706572747950726f6475637431323334), DecoratedKey(70791399548943621833439300945136455431, 50726f706572747950726f6475637431323334))]] # [11:31:00.013] [DEBUG] [o.a.c.d.DataTracker] [addNewSSTablesSize] [adding target\test\cassandra\data\RevKeyspace\PropertyProductDefaultInventoryCounts\RevKeyspace-PropertyProductDefaultInventoryCounts-he-1 to list of files tracked for RevKeyspace.PropertyProductDefaultInventoryCounts] # [11:31:00.014] [DEBUG] [o.a.c.d.c.CompactionManager] [submitBackground] [Scheduling a background task check for RevKeyspace.PropertyProductDefaultInventoryCounts with SizeTieredCompactionStrategy] # [11:31:00.014] [DEBUG] [o.a.c.d.c.CompactionManager] [runMayThrow] [Checking RevKeyspace.PropertyProductDefaultInventoryCounts] # [11:31:00.014] [DEBUG] [o.a.c.d.c.CommitLog] [call] [discard completed log segments for ReplayPosition(segmentId=592725621297887, position=6701), column family 1001] # [11:31:00.014] [DEBUG] [o.a.c.d.c.SizeTieredCompactionStrategy] [getNextBackgroundTask] [Compaction buckets are
[1/3] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/cassandra-1.1 60dadc5dc - b0342978a refs/heads/trunk f471b927b - aff58e8ee Merge branch 'cassandra-1.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aff58e8e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aff58e8e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aff58e8e Branch: refs/heads/trunk Commit: aff58e8ee4415815766e10b842baf39eec209ef9 Parents: f471b92 b034297 Author: Jonathan Ellis jbel...@apache.org Authored: Thu Aug 30 13:46:06 2012 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Thu Aug 30 13:46:06 2012 -0500 -- CHANGES.txt|1 + .../db/compaction/LeveledCompactionStrategy.java | 26 +- 2 files changed, 10 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/aff58e8e/CHANGES.txt -- diff --cc CHANGES.txt index ce7ec50,d3eafe4..5019369 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,53 -1,5 +1,54 @@@ +1.2-dev + * new metrics (CASSANDRA-4009) + * redesign KEYS indexes to avoid read-before-write (CASSANDRA-2897) + * debug tracing (CASSANDRA-1123) + * parallelize row cache loading (CASSANDRA-4282) + * Make compaction, flush JBOD-aware (CASSANDRA-4292) + * run local range scans on the read stage (CASSANDRA-3687) + * clean up ioexceptions (CASSANDRA-2116) + * add disk_failure_policy (CASSANDRA-2118) + * Introduce new json format with row level deletion (CASSANDRA-4054) + * remove redundant name column from schema_keyspaces (CASSANDRA-4433) + * improve nodetool ring handling of multi-dc clusters (CASSANDRA-3047) + * update NTS calculateNaturalEndpoints to be O(N log N) (CASSANDRA-3881) + * add UseCondCardMark XX jvm settings on jdk 1.7 (CASSANDRA-4366) + * split up rpc timeout by operation type (CASSANDRA-2819) + * rewrite key cache save/load to use only sequential i/o (CASSANDRA-3762) + * update MS protocol with a version handshake + broadcast address id + (CASSANDRA-4311) + * multithreaded hint replay (CASSANDRA-4189) + * add inter-node message compression (CASSANDRA-3127) + * remove COPP (CASSANDRA-2479) + * Track tombstone expiration and compact when tombstone content is + higher than a configurable threshold, default 20% (CASSANDRA-3442, 4234) + * update MurmurHash to version 3 (CASSANDRA-2975) + * (CLI) track elapsed time for `delete' operation (CASSANDRA-4060) + * (CLI) jline version is bumped to 1.0 to properly support + 'delete' key function (CASSANDRA-4132) + * Save IndexSummary into new SSTable 'Summary' component (CASSANDRA-2392, 4289) + * Add support for range tombstones (CASSANDRA-3708) + * Improve MessagingService efficiency (CASSANDRA-3617) + * Avoid ID conflicts from concurrent schema changes (CASSANDRA-3794) + * Set thrift HSHA server thread limit to unlimited by default (CASSANDRA-4277) + * Avoids double serialization of CF id in RowMutation messages + (CASSANDRA-4293) + * stream compressed sstables directly with java nio (CASSANDRA-4297) + * Support multiple ranges in SliceQueryFilter (CASSANDRA-3885) + * Add column metadata to system column families (CASSANDRA-4018) + * (cql3) Always use composite types by default (CASSANDRA-4329) + * (cql3) Add support for set, map and list (CASSANDRA-3647) + * Validate date type correctly (CASSANDRA-4441) + * (cql3) Allow definitions with only a PK (CASSANDRA-4361) + * (cql3) Add support for row key composites (CASSANDRA-4179) + * improve DynamicEndpointSnitch by using reservoir sampling (CASSANDRA-4038) + * (cql3) Add support for 2ndary indexes (CASSANDRA-3680) + * (cql3) fix defining more than one PK to be invalid (CASSANDRA-4477) + * remove schema agreement checking from all external APIs (Thrift, CQL and CQL3) (CASSANDRA-4487) + * add Murmur3Partitioner and make it default for new installations (CASSANDRA-3772) + + 1.1.5 + * avoid recursion in leveled compaction (CASSANDRA-4587) * increase stack size under Java7 to 180K * Log(info) schema changes (CASSANDRA-4547) * Change nodetool setcachecapcity to manipulate global caches (CASSANDRA-4563) http://git-wip-us.apache.org/repos/asf/cassandra/blob/aff58e8e/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java --
[3/3] git commit: avoid recursion in leveled compaction patch by jbellis; reviewed by yukim for CASSANDRA-4587
avoid recursion in leveled compaction
patch by jbellis; reviewed by yukim for CASSANDRA-4587
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b0342978
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b0342978
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b0342978
Branch: refs/heads/cassandra-1.1
Commit: b0342978a0a444b067fae25f4bf9a2f7e5dca0e3
Parents: 60dadc5
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Aug 30 13:45:58 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Aug 30 13:45:58 2012 -0500
--
CHANGES.txt|1 +
.../db/compaction/LeveledCompactionStrategy.java | 26 +-
2 files changed, 10 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0342978/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e8f277f..d3eafe4 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.1.5
+ * avoid recursion in leveled compaction (CASSANDRA-4587)
* increase stack size under Java7 to 180K
* Log(info) schema changes (CASSANDRA-4547)
* Change nodetool setcachecapcity to manipulate global caches (CASSANDRA-4563)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0342978/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
index 7fb58be..d14545c 100644
--- a/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
@@ -214,7 +214,8 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
this.range = range;
this.sstables = new ArrayListSSTableReader(sstables);
Collections.sort(this.sstables, SSTable.sstableComparator);
-this.sstableIterator = this.sstables.iterator();
+sstableIterator = this.sstables.iterator();
+currentScanner = sstableIterator.next().getDirectScanner(range);
long length = 0;
for (SSTableReader sstable : sstables)
@@ -226,26 +227,17 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
{
try
{
-if (currentScanner != null)
+while (true)
{
if (currentScanner.hasNext())
-{
return currentScanner.next();
-}
-else
-{
-positionOffset += currentScanner.getLengthInBytes();
-currentScanner.close();
-currentScanner = null;
-return computeNext();
-}
-}
-
-if (!sstableIterator.hasNext())
-return endOfData();
-currentScanner =
sstableIterator.next().getDirectScanner(range);
-return computeNext();
+positionOffset += currentScanner.getLengthInBytes();
+currentScanner.close();
+if (!sstableIterator.hasNext())
+return endOfData();
+currentScanner =
sstableIterator.next().getDirectScanner(range);
+}
}
catch (IOException e)
{
[2/3] git commit: avoid recursion in leveled compaction patch by jbellis; reviewed by yukim for CASSANDRA-4587
avoid recursion in leveled compaction
patch by jbellis; reviewed by yukim for CASSANDRA-4587
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b0342978
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b0342978
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b0342978
Branch: refs/heads/trunk
Commit: b0342978a0a444b067fae25f4bf9a2f7e5dca0e3
Parents: 60dadc5
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Aug 30 13:45:58 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Aug 30 13:45:58 2012 -0500
--
CHANGES.txt|1 +
.../db/compaction/LeveledCompactionStrategy.java | 26 +-
2 files changed, 10 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0342978/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e8f277f..d3eafe4 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.1.5
+ * avoid recursion in leveled compaction (CASSANDRA-4587)
* increase stack size under Java7 to 180K
* Log(info) schema changes (CASSANDRA-4547)
* Change nodetool setcachecapcity to manipulate global caches (CASSANDRA-4563)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0342978/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
index 7fb58be..d14545c 100644
--- a/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
@@ -214,7 +214,8 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
this.range = range;
this.sstables = new ArrayListSSTableReader(sstables);
Collections.sort(this.sstables, SSTable.sstableComparator);
-this.sstableIterator = this.sstables.iterator();
+sstableIterator = this.sstables.iterator();
+currentScanner = sstableIterator.next().getDirectScanner(range);
long length = 0;
for (SSTableReader sstable : sstables)
@@ -226,26 +227,17 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
{
try
{
-if (currentScanner != null)
+while (true)
{
if (currentScanner.hasNext())
-{
return currentScanner.next();
-}
-else
-{
-positionOffset += currentScanner.getLengthInBytes();
-currentScanner.close();
-currentScanner = null;
-return computeNext();
-}
-}
-
-if (!sstableIterator.hasNext())
-return endOfData();
-currentScanner =
sstableIterator.next().getDirectScanner(range);
-return computeNext();
+positionOffset += currentScanner.getLengthInBytes();
+currentScanner.close();
+if (!sstableIterator.hasNext())
+return endOfData();
+currentScanner =
sstableIterator.next().getDirectScanner(range);
+}
}
catch (IOException e)
{
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445184#comment-13445184 ] Brandon Williams commented on CASSANDRA-4589: - bq. if we cannot Read or write to a KS why allow creating it? Because that's how you add them for new DCs, and you might want to add this keyspace only to a new DC that hasn't bootstrapped in yet. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445184#comment-13445184 ] Brandon Williams edited comment on CASSANDRA-4589 at 8/31/12 5:49 AM: -- bq. if we cannot Read or write to a KS why allow creating it? Because that's how you add them for new DCs, with an rf of zero until you have the nodes in place. was (Author: brandon.williams): bq. if we cannot Read or write to a KS why allow creating it? Because that's how you add them for new DCs, and you might want to add this keyspace only to a new DC that hasn't bootstrapped in yet. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445188#comment-13445188 ] Brandon Williams commented on CASSANDRA-4589: - Also, you can have the situation where RF=N, but then a node dies and you have to remove it (again, CASSANDRA-2129). Now you can't write, obviously, but at a CL of ONE you should still be able to read. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[1/3] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches:
refs/heads/cassandra-1.1 b0342978a - 6e1f3a019
refs/heads/trunk aff58e8ee - 39fdebfd4
Merge branch 'cassandra-1.1' into trunk
Conflicts:
src/java/org/apache/cassandra/db/Directories.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/39fdebfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/39fdebfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/39fdebfd
Branch: refs/heads/trunk
Commit: 39fdebfd4917c40647c5f7db834d351d235562a0
Parents: aff58e8 6e1f3a0
Author: Yuki Morishita yu...@apache.org
Authored: Thu Aug 30 13:59:58 2012 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Thu Aug 30 13:59:58 2012 -0500
--
src/java/org/apache/cassandra/db/Directories.java | 54 ++-
1 files changed, 36 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/39fdebfd/src/java/org/apache/cassandra/db/Directories.java
--
diff --cc src/java/org/apache/cassandra/db/Directories.java
index 895f931,7ee2823..714b363
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@@ -532,21 -473,29 +532,29 @@@ public class Directorie
{
logger.info(Upgrade from pre-1.1 version detected: migrating
sstables to new directory layout);
-for (File location : dataFileLocations)
+for (DataDirectory dir : dataFileLocations)
{
-if (!location.exists() || !location.isDirectory())
+if (!dir.location.exists() || !dir.location.isDirectory())
continue;
- for (File ksDir : dir.location.listFiles())
-File[] ksDirs = location.listFiles();
++File[] ksDirs = dir.location.listFiles();
+ if (ksDirs != null)
{
- if (!ksDir.isDirectory())
- continue;
+ for (File ksDir : ksDirs)
+ {
+ if (!ksDir.isDirectory())
+ continue;
- for (File file : ksDir.listFiles())
- migrateFile(file, ksDir, null);
+ File[] files = ksDir.listFiles();
+ if (files != null)
+ {
+ for (File file : files)
+ migrateFile(file, ksDir, null);
+ }
- migrateSnapshots(ksDir);
- migrateBackups(ksDir);
+ migrateSnapshots(ksDir);
+ migrateBackups(ksDir);
+ }
}
}
}
[2/3] git commit: Fix NPE when listing directory; patch by yukim reviewed by jbellis for CASSANDRA-4572
Fix NPE when listing directory; patch by yukim reviewed by jbellis for
CASSANDRA-4572
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6e1f3a01
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6e1f3a01
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6e1f3a01
Branch: refs/heads/trunk
Commit: 6e1f3a0195b777c9ae79ab89230b67ca20c1adc4
Parents: b034297
Author: Yuki Morishita yu...@apache.org
Authored: Wed Aug 29 11:05:51 2012 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Thu Aug 30 13:54:56 2012 -0500
--
src/java/org/apache/cassandra/db/Directories.java | 55 ++--
1 files changed, 36 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6e1f3a01/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java
b/src/java/org/apache/cassandra/db/Directories.java
index 9c9f9b8..7ee2823 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -31,7 +31,6 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.*;
import org.apache.cassandra.db.compaction.LeveledManifest;
import org.apache.cassandra.io.util.FileUtils;
-import org.apache.cassandra.io.util.MmappedSegmentedFile;
import org.apache.cassandra.io.sstable.*;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.utils.CLibrary;
@@ -479,16 +478,24 @@ public class Directories
if (!location.exists() || !location.isDirectory())
continue;
-for (File ksDir : location.listFiles())
+File[] ksDirs = location.listFiles();
+if (ksDirs != null)
{
-if (!ksDir.isDirectory())
-continue;
+for (File ksDir : ksDirs)
+{
+if (!ksDir.isDirectory())
+continue;
-for (File file : ksDir.listFiles())
-migrateFile(file, ksDir, null);
+File[] files = ksDir.listFiles();
+if (files != null)
+{
+for (File file : files)
+migrateFile(file, ksDir, null);
+}
-migrateSnapshots(ksDir);
-migrateBackups(ksDir);
+migrateSnapshots(ksDir);
+migrateBackups(ksDir);
+}
}
}
}
@@ -499,16 +506,23 @@ public class Directories
if (!snapshotDir.exists())
return;
-for (File snapshot : snapshotDir.listFiles())
+File[] snapshots = snapshotDir.listFiles();
+if (snapshots != null)
{
-if (!snapshot.isDirectory())
-continue;
-
-for (File f : snapshot.listFiles())
-migrateFile(f, ksDir, join(SNAPSHOT_SUBDIR,
snapshot.getName()));
+for (File snapshot : snapshots)
+{
+if (!snapshot.isDirectory())
+continue;
-if (!snapshot.delete())
-logger.info(Old snapsot directory {} not deleted by
migraation as it is not empty, snapshot);
+File[] files = snapshot.listFiles();
+if (files != null)
+{
+for (File f : files)
+migrateFile(f, ksDir, join(SNAPSHOT_SUBDIR,
snapshot.getName()));
+}
+if (!snapshot.delete())
+logger.info(Old snapsot directory {} not deleted by
migraation as it is not empty, snapshot);
+}
}
if (!snapshotDir.delete())
logger.info(Old directory {} not deleted by migration as it is
not empty, snapshotDir);
@@ -520,9 +534,12 @@ public class Directories
if (!backupDir.exists())
return;
-for (File f : backupDir.listFiles())
-migrateFile(f, ksDir, BACKUPS_SUBDIR);
-
+File[] files = backupDir.listFiles();
+if (files != null)
+{
+for (File f : files)
+migrateFile(f, ksDir, BACKUPS_SUBDIR);
+}
if (!backupDir.delete())
logger.info(Old directory {} not deleted by migration as it is
not empty, backupDir);
}
[3/3] git commit: Fix NPE when listing directory; patch by yukim reviewed by jbellis for CASSANDRA-4572
Fix NPE when listing directory; patch by yukim reviewed by jbellis for
CASSANDRA-4572
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6e1f3a01
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6e1f3a01
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6e1f3a01
Branch: refs/heads/cassandra-1.1
Commit: 6e1f3a0195b777c9ae79ab89230b67ca20c1adc4
Parents: b034297
Author: Yuki Morishita yu...@apache.org
Authored: Wed Aug 29 11:05:51 2012 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Thu Aug 30 13:54:56 2012 -0500
--
src/java/org/apache/cassandra/db/Directories.java | 55 ++--
1 files changed, 36 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6e1f3a01/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java
b/src/java/org/apache/cassandra/db/Directories.java
index 9c9f9b8..7ee2823 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -31,7 +31,6 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.*;
import org.apache.cassandra.db.compaction.LeveledManifest;
import org.apache.cassandra.io.util.FileUtils;
-import org.apache.cassandra.io.util.MmappedSegmentedFile;
import org.apache.cassandra.io.sstable.*;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.utils.CLibrary;
@@ -479,16 +478,24 @@ public class Directories
if (!location.exists() || !location.isDirectory())
continue;
-for (File ksDir : location.listFiles())
+File[] ksDirs = location.listFiles();
+if (ksDirs != null)
{
-if (!ksDir.isDirectory())
-continue;
+for (File ksDir : ksDirs)
+{
+if (!ksDir.isDirectory())
+continue;
-for (File file : ksDir.listFiles())
-migrateFile(file, ksDir, null);
+File[] files = ksDir.listFiles();
+if (files != null)
+{
+for (File file : files)
+migrateFile(file, ksDir, null);
+}
-migrateSnapshots(ksDir);
-migrateBackups(ksDir);
+migrateSnapshots(ksDir);
+migrateBackups(ksDir);
+}
}
}
}
@@ -499,16 +506,23 @@ public class Directories
if (!snapshotDir.exists())
return;
-for (File snapshot : snapshotDir.listFiles())
+File[] snapshots = snapshotDir.listFiles();
+if (snapshots != null)
{
-if (!snapshot.isDirectory())
-continue;
-
-for (File f : snapshot.listFiles())
-migrateFile(f, ksDir, join(SNAPSHOT_SUBDIR,
snapshot.getName()));
+for (File snapshot : snapshots)
+{
+if (!snapshot.isDirectory())
+continue;
-if (!snapshot.delete())
-logger.info(Old snapsot directory {} not deleted by
migraation as it is not empty, snapshot);
+File[] files = snapshot.listFiles();
+if (files != null)
+{
+for (File f : files)
+migrateFile(f, ksDir, join(SNAPSHOT_SUBDIR,
snapshot.getName()));
+}
+if (!snapshot.delete())
+logger.info(Old snapsot directory {} not deleted by
migraation as it is not empty, snapshot);
+}
}
if (!snapshotDir.delete())
logger.info(Old directory {} not deleted by migration as it is
not empty, snapshotDir);
@@ -520,9 +534,12 @@ public class Directories
if (!backupDir.exists())
return;
-for (File f : backupDir.listFiles())
-migrateFile(f, ksDir, BACKUPS_SUBDIR);
-
+File[] files = backupDir.listFiles();
+if (files != null)
+{
+for (File f : files)
+migrateFile(f, ksDir, BACKUPS_SUBDIR);
+}
if (!backupDir.delete())
logger.info(Old directory {} not deleted by migration as it is
not empty, backupDir);
}
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[
https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445214#comment-13445214
]
Jonathan Ellis commented on CASSANDRA-4589:
---
{{assureSufficientLiveNodes}} used to be what would fail the write if you asked
for RF=1 and CL.QUORUM, for instance.
Did you actually try a higher CL? I kinda think that if you are asking for
CL.ONE then it is doing the Right Thing.
writes are allowed when RFN
Key: CASSANDRA-4589
URL: https://issues.apache.org/jira/browse/CASSANDRA-4589
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.2
Reporter: Brandon Williams
Assignee: Vijay
Priority: Minor
Fix For: 1.2.0
Easily repro'd by running stress with a ridiculous rf:
# tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456
We're supposed to allow creation of a ks where the rf exceeds the amount of
nodes, but we shouldn't be able to write to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-4589. - Resolution: Not A Problem bq. Did you actually try a higher CL? I kinda think that if you are asking for CL.ONE then it is doing the Right Thing. Nope :/ It's doing the right thing and throwing UE at higher CLs. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445242#comment-13445242 ] Jonathan Ellis commented on CASSANDRA-4589: --- Accepting a CL.ONE write, when there is = 1 replica available, sounds like correct behavior to me. writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445261#comment-13445261 ] Vijay commented on CASSANDRA-4589: -- Just realized that i was commenting on a resolved ticket :) Thanks! writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (CASSANDRA-4589) writes are allowed when RFN
[ https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams closed CASSANDRA-4589. --- writes are allowed when RFN Key: CASSANDRA-4589 URL: https://issues.apache.org/jira/browse/CASSANDRA-4589 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.2 Reporter: Brandon Williams Assignee: Vijay Priority: Minor Fix For: 1.2.0 Easily repro'd by running stress with a ridiculous rf: # tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456 We're supposed to allow creation of a ks where the rf exceeds the amount of nodes, but we shouldn't be able to write to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4589) writes are allowed when RFN
[
https://issues.apache.org/jira/browse/CASSANDRA-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445260#comment-13445260
]
Vijay commented on CASSANDRA-4589:
--
{quote}
that's how you add them for new DCs, with an rf of zero until you have the
nodes in place
{quote}
if we do this patch then we will start failing reads and writes (because
writeEndpoints will be actual end points for the whole cluster) when the user
is expanding the clusters to other DC's.
{quote}
CL.ONE write, when there is = 1 replica available, sounds like correct
behavior to me.
{quote}
+1
writes are allowed when RFN
Key: CASSANDRA-4589
URL: https://issues.apache.org/jira/browse/CASSANDRA-4589
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.2
Reporter: Brandon Williams
Assignee: Vijay
Priority: Minor
Fix For: 1.2.0
Easily repro'd by running stress with a ridiculous rf:
# tools/bin/cassandra-stress -n 10 -i 1 -o insert -l123456
We're supposed to allow creation of a ks where the rf exceeds the amount of
nodes, but we shouldn't be able to write to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4591) Pass -umask option to jsvc
Nick Bailey created CASSANDRA-4591: -- Summary: Pass -umask option to jsvc Key: CASSANDRA-4591 URL: https://issues.apache.org/jira/browse/CASSANDRA-4591 Project: Cassandra Issue Type: Improvement Components: Packaging Affects Versions: 1.1.4 Reporter: Nick Bailey Priority: Minor Fix For: 1.1.5, 1.2.0 Currently jsvc defaults to a very restrictive umask. This makes it hard for external tools to work with cassandra data files (snapshots). It would be useful to pass in a -umask option to jsvc with slightly less restrictive permissions (just adding group read permissions). It should just be passing 'umask 0037' (u=rwx,g=r,o=), in the debian init script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4591) Pass -umask option to jsvc
[ https://issues.apache.org/jira/browse/CASSANDRA-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445292#comment-13445292 ] Nick Bailey commented on CASSANDRA-4591: Correction, should be 0027, which gives group execute permissions, since directories can't be read without them. Pass -umask option to jsvc -- Key: CASSANDRA-4591 URL: https://issues.apache.org/jira/browse/CASSANDRA-4591 Project: Cassandra Issue Type: Improvement Components: Packaging Affects Versions: 1.1.4 Reporter: Nick Bailey Priority: Minor Fix For: 1.1.5, 1.2.0 Currently jsvc defaults to a very restrictive umask. This makes it hard for external tools to work with cassandra data files (snapshots). It would be useful to pass in a -umask option to jsvc with slightly less restrictive permissions (just adding group read permissions). It should just be passing 'umask 0037' (u=rwx,g=r,o=), in the debian init script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4592) CQL COPY TO command doesn't work with python cassandra-dbapi2
Tyler Patterson created CASSANDRA-4592:
--
Summary: CQL COPY TO command doesn't work with python
cassandra-dbapi2
Key: CASSANDRA-4592
URL: https://issues.apache.org/jira/browse/CASSANDRA-4592
Project: Cassandra
Issue Type: Bug
Environment: ubuntu, dtest, ccm cluster, cassandra 1.1.4
Reporter: Tyler Patterson
Running the COPY TO command produces the below error, though running the same
COPY TO command in cqlsh in the exact same cluster works fine:
{code}
==
ERROR: copyto_test.TestCopyTo.copy_to_test
--
Traceback (most recent call last):
File /usr/local/lib/python2.7/dist-packages/nose/case.py, line 197, in
runTest
self.test(*self.arg)
File /home/tahooie/datastax/cassandra-dtest/tools.py, line 132, in wrapped
f(obj)
File /home/tahooie/datastax/cassandra-dtest/copyto_test.py, line 33, in
copy_to_test
cursor_v2.execute(copy_from_cql)
File /usr/local/lib/python2.7/dist-packages/cql/cursor.py, line 117, in
execute
response = self.handle_cql_execution_errors(doquery, prepared_q, compress)
File /usr/local/lib/python2.7/dist-packages/cql/cursor.py, line 134, in
handle_cql_execution_errors
raise cql.ProgrammingError(Bad Request: %s % ire.why)
ProgrammingError: Bad Request: line 1:0 no viable alternative at input 'COPY'
--
Ran 1 test in 8.239s
{code}
Here is an example of the COPY TO command:
{code}
COPY airplanes TO '/tmp/tmpOCdbFt';
{code}
This is the dtest used to produce the error:
{code}
import os
import tempfile
from dtest import Tester, debug
from tools import *
class TestCopyTo(Tester):
@since('1.1')
def copy_to_test(self):
self.num_rows = 0
cluster = self.cluster
cluster.populate(2).start()
time.sleep(1)
node1 = cluster.nodelist()[0]
cursor_v2 = self.cql_connection(node1).cursor()
self.create_ks(cursor_v2, 'ks', 2)
cursor_v2.execute(
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
)
cursor_v2.execute(INSERT INTO airplanes (name, manufacturer, year,
mach) VALUES ('P38-Lightning', 'Lockheed', 1937, '.7'))
fn = tempfile.NamedTemporaryFile().name
copy_from_cql = COPY airplanes TO '%s' % fn
cursor_v2.execute(copy_from_cql)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4592) CQL COPY TO command doesn't work with python cassandra-dbapi2
[
https://issues.apache.org/jira/browse/CASSANDRA-4592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams resolved CASSANDRA-4592.
-
Resolution: Invalid
COPY is cqlsh sugar, not a cql command.
CQL COPY TO command doesn't work with python cassandra-dbapi2
-
Key: CASSANDRA-4592
URL: https://issues.apache.org/jira/browse/CASSANDRA-4592
Project: Cassandra
Issue Type: Bug
Environment: ubuntu, dtest, ccm cluster, cassandra 1.1.4
Reporter: Tyler Patterson
Running the COPY TO command produces the below error, though running the same
COPY TO command in cqlsh in the exact same cluster works fine:
{code}
==
ERROR: copyto_test.TestCopyTo.copy_to_test
--
Traceback (most recent call last):
File /usr/local/lib/python2.7/dist-packages/nose/case.py, line 197, in
runTest
self.test(*self.arg)
File /home/tahooie/datastax/cassandra-dtest/tools.py, line 132, in wrapped
f(obj)
File /home/tahooie/datastax/cassandra-dtest/copyto_test.py, line 33, in
copy_to_test
cursor_v2.execute(copy_from_cql)
File /usr/local/lib/python2.7/dist-packages/cql/cursor.py, line 117, in
execute
response = self.handle_cql_execution_errors(doquery, prepared_q, compress)
File /usr/local/lib/python2.7/dist-packages/cql/cursor.py, line 134, in
handle_cql_execution_errors
raise cql.ProgrammingError(Bad Request: %s % ire.why)
ProgrammingError: Bad Request: line 1:0 no viable alternative at input 'COPY'
--
Ran 1 test in 8.239s
{code}
Here is an example of the COPY TO command:
{code}
COPY airplanes TO '/tmp/tmpOCdbFt';
{code}
This is the dtest used to produce the error:
{code}
import os
import tempfile
from dtest import Tester, debug
from tools import *
class TestCopyTo(Tester):
@since('1.1')
def copy_to_test(self):
self.num_rows = 0
cluster = self.cluster
cluster.populate(2).start()
time.sleep(1)
node1 = cluster.nodelist()[0]
cursor_v2 = self.cql_connection(node1).cursor()
self.create_ks(cursor_v2, 'ks', 2)
cursor_v2.execute(
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
)
cursor_v2.execute(INSERT INTO airplanes (name, manufacturer, year,
mach) VALUES ('P38-Lightning', 'Lockheed', 1937, '.7'))
fn = tempfile.NamedTemporaryFile().name
copy_from_cql = COPY airplanes TO '%s' % fn
cursor_v2.execute(copy_from_cql)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4592) CQL COPY TO command doesn't work with python cassandra-dbapi2
[
https://issues.apache.org/jira/browse/CASSANDRA-4592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445319#comment-13445319
]
Tyler Patterson commented on CASSANDRA-4592:
doh!
CQL COPY TO command doesn't work with python cassandra-dbapi2
-
Key: CASSANDRA-4592
URL: https://issues.apache.org/jira/browse/CASSANDRA-4592
Project: Cassandra
Issue Type: Bug
Environment: ubuntu, dtest, ccm cluster, cassandra 1.1.4
Reporter: Tyler Patterson
Running the COPY TO command produces the below error, though running the same
COPY TO command in cqlsh in the exact same cluster works fine:
{code}
==
ERROR: copyto_test.TestCopyTo.copy_to_test
--
Traceback (most recent call last):
File /usr/local/lib/python2.7/dist-packages/nose/case.py, line 197, in
runTest
self.test(*self.arg)
File /home/tahooie/datastax/cassandra-dtest/tools.py, line 132, in wrapped
f(obj)
File /home/tahooie/datastax/cassandra-dtest/copyto_test.py, line 33, in
copy_to_test
cursor_v2.execute(copy_from_cql)
File /usr/local/lib/python2.7/dist-packages/cql/cursor.py, line 117, in
execute
response = self.handle_cql_execution_errors(doquery, prepared_q, compress)
File /usr/local/lib/python2.7/dist-packages/cql/cursor.py, line 134, in
handle_cql_execution_errors
raise cql.ProgrammingError(Bad Request: %s % ire.why)
ProgrammingError: Bad Request: line 1:0 no viable alternative at input 'COPY'
--
Ran 1 test in 8.239s
{code}
Here is an example of the COPY TO command:
{code}
COPY airplanes TO '/tmp/tmpOCdbFt';
{code}
This is the dtest used to produce the error:
{code}
import os
import tempfile
from dtest import Tester, debug
from tools import *
class TestCopyTo(Tester):
@since('1.1')
def copy_to_test(self):
self.num_rows = 0
cluster = self.cluster
cluster.populate(2).start()
time.sleep(1)
node1 = cluster.nodelist()[0]
cursor_v2 = self.cql_connection(node1).cursor()
self.create_ks(cursor_v2, 'ks', 2)
cursor_v2.execute(
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
)
cursor_v2.execute(INSERT INTO airplanes (name, manufacturer, year,
mach) VALUES ('P38-Lightning', 'Lockheed', 1937, '.7'))
fn = tempfile.NamedTemporaryFile().name
copy_from_cql = COPY airplanes TO '%s' % fn
cursor_v2.execute(copy_from_cql)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4544) fix acknowledge_by for batch_mutate
[ https://issues.apache.org/jira/browse/CASSANDRA-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445339#comment-13445339 ] Aleksey Yeschenko commented on CASSANDRA-4544: -- I think it also makes sense to differentiate between UnavailableException thrown while batchlog write and UE exception thrown after that. fix acknowledge_by for batch_mutate --- Key: CASSANDRA-4544 URL: https://issues.apache.org/jira/browse/CASSANDRA-4544 Project: Cassandra Issue Type: Sub-task Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Priority: Minor CASSANDRA-4414 added TimedOutException.acknowledged_by, but for a batch write will send back the acknowledged_by for a random row, which usually does not reflect the status for the batch as a whole. Should override this to -1 in that case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4544) fix acknowledge_by for batch_mutate
[ https://issues.apache.org/jira/browse/CASSANDRA-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445370#comment-13445370 ] Jonathan Ellis commented on CASSANDRA-4544: --- But if the batchlog write is successful, it should return TOE, not UE. UE means I stopped because I knew the replicas were down before I wrote anything. TOE means write is in progress, but isn't complete yet. Once BL write starts, write is in progress so can't return UE anymore. fix acknowledge_by for batch_mutate --- Key: CASSANDRA-4544 URL: https://issues.apache.org/jira/browse/CASSANDRA-4544 Project: Cassandra Issue Type: Sub-task Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Priority: Minor CASSANDRA-4414 added TimedOutException.acknowledged_by, but for a batch write will send back the acknowledged_by for a random row, which usually does not reflect the status for the batch as a whole. Should override this to -1 in that case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4548) Mutation response(WriteResponse.java) could be smaller and not contain keyspace and key
[ https://issues.apache.org/jira/browse/CASSANDRA-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4548: -- Reviewer: brandon.williams Priority: Minor (was: Major) Affects Version/s: (was: 1.2.0 beta 1) Fix Version/s: 1.2.0 beta 1 Assignee: Jonathan Ellis Mutation response(WriteResponse.java) could be smaller and not contain keyspace and key --- Key: CASSANDRA-4548 URL: https://issues.apache.org/jira/browse/CASSANDRA-4548 Project: Cassandra Issue Type: Improvement Components: Core Reporter: sankalp kohli Assignee: Jonathan Ellis Priority: Minor Labels: network Fix For: 1.2.0 beta 1 In the mutation response, WriteResponse.java object is send back to the co-ordinator. This object has keyspace and key in it which is not required. It is not being used at the co-ordiantor. This wastes IO specially in case of WAN links between DC. Also since response from each node in multi-DC deployments goes back to the co-ordinator in another DC makes it even worse. It also becomes worse if the the keyspace and key are of large size and the data is small. In that case, a node which is not the co-ordinator and purely receiving mutations, the outbound n/w bandwidth could be half of incoming bandwidth. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4548) Mutation response(WriteResponse.java) could be smaller and not contain keyspace and key
[ https://issues.apache.org/jira/browse/CASSANDRA-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4548: -- Attachment: 4548.txt attached. (the boolean isn't used either, we don't bother sending responses on failure, letting them time out.) Mutation response(WriteResponse.java) could be smaller and not contain keyspace and key --- Key: CASSANDRA-4548 URL: https://issues.apache.org/jira/browse/CASSANDRA-4548 Project: Cassandra Issue Type: Improvement Components: Core Reporter: sankalp kohli Assignee: Jonathan Ellis Priority: Minor Labels: network Fix For: 1.2.0 beta 1 Attachments: 4548.txt In the mutation response, WriteResponse.java object is send back to the co-ordinator. This object has keyspace and key in it which is not required. It is not being used at the co-ordiantor. This wastes IO specially in case of WAN links between DC. Also since response from each node in multi-DC deployments goes back to the co-ordinator in another DC makes it even worse. It also becomes worse if the the keyspace and key are of large size and the data is small. In that case, a node which is not the co-ordinator and purely receiving mutations, the outbound n/w bandwidth could be half of incoming bandwidth. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4537) We should emit number of sstables in each level from JMX
[ https://issues.apache.org/jira/browse/CASSANDRA-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4537: -- Reviewer: jbellis Affects Version/s: (was: 1.2.0 beta 1) 1.0.0 Fix Version/s: 1.2.1 Assignee: Yuki Morishita We should emit number of sstables in each level from JMX Key: CASSANDRA-4537 URL: https://issues.apache.org/jira/browse/CASSANDRA-4537 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Reporter: sankalp kohli Assignee: Yuki Morishita Priority: Minor Labels: compaction, leveled Fix For: 1.2.1 Original Estimate: 12h Remaining Estimate: 12h We should add methods to this Mbean org.apache.cassandra.db.ColumnFamilyStoreMBean These metrics will be helpful to see how sstables are distributed in different levels and how they move to higher level with time. Currently we can see this by looking at the json file but with JMX, we can monitor the historic values over time using any monitoring tool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Frank updated CASSANDRA-4593: - Priority: Major (was: Critical) Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
Ben Frank created CASSANDRA-4593: Summary: Reading the ByteBuffer key from a map job causes an infinite fetch loop Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Priority: Critical Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Frank updated CASSANDRA-4593: - Attachment: cassandra-1.1-4593.txt patch against the cassandra-1.1 branch Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Attachments: cassandra-1.1-4593.txt Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4516) OutboundTcpConnection could drop outgoing messages and not log it.
[ https://issues.apache.org/jira/browse/CASSANDRA-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-4516. --- Resolution: Not A Problem added to ConnectionMetrics in CASSANDRA-4009 OutboundTcpConnection could drop outgoing messages and not log it. --- Key: CASSANDRA-4516 URL: https://issues.apache.org/jira/browse/CASSANDRA-4516 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.4 Reporter: sankalp kohli Priority: Minor Since there is one connection between two nodes and all writes are handled by single thread, there is a chance that a message gets old enough and is dropped in OutboundTcpConnection. These dropped message does not get logged by MessageService. We should definitely log these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445404#comment-13445404 ] Ben Frank edited comment on CASSANDRA-4593 at 8/31/12 10:12 AM: patch against the cassandra-1.1 branch attached. This does a mark on the buffer, saves off the token value and then resets the buffer to back to the mark. Downstream users then aren't able to effect the operation of the iterator. was (Author: airlust): patch against the cassandra-1.1 branch Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Attachments: cassandra-1.1-4593.txt Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4594) COPY TO and COPY FROM don't default to consistent ordering of columns
Tyler Patterson created CASSANDRA-4594:
--
Summary: COPY TO and COPY FROM don't default to consistent
ordering of columns
Key: CASSANDRA-4594
URL: https://issues.apache.org/jira/browse/CASSANDRA-4594
Project: Cassandra
Issue Type: Bug
Environment: Happens in CQLSH 2, may or may not happen in CQLSH 3
Reporter: Tyler Patterson
Assignee: Brandon Williams
Priority: Minor
Here is the input:
{code}
CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
USE test;
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
INSERT INTO airplanes (name, manufacturer, year, mach) VALUES ('P38-Lightning',
'Lockheed', 1937, '.7');
COPY airplanes TO 'temp.cfg' WITH HEADER=true;
TRUNCATE airplanes;
COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
SELECT * FROM airplanes;
{code}
Here is what happens when executed. Note how it tried to import the float into
the int column:
{code}
cqlsh:test DROP KEYSPACE test;
cqlsh:test CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
cqlsh:test USE test;
cqlsh:test
cqlsh:test CREATE TABLE airplanes (
... name text PRIMARY KEY,
... manufacturer ascii,
... year int,
... mach float
... );
cqlsh:test
cqlsh:test INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
cqlsh:test
cqlsh:test COPY airplanes TO 'temp.cfg' WITH HEADER=true;
1 rows exported in 0.003 seconds.
cqlsh:test TRUNCATE airplanes;
cqlsh:test
cqlsh:test COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
Bad Request: unable to make int from '0.7'
Aborting import at record #0 (line 1). Previously-inserted values still present.
0 rows imported in 0.002 seconds.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-4593. --- Resolution: Invalid You're not free to destructively mutate the key buffer. Use positional reads or duplicate() it. Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Attachments: cassandra-1.1-4593.txt Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4595) Nodetool commands like scrub uses hard coded data directory path to be /var/lib/cassandra
sankalp kohli created CASSANDRA-4595: Summary: Nodetool commands like scrub uses hard coded data directory path to be /var/lib/cassandra Key: CASSANDRA-4595 URL: https://issues.apache.org/jira/browse/CASSANDRA-4595 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.4 Reporter: sankalp kohli Priority: Minor If your data directory is not /var/lib/cassandra, the scrub and other nodetool commands won't work. This should not be hard coded but rather be picked up from the yaml. We have to create sym links to get around it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4594) COPY TO and COPY FROM don't default to consistent ordering of columns
[
https://issues.apache.org/jira/browse/CASSANDRA-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-4594:
Reviewer: brandon.williams
COPY TO and COPY FROM don't default to consistent ordering of columns
-
Key: CASSANDRA-4594
URL: https://issues.apache.org/jira/browse/CASSANDRA-4594
Project: Cassandra
Issue Type: Bug
Environment: Happens in CQLSH 2, may or may not happen in CQLSH 3
Reporter: Tyler Patterson
Assignee: paul cannon
Priority: Minor
Here is the input:
{code}
CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
USE test;
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
COPY airplanes TO 'temp.cfg' WITH HEADER=true;
TRUNCATE airplanes;
COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
SELECT * FROM airplanes;
{code}
Here is what happens when executed. Note how it tried to import the float
into the int column:
{code}
cqlsh:test DROP KEYSPACE test;
cqlsh:test CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
cqlsh:test USE test;
cqlsh:test
cqlsh:test CREATE TABLE airplanes (
... name text PRIMARY KEY,
... manufacturer ascii,
... year int,
... mach float
... );
cqlsh:test
cqlsh:test INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
cqlsh:test
cqlsh:test COPY airplanes TO 'temp.cfg' WITH HEADER=true;
1 rows exported in 0.003 seconds.
cqlsh:test TRUNCATE airplanes;
cqlsh:test
cqlsh:test COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
Bad Request: unable to make int from '0.7'
Aborting import at record #0 (line 1). Previously-inserted values still
present.
0 rows imported in 0.002 seconds.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4594) COPY TO and COPY FROM don't default to consistent ordering of columns
[
https://issues.apache.org/jira/browse/CASSANDRA-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams reassigned CASSANDRA-4594:
---
Assignee: paul cannon (was: Brandon Williams)
COPY TO and COPY FROM don't default to consistent ordering of columns
-
Key: CASSANDRA-4594
URL: https://issues.apache.org/jira/browse/CASSANDRA-4594
Project: Cassandra
Issue Type: Bug
Environment: Happens in CQLSH 2, may or may not happen in CQLSH 3
Reporter: Tyler Patterson
Assignee: paul cannon
Priority: Minor
Here is the input:
{code}
CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
USE test;
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
COPY airplanes TO 'temp.cfg' WITH HEADER=true;
TRUNCATE airplanes;
COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
SELECT * FROM airplanes;
{code}
Here is what happens when executed. Note how it tried to import the float
into the int column:
{code}
cqlsh:test DROP KEYSPACE test;
cqlsh:test CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
cqlsh:test USE test;
cqlsh:test
cqlsh:test CREATE TABLE airplanes (
... name text PRIMARY KEY,
... manufacturer ascii,
... year int,
... mach float
... );
cqlsh:test
cqlsh:test INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
cqlsh:test
cqlsh:test COPY airplanes TO 'temp.cfg' WITH HEADER=true;
1 rows exported in 0.003 seconds.
cqlsh:test TRUNCATE airplanes;
cqlsh:test
cqlsh:test COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
Bad Request: unable to make int from '0.7'
Aborting import at record #0 (line 1). Previously-inserted values still
present.
0 rows imported in 0.002 seconds.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4595) Nodetool commands like scrub uses hard coded data directory path to be /var/lib/cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445421#comment-13445421 ] Jonathan Ellis commented on CASSANDRA-4595: --- Nodetool just sends JMX commands to the server, it never tries to access data directories itself. I further note that I don't see /var/lib/cassandra hardcoded anywhere in the source tree. Nodetool commands like scrub uses hard coded data directory path to be /var/lib/cassandra - Key: CASSANDRA-4595 URL: https://issues.apache.org/jira/browse/CASSANDRA-4595 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.4 Reporter: sankalp kohli Priority: Minor Labels: nodetool Original Estimate: 24h Remaining Estimate: 24h If your data directory is not /var/lib/cassandra, the scrub and other nodetool commands won't work. This should not be hard coded but rather be picked up from the yaml. We have to create sym links to get around it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4544) fix acknowledge_by for batch_mutate
[ https://issues.apache.org/jira/browse/CASSANDRA-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445423#comment-13445423 ] Aleksey Yeschenko commented on CASSANDRA-4544: -- You are right. It's just that I called responseHandler.assureSufficientLiveNodes() for each mutation before writing to the batchlog, but had another check after the batchlog write (in sendToHintedEndpoints, throws UnavaialbeException). Now they both happen before the batchlog write (for a_b_m) and the question is irrelevant. fix acknowledge_by for batch_mutate --- Key: CASSANDRA-4544 URL: https://issues.apache.org/jira/browse/CASSANDRA-4544 Project: Cassandra Issue Type: Sub-task Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Priority: Minor CASSANDRA-4414 added TimedOutException.acknowledged_by, but for a batch write will send back the acknowledged_by for a random row, which usually does not reflect the status for the batch as a whole. Should override this to -1 in that case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4548) Mutation response(WriteResponse.java) could be smaller and not contain keyspace and key
[ https://issues.apache.org/jira/browse/CASSANDRA-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445428#comment-13445428 ] Brandon Williams commented on CASSANDRA-4548: - +1 Mutation response(WriteResponse.java) could be smaller and not contain keyspace and key --- Key: CASSANDRA-4548 URL: https://issues.apache.org/jira/browse/CASSANDRA-4548 Project: Cassandra Issue Type: Improvement Components: Core Reporter: sankalp kohli Assignee: Jonathan Ellis Priority: Minor Labels: network Fix For: 1.2.0 beta 1 Attachments: 4548.txt In the mutation response, WriteResponse.java object is send back to the co-ordinator. This object has keyspace and key in it which is not required. It is not being used at the co-ordiantor. This wastes IO specially in case of WAN links between DC. Also since response from each node in multi-DC deployments goes back to the co-ordinator in another DC makes it even worse. It also becomes worse if the the keyspace and key are of large size and the data is small. In that case, a node which is not the co-ordinator and purely receiving mutations, the outbound n/w bandwidth could be half of incoming bandwidth. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445432#comment-13445432 ] Ben Frank commented on CASSANDRA-4593: -- Fair enough, I was just doing a byteBuffer.getLong() which I know resets the position but I didn't really consider it destructive. I can't believe I'll be the only person caught out by this, is there some documentation I've missed, or a relevant wiki page I should update with this information? Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Attachments: cassandra-1.1-4593.txt Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445438#comment-13445438 ] Jonathan Ellis commented on CASSANDRA-4593: --- Not really. We mostly intend CFRR to be used by Pig and Hive and not manually. Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Attachments: cassandra-1.1-4593.txt Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4596) thrift call to get_paged_slice() hangs if end token is 0
Normen Seemann created CASSANDRA-4596: - Summary: thrift call to get_paged_slice() hangs if end token is 0 Key: CASSANDRA-4596 URL: https://issues.apache.org/jira/browse/CASSANDRA-4596 Project: Cassandra Issue Type: Bug Components: API, Core, Hadoop Affects Versions: 1.1.3 Environment: linux Reporter: Normen Seemann I am using get_paged_slice() for range scans driven from within hadoop mappers. The mapper that scans the last segment with get_paged_slice() where start key is set *and* end_token is set to 0 token hangs within the thirft call. Client shows following jstack: - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Interpreted frame) - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129 (Interpreted frame) - org.apache.thrift.transport.TIOStreamTransport.read(byte[], int, int) @bci=25, line=127 (Interpreted frame) - org.apache.thrift.transport.TTransport.readAll(byte[], int, int) @bci=22, line=84 (Interpreted frame) - org.apache.thrift.transport.TFramedTransport.readFrame() @bci=10, line=129 (Interpreted frame) - org.apache.thrift.transport.TFramedTransport.read(byte[], int, int) @bci=28, line=101 (Interpreted frame) - org.apache.thrift.transport.TTransport.readAll(byte[], int, int) @bci=22, line=84 (Interpreted frame) - org.apache.thrift.protocol.TBinaryProtocol.readAll(byte[], int, int) @bci=12, line=378 (Interpreted frame) - org.apache.thrift.protocol.TBinaryProtocol.readI32() @bci=52, line=297 (Interpreted frame) - org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin() @bci=1, line=204 (Interpreted frame) - org.apache.thrift.TServiceClient.receiveBase(org.apache.thrift.TBase, java.lang.String) @bci=4, line=69 (Interpreted frame) - org.apache.cassandra.thrift.Cassandra$Client.recv_get_paged_slice() @bci=12, line=727 (Interpreted frame) - org.apache.cassandra.thrift.Cassandra$Client.get_paged_slice(java.lang.String, org.apache.cassandra.thrift.KeyRange, java.nio.ByteBuffer, org.apache.cassandra.thrift.ConsistencyLevel) @bci=10, line=711 (Interpreted frame) Changing the end_token from 0 to 2**127-1 fixes the problem, however, I would only consider this a workaround. Now, there are actually two issues: 1.) Is the call to get_paged_slice() I described supported at all? 2.) if it is not supported please fix with reasonable error instead of just hanging -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[
https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445572#comment-13445572
]
Tyler Hobbs commented on CASSANDRA-4206:
I'm not sure how useful it is at this point, but just for the record I'm seeing
what appears to be the same issue with 1.0.7 (which means it shouldn't be
CASSANDRA-3579):
{noformat}
INFO [GossipTasks:1] 2012-08-27 14:22:50,242 Gossiper.java (line 818)
InetAddress /xx.xx.178.59 is now dead.
INFO [GossipStage:1] 2012-08-27 14:22:59,090 Gossiper.java (line 804)
InetAddress /xx.xx.178.59 is now UP
INFO [HintedHandoff:1] 2012-08-27 14:23:41,548 HintedHandOffManager.java (line
296) Started hinted handoff for token: 132332031580364958013534569556798748899
with IP: /xx.xx.178.59
INFO [HintedHandoff:1] 2012-08-27 14:23:41,870 ColumnFamilyStore.java (line
704) Enqueuing flush of Memtable-HintsColumnFamily@2081164539(597050/47764000
serialized/live bytes, 857 ops)
INFO [FlushWriter:181] 2012-08-27 14:23:41,870 Memtable.java (line 246)
Writing Memtable-HintsColumnFamily@2081164539(597050/47764000 serialized/live
bytes, 857 ops)
INFO [FlushWriter:181] 2012-08-27 14:23:41,959 Memtable.java (line 283)
Completed flushing
/xx/xx/xx/cassandra/datafile/system/HintsColumnFamily-hc-6730-Data.db (624946
bytes)
INFO [CompactionExecutor:884] 2012-08-27 14:23:41,961 CompactionTask.java
(line 113) Compacting
[SSTableReader(path='/xx/xx/xx/cassandra/datafile/system/HintsColumnFamily-hc-6729-Data.db'),
SSTableReader(path='/ngs/app/xcardp/cassandra/datafile/system/HintsColumnFamily-hc-6730-Data.db')]
INFO [CompactionExecutor:884] 2012-08-27 14:23:41,987
CompactionController.java (line 133) Compacting large row
system/HintsColumnFamily:31372e33342e3137382e3534 (274816343 bytes)
incrementally
ERROR [CompactionExecutor:884] 2012-08-27 14:23:56,322
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[CompactionExecutor:884,1,main]
java.lang.AssertionError: originally calculated column size of 197713629 but
now it is 197711561
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at
org.apache.cassandra.db.compaction.CompactionManager$6.call(CompactionManager.java:275)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [HintedHandoff:1] 2012-08-27 14:23:56,323 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of 197713629 but
now it is 197711561
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:369)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:248)
at
org.apache.cassandra.db.HintedHandOffManager.access$200(HintedHandOffManager.java:84)
at
org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:416)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError:
originally calculated column size of 197713629 but now it is 197711561
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:365)
... 7 more
Caused by: java.lang.AssertionError: originally calculated column size of
197713629 but now it is 197711561
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at
org.apache.cassandra.db.compaction.CompactionManager$6.call(CompactionManager.java:275)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
ERROR [HintedHandoff:1] 2012-08-27 14:23:56,345
[jira] [Commented] (CASSANDRA-4594) COPY TO and COPY FROM don't default to consistent ordering of columns
[
https://issues.apache.org/jira/browse/CASSANDRA-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445584#comment-13445584
]
paul cannon commented on CASSANDRA-4594:
This is because a {{select * from airplanes;}} does not give the columns in the
order they were defined. I'm not sure why not; if that's a bug in C*, then we
should fix that. If there isn't supposed to be any expectation of order, then
cqlsh should be inspecting the columns and specifying them explicitly.
COPY TO and COPY FROM don't default to consistent ordering of columns
-
Key: CASSANDRA-4594
URL: https://issues.apache.org/jira/browse/CASSANDRA-4594
Project: Cassandra
Issue Type: Bug
Environment: Happens in CQLSH 2, may or may not happen in CQLSH 3
Reporter: Tyler Patterson
Assignee: paul cannon
Priority: Minor
Here is the input:
{code}
CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
USE test;
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
COPY airplanes TO 'temp.cfg' WITH HEADER=true;
TRUNCATE airplanes;
COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
SELECT * FROM airplanes;
{code}
Here is what happens when executed. Note how it tried to import the float
into the int column:
{code}
cqlsh:test DROP KEYSPACE test;
cqlsh:test CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
cqlsh:test USE test;
cqlsh:test
cqlsh:test CREATE TABLE airplanes (
... name text PRIMARY KEY,
... manufacturer ascii,
... year int,
... mach float
... );
cqlsh:test
cqlsh:test INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
cqlsh:test
cqlsh:test COPY airplanes TO 'temp.cfg' WITH HEADER=true;
1 rows exported in 0.003 seconds.
cqlsh:test TRUNCATE airplanes;
cqlsh:test
cqlsh:test COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
Bad Request: unable to make int from '0.7'
Aborting import at record #0 (line 1). Previously-inserted values still
present.
0 rows imported in 0.002 seconds.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4594) COPY TO and COPY FROM don't default to consistent ordering of columns
[
https://issues.apache.org/jira/browse/CASSANDRA-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4594:
--
Fix Version/s: 1.2.0
COPY TO and COPY FROM don't default to consistent ordering of columns
-
Key: CASSANDRA-4594
URL: https://issues.apache.org/jira/browse/CASSANDRA-4594
Project: Cassandra
Issue Type: Bug
Environment: Happens in CQLSH 2, may or may not happen in CQLSH 3
Reporter: Tyler Patterson
Assignee: paul cannon
Priority: Minor
Fix For: 1.2.0
Here is the input:
{code}
CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
USE test;
CREATE TABLE airplanes (
name text PRIMARY KEY,
manufacturer ascii,
year int,
mach float
);
INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
COPY airplanes TO 'temp.cfg' WITH HEADER=true;
TRUNCATE airplanes;
COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
SELECT * FROM airplanes;
{code}
Here is what happens when executed. Note how it tried to import the float
into the int column:
{code}
cqlsh:test DROP KEYSPACE test;
cqlsh:test CREATE KEYSPACE test WITH strategy_class = 'SimpleStrategy' AND
strategy_options:replication_factor = 1;
cqlsh:test USE test;
cqlsh:test
cqlsh:test CREATE TABLE airplanes (
... name text PRIMARY KEY,
... manufacturer ascii,
... year int,
... mach float
... );
cqlsh:test
cqlsh:test INSERT INTO airplanes (name, manufacturer, year, mach) VALUES
('P38-Lightning', 'Lockheed', 1937, '.7');
cqlsh:test
cqlsh:test COPY airplanes TO 'temp.cfg' WITH HEADER=true;
1 rows exported in 0.003 seconds.
cqlsh:test TRUNCATE airplanes;
cqlsh:test
cqlsh:test COPY airplanes FROM 'temp.cfg' WITH HEADER=true;
Bad Request: unable to make int from '0.7'
Aborting import at record #0 (line 1). Previously-inserted values still
present.
0 rows imported in 0.002 seconds.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4593) Reading the ByteBuffer key from a map job causes an infinite fetch loop
[ https://issues.apache.org/jira/browse/CASSANDRA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445627#comment-13445627 ] Jeremy Hanna commented on CASSANDRA-4593: - It may be worth adding to the MapReduce or Troubleshooting section of http://wiki.apache.org/cassandra/HadoopSupport. We were bitten by something like this at a previous job and it was hard to track down. Reading the ByteBuffer key from a map job causes an infinite fetch loop --- Key: CASSANDRA-4593 URL: https://issues.apache.org/jira/browse/CASSANDRA-4593 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.1.2 Reporter: Ben Frank Attachments: cassandra-1.1-4593.txt Reading the ByteBuffer key from a map job empties the buffer. One of these key buffers is later used in ColumnFamilyRecordReader to figure out the last token that was received, then using that as a start point to fetch more rows. With a now empty buffer, the token defaults to the start of the range and thus the end of the data is never reached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
