[jira] [Updated] (CASSANDRA-8320) 2.1.2: NullPointerException in SSTableWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-8320: --- Attachment: 8320.patch attaching patch to properly handle abort had to add a 'sharesBfWith' field to SSTR, since we have not yet replaced the file, but we do share the bloom filter, otherwise we will close the bf when removing the tmplink file and fail removing the finished file 2.1.2: NullPointerException in SSTableWriter Key: CASSANDRA-8320 URL: https://issues.apache.org/jira/browse/CASSANDRA-8320 Project: Cassandra Issue Type: Bug Reporter: Evgeny Pasynkov Assignee: Marcus Eriksson Fix For: 2.1.3 Attachments: 8320.patch After upgrading to 2.1.2, I've got tons of these exception in log: java.lang.NullPointerException: null at org.apache.cassandra.io.sstable.SSTableWriter.openEarly(SSTableWriter.java:381) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:295) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:186) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[cassandra-all-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) ~[cassandra-all-2.1.2.jar:2.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60] This error is prepended by the following problems: 12:59:59.632 [NonPeriodicTasks:1] ERROR o.a.c.io.sstable.SSTableDeletingTask - Unable to delete E:\Upsource_11959\data\cassandra\data\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-4-Data.db (it will be removed on server restart; we'll also retry after GC) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[
https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222799#comment-14222799
]
Robert Stupp commented on CASSANDRA-7438:
-
rehashing: growing (x2) is already implemented, shrinking (/2) shouldn't be a
big issue, too. The implementation only locks the currently processed
partitions during rehash.
put operation: fixed (was definitely a bug), cleanup is running concurrently
and trigger on out of memory condition
block sizes: will give it a try (fixed vs. different sizes vs. variable sized
(no blocks))
per-partition locks: already thought about it - not sure whether it's worth the
additional RW-lock overhead since partition lock time is very low during normal
operation
metrics: some (very basic) metrics are already in it - will add some more timer
metrics (configurable)
[~vijay2...@yahoo.com] can you catch {{OutOfMemoryError}} for Unsafe.allocate()
? It should not go up the whole call stack.
Serializing Row cache alternative (Fully off heap)
--
Key: CASSANDRA-7438
URL: https://issues.apache.org/jira/browse/CASSANDRA-7438
Project: Cassandra
Issue Type: Improvement
Components: Core
Environment: Linux
Reporter: Vijay
Assignee: Vijay
Labels: performance
Fix For: 3.0
Attachments: 0001-CASSANDRA-7438.patch
Currently SerializingCache is partially off heap, keys are still stored in
JVM heap as BB,
* There is a higher GC costs for a reasonably big cache.
* Some users have used the row cache efficiently in production for better
results, but this requires careful tunning.
* Overhead in Memory for the cache entries are relatively high.
So the proposal for this ticket is to move the LRU cache logic completely off
heap and use JNI to interact with cache. We might want to ensure that the new
implementation match the existing API's (ICache), and the implementation
needs to have safe memory access, low overhead in memory and less memcpy's
(As much as possible).
We might also want to make this cache configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[
https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222799#comment-14222799
]
Robert Stupp edited comment on CASSANDRA-7438 at 11/24/14 8:58 AM:
---
rehashing: growing (x2) is already implemented, shrinking (/2) shouldn't be a
big issue, too. The implementation only locks the currently processed
partitions during rehash.
put operation: fixed (was definitely a bug), cleanup is running concurrently
and trigger on out of memory condition
block sizes: will give it a try (fixed vs. different sizes vs. variable sized
(no blocks))
per-partition locks: already thought about it - not sure whether it's worth the
additional RW-lock overhead since partition lock time is very low during normal
operation
metrics: some (very basic) metrics are already in it - will add some more timer
metrics (configurable)
[~vijay2...@yahoo.com] can you catch {{OutOfMemoryError}} for Unsafe.allocate()
? It should not go up the whole call stack as is to prevent C* handling that as
Java heap full.
was (Author: snazy):
rehashing: growing (x2) is already implemented, shrinking (/2) shouldn't be a
big issue, too. The implementation only locks the currently processed
partitions during rehash.
put operation: fixed (was definitely a bug), cleanup is running concurrently
and trigger on out of memory condition
block sizes: will give it a try (fixed vs. different sizes vs. variable sized
(no blocks))
per-partition locks: already thought about it - not sure whether it's worth the
additional RW-lock overhead since partition lock time is very low during normal
operation
metrics: some (very basic) metrics are already in it - will add some more timer
metrics (configurable)
[~vijay2...@yahoo.com] can you catch {{OutOfMemoryError}} for Unsafe.allocate()
? It should not go up the whole call stack.
Serializing Row cache alternative (Fully off heap)
--
Key: CASSANDRA-7438
URL: https://issues.apache.org/jira/browse/CASSANDRA-7438
Project: Cassandra
Issue Type: Improvement
Components: Core
Environment: Linux
Reporter: Vijay
Assignee: Vijay
Labels: performance
Fix For: 3.0
Attachments: 0001-CASSANDRA-7438.patch
Currently SerializingCache is partially off heap, keys are still stored in
JVM heap as BB,
* There is a higher GC costs for a reasonably big cache.
* Some users have used the row cache efficiently in production for better
results, but this requires careful tunning.
* Overhead in Memory for the cache entries are relatively high.
So the proposal for this ticket is to move the LRU cache logic completely off
heap and use JNI to interact with cache. We might want to ensure that the new
implementation match the existing API's (ICache), and the implementation
needs to have safe memory access, low overhead in memory and less memcpy's
(As much as possible).
We might also want to make this cache configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222802#comment-14222802 ] Pavel Yaskevich commented on CASSANDRA-7438: bq. per-partition locks: already thought about it - not sure whether it's worth the additional RW-lock overhead since partition lock time is very low during normal operation It depends on the operation mode if there are e.g. 75% reads and 25% writes it makes more sense to use locks, because RW lock is going to be optimized by JVM to CAS operation when there is no contention, anyhow it's a valid test to do with different modes to check CAS vs. RW. Serializing Row cache alternative (Fully off heap) -- Key: CASSANDRA-7438 URL: https://issues.apache.org/jira/browse/CASSANDRA-7438 Project: Cassandra Issue Type: Improvement Components: Core Environment: Linux Reporter: Vijay Assignee: Vijay Labels: performance Fix For: 3.0 Attachments: 0001-CASSANDRA-7438.patch Currently SerializingCache is partially off heap, keys are still stored in JVM heap as BB, * There is a higher GC costs for a reasonably big cache. * Some users have used the row cache efficiently in production for better results, but this requires careful tunning. * Overhead in Memory for the cache entries are relatively high. So the proposal for this ticket is to move the LRU cache logic completely off heap and use JNI to interact with cache. We might want to ensure that the new implementation match the existing API's (ICache), and the implementation needs to have safe memory access, low overhead in memory and less memcpy's (As much as possible). We might also want to make this cache configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8053) Support for user defined aggregate functions
[
https://issues.apache.org/jira/browse/CASSANDRA-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222822#comment-14222822
]
Sylvain Lebresne commented on CASSANDRA-8053:
-
bq. But we cannot remove STYPE at all - this would not work if there are
overloads of SFUNC with similar argument type
Good point. I guess that's why Postgres has it.
bq. Propose to make STYPE optional and error out, if SFUNC is ambigious
Giving the point above, I'd rather then just stick to Postgres syntax and keep
it mandatory for now. Making it optional would be kind of messy if method
overloads of the state transition function are added after the aggregate has
been defined.
Support for user defined aggregate functions
Key: CASSANDRA-8053
URL: https://issues.apache.org/jira/browse/CASSANDRA-8053
Project: Cassandra
Issue Type: New Feature
Reporter: Robert Stupp
Assignee: Robert Stupp
Labels: cql, udf
Fix For: 3.0
Attachments: 8053v1.txt
CASSANDRA-4914 introduces aggregate functions.
This ticket is about to decide how we can support user defined aggregate
functions. UD aggregate functions should be supported for all UDF flavors
(class, java, jsr223).
Things to consider:
* Special implementations for each scripting language should be omitted
* No exposure of internal APIs (e.g. {{AggregateFunction}} interface)
* No need for users to deal with serializers / codecs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8366) Repair grows data on nodes, causes load to become unbalanced
Jan Karlsson created CASSANDRA-8366: --- Summary: Repair grows data on nodes, causes load to become unbalanced Key: CASSANDRA-8366 URL: https://issues.apache.org/jira/browse/CASSANDRA-8366 Project: Cassandra Issue Type: Bug Environment: 4 node cluster 2.1.2 Cassandra Inserts and reads are done with CQL driver Reporter: Jan Karlsson There seems to be something weird going on when repairing data. I have a program that runs 2 hours which inserts 250 random numbers and reads 250 times per second. It creates 2 keyspaces with SimpleStrategy and RF of 3. I use size-tiered compaction for my cluster. After those 2 hours I run a repair and the load of all nodes goes up. If I run incremental repair the load goes up alot more. I saw the load shoot up 8 times the original size multiple times with incremental repair. (from 2G to 16G) with node 9 8 7 and 7 the repro procedure looked like this: (Note that running full repair first is not a requirement to reproduce.) After 2 hours of 250 reads + 250 writes per second: UN 9 583.39 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 584.01 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 583.72 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 583.84 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Repair -pr -par on all nodes sequentially UN 9 746.29 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 751.02 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 748.89 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.34 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 2.41 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.53 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.6 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 2.17 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 after rolling restart UN 9 1.47 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 1.5 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.19 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 compact all nodes sequentially UN 9 989.99 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 994.75 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 1.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.82 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 1.98 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.3 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 3.71 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 restart once more UN 9 2 GB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.05 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 4.1 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Is there something im missing or is this strange behavior? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/528cc3dd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/528cc3dd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/528cc3dd Branch: refs/heads/trunk Commit: 528cc3dd19e407271a0859395dbad3d3773b8ffe Parents: be0b451 6ae1b42 Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon Nov 24 10:33:32 2014 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Mon Nov 24 10:33:32 2014 +0100 -- lib/jamm-0.3.0.jar | Bin 21149 - 21033 bytes 1 file changed, 0 insertions(+), 0 deletions(-) --
cassandra git commit: Better jamm 0.3.0 jar
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 f02d19451 - 6ae1b420a Better jamm 0.3.0 jar Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6ae1b420 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6ae1b420 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6ae1b420 Branch: refs/heads/cassandra-2.1 Commit: 6ae1b420a68b15ce206ed3756d56adc94441c059 Parents: f02d194 Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon Nov 24 10:32:46 2014 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Mon Nov 24 10:32:46 2014 +0100 -- lib/jamm-0.3.0.jar | Bin 21149 - 21033 bytes 1 file changed, 0 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6ae1b420/lib/jamm-0.3.0.jar -- diff --git a/lib/jamm-0.3.0.jar b/lib/jamm-0.3.0.jar index 782f00c..f770592 100644 Binary files a/lib/jamm-0.3.0.jar and b/lib/jamm-0.3.0.jar differ
[1/2] cassandra git commit: Better jamm 0.3.0 jar
Repository: cassandra Updated Branches: refs/heads/trunk be0b451b7 - 528cc3dd1 Better jamm 0.3.0 jar Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6ae1b420 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6ae1b420 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6ae1b420 Branch: refs/heads/trunk Commit: 6ae1b420a68b15ce206ed3756d56adc94441c059 Parents: f02d194 Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon Nov 24 10:32:46 2014 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Mon Nov 24 10:32:46 2014 +0100 -- lib/jamm-0.3.0.jar | Bin 21149 - 21033 bytes 1 file changed, 0 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6ae1b420/lib/jamm-0.3.0.jar -- diff --git a/lib/jamm-0.3.0.jar b/lib/jamm-0.3.0.jar index 782f00c..f770592 100644 Binary files a/lib/jamm-0.3.0.jar and b/lib/jamm-0.3.0.jar differ
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222825#comment-14222825 ] Sylvain Lebresne commented on CASSANDRA-8231: - Seems that the jar pushed had some windows end of line in the manifest (or something like that). Pushed a new version of the jar (the one from maven actually) which appears to fix it. Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Make LCS split compaction results over many directories
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 025b4060c - 2ce1ad8e6
Make LCS split compaction results over many directories
Patch by marcuse; reviewed by yukim for CASSANDRA-8329
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2ce1ad8e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2ce1ad8e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2ce1ad8e
Branch: refs/heads/cassandra-2.0
Commit: 2ce1ad8e6f5d3c5cf781e1ff87cda4f61c89d9ee
Parents: 025b406
Author: Marcus Eriksson marc...@apache.org
Authored: Tue Nov 18 11:01:17 2014 +0100
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Nov 24 09:43:47 2014 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/Memtable.java | 20 --
.../cassandra/db/compaction/CompactionTask.java | 74
.../cassandra/io/util/DiskAwareRunnable.java| 37 +++---
4 files changed, 69 insertions(+), 63 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 01ea887..6a5ac0d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
* Fix InvalidRequestException with ORDER BY (CASSANDRA-8286)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index 0b186dc..425b352 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -336,13 +336,23 @@ public class Memtable
return estimatedSize;
}
-protected void runWith(File sstableDirectory) throws Exception
+protected void runMayThrow() throws Exception
{
+long writeSize = getExpectedWriteSize();
+Directories.DataDirectory dataDirectory =
getWriteDirectory(writeSize);
+File sstableDirectory =
cfs.directories.getLocationForDisk(dataDirectory);
assert sstableDirectory != null : Flush task is not bound to any
disk;
-
-SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
-cfs.replaceFlushed(Memtable.this, sstable);
-latch.countDown();
+try
+{
+SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
+cfs.replaceFlushed(Memtable.this, sstable);
+latch.countDown();
+}
+finally
+{
+if (dataDirectory != null)
+returnWriteDirectory(dataDirectory, writeSize);
+}
}
protected Directories getDirectories()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index 5ef4aad..08fe81a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@ -87,11 +87,11 @@ public class CompactionTask extends AbstractCompactionTask
* which are properly serialized.
* Caller is in charge of marking/unmarking the sstables as compacting.
*/
-protected void runWith(File sstableDirectory) throws Exception
+protected void runMayThrow() throws Exception
{
// The collection of sstables passed may be empty (but not null); even
if
// it is not empty, it may compact down to nothing if all rows are
deleted.
-assert sstables != null sstableDirectory != null;
+assert sstables != null;
// Note that the current compaction strategy, is not necessarily the
one this task was created under.
// This should be harmless; see comments to
CFS.maybeReloadCompactionStrategy.
@@ -149,45 +149,60 @@ public class CompactionTask extends AbstractCompactionTask
return;
}
-SSTableWriter writer = createCompactionWriter(sstableDirectory,
keysPerSSTable);
+long writeSize = getExpectedWriteSize() / estimatedSSTables;
+Directories.DataDirectory dataDirectory =
[1/3] cassandra git commit: Make LCS split compaction results over many directories
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 6ae1b420a - 35f173a0e
Make LCS split compaction results over many directories
Patch by marcuse; reviewed by yukim for CASSANDRA-8329
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2ce1ad8e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2ce1ad8e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2ce1ad8e
Branch: refs/heads/cassandra-2.1
Commit: 2ce1ad8e6f5d3c5cf781e1ff87cda4f61c89d9ee
Parents: 025b406
Author: Marcus Eriksson marc...@apache.org
Authored: Tue Nov 18 11:01:17 2014 +0100
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Nov 24 09:43:47 2014 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/Memtable.java | 20 --
.../cassandra/db/compaction/CompactionTask.java | 74
.../cassandra/io/util/DiskAwareRunnable.java| 37 +++---
4 files changed, 69 insertions(+), 63 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 01ea887..6a5ac0d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
* Fix InvalidRequestException with ORDER BY (CASSANDRA-8286)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index 0b186dc..425b352 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -336,13 +336,23 @@ public class Memtable
return estimatedSize;
}
-protected void runWith(File sstableDirectory) throws Exception
+protected void runMayThrow() throws Exception
{
+long writeSize = getExpectedWriteSize();
+Directories.DataDirectory dataDirectory =
getWriteDirectory(writeSize);
+File sstableDirectory =
cfs.directories.getLocationForDisk(dataDirectory);
assert sstableDirectory != null : Flush task is not bound to any
disk;
-
-SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
-cfs.replaceFlushed(Memtable.this, sstable);
-latch.countDown();
+try
+{
+SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
+cfs.replaceFlushed(Memtable.this, sstable);
+latch.countDown();
+}
+finally
+{
+if (dataDirectory != null)
+returnWriteDirectory(dataDirectory, writeSize);
+}
}
protected Directories getDirectories()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index 5ef4aad..08fe81a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@ -87,11 +87,11 @@ public class CompactionTask extends AbstractCompactionTask
* which are properly serialized.
* Caller is in charge of marking/unmarking the sstables as compacting.
*/
-protected void runWith(File sstableDirectory) throws Exception
+protected void runMayThrow() throws Exception
{
// The collection of sstables passed may be empty (but not null); even
if
// it is not empty, it may compact down to nothing if all rows are
deleted.
-assert sstables != null sstableDirectory != null;
+assert sstables != null;
// Note that the current compaction strategy, is not necessarily the
one this task was created under.
// This should be harmless; see comments to
CFS.maybeReloadCompactionStrategy.
@@ -149,45 +149,60 @@ public class CompactionTask extends AbstractCompactionTask
return;
}
-SSTableWriter writer = createCompactionWriter(sstableDirectory,
keysPerSSTable);
+long writeSize = getExpectedWriteSize() / estimatedSSTables;
+Directories.DataDirectory dataDirectory =
[2/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/db/Memtable.java
src/java/org/apache/cassandra/db/compaction/CompactionTask.java
src/java/org/apache/cassandra/io/util/DiskAwareRunnable.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d01c365
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d01c365
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d01c365
Branch: refs/heads/cassandra-2.1
Commit: 0d01c36599a7721a864780a3a10e134fdfa6797a
Parents: f02d194 2ce1ad8
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Nov 24 10:27:50 2014 +0100
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Nov 24 10:27:50 2014 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/Memtable.java | 6 +++--
.../cassandra/db/compaction/CompactionTask.java | 10 -
.../cassandra/io/util/DiskAwareRunnable.java| 23 ++--
4 files changed, 12 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d01c365/CHANGES.txt
--
diff --cc CHANGES.txt
index 96da1bd,6a5ac0d..313000a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,5 +1,19 @@@
-2.0.12:
+2.1.3
+ * Fix high size calculations for prepared statements (CASSANDRA-8231)
+ * Centralize shared executors (CASSANDRA-8055)
+ * Fix filtering for CONTAINS (KEY) relations on frozen collection
+ clustering columns when the query is restricted to a single
+ partition (CASSANDRA-8203)
+ * Do more aggressive entire-sstable TTL expiry checks (CASSANDRA-8243)
+ * Add more log info if readMeter is null (CASSANDRA-8238)
+ * add check of the system wall clock time at startup (CASSANDRA-8305)
+ * Support for frozen collections (CASSANDRA-7859)
+ * Fix overflow on histogram computation (CASSANDRA-8028)
+ * Have paxos reuse the timestamp generation of normal queries
(CASSANDRA-7801)
+ * Fix incremental repair not remove parent session on remote (CASSANDRA-8291)
+ * Improve JBOD disk utilization (CASSANDRA-7386)
+Merged from 2.0:
+ * Make LCS split compaction results over all data directories
(CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
* Fix InvalidRequestException with ORDER BY (CASSANDRA-8286)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d01c365/src/java/org/apache/cassandra/db/Memtable.java
--
diff --cc src/java/org/apache/cassandra/db/Memtable.java
index ba3864f,425b352..3ae5da4
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@@ -306,12 -336,23 +306,14 @@@ public class Memtabl
return estimatedSize;
}
- protected void runWith(File sstableDirectory) throws Exception
+ protected void runMayThrow() throws Exception
{
+ long writeSize = getExpectedWriteSize();
+ Directories.DataDirectory dataDirectory =
getWriteDirectory(writeSize);
+ File sstableDirectory =
cfs.directories.getLocationForDisk(dataDirectory);
assert sstableDirectory != null : Flush task is not bound to any
disk;
-
-try
-{
-SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
-cfs.replaceFlushed(Memtable.this, sstable);
-latch.countDown();
-}
-finally
-{
-if (dataDirectory != null)
-returnWriteDirectory(dataDirectory, writeSize);
-}
+SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
+cfs.replaceFlushed(Memtable.this, sstable);
}
protected Directories getDirectories()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d01c365/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index b442482,08fe81a..0e8900d
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@@ -106,11 -91,8 +106,11 @@@ public class CompactionTask extends Abs
{
// The collection of sstables passed may be empty (but not null);
even if
// it is not empty, it may compact down to nothing if all rows are
deleted.
- assert sstables != null sstableDirectory != null;
+ assert
[3/4] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Conflicts:
src/java/org/apache/cassandra/db/compaction/CompactionTask.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cd4f729e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cd4f729e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cd4f729e
Branch: refs/heads/trunk
Commit: cd4f729e9d2328564a77ac7bc52a392edd3d2b82
Parents: be0b451 0d01c36
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Nov 24 10:37:13 2014 +0100
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Nov 24 10:37:13 2014 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/Memtable.java | 6 +++--
.../cassandra/db/compaction/CompactionTask.java | 9
.../cassandra/io/util/DiskAwareRunnable.java| 23 ++--
4 files changed, 11 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4f729e/CHANGES.txt
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4f729e/src/java/org/apache/cassandra/db/Memtable.java
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4f729e/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index 808626b,0e8900d..1abb4ee
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@@ -181,7 -173,7 +181,7 @@@ public class CompactionTask extends Abs
return;
}
-
writer.switchWriter(createCompactionWriter(sstableDirectory, keysPerSSTable,
minRepairedAt, sstableFormat));
-
writer.switchWriter(createCompactionWriter(cfs.directories.getLocationForDisk(getWriteDirectory(estimatedTotalKeys/estimatedSSTables)),
keysPerSSTable, minRepairedAt));
++
writer.switchWriter(createCompactionWriter(cfs.directories.getLocationForDisk(getWriteDirectory(estimatedTotalKeys/estimatedSSTables)),
keysPerSSTable, minRepairedAt, sstableFormat));
while (iter.hasNext())
{
if (ci.isStopRequested())
@@@ -193,7 -185,7 +193,7 @@@
totalKeysWritten++;
if
(newSSTableSegmentThresholdReached(writer.currentWriter()))
{
-
writer.switchWriter(createCompactionWriter(sstableDirectory, keysPerSSTable,
minRepairedAt, sstableFormat));
-
writer.switchWriter(createCompactionWriter(cfs.directories.getLocationForDisk(getWriteDirectory(estimatedTotalKeys/estimatedSSTables)),
keysPerSSTable, minRepairedAt));
++
writer.switchWriter(createCompactionWriter(cfs.directories.getLocationForDisk(getWriteDirectory(estimatedTotalKeys/estimatedSSTables)),
keysPerSSTable, minRepairedAt, sstableFormat));
}
}
[2/4] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/db/Memtable.java
src/java/org/apache/cassandra/db/compaction/CompactionTask.java
src/java/org/apache/cassandra/io/util/DiskAwareRunnable.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d01c365
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d01c365
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d01c365
Branch: refs/heads/trunk
Commit: 0d01c36599a7721a864780a3a10e134fdfa6797a
Parents: f02d194 2ce1ad8
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Nov 24 10:27:50 2014 +0100
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Nov 24 10:27:50 2014 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/Memtable.java | 6 +++--
.../cassandra/db/compaction/CompactionTask.java | 10 -
.../cassandra/io/util/DiskAwareRunnable.java| 23 ++--
4 files changed, 12 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d01c365/CHANGES.txt
--
diff --cc CHANGES.txt
index 96da1bd,6a5ac0d..313000a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,5 +1,19 @@@
-2.0.12:
+2.1.3
+ * Fix high size calculations for prepared statements (CASSANDRA-8231)
+ * Centralize shared executors (CASSANDRA-8055)
+ * Fix filtering for CONTAINS (KEY) relations on frozen collection
+ clustering columns when the query is restricted to a single
+ partition (CASSANDRA-8203)
+ * Do more aggressive entire-sstable TTL expiry checks (CASSANDRA-8243)
+ * Add more log info if readMeter is null (CASSANDRA-8238)
+ * add check of the system wall clock time at startup (CASSANDRA-8305)
+ * Support for frozen collections (CASSANDRA-7859)
+ * Fix overflow on histogram computation (CASSANDRA-8028)
+ * Have paxos reuse the timestamp generation of normal queries
(CASSANDRA-7801)
+ * Fix incremental repair not remove parent session on remote (CASSANDRA-8291)
+ * Improve JBOD disk utilization (CASSANDRA-7386)
+Merged from 2.0:
+ * Make LCS split compaction results over all data directories
(CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
* Fix InvalidRequestException with ORDER BY (CASSANDRA-8286)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d01c365/src/java/org/apache/cassandra/db/Memtable.java
--
diff --cc src/java/org/apache/cassandra/db/Memtable.java
index ba3864f,425b352..3ae5da4
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@@ -306,12 -336,23 +306,14 @@@ public class Memtabl
return estimatedSize;
}
- protected void runWith(File sstableDirectory) throws Exception
+ protected void runMayThrow() throws Exception
{
+ long writeSize = getExpectedWriteSize();
+ Directories.DataDirectory dataDirectory =
getWriteDirectory(writeSize);
+ File sstableDirectory =
cfs.directories.getLocationForDisk(dataDirectory);
assert sstableDirectory != null : Flush task is not bound to any
disk;
-
-try
-{
-SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
-cfs.replaceFlushed(Memtable.this, sstable);
-latch.countDown();
-}
-finally
-{
-if (dataDirectory != null)
-returnWriteDirectory(dataDirectory, writeSize);
-}
+SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
+cfs.replaceFlushed(Memtable.this, sstable);
}
protected Directories getDirectories()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d01c365/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --cc src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index b442482,08fe81a..0e8900d
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@@ -106,11 -91,8 +106,11 @@@ public class CompactionTask extends Abs
{
// The collection of sstables passed may be empty (but not null);
even if
// it is not empty, it may compact down to nothing if all rows are
deleted.
- assert sstables != null sstableDirectory != null;
+ assert sstables
[1/4] cassandra git commit: Make LCS split compaction results over many directories
Repository: cassandra
Updated Branches:
refs/heads/trunk 528cc3dd1 - 065aeeb4a
Make LCS split compaction results over many directories
Patch by marcuse; reviewed by yukim for CASSANDRA-8329
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2ce1ad8e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2ce1ad8e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2ce1ad8e
Branch: refs/heads/trunk
Commit: 2ce1ad8e6f5d3c5cf781e1ff87cda4f61c89d9ee
Parents: 025b406
Author: Marcus Eriksson marc...@apache.org
Authored: Tue Nov 18 11:01:17 2014 +0100
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Nov 24 09:43:47 2014 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/Memtable.java | 20 --
.../cassandra/db/compaction/CompactionTask.java | 74
.../cassandra/io/util/DiskAwareRunnable.java| 37 +++---
4 files changed, 69 insertions(+), 63 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 01ea887..6a5ac0d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
* Fix InvalidRequestException with ORDER BY (CASSANDRA-8286)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index 0b186dc..425b352 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -336,13 +336,23 @@ public class Memtable
return estimatedSize;
}
-protected void runWith(File sstableDirectory) throws Exception
+protected void runMayThrow() throws Exception
{
+long writeSize = getExpectedWriteSize();
+Directories.DataDirectory dataDirectory =
getWriteDirectory(writeSize);
+File sstableDirectory =
cfs.directories.getLocationForDisk(dataDirectory);
assert sstableDirectory != null : Flush task is not bound to any
disk;
-
-SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
-cfs.replaceFlushed(Memtable.this, sstable);
-latch.countDown();
+try
+{
+SSTableReader sstable = writeSortedContents(context,
sstableDirectory);
+cfs.replaceFlushed(Memtable.this, sstable);
+latch.countDown();
+}
+finally
+{
+if (dataDirectory != null)
+returnWriteDirectory(dataDirectory, writeSize);
+}
}
protected Directories getDirectories()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ce1ad8e/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index 5ef4aad..08fe81a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@ -87,11 +87,11 @@ public class CompactionTask extends AbstractCompactionTask
* which are properly serialized.
* Caller is in charge of marking/unmarking the sstables as compacting.
*/
-protected void runWith(File sstableDirectory) throws Exception
+protected void runMayThrow() throws Exception
{
// The collection of sstables passed may be empty (but not null); even
if
// it is not empty, it may compact down to nothing if all rows are
deleted.
-assert sstables != null sstableDirectory != null;
+assert sstables != null;
// Note that the current compaction strategy, is not necessarily the
one this task was created under.
// This should be harmless; see comments to
CFS.maybeReloadCompactionStrategy.
@@ -149,45 +149,60 @@ public class CompactionTask extends AbstractCompactionTask
return;
}
-SSTableWriter writer = createCompactionWriter(sstableDirectory,
keysPerSSTable);
+long writeSize = getExpectedWriteSize() / estimatedSSTables;
+Directories.DataDirectory dataDirectory =
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.1
Merge branch 'cassandra-2.1' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35f173a0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35f173a0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35f173a0 Branch: refs/heads/cassandra-2.1 Commit: 35f173a0e25f6ac01d79d57ed5992836238295db Parents: 0d01c36 6ae1b42 Author: Marcus Eriksson marc...@apache.org Authored: Mon Nov 24 10:38:30 2014 +0100 Committer: Marcus Eriksson marc...@apache.org Committed: Mon Nov 24 10:38:30 2014 +0100 -- lib/jamm-0.3.0.jar | Bin 21149 - 21033 bytes 1 file changed, 0 insertions(+), 0 deletions(-) --
[4/4] cassandra git commit: Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/cassandra into trunk
Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/cassandra into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/065aeeb4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/065aeeb4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/065aeeb4 Branch: refs/heads/trunk Commit: 065aeeb4a9d2aca998f96f817649714badb2cc80 Parents: cd4f729 528cc3d Author: Marcus Eriksson marc...@apache.org Authored: Mon Nov 24 10:39:05 2014 +0100 Committer: Marcus Eriksson marc...@apache.org Committed: Mon Nov 24 10:39:05 2014 +0100 -- lib/jamm-0.3.0.jar | Bin 21149 - 21033 bytes 1 file changed, 0 insertions(+), 0 deletions(-) --
[jira] [Commented] (CASSANDRA-8181) Intermittent failure of SSTableImportTest unit test
[
https://issues.apache.org/jira/browse/CASSANDRA-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222832#comment-14222832
]
Benjamin Lerer commented on CASSANDRA-8181:
---
[~azotcsit] as you are the one that wrote the patch for #7498 could you review
the fix.
Intermittent failure of SSTableImportTest unit test
---
Key: CASSANDRA-8181
URL: https://issues.apache.org/jira/browse/CASSANDRA-8181
Project: Cassandra
Issue Type: Bug
Components: Tests
Reporter: Michael Shuler
Assignee: Benjamin Lerer
Fix For: 2.1.3
Attachments: CASSANDRA-8181.txt,
SSTableImportTest_failure_system.log.gz
{noformat}
[junit] Testsuite: org.apache.cassandra.tools.SSTableImportTest
[junit] Tests run: 8, Failures: 1, Errors: 0, Time elapsed: 6.712 sec
[junit]
[junit] - Standard Output ---
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 1 keys...
[junit] 1 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] - ---
[junit] Testcase:
testImportWithAsciiKeyValidator(org.apache.cassandra.tools.SSTableImportTest):
FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at
org.apache.cassandra.tools.SSTableImportTest.testImportWithAsciiKeyValidator(SSTableImportTest.java:166)
[junit]
[junit]
[junit] Test org.apache.cassandra.tools.SSTableImportTest FAILED
{noformat}
testImportWithAsciiKeyValidator was added in CASSANDRA-7498 and fails as
above occasionally (~10-15% of runs) in CI. Attached is the system.log from
the failed test on 2.1 HEAD (8e5fdc2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8181) Intermittent failure of SSTableImportTest unit test
[
https://issues.apache.org/jira/browse/CASSANDRA-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222833#comment-14222833
]
Alexey Zotov commented on CASSANDRA-8181:
-
[~blerer] good catch, the fix LGTM. Thanks.
Intermittent failure of SSTableImportTest unit test
---
Key: CASSANDRA-8181
URL: https://issues.apache.org/jira/browse/CASSANDRA-8181
Project: Cassandra
Issue Type: Bug
Components: Tests
Reporter: Michael Shuler
Assignee: Benjamin Lerer
Fix For: 2.1.3
Attachments: CASSANDRA-8181.txt,
SSTableImportTest_failure_system.log.gz
{noformat}
[junit] Testsuite: org.apache.cassandra.tools.SSTableImportTest
[junit] Tests run: 8, Failures: 1, Errors: 0, Time elapsed: 6.712 sec
[junit]
[junit] - Standard Output ---
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 1 keys...
[junit] 1 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] - ---
[junit] Testcase:
testImportWithAsciiKeyValidator(org.apache.cassandra.tools.SSTableImportTest):
FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at
org.apache.cassandra.tools.SSTableImportTest.testImportWithAsciiKeyValidator(SSTableImportTest.java:166)
[junit]
[junit]
[junit] Test org.apache.cassandra.tools.SSTableImportTest FAILED
{noformat}
testImportWithAsciiKeyValidator was added in CASSANDRA-7498 and fails as
above occasionally (~10-15% of runs) in CI. Attached is the system.log from
the failed test on 2.1 HEAD (8e5fdc2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222837#comment-14222837 ] Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-8231: - I tried a get latest from the repo and it appears there is no check-in happened related to this jar file. Can you please check that? Thanks a lot Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222841#comment-14222841 ] Sylvain Lebresne commented on CASSANDRA-8231: - Well, you might want to check again because it's beein committed. Make sure you're using http://git-wip-us.apache.org/repos/asf/cassandra.git, not some mirror that lags behind. Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222841#comment-14222841 ] Sylvain Lebresne edited comment on CASSANDRA-8231 at 11/24/14 10:00 AM: Well, you might want to check again because it's been committed. Make sure you're using http://git-wip-us.apache.org/repos/asf/cassandra.git, not some mirror that lags behind. was (Author: slebresne): Well, you might want to check again because it's beein committed. Make sure you're using http://git-wip-us.apache.org/repos/asf/cassandra.git, not some mirror that lags behind. Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222846#comment-14222846 ] Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-8231: - Thanks a lot for the info. I was using https://github.com/apache/cassandra/tree/trunk Any idea how long the above one takes to get it synced? Thanks -Raj Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8168) Require Java 8
[
https://issues.apache.org/jira/browse/CASSANDRA-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222855#comment-14222855
]
Sylvain Lebresne commented on CASSANDRA-8168:
-
bq. In other words, extend Cassandra 2.1 EOL by at least one year longer than
we usually do. People stuck with Java 7 won't be able to go to C* 3.0 (until
they get unstuck), but won't have to deal with an abandoned Cassandra version,
either.
For what it's worth, I'm mildly a fan of this. It's a bit unclear to me that
getting java 8 in 3.0 is such a big deal, but buying one more year of painful
merge is a big deal. Getting nashorn is nice, but for UDF I think we'll end up
with mostly recommending people to use JAVA directly and so I don't think it's
a huge deal. As for CASSANDRA-5239, how sure are we that we'll have this done
in 3.0? And that {{CompletableFuture}} gives us something that guava's
{{ListenableFuture}} don't give us?
Require Java 8
--
Key: CASSANDRA-8168
URL: https://issues.apache.org/jira/browse/CASSANDRA-8168
Project: Cassandra
Issue Type: Task
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 3.0
This is to discuss requiring Java 8 for version = 3.0
There are a couple big reasons for this.
* Better support for complex async work e.g (CASSANDRA-5239)
http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
* Use Nashorn for Javascript UDFs CASSANDRA-7395
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8280) Cassandra crashing on inserting data over 64K into indexed strings
[
https://issues.apache.org/jira/browse/CASSANDRA-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-8280:
---
Attachment: 8280-2.0-v4.txt
v4 of the 2.0 patch, which fixes the length check in MS.buildPartitionKeyNames
removes the redundant call to ThriftValidation.validateKey in
MS.getMutations()
Cassandra crashing on inserting data over 64K into indexed strings
--
Key: CASSANDRA-8280
URL: https://issues.apache.org/jira/browse/CASSANDRA-8280
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Debian 7, Cassandra 2.1.1, java 1.7.0_60
Reporter: Cristian Marinescu
Assignee: Sam Tunnicliffe
Priority: Critical
Fix For: 2.1.3
Attachments: 8280-2.0-v2.txt, 8280-2.0-v3.txt, 8280-2.0-v4.txt,
8280-2.0.txt, 8280-2.1-v2.txt, 8280-2.1.txt
An attemtp to instert 65536 bytes in a field that is a primary index throws
(correctly?) the cassandra.InvalidRequest exception. However, inserting the
same data *in a indexed field that is not a primary index* works just fine.
However, Cassandra will crash on next commit and never recover. So I rated it
as Critical as it can be used for DoS attacks.
Reproduce: see the snippet below:
{code}
import uuid
from cassandra import ConsistencyLevel
from cassandra import InvalidRequest
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
from cassandra.policies import ConstantReconnectionPolicy
from cassandra.cqltypes import UUID
# DROP KEYSPACE IF EXISTS cs;
# CREATE KEYSPACE cs WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
# USE cs;
# CREATE TABLE test3 (name text, value uuid, sentinel text, PRIMARY KEY
(name));
# CREATE INDEX test3_sentinels ON test3(sentinel);
class CassandraDemo(object):
def __init__(self):
ips = [127.0.0.1]
ap = PlainTextAuthProvider(username=cs, password=cs)
reconnection_policy = ConstantReconnectionPolicy(20.0,
max_attempts=100)
cluster = Cluster(ips, auth_provider=ap, protocol_version=3,
reconnection_policy=reconnection_policy)
self.session = cluster.connect(cs)
def exec_query(self, query, args):
prepared_statement = self.session.prepare(query)
prepared_statement.consistency_level = ConsistencyLevel.LOCAL_QUORUM
self.session.execute(prepared_statement, args)
def bug(self):
k1 = UUID( str(uuid.uuid4()) )
long_string = X * 65536
query = INSERT INTO test3 (name, value, sentinel) VALUES (?, ?, ?);
args = (foo, k1, long_string)
self.exec_query(query, args)
self.session.execute(DROP KEYSPACE IF EXISTS cs_test, timeout=30)
self.session.execute(CREATE KEYSPACE cs_test WITH replication =
{'class': 'SimpleStrategy', 'replication_factor': 1})
c = CassandraDemo()
#first run
c.bug()
#second run, Cassandra crashes with java.lang.AssertionError
c.bug()
{code}
And here is the cassandra log:
{code}
ERROR [MemtableFlushWriter:3] 2014-11-06 16:44:49,263
CassandraDaemon.java:153 - Exception in thread
Thread[MemtableFlushWriter:3,5,main]
java.lang.AssertionError: 65536
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.db.ColumnIndex$Builder.maybeWriteRowHeader(ColumnIndex.java:214)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:201)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:142)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:233)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:218)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:354)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:312)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
at
[jira] [Updated] (CASSANDRA-8352) Timeout Exception on Node Failure in Remote Data Center
[ https://issues.apache.org/jira/browse/CASSANDRA-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhtar Hussain updated CASSANDRA-8352: -- Summary: Timeout Exception on Node Failure in Remote Data Center (was: Strange problem regarding Cassandra nodes) Timeout Exception on Node Failure in Remote Data Center --- Key: CASSANDRA-8352 URL: https://issues.apache.org/jira/browse/CASSANDRA-8352 Project: Cassandra Issue Type: Bug Environment: Unix, Cassandra 2.0.3 Reporter: Akhtar Hussain Labels: DataCenter, GEO-Red We have a Geo-red setup with 2 Data centers having 3 nodes each. When we bring down a single Cassandra node down in DC2 by kill -9 Cassandra-pid, reads fail on DC1 with TimedOutException for a brief amount of time (15-20 sec~). Questions: 1.We need to understand why reads fail on DC1 when a node in another DC i.e. DC2 fails? As we are using LOCAL_QUORUM for both reads/writes in DC1, request should return once 2 nodes in local DC have replied instead of timing out because of node in remote DC. 2.We want to make sure that no Cassandra requests fail in case of node failures. We used rapid read protection of ALWAYS/99percentile/10ms as mentioned in http://www.datastax.com/dev/blog/rapid-read-protection-in-cassandra-2-0-2. But nothing worked. How to ensure zero request failures in case a node fails? 3.What is the right way of handling HTimedOutException exceptions in Hector? 4.Please confirm are we using public private hostnames as expected? We are using Cassandra 2.0.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8352) Timeout Exception on Node Failure in Remote Data Center
[ https://issues.apache.org/jira/browse/CASSANDRA-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhtar Hussain updated CASSANDRA-8352: -- Reviewer: Anuj Timeout Exception on Node Failure in Remote Data Center --- Key: CASSANDRA-8352 URL: https://issues.apache.org/jira/browse/CASSANDRA-8352 Project: Cassandra Issue Type: Bug Environment: Unix, Cassandra 2.0.3 Reporter: Akhtar Hussain Labels: DataCenter, GEO-Red We have a Geo-red setup with 2 Data centers having 3 nodes each. When we bring down a single Cassandra node down in DC2 by kill -9 Cassandra-pid, reads fail on DC1 with TimedOutException for a brief amount of time (15-20 sec~). Questions: 1.We need to understand why reads fail on DC1 when a node in another DC i.e. DC2 fails? As we are using LOCAL_QUORUM for both reads/writes in DC1, request should return once 2 nodes in local DC have replied instead of timing out because of node in remote DC. 2.We want to make sure that no Cassandra requests fail in case of node failures. We used rapid read protection of ALWAYS/99percentile/10ms as mentioned in http://www.datastax.com/dev/blog/rapid-read-protection-in-cassandra-2-0-2. But nothing worked. How to ensure zero request failures in case a node fails? 3.What is the right way of handling HTimedOutException exceptions in Hector? 4.Please confirm are we using public private hostnames as expected? We are using Cassandra 2.0.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Validate size of indexed column values
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 2ce1ad8e6 - 0e3d9fc14
Validate size of indexed column values
patch by Sam Tunnicliffe; reviewed by Aleksey Yeschenko for
CASSANDRA-8280
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0e3d9fc1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0e3d9fc1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0e3d9fc1
Branch: refs/heads/cassandra-2.0
Commit: 0e3d9fc14bfcb38b9f179c0428cf586890c4a8ab
Parents: 2ce1ad8
Author: Sam Tunnicliffe s...@beobal.com
Authored: Mon Nov 24 14:50:14 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 14:50:14 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/cql3/CFDefinition.java | 5 +
.../cassandra/cql3/ColumnNameBuilder.java | 7 ++
.../cql3/statements/ModificationStatement.java | 20 +--
.../cql3/statements/UpdateStatement.java| 22 +++-
.../db/index/SecondaryIndexManager.java | 6 +-
.../cassandra/db/marshal/CompositeType.java | 9 ++
.../cassandra/io/sstable/SSTableWriter.java | 9 ++
.../cql3/IndexedValuesValidationTest.java | 124 +++
9 files changed, 192 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6a5ac0d..412eb59 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/CFDefinition.java
--
diff --git a/src/java/org/apache/cassandra/cql3/CFDefinition.java
b/src/java/org/apache/cassandra/cql3/CFDefinition.java
index 23bedaf..e0bb409 100644
--- a/src/java/org/apache/cassandra/cql3/CFDefinition.java
+++ b/src/java/org/apache/cassandra/cql3/CFDefinition.java
@@ -358,5 +358,10 @@ public class CFDefinition implements
IterableCFDefinition.Name
return columnName;
}
+
+public int getLength()
+{
+return columnName == null ? 0 : columnName.remaining();
+}
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
b/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
index 3d5eff6..50cdc74 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
@@ -78,4 +78,11 @@ public interface ColumnNameBuilder
*/
public ByteBuffer getComponent(int i);
+/**
+ * Returns the total length of the ByteBuffer that will
+ * be returned by build().
+ * @return the total length of the column name to be built
+ */
+public int getLength();
+
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
index 61f65c1..db22e7d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
@@ -22,8 +22,6 @@ import java.util.*;
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
-import org.apache.cassandra.db.marshal.AbstractType;
-import org.github.jamm.MemoryMeter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,7 +41,9 @@ import org.apache.cassandra.service.QueryState;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.thrift.ThriftValidation;
import org.apache.cassandra.transport.messages.ResultMessage;
+import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
+import org.github.jamm.MemoryMeter;
/*
* Abstract parent class of individual modifications, i.e. INSERT, UPDATE and
DELETE.
@@ -328,7 +328,7 @@ public abstract class ModificationStatement implements
CQLStatement, MeasurableF
throws InvalidRequestException
{
CFDefinition
[jira] [Resolved] (CASSANDRA-8081) AssertionError with 2ndary indexes
[
https://issues.apache.org/jira/browse/CASSANDRA-8081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe resolved CASSANDRA-8081.
Resolution: Duplicate
Reproduced In: 2.1.0, 2.0.10, 1.2.19 (was: 1.2.19, 2.0.10, 2.1.0)
This will be fixed by CASSANDRA-8280, please reopen if it isn't.
AssertionError with 2ndary indexes
---
Key: CASSANDRA-8081
URL: https://issues.apache.org/jira/browse/CASSANDRA-8081
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: kais
Assignee: Sam Tunnicliffe
If you create a table with a column type text or blob and add a secondary
index on it. If you insert a value that is longer than
FBUtilities.MAX_UNSIGNED_SHORT, then flush, you get an assertion error
{code}
CREATE TABLE test_text (key text PRIMARY KEY, col text);
CREATE INDEX test_text_col_idx ON test_text (col);
{code}
{code}
INFO [FlushWriter:3] 2014-10-08 10:53:38,471 Memtable.java (line 331)
Writing Memtable-test_text.test_text_col_idx@849649959(15/150 serialized/live
bytes, 1 ops)
INFO [FlushWriter:4] 2014-10-08 10:53:38,554 Memtable.java (line 331)
Writing Memtable-test_text@1448092010(100025/1000250 serialized/live bytes, 2
ops)
ERROR [FlushWriter:3] 2014-10-08 10:53:38,554 CassandraDaemon.java (line 196)
Exception in thread Thread[FlushWriter:3,5,RMI Runtime]
java.lang.AssertionError: 10
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:342)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[1/2] cassandra git commit: Validate size of indexed column values
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 35f173a0e - cab2b25b0
Validate size of indexed column values
patch by Sam Tunnicliffe; reviewed by Aleksey Yeschenko for
CASSANDRA-8280
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0e3d9fc1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0e3d9fc1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0e3d9fc1
Branch: refs/heads/cassandra-2.1
Commit: 0e3d9fc14bfcb38b9f179c0428cf586890c4a8ab
Parents: 2ce1ad8
Author: Sam Tunnicliffe s...@beobal.com
Authored: Mon Nov 24 14:50:14 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 14:50:14 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/cql3/CFDefinition.java | 5 +
.../cassandra/cql3/ColumnNameBuilder.java | 7 ++
.../cql3/statements/ModificationStatement.java | 20 +--
.../cql3/statements/UpdateStatement.java| 22 +++-
.../db/index/SecondaryIndexManager.java | 6 +-
.../cassandra/db/marshal/CompositeType.java | 9 ++
.../cassandra/io/sstable/SSTableWriter.java | 9 ++
.../cql3/IndexedValuesValidationTest.java | 124 +++
9 files changed, 192 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6a5ac0d..412eb59 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/CFDefinition.java
--
diff --git a/src/java/org/apache/cassandra/cql3/CFDefinition.java
b/src/java/org/apache/cassandra/cql3/CFDefinition.java
index 23bedaf..e0bb409 100644
--- a/src/java/org/apache/cassandra/cql3/CFDefinition.java
+++ b/src/java/org/apache/cassandra/cql3/CFDefinition.java
@@ -358,5 +358,10 @@ public class CFDefinition implements
IterableCFDefinition.Name
return columnName;
}
+
+public int getLength()
+{
+return columnName == null ? 0 : columnName.remaining();
+}
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
b/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
index 3d5eff6..50cdc74 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
@@ -78,4 +78,11 @@ public interface ColumnNameBuilder
*/
public ByteBuffer getComponent(int i);
+/**
+ * Returns the total length of the ByteBuffer that will
+ * be returned by build().
+ * @return the total length of the column name to be built
+ */
+public int getLength();
+
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
index 61f65c1..db22e7d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
@@ -22,8 +22,6 @@ import java.util.*;
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
-import org.apache.cassandra.db.marshal.AbstractType;
-import org.github.jamm.MemoryMeter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,7 +41,9 @@ import org.apache.cassandra.service.QueryState;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.thrift.ThriftValidation;
import org.apache.cassandra.transport.messages.ResultMessage;
+import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
+import org.github.jamm.MemoryMeter;
/*
* Abstract parent class of individual modifications, i.e. INSERT, UPDATE and
DELETE.
@@ -328,7 +328,7 @@ public abstract class ModificationStatement implements
CQLStatement, MeasurableF
throws InvalidRequestException
{
CFDefinition
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/cql3/CFDefinition.java
src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cab2b25b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cab2b25b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cab2b25b
Branch: refs/heads/cassandra-2.1
Commit: cab2b25b0a5b2029a9c3e1324d080b4982fbdc50
Parents: 35f173a 0e3d9fc
Author: Aleksey Yeschenko alek...@apache.org
Authored: Mon Nov 24 15:14:14 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 15:14:14 2014 +0300
--
CHANGES.txt | 1 +
.../cql3/statements/UpdateStatement.java| 17 +++-
.../cassandra/io/sstable/SSTableWriter.java | 9 ++
.../cql3/IndexedValuesValidationTest.java | 86
4 files changed, 112 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cab2b25b/CHANGES.txt
--
diff --cc CHANGES.txt
index 313000a,412eb59..9db65e9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,5 +1,19 @@@
-2.0.12:
+2.1.3
+ * Fix high size calculations for prepared statements (CASSANDRA-8231)
+ * Centralize shared executors (CASSANDRA-8055)
+ * Fix filtering for CONTAINS (KEY) relations on frozen collection
+ clustering columns when the query is restricted to a single
+ partition (CASSANDRA-8203)
+ * Do more aggressive entire-sstable TTL expiry checks (CASSANDRA-8243)
+ * Add more log info if readMeter is null (CASSANDRA-8238)
+ * add check of the system wall clock time at startup (CASSANDRA-8305)
+ * Support for frozen collections (CASSANDRA-7859)
+ * Fix overflow on histogram computation (CASSANDRA-8028)
+ * Have paxos reuse the timestamp generation of normal queries
(CASSANDRA-7801)
+ * Fix incremental repair not remove parent session on remote (CASSANDRA-8291)
+ * Improve JBOD disk utilization (CASSANDRA-7386)
+Merged from 2.0:
+ * Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories
(CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cab2b25b/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
--
diff --cc src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
index 2c87173,9d98c84..09f26d6
--- a/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
@@@ -22,12 -22,11 +22,12 @@@ import java.util.*
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.composites.Composite;
- import org.apache.cassandra.db.marshal.AbstractType;
+ import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.utils.ByteBufferUtil;
-import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
/**
@@@ -95,8 -103,22 +95,23 @@@ public class UpdateStatement extends Mo
else
{
for (Operation update : updates)
-update.execute(key, cf, builder.copy(), params);
+update.execute(key, cf, prefix, params);
}
+
+ SecondaryIndexManager indexManager =
Keyspace.open(cfm.ksName).getColumnFamilyStore(cfm.cfId).indexManager;
+ if (indexManager.hasIndexes())
+ {
-for (Column column : cf)
++for (Cell cell : cf)
+ {
-if (!indexManager.validate(column))
++// Indexed values must be validated by any applicable index.
See CASSANDRA-3057/4240/8081 for more details
++if (!indexManager.validate(cell))
+ throw new InvalidRequestException(String.format(Can't
index column value of size %d for index %s on %s.%s,
-
column.value().remaining(),
-
[3/4] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/cql3/CFDefinition.java
src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cab2b25b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cab2b25b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cab2b25b
Branch: refs/heads/trunk
Commit: cab2b25b0a5b2029a9c3e1324d080b4982fbdc50
Parents: 35f173a 0e3d9fc
Author: Aleksey Yeschenko alek...@apache.org
Authored: Mon Nov 24 15:14:14 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 15:14:14 2014 +0300
--
CHANGES.txt | 1 +
.../cql3/statements/UpdateStatement.java| 17 +++-
.../cassandra/io/sstable/SSTableWriter.java | 9 ++
.../cql3/IndexedValuesValidationTest.java | 86
4 files changed, 112 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cab2b25b/CHANGES.txt
--
diff --cc CHANGES.txt
index 313000a,412eb59..9db65e9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,5 +1,19 @@@
-2.0.12:
+2.1.3
+ * Fix high size calculations for prepared statements (CASSANDRA-8231)
+ * Centralize shared executors (CASSANDRA-8055)
+ * Fix filtering for CONTAINS (KEY) relations on frozen collection
+ clustering columns when the query is restricted to a single
+ partition (CASSANDRA-8203)
+ * Do more aggressive entire-sstable TTL expiry checks (CASSANDRA-8243)
+ * Add more log info if readMeter is null (CASSANDRA-8238)
+ * add check of the system wall clock time at startup (CASSANDRA-8305)
+ * Support for frozen collections (CASSANDRA-7859)
+ * Fix overflow on histogram computation (CASSANDRA-8028)
+ * Have paxos reuse the timestamp generation of normal queries
(CASSANDRA-7801)
+ * Fix incremental repair not remove parent session on remote (CASSANDRA-8291)
+ * Improve JBOD disk utilization (CASSANDRA-7386)
+Merged from 2.0:
+ * Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories
(CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/cab2b25b/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
--
diff --cc src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
index 2c87173,9d98c84..09f26d6
--- a/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
@@@ -22,12 -22,11 +22,12 @@@ import java.util.*
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.composites.Composite;
- import org.apache.cassandra.db.marshal.AbstractType;
+ import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.utils.ByteBufferUtil;
-import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
/**
@@@ -95,8 -103,22 +95,23 @@@ public class UpdateStatement extends Mo
else
{
for (Operation update : updates)
-update.execute(key, cf, builder.copy(), params);
+update.execute(key, cf, prefix, params);
}
+
+ SecondaryIndexManager indexManager =
Keyspace.open(cfm.ksName).getColumnFamilyStore(cfm.cfId).indexManager;
+ if (indexManager.hasIndexes())
+ {
-for (Column column : cf)
++for (Cell cell : cf)
+ {
-if (!indexManager.validate(column))
++// Indexed values must be validated by any applicable index.
See CASSANDRA-3057/4240/8081 for more details
++if (!indexManager.validate(cell))
+ throw new InvalidRequestException(String.format(Can't
index column value of size %d for index %s on %s.%s,
-
column.value().remaining(),
-
[2/4] cassandra git commit: Validate size of indexed column values
Validate size of indexed column values
patch by Sam Tunnicliffe; reviewed by Aleksey Yeschenko for
CASSANDRA-8280
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0e3d9fc1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0e3d9fc1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0e3d9fc1
Branch: refs/heads/trunk
Commit: 0e3d9fc14bfcb38b9f179c0428cf586890c4a8ab
Parents: 2ce1ad8
Author: Sam Tunnicliffe s...@beobal.com
Authored: Mon Nov 24 14:50:14 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 14:50:14 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/cql3/CFDefinition.java | 5 +
.../cassandra/cql3/ColumnNameBuilder.java | 7 ++
.../cql3/statements/ModificationStatement.java | 20 +--
.../cql3/statements/UpdateStatement.java| 22 +++-
.../db/index/SecondaryIndexManager.java | 6 +-
.../cassandra/db/marshal/CompositeType.java | 9 ++
.../cassandra/io/sstable/SSTableWriter.java | 9 ++
.../cql3/IndexedValuesValidationTest.java | 124 +++
9 files changed, 192 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6a5ac0d..412eb59 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
on COMPACT STORAGE tables (CASSANDRA-8264)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/CFDefinition.java
--
diff --git a/src/java/org/apache/cassandra/cql3/CFDefinition.java
b/src/java/org/apache/cassandra/cql3/CFDefinition.java
index 23bedaf..e0bb409 100644
--- a/src/java/org/apache/cassandra/cql3/CFDefinition.java
+++ b/src/java/org/apache/cassandra/cql3/CFDefinition.java
@@ -358,5 +358,10 @@ public class CFDefinition implements
IterableCFDefinition.Name
return columnName;
}
+
+public int getLength()
+{
+return columnName == null ? 0 : columnName.remaining();
+}
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
b/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
index 3d5eff6..50cdc74 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnNameBuilder.java
@@ -78,4 +78,11 @@ public interface ColumnNameBuilder
*/
public ByteBuffer getComponent(int i);
+/**
+ * Returns the total length of the ByteBuffer that will
+ * be returned by build().
+ * @return the total length of the column name to be built
+ */
+public int getLength();
+
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0e3d9fc1/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
index 61f65c1..db22e7d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
@@ -22,8 +22,6 @@ import java.util.*;
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
-import org.apache.cassandra.db.marshal.AbstractType;
-import org.github.jamm.MemoryMeter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,7 +41,9 @@ import org.apache.cassandra.service.QueryState;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.thrift.ThriftValidation;
import org.apache.cassandra.transport.messages.ResultMessage;
+import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
+import org.github.jamm.MemoryMeter;
/*
* Abstract parent class of individual modifications, i.e. INSERT, UPDATE and
DELETE.
@@ -328,7 +328,7 @@ public abstract class ModificationStatement implements
CQLStatement, MeasurableF
throws InvalidRequestException
{
CFDefinition cfDef = cfm.getCfDef();
-ColumnNameBuilder keyBuilder = cfDef.getKeyNameBuilder();
+
[1/4] cassandra git commit: Merge branch 'cassandra-2.1' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.1
Repository: cassandra Updated Branches: refs/heads/trunk 065aeeb4a - 41435ef6c Merge branch 'cassandra-2.1' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35f173a0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35f173a0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35f173a0 Branch: refs/heads/trunk Commit: 35f173a0e25f6ac01d79d57ed5992836238295db Parents: 0d01c36 6ae1b42 Author: Marcus Eriksson marc...@apache.org Authored: Mon Nov 24 10:38:30 2014 +0100 Committer: Marcus Eriksson marc...@apache.org Committed: Mon Nov 24 10:38:30 2014 +0100 -- lib/jamm-0.3.0.jar | Bin 21149 - 21033 bytes 1 file changed, 0 insertions(+), 0 deletions(-) --
[4/4] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/41435ef6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/41435ef6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/41435ef6
Branch: refs/heads/trunk
Commit: 41435ef6c1fec1cadf6606eb6eb66fe15bd8c46d
Parents: 065aeeb cab2b25
Author: Aleksey Yeschenko alek...@apache.org
Authored: Mon Nov 24 15:19:33 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 15:19:33 2014 +0300
--
CHANGES.txt | 1 +
.../cql3/statements/UpdateStatement.java| 17 +++-
.../io/sstable/format/big/BigTableWriter.java | 9 ++
.../cql3/IndexedValuesValidationTest.java | 86
4 files changed, 112 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/41435ef6/CHANGES.txt
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/41435ef6/src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/41435ef6/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
--
diff --cc
src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index ec53b4e,000..5221509
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@@ -1,541 -1,0 +1,550 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable.format.big;
+
+import java.io.Closeable;
+import java.io.DataInput;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.cassandra.db.*;
+import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.io.sstable.format.SSTableWriter;
+import org.apache.cassandra.io.sstable.format.Version;
+import org.apache.cassandra.io.util.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compaction.AbstractCompactedRow;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.io.FSWriteError;
+import org.apache.cassandra.io.compress.CompressedSequentialWriter;
+import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
+import org.apache.cassandra.io.sstable.metadata.MetadataComponent;
+import org.apache.cassandra.io.sstable.metadata.MetadataType;
+import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
+import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.io.util.DataOutputStreamAndChannel;
+import org.apache.cassandra.io.util.FileMark;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.io.util.SegmentedFile;
+import org.apache.cassandra.io.util.SequentialWriter;
+import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.ByteBufferUtil;
++import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.FilterFactory;
+import org.apache.cassandra.utils.IFilter;
+import org.apache.cassandra.utils.Pair;
+import org.apache.cassandra.utils.StreamingHistogram;
+
+public class BigTableWriter extends SSTableWriter
+{
+private static final Logger logger =
LoggerFactory.getLogger(BigTableWriter.class);
+
+// not very
[jira] [Updated] (CASSANDRA-8366) Repair grows data on nodes, causes load to become unbalanced
[ https://issues.apache.org/jira/browse/CASSANDRA-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Karlsson updated CASSANDRA-8366: Description: There seems to be something weird going on when repairing data. I have a program that runs 2 hours which inserts 250 random numbers and reads 250 times per second. It creates 2 keyspaces with SimpleStrategy and RF of 3. I use size-tiered compaction for my cluster. After those 2 hours I run a repair and the load of all nodes goes up. If I run incremental repair the load goes up alot more. I saw the load shoot up 8 times the original size multiple times with incremental repair. (from 2G to 16G) with node 9 8 7 and 6 the repro procedure looked like this: (Note that running full repair first is not a requirement to reproduce.) After 2 hours of 250 reads + 250 writes per second: UN 9 583.39 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 584.01 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 583.72 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 583.84 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Repair -pr -par on all nodes sequentially UN 9 746.29 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 751.02 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 748.89 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.34 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 2.41 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.53 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.6 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 2.17 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 after rolling restart UN 9 1.47 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 1.5 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.19 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 compact all nodes sequentially UN 9 989.99 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 994.75 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 1.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.82 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 1.98 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.3 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 3.71 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 restart once more UN 9 2 GB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.05 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 4.1 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Is there something im missing or is this strange behavior? was: There seems to be something weird going on when repairing data. I have a program that runs 2 hours which inserts 250 random numbers and reads 250 times per second. It creates 2 keyspaces with SimpleStrategy and RF of 3. I use size-tiered compaction for my cluster. After those 2 hours I run a repair and the load of all nodes goes up. If I run incremental repair the load goes up alot more. I saw the load shoot up 8 times the original size multiple times with incremental repair. (from 2G to 16G) with node 9 8 7 and 7 the repro procedure looked like this: (Note that running full repair first is not a requirement to reproduce.) After 2 hours of 250 reads + 250 writes per second: UN 9 583.39 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 584.01 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 583.72 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 583.84 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Repair -pr -par on all nodes sequentially UN 9 746.29 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 751.02 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 748.89 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.34 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 2.41 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.53 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.6 GB 256 ?
[jira] [Updated] (CASSANDRA-7886) TombstoneOverwhelmingException should not wait for timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Spriegel updated CASSANDRA-7886: -- Attachment: 7886_v2_trunk.txt TombstoneOverwhelmingException should not wait for timeout -- Key: CASSANDRA-7886 URL: https://issues.apache.org/jira/browse/CASSANDRA-7886 Project: Cassandra Issue Type: Improvement Components: Core Environment: Tested with Cassandra 2.0.8 Reporter: Christian Spriegel Assignee: Christian Spriegel Priority: Minor Fix For: 3.0 Attachments: 7886_v1.txt, 7886_v2_trunk.txt *Issue* When you have TombstoneOverwhelmingExceptions occuring in queries, this will cause the query to be simply dropped on every data-node, but no response is sent back to the coordinator. Instead the coordinator waits for the specified read_request_timeout_in_ms. On the application side this can cause memory issues, since the application is waiting for the timeout interval for every request.Therefore, if our application runs into TombstoneOverwhelmingExceptions, then (sooner or later) our entire application cluster goes down :-( *Proposed solution* I think the data nodes should send a error message to the coordinator when they run into a TombstoneOverwhelmingException. Then the coordinator does not have to wait for the timeout-interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222950#comment-14222950
]
Nicolas Lalevée commented on CASSANDRA-8356:
I got the snapshot data from a node on my local machine, and I tried to load it
up in a local cassandra node 2.0.11.
The node did the opening of the files correctly. But querying against it is
impossible, I hit the following error:
{noformat}
ERROR 11:28:45,693 Exception in thread Thread[ReadStage:2,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:267)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:587)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(ByteBufferUtil.java:596)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:61)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:1)
at
org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:436)
at
org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:141)
at
org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:113)
at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:202)
at
org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:54)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:155)
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:168)
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:87)
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:56)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1413)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1977)
... 3 more
{nofomart}
This reminded me of an error we had on our test cluster, when we tested the
upgrade to 2.0.x : CASSANDRA-6733
So here, I ran an upgradesstable on our production cluster, and now the slice
queries return all the expected data. So everything is back to normal (and I am
very pleased by the lower cpu activity with 2.0.x for the same load).
I looked up again the logs in prod, I still don't see any such Buffer.limit
errors. I don't know what was going wrong.
As for CASSANDRA-6733, I have a snapshot of the data before the upgrade_sstable
(unfortunately I don't have a snapshot pre-upgrade, but somme sstables are sill
in the old format). If someone wants the data to analyse it, concat me,
nlalevee at scoop.it.
Slice query on a super column family with counters doesn't get all the data
---
Key: CASSANDRA-8356
URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
Project: Cassandra
Issue Type: Bug
Reporter: Nicolas Lalevée
Assignee: Aleksey Yeschenko
Fix For: 2.0.12
We've finally been able to upgrade our cluster to 2.0.11, after
CASSANDRA-7188 being fixed.
But now slice queries on a super column family with counters doesn't return
all the expected data. We first though because of all the trouble we had that
we lost data, but there a way to actually
[jira] [Comment Edited] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222950#comment-14222950
]
Nicolas Lalevée edited comment on CASSANDRA-8356 at 11/24/14 12:50 PM:
---
I got the snapshot data from a node on my local machine, and I tried to load it
up in a local cassandra node 2.0.11.
The node did the opening of the files correctly. But querying against it is
impossible, I hit the following error:
{noformat}
ERROR 11:28:45,693 Exception in thread Thread[ReadStage:2,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:267)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:587)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(ByteBufferUtil.java:596)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:61)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:1)
at
org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:436)
at
org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:141)
at
org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:113)
at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:202)
at
org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:54)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:155)
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:168)
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:87)
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:56)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1413)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1977)
... 3 more
{noformat}
This reminded me of an error we had on our test cluster, when we tested the
upgrade to 2.0.x : CASSANDRA-6733
So here, I ran an upgradesstable on our production cluster, and now the slice
queries return all the expected data. So everything is back to normal (and I am
very pleased by the lower cpu activity with 2.0.x for the same load).
I looked up again the logs in prod, I still don't see any such Buffer.limit
errors. I don't know what was going wrong.
As for CASSANDRA-6733, I have a snapshot of the data before the upgrade_sstable
(unfortunately I don't have a snapshot pre-upgrade, but somme sstables are sill
in the old format). If someone wants the data to analyse it, concat me,
nlalevee at scoop.it.
was (Author: hibou):
I got the snapshot data from a node on my local machine, and I tried to load it
up in a local cassandra node 2.0.11.
The node did the opening of the files correctly. But querying against it is
impossible, I hit the following error:
{noformat}
ERROR 11:28:45,693 Exception in thread Thread[ReadStage:2,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
[jira] [Commented] (CASSANDRA-7886) TombstoneOverwhelmingException should not wait for timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222955#comment-14222955 ] Christian Spriegel commented on CASSANDRA-7886: --- Hi [~slebresne], I finally had the time to port my patch to trunk and add error handling to the ErrorMessage class. Thrift and CQL protocol 3 will get an Unavailable error instead of my new READ_FAILURE. CQL protocol = 4 will get the new READ_FAILURE. It seems there is no CQL protcol 4 yet, so my code always returns Unavailable at the moment. Let me know if I you want me to improve anything. TombstoneOverwhelmingException should not wait for timeout -- Key: CASSANDRA-7886 URL: https://issues.apache.org/jira/browse/CASSANDRA-7886 Project: Cassandra Issue Type: Improvement Components: Core Environment: Tested with Cassandra 2.0.8 Reporter: Christian Spriegel Assignee: Christian Spriegel Priority: Minor Fix For: 3.0 Attachments: 7886_v1.txt, 7886_v2_trunk.txt *Issue* When you have TombstoneOverwhelmingExceptions occuring in queries, this will cause the query to be simply dropped on every data-node, but no response is sent back to the coordinator. Instead the coordinator waits for the specified read_request_timeout_in_ms. On the application side this can cause memory issues, since the application is waiting for the timeout interval for every request.Therefore, if our application runs into TombstoneOverwhelmingExceptions, then (sooner or later) our entire application cluster goes down :-( *Proposed solution* I think the data nodes should send a error message to the coordinator when they run into a TombstoneOverwhelmingException. Then the coordinator does not have to wait for the timeout-interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222962#comment-14222962 ] Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-8231: - Looks like the issue is still persisting even after getting the latest… Here is my git log and I see the jamm jar file changes in there. Rajanarayanans-MacBook-Pro:cassandra-trunk RajT$ git log --oneline 41435ef Merge branch 'cassandra-2.1' into trunk cab2b25 Merge branch 'cassandra-2.0' into cassandra-2.1 0e3d9fc Validate size of indexed column values 065aeeb Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/cassandra into trunk 35f173a Merge branch 'cassandra-2.1' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.1 cd4f729 Merge branch 'cassandra-2.1' into trunk 528cc3d Merge branch 'cassandra-2.1' into trunk 6ae1b42 Better jamm 0.3.0 jar The error message is same though: [junit] Error opening zip file or JAR manifest missing : /Users/RajT/cassandra-source/cassandra-trunk/lib/jamm-0.3.0.jar Thanks -Raj Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Ignore Paxos commits for truncated tables
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 0e3d9fc14 - 17de36f24
Ignore Paxos commits for truncated tables
patch by Sam Tunnicliffe; reviewed by Aleksey Yeschenko for
CASSANDRA-7538
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17de36f2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17de36f2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17de36f2
Branch: refs/heads/cassandra-2.0
Commit: 17de36f246c912287b85eb7015583a35f5040919
Parents: 0e3d9fc
Author: Sam Tunnicliffe s...@beobal.com
Authored: Mon Nov 24 16:07:17 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 16:07:17 2014 +0300
--
CHANGES.txt | 1 +
.../cassandra/service/paxos/PaxosState.java | 17 ++-
.../cassandra/service/PaxosStateTest.java | 108 +++
3 files changed, 122 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 412eb59..fe23248 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Ignore Paxos commits for truncated tables (CASSANDRA-7538)
* Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/src/java/org/apache/cassandra/service/paxos/PaxosState.java
--
diff --git a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
index 0196122..2adecec 100644
--- a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
+++ b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
@@ -31,6 +31,7 @@ import org.apache.cassandra.db.RowMutation;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.SystemKeyspace;
import org.apache.cassandra.tracing.Tracing;
+import org.apache.cassandra.utils.UUIDGen;
public class PaxosState
{
@@ -132,10 +133,18 @@ public class PaxosState
// Committing it is however always safe due to column timestamps,
so always do it. However,
// if our current in-progress ballot is strictly greater than the
proposal one, we shouldn't
// erase the in-progress update.
-Tracing.trace(Committing proposal {}, proposal);
-RowMutation rm = proposal.makeMutation();
-Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
-
+// The table may have been truncated since the proposal was
initiated. In that case, we
+// don't want to perform the mutation and potentially resurrect
truncated data
+if (UUIDGen.unixTimestamp(proposal.ballot) =
SystemKeyspace.getTruncatedAt(proposal.update.metadata().cfId))
+{
+Tracing.trace(Committing proposal {}, proposal);
+RowMutation rm = proposal.makeMutation();
+Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
+}
+else
+{
+Tracing.trace(Not committing proposal {} as ballot timestamp
predates last truncation time, proposal);
+}
// We don't need to lock, we're just blindly updating
SystemKeyspace.savePaxosCommit(proposal);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/test/unit/org/apache/cassandra/service/PaxosStateTest.java
--
diff --git a/test/unit/org/apache/cassandra/service/PaxosStateTest.java
b/test/unit/org/apache/cassandra/service/PaxosStateTest.java
new file mode 100644
index 000..306c424
--- /dev/null
+++ b/test/unit/org/apache/cassandra/service/PaxosStateTest.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific
[jira] [Assigned] (CASSANDRA-8267) Only stream from unrepaired sstables during incremental repair
[ https://issues.apache.org/jira/browse/CASSANDRA-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson reassigned CASSANDRA-8267: -- Assignee: Marcus Eriksson Only stream from unrepaired sstables during incremental repair -- Key: CASSANDRA-8267 URL: https://issues.apache.org/jira/browse/CASSANDRA-8267 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1.3 Seems we stream from all sstables even if we do incremental repair, we should limit this to only stream from the unrepaired sstables if we do incremental repair -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/service/paxos/PaxosState.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eac7781e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eac7781e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eac7781e
Branch: refs/heads/cassandra-2.1
Commit: eac7781e7c429ac25b56ddc2ec20bc34f3244df6
Parents: cab2b25 17de36f
Author: Aleksey Yeschenko alek...@apache.org
Authored: Mon Nov 24 16:27:56 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 16:27:56 2014 +0300
--
CHANGES.txt | 1 +
.../cassandra/service/paxos/PaxosState.java | 17 ++-
.../cassandra/service/PaxosStateTest.java | 104 +++
3 files changed, 118 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eac7781e/CHANGES.txt
--
diff --cc CHANGES.txt
index 9db65e9,fe23248..c9e35d5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,5 +1,19 @@@
-2.0.12:
+2.1.3
+ * Fix high size calculations for prepared statements (CASSANDRA-8231)
+ * Centralize shared executors (CASSANDRA-8055)
+ * Fix filtering for CONTAINS (KEY) relations on frozen collection
+ clustering columns when the query is restricted to a single
+ partition (CASSANDRA-8203)
+ * Do more aggressive entire-sstable TTL expiry checks (CASSANDRA-8243)
+ * Add more log info if readMeter is null (CASSANDRA-8238)
+ * add check of the system wall clock time at startup (CASSANDRA-8305)
+ * Support for frozen collections (CASSANDRA-7859)
+ * Fix overflow on histogram computation (CASSANDRA-8028)
+ * Have paxos reuse the timestamp generation of normal queries
(CASSANDRA-7801)
+ * Fix incremental repair not remove parent session on remote (CASSANDRA-8291)
+ * Improve JBOD disk utilization (CASSANDRA-7386)
+Merged from 2.0:
+ * Ignore Paxos commits for truncated tables (CASSANDRA-7538)
* Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories
(CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eac7781e/src/java/org/apache/cassandra/service/paxos/PaxosState.java
--
diff --cc src/java/org/apache/cassandra/service/paxos/PaxosState.java
index abd173c,2adecec..01e03f4
--- a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
+++ b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
@@@ -18,17 -19,19 +18,18 @@@
* under the License.
*
*/
-
+package org.apache.cassandra.service.paxos;
import java.nio.ByteBuffer;
+import java.util.concurrent.locks.Lock;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import com.google.common.util.concurrent.Striped;
import org.apache.cassandra.config.CFMetaData;
-import org.apache.cassandra.db.RowMutation;
-import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.*;
import org.apache.cassandra.tracing.Tracing;
+ import org.apache.cassandra.utils.UUIDGen;
public class PaxosState
{
@@@ -131,10 -133,18 +132,18 @@@
// Committing it is however always safe due to column timestamps,
so always do it. However,
// if our current in-progress ballot is strictly greater than the
proposal one, we shouldn't
// erase the in-progress update.
- Tracing.trace(Committing proposal {}, proposal);
- Mutation mutation = proposal.makeMutation();
- Keyspace.open(mutation.getKeyspaceName()).apply(mutation, true);
-
+ // The table may have been truncated since the proposal was
initiated. In that case, we
+ // don't want to perform the mutation and potentially resurrect
truncated data
+ if (UUIDGen.unixTimestamp(proposal.ballot) =
SystemKeyspace.getTruncatedAt(proposal.update.metadata().cfId))
+ {
+ Tracing.trace(Committing proposal {}, proposal);
-RowMutation rm = proposal.makeMutation();
-Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
++Mutation mutation = proposal.makeMutation();
++Keyspace.open(mutation.getKeyspaceName()).apply(mutation,
true);
+ }
+ else
+ {
+ Tracing.trace(Not committing proposal {} as ballot timestamp
predates last truncation time, proposal);
+ }
[1/3] cassandra git commit: Ignore Paxos commits for truncated tables
Repository: cassandra
Updated Branches:
refs/heads/trunk 41435ef6c - 584113103
Ignore Paxos commits for truncated tables
patch by Sam Tunnicliffe; reviewed by Aleksey Yeschenko for
CASSANDRA-7538
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17de36f2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17de36f2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17de36f2
Branch: refs/heads/trunk
Commit: 17de36f246c912287b85eb7015583a35f5040919
Parents: 0e3d9fc
Author: Sam Tunnicliffe s...@beobal.com
Authored: Mon Nov 24 16:07:17 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 16:07:17 2014 +0300
--
CHANGES.txt | 1 +
.../cassandra/service/paxos/PaxosState.java | 17 ++-
.../cassandra/service/PaxosStateTest.java | 108 +++
3 files changed, 122 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 412eb59..fe23248 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Ignore Paxos commits for truncated tables (CASSANDRA-7538)
* Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/src/java/org/apache/cassandra/service/paxos/PaxosState.java
--
diff --git a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
index 0196122..2adecec 100644
--- a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
+++ b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
@@ -31,6 +31,7 @@ import org.apache.cassandra.db.RowMutation;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.SystemKeyspace;
import org.apache.cassandra.tracing.Tracing;
+import org.apache.cassandra.utils.UUIDGen;
public class PaxosState
{
@@ -132,10 +133,18 @@ public class PaxosState
// Committing it is however always safe due to column timestamps,
so always do it. However,
// if our current in-progress ballot is strictly greater than the
proposal one, we shouldn't
// erase the in-progress update.
-Tracing.trace(Committing proposal {}, proposal);
-RowMutation rm = proposal.makeMutation();
-Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
-
+// The table may have been truncated since the proposal was
initiated. In that case, we
+// don't want to perform the mutation and potentially resurrect
truncated data
+if (UUIDGen.unixTimestamp(proposal.ballot) =
SystemKeyspace.getTruncatedAt(proposal.update.metadata().cfId))
+{
+Tracing.trace(Committing proposal {}, proposal);
+RowMutation rm = proposal.makeMutation();
+Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
+}
+else
+{
+Tracing.trace(Not committing proposal {} as ballot timestamp
predates last truncation time, proposal);
+}
// We don't need to lock, we're just blindly updating
SystemKeyspace.savePaxosCommit(proposal);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/test/unit/org/apache/cassandra/service/PaxosStateTest.java
--
diff --git a/test/unit/org/apache/cassandra/service/PaxosStateTest.java
b/test/unit/org/apache/cassandra/service/PaxosStateTest.java
new file mode 100644
index 000..306c424
--- /dev/null
+++ b/test/unit/org/apache/cassandra/service/PaxosStateTest.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/58411310 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/58411310 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/58411310 Branch: refs/heads/trunk Commit: 5841131037155ab76fcfb0e84cacb3b00400830b Parents: 41435ef eac7781 Author: Aleksey Yeschenko alek...@apache.org Authored: Mon Nov 24 16:28:23 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Mon Nov 24 16:28:51 2014 +0300 -- CHANGES.txt | 4 +- .../cassandra/service/paxos/PaxosState.java | 17 ++- .../cassandra/service/PaxosStateTest.java | 104 +++ 3 files changed, 120 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/58411310/CHANGES.txt -- diff --cc CHANGES.txt index 1beb2e2,c9e35d5..af73426 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,38 -1,5 +1,39 @@@ +3.0 + * Fix aggregate fn results on empty selection, result column name, + and cqlsh parsing (CASSANDRA-8229) + * Mark sstables as repaired after full repair (CASSANDRA-7586) + * Extend Descriptor to include a format value and refactor reader/writer apis (CASSANDRA-7443) + * Integrate JMH for microbenchmarks (CASSANDRA-8151) + * Keep sstable levels when bootstrapping (CASSANDRA-7460) + * Add Sigar library and perform basic OS settings check on startup (CASSANDRA-7838) + * Support for aggregation functions (CASSANDRA-4914) + * Remove cassandra-cli (CASSANDRA-7920) + * Accept dollar quoted strings in CQL (CASSANDRA-7769) + * Make assassinate a first class command (CASSANDRA-7935) + * Support IN clause on any clustering column (CASSANDRA-4762) + * Improve compaction logging (CASSANDRA-7818) + * Remove YamlFileNetworkTopologySnitch (CASSANDRA-7917) + * Do anticompaction in groups (CASSANDRA-6851) + * Support pure user-defined functions (CASSANDRA-7395, 7526, 7562, 7740, 7781, 7929, + 7924, 7812, 8063, 7813) + * Permit configurable timestamps with cassandra-stress (CASSANDRA-7416) + * Move sstable RandomAccessReader to nio2, which allows using the + FILE_SHARE_DELETE flag on Windows (CASSANDRA-4050) + * Remove CQL2 (CASSANDRA-5918) + * Add Thrift get_multi_slice call (CASSANDRA-6757) + * Optimize fetching multiple cells by name (CASSANDRA-6933) + * Allow compilation in java 8 (CASSANDRA-7028) + * Make incremental repair default (CASSANDRA-7250) + * Enable code coverage thru JaCoCo (CASSANDRA-7226) + * Switch external naming of 'column families' to 'tables' (CASSANDRA-4369) + * Shorten SSTable path (CASSANDRA-6962) + * Use unsafe mutations for most unit tests (CASSANDRA-6969) + * Fix race condition during calculation of pending ranges (CASSANDRA-7390) + * Fail on very large batch sizes (CASSANDRA-8011) - * improve concurrency of repair (CASSANDRA-6455, 8208) ++ * Improve concurrency of repair (CASSANDRA-6455, 8208) ++ + 2.1.3 - * Fix high size calculations for prepared statements (CASSANDRA-8231) * Centralize shared executors (CASSANDRA-8055) * Fix filtering for CONTAINS (KEY) relations on frozen collection clustering columns when the query is restricted to a single
[2/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/service/paxos/PaxosState.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eac7781e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eac7781e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eac7781e
Branch: refs/heads/trunk
Commit: eac7781e7c429ac25b56ddc2ec20bc34f3244df6
Parents: cab2b25 17de36f
Author: Aleksey Yeschenko alek...@apache.org
Authored: Mon Nov 24 16:27:56 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 16:27:56 2014 +0300
--
CHANGES.txt | 1 +
.../cassandra/service/paxos/PaxosState.java | 17 ++-
.../cassandra/service/PaxosStateTest.java | 104 +++
3 files changed, 118 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eac7781e/CHANGES.txt
--
diff --cc CHANGES.txt
index 9db65e9,fe23248..c9e35d5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,5 +1,19 @@@
-2.0.12:
+2.1.3
+ * Fix high size calculations for prepared statements (CASSANDRA-8231)
+ * Centralize shared executors (CASSANDRA-8055)
+ * Fix filtering for CONTAINS (KEY) relations on frozen collection
+ clustering columns when the query is restricted to a single
+ partition (CASSANDRA-8203)
+ * Do more aggressive entire-sstable TTL expiry checks (CASSANDRA-8243)
+ * Add more log info if readMeter is null (CASSANDRA-8238)
+ * add check of the system wall clock time at startup (CASSANDRA-8305)
+ * Support for frozen collections (CASSANDRA-7859)
+ * Fix overflow on histogram computation (CASSANDRA-8028)
+ * Have paxos reuse the timestamp generation of normal queries
(CASSANDRA-7801)
+ * Fix incremental repair not remove parent session on remote (CASSANDRA-8291)
+ * Improve JBOD disk utilization (CASSANDRA-7386)
+Merged from 2.0:
+ * Ignore Paxos commits for truncated tables (CASSANDRA-7538)
* Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories
(CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eac7781e/src/java/org/apache/cassandra/service/paxos/PaxosState.java
--
diff --cc src/java/org/apache/cassandra/service/paxos/PaxosState.java
index abd173c,2adecec..01e03f4
--- a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
+++ b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
@@@ -18,17 -19,19 +18,18 @@@
* under the License.
*
*/
-
+package org.apache.cassandra.service.paxos;
import java.nio.ByteBuffer;
+import java.util.concurrent.locks.Lock;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import com.google.common.util.concurrent.Striped;
import org.apache.cassandra.config.CFMetaData;
-import org.apache.cassandra.db.RowMutation;
-import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.*;
import org.apache.cassandra.tracing.Tracing;
+ import org.apache.cassandra.utils.UUIDGen;
public class PaxosState
{
@@@ -131,10 -133,18 +132,18 @@@
// Committing it is however always safe due to column timestamps,
so always do it. However,
// if our current in-progress ballot is strictly greater than the
proposal one, we shouldn't
// erase the in-progress update.
- Tracing.trace(Committing proposal {}, proposal);
- Mutation mutation = proposal.makeMutation();
- Keyspace.open(mutation.getKeyspaceName()).apply(mutation, true);
-
+ // The table may have been truncated since the proposal was
initiated. In that case, we
+ // don't want to perform the mutation and potentially resurrect
truncated data
+ if (UUIDGen.unixTimestamp(proposal.ballot) =
SystemKeyspace.getTruncatedAt(proposal.update.metadata().cfId))
+ {
+ Tracing.trace(Committing proposal {}, proposal);
-RowMutation rm = proposal.makeMutation();
-Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
++Mutation mutation = proposal.makeMutation();
++Keyspace.open(mutation.getKeyspaceName()).apply(mutation,
true);
+ }
+ else
+ {
+ Tracing.trace(Not committing proposal {} as ballot timestamp
predates last truncation time, proposal);
+ }
[1/2] cassandra git commit: Ignore Paxos commits for truncated tables
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 cab2b25b0 - eac7781e7
Ignore Paxos commits for truncated tables
patch by Sam Tunnicliffe; reviewed by Aleksey Yeschenko for
CASSANDRA-7538
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17de36f2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17de36f2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17de36f2
Branch: refs/heads/cassandra-2.1
Commit: 17de36f246c912287b85eb7015583a35f5040919
Parents: 0e3d9fc
Author: Sam Tunnicliffe s...@beobal.com
Authored: Mon Nov 24 16:07:17 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Mon Nov 24 16:07:17 2014 +0300
--
CHANGES.txt | 1 +
.../cassandra/service/paxos/PaxosState.java | 17 ++-
.../cassandra/service/PaxosStateTest.java | 108 +++
3 files changed, 122 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 412eb59..fe23248 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.12:
+ * Ignore Paxos commits for truncated tables (CASSANDRA-7538)
* Validate size of indexed column values (CASSANDRA-8280)
* Make LCS split compaction results over all data directories (CASSANDRA-8329)
* Fix some failing queries that use multi-column relations
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/src/java/org/apache/cassandra/service/paxos/PaxosState.java
--
diff --git a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
index 0196122..2adecec 100644
--- a/src/java/org/apache/cassandra/service/paxos/PaxosState.java
+++ b/src/java/org/apache/cassandra/service/paxos/PaxosState.java
@@ -31,6 +31,7 @@ import org.apache.cassandra.db.RowMutation;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.SystemKeyspace;
import org.apache.cassandra.tracing.Tracing;
+import org.apache.cassandra.utils.UUIDGen;
public class PaxosState
{
@@ -132,10 +133,18 @@ public class PaxosState
// Committing it is however always safe due to column timestamps,
so always do it. However,
// if our current in-progress ballot is strictly greater than the
proposal one, we shouldn't
// erase the in-progress update.
-Tracing.trace(Committing proposal {}, proposal);
-RowMutation rm = proposal.makeMutation();
-Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
-
+// The table may have been truncated since the proposal was
initiated. In that case, we
+// don't want to perform the mutation and potentially resurrect
truncated data
+if (UUIDGen.unixTimestamp(proposal.ballot) =
SystemKeyspace.getTruncatedAt(proposal.update.metadata().cfId))
+{
+Tracing.trace(Committing proposal {}, proposal);
+RowMutation rm = proposal.makeMutation();
+Keyspace.open(rm.getKeyspaceName()).apply(rm, true);
+}
+else
+{
+Tracing.trace(Not committing proposal {} as ballot timestamp
predates last truncation time, proposal);
+}
// We don't need to lock, we're just blindly updating
SystemKeyspace.savePaxosCommit(proposal);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/17de36f2/test/unit/org/apache/cassandra/service/PaxosStateTest.java
--
diff --git a/test/unit/org/apache/cassandra/service/PaxosStateTest.java
b/test/unit/org/apache/cassandra/service/PaxosStateTest.java
new file mode 100644
index 000..306c424
--- /dev/null
+++ b/test/unit/org/apache/cassandra/service/PaxosStateTest.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222982#comment-14222982 ] Benjamin Lerer commented on CASSANDRA-8231: --- Could you try the following: 1) delete the jar 2) Dowload the jar from: http://search.maven.org/remotecontent?filepath=com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar and put it in the lib folder 3) tell me if you still have the error Wrong size of cached prepared statements Key: CASSANDRA-8231 URL: https://issues.apache.org/jira/browse/CASSANDRA-8231 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jaroslav Kamenik Assignee: Benjamin Lerer Fix For: 2.1.3 Attachments: 8231-notes.txt, CASSANDRA-8231-V2-trunk.txt, CASSANDRA-8231-V2.txt, CASSANDRA-8231.txt, Unsafes.java Cassandra counts memory footprint of prepared statements for caching purposes. It seems, that there is problem with some statements, ie SelectStatement. Even simple selects is counted as 100KB object, updates, deletes etc have few hundreds or thousands bytes. Result is that cache - QueryProcessor.preparedStatements - holds just fraction of statements.. I dig a little into the code, and it seems that problem is in jamm in class MemoryMeter. It seems that if instance contains reference to class, it counts size of whole class too. SelectStatement references EnumSet through ResultSet.Metadata and EnumSet holds reference to Enum class... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8192) AssertionError in Memory.java
[
https://issues.apache.org/jira/browse/CASSANDRA-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222995#comment-14222995
]
Andreas Schnitzerling commented on CASSANDRA-8192:
--
I managed to start 2.1.2 w/finalizer-patch and -Xms1229M -Xmx1229M^. Same
error. Windows Task-Manager shows only 308MB used after that error. So one node
w/ 1GB jvm set no error (my post before) and another same node w/1,2GB jvm set
throws that error.
AssertionError in Memory.java
-
Key: CASSANDRA-8192
URL: https://issues.apache.org/jira/browse/CASSANDRA-8192
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Windows-7-32 bit, 3GB RAM, Java 1.7.0_67
Reporter: Andreas Schnitzerling
Assignee: Joshua McKenzie
Attachments: cassandra.bat, cassandra.yaml, system.log
Since update of 1 of 12 nodes from 2.1.0-rel to 2.1.1-rel Exception during
start up.
{panel:title=system.log}
ERROR [SSTableBatchOpen:1] 2014-10-27 09:44:00,079 CassandraDaemon.java:153 -
Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.compress.CompressionMetadata.init(CompressionMetadata.java:135)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438)
~[apache-cassandra-2.1.1.jar:2.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_55]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_55]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
{panel}
In the attached log you can still see as well CASSANDRA-8069 and
CASSANDRA-6283.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8367) Clash between Cassandra and Crunch mapreduce config
Radovan Zvoncek created CASSANDRA-8367: -- Summary: Clash between Cassandra and Crunch mapreduce config Key: CASSANDRA-8367 URL: https://issues.apache.org/jira/browse/CASSANDRA-8367 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Radovan Zvoncek Priority: Minor We would like to use Cassandra's (Cql)BulkOutputFormats to implement Resource IOs for Crunch. We want to do this to allow Crunch users write results of their jobs directly to Cassandra (thus skipping writing them to file system). In the process of doing this, we found out there is a clash in the mapreduce job config. The affected config key is 'mapreduce.output.basename'. Cassandra is using it [1] for something different than Crunch [2]. This is resulting in some obscure behavior I personally don't understand, but it causes the jobs to fail. We went ahead and re-implemented the output format classes to use different config key, but we'd very much like to stop using them. [1] https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hadoop/ConfigHelper.java#L54 [2] https://github.com/apache/crunch/blob/3f13ee65c9debcf6bd7366607f58beae6c73ffe2/crunch-core/src/main/java/org/apache/crunch/io/CrunchOutputs.java#L99 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity
Aleksey Yeschenko created CASSANDRA-8368: Summary: Consider not using hints for batchlog replay, in any capacity Key: CASSANDRA-8368 URL: https://issues.apache.org/jira/browse/CASSANDRA-8368 Project: Cassandra Issue Type: Bug Reporter: Aleksey Yeschenko Fix For: 3.0 Currently, when replaying a batch, if a request times out, we simply write a hint for it and call it a day. It's simple, but it does tie us to hints, which some people prefer to disable altogether (and some still will even after CASSANDRA-6230). It also potentially violates the consistency level of the original request. As an alternative, once CASSANDRA-7237 is complete, I suggest we stop relying on hints at all, and do this instead: 1. Store the consistency level as batch metadata 2. On replay, hint in case of a timeout, but not if the node is down as per FD 3. If CL is met, consider the batch replayed and discard it, but not account the hints towards CL (as per usual write patch), unless CL.ANY is being used 4. If CL is *not* met, write a new batch with contents of the current one, but with timeuuid set in the future, for later replay (delayed by fixed configurable time or exponentially backed off). With that new batch store the list of nodes we've delivered the hint to, so that next time we replay it we don't waste writes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8267) Only stream from unrepaired sstables during incremental repair
[ https://issues.apache.org/jira/browse/CASSANDRA-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223034#comment-14223034 ] Marcus Eriksson commented on CASSANDRA-8267: To solve this we need to tell a node whether or not this is an incremental repair when requesting ranges from it. This breaks streaming message versioning, meaning we would not be able to stream between two nodes unless they were both upgraded, which would suck in a minor release. One solution could be to only break streaming for incremental repairs (when they are initiated on an upgraded node) by adding a new IncrementalStreamRequest message and failing early if we notice that not all endpoints included in the incremental repair are upgraded. This would make old-style repairs still work since they don't use the new message (and full repairs are the default in 2.1). WDYT [~yukim]? Would this be acceptable or do you have a better solution? We kind of have to fix this in 2.1 since it makes incremental repairs quite bad. Only stream from unrepaired sstables during incremental repair -- Key: CASSANDRA-8267 URL: https://issues.apache.org/jira/browse/CASSANDRA-8267 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1.3 Seems we stream from all sstables even if we do incremental repair, we should limit this to only stream from the unrepaired sstables if we do incremental repair -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7949) LCS compaction low performance, many pending compactions, nodes are almost idle
[ https://issues.apache.org/jira/browse/CASSANDRA-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223036#comment-14223036 ] Nikolai Grigoriev commented on CASSANDRA-7949: -- I have recently realized that there may be relatively cheap (operationally and development-wise) workaround for that limitation. It would also partially address the problem with bootstrapping new node. The root cause of all this is a large amount of data in a single CF on a single node when using LCS for that CF. The performance of a single compaction task running on a single thread is limited anyway. One of the obvious ways to break this limitation is to shard the data across multiple clones of that CF at the application level. Something as dumb as row key hash mod X and add this suffix to the CF name. In my case looks like having X=4 would be more than enough to solve the problem. LCS compaction low performance, many pending compactions, nodes are almost idle --- Key: CASSANDRA-7949 URL: https://issues.apache.org/jira/browse/CASSANDRA-7949 Project: Cassandra Issue Type: Bug Components: Core Environment: DSE 4.5.1-1, Cassandra 2.0.8 Reporter: Nikolai Grigoriev Attachments: iostats.txt, nodetool_compactionstats.txt, nodetool_tpstats.txt, pending compactions 2day.png, system.log.gz, vmstat.txt I've been evaluating new cluster of 15 nodes (32 core, 6x800Gb SSD disks + 2x600Gb SAS, 128Gb RAM, OEL 6.5) and I've built a simulator that creates the load similar to the load in our future product. Before running the simulator I had to pre-generate enough data. This was done using Java code and DataStax Java driver. To avoid going deep into details, two tables have been generated. Each table currently has about 55M rows and between few dozens and few thousands of columns in each row. This data generation process was generating massive amount of non-overlapping data. Thus, the activity was write-only and highly parallel. This is not the type of the traffic that the system will have ultimately to deal with, it will be mix of reads and updates to the existing data in the future. This is just to explain the choice of LCS, not mentioning the expensive SSD disk space. At some point while generating the data I have noticed that the compactions started to pile up. I knew that I was overloading the cluster but I still wanted the genration test to complete. I was expecting to give the cluster enough time to finish the pending compactions and get ready for real traffic. However, after the storm of write requests have been stopped I have noticed that the number of pending compactions remained constant (and even climbed up a little bit) on all nodes. After trying to tune some parameters (like setting the compaction bandwidth cap to 0) I have noticed a strange pattern: the nodes were compacting one of the CFs in a single stream using virtually no CPU and no disk I/O. This process was taking hours. After that it would be followed by a short burst of few dozens of compactions running in parallel (CPU at 2000%, some disk I/O - up to 10-20%) and then getting stuck again for many hours doing one compaction at time. So it looks like this: # nodetool compactionstats pending tasks: 3351 compaction typekeyspace table completed total unit progress Compaction myks table_list1 66499295588 1910515889913 bytes 3.48% Active compaction remaining time :n/a # df -h ... /dev/sdb1.5T 637G 854G 43% /cassandra-data/disk1 /dev/sdc1.5T 425G 1.1T 29% /cassandra-data/disk2 /dev/sdd1.5T 429G 1.1T 29% /cassandra-data/disk3 # find . -name **table_list1**Data** | grep -v snapshot | wc -l 1310 Among these files I see: 1043 files of 161Mb (my sstable size is 160Mb) 9 large files - 3 between 1 and 2Gb, 3 of 5-8Gb, 55Gb, 70Gb and 370Gb 263 files of various sized - between few dozens of Kb and 160Mb I've been running the heavy load for about 1,5days and it's been close to 3 days after that and the number of pending compactions does not go down. I have applied one of the not-so-obvious recommendations to disable multithreaded compactions and that seems to be helping a bit - I see some nodes started to have fewer pending compactions. About half of the cluster, in fact. But even there I see they are sitting idle most of the time lazily compacting in one stream with CPU at ~140% and occasionally doing the bursts of compaction work for few minutes. I am wondering if this is really a bug or something in the LCS logic that would manifest itself only in such an edge case scenario where I have loaded lots
[jira] [Commented] (CASSANDRA-7874) Validate functionality of different JSR-223 providers in UDFs
[ https://issues.apache.org/jira/browse/CASSANDRA-7874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223046#comment-14223046 ] Joshua McKenzie commented on CASSANDRA-7874: +1 from me on the Windows changes Validate functionality of different JSR-223 providers in UDFs - Key: CASSANDRA-7874 URL: https://issues.apache.org/jira/browse/CASSANDRA-7874 Project: Cassandra Issue Type: Task Components: Core Reporter: Robert Stupp Assignee: Robert Stupp Labels: udf Fix For: 3.0 Attachments: 7874.txt, 7874v2.txt, 7874v3.txt, 7874v4.txt, 7874v5.txt, 7874v6.txt CASSANDRA-7526 introduces ability to support optional JSR-223 providers like Clojure, Jython, Groovy or JRuby. This ticket is about to test functionality with these providers but not to include them in C* distribution. Expected result is a how to document, wiki page or similar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8192) AssertionError in Memory.java
[
https://issues.apache.org/jira/browse/CASSANDRA-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223047#comment-14223047
]
Joshua McKenzie commented on CASSANDRA-8192:
I assume a 32-bit JVM due to the heap size limitation. Were you able to test
out the 64-bit environment w/3G heap outside of Upsource?
AssertionError in Memory.java
-
Key: CASSANDRA-8192
URL: https://issues.apache.org/jira/browse/CASSANDRA-8192
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Windows-7-32 bit, 3GB RAM, Java 1.7.0_67
Reporter: Andreas Schnitzerling
Assignee: Joshua McKenzie
Attachments: cassandra.bat, cassandra.yaml, system.log
Since update of 1 of 12 nodes from 2.1.0-rel to 2.1.1-rel Exception during
start up.
{panel:title=system.log}
ERROR [SSTableBatchOpen:1] 2014-10-27 09:44:00,079 CassandraDaemon.java:153 -
Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.compress.CompressionMetadata.init(CompressionMetadata.java:135)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438)
~[apache-cassandra-2.1.1.jar:2.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_55]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_55]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
{panel}
In the attached log you can still see as well CASSANDRA-8069 and
CASSANDRA-6283.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8369) Better error handling in CQLSH for invalid password
Johnny Miller created CASSANDRA-8369: Summary: Better error handling in CQLSH for invalid password Key: CASSANDRA-8369 URL: https://issues.apache.org/jira/browse/CASSANDRA-8369 Project: Cassandra Issue Type: Improvement Reporter: Johnny Miller Priority: Minor On C* 2.0.11/Cqlsh 4.1.1 when logging with an invalid password you get back a stacktrace rather than a more user friendly error. It might be better if this was more user friendly. For example - this is what you get back now: root@cass1:~# cqlsh -u cassandra -p johnny Traceback (most recent call last): File /usr/bin/cqlsh, line 2113, in module main(*read_options(sys.argv[1:], os.environ)) File /usr/bin/cqlsh, line 2093, in main single_statement=options.execute) File /usr/bin/cqlsh, line 505, in __init__ password=password, cql_version=cqlver, transport=transport) File /usr/share/dse/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/connection.py, line 143, in connect File /usr/share/dse/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/connection.py, line 59, in __init__ File /usr/share/dse/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/thrifteries.py, line 157, in establish_connection File /usr/share/dse/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py, line 465, in login File /usr/share/dse/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py, line 486, in recv_login cql.cassandra.ttypes.AuthenticationException: AuthenticationException(why='Username and/or password are incorrect') -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8168) Require Java 8
[ https://issues.apache.org/jira/browse/CASSANDRA-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223062#comment-14223062 ] T Jake Luciani commented on CASSANDRA-8168: --- bq. And that CompletableFuture gives us something that guava's ListenableFuture don't give us? Agreed. I do think lambas help make this code more maintainable but not required. One thing I've learned is java 8 should give us some performance boost due to improvements in CAS contention improvements. We use these types extensively on the hot path http://ashkrit.blogspot.com/2014/02/atomicinteger-java-7-vs-java-8.html Require Java 8 -- Key: CASSANDRA-8168 URL: https://issues.apache.org/jira/browse/CASSANDRA-8168 Project: Cassandra Issue Type: Task Reporter: T Jake Luciani Assignee: T Jake Luciani Fix For: 3.0 This is to discuss requiring Java 8 for version = 3.0 There are a couple big reasons for this. * Better support for complex async work e.g (CASSANDRA-5239) http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html * Use Nashorn for Javascript UDFs CASSANDRA-7395 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8192) AssertionError in Memory.java
[
https://issues.apache.org/jira/browse/CASSANDRA-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223070#comment-14223070
]
Andreas Ländle commented on CASSANDRA-8192:
---
At least I'm using a 64-bit JVM JDK 7u60. For now I tested to run cassandra
with 4GB heap-size (instead 3GB before) and at least until now I couldn't
reproduce the error.
AssertionError in Memory.java
-
Key: CASSANDRA-8192
URL: https://issues.apache.org/jira/browse/CASSANDRA-8192
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Windows-7-32 bit, 3GB RAM, Java 1.7.0_67
Reporter: Andreas Schnitzerling
Assignee: Joshua McKenzie
Attachments: cassandra.bat, cassandra.yaml, system.log
Since update of 1 of 12 nodes from 2.1.0-rel to 2.1.1-rel Exception during
start up.
{panel:title=system.log}
ERROR [SSTableBatchOpen:1] 2014-10-27 09:44:00,079 CassandraDaemon.java:153 -
Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.compress.CompressionMetadata.init(CompressionMetadata.java:135)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302)
~[apache-cassandra-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438)
~[apache-cassandra-2.1.1.jar:2.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[na:1.7.0_55]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_55]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
{panel}
In the attached log you can still see as well CASSANDRA-8069 and
CASSANDRA-6283.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8285) OOME in Cassandra 2.0.11
[ https://issues.apache.org/jira/browse/CASSANDRA-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223083#comment-14223083 ] Pierre Laporte commented on CASSANDRA-8285: --- I have the issue after ~1.5 day on the endurance test of java-driver 2.1.3 against 2.0.12. Please find the associated heap dump [here|https://drive.google.com/open?id=0BxvGkaXP3ayeOElqY1ZNQTlBNTgauthuser=1] OOME in Cassandra 2.0.11 Key: CASSANDRA-8285 URL: https://issues.apache.org/jira/browse/CASSANDRA-8285 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.11 + java-driver 2.0.8-SNAPSHOT Cassandra 2.0.11 + ruby-driver 1.0-beta Reporter: Pierre Laporte Assignee: Aleksey Yeschenko Attachments: OOME_node_system.log, gc.log.gz, heap-usage-after-gc-zoom.png, heap-usage-after-gc.png We ran drivers 3-days endurance tests against Cassandra 2.0.11 and C* crashed with an OOME. This happened both with ruby-driver 1.0-beta and java-driver 2.0.8-snapshot. Attached are : | OOME_node_system.log | The system.log of one Cassandra node that crashed | | gc.log.gz | The GC log on the same node | | heap-usage-after-gc.png | The heap occupancy evolution after every GC cycle | | heap-usage-after-gc-zoom.png | A focus on when things start to go wrong | Workload : Our test executes 5 CQL statements (select, insert, select, delete, select) for a given unique id, during 3 days, using multiple threads. There is not change in the workload during the test. Symptoms : In the attached log, it seems something starts in Cassandra between 2014-11-06 10:29:22 and 2014-11-06 10:45:32. This causes an allocation that fills the heap. We eventually get stuck in a Full GC storm and get an OOME in the logs. I have run the java-driver tests against Cassandra 1.2.19 and 2.1.1. The error does not occur. It seems specific to 2.0.11. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8332) Null pointer after droping keyspace
[
https://issues.apache.org/jira/browse/CASSANDRA-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-8332:
--
Fix Version/s: 2.0.12
Null pointer after droping keyspace
---
Key: CASSANDRA-8332
URL: https://issues.apache.org/jira/browse/CASSANDRA-8332
Project: Cassandra
Issue Type: Bug
Reporter: Chris Lohfink
Assignee: T Jake Luciani
Priority: Minor
Fix For: 2.0.12, 2.1.3
Attachments: 8332.txt, CassandraStressTest-8332.zip
After dropping keyspace, sometimes I see this in logs:
{code}
ERROR 03:40:29 Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: null
at
org.apache.cassandra.io.compress.CompressionParameters.setLiveMetadata(CompressionParameters.java:108)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadata(SSTableReader.java:1142)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1896)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:68)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1681)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1693)
~[main/:na]
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:181)
~[main/:na]
at
org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getScanners(WrappingCompactionStrategy.java:320)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:340)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:151)
~[main/:na]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:233)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
{code}
Minor issue since doesn't really affect anything, but the error makes it look
like somethings wrong. Seen on 2.1 branch
(1b21aef8152d96a180e75f2fcc5afad9ded6c595), not sure how far back (may be
post 2.1.2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8332) Null pointer after droping keyspace
[
https://issues.apache.org/jira/browse/CASSANDRA-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-8332:
--
Attachment: 8332.txt
Null pointer after droping keyspace
---
Key: CASSANDRA-8332
URL: https://issues.apache.org/jira/browse/CASSANDRA-8332
Project: Cassandra
Issue Type: Bug
Reporter: Chris Lohfink
Assignee: T Jake Luciani
Priority: Minor
Fix For: 2.0.12, 2.1.3
Attachments: 8332.txt, CassandraStressTest-8332.zip
After dropping keyspace, sometimes I see this in logs:
{code}
ERROR 03:40:29 Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: null
at
org.apache.cassandra.io.compress.CompressionParameters.setLiveMetadata(CompressionParameters.java:108)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadata(SSTableReader.java:1142)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1896)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:68)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1681)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1693)
~[main/:na]
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:181)
~[main/:na]
at
org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getScanners(WrappingCompactionStrategy.java:320)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:340)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:151)
~[main/:na]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:233)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
{code}
Minor issue since doesn't really affect anything, but the error makes it look
like somethings wrong. Seen on 2.1 branch
(1b21aef8152d96a180e75f2fcc5afad9ded6c595), not sure how far back (may be
post 2.1.2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8231) Wrong size of cached prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223140#comment-14223140 ] Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-8231: - Hi Benjamin, Still the problem persists. Here are the steps I have gone through. Rajanarayanans-MacBook-Pro:lib RajT$ rm jamm-0.3.0.jar Rajanarayanans-MacBook-Pro:lib RajT$ wget http://search.maven.org/remotecontent?filepath=com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar --2014-11-24 16:24:19-- http://search.maven.org/remotecontent?filepath=com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar Resolving search.maven.org... 207.223.241.72 Connecting to search.maven.org|207.223.241.72|:80... connected. HTTP request sent, awaiting response... 302 Moved Temporarily Location: https://repo1.maven.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar [following] --2014-11-24 16:24:19-- https://repo1.maven.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar Resolving repo1.maven.org... 185.31.18.209 Connecting to repo1.maven.org|185.31.18.209|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 21033 (21K) [application/java-archive] Saving to: 'remotecontent?filepath=com%2Fgithub%2Fjbellis%2Fjamm%2F0.3.0%2Fjamm-0.3.0.jar' 100%[] 21,033 --.-K/s in 0.04s 2014-11-24 16:24:20 (582 KB/s) - 'remotecontent?filepath=com%2Fgithub%2Fjbellis%2Fjamm%2F0.3.0%2Fjamm-0.3.0.jar' saved [21033/21033] Rajanarayanans-MacBook-Pro:lib RajT$ mv remotecontent?filepath=com%2Fgithub%2Fjbellis%2Fjamm%2F0.3.0%2Fjamm-0.3.0.jar jamm-0.3.0.jar Rajanarayanans-MacBook-Pro:lib RajT$ cd .. Rajanarayanans-MacBook-Pro:cassandra-trunk RajT$ ant test -Dtest.name=CleanupTest Buildfile: /Users/RajT/cassandra-source/cassandra-trunk/build.xml init: maven-ant-tasks-localrepo: maven-ant-tasks-download: maven-ant-tasks-init: maven-declare-dependencies: maven-ant-tasks-retrieve-build: init-dependencies: [echo] Loading dependency paths from file: /Users/RajT/cassandra-source/cassandra-trunk/build/build-dependencies.xml [unzip] Expanding: /Users/RajT/cassandra-source/cassandra-trunk/build/lib/jars/org.jacoco.agent-0.7.1.201405082137.jar into /Users/RajT/cassandra-source/cassandra-trunk/build/lib/jars check-gen-cql3-grammar: gen-cql3-grammar: build-project: [echo] apache-cassandra: /Users/RajT/cassandra-source/cassandra-trunk/build.xml createVersionPropFile: [propertyfile] Updating property file: /Users/RajT/cassandra-source/cassandra-trunk/src/resources/org/apache/cassandra/config/version.properties [copy] Copying 1 file to /Users/RajT/cassandra-source/cassandra-trunk/build/classes/main build: build-test: test: testlist: [echo] running test bucket 0 tests [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/usr/local/Cellar/ant/1.9.4/libexec/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/Users/RajT/cassandra-source/cassandra-trunk/build/lib/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Error occurred during initialization of VM [junit] agent library failed to init: instrument [junit] objc[7769]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/jre/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined. [junit] Error opening zip file or JAR manifest missing : /Users/RajT/cassandra-source/cassandra-trunk/lib/jamm-0.3.0.jar [junit] Testsuite: org.apache.cassandra.db.CleanupTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec [junit] [junit] Testcase: org.apache.cassandra.db.CleanupTest:null: Caused an ERROR [junit] Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit. [junit] junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit. [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.CleanupTest FAILED (crashed) [junitreport] Processing /Users/RajT/cassandra-source/cassandra-trunk/build/test/TESTS-TestSuites.xml to /var/folders/nf/trtmyt9534z03kq8p8zgbnxhgn/T/null1829125363 [junitreport] Loading stylesheet jar:file:/usr/local/Cellar/ant/1.9.4/libexec/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl [junitreport] Transform time: 2046ms [junitreport] Deleting: /var/folders/nf/trtmyt9534z03kq8p8zgbnxhgn/T/null1829125363 BUILD FAILED
[jira] [Updated] (CASSANDRA-8061) tmplink files are not removed
[
https://issues.apache.org/jira/browse/CASSANDRA-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua McKenzie updated CASSANDRA-8061:
---
Attachment: 8061_v1.txt
[~enigmacurry] [~mshuler] Have we been able to reproduce this?
From tracing through the code I see a couple of places where it looks like we
could potentially leak tmplink files in SSTableWriter.openEarly. We create
the hard links, grab an sstablereader w/those internally, and then if our
iwriter.getMaxReadableKey comes back null, immediately return null from the
method without calling releaseReference on the SSTR. I've added a
releaseReference call in there to prevent that.
I've also converted SSTableRewriter to implement AutoClosable as the current
pattern of having to manually finish or abort is prone to error, though a
manual inspection of our usage doesn't uncover any errors. I've also tightened
up the finish() method by protecting against duplicate usage, removed the
redundant / premature optimization of tracking early opened SSTableReaders
separately from their SSTableWriters (prevented construction of list on abort
path), and did a little renaming cleanup.
While the logic to track tmplink / early opened SSTR's and release references
to them is complex it appears to be sound from my first inspection - either the
finish() or abort() path should correctly remove those files.
I'm setting up a long running test with the above schema from Alexander to see
if I can replicate this locally.
v1 attached with the above changes. If anyone has a reproduction of this
problem and could take a spin with this patch, that would be great!
tmplink files are not removed
-
Key: CASSANDRA-8061
URL: https://issues.apache.org/jira/browse/CASSANDRA-8061
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Linux
Reporter: Gianluca Borello
Assignee: Joshua McKenzie
Priority: Critical
Fix For: 2.1.3
Attachments: 8061_v1.txt, 8248-thread_dump.txt
After installing 2.1.0, I'm experiencing a bunch of tmplink files that are
filling my disk. I found https://issues.apache.org/jira/browse/CASSANDRA-7803
and that is very similar, and I confirm it happens both on 2.1.0 as well as
from the latest commit on the cassandra-2.1 branch
(https://github.com/apache/cassandra/commit/aca80da38c3d86a40cc63d9a122f7d45258e4685
from the cassandra-2.1)
Even starting with a clean keyspace, after a few hours I get:
{noformat}
$ sudo find /raid0 | grep tmplink | xargs du -hs
2.7G
/raid0/cassandra/data/draios/protobuf1-ccc6dce04beb11e4abf997b38fbf920b/draios-protobuf1-tmplink-ka-4515-Data.db
13M
/raid0/cassandra/data/draios/protobuf1-ccc6dce04beb11e4abf997b38fbf920b/draios-protobuf1-tmplink-ka-4515-Index.db
1.8G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-1788-Data.db
12M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-1788-Index.db
5.2M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-2678-Index.db
822M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-2678-Data.db
7.3M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3283-Index.db
1.2G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3283-Data.db
6.7M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3951-Index.db
1.1G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3951-Data.db
11M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-4799-Index.db
1.7G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-4799-Data.db
812K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-234-Index.db
122M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-208-Data.db
744K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-739-Index.db
660K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-193-Index.db
796K
[jira] [Commented] (CASSANDRA-7124) Use JMX Notifications to Indicate Success/Failure of Long-Running Operations
[
https://issues.apache.org/jira/browse/CASSANDRA-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223153#comment-14223153
]
Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-7124:
-
[~yukim], Once you have a look at the patch attached and if the methodology is
right, I can start working on the other ones like compact, decommission, move,
relocate etc. Thanks
Use JMX Notifications to Indicate Success/Failure of Long-Running Operations
Key: CASSANDRA-7124
URL: https://issues.apache.org/jira/browse/CASSANDRA-7124
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Rajanarayanan Thottuvaikkatumana
Priority: Minor
Labels: lhf
Fix For: 3.0
Attachments: cassandra-trunk-cleanup-7124.txt
If {{nodetool cleanup}} or some other long-running operation takes too long
to complete, you'll see an error like the one in CASSANDRA-2126, so you can't
tell if the operation completed successfully or not. CASSANDRA-4767 fixed
this for repairs with JMX notifications. We should do something similar for
nodetool cleanup, compact, decommission, move, relocate, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8168) Require Java 8
[
https://issues.apache.org/jira/browse/CASSANDRA-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223159#comment-14223159
]
Dave Brosius commented on CASSANDRA-8168:
-
{quote}
One thing I've learned is java 8 should give us some performance boost due to
improvements in CAS contention improvements. We use these types extensively on
the hot path
http://ashkrit.blogspot.com/2014/02/atomicinteger-java-7-vs-java-8.html
{quote}
Yes but isn't that the JVM doing that? Compiling with 8 shouldn't effect that.
I'm not a big fan of going to 8 at the moment, to me, just because i'm not sure
the added value outweighs the pain you add to users.
Require Java 8
--
Key: CASSANDRA-8168
URL: https://issues.apache.org/jira/browse/CASSANDRA-8168
Project: Cassandra
Issue Type: Task
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 3.0
This is to discuss requiring Java 8 for version = 3.0
There are a couple big reasons for this.
* Better support for complex async work e.g (CASSANDRA-5239)
http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
* Use Nashorn for Javascript UDFs CASSANDRA-7395
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8370) cqlsh doesn't handle LIST statements correctly
Sam Tunnicliffe created CASSANDRA-8370:
--
Summary: cqlsh doesn't handle LIST statements correctly
Key: CASSANDRA-8370
URL: https://issues.apache.org/jira/browse/CASSANDRA-8370
Project: Cassandra
Issue Type: Bug
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Priority: Minor
Fix For: 2.1.3
{{LIST USERS}} and {{LIST PERMISSIONS}} statements are not handled correctly by
cqlsh in 2.1 (since CASSANDRA-6307).
Running such a query results in errors along the lines of:
{{noformat}}
sam@easy:~/projects/cassandra$ bin/cqlsh --debug -u cassandra -p cassandra
Using CQL driver: module 'cassandra' from
'/home/sam/projects/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.2.zip/cassandra-driver-2.1.2/cassandra/__init__.py'
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh list users;
Traceback (most recent call last):
File bin/cqlsh, line 879, in onecmd
self.handle_statement(st, statementtext)
File bin/cqlsh, line 920, in handle_statement
return self.perform_statement(cqlruleset.cql_extract_orig(tokens, srcstr))
File bin/cqlsh, line 953, in perform_statement
result = self.perform_simple_statement(stmt)
File bin/cqlsh, line 989, in perform_simple_statement
self.print_result(rows, self.parse_for_table_meta(statement.query_string))
File bin/cqlsh, line 970, in parse_for_table_meta
return self.get_table_meta(ks, cf)
File bin/cqlsh, line 732, in get_table_meta
ksmeta = self.get_keyspace_meta(ksname)
File bin/cqlsh, line 717, in get_keyspace_meta
raise KeyspaceNotFound('Keyspace %r not found.' % ksname)
KeyspaceNotFound: Keyspace None not found.
{{noformat}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8370) cqlsh doesn't handle LIST statements correctly
[
https://issues.apache.org/jira/browse/CASSANDRA-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-8370:
---
Attachment: 8370.txt
Attached a trivial patch opened dtest PR:
https://github.com/riptano/cassandra-dtest/pull/120
cqlsh doesn't handle LIST statements correctly
--
Key: CASSANDRA-8370
URL: https://issues.apache.org/jira/browse/CASSANDRA-8370
Project: Cassandra
Issue Type: Bug
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Priority: Minor
Labels: cqlsh
Fix For: 2.1.3
Attachments: 8370.txt
{{LIST USERS}} and {{LIST PERMISSIONS}} statements are not handled correctly
by cqlsh in 2.1 (since CASSANDRA-6307).
Running such a query results in errors along the lines of:
{{noformat}}
sam@easy:~/projects/cassandra$ bin/cqlsh --debug -u cassandra -p cassandra
Using CQL driver: module 'cassandra' from
'/home/sam/projects/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.2.zip/cassandra-driver-2.1.2/cassandra/__init__.py'
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh list users;
Traceback (most recent call last):
File bin/cqlsh, line 879, in onecmd
self.handle_statement(st, statementtext)
File bin/cqlsh, line 920, in handle_statement
return self.perform_statement(cqlruleset.cql_extract_orig(tokens, srcstr))
File bin/cqlsh, line 953, in perform_statement
result = self.perform_simple_statement(stmt)
File bin/cqlsh, line 989, in perform_simple_statement
self.print_result(rows, self.parse_for_table_meta(statement.query_string))
File bin/cqlsh, line 970, in parse_for_table_meta
return self.get_table_meta(ks, cf)
File bin/cqlsh, line 732, in get_table_meta
ksmeta = self.get_keyspace_meta(ksname)
File bin/cqlsh, line 717, in get_keyspace_meta
raise KeyspaceNotFound('Keyspace %r not found.' % ksname)
KeyspaceNotFound: Keyspace None not found.
{{noformat}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8370) cqlsh doesn't handle LIST statements correctly
[
https://issues.apache.org/jira/browse/CASSANDRA-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-8370:
---
Reviewer: Mikhail Stepura
Reproduced In: 2.1.2, 2.1.1, 2.1.0 (was: 2.1.0, 2.1.1, 2.1.2)
cqlsh doesn't handle LIST statements correctly
--
Key: CASSANDRA-8370
URL: https://issues.apache.org/jira/browse/CASSANDRA-8370
Project: Cassandra
Issue Type: Bug
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Priority: Minor
Labels: cqlsh
Fix For: 2.1.3
Attachments: 8370.txt
{{LIST USERS}} and {{LIST PERMISSIONS}} statements are not handled correctly
by cqlsh in 2.1 (since CASSANDRA-6307).
Running such a query results in errors along the lines of:
{{noformat}}
sam@easy:~/projects/cassandra$ bin/cqlsh --debug -u cassandra -p cassandra
Using CQL driver: module 'cassandra' from
'/home/sam/projects/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.2.zip/cassandra-driver-2.1.2/cassandra/__init__.py'
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh list users;
Traceback (most recent call last):
File bin/cqlsh, line 879, in onecmd
self.handle_statement(st, statementtext)
File bin/cqlsh, line 920, in handle_statement
return self.perform_statement(cqlruleset.cql_extract_orig(tokens, srcstr))
File bin/cqlsh, line 953, in perform_statement
result = self.perform_simple_statement(stmt)
File bin/cqlsh, line 989, in perform_simple_statement
self.print_result(rows, self.parse_for_table_meta(statement.query_string))
File bin/cqlsh, line 970, in parse_for_table_meta
return self.get_table_meta(ks, cf)
File bin/cqlsh, line 732, in get_table_meta
ksmeta = self.get_keyspace_meta(ksname)
File bin/cqlsh, line 717, in get_keyspace_meta
raise KeyspaceNotFound('Keyspace %r not found.' % ksname)
KeyspaceNotFound: Keyspace None not found.
{{noformat}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8370) cqlsh doesn't handle LIST statements correctly
[
https://issues.apache.org/jira/browse/CASSANDRA-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-8370:
---
Description:
{{LIST USERS}} and {{LIST PERMISSIONS}} statements are not handled correctly by
cqlsh in 2.1 (since CASSANDRA-6307).
Running such a query results in errors along the lines of:
{noformat}
sam@easy:~/projects/cassandra$ bin/cqlsh --debug -u cassandra -p cassandra
Using CQL driver: module 'cassandra' from
'/home/sam/projects/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.2.zip/cassandra-driver-2.1.2/cassandra/__init__.py'
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh list users;
Traceback (most recent call last):
File bin/cqlsh, line 879, in onecmd
self.handle_statement(st, statementtext)
File bin/cqlsh, line 920, in handle_statement
return self.perform_statement(cqlruleset.cql_extract_orig(tokens, srcstr))
File bin/cqlsh, line 953, in perform_statement
result = self.perform_simple_statement(stmt)
File bin/cqlsh, line 989, in perform_simple_statement
self.print_result(rows, self.parse_for_table_meta(statement.query_string))
File bin/cqlsh, line 970, in parse_for_table_meta
return self.get_table_meta(ks, cf)
File bin/cqlsh, line 732, in get_table_meta
ksmeta = self.get_keyspace_meta(ksname)
File bin/cqlsh, line 717, in get_keyspace_meta
raise KeyspaceNotFound('Keyspace %r not found.' % ksname)
KeyspaceNotFound: Keyspace None not found.
{noformat}
was:
{{LIST USERS}} and {{LIST PERMISSIONS}} statements are not handled correctly by
cqlsh in 2.1 (since CASSANDRA-6307).
Running such a query results in errors along the lines of:
{{noformat}}
sam@easy:~/projects/cassandra$ bin/cqlsh --debug -u cassandra -p cassandra
Using CQL driver: module 'cassandra' from
'/home/sam/projects/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.2.zip/cassandra-driver-2.1.2/cassandra/__init__.py'
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh list users;
Traceback (most recent call last):
File bin/cqlsh, line 879, in onecmd
self.handle_statement(st, statementtext)
File bin/cqlsh, line 920, in handle_statement
return self.perform_statement(cqlruleset.cql_extract_orig(tokens, srcstr))
File bin/cqlsh, line 953, in perform_statement
result = self.perform_simple_statement(stmt)
File bin/cqlsh, line 989, in perform_simple_statement
self.print_result(rows, self.parse_for_table_meta(statement.query_string))
File bin/cqlsh, line 970, in parse_for_table_meta
return self.get_table_meta(ks, cf)
File bin/cqlsh, line 732, in get_table_meta
ksmeta = self.get_keyspace_meta(ksname)
File bin/cqlsh, line 717, in get_keyspace_meta
raise KeyspaceNotFound('Keyspace %r not found.' % ksname)
KeyspaceNotFound: Keyspace None not found.
{{noformat}}
cqlsh doesn't handle LIST statements correctly
--
Key: CASSANDRA-8370
URL: https://issues.apache.org/jira/browse/CASSANDRA-8370
Project: Cassandra
Issue Type: Bug
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Priority: Minor
Labels: cqlsh
Fix For: 2.1.3
Attachments: 8370.txt
{{LIST USERS}} and {{LIST PERMISSIONS}} statements are not handled correctly
by cqlsh in 2.1 (since CASSANDRA-6307).
Running such a query results in errors along the lines of:
{noformat}
sam@easy:~/projects/cassandra$ bin/cqlsh --debug -u cassandra -p cassandra
Using CQL driver: module 'cassandra' from
'/home/sam/projects/cassandra/bin/../lib/cassandra-driver-internal-only-2.1.2.zip/cassandra-driver-2.1.2/cassandra/__init__.py'
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.2-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh list users;
Traceback (most recent call last):
File bin/cqlsh, line 879, in onecmd
self.handle_statement(st, statementtext)
File bin/cqlsh, line 920, in handle_statement
return self.perform_statement(cqlruleset.cql_extract_orig(tokens, srcstr))
File bin/cqlsh, line 953, in perform_statement
result = self.perform_simple_statement(stmt)
File bin/cqlsh, line 989, in perform_simple_statement
self.print_result(rows, self.parse_for_table_meta(statement.query_string))
File bin/cqlsh, line 970, in parse_for_table_meta
return self.get_table_meta(ks, cf)
File bin/cqlsh, line 732, in get_table_meta
ksmeta = self.get_keyspace_meta(ksname)
File bin/cqlsh, line 717, in get_keyspace_meta
raise KeyspaceNotFound('Keyspace %r not found.' % ksname)
KeyspaceNotFound: Keyspace
[jira] [Commented] (CASSANDRA-8365) CamelCase name is used as index name instead of lowercase
[
https://issues.apache.org/jira/browse/CASSANDRA-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223214#comment-14223214
]
Philip Thompson commented on CASSANDRA-8365:
What Cassandra version were you running [~pingtimeout]
CamelCase name is used as index name instead of lowercase
-
Key: CASSANDRA-8365
URL: https://issues.apache.org/jira/browse/CASSANDRA-8365
Project: Cassandra
Issue Type: Bug
Reporter: Pierre Laporte
Priority: Minor
Labels: cqlsh
In cqlsh, when I execute a CREATE INDEX FooBar ... statement, the CamelCase
name is used as index name, even though it is unquoted. Trying to quote the
index name results in a syntax error.
However, when I try to delete the index, I have to quote the index name,
otherwise I get an invalid-query error telling me that the index (lowercase)
does not exist.
This seems inconsistent. Shouldn't the index name be lowercased before the
index is created ?
Here is the code to reproduce the issue :
{code}
cqlsh:schemabuilderit CREATE TABLE IndexTest (a int primary key, b int);
cqlsh:schemabuilderit CREATE INDEX FooBar on indextest (b);
cqlsh:schemabuilderit DESCRIBE TABLE indextest ;
CREATE TABLE schemabuilderit.indextest (
a int PRIMARY KEY,
b int
) ;
CREATE INDEX FooBar ON schemabuilderit.indextest (b);
cqlsh:schemabuilderit DROP INDEX FooBar;
code=2200 [Invalid query] message=Index 'foobar' could not be found in any
of the tables of keyspace 'schemabuilderit'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8365) CamelCase name is used as index name instead of lowercase
[
https://issues.apache.org/jira/browse/CASSANDRA-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8365:
---
Labels: cqlsh (was: )
CamelCase name is used as index name instead of lowercase
-
Key: CASSANDRA-8365
URL: https://issues.apache.org/jira/browse/CASSANDRA-8365
Project: Cassandra
Issue Type: Bug
Reporter: Pierre Laporte
Priority: Minor
Labels: cqlsh
In cqlsh, when I execute a CREATE INDEX FooBar ... statement, the CamelCase
name is used as index name, even though it is unquoted. Trying to quote the
index name results in a syntax error.
However, when I try to delete the index, I have to quote the index name,
otherwise I get an invalid-query error telling me that the index (lowercase)
does not exist.
This seems inconsistent. Shouldn't the index name be lowercased before the
index is created ?
Here is the code to reproduce the issue :
{code}
cqlsh:schemabuilderit CREATE TABLE IndexTest (a int primary key, b int);
cqlsh:schemabuilderit CREATE INDEX FooBar on indextest (b);
cqlsh:schemabuilderit DESCRIBE TABLE indextest ;
CREATE TABLE schemabuilderit.indextest (
a int PRIMARY KEY,
b int
) ;
CREATE INDEX FooBar ON schemabuilderit.indextest (b);
cqlsh:schemabuilderit DROP INDEX FooBar;
code=2200 [Invalid query] message=Index 'foobar' could not be found in any
of the tables of keyspace 'schemabuilderit'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8285) OOME in Cassandra 2.0.11
[
https://issues.apache.org/jira/browse/CASSANDRA-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Laporte updated CASSANDRA-8285:
--
Attachment: system.log.gz
gc-1416849312.log.gz
I just reproduced the issue on my machine against Cassandra 2.1.2.
*Howto*
Create 3-nodes C* cluster
{code}ccm create -n 3 -v 2.1.2 -b -s -i 127.0.0. cassandra-2.1{code}
Insert/delete a lot of rows inside a single table. I was actually trying to
reproduce the TombstoneOverwhelmingException but got an OOME instead.
{code}
public class CassandraTest implements AutoCloseable {
public static final String KEYSPACE = TombstonesOverwhelming;
private Cluster cluster;
protected Session session;
public CassandraTest() {
this(new RoundRobinPolicy());
}
public CassandraTest(LoadBalancingPolicy loadBalancingPolicy) {
System.out.println(Creating builder...);
cluster =
Cluster.builder().addContactPoint(127.0.0.1).withLoadBalancingPolicy(loadBalancingPolicy).build();
for (Host host : cluster.getMetadata().getAllHosts()) {
System.out.println(Found host + host.getAddress() + in DC +
host.getDatacenter());
}
session = cluster.connect();
}
private void executeQuietly(String query) {
try {
execute(query);
} catch (Exception e) {
e.printStackTrace();
}
}
private ResultSet execute(String query) {
return session.execute(query);
}
private ResultSet execute(Statement statement) {
return session.execute(statement);
}
@Override
public void close() throws IOException {
cluster.close();
}
public static void main(String... args) throws Exception {
try (CassandraTest test = new CassandraTest()) {
test.executeQuietly(DROP KEYSPACE IF EXISTS + KEYSPACE);
test.execute(CREATE KEYSPACE + KEYSPACE + +
WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor' : 3 });
test.execute(USE + KEYSPACE);
test.execute(CREATE TABLE useful (run int, iteration int, copy
int, PRIMARY KEY (run, iteration, copy)));
System.out.println(Press ENTER to start the test);
System.in.read();
for (int run = 0; run 1_000_000; run++) {
System.out.printf(Starting run % 7d... , run);
System.out.print(Inserting...);
for (int iteration = 0; iteration 1_000_000; iteration++) {
Batch batch = QueryBuilder.batch();
batch.setConsistencyLevel(ConsistencyLevel.QUORUM);
for (int copy = 0; copy 100; copy++) {
batch.add(QueryBuilder.insertInto(useful)
.value(run, run).value(iteration,
iteration).value(copy, copy));
}
test.execute(batch);
}
System.out.println(Deleting...);
for (int iteration = 0; iteration 1_000_000; iteration++) {
Batch batch = QueryBuilder.batch();
batch.setConsistencyLevel(ConsistencyLevel.QUORUM);
for (int copy = 0; copy 100; copy++) {
batch.add(QueryBuilder.delete().from(useful)
.where(eq(run, run)).and(eq(iteration,
iteration)).and(eq(copy, copy)));
}
test.execute(batch);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
{code}
I took ~50 minutes before two instances OOME'd. Please find attached the gc
log and the system log. If needed, I can upload a heap dump too.
Hope that helps
OOME in Cassandra 2.0.11
Key: CASSANDRA-8285
URL: https://issues.apache.org/jira/browse/CASSANDRA-8285
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 2.0.11 + java-driver 2.0.8-SNAPSHOT
Cassandra 2.0.11 + ruby-driver 1.0-beta
Reporter: Pierre Laporte
Assignee: Aleksey Yeschenko
Attachments: OOME_node_system.log, gc-1416849312.log.gz, gc.log.gz,
heap-usage-after-gc-zoom.png, heap-usage-after-gc.png, system.log.gz
We ran drivers 3-days endurance tests against Cassandra 2.0.11 and C* crashed
with an OOME. This happened both with ruby-driver 1.0-beta and java-driver
2.0.8-snapshot.
Attached are :
| OOME_node_system.log | The system.log of one Cassandra node that crashed |
| gc.log.gz | The GC log on the same node |
| heap-usage-after-gc.png | The heap occupancy evolution after every GC cycle
|
| heap-usage-after-gc-zoom.png | A focus on when things start to go wrong |
Workload :
Our test executes 5 CQL statements (select, insert, select, delete,
[jira] [Comment Edited] (CASSANDRA-8285) OOME in Cassandra 2.0.11
[
https://issues.apache.org/jira/browse/CASSANDRA-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223321#comment-14223321
]
Pierre Laporte edited comment on CASSANDRA-8285 at 11/24/14 7:09 PM:
-
I just reproduced the issue on my machine against Cassandra 2.1.2.
*Howto*
Create 3-nodes C* cluster
{code}ccm create -n 3 -v 2.1.2 -b -s -i 127.0.0. cassandra-2.1{code}
Insert/delete a lot of rows inside a single table. I was actually trying to
reproduce the TombstoneOverwhelmingException but got an OOME instead.
{code}
public class CassandraTest implements AutoCloseable {
public static final String KEYSPACE = TombstonesOverwhelming;
private Cluster cluster;
protected Session session;
public CassandraTest() {
this(new RoundRobinPolicy());
}
public CassandraTest(LoadBalancingPolicy loadBalancingPolicy) {
System.out.println(Creating builder...);
cluster =
Cluster.builder().addContactPoint(127.0.0.1).withLoadBalancingPolicy(loadBalancingPolicy).build();
for (Host host : cluster.getMetadata().getAllHosts()) {
System.out.println(Found host + host.getAddress() + in DC +
host.getDatacenter());
}
session = cluster.connect();
}
private void executeQuietly(String query) {
try {
execute(query);
} catch (Exception e) {
e.printStackTrace();
}
}
private ResultSet execute(String query) {
return session.execute(query);
}
private ResultSet execute(Statement statement) {
return session.execute(statement);
}
@Override
public void close() throws IOException {
cluster.close();
}
public static void main(String... args) throws Exception {
try (CassandraTest test = new CassandraTest()) {
test.executeQuietly(DROP KEYSPACE IF EXISTS + KEYSPACE);
test.execute(CREATE KEYSPACE + KEYSPACE + +
WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor' : 3 });
test.execute(USE + KEYSPACE);
test.execute(CREATE TABLE useful (run int, iteration int, copy
int, PRIMARY KEY (run, iteration, copy)));
System.out.println(Press ENTER to start the test);
System.in.read();
for (int run = 0; run 1_000_000; run++) {
System.out.printf(Starting run % 7d... , run);
System.out.print(Inserting...);
for (int iteration = 0; iteration 1_000_000; iteration++) {
Batch batch = QueryBuilder.batch();
batch.setConsistencyLevel(ConsistencyLevel.QUORUM);
for (int copy = 0; copy 100; copy++) {
batch.add(QueryBuilder.insertInto(useful)
.value(run, run).value(iteration,
iteration).value(copy, copy));
}
test.execute(batch);
}
System.out.println(Deleting...);
for (int iteration = 0; iteration 1_000_000; iteration++) {
Batch batch = QueryBuilder.batch();
batch.setConsistencyLevel(ConsistencyLevel.QUORUM);
for (int copy = 0; copy 100; copy++) {
batch.add(QueryBuilder.delete().from(useful)
.where(eq(run, run)).and(eq(iteration,
iteration)).and(eq(copy, copy)));
}
test.execute(batch);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
{code}
I took ~50 minutes before two instances OOME'd. Please find attached the gc
log (gc-1416849312.log.gz) and the system log (system.log.gz). If needed, I
can upload a heap dump too.
Hope that helps
was (Author: pingtimeout):
I just reproduced the issue on my machine against Cassandra 2.1.2.
*Howto*
Create 3-nodes C* cluster
{code}ccm create -n 3 -v 2.1.2 -b -s -i 127.0.0. cassandra-2.1{code}
Insert/delete a lot of rows inside a single table. I was actually trying to
reproduce the TombstoneOverwhelmingException but got an OOME instead.
{code}
public class CassandraTest implements AutoCloseable {
public static final String KEYSPACE = TombstonesOverwhelming;
private Cluster cluster;
protected Session session;
public CassandraTest() {
this(new RoundRobinPolicy());
}
public CassandraTest(LoadBalancingPolicy loadBalancingPolicy) {
System.out.println(Creating builder...);
cluster =
Cluster.builder().addContactPoint(127.0.0.1).withLoadBalancingPolicy(loadBalancingPolicy).build();
for (Host host : cluster.getMetadata().getAllHosts()) {
System.out.println(Found host + host.getAddress() + in DC +
host.getDatacenter());
}
session
[jira] [Commented] (CASSANDRA-8365) CamelCase name is used as index name instead of lowercase
[
https://issues.apache.org/jira/browse/CASSANDRA-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223328#comment-14223328
]
Pierre Laporte commented on CASSANDRA-8365:
---
[~philipthompson] I am using 2.1.2
CamelCase name is used as index name instead of lowercase
-
Key: CASSANDRA-8365
URL: https://issues.apache.org/jira/browse/CASSANDRA-8365
Project: Cassandra
Issue Type: Bug
Reporter: Pierre Laporte
Priority: Minor
Labels: cqlsh
In cqlsh, when I execute a CREATE INDEX FooBar ... statement, the CamelCase
name is used as index name, even though it is unquoted. Trying to quote the
index name results in a syntax error.
However, when I try to delete the index, I have to quote the index name,
otherwise I get an invalid-query error telling me that the index (lowercase)
does not exist.
This seems inconsistent. Shouldn't the index name be lowercased before the
index is created ?
Here is the code to reproduce the issue :
{code}
cqlsh:schemabuilderit CREATE TABLE IndexTest (a int primary key, b int);
cqlsh:schemabuilderit CREATE INDEX FooBar on indextest (b);
cqlsh:schemabuilderit DESCRIBE TABLE indextest ;
CREATE TABLE schemabuilderit.indextest (
a int PRIMARY KEY,
b int
) ;
CREATE INDEX FooBar ON schemabuilderit.indextest (b);
cqlsh:schemabuilderit DROP INDEX FooBar;
code=2200 [Invalid query] message=Index 'foobar' could not be found in any
of the tables of keyspace 'schemabuilderit'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8342) Remove historical guidance for concurrent reader and writer tunings.
[ https://issues.apache.org/jira/browse/CASSANDRA-8342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223342#comment-14223342 ] Jonathan Ellis commented on CASSANDRA-8342: --- [~enigmacurry] as a sanity check can you stress i2.8xl reads at 64 128 and 256 concurrent read threads? Remove historical guidance for concurrent reader and writer tunings. Key: CASSANDRA-8342 URL: https://issues.apache.org/jira/browse/CASSANDRA-8342 Project: Cassandra Issue Type: Improvement Reporter: Matt Stump The cassandra.yaml and documentation provide guidance on tuning concurrent readers or concurrent writers to system resources (cores, spindles). Testing performed by both myself and customers demonstrates no benefit for thread pool sizes above 64 in size, and for thread pools greater than 128 in size a decrease in throughput. This is due to thread scheduling and synchronization bottlenecks within Cassandra. Additionally, for lower end systems reducing these thread pools provides very little benefit because the bottleneck is typically moved to either IO or CPU. I propose that we set the default value to 64 (current default is 32), and remove all guidance/recommendations regarding tuning. This recommendation may change in 3.0, but that would require further experimentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8285) OOME in Cassandra 2.0.11
[ https://issues.apache.org/jira/browse/CASSANDRA-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223346#comment-14223346 ] Russ Hatch commented on CASSANDRA-8285: --- The heap dump Pierre included directly above looks similar to the earlier one. One difference here is there appears to be 2 active memtables (the earlier heap dump had just one) consuming most of the heap. In this newer heap dump the MemTable objects appear to be contained/referenced by DataTracker objects, whereas in the earlier heap the voracious MemTable looked kinda like a top level object. But I could be misreading/misunderstanding the MAT report. OOME in Cassandra 2.0.11 Key: CASSANDRA-8285 URL: https://issues.apache.org/jira/browse/CASSANDRA-8285 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.11 + java-driver 2.0.8-SNAPSHOT Cassandra 2.0.11 + ruby-driver 1.0-beta Reporter: Pierre Laporte Assignee: Aleksey Yeschenko Attachments: OOME_node_system.log, gc-1416849312.log.gz, gc.log.gz, heap-usage-after-gc-zoom.png, heap-usage-after-gc.png, system.log.gz We ran drivers 3-days endurance tests against Cassandra 2.0.11 and C* crashed with an OOME. This happened both with ruby-driver 1.0-beta and java-driver 2.0.8-snapshot. Attached are : | OOME_node_system.log | The system.log of one Cassandra node that crashed | | gc.log.gz | The GC log on the same node | | heap-usage-after-gc.png | The heap occupancy evolution after every GC cycle | | heap-usage-after-gc-zoom.png | A focus on when things start to go wrong | Workload : Our test executes 5 CQL statements (select, insert, select, delete, select) for a given unique id, during 3 days, using multiple threads. There is not change in the workload during the test. Symptoms : In the attached log, it seems something starts in Cassandra between 2014-11-06 10:29:22 and 2014-11-06 10:45:32. This causes an allocation that fills the heap. We eventually get stuck in a Full GC storm and get an OOME in the logs. I have run the java-driver tests against Cassandra 1.2.19 and 2.1.1. The error does not occur. It seems specific to 2.0.11. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8338) Simplify Token Selection
[
https://issues.apache.org/jira/browse/CASSANDRA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reassigned CASSANDRA-8338:
-
Assignee: Jonathan Ellis
Simplify Token Selection
Key: CASSANDRA-8338
URL: https://issues.apache.org/jira/browse/CASSANDRA-8338
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Joaquin Casares
Assignee: Jonathan Ellis
Priority: Trivial
Labels: lhf
When creating provisioning scripts, especially when running tools like Chef,
each node is launched individually. When not using vnodes your initial setup
will always be unbalanced unless you handle token assignment within your
scripts.
I spoke to someone recently who was using this in production and his
operations team wasn't too pleased that they had to use OpsCenter as an extra
step for rebalancing. Instead, we should provide this functionality out of
the box for new clusters.
Instead, could we have the following options below the initial_token section?
{CODE}
# datacenter_index: 0
# node_index: 0
# datacenter_size: 1
{CODE}
The above configuration options, when uncommented, would do the math of:
{CODE}
token = node_index * (range / datacenter_size) + (datacenter_index * 100)
+ start_of_range
{CODE}
This means that users don't have to repeatedly implement the initial_token
selection code nor know the range and offsets of their partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7933) Update cassandra-stress README
[ https://issues.apache.org/jira/browse/CASSANDRA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223349#comment-14223349 ] Jonathan Ellis commented on CASSANDRA-7933: --- Can you review [~tjake]? Update cassandra-stress README -- Key: CASSANDRA-7933 URL: https://issues.apache.org/jira/browse/CASSANDRA-7933 Project: Cassandra Issue Type: Task Reporter: Benedict Assignee: Liang Xie Priority: Minor Attachments: CASSANDRA-7933.txt There is a README in the tools/stress directory. It is completely out of date. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8253) cassandra-stress 2.1 doesn't support LOCAL_ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-8253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223350#comment-14223350 ] Jonathan Ellis commented on CASSANDRA-8253: --- [~tjake] to review cassandra-stress 2.1 doesn't support LOCAL_ONE -- Key: CASSANDRA-8253 URL: https://issues.apache.org/jira/browse/CASSANDRA-8253 Project: Cassandra Issue Type: Bug Reporter: J.B. Langston Assignee: Liang Xie Attachments: CASSANDRA-8253.txt Looks like a simple oversight in argument parsing: ➜ bin ./cassandra-stress write cl=LOCAL_ONE Invalid value LOCAL_ONE; must match pattern ONE|QUORUM|LOCAL_QUORUM|EACH_QUORUM|ALL|ANY Also, CASSANDRA-7077 argues that it should be using LOCAL_ONE by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8338) Simplify Token Selection
[
https://issues.apache.org/jira/browse/CASSANDRA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-8338:
--
Assignee: Jeremiah Jordan (was: Jonathan Ellis)
Simplify Token Selection
Key: CASSANDRA-8338
URL: https://issues.apache.org/jira/browse/CASSANDRA-8338
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Joaquin Casares
Assignee: Jeremiah Jordan
Priority: Trivial
Labels: lhf
When creating provisioning scripts, especially when running tools like Chef,
each node is launched individually. When not using vnodes your initial setup
will always be unbalanced unless you handle token assignment within your
scripts.
I spoke to someone recently who was using this in production and his
operations team wasn't too pleased that they had to use OpsCenter as an extra
step for rebalancing. Instead, we should provide this functionality out of
the box for new clusters.
Instead, could we have the following options below the initial_token section?
{CODE}
# datacenter_index: 0
# node_index: 0
# datacenter_size: 1
{CODE}
The above configuration options, when uncommented, would do the math of:
{CODE}
token = node_index * (range / datacenter_size) + (datacenter_index * 100)
+ start_of_range
{CODE}
This means that users don't have to repeatedly implement the initial_token
selection code nor know the range and offsets of their partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8346) Paxos operation can use stale data during multiple range movements
[
https://issues.apache.org/jira/browse/CASSANDRA-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-8346:
--
Reviewer: sankalp kohli
[~kohlisankalp] to review
Paxos operation can use stale data during multiple range movements
--
Key: CASSANDRA-8346
URL: https://issues.apache.org/jira/browse/CASSANDRA-8346
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Fix For: 2.0.12
Attachments: 8346.txt
Paxos operations correctly account for pending ranges for all operation
pertaining to the Paxos state, but those pending ranges are not taken into
account when reading the data to check for the conditions or during a serial
read. It's thus possible to break the LWT guarantees by reading a stale
value. This require 2 node movements (on the same token range) to be a
problem though.
Basically, we have {{RF}} replicas + {{P}} pending nodes. For the Paxos
prepare/propose phases, the number of required participants (the Paxos
QUORUM) is {{(RF + P + 1) / 2}} ({{SP.getPaxosParticipants}}), but the read
done to check conditions or for serial reads is done at a normal QUORUM (or
LOCAL_QUORUM), and so a weaker {{(RF + 1) / 2}}. We have a problem if it's
possible that said read can read only from nodes that were not part of the
paxos participants, and so we have a problem if:
{noformat}
normal quorum == (RF + 1) / 2 = (RF + P) - ((RF + P + 1) / 2) ==
participants considered - blocked for
{noformat}
We're good if {{P = 0}} or {{P = 1}} since this inequality gives us
respectively {{RF + 1 = RF - 1}} and {{RF + 1 = RF}}, both of which are
impossible. But at {{P = 2}} (2 pending nodes), this inequality is equivalent
to {{RF = RF}} and so we might read stale data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8188) don't block SocketThread for MessagingService
[
https://issues.apache.org/jira/browse/CASSANDRA-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223362#comment-14223362
]
Jonathan Ellis commented on CASSANDRA-8188:
---
I'd be okay with adding this to 2.0.12. Brandon?
don't block SocketThread for MessagingService
-
Key: CASSANDRA-8188
URL: https://issues.apache.org/jira/browse/CASSANDRA-8188
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: yangwei
Assignee: yangwei
Fix For: 2.1.2
Attachments:
0001-don-t-block-SocketThread-for-MessagingService.patch, handshake.stack.txt
We have two datacenters A and B.
The node in A cannot handshake version with nodes in B, logs in A as follow:
{noformat}
INFO [HANDSHAKE-/B] 2014-10-24 04:29:49,075 OutboundTcpConnection.java
(line 395) Cannot handshake version with B
TRACE [WRITE-/B] 2014-10-24 11:02:49,044 OutboundTcpConnection.java (line
368) unable to connect to /B
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at java.nio.channels.SocketChannel.open(SocketChannel.java:184)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:134)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:119)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:299)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:150)
{noformat}
The jstack output of nodes in B shows it blocks in inputStream.readInt
resulting in SocketThread not accept socket any more, logs as follow:
{noformat}
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
- locked 0x0007963747e8 (a java.lang.Object)
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203)
- locked 0x000796374848 (a java.lang.Object)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
- locked 0x0007a5c7ca88 (a
sun.nio.ch.SocketAdaptor$SocketInputStream)
at java.io.InputStream.read(InputStream.java:101)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
- locked 0x0007a5c7ca88 (a
sun.nio.ch.SocketAdaptor$SocketInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:879)
{noformat}
In nodes of B tcpdump shows retransmission of SYN,ACK during the tcp
three-way handshake phase because tcp implementation drops the last ack when
the backlog queue is full.
In nodes of B ss -tl shows Recv-Q 51 Send-Q 50.
In nodes of B netstat -s shows “SYNs to LISTEN sockets dropped” and “times
the listen queue of a socket overflowed” are both increasing.
This patch sets read timeout to 2 *
OutboundTcpConnection.WAIT_FOR_VERSION_MAX_TIME for the accepted socket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8343) Secondary index creation causes moves/bootstraps to fail
[ https://issues.apache.org/jira/browse/CASSANDRA-8343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-8343: -- Assignee: Yuki Morishita Secondary index creation causes moves/bootstraps to fail Key: CASSANDRA-8343 URL: https://issues.apache.org/jira/browse/CASSANDRA-8343 Project: Cassandra Issue Type: Bug Reporter: Michael Frisch Assignee: Yuki Morishita Node moves/bootstraps are failing if the stream timeout is set to a value in which secondary index creation cannot complete. This happens because at the end of the very last stream the StreamInSession.closeIfFinished() function calls maybeBuildSecondaryIndexes on every column family. If the stream time + all CF's index creation takes longer than your stream timeout then the socket closes from the sender's side, the receiver of the stream tries to write to said socket because it's not null, an IOException is thrown but not caught in closeIfFinished(), the exception is caught somewhere and not logged, AbstractStreamSession.close() is never called, and the CountDownLatch is never decremented. This causes the move/bootstrap to continue forever until the node is restarted. This problem of stream time + secondary index creation time exists on decommissioning/unbootstrap as well but since it's on the sending side the timeout triggers the onFailure() callback which does decrement the CountDownLatch leading to completion. A cursory glance at the 2.0 code leads me to believe this problem would exist there as well. Temporary workaround: set a really high/infinite stream timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8150) Revaluate Default JVM tuning parameters
[ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223371#comment-14223371 ] Jonathan Ellis commented on CASSANDRA-8150: --- [~tjake] are you going to run the tests or do you want to delegate to [~enigmacurry]'s team? Revaluate Default JVM tuning parameters --- Key: CASSANDRA-8150 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150 Project: Cassandra Issue Type: Improvement Components: Config Reporter: Matt Stump Assignee: Brandon Williams Attachments: upload.png It's been found that the old twitter recommendations of 100m per core up to 800m is harmful and should no longer be used. Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better for releases greater than 2.1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8352) Timeout Exception on Node Failure in Remote Data Center
[ https://issues.apache.org/jira/browse/CASSANDRA-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-8352. --- Resolution: Cannot Reproduce Upgrade to 2.0.11 and let us know if you still see unexpected behavior. Timeout Exception on Node Failure in Remote Data Center --- Key: CASSANDRA-8352 URL: https://issues.apache.org/jira/browse/CASSANDRA-8352 Project: Cassandra Issue Type: Bug Environment: Unix, Cassandra 2.0.3 Reporter: Akhtar Hussain Labels: DataCenter, GEO-Red We have a Geo-red setup with 2 Data centers having 3 nodes each. When we bring down a single Cassandra node down in DC2 by kill -9 Cassandra-pid, reads fail on DC1 with TimedOutException for a brief amount of time (15-20 sec~). Questions: 1.We need to understand why reads fail on DC1 when a node in another DC i.e. DC2 fails? As we are using LOCAL_QUORUM for both reads/writes in DC1, request should return once 2 nodes in local DC have replied instead of timing out because of node in remote DC. 2.We want to make sure that no Cassandra requests fail in case of node failures. We used rapid read protection of ALWAYS/99percentile/10ms as mentioned in http://www.datastax.com/dev/blog/rapid-read-protection-in-cassandra-2-0-2. But nothing worked. How to ensure zero request failures in case a node fails? 3.What is the right way of handling HTimedOutException exceptions in Hector? 4.Please confirm are we using public private hostnames as expected? We are using Cassandra 2.0.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8338) Simplify Token Selection
[
https://issues.apache.org/jira/browse/CASSANDRA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223366#comment-14223366
]
Nick Bailey commented on CASSANDRA-8338:
It might be worth putting this in a different file than cassandra.yaml. It's
already confusing that some options in there (initial_token, num_tokens) only
matter the very first time a node starts up. I'm not sure if we should be
adding more. Also we should make sure we convey that this only helps when the
entire cluster is being set up for the first time, not when adding nodes.
Lastly, this will need to incorporate rack information as well if we want it to
work correctly when not everything is in the same rack.
Simplify Token Selection
Key: CASSANDRA-8338
URL: https://issues.apache.org/jira/browse/CASSANDRA-8338
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Joaquin Casares
Assignee: Jeremiah Jordan
Priority: Trivial
Labels: lhf
When creating provisioning scripts, especially when running tools like Chef,
each node is launched individually. When not using vnodes your initial setup
will always be unbalanced unless you handle token assignment within your
scripts.
I spoke to someone recently who was using this in production and his
operations team wasn't too pleased that they had to use OpsCenter as an extra
step for rebalancing. Instead, we should provide this functionality out of
the box for new clusters.
Instead, could we have the following options below the initial_token section?
{CODE}
# datacenter_index: 0
# node_index: 0
# datacenter_size: 1
{CODE}
The above configuration options, when uncommented, would do the math of:
{CODE}
token = node_index * (range / datacenter_size) + (datacenter_index * 100)
+ start_of_range
{CODE}
This means that users don't have to repeatedly implement the initial_token
selection code nor know the range and offsets of their partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8150) Revaluate Default JVM tuning parameters
[ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223415#comment-14223415 ] T Jake Luciani commented on CASSANDRA-8150: --- I can run it through some workloads... Revaluate Default JVM tuning parameters --- Key: CASSANDRA-8150 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150 Project: Cassandra Issue Type: Improvement Components: Config Reporter: Matt Stump Assignee: Brandon Williams Attachments: upload.png It's been found that the old twitter recommendations of 100m per core up to 800m is harmful and should no longer be used. Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better for releases greater than 2.1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8338) Simplify Token Selection
[
https://issues.apache.org/jira/browse/CASSANDRA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremiah Jordan resolved CASSANDRA-8338.
Resolution: Won't Fix
Simplify Token Selection
Key: CASSANDRA-8338
URL: https://issues.apache.org/jira/browse/CASSANDRA-8338
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Joaquin Casares
Assignee: Jeremiah Jordan
Priority: Trivial
Labels: lhf
When creating provisioning scripts, especially when running tools like Chef,
each node is launched individually. When not using vnodes your initial setup
will always be unbalanced unless you handle token assignment within your
scripts.
I spoke to someone recently who was using this in production and his
operations team wasn't too pleased that they had to use OpsCenter as an extra
step for rebalancing. Instead, we should provide this functionality out of
the box for new clusters.
Instead, could we have the following options below the initial_token section?
{CODE}
# datacenter_index: 0
# node_index: 0
# datacenter_size: 1
{CODE}
The above configuration options, when uncommented, would do the math of:
{CODE}
token = node_index * (range / datacenter_size) + (datacenter_index * 100)
+ start_of_range
{CODE}
This means that users don't have to repeatedly implement the initial_token
selection code nor know the range and offsets of their partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8338) Simplify Token Selection
[
https://issues.apache.org/jira/browse/CASSANDRA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223425#comment-14223425
]
Jeremiah Jordan commented on CASSANDRA-8338:
If you know the DC size to put it in the file:
{noformat}
# datacenter_index: 0
# node_index: 0
# datacenter_size: 1
{noformat}
Then you can just include the formula in your chef script.
Simplify Token Selection
Key: CASSANDRA-8338
URL: https://issues.apache.org/jira/browse/CASSANDRA-8338
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Joaquin Casares
Assignee: Jeremiah Jordan
Priority: Trivial
Labels: lhf
When creating provisioning scripts, especially when running tools like Chef,
each node is launched individually. When not using vnodes your initial setup
will always be unbalanced unless you handle token assignment within your
scripts.
I spoke to someone recently who was using this in production and his
operations team wasn't too pleased that they had to use OpsCenter as an extra
step for rebalancing. Instead, we should provide this functionality out of
the box for new clusters.
Instead, could we have the following options below the initial_token section?
{CODE}
# datacenter_index: 0
# node_index: 0
# datacenter_size: 1
{CODE}
The above configuration options, when uncommented, would do the math of:
{CODE}
token = node_index * (range / datacenter_size) + (datacenter_index * 100)
+ start_of_range
{CODE}
This means that users don't have to repeatedly implement the initial_token
selection code nor know the range and offsets of their partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7970) JSON support for CQL
[
https://issues.apache.org/jira/browse/CASSANDRA-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan McGuire updated CASSANDRA-7970:
Issue Type: New Feature (was: Bug)
JSON support for CQL
Key: CASSANDRA-7970
URL: https://issues.apache.org/jira/browse/CASSANDRA-7970
Project: Cassandra
Issue Type: New Feature
Components: API
Reporter: Jonathan Ellis
Assignee: Tyler Hobbs
Fix For: 3.0
JSON is popular enough that not supporting it is becoming a competitive
weakness. We can add JSON support in a way that is compatible with our
performance goals by *mapping* JSON to an existing schema: one JSON documents
maps to one CQL row.
Thus, it is NOT a goal to support schemaless documents, which is a misfeature
[1] [2] [3]. Rather, it is to allow a convenient way to easily turn a JSON
document from a service or a user into a CQL row, with all the validation
that entails.
Since we are not looking to support schemaless documents, we will not be
adding a JSON data type (CASSANDRA-6833) a la postgresql. Rather, we will
map the JSON to UDT, collections, and primitive CQL types.
Here's how this might look:
{code}
CREATE TYPE address (
street text,
city text,
zip_code int,
phones settext
);
CREATE TABLE users (
id uuid PRIMARY KEY,
name text,
addresses maptext, address
);
INSERT INTO users JSON
{‘id’: 4b856557-7153,
‘name’: ‘jbellis’,
‘address’: {“home”: {“street”: “123 Cassandra Dr”,
“city”: “Austin”,
“zip_code”: 78747,
“phones”: [2101234567]}}};
SELECT JSON id, address FROM users;
{code}
(We would also want to_json and from_json functions to allow mapping a single
column's worth of data. These would not require extra syntax.)
[1] http://rustyrazorblade.com/2014/07/the-myth-of-schema-less/
[2] https://blog.compose.io/schema-less-is-usually-a-lie/
[3] http://dl.acm.org/citation.cfm?id=2481247
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8365) CamelCase name is used as index name instead of lowercase
[
https://issues.apache.org/jira/browse/CASSANDRA-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8365:
---
Reproduced In: 2.1.2
Fix Version/s: 2.1.3
CamelCase name is used as index name instead of lowercase
-
Key: CASSANDRA-8365
URL: https://issues.apache.org/jira/browse/CASSANDRA-8365
Project: Cassandra
Issue Type: Bug
Reporter: Pierre Laporte
Priority: Minor
Labels: cqlsh
Fix For: 2.1.3
In cqlsh, when I execute a CREATE INDEX FooBar ... statement, the CamelCase
name is used as index name, even though it is unquoted. Trying to quote the
index name results in a syntax error.
However, when I try to delete the index, I have to quote the index name,
otherwise I get an invalid-query error telling me that the index (lowercase)
does not exist.
This seems inconsistent. Shouldn't the index name be lowercased before the
index is created ?
Here is the code to reproduce the issue :
{code}
cqlsh:schemabuilderit CREATE TABLE IndexTest (a int primary key, b int);
cqlsh:schemabuilderit CREATE INDEX FooBar on indextest (b);
cqlsh:schemabuilderit DESCRIBE TABLE indextest ;
CREATE TABLE schemabuilderit.indextest (
a int PRIMARY KEY,
b int
) ;
CREATE INDEX FooBar ON schemabuilderit.indextest (b);
cqlsh:schemabuilderit DROP INDEX FooBar;
code=2200 [Invalid query] message=Index 'foobar' could not be found in any
of the tables of keyspace 'schemabuilderit'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8354) A better story for dealing with empty values
[
https://issues.apache.org/jira/browse/CASSANDRA-8354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223462#comment-14223462
]
Jonathan Ellis commented on CASSANDRA-8354:
---
Is there a way we can avoid permanently enshrining this wart?
What if for instance we added an option {{strict_cql_values}} to 3.0 that
defaults to false. When enabled it rejects nonsensical empty values. For 3.1
we default to true, and give people a tool to convert empty to null or some
other value. For 4.0 it stays permanently true.
A better story for dealing with empty values
Key: CASSANDRA-8354
URL: https://issues.apache.org/jira/browse/CASSANDRA-8354
Project: Cassandra
Issue Type: Improvement
Reporter: Sylvain Lebresne
Fix For: 3.0
In CQL, a value of any type can be empty, even for types for which such
values doesn't make any sense (int, uuid, ...). Note that it's different from
having no value (i.e. a {{null}}). This is due to historical reasons, and we
can't entirely disallow it for backward compatibility, but it's pretty
painful when working with CQL since you always need to be defensive about
such largely non-sensical values.
This is particularly annoying with UDF: those empty values are represented as
{{null}} for UDF and that plays weirdly with UDF that use unboxed native
types.
So I would suggest that we introduce variations of the types that don't
accept empty byte buffers for those type for which it's not a particularly
sensible value.
Ideally we'd use those variant by default, that is:
{noformat}
CREATE TABLE foo (k text PRIMARY, v int)
{noformat}
would not accept empty values for {{v}}. But
{noformat}
CREATE TABLE foo (k text PRIMARY, v int ALLOW EMPTY)
{noformat}
would.
Similarly, for UDF, a function like:
{noformat}
CREATE FUNCTION incr(v int) RETURNS int LANGUAGE JAVA AS 'return v + 1';
{noformat}
would be guaranteed it can only be applied where no empty values are allowed.
A
function that wants to handle empty values could be created with:
{noformat}
CREATE FUNCTION incr(v int ALLOW EMPTY) RETURNS int ALLOW EMPTY LANGUAGE JAVA
AS 'return (v == null) ? null : v + 1';
{noformat}
Of course, doing that has the problem of backward compatibility. One option
could be to say that if a type doesn't accept empties, but we do have an
empty internally, then we convert it to some reasonably sensible default
value (0 for numeric values, the smallest possible uuid for uuids, etc...).
This way, we could allow convesion of types to and from 'ALLOW EMPTY'. And
maybe we'd say that existing compact tables gets the 'ALLOW EMPTY' flag for
their types by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8067) NullPointerException in KeyCacheSerializer
[
https://issues.apache.org/jira/browse/CASSANDRA-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223464#comment-14223464
]
Julien Anguenot commented on CASSANDRA-8067:
[~mokemokechicken] [~Andie78] error definitely still occurring in a cluster
where all nodes are running 2.1.2: error happening against every node of the
cluster on regular basis. The same cluster running 2.0.11 did not have that
particular issue before migration.
NullPointerException in KeyCacheSerializer
--
Key: CASSANDRA-8067
URL: https://issues.apache.org/jira/browse/CASSANDRA-8067
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Eric Leleu
Fix For: 2.1.1
Hi,
I have this stack trace in the logs of Cassandra server (v2.1)
{code}
ERROR [CompactionExecutor:14] 2014-10-06 23:32:02,098
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:14,1,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:225)
~[apache-cassandra-2.1.0.jar:2.1.0]
at
org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1061)
~[apache-cassandra-2.1.0.jar:2.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
~[na:1.7.0]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0]
at java.lang.Thread.run(Unknown Source) [na:1.7.0]
{code}
It may not be critical because this error occured in the AutoSavingCache.
However the line 475 is about the CFMetaData so it may hide bigger issue...
{code}
474 CFMetaData cfm =
Schema.instance.getCFMetaData(key.desc.ksname, key.desc.cfname);
475 cfm.comparator.rowIndexEntrySerializer().serialize(entry,
out);
{code}
Regards,
Eric
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8354) A better story for dealing with empty values
[
https://issues.apache.org/jira/browse/CASSANDRA-8354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223472#comment-14223472
]
Aleksey Yeschenko commented on CASSANDRA-8354:
--
bq. What if for instance we added an option strict_cql_values to 3.0 that
defaults to false. When enabled it rejects nonsensical empty values. For 3.1 we
default to true, and give people a tool to convert empty to null or some other
value. For 4.0 it stays permanently true.
That. Except it's not just CQL, there is thrift too, where we should enforce
this, so maybe should name it 'reject_empty_types' or something. As a tool,
upgradesstables will probably do.
Don't want to legitimize it on CQL syntax level either.
A better story for dealing with empty values
Key: CASSANDRA-8354
URL: https://issues.apache.org/jira/browse/CASSANDRA-8354
Project: Cassandra
Issue Type: Improvement
Reporter: Sylvain Lebresne
Fix For: 3.0
In CQL, a value of any type can be empty, even for types for which such
values doesn't make any sense (int, uuid, ...). Note that it's different from
having no value (i.e. a {{null}}). This is due to historical reasons, and we
can't entirely disallow it for backward compatibility, but it's pretty
painful when working with CQL since you always need to be defensive about
such largely non-sensical values.
This is particularly annoying with UDF: those empty values are represented as
{{null}} for UDF and that plays weirdly with UDF that use unboxed native
types.
So I would suggest that we introduce variations of the types that don't
accept empty byte buffers for those type for which it's not a particularly
sensible value.
Ideally we'd use those variant by default, that is:
{noformat}
CREATE TABLE foo (k text PRIMARY, v int)
{noformat}
would not accept empty values for {{v}}. But
{noformat}
CREATE TABLE foo (k text PRIMARY, v int ALLOW EMPTY)
{noformat}
would.
Similarly, for UDF, a function like:
{noformat}
CREATE FUNCTION incr(v int) RETURNS int LANGUAGE JAVA AS 'return v + 1';
{noformat}
would be guaranteed it can only be applied where no empty values are allowed.
A
function that wants to handle empty values could be created with:
{noformat}
CREATE FUNCTION incr(v int ALLOW EMPTY) RETURNS int ALLOW EMPTY LANGUAGE JAVA
AS 'return (v == null) ? null : v + 1';
{noformat}
Of course, doing that has the problem of backward compatibility. One option
could be to say that if a type doesn't accept empties, but we do have an
empty internally, then we convert it to some reasonably sensible default
value (0 for numeric values, the smallest possible uuid for uuids, etc...).
This way, we could allow convesion of types to and from 'ALLOW EMPTY'. And
maybe we'd say that existing compact tables gets the 'ALLOW EMPTY' flag for
their types by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223479#comment-14223479 ] Ariel Weisberg commented on CASSANDRA-7438: --- I think that for caches the behavior you want to avoid most is slowly growing heap. People hate that because it's unpredictable and they don't know when it's going to stop. You can always start with jemalloc and get the feature working and then iterate on memory management. Fixed block sizes is a baby and bath water scenario to get the desirable fixed memory utilization property. When you want to build everything out of fixed size pages you have to slot the pages or do some other internal page management strategy so you can pack multiple things and rewrite pages as they fragment. You also need size tiered free lists and fragmentation metadata for pages so you can find partial free pages. That kind of thing only makes sense in ye olden database land where rewriting an already dirty page is cheaper than more IOPs. In memory you can relocate objects. Memcached used to have the problem that instead of the heap growing the cache would lose capacity to fragmentation. FB implemented slab rebalancing in their fork, and then Memcached did its own implementation. The issue was internal fragmentation due to having too many of the wrong size slabs. For Robert * Executor service shutdown, never really got why it takes a timeout nor why there is no blocking version. 99% of the time if it doesn't shutdown within the timeout it's a bug and you don't want to ignore it. We are pedantic about everything else why not this? It's also unused right now. * Stats could go into an atomic long array with padding. It really depends on the access pattern. You want data that is read/written at the same time on the same cache line. These are global counters so they will be contended by everyone accessing the cache, better that they only have to pull in one cache line with all counters then multiple and have to wait for exclusive access before writing to each one. Also consider LongAdder. * If you want to do your own memory management strategy I think something like segregated storage as in boost pool with size tiers for powers of two and power of two plus previous power of two. You can CAS the head of the free list for each tier to make it thread safe, and lock when allocating out a new block instead of the free list. This won't adapt to changing size distributions. For that stuff needs to be relocatable * I'll bet you could use a stamped lock pattern and readers might not have to lock all. I think getting it working with just a lock is fine. * I am not sure shrinking is very important? The table is pretty dense and should be a small portion of total memory once all the other memory is accounted for. You would need a lot of tiny cache entries to really bloat the table and then the population distribution would need to change to make that a waste. * LRU lists per segment seems like it's not viable. That isn't a close enough approximation to LRU since we want at most two or three entries per partition. * Some loops of very similar byte munging in HashEntryAccess * Periodic cleanup check is maybe not so nice. An edge trigger via a CAS field would be nicer and move that up to 80% since on a big-memory machine that is a lot of wasted cache space. Walking the entire LRU could take several seconds, but if it is amortized across a lot of expiration maybe it is ok. * Some rehash required checking is duplicated in OHCacheImpl For Vijay * sun.misc.Hashing doesn't seem to exist for me, maybe a Java 8 issue? * The queue really needs to be bounded, producer and consumer could proceed at different rates. With striped * Tasks submitted to executor services via submit will wrap the result including exceptions in a future which silently discards them. The library might take at initialization time a listener for these errors, or if it is going to be C* specific it could use the wrapped runnable or similar. * A lot of locking that was spin locking (which unbounded I don't think is great) is now blocking locking. There is no adaptive spinning if you don't use synchronized. If you are already using unsafe maybe you could do monitor enter/exit. Never tried it. * It looks like concurrent calls to rehash could cause the table to rehash twice since the rebalance field is not CASed. You should do the volatile read, and then attempt the CAS (avoids putting the cache line in exclusive state every time). * StatsHolder, same AtomicLongArray suggestion. Also consider LongAdder. * In Segment.java in the replace path AtomicLong.addAndGet is called back to back, could be called once with the math already done. I believe each of those stalls processing until the store buffers have flushed. The put path does something similar and could have the same
[jira] [Assigned] (CASSANDRA-8366) Repair grows data on nodes, causes load to become unbalanced
[ https://issues.apache.org/jira/browse/CASSANDRA-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan McGuire reassigned CASSANDRA-8366: --- Assignee: Alan Boudreault Repair grows data on nodes, causes load to become unbalanced Key: CASSANDRA-8366 URL: https://issues.apache.org/jira/browse/CASSANDRA-8366 Project: Cassandra Issue Type: Bug Environment: 4 node cluster 2.1.2 Cassandra Inserts and reads are done with CQL driver Reporter: Jan Karlsson Assignee: Alan Boudreault There seems to be something weird going on when repairing data. I have a program that runs 2 hours which inserts 250 random numbers and reads 250 times per second. It creates 2 keyspaces with SimpleStrategy and RF of 3. I use size-tiered compaction for my cluster. After those 2 hours I run a repair and the load of all nodes goes up. If I run incremental repair the load goes up alot more. I saw the load shoot up 8 times the original size multiple times with incremental repair. (from 2G to 16G) with node 9 8 7 and 6 the repro procedure looked like this: (Note that running full repair first is not a requirement to reproduce.) After 2 hours of 250 reads + 250 writes per second: UN 9 583.39 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 584.01 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 583.72 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 583.84 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Repair -pr -par on all nodes sequentially UN 9 746.29 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 751.02 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 748.89 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.34 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 2.41 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.53 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.6 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 2.17 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 after rolling restart UN 9 1.47 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 1.5 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 2.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.19 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 compact all nodes sequentially UN 9 989.99 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 994.75 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 1.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 758.82 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 repair -inc -par on all nodes sequentially UN 9 1.98 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.3 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 3.71 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 restart once more UN 9 2 GB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1 UN 8 2.05 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1 UN 7 4.1 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1 UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1 Is there something im missing or is this strange behavior? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8342) Remove historical guidance for concurrent reader and writer tunings.
[ https://issues.apache.org/jira/browse/CASSANDRA-8342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan McGuire reassigned CASSANDRA-8342: --- Assignee: Ryan McGuire Remove historical guidance for concurrent reader and writer tunings. Key: CASSANDRA-8342 URL: https://issues.apache.org/jira/browse/CASSANDRA-8342 Project: Cassandra Issue Type: Improvement Reporter: Matt Stump Assignee: Ryan McGuire The cassandra.yaml and documentation provide guidance on tuning concurrent readers or concurrent writers to system resources (cores, spindles). Testing performed by both myself and customers demonstrates no benefit for thread pool sizes above 64 in size, and for thread pools greater than 128 in size a decrease in throughput. This is due to thread scheduling and synchronization bottlenecks within Cassandra. Additionally, for lower end systems reducing these thread pools provides very little benefit because the bottleneck is typically moved to either IO or CPU. I propose that we set the default value to 64 (current default is 32), and remove all guidance/recommendations regarding tuning. This recommendation may change in 3.0, but that would require further experimentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8303) Provide strict mode for CQL Queries
[ https://issues.apache.org/jira/browse/CASSANDRA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8303: --- Fix Version/s: 3.0 Provide strict mode for CQL Queries - Key: CASSANDRA-8303 URL: https://issues.apache.org/jira/browse/CASSANDRA-8303 Project: Cassandra Issue Type: Improvement Reporter: Anupam Arora Fix For: 3.0 Please provide a strict mode option in cassandra that will kick out any CQL queries that are expensive, e.g. any query with ALLOWS FILTERING, multi-partition queries, secondary index queries, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
