[jira] [Updated] (CASSANDRA-5506) Reduce memory consumption of IndexSummary
[ https://issues.apache.org/jira/browse/CASSANDRA-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-5506: -- Reviewer: vijay2...@yahoo.com (was: yukim) Reduce memory consumption of IndexSummary - Key: CASSANDRA-5506 URL: https://issues.apache.org/jira/browse/CASSANDRA-5506 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Nick Puz Assignee: Jonathan Ellis Fix For: 1.2.5 I am evaluating cassandra for a use case with many tiny rows which would result in a node with 1-3TB of storage having billions of rows. Before loading that much data I am hitting GC issues and when looking at the heap dump I noticed that 70+% of the memory was used by IndexSummaries. The two major issues seem to be: 1) that the positions are stored as an ArrayListLong which results in each position taking 24 bytes (class + flags + 8 byte long). This might make sense when the file is initially written but once it has been serialized it would be a lot more memory efficient to just have an long[] (really a int[] would be fine unless 2GB sstables are allowed). 2) The DecoratedKey for a byte[16] key takes 195 bytes -- this is for the overhead of the ByteBuffer in the key and overhead in the token. To somewhat work around the problem I have increased index_sample but will this many rows that didn't really help starts to have diminishing returns. NOTE: This heap dump was from linux with a 64bit oracle vm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5506) Reduce memory consumption of IndexSummary
[ https://issues.apache.org/jira/browse/CASSANDRA-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643917#comment-13643917 ] Jonathan Ellis commented on CASSANDRA-5506: --- I'm pretty comfortable switching the representation for 1.2.5; let's make a separate ticket to move off heap for 2.0. Reduce memory consumption of IndexSummary - Key: CASSANDRA-5506 URL: https://issues.apache.org/jira/browse/CASSANDRA-5506 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Nick Puz Assignee: Jonathan Ellis Fix For: 1.2.5 I am evaluating cassandra for a use case with many tiny rows which would result in a node with 1-3TB of storage having billions of rows. Before loading that much data I am hitting GC issues and when looking at the heap dump I noticed that 70+% of the memory was used by IndexSummaries. The two major issues seem to be: 1) that the positions are stored as an ArrayListLong which results in each position taking 24 bytes (class + flags + 8 byte long). This might make sense when the file is initially written but once it has been serialized it would be a lot more memory efficient to just have an long[] (really a int[] would be fine unless 2GB sstables are allowed). 2) The DecoratedKey for a byte[16] key takes 195 bytes -- this is for the overhead of the ByteBuffer in the key and overhead in the token. To somewhat work around the problem I have increased index_sample but will this many rows that didn't really help starts to have diminishing returns. NOTE: This heap dump was from linux with a 64bit oracle vm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5506) Reduce memory consumption of IndexSummary
[ https://issues.apache.org/jira/browse/CASSANDRA-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643920#comment-13643920 ] Vijay commented on CASSANDRA-5506: -- +1 for the patch and +1 for a separate ticket. Reduce memory consumption of IndexSummary - Key: CASSANDRA-5506 URL: https://issues.apache.org/jira/browse/CASSANDRA-5506 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Nick Puz Assignee: Jonathan Ellis Fix For: 1.2.5 I am evaluating cassandra for a use case with many tiny rows which would result in a node with 1-3TB of storage having billions of rows. Before loading that much data I am hitting GC issues and when looking at the heap dump I noticed that 70+% of the memory was used by IndexSummaries. The two major issues seem to be: 1) that the positions are stored as an ArrayListLong which results in each position taking 24 bytes (class + flags + 8 byte long). This might make sense when the file is initially written but once it has been serialized it would be a lot more memory efficient to just have an long[] (really a int[] would be fine unless 2GB sstables are allowed). 2) The DecoratedKey for a byte[16] key takes 195 bytes -- this is for the overhead of the ByteBuffer in the key and overhead in the token. To somewhat work around the problem I have increased index_sample but will this many rows that didn't really help starts to have diminishing returns. NOTE: This heap dump was from linux with a 64bit oracle vm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5520) Query tracing session info inconsistent with events info
Ilya Kirnos created CASSANDRA-5520: -- Summary: Query tracing session info inconsistent with events info Key: CASSANDRA-5520 URL: https://issues.apache.org/jira/browse/CASSANDRA-5520 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: Linux Reporter: Ilya Kirnos Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request --+--+ c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread --+--+--+++ c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /50.112.90.147 |50.112.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.153.16 | 50.112.90.147 |246 | WRITE-/50.112.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /50.112.90.147 |50.112.4.16 |259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | 50.112.90.147 |253 | WRITE-/50.112.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /50.112.90.147 | 50.112.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash |50.112.4.16 |421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.151.214 | 50.112.90.147 |310 | WRITE-/50.112.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /50.112.90.147 | 50.112.213.136 |106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references |50.112.4.16 |444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | 50.112.90.147 |352 | WRITE-/50.112.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | 50.112.213.136 |144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 | Merging memtable contents |50.112.4.16 |472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | 50.112.90.147 |362 | WRITE-/50.112.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | 50.112.213.136 |164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables |50.112.4.16 |510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.151.214 | 50.112.90.147 |376 | WRITE-/50.112.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 | Merging memtable contents | 50.112.213.136 |195 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9480-e9d811e0fc18 | Read 0 live cells and 0 tombstoned |50.112.4.16 |530 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | 50.112.90.147 |401 | WRITE-/50.112.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash |
[jira] [Created] (CASSANDRA-5521) move IndexSummary off heap
Jonathan Ellis created CASSANDRA-5521: - Summary: move IndexSummary off heap Key: CASSANDRA-5521 URL: https://issues.apache.org/jira/browse/CASSANDRA-5521 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jonathan Ellis Assignee: Vijay Fix For: 2.0 IndexSummary can still use a lot of heap for narrow-row sstables. (It can also contribute to memory fragmentation because of the large arrays it creates.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: Reduce memory consumption of IndexSummary patch by jbellis; reviewed by vijay for CASSANDRA-5506
Updated Branches:
refs/heads/cassandra-1.2 dc5b1e9b8 - 9851b73fc
Reduce memory consumption of IndexSummary
patch by jbellis; reviewed by vijay for CASSANDRA-5506
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9851b73f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9851b73f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9851b73f
Branch: refs/heads/cassandra-1.2
Commit: 9851b73fc3bbaf61ec80b54c64483bcf56aaa396
Parents: dc5b1e9
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 28 09:10:38 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 28 09:10:38 2013 -0500
--
CHANGES.txt|1 +
src/java/org/apache/cassandra/db/DataTracker.java |1 -
.../db/compaction/AbstractCompactionStrategy.java |2 +-
.../apache/cassandra/io/sstable/IndexSummary.java | 115 ---
.../cassandra/io/sstable/IndexSummaryBuilder.java | 76 ++
.../apache/cassandra/io/sstable/SSTableReader.java | 43 +++---
.../apache/cassandra/io/sstable/SSTableWriter.java | 11 +-
.../org/apache/cassandra/utils/ByteBufferUtil.java |8 +
.../cassandra/io/sstable/SSTableReaderTest.java|4 +-
9 files changed, 170 insertions(+), 91 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7634742..c843e5e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2.5
+ * reduce memory consumption of IndexSummary (CASSANDRA-5506)
* remove per-row column name bloom filters (CASSANDRA-5492)
* Include fatal errors in trace events (CASSANDRA-5447)
* Ensure that PerRowSecondaryIndex is notified of row-level deletes
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/src/java/org/apache/cassandra/db/DataTracker.java
--
diff --git a/src/java/org/apache/cassandra/db/DataTracker.java
b/src/java/org/apache/cassandra/db/DataTracker.java
index adbe037..7f6e94c 100644
--- a/src/java/org/apache/cassandra/db/DataTracker.java
+++ b/src/java/org/apache/cassandra/db/DataTracker.java
@@ -329,7 +329,6 @@ public class DataTracker
{
for (SSTableReader sstable : newSSTables)
{
-assert sstable.getKeySamples() != null;
if (logger.isDebugEnabled())
logger.debug(String.format(adding %s to list of files tracked
for %s.%s,
sstable.descriptor, cfstore.table.name,
cfstore.getColumnFamilyName()));
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index cb15109..a588216 100644
---
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -190,7 +190,7 @@ public abstract class AbstractCompactionStrategy
else
{
// what percentage of columns do we expect to compact outside of
overlap?
-if (sstable.getKeySamples().size() 2)
+if (sstable.getKeySamples().length 2)
{
// we have too few samples to estimate correct percentage
return false;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
b/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
index 1b9291b..3213d20 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
@@ -21,109 +21,110 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.nio.ByteBuffer;
-import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.RowPosition;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.utils.ByteBufferUtil;
-/**
- * Two approaches to building an IndexSummary:
- * 1. Call maybeAddEntry with every potential index entry
- * 2. Call shouldAddEntry, [addEntry,] incrementRowid
- */
public class IndexSummary
{
[jira] [Commented] (CASSANDRA-5506) Reduce memory consumption of IndexSummary
[ https://issues.apache.org/jira/browse/CASSANDRA-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644030#comment-13644030 ] Jonathan Ellis commented on CASSANDRA-5506: --- committed; created CASSANDRA-5521 for off-heap feature. Reduce memory consumption of IndexSummary - Key: CASSANDRA-5506 URL: https://issues.apache.org/jira/browse/CASSANDRA-5506 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Nick Puz Assignee: Jonathan Ellis Fix For: 1.2.5 I am evaluating cassandra for a use case with many tiny rows which would result in a node with 1-3TB of storage having billions of rows. Before loading that much data I am hitting GC issues and when looking at the heap dump I noticed that 70+% of the memory was used by IndexSummaries. The two major issues seem to be: 1) that the positions are stored as an ArrayListLong which results in each position taking 24 bytes (class + flags + 8 byte long). This might make sense when the file is initially written but once it has been serialized it would be a lot more memory efficient to just have an long[] (really a int[] would be fine unless 2GB sstables are allowed). 2) The DecoratedKey for a byte[16] key takes 195 bytes -- this is for the overhead of the ByteBuffer in the key and overhead in the token. To somewhat work around the problem I have increased index_sample but will this many rows that didn't really help starts to have diminishing returns. NOTE: This heap dump was from linux with a 64bit oracle vm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[1/3] git commit: Reduce memory consumption of IndexSummary patch by jbellis; reviewed by vijay for CASSANDRA-5506
Updated Branches:
refs/heads/trunk 1aa987402 - 4439b4117
Reduce memory consumption of IndexSummary
patch by jbellis; reviewed by vijay for CASSANDRA-5506
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9851b73f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9851b73f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9851b73f
Branch: refs/heads/trunk
Commit: 9851b73fc3bbaf61ec80b54c64483bcf56aaa396
Parents: dc5b1e9
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 28 09:10:38 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 28 09:10:38 2013 -0500
--
CHANGES.txt|1 +
src/java/org/apache/cassandra/db/DataTracker.java |1 -
.../db/compaction/AbstractCompactionStrategy.java |2 +-
.../apache/cassandra/io/sstable/IndexSummary.java | 115 ---
.../cassandra/io/sstable/IndexSummaryBuilder.java | 76 ++
.../apache/cassandra/io/sstable/SSTableReader.java | 43 +++---
.../apache/cassandra/io/sstable/SSTableWriter.java | 11 +-
.../org/apache/cassandra/utils/ByteBufferUtil.java |8 +
.../cassandra/io/sstable/SSTableReaderTest.java|4 +-
9 files changed, 170 insertions(+), 91 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7634742..c843e5e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2.5
+ * reduce memory consumption of IndexSummary (CASSANDRA-5506)
* remove per-row column name bloom filters (CASSANDRA-5492)
* Include fatal errors in trace events (CASSANDRA-5447)
* Ensure that PerRowSecondaryIndex is notified of row-level deletes
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/src/java/org/apache/cassandra/db/DataTracker.java
--
diff --git a/src/java/org/apache/cassandra/db/DataTracker.java
b/src/java/org/apache/cassandra/db/DataTracker.java
index adbe037..7f6e94c 100644
--- a/src/java/org/apache/cassandra/db/DataTracker.java
+++ b/src/java/org/apache/cassandra/db/DataTracker.java
@@ -329,7 +329,6 @@ public class DataTracker
{
for (SSTableReader sstable : newSSTables)
{
-assert sstable.getKeySamples() != null;
if (logger.isDebugEnabled())
logger.debug(String.format(adding %s to list of files tracked
for %s.%s,
sstable.descriptor, cfstore.table.name,
cfstore.getColumnFamilyName()));
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index cb15109..a588216 100644
---
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -190,7 +190,7 @@ public abstract class AbstractCompactionStrategy
else
{
// what percentage of columns do we expect to compact outside of
overlap?
-if (sstable.getKeySamples().size() 2)
+if (sstable.getKeySamples().length 2)
{
// we have too few samples to estimate correct percentage
return false;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9851b73f/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
b/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
index 1b9291b..3213d20 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummary.java
@@ -21,109 +21,110 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.nio.ByteBuffer;
-import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.RowPosition;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.utils.ByteBufferUtil;
-/**
- * Two approaches to building an IndexSummary:
- * 1. Call maybeAddEntry with every potential index entry
- * 2. Call shouldAddEntry, [addEntry,] incrementRowid
- */
public class IndexSummary
{
public
[3/3] git commit: ghetto-merge from cassandra 1.2
ghetto-merge from cassandra 1.2
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4439b411
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4439b411
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4439b411
Branch: refs/heads/trunk
Commit: 4439b4117dad85b206f3cd8d4bfbb4a5d1a9e822
Parents: 4ab535a
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 28 09:49:56 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 28 09:49:56 2013 -0500
--
CHANGES.txt|1 +
.../cassandra/cql3/statements/SelectStatement.java |1 +
src/java/org/apache/cassandra/db/DataTracker.java |1 -
.../db/compaction/AbstractCompactionStrategy.java |2 +-
.../apache/cassandra/io/sstable/IndexSummary.java | 116 +++
.../apache/cassandra/io/sstable/SSTableReader.java | 43 +++---
.../apache/cassandra/io/sstable/SSTableWriter.java | 12 +-
.../apache/cassandra/service/StorageService.java |4 +-
.../org/apache/cassandra/utils/ByteBufferUtil.java |8 +
.../cassandra/io/sstable/SSTableReaderTest.java|4 +-
10 files changed, 97 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4439b411/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 10e2e6e..56e671e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -40,6 +40,7 @@
* Add binary protocol versioning (CASSANDRA-5436)
1.2.5
+ * reduce memory consumption of IndexSummary (CASSANDRA-5506)
* remove per-row column name bloom filters (CASSANDRA-5492)
* Include fatal errors in trace events (CASSANDRA-5447)
* Ensure that PerRowSecondaryIndex is notified of row-level deletes
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4439b411/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
index e857104..38fa361 100644
--- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
@@ -273,6 +273,7 @@ public class SelectStatement implements CQLStatement
// to account for the grouping of columns.
// Since that doesn't work for maps/sets/lists, we now use the
compositesToGroup option of SliceQueryFilter.
// But we must preserve backward compatibility too (for mixed
version cluster that is).
+int multiplier = cfDef.isCompact ? 1 : (cfDef.metadata.size() + 1);
int toGroup = cfDef.isCompact ? -1 : cfDef.columns.size();
ColumnSlice slice = new ColumnSlice(getRequestedBound(Bound.START,
variables),
getRequestedBound(Bound.END,
variables));
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4439b411/src/java/org/apache/cassandra/db/DataTracker.java
--
diff --git a/src/java/org/apache/cassandra/db/DataTracker.java
b/src/java/org/apache/cassandra/db/DataTracker.java
index d4974d8..c5a9c2b 100644
--- a/src/java/org/apache/cassandra/db/DataTracker.java
+++ b/src/java/org/apache/cassandra/db/DataTracker.java
@@ -342,7 +342,6 @@ public class DataTracker
{
for (SSTableReader sstable : newSSTables)
{
-assert sstable.getKeySamples() != null;
if (logger.isDebugEnabled())
logger.debug(String.format(adding %s to list of files tracked
for %s.%s,
sstable.descriptor, cfstore.table.getName(),
cfstore.name));
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4439b411/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index 18af985..aefacd6 100644
---
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -251,7 +251,7 @@ public abstract class AbstractCompactionStrategy
else
{
// what percentage of columns do we expect to compact outside of
overlap?
-if (sstable.getKeySamples().size() 2)
+if (sstable.getKeySamples().length 2)
{
// we have too few samples to estimate correct percentage
git commit: fix merge issues
Updated Branches:
refs/heads/trunk 4439b4117 - 385ce13f5
fix merge issues
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/385ce13f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/385ce13f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/385ce13f
Branch: refs/heads/trunk
Commit: 385ce13f5ac3be91e40caa4f2cb34ec0ce654719
Parents: 4439b41
Author: Dave Brosius dbros...@apache.org
Authored: Sun Apr 28 12:23:10 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Sun Apr 28 12:23:10 2013 -0400
--
.../cassandra/io/sstable/IndexSummaryBuilder.java | 76 +++
.../apache/cassandra/io/sstable/SSTableReader.java |4 +-
.../apache/cassandra/io/sstable/SSTableWriter.java |4 +-
3 files changed, 80 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/385ce13f/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
new file mode 100644
index 000..12be453
--- /dev/null
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable;
+
+import java.util.ArrayList;
+
+import com.google.common.primitives.Bytes;
+import com.google.common.primitives.Longs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.utils.ByteBufferUtil;
+
+public class IndexSummaryBuilder
+{
+private static final Logger logger =
LoggerFactory.getLogger(IndexSummaryBuilder.class);
+
+private final ArrayListLong positions;
+private final ArrayListbyte[] keys;
+private long keysWritten = 0;
+
+public IndexSummaryBuilder(long expectedKeys, int indexInterval)
+{
+long expectedEntries = expectedKeys / indexInterval;
+if (expectedEntries Integer.MAX_VALUE)
+{
+// that's a _lot_ of keys, and a very low interval
+int effectiveInterval = (int) Math.ceil((double) Integer.MAX_VALUE
/ expectedKeys);
+expectedEntries = expectedKeys / effectiveInterval;
+assert expectedEntries = Integer.MAX_VALUE : expectedEntries;
+logger.warn(Index interval of {} is too low for {} expected keys;
using interval of {} instead,
+indexInterval, expectedKeys, effectiveInterval);
+}
+positions = new ArrayListLong((int)expectedEntries);
+keys = new ArrayListbyte[]((int)expectedEntries);
+}
+
+public IndexSummaryBuilder maybeAddEntry(DecoratedKey decoratedKey, int
indexInterval, long indexPosition)
+{
+if (keysWritten % indexInterval == 0)
+{
+keys.add(ByteBufferUtil.getArray(decoratedKey.key));
+positions.add(indexPosition);
+}
+keysWritten++;
+
+return this;
+}
+
+public IndexSummary build(IPartitioner partitioner, int indexInterval)
+{
+byte[][] keysArray = new byte[keys.size()][];
+for (int i = 0; i keys.size(); i++)
+keysArray[i] = keys.get(i);
+
+return new IndexSummary(partitioner, keysArray,
Longs.toArray(positions), indexInterval);
+}
+}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/385ce13f/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
index 2d6c1df..27589f5 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
+++
git commit: keep a local copy of indexInterval in summary builder so we don't need to pass it to maybeAddEntry
Updated Branches:
refs/heads/trunk 385ce13f5 - facaaf504
keep a local copy of indexInterval in summary builder so we don't need to pass
it to maybeAddEntry
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/facaaf50
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/facaaf50
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/facaaf50
Branch: refs/heads/trunk
Commit: facaaf504203f303c78fbda039d26e005ec3e74e
Parents: 385ce13
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 28 14:21:32 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 28 14:21:32 2013 -0500
--
.../cassandra/io/sstable/IndexSummaryBuilder.java |9 +
.../apache/cassandra/io/sstable/SSTableReader.java |2 +-
.../apache/cassandra/io/sstable/SSTableWriter.java |4 ++--
3 files changed, 8 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/facaaf50/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
index 12be453..7b06ee5 100644
--- a/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
+++ b/src/java/org/apache/cassandra/io/sstable/IndexSummaryBuilder.java
@@ -19,7 +19,6 @@ package org.apache.cassandra.io.sstable;
import java.util.ArrayList;
-import com.google.common.primitives.Bytes;
import com.google.common.primitives.Longs;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -35,10 +34,12 @@ public class IndexSummaryBuilder
private final ArrayListLong positions;
private final ArrayListbyte[] keys;
+private final int indexInterval;
private long keysWritten = 0;
public IndexSummaryBuilder(long expectedKeys, int indexInterval)
{
+this.indexInterval = indexInterval;
long expectedEntries = expectedKeys / indexInterval;
if (expectedEntries Integer.MAX_VALUE)
{
@@ -47,13 +48,13 @@ public class IndexSummaryBuilder
expectedEntries = expectedKeys / effectiveInterval;
assert expectedEntries = Integer.MAX_VALUE : expectedEntries;
logger.warn(Index interval of {} is too low for {} expected keys;
using interval of {} instead,
-indexInterval, expectedKeys, effectiveInterval);
+indexInterval, expectedKeys, effectiveInterval);
}
positions = new ArrayListLong((int)expectedEntries);
keys = new ArrayListbyte[]((int)expectedEntries);
}
-public IndexSummaryBuilder maybeAddEntry(DecoratedKey decoratedKey, int
indexInterval, long indexPosition)
+public IndexSummaryBuilder maybeAddEntry(DecoratedKey decoratedKey, long
indexPosition)
{
if (keysWritten % indexInterval == 0)
{
@@ -65,7 +66,7 @@ public class IndexSummaryBuilder
return this;
}
-public IndexSummary build(IPartitioner partitioner, int indexInterval)
+public IndexSummary build(IPartitioner partitioner)
{
byte[][] keysArray = new byte[keys.size()][];
for (int i = 0; i keys.size(); i++)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/facaaf50/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
index 27589f5..bc467c1 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
@@ -400,7 +400,7 @@ public class SSTableReader extends SSTable
// if summary was already read from disk we don't want to
re-populate it using primary index
if (!summaryLoaded)
{
-summaryBuilder.maybeAddEntry(decoratedKey,
metadata.getIndexInterval(), indexPosition);
+summaryBuilder.maybeAddEntry(decoratedKey, indexPosition);
ibuilder.addPotentialBoundary(indexPosition);
dbuilder.addPotentialBoundary(indexEntry.position);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/facaaf50/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
b/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
index c67e868..4b0b756 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
+++
git commit: meant to squash this in too
Updated Branches:
refs/heads/trunk facaaf504 - 0f8fc77b8
meant to squash this in too
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0f8fc77b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0f8fc77b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0f8fc77b
Branch: refs/heads/trunk
Commit: 0f8fc77b8bc2c5ddea8cecf69afb02567143b101
Parents: facaaf5
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 28 14:23:17 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 28 14:23:17 2013 -0500
--
.../apache/cassandra/io/sstable/SSTableReader.java |2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f8fc77b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
index bc467c1..2d6c1df 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableReader.java
@@ -407,7 +407,7 @@ public class SSTableReader extends SSTable
}
if (!summaryLoaded)
-indexSummary = summaryBuilder.build(partitioner,
metadata.getIndexInterval());
+indexSummary = summaryBuilder.build(partitioner);
}
finally
{
[jira] [Commented] (CASSANDRA-5521) move IndexSummary off heap
[ https://issues.apache.org/jira/browse/CASSANDRA-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644123#comment-13644123 ] Vijay commented on CASSANDRA-5521: -- Pushed the changes to https://github.com/Vijay2win/cassandra/commits/5521 The idea is as mentioned in CASSANDRA-5506 comments, Since this ticket is marked for 2.0 i took the liberty of changing the index summary file format. I am going to spend sometime on testing the performance difference (If any). Thanks! move IndexSummary off heap -- Key: CASSANDRA-5521 URL: https://issues.apache.org/jira/browse/CASSANDRA-5521 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jonathan Ellis Assignee: Vijay Fix For: 2.0 IndexSummary can still use a lot of heap for narrow-row sstables. (It can also contribute to memory fragmentation because of the large arrays it creates.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5472) Timeuuid with CLUSTERING ORDER DESC cannot be used with the dateOf CQL3 function
[ https://issues.apache.org/jira/browse/CASSANDRA-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644147#comment-13644147 ] Aleksey Yeschenko commented on CASSANDRA-5472: -- Makes sense and lgtm. +1 Timeuuid with CLUSTERING ORDER DESC cannot be used with the dateOf CQL3 function Key: CASSANDRA-5472 URL: https://issues.apache.org/jira/browse/CASSANDRA-5472 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.4 Reporter: Gareth Collins Assignee: Sylvain Lebresne Fix For: 1.2.5 Attachments: 5472.txt I originally raised this issue in the mailing lists: http://www.mail-archive.com/user@cassandra.apache.org/msg29185.html Here is what I tried: cqlsh:location create table test_y (message_id timeuuid, name text, PRIMARY KEY (name,message_id)); cqlsh:location insert into test_y (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_y (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_y (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_y (message_id,name) VALUES (now(),'foo'); cqlsh:location select dateOf(message_id) from test_y; dateOf(message_id) -- 2013-04-13 00:33:42-0400 2013-04-13 00:33:43-0400 2013-04-13 00:33:43-0400 2013-04-13 00:33:44-0400 cqlsh:location create table test_x (message_id timeuuid, name text, PRIMARY KEY (name,message_id)) WITH CLUSTERING ORDER BY (message_id DESC); cqlsh:location insert into test_x (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_x (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_x (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_x (message_id,name) VALUES (now(),'foo'); cqlsh:location insert into test_x (message_id,name) VALUES (now(),'foo'); cqlsh:location select dateOf(message_id) from test_x; Bad Request: Type error: org.apache.cassandra.cql3.statements.Selection$SimpleSelector@1e7318 cannot be passed as argument 0 of function dateof of type timeuuid It should be possible to use dateOf on message_id in table test_x -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: remove unused code/deprecations
Updated Branches:
refs/heads/trunk 0f8fc77b8 - 6c8b3ad0e
remove unused code/deprecations
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6c8b3ad0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6c8b3ad0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6c8b3ad0
Branch: refs/heads/trunk
Commit: 6c8b3ad0efaabd0e728eadd437ae0ed45237d01d
Parents: 0f8fc77
Author: Dave Brosius dbros...@apache.org
Authored: Sun Apr 28 22:49:29 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Sun Apr 28 22:49:29 2013 -0400
--
src/java/org/apache/cassandra/cli/CliClient.java |6 +-
1 files changed, 1 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6c8b3ad0/src/java/org/apache/cassandra/cli/CliClient.java
--
diff --git a/src/java/org/apache/cassandra/cli/CliClient.java
b/src/java/org/apache/cassandra/cli/CliClient.java
index 7ff9048..aa4e780 100644
--- a/src/java/org/apache/cassandra/cli/CliClient.java
+++ b/src/java/org/apache/cassandra/cli/CliClient.java
@@ -50,8 +50,6 @@ import org.apache.cassandra.utils.UUIDGen;
import org.apache.thrift.TBaseHelper;
import org.apache.thrift.TException;
import org.codehaus.jackson.*;
-import org.yaml.snakeyaml.Loader;
-import org.yaml.snakeyaml.TypeDescription;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.constructor.Constructor;
@@ -182,9 +180,7 @@ public class CliClient
try
{
final Constructor constructor = new Constructor(CliUserHelp.class);
-TypeDescription desc = new TypeDescription(CliUserHelp.class);
-desc.putListPropertyType(commands, CliCommandHelp.class);
-final Yaml yaml = new Yaml(new Loader(constructor));
+final Yaml yaml = new Yaml(constructor);
return (CliUserHelp) yaml.load(is);
}
finally
git commit: remove unused cql2 describeSchemaVersions
Updated Branches:
refs/heads/trunk 6c8b3ad0e - 0749c0415
remove unused cql2 describeSchemaVersions
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0749c041
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0749c041
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0749c041
Branch: refs/heads/trunk
Commit: 0749c0415edb480f98da501b09a212af4ed6d9d7
Parents: 6c8b3ad
Author: Dave Brosius dbros...@apache.org
Authored: Sun Apr 28 23:10:29 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Sun Apr 28 23:10:29 2013 -0400
--
.../org/apache/cassandra/cql/QueryProcessor.java |7 ---
1 files changed, 0 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0749c041/src/java/org/apache/cassandra/cql/QueryProcessor.java
--
diff --git a/src/java/org/apache/cassandra/cql/QueryProcessor.java
b/src/java/org/apache/cassandra/cql/QueryProcessor.java
index cf2231b..f896f24 100644
--- a/src/java/org/apache/cassandra/cql/QueryProcessor.java
+++ b/src/java/org/apache/cassandra/cql/QueryProcessor.java
@@ -338,13 +338,6 @@ public class QueryProcessor
throw new InvalidRequestException(range finish must come after
start in traversal order);
}
-private static MapString, ListString describeSchemaVersions()
-{
-// unreachable hosts don't count towards disagreement
-return Maps.filterKeys(StorageProxy.describeSchemaVersions(),
-
Predicates.not(Predicates.equalTo(StorageProxy.UNREACHABLE)));
-}
-
public static CqlResult processStatement(CQLStatement
statement,ThriftClientState clientState, ListByteBuffer variables )
throws RequestExecutionException, RequestValidationException
{
[jira] [Commented] (CASSANDRA-5521) move IndexSummary off heap
[ https://issues.apache.org/jira/browse/CASSANDRA-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644229#comment-13644229 ] Jonathan Ellis commented on CASSANDRA-5521: --- Seems inefficient to make the get pair the unit of fetching memory, since usually you want the key or the position but not both. (You wouldn't have to change the signature of getKey either, if you just fetched the key into a byte[] directly instead of wrapping half of it.) What's the upgrade path? Would prefer automatically rebuilds old summaries to user has to manually blow away summaries or it dies trying to start. move IndexSummary off heap -- Key: CASSANDRA-5521 URL: https://issues.apache.org/jira/browse/CASSANDRA-5521 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jonathan Ellis Assignee: Vijay Fix For: 2.0 IndexSummary can still use a lot of heap for narrow-row sstables. (It can also contribute to memory fragmentation because of the large arrays it creates.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: KeyIterator no longer needs to hold onto the Descriptor
Updated Branches:
refs/heads/trunk 0749c0415 - 383d35ea5
KeyIterator no longer needs to hold onto the Descriptor
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/383d35ea
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/383d35ea
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/383d35ea
Branch: refs/heads/trunk
Commit: 383d35ea516deb4f127c1c9884ee64561da3cb2b
Parents: 0749c04
Author: Dave Brosius dbros...@apache.org
Authored: Sun Apr 28 23:30:20 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Sun Apr 28 23:30:20 2013 -0400
--
.../apache/cassandra/io/sstable/KeyIterator.java |2 --
1 files changed, 0 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/383d35ea/src/java/org/apache/cassandra/io/sstable/KeyIterator.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/KeyIterator.java
b/src/java/org/apache/cassandra/io/sstable/KeyIterator.java
index 0af4537..f4f7ee5 100644
--- a/src/java/org/apache/cassandra/io/sstable/KeyIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/KeyIterator.java
@@ -32,11 +32,9 @@ import org.apache.cassandra.utils.CloseableIterator;
public class KeyIterator extends AbstractIteratorDecoratedKey implements
CloseableIteratorDecoratedKey
{
private final RandomAccessReader in;
-private final Descriptor desc;
public KeyIterator(Descriptor desc)
{
-this.desc = desc;
File path = new File(desc.filenameFor(SSTable.COMPONENT_INDEX));
in = RandomAccessReader.open(path);
}
[jira] [Commented] (CASSANDRA-5521) move IndexSummary off heap
[
https://issues.apache.org/jira/browse/CASSANDRA-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644237#comment-13644237
]
Vijay commented on CASSANDRA-5521:
--
{quote}
Seems inefficient to make the get pair the unit of fetching memory,
{quote}
Ahaa good point will fix it.
{quote}
What's the upgrade path? Would prefer automatically rebuilds old summaries to
user has to manually blow away summaries or it dies trying to start.
{quote}
It is automatic, but until the user runs scrub or until the new SST's are
created the startup will be slow.
move IndexSummary off heap
--
Key: CASSANDRA-5521
URL: https://issues.apache.org/jira/browse/CASSANDRA-5521
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jonathan Ellis
Assignee: Vijay
Fix For: 2.0
IndexSummary can still use a lot of heap for narrow-row sstables. (It can
also contribute to memory fragmentation because of the large arrays it
creates.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5517) Cassandra crashes at start with segmentation fault
[ https://issues.apache.org/jira/browse/CASSANDRA-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644293#comment-13644293 ] Sergey Naumov commented on CASSANDRA-5517: -- # java -version java version 1.6.0_27 OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.10.1) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) Cassandra crashes at start with segmentation fault -- Key: CASSANDRA-5517 URL: https://issues.apache.org/jira/browse/CASSANDRA-5517 Project: Cassandra Issue Type: Bug Components: Core Environment: VirtualBox 4.2.6 VM with 4GB RAM, Xubuntu 12.10 as host and guest OS. Cassandra 1.2.4 installed on guest as Debian package. Reporter: Sergey Naumov Sometimes Cassandra fails at start with segmentation fault: # /usr/sbin/cassandra -f xss = -ea -javaajent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k Segmentation fault It seems that not only me encountered this bug: http://snapwebsites.org/known-issues/cassandra-crashes-java-segmentation-fault Solution proposed on this link works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
