date:20120215

[jira] [Commented] (CASSANDRA-3907) Support compression using BulkWriter

2012-02-15 Thread Erik Forsberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208267#comment-13208267
 ] 

Erik Forsberg commented on CASSANDRA-3907:
--

So, with this patch, if I enable compression with BulkWriter, sstables will be 
compressed on the hadoop side, streamed in compressed form and the cassandra 
daemon does not have to compress them before writing to disk?

Or am I misunderstanding how this works?

 Support compression using BulkWriter
 

 Key: CASSANDRA-3907
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3907
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Chris Goffinet
Assignee: Chris Goffinet
 Fix For: 1.1.0

 Attachments: 0001-Add-compression-support-to-BulkWriter.patch


 Currently there is no way to enable compression using BulkWriter. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3721) Staggering repair

2012-02-15 Thread Sylvain Lebresne (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208268#comment-13208268
 ] 

Sylvain Lebresne commented on CASSANDRA-3721:
-

I think there have been a misunderstanding. I've attached 3721.patch that was 
addressing my concerns with the initial 
0001-staggering-repair-with-snapshot.patch. I'm personally good with 3721.patch 
(It may be that my poor wording suggested otherwise, sorry if that's the case), 
except that it needs review of course.

Vijay did commit 3721.patch, so I think that was right (though maybe Vijay you 
could have made it more clear in your comment that you did review the last 
version, +1ed it and thus committed it). 

 Staggering repair
 -

 Key: CASSANDRA-3721
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3721
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 1.1.0
Reporter: Vijay
Assignee: Vijay
Priority: Minor
 Fix For: 1.1.1

 Attachments: 0001-add-snapshot-command.patch, 
 0001-staggering-repair-with-snapshot.patch, 3721.patch


 Currently repair runs on all the nodes at once and causing the range of data 
 to be hot (higher latency on reads).
 Sequence:
 1) Send a repair request to all of the nodes so we can hold the references of 
 the SSTables (point at which repair was initiated)
 2) Send Validation on one node at a time (once completed will release 
 references).
 3) Hold the reference of the tree in the requesting node and once everything 
 is complete start diff.
 We can also serialize the streaming part not more than 1 node is involved in 
 the streaming.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-3862) RowCache misses Updates

2012-02-15 Thread Sylvain Lebresne (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-3862:

Attachment: 3862-v4.patch

Actually then handling of the copying patch by the preceding patches is wrong.
When a put arrives and there is a sentinel, the patch does not add the put to
the sentinel correctly. But thinking about it, for the copying cache, we should
avoid having writes check the current value in the cache, because that have a
non-negligible performance impact. What we should do is let invalidate actually
invalidate sentinels. The only problem we're faced with if we do that, is that
when a read-for-caching returns, it must make sure his own sentinel hasn't been
invalidated. And in particular it must be careful of the case where the
sentinel has been invalidated and another read has set another sentinel.

Anyway, attaching a v4 (that include the comments cleanups) that choose that
strategy instead (and thus is (hopefully) not buggy even in the copying cache
case). Note that it means that reads must be able to identify sentinels
uniquely (not based on the content), so the code assign a unique ID to sentinel
and use that for comparison.

RowCache misses Updates
---

Key: CASSANDRA-3862
URL: https://issues.apache.org/jira/browse/CASSANDRA-3862
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.6
Reporter: Daniel Doubleday
Assignee: Sylvain Lebresne
Fix For: 1.1.0

Attachments: 3862-cleanup.txt, 3862-v2.patch, 3862-v4.patch,
3862.patch, 3862_v3.patch, include_memtables_in_rowcache_read.patch

While performing stress tests to find any race problems for CASSANDRA-2864 I
guess I (re-)found one for the standard on-heap row cache.
During my stress test I hava lots of threads running with some of them only
reading other writing and re-reading the value.
This seems to happen:
- Reader tries to read row A for the first time doing a getTopLevelColumns
- Row A which is not in the cache yet is updated by Writer. The row is not
eagerly read during write (because we want fast writes) so the writer cannot
perform a cache update
- Reader puts the row in the cache which is now missing the update
I already asked this some time ago on the mailing list but unfortunately
didn't dig after I got no answer since I assumed that I just missed
something. In a way I still do but haven't found any locking mechanism that
makes sure that this should not happen.
The problem can be reproduced with every run of my stress test. When I
restart the server the expected column is there. It's just missing from the
cache.
To test I have created a patch that merges memtables with the row cache. With
the patch the problem is gone.
I can also reproduce in 0.8. Haven't checked 1.1 but I haven't found any
relevant change their either so I assume the same aplies there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

git commit: Staggering repair patch by Vijay and Sylvain Lebresne; reviewed by Sylvain Lebresne for CASSANDRA-3721

2012-02-15 Thread vijay

Updated Branches:
  refs/heads/trunk 6642d0f95 - ddee43e84


Staggering repair
patch by Vijay and Sylvain Lebresne; reviewed by Sylvain Lebresne for 
CASSANDRA-3721


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ddee43e8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ddee43e8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ddee43e8

Branch: refs/heads/trunk
Commit: ddee43e8463777b0419fac2423a59511202c8fab
Parents: 6642d0f
Author: Vijay Parthasarathy vijay2...@gmail.com
Authored: Wed Feb 15 01:48:32 2012 -0800
Committer: Vijay Parthasarathy vijay2...@gmail.com
Committed: Wed Feb 15 01:48:32 2012 -0800

--
 .../org/apache/cassandra/db/ColumnFamilyStore.java |8 +
 src/java/org/apache/cassandra/db/Directories.java  |   15 +
 .../cassandra/db/compaction/CompactionManager.java |   37 ++-
 .../cassandra/service/AntiEntropyService.java  |  223 --
 .../apache/cassandra/service/StorageService.java   |   13 +-
 .../cassandra/service/StorageServiceMBean.java |4 +-
 src/java/org/apache/cassandra/tools/NodeCmd.java   |7 +-
 src/java/org/apache/cassandra/tools/NodeProbe.java |8 +-
 .../apache/cassandra/io/CompactSerializerTest.java |1 +
 9 files changed, 257 insertions(+), 59 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ddee43e8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index e4e3204..218fadf 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1423,6 +1423,14 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 }
 }
 
+public ListSSTableReader getSnapshotSSTableReader(String tag) throws 
IOException
+{
+ListSSTableReader readers = new ArrayListSSTableReader();
+for (Map.EntryDescriptor, SetComponent entries : 
directories.sstableLister().snapshots(tag).list().entrySet())
+readers.add(SSTableReader.open(entries.getKey(), 
entries.getValue(), metadata, partitioner));
+return readers;
+}
+
 /**
  * Take a snap shot of this columnfamily store.
  *

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ddee43e8/src/java/org/apache/cassandra/db/Directories.java
--
diff --git a/src/java/org/apache/cassandra/db/Directories.java 
b/src/java/org/apache/cassandra/db/Directories.java
index 2afefd2..7c51830 100644
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@ -185,6 +185,7 @@ public class Directories
 private int nbFiles;
 private final MapDescriptor, SetComponent components = new 
HashMapDescriptor, SetComponent();
 private boolean filtered;
+private String snapshotName;
 
 public SSTableLister skipCompacted(boolean b)
 {
@@ -219,6 +220,14 @@ public class Directories
 return this;
 }
 
+public SSTableLister snapshots(String sn)
+{
+if (filtered)
+throw new IllegalStateException(list() has already been 
called);
+snapshotName = sn;
+return this;
+}
+
 public MapDescriptor, SetComponent list()
 {
 filter();
@@ -246,6 +255,12 @@ public class Directories
 
 for (File location : sstableDirectories)
 {
+if (snapshotName != null)
+{
+new File(location, join(SNAPSHOT_SUBDIR, 
snapshotName)).listFiles(getFilter());
+continue;
+}
+
 if (!onlyBackups)
 location.listFiles(getFilter());
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ddee43e8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index f30510b..ed699f2 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -807,23 +807,33 @@ public class CompactionManager implements 
CompactionManagerMBean
 if (!cfs.isValid())
 return;
 
-// flush first so everyone is validating data that is as similar as 
possible
-try
-{
-

[jira] [Resolved] (CASSANDRA-3721) Staggering repair

2012-02-15 Thread Vijay (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay resolved CASSANDRA-3721.
--

Resolution: Fixed

Sorry for the confusion, +1 for me and i committed it again... I did test it 
and unit test passes. Thanks!

 Staggering repair
 -

 Key: CASSANDRA-3721
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3721
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 1.1.0
Reporter: Vijay
Assignee: Vijay
Priority: Minor
 Fix For: 1.1.1

 Attachments: 0001-add-snapshot-command.patch, 
 0001-staggering-repair-with-snapshot.patch, 3721.patch


 Currently repair runs on all the nodes at once and causing the range of data 
 to be hot (higher latency on reads).
 Sequence:
 1) Send a repair request to all of the nodes so we can hold the references of 
 the SSTables (point at which repair was initiated)
 2) Send Validation on one node at a time (once completed will release 
 references).
 3) Hold the reference of the tree in the requesting node and once everything 
 is complete start diff.
 We can also serialize the streaming part not more than 1 node is involved in 
 the streaming.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3912) support incremental repair controlled by external agent

2012-02-15 Thread Sylvain Lebresne (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208387#comment-13208387
]

Sylvain Lebresne commented on CASSANDRA-3912:
-

I don't think that on the server side we should do more than expose the
forceTableRepair method taking the Range in argument (and I do think there is
value in doing that). The user friendly transformation of steps to range could
probably be left to external scripts imo, but at least it should be moved to
nodetool itself if we want it in.

As a side note, I'll remark that every repair of a range triggers a flush, so
one should probably be careful to not repair incrementally on too small a range.

support incremental repair controlled by external agent
---

As a poor man's pre-cursor to CASSANDRA-2699, exposing the ability to repair
small parts of a range is extremely useful because it allows (with external
scripting logic) to slowly repair a node's content over time. Other than
avoiding the bulkyness of complete repairs, it means that you can safely do
repairs even if you absolutely cannot afford e.g. disk spaces spikes (see
CASSANDRA-2699 for what the issues are).
Attaching a patch that exposes a repairincremental command to nodetool,
where you specify a step and the number of total steps. Incrementally
performing a repair in 100 steps, for example, would be done by:
{code}
nodetool repairincremental 0 100
nodetool repairincremental 1 100
...
nodetool repairincremental 99 100
{code}
An external script can be used to keep track of what has been repaired and
when. This should allow (1) allow incremental repair to happen now/soon, and
(2) allow experimentation and evaluation for an implementation of
CASSANDRA-2699 which I still think is a good idea. This patch does nothing to
help the average deployment, but at least makes incremental repair possible
given sufficient effort spent on external scripting.
The big no-no about the patch is that it is entirely specific to
RandomPartitioner and BigIntegerToken. If someone can suggest a way to
implement this command generically using the Range/Token abstractions, I'd be
happy to hear suggestions.
An alternative would be to provide a nodetool command that allows you to
simply specify the specific token ranges on the command line. It makes using
it a bit more difficult, but would mean that it works for any partitioner and
token type.
Unless someone can suggest a better way to do this, I think I'll provide a
patch that does this. I'm still leaning towards supporting the simple step N
out of M form though.

[jira] [Created] (CASSANDRA-3913) Incorrect InetAddress equality test

2012-02-15 Thread Brandon Williams (Created) (JIRA)

Incorrect InetAddress equality test
---

 Key: CASSANDRA-3913
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3913
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.0.6, 0.8.9
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Trivial
 Fix For: 1.0.8
 Attachments: 3913.txt

CASSANDRA-3485 introduced some InetAddress checks using == instead of .equals.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-3913) Incorrect InetAddress equality test

2012-02-15 Thread Brandon Williams (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-3913:


Attachment: 3913.txt

 Incorrect InetAddress equality test
 ---

 Key: CASSANDRA-3913
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3913
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.9, 1.0.6
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Trivial
 Fix For: 1.0.8

 Attachments: 3913.txt


 CASSANDRA-3485 introduced some InetAddress checks using == instead of .equals.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-3907) Support compression using BulkWriter

2012-02-15 Thread Brandon Williams (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208419#comment-13208419
 ] 

Brandon Williams commented on CASSANDRA-3907:
-

Your understanding is correct.

 Support compression using BulkWriter
 

 Key: CASSANDRA-3907
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3907
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Chris Goffinet
Assignee: Chris Goffinet
 Fix For: 1.1.0

 Attachments: 0001-Add-compression-support-to-BulkWriter.patch


 Currently there is no way to enable compression using BulkWriter. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

69 matches

Mail list logo