[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13012867#comment-13012867 ] Peter Schuller commented on CASSANDRA-1902: --- bq. Documentation for posix_fadvice/madvice calls suggests to do more frequent little requests instead of big requests - kernel in the high possibility going to ignore an advice on the big region. I suspect this comes from WILLNEED being intended as a read-ahead hint. The implementation puts a maximum cap on the amount of read-ahead it'll do: http://lxr.free-electrons.com/source/mm/fadvise.c#L95 http://lxr.free-electrons.com/source/mm/readahead.c#L203 http://lxr.free-electrons.com/source/mm/readahead.c#L236 My knowledge of the VM subsystem fails me but my interpretation of max_sane_readahead() is that it's trying to essentially cap based on available memory resources such that it may or may not do the read-ahead (or do it to a lesser extent) based on memory pressure. This is different from e.g. DONTNEED which is fine to call for huge segments and seems to be O(n) (as per CASSANDRA-1470 discussion). Btw, it's also possible to populate with an mmap() call with MAP_POPULATE (Linux-specific, but we're totally Linux specific anyway). I haven't checked but empirically I suspect this is not subject to limits on how much you populate - and the call is also blocking. bq. Note that WILLNEED is non-blocking call. Implementation: http://lxr.free-electrons.com/source/mm/readahead.c#L145 which boils down to a call to: http://lxr.free-electrons.com/source/mm/readahead.c#L109 So yes it's non-blocking, but you're still either going to take the random I/O (remember that as of CASSANDRA-1470 we know nothing will be in page cache) or else requests would be dropped somewhere (causing less hotness). If you're taking the I/O, again either the I/O is done in such a way as to essentially be serial or else it's done with some level of parallelism. Which one will affect the impact is has on other I/O. bq. We can't stop using DONTNEED while writing compacted file because it will suck pages from sstables which are currently in use. And we do WILLNEED's only when we have SSTableReader for a compacted file ready - right before old sstables going to be replaced with new ones so this is not going to make a big performance impact on the reads. The whole process is going to be suboptimal yes since both the old and the new sstable will be cached. But note that even though the read-ahead is done at the point of the switch, the actual freeing of cached pages associated with the old sstable is still happening at some point in the future as a result of GC (right? or am I misreading?). My thinking has been that something along the lines of CASSANDRA-1608 and capping sstable sizes may helt mitigate these effects at some point in the future. Meanwhile, the only way I see to avoid double caching is to DONTNEED the old table first, but that has the obvious effects on live traffic. There is definitely the trade-off here and the timing is different in the WILLNEED-just-before-switch case, maybe leading to more data being retained hot, but at the cost of all that seek-bound I/O (or am I wrong about that bit?) which, for large sstables, will happen over some non-trivial amount of time. Anyways... there are lots of details and they're difficult to express in a organized fashion. I think the main thing at least is to just be explicitly aware of the seek-bound I/O implied by compaction and probably confirm what the behavior actually is for a large sstable - specifically whether it will cause a storm of highly concurrent I/O with an insanely high queue depth (= *very* bad for live reads) or a slower mostly serial background read-in (= much better for live reads) or somewhere in between. (I'll try to look into that, subject to time.) Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: T Jake Luciani Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads
[jira] [Commented] (CASSANDRA-2347) index scan uses incorrect comparator on non-indexed expressions
[ https://issues.apache.org/jira/browse/CASSANDRA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13012885#comment-13012885 ] Roland Gude commented on CASSANDRA-2347: i just created an account. just in case it is needed and maybe i can contribute more in the future. index scan uses incorrect comparator on non-indexed expressions --- Key: CASSANDRA-2347 URL: https://issues.apache.org/jira/browse/CASSANDRA-2347 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 0.7.5 Attachments: 2347.txt When multiple index expressions are specified, the column name comparator is used when evaluating secondary (non-indexed) expressions after an indexed expression match. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2404) if out of disk space reclaim compacted SSTables during memtable flush
if out of disk space reclaim compacted SSTables during memtable flush - Key: CASSANDRA-2404 URL: https://issues.apache.org/jira/browse/CASSANDRA-2404 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.4 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Fix For: 0.7.5 During compaction if there is not enough disk space we invoke GC to reclaim unused space. During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table. Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1902: --- Attachment: CASSANDRA-1902-v6.patch Fixed doc string for SegmentedFile.complete() and renamed isRangeInCache() to wasRangeInCache(). I have made changes to NativeMappedSegment.migrateCachedPages() and BufferedSegmentedFile.tryPreserveFilePageCache() so it does not do syscall for each page, instead it collectes continues regions of the cache and does a single call on each to minimise seeking as much as possible. Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: T Jake Luciani Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1086955 - /cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
Author: gdusbabek
Date: Wed Mar 30 13:46:32 2011
New Revision: 1086955
URL: http://svn.apache.org/viewvc?rev=1086955view=rev
Log:
fix broken jdbc tests
Modified:
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
Modified:
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java?rev=1086955r1=1086954r2=1086955view=diff
==
---
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
(original)
+++
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
Wed Mar 30 13:46:32 2011
@@ -56,6 +56,7 @@ public class JdbcDriverTest extends Embe
private static final String firstrec =
FBUtilities.bytesToHex(firstrec.getBytes());
private static final String last =
FBUtilities.bytesToHex(last.getBytes());
private static final String lastrec =
FBUtilities.bytesToHex(lastrec.getBytes());
+private static final String jsmith =
FBUtilities.bytesToHex(jsmith.getBytes());
/** SetUp */
@BeforeClass
@@ -66,16 +67,25 @@ public class JdbcDriverTest extends Embe
con =
DriverManager.getConnection(jdbc:cassandra:root/root@localhost:9170/Keyspace1);
String[] inserts =
{
-String.format(UPDATE Standard1 SET '%s' = '%s', '%s' = '%s' WHERE
KEY = 'jsmith', first, firstrec, last, lastrec),
-UPDATE JdbcInteger SET 1 = 11, 2 = 22 WHERE KEY = 'jsmith',
-UPDATE JdbcInteger SET 3 = 33, 4 = 44 WHERE KEY = 'jsmith',
-UPDATE JdbcLong SET 1 = 11, 2 = 22 WHERE KEY = 'jsmith',
-UPDATE JdbcAscii SET 'first' = 'firstrec', 'last' = 'lastrec'
WHERE key = 'jsmith',
-String.format(UPDATE JdbcBytes SET '%s' = '%s', '%s' = '%s' WHERE
key = 'jsmith', first, firstrec, last, lastrec),
-UPDATE JdbcUtf8 SET 'first' = 'firstrec', 'last' = 'lastrec'
WHERE key = 'jsmith',
+String.format(UPDATE Standard1 SET '%s' = '%s', '%s' = '%s' WHERE
KEY = '%s', first, firstrec, last, lastrec, jsmith),
+UPDATE JdbcInteger SET 1 = 11, 2 = 22 WHERE KEY = ' + jsmith +
',
+UPDATE JdbcInteger SET 3 = 33, 4 = 44 WHERE KEY = ' + jsmith +
',
+UPDATE JdbcLong SET 1 = 11, 2 = 22 WHERE KEY = ' + jsmith + ',
+UPDATE JdbcAscii SET 'first' = 'firstrec', 'last' = 'lastrec'
WHERE key = ' + jsmith + ',
+String.format(UPDATE JdbcBytes SET '%s' = '%s', '%s' = '%s' WHERE
key = '%s', first, firstrec, last, lastrec, jsmith),
+UPDATE JdbcUtf8 SET 'first' = 'firstrec', 'last' = 'lastrec'
WHERE key = ' + jsmith + ',
};
for (String q : inserts)
-executeNoResults(con, q);
+{
+try
+{
+executeNoResults(con, q);
+}
+catch (SQLException ex)
+{
+throw new AssertionError(ex.getMessage() + , query: + q);
+}
+}
}
private static void expectedMetaData(ResultSetMetaData md, int col, String
colClass, String table, String schema,
@@ -112,9 +122,10 @@ public class JdbcDriverTest extends Embe
@Test
public void testIntegerMetadata() throws SQLException
{
+String key = FBUtilities.bytesToHex(Integer.getBytes());
Statement stmt = con.createStatement();
-stmt.executeUpdate(UPDATE JdbcInteger SET 1=111, 2=222 WHERE KEY =
'Integer');
-ResultSet rs = stmt.executeQuery(SELECT 1, 2 from JdbcInteger WHERE
KEY = 'Integer');
+stmt.executeUpdate(UPDATE JdbcInteger SET 1=111, 2=222 WHERE KEY = '
+ key + ');
+ResultSet rs = stmt.executeQuery(SELECT 1, 2 from JdbcInteger WHERE
KEY = ' + key + ');
assert rs.next();
assert rs.getInt(1) == 111;
assert rs.getInt(2) == 222;
@@ -136,9 +147,10 @@ public class JdbcDriverTest extends Embe
@Test
public void testLongMetadata() throws SQLException
{
+String key = FBUtilities.bytesToHex(Long.getBytes());
Statement stmt = con.createStatement();
-stmt.executeUpdate(UPDATE JdbcLong SET 1=111, 2=222 WHERE KEY =
'Long');
-ResultSet rs = stmt.executeQuery(SELECT 1, 2 from JdbcLong WHERE KEY
= 'Long');
+stmt.executeUpdate(UPDATE JdbcLong SET 1=111, 2=222 WHERE KEY = ' +
key + ');
+ResultSet rs = stmt.executeQuery(SELECT 1, 2 from JdbcLong WHERE KEY
= ' + key + ');
assert rs.next();
assert rs.getLong(1) == 111;
assert rs.getLong(2) == 222;
@@ -159,11 +171,13 @@ public class JdbcDriverTest extends Embe
@Test
public void testStringMetadata() throws SQLException
{
+String aKey = FBUtilities.bytesToHex(ascii.getBytes());
+String uKey = FBUtilities.bytesToHex(utf8.getBytes());
buildbot success in ASF Buildbot on cassandra-trunk
The Buildbot has detected a restored build on builder cassandra-trunk while building ASF Buildbot. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/1198 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: isis_ubuntu Build Reason: scheduler Build Source Stamp: [branch cassandra/trunk] 1086955 Blamelist: gdusbabek Build succeeded! sincerely, -The Buildbot
[jira] [Commented] (CASSANDRA-2361) AES depends on java serialization
[ https://issues.apache.org/jira/browse/CASSANDRA-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13012979#comment-13012979 ] Hudson commented on CASSANDRA-2361: --- Integrated in Cassandra #819 (See [https://hudson.apache.org/hudson/job/Cassandra/819/]) Use ICompactSerializer in AES instead of java serialization (and fix tests warnings) patch by gdusbabek; reviewed by slebresne for CASSANDRA-2361 AES depends on java serialization - Key: CASSANDRA-2361 URL: https://issues.apache.org/jira/browse/CASSANDRA-2361 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.4 Reporter: Gary Dusbabek Assignee: Gary Dusbabek Priority: Minor Fix For: 0.7.5, 0.8 Attachments: 2361-part1.patch, v2-0001-ICompactSerializers-for-MerkleTrees.txt, v2-0002-get-rid-of-annoying-AES-emissions.txt 0.8 should be able to run in the same cluster as 0.7. AES uses java serialization which means that Token serialization stands a good chance of being brittle. This needs to be fixed. 1. place a hard-coded serialVersionUID for Token in 0.7.5. 2. have AES use ICompactSerializer in place of ObjectInputStream in 0.8. This would be a good opportunity to audit the code for imprudent uses of Object[Input|Output]Stream and get those cleaned up. Also, it will exercise the versioning code a bit. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1086964 - in /cassandra/trunk: src/java/org/apache/cassandra/db/marshal/ test/unit/org/apache/cassandra/db/marshal/
Author: gdusbabek
Date: Wed Mar 30 14:30:33 2011
New Revision: 1086964
URL: http://svn.apache.org/viewvc?rev=1086964view=rev
Log:
AbstractType converts types to Strings (not just ByteBuffers)
Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AbstractType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AsciiType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/BytesType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/CounterColumnType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/IntegerType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/LexicalUUIDType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/LocalByPartionerType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/LongType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/UTF8Type.java
cassandra/trunk/test/unit/org/apache/cassandra/db/marshal/RoundTripTest.java
Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AbstractType.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AbstractType.java?rev=1086964r1=1086963r2=1086964view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AbstractType.java
(original)
+++ cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AbstractType.java
Wed Mar 30 14:30:33 2011
@@ -94,6 +94,9 @@ public abstract class AbstractTypeT im
public abstract T compose(ByteBuffer bytes);
+/** get a string representation of a particular type. */
+public abstract String toString(T t);
+
/** get a string representation of the bytes suitable for log messages */
public abstract String getString(ByteBuffer bytes);
Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AsciiType.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AsciiType.java?rev=1086964r1=1086963r2=1086964view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AsciiType.java
(original)
+++ cassandra/trunk/src/java/org/apache/cassandra/db/marshal/AsciiType.java Wed
Mar 30 14:30:33 2011
@@ -34,7 +34,6 @@ public class AsciiType extends AbstractT
AsciiType() {} // singleton
-@Override
public String getString(ByteBuffer bytes)
{
try
@@ -47,6 +46,11 @@ public class AsciiType extends AbstractT
}
}
+public String toString(String s)
+{
+return s;
+}
+
public int compare(ByteBuffer o1, ByteBuffer o2)
{
return BytesType.bytesCompare(o1, o2);
Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/BytesType.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/marshal/BytesType.java?rev=1086964r1=1086963r2=1086964view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/db/marshal/BytesType.java
(original)
+++ cassandra/trunk/src/java/org/apache/cassandra/db/marshal/BytesType.java Wed
Mar 30 14:30:33 2011
@@ -57,6 +57,11 @@ public class BytesType extends AbstractT
return ByteBufferUtil.bytesToHex(bytes);
}
+public String toString(ByteBuffer byteBuffer)
+{
+return getString(byteBuffer);
+}
+
public ByteBuffer fromString(String source)
{
try
Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/CounterColumnType.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/marshal/CounterColumnType.java?rev=1086964r1=1086963r2=1086964view=diff
==
---
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/CounterColumnType.java
(original)
+++
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/CounterColumnType.java
Wed Mar 30 14:30:33 2011
@@ -45,6 +45,11 @@ public class CounterColumnType extends A
return ByteBufferUtil.bytesToHex(bytes);
}
+public String toString(Long l)
+{
+return l.toString();
+}
+
/**
* create commutative column
*/
Modified:
cassandra/trunk/src/java/org/apache/cassandra/db/marshal/IntegerType.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/marshal/IntegerType.java?rev=1086964r1=1086963r2=1086964view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/db/marshal/IntegerType.java
(original)
+++ cassandra/trunk/src/java/org/apache/cassandra/db/marshal/IntegerType.java
Wed Mar 30 14:30:33 2011
@@ -129,6
svn commit: r1086966 - /cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/Utils.java
Author: gdusbabek
Date: Wed Mar 30 14:30:48 2011
New Revision: 1086966
URL: http://svn.apache.org/viewvc?rev=1086966view=rev
Log:
compress utf8 bytes and not platform bytes.
Modified:
cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/Utils.java
Modified:
cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/Utils.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/Utils.java?rev=1086966r1=1086965r2=1086966view=diff
==
--- cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/Utils.java
(original)
+++ cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/Utils.java
Wed Mar 30 14:30:48 2011
@@ -28,6 +28,7 @@ import java.sql.Types;
import java.util.UUID;
import java.util.zip.Deflater;
+import com.google.common.base.Charsets;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.BytesType;
import org.apache.cassandra.db.marshal.IntegerType;
@@ -42,7 +43,7 @@ class Utils
public static ByteBuffer compressQuery(String queryStr, Compression
compression)
{
-byte[] data = queryStr.getBytes();
+byte[] data = queryStr.getBytes(Charsets.UTF_8);
Deflater compressor = new Deflater();
compressor.setInput(data);
compressor.finish();
svn commit: r1086967 - in /cassandra/trunk/drivers/java: src/org/apache/cassandra/cql/jdbc/CassandraResultSet.java test/org/apache/cassandra/cql/JdbcDriverTest.java test/org/apache/cassandra/cql/jdbc/
Author: gdusbabek
Date: Wed Mar 30 14:30:55 2011
New Revision: 1086967
URL: http://svn.apache.org/viewvc?rev=1086967view=rev
Log:
fix off-by-one in CassandraResultSet
Modified:
cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/CassandraResultSet.java
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/jdbc/PreparedStatementTest.java
Modified:
cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/CassandraResultSet.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/CassandraResultSet.java?rev=1086967r1=1086966r2=1086967view=diff
==
---
cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/CassandraResultSet.java
(original)
+++
cassandra/trunk/drivers/java/src/org/apache/cassandra/cql/jdbc/CassandraResultSet.java
Wed Mar 30 14:30:55 2011
@@ -365,7 +365,7 @@ class CassandraResultSet implements Resu
*/
public byte[] getBytes(int index) throws SQLException
{
-return values.get(index) != null ?
((ByteBuffer)values.get(index).getValue()).array() : null;
+return values.get(index-1) != null ?
((ByteBuffer)values.get(index-1).getValue()).array() : null;
}
/**
@@ -553,7 +553,7 @@ class CassandraResultSet implements Resu
*/
public int getInt(int index) throws SQLException
{
-return values.get(index) != null ?
((BigInteger)values.get(index).getValue()).intValue() : null;
+return values.get(index-1) != null ?
((BigInteger)values.get(index-1).getValue()).intValue() : null;
}
/**
@@ -574,7 +574,7 @@ class CassandraResultSet implements Resu
*/
public long getLong(int index) throws SQLException
{
-return values.get(index) != null ? (Long)values.get(index).getValue()
: null;
+return values.get(index-1) != null ?
(Long)values.get(index-1).getValue() : null;
}
/**
@@ -664,7 +664,7 @@ class CassandraResultSet implements Resu
*/
public Object getObject(int index) throws SQLException
{
-return values.get(index) == null ? null : values.get(index).getValue();
+return values.get(index-1) == null ? null :
values.get(index-1).getValue();
}
/**
@@ -805,7 +805,7 @@ class CassandraResultSet implements Resu
*/
public String getString(int index) throws SQLException
{
-return values.get(index) != null ?
ColumnDecoder.colValueAsString(values.get(index).getValue()) : null;
+return values.get(index-1) != null ?
ColumnDecoder.colValueAsString(values.get(index-1).getValue()) : null;
}
/**
Modified:
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java?rev=1086967r1=1086966r2=1086967view=diff
==
---
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
(original)
+++
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/JdbcDriverTest.java
Wed Mar 30 14:30:55 2011
@@ -228,8 +228,8 @@ public class JdbcDriverTest extends Embe
FBUtilities.bytesToHex(a),
FBUtilities.bytesToHex(b)));
assert rs.next();
-assert Arrays.equals(aa, rs.getBytes(0));
-assert Arrays.equals(bb, rs.getBytes(1));
+assert Arrays.equals(aa, rs.getBytes(1));
+assert Arrays.equals(bb, rs.getBytes(2));
assert Arrays.equals(aa, rs.getBytes(FBUtilities.bytesToHex(a)));
assert Arrays.equals(bb, rs.getBytes(FBUtilities.bytesToHex(b)));
ResultSetMetaData md = rs.getMetaData();
@@ -423,16 +423,16 @@ public class JdbcDriverTest extends Embe
{
// getString and getObject should always work.
assert rs.getString(cols[c]) != null;
-assert rs.getString(c) != null;
+assert rs.getString(c+1) != null;
assert rs.getObject(cols[c]) != null;
-assert rs.getObject(c) != null;
+assert rs.getObject(c+1) != null;
// now call the accessor.
try
{
Method byInt = rs.getClass().getDeclaredMethod(get +
accessor, int.class);
byInt.setAccessible(true);
-assert byInt.invoke(rs, c) != null;
+assert byInt.invoke(rs, c+1) != null;
Method byString = rs.getClass().getDeclaredMethod(get +
accessor, String.class);
byString.setAccessible(true);
Modified:
cassandra/trunk/drivers/java/test/org/apache/cassandra/cql/jdbc/PreparedStatementTest.java
URL:
svn commit: r1086968 - /cassandra/trunk/CHANGES.txt
Author: gdusbabek Date: Wed Mar 30 14:30:58 2011 New Revision: 1086968 URL: http://svn.apache.org/viewvc?rev=1086968view=rev Log: CHANGES.txt Modified: cassandra/trunk/CHANGES.txt Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1086968r1=1086967r2=1086968view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Wed Mar 30 14:30:58 2011 @@ -10,7 +10,7 @@ * backwards compatible internal messaging (CASSANDRA-1015) * check for null encryption in MessagingService (CASSANDRA-2152) * Fix for Cli to support updating replicate_on_write (CASSANDRA-2236) - * JDBC driver for CQL (CASSANDRA-2124, 2302) + * JDBC driver for CQL (CASSANDRA-2124, 2302, 2277) * atomic switch of memtables and sstables (CASSANDRA-2284) * add pluggable SeedProvider (CASSANDRA-1669)
[Cassandra Wiki] Update of FAQ_JP by MakiWatanabe
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The FAQ_JP page has been changed by MakiWatanabe. The comment on this change is: Translate cleaning_compacted_tables, jna. http://wiki.apache.org/cassandra/FAQ_JP?action=diffrev1=77rev2=78 -- * [[#unsubscribe|メーリングリストの配信を停止するにはどうしたらいいですか？]] * [[#cleaning_compacted_tables|Compactionを実行してもディスク使用量が減らないのはなぜでしょうか？]] * [[#mmap|topコマンドの出力で，CassandraがJava heapの最大値よりも大きなメモリを使用しているのはなぜでしょうか？]] - * [[#jna|I'm getting java.io.IOException: Cannot run program ln when trying to snapshot or update a keyspace]] + * [[#jna|Keyspaceを更新したり、スナップショットをとろうとしたときに java.io.IOException: Cannot run program ln が発生します]] Anchor(cant_listen_on_ip_any) == CassandraはListenAddress設定で何故0.0.0.0(全アドレス)でリッスンできないのですか? == @@ -449, +449 @@ Anchor(cleaning_compacted_tables) == Compactionを実行してもディスク使用量が減らないのはなぜでしょうか？ == - SSTables that are obsoleted by a compaction are deleted asynchronously when the JVM performs a GC. You can force a GC from jconsole if necessary, but Cassandra will force one itself if it detects that it is low on space. A compaction marker is also added to obsolete sstables so they can be deleted on startup if the server does not perform a GC before being restarted. Read more on this subject [[http://wiki.apache.org/cassandra/MemtableSSTable|here]]. - + compactionによって不要になったSSTableはJVMがGCを実行する際に非同期的に削除されます。必要ならjconsoleで強制的にGCさせることもできますが、Cassandraは空きスペースが少なくなれば自動的にGCを実行します。 + 不要になったsstableにはcompactionマーカーが付与されます。GCが実行される前にサーバーがリスタートした場合、起動時にこれらのsstableを削除します。 + 詳細については[[MemtableSSTable_JP|MemtableSSTable]]を参照してください。 Anchor(mmap) @@ -463, +464 @@ Anchor(jna) - == I'm getting java.io.IOException: Cannot run program ln when trying to snapshot or update a keyspace == - Updating a keyspace first takes a snapshot. This involves creating hardlinks to the existing SSTables, but Java has no native way to create hard links, so it must fork ln. When forking, there must be as much memory free as the parent process, even though the child isn't going to use it all. Because Java is a large process, this is problematic. The solution is to install [[http://jna.java.net/|Java Native Access]] so it can create the hard links itself. + == Keyspaceを更新したり、スナップショットをとろうとしたときに java.io.IOException: Cannot run program ln が発生します。 == + Keyspaceを更新する場合、最初にスナップショットが取られます。これには既存のSSTあｂれへのはーどりんくを生成することも含まれますが、Javaにはハードリンクをネイティブで生成する機能がないため、lnをforkする必要があります。forkを実行する際には、例え子プロセスがそのすべてを使用しないとしても親プロセスと同じ大きさのフリーメモリが必要です。Javaは巨大なプロセスですので、これは難しい問題です。解決策は[[http://jna.java.net/|Java Native Access]]をインストールし、ハードリンクの生成をネイティブで実行することです。
[jira] [Updated] (CASSANDRA-1034) Remove assumption that Key to Token is one-to-one
[
https://issues.apache.org/jira/browse/CASSANDRA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-1034:
Attachment: Remove-assumption-that-key-and-token-are-one-to-one-v2.patch
I realize that this is a little more subtle than I first though.
You just cannot compare a Token and a DecoratedKey simply, because a Token is
actually a range of keys. Hence dealing with a Range that mixes Token and
DecoratedKey correctly is doable, but a bit complicated (typically, it involves
declaring multiple different comparison functions). To take quick example,
consider that if you mix DK and Token, you must have the following that stands:
{noformat}
(T(2), T(8)] should not contain DK(T(2), foo) = DK(T(2), foo) T(2)
[T(2), T(8)] should contain DK(T(2), foo) = DK(T(2), foo) = T(2)
{noformat}
So there is no way to write a compareTo() function dealing with both DK and
token.
So I think that it will be simplest to not mix DK and Token in the same ranges.
We'll have ranges of Token (for everything related to ring management) and
ranges of DK (for rangeSlice and scan). This is what the patch does (and the
patch 'generify' AbstractBounds, Range and Bounds (a fair part of the patch) to
keep type information around and avoid unnecessary casts all over the place).
We still want to make a rangeSlice/scan over a range of token. To do that, we
simply convert a range of Token to a range of DK. This involves declaring for a
given token a smallest key and biggest key, and this in turn comes a slight
complication related to the minimum token, but the detail are in the docstrings
of the patch. I am reasonably confident on that new patch.
(Note that this patch is much bigger than the previous one, but this is mostly
due to the generification of Range)
Remove assumption that Key to Token is one-to-one
-
Key: CASSANDRA-1034
URL: https://issues.apache.org/jira/browse/CASSANDRA-1034
Project: Cassandra
Issue Type: Bug
Reporter: Stu Hood
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 0.8
Attachments:
0001-Make-range-accept-both-Token-and-DecoratedKey.patch,
0002-LengthPartitioner.patch, 1034_v1.txt,
Remove-assumption-that-key-and-token-are-one-to-one-v2.patch
get_range_slices assumes that Tokens do not collide and converts a KeyRange
to an AbstractBounds. For RandomPartitioner, this assumption isn't safe, and
would lead to a very weird heisenberg.
Converting AbstractBounds to use a DecoratedKey would solve this, because the
byte[] key portion of the DecoratedKey can act as a tiebreaker.
Alternatively, we could make DecoratedKey extend Token, and then use
DecoratedKeys in places where collisions are unacceptable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1087008 - /cassandra/branches/cassandra-0.7/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
Author: brandonwilliams
Date: Wed Mar 30 16:18:39 2011
New Revision: 1087008
URL: http://svn.apache.org/viewvc?rev=1087008view=rev
Log:
stress.java rejects arbitrary arguments.
Patch by Pavel Yaskevich, reviewed by brandonwilliams for CASSANDRA-2323
Modified:
cassandra/branches/cassandra-0.7/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
Modified:
cassandra/branches/cassandra-0.7/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java?rev=1087008r1=1087007r2=1087008view=diff
==
---
cassandra/branches/cassandra-0.7/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
(original)
+++
cassandra/branches/cassandra-0.7/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
Wed Mar 30 16:18:39 2011
@@ -115,6 +115,12 @@ public class Session
{
CommandLine cmd = parser.parse(availableOptions, arguments);
+if (cmd.getArgs().length 0)
+{
+System.err.println(Application does not allow arbitrary
arguments: + StringUtils.join(cmd.getArgList(), , ));
+System.exit(1);
+}
+
if (cmd.hasOption(h))
throw new IllegalArgumentException(help);
[Cassandra Wiki] Update of Cassandra EC2 by MatthewCooke
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The Cassandra EC2 page has been changed by MatthewCooke. http://wiki.apache.org/cassandra/Cassandra%20EC2?action=diffrev1=2rev2=3 -- With these simple bash scripts you can easily start up a small Cassandra cluster on EC2 for testing. [[http://github.com/johanoskarsson/cassandra-ec2|Cassandra EC2]] See the readme for instructions on how to use it. + + NB: I found these scripts a useful template but they are a bit dated now and required a variety of fixes, in particular: + * SVN URL has changed from http://svn.apache.org/repos/asf/incubator/cassandra/trunk/ to https://svn.apache.org/repos/asf/cassandra/trunk (no longer in incubation) + * storage-conf.xml is now cassandra.yaml + * The debian (lenny) AMI would not apply the updates required due to an issue with the signing key (I don't know the way to fix this, as I'm not a debian person but it's probably trivial to fix) Worth noting that this is not intended for production use.
[Cassandra Wiki] Update of Cassandra EC2 by MatthewCooke
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The Cassandra EC2 page has been changed by MatthewCooke. http://wiki.apache.org/cassandra/Cassandra%20EC2?action=diffrev1=3rev2=4 -- + NB: I found these scripts a useful template but they are a bit dated now and required a variety of fixes, in particular: + * SVN URL has changed from http://svn.apache.org/repos/asf/incubator/cassandra/trunk/ to https://svn.apache.org/repos/asf/cassandra/trunk (no longer in incubation) + * storage-conf.xml is now cassandra.yaml + * The debian (lenny) AMI would not apply the updates required due to an issue with the signing key (I don't know the way to fix this, as I'm not a debian person but it's probably trivial to fix) + With these simple bash scripts you can easily start up a small Cassandra cluster on EC2 for testing. [[http://github.com/johanoskarsson/cassandra-ec2|Cassandra EC2]] See the readme for instructions on how to use it. - - NB: I found these scripts a useful template but they are a bit dated now and required a variety of fixes, in particular: - * SVN URL has changed from http://svn.apache.org/repos/asf/incubator/cassandra/trunk/ to https://svn.apache.org/repos/asf/cassandra/trunk (no longer in incubation) - * storage-conf.xml is now cassandra.yaml - * The debian (lenny) AMI would not apply the updates required due to an issue with the signing key (I don't know the way to fix this, as I'm not a debian person but it's probably trivial to fix) Worth noting that this is not intended for production use.
[Cassandra Wiki] Update of Cassandra EC2 by MatthewCooke
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The Cassandra EC2 page has been changed by MatthewCooke. http://wiki.apache.org/cassandra/Cassandra%20EC2?action=diffrev1=4rev2=5 -- + '' - NB: I found these scripts a useful template but they are a bit dated now and required a variety of fixes, in particular: + Note: I found these scripts a useful template but they are a bit dated now and required a variety of fixes, in particular: * SVN URL has changed from http://svn.apache.org/repos/asf/incubator/cassandra/trunk/ to https://svn.apache.org/repos/asf/cassandra/trunk (no longer in incubation) * storage-conf.xml is now cassandra.yaml * The debian (lenny) AMI would not apply the updates required due to an issue with the signing key (I don't know the way to fix this, as I'm not a debian person but it's probably trivial to fix) + '' With these simple bash scripts you can easily start up a small Cassandra cluster on EC2 for testing. [[http://github.com/johanoskarsson/cassandra-ec2|Cassandra EC2]] See the readme for instructions on how to use it.
[Cassandra Wiki] Update of Cassandra EC2 by MatthewCooke
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The Cassandra EC2 page has been changed by MatthewCooke. http://wiki.apache.org/cassandra/Cassandra%20EC2?action=diffrev1=5rev2=6 -- + With these simple bash scripts you can easily start up a small Cassandra cluster on EC2 for testing. [[http://github.com/johanoskarsson/cassandra-ec2|Cassandra EC2]] See the readme for instructions on how to use it. + '' Note: I found these scripts a useful template but they are a bit dated now and required a variety of fixes, in particular: * SVN URL has changed from http://svn.apache.org/repos/asf/incubator/cassandra/trunk/ to https://svn.apache.org/repos/asf/cassandra/trunk (no longer in incubation) @@ -7, +9 @@ * The debian (lenny) AMI would not apply the updates required due to an issue with the signing key (I don't know the way to fix this, as I'm not a debian person but it's probably trivial to fix) '' - With these simple bash scripts you can easily start up a small Cassandra cluster on EC2 for testing. [[http://github.com/johanoskarsson/cassandra-ec2|Cassandra EC2]] See the readme for instructions on how to use it. Worth noting that this is not intended for production use.
[jira] [Updated] (CASSANDRA-2398) Type specific compression
[
https://issues.apache.org/jira/browse/CASSANDRA-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stu Hood updated CASSANDRA-2398:
Attachment: (was: example.diff)
Type specific compression
-
Key: CASSANDRA-2398
URL: https://issues.apache.org/jira/browse/CASSANDRA-2398
Project: Cassandra
Issue Type: New Feature
Components: Core
Affects Versions: 0.8
Reporter: Stu Hood
Labels: compression
Cassandra has a lot of locations that are ripe for type specific compression.
A short list:
Indexes
* Keys compressed as BytesType, which could default to LZO/LZMA
* Offsets (delta and varint encoding)
* Column names added by 2319
Data
* Keys, columns, timestamps: see
http://wiki.apache.org/cassandra/FileFormatDesignDoc
A basic interface for type specific compression could be as simple as:
{code:java}
public void compress(int version, final ListByteBuffer from, DataOutput to)
throws IOException
public void decompress(int version, DataInput from, ListByteBuffer to)
throws IOException
public void skip(int version, DataInput from) throws IOException
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2398) Type specific compression
[
https://issues.apache.org/jira/browse/CASSANDRA-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stu Hood updated CASSANDRA-2398:
Description:
Cassandra has a lot of locations that are ripe for type specific compression. A
short list:
Indexes
* Keys compressed as BytesType, which could default to LZO/LZMA
* Offsets (delta and varint encoding)
* Column names added by 2319
Data
* Keys, columns, timestamps: see
http://wiki.apache.org/cassandra/FileFormatDesignDoc
A basic interface for type specific compression could be as simple as:
{code:java}
public void compress(int version, final ListByteBuffer from, DataOutput to)
throws IOException
public void decompress(int version, DataInput from, ListByteBuffer to) throws
IOException
public void skip(int version, DataInput from) throws IOException
{code}
was:
Cassandra has a lot of locations that are ripe for type specific compression. A
short list:
Indexes
* Keys compressed as BytesType, which could default to LZO/LZMA
* Offsets (delta and varint encoding)
* Column names added by 2319
Data
* Keys, columns, timestamps: see
http://wiki.apache.org/cassandra/FileFormatDesignDoc
A basic interface for type specific compression could be as simple as:
{code:java}
public void compress(int version, IteratorByteBuffer from, int count,
DataOutput to) throws IOException
public void decompress(int version, DataInput from, ListByteBuffer to) throws
IOException
public void skip(int version, DataInput from) throws IOException
{code}
Type specific compression
-
Key: CASSANDRA-2398
URL: https://issues.apache.org/jira/browse/CASSANDRA-2398
Project: Cassandra
Issue Type: New Feature
Components: Core
Affects Versions: 0.8
Reporter: Stu Hood
Labels: compression
Cassandra has a lot of locations that are ripe for type specific compression.
A short list:
Indexes
* Keys compressed as BytesType, which could default to LZO/LZMA
* Offsets (delta and varint encoding)
* Column names added by 2319
Data
* Keys, columns, timestamps: see
http://wiki.apache.org/cassandra/FileFormatDesignDoc
A basic interface for type specific compression could be as simple as:
{code:java}
public void compress(int version, final ListByteBuffer from, DataOutput to)
throws IOException
public void decompress(int version, DataInput from, ListByteBuffer to)
throws IOException
public void skip(int version, DataInput from) throws IOException
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2398) Type specific compression
[
https://issues.apache.org/jira/browse/CASSANDRA-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stu Hood updated CASSANDRA-2398:
Attachment: 0001-Add-basic-type-specific-compression-to-AbstractType-ne.txt
Implemented delta and varint encoding for LongType, and used that primitive to
compress the lengths for the AbstractType implementation. I'd love to find a
pure java LZ* implementation that we're happy with and call this finished.
Type specific compression
-
Key: CASSANDRA-2398
URL: https://issues.apache.org/jira/browse/CASSANDRA-2398
Project: Cassandra
Issue Type: New Feature
Components: Core
Affects Versions: 0.8
Reporter: Stu Hood
Labels: compression
Attachments:
0001-Add-basic-type-specific-compression-to-AbstractType-ne.txt
Cassandra has a lot of locations that are ripe for type specific compression.
A short list:
Indexes
* Keys compressed as BytesType, which could default to LZO/LZMA
* Offsets (delta and varint encoding)
* Column names added by 2319
Data
* Keys, columns, timestamps: see
http://wiki.apache.org/cassandra/FileFormatDesignDoc
A basic interface for type specific compression could be as simple as:
{code:java}
public void compress(int version, final ListByteBuffer from, DataOutput to)
throws IOException
public void decompress(int version, DataInput from, ListByteBuffer to)
throws IOException
public void skip(int version, DataInput from) throws IOException
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Reporter: Peter Schuller Priority: Minor The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1278) Make bulk loading into Cassandra less crappy, more pluggable
[ https://issues.apache.org/jira/browse/CASSANDRA-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013568#comment-13013568 ] Edward Capriolo commented on CASSANDRA-1278: I definitely agree that having a 100% pure cassandra way to do this is good. I would personally like to see a JMX function like 'nodetool addsstable mykeyspace mycf mysstable-file' . Most people can generating and move an SSTable on their own (sstableWriter +scp) . Those people only need a way to alert cassandra about new SSTables in the data directory. Doing that is a subset of the work needed to find a way to stream it. Make bulk loading into Cassandra less crappy, more pluggable Key: CASSANDRA-1278 URL: https://issues.apache.org/jira/browse/CASSANDRA-1278 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Jeremy Hanna Assignee: Matthew F. Dennis Fix For: 0.7.5 Attachments: 1278-cassandra-0.7.1.txt, 1278-cassandra-0.7.txt Original Estimate: 40h Time Spent: 40h 40m Remaining Estimate: 0h Currently bulk loading into Cassandra is a black art. People are either directed to just do it responsibly with thrift or a higher level client, or they have to explore the contrib/bmt example - http://wiki.apache.org/cassandra/BinaryMemtable That contrib module requires delving into the code to find out how it works and then applying it to the given problem. Using either method, the user also needs to keep in mind that overloading the cluster is possible - which will hopefully be addressed in CASSANDRA-685 This improvement would be to create a contrib module or set of documents dealing with bulk loading. Perhaps it could include code in the Core to make it more pluggable for external clients of different types. It is just that this is something that many that are new to Cassandra need to do - bulk load their data into Cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2347) index scan uses incorrect comparator on non-indexed expressions
[ https://issues.apache.org/jira/browse/CASSANDRA-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2347: -- Reviewer: jbellis Assignee: Roland Gude (was: Jonathan Ellis) Great! index scan uses incorrect comparator on non-indexed expressions --- Key: CASSANDRA-2347 URL: https://issues.apache.org/jira/browse/CASSANDRA-2347 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.0 Reporter: Jonathan Ellis Assignee: Roland Gude Fix For: 0.7.5 Attachments: 2347.txt When multiple index expressions are specified, the column name comparator is used when evaluating secondary (non-indexed) expressions after an indexed expression match. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1034) Remove assumption that Key to Token is one-to-one
[ https://issues.apache.org/jira/browse/CASSANDRA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013583#comment-13013583 ] Jonathan Ellis commented on CASSANDRA-1034: --- Can you break the generification out into a separate patch? Remove assumption that Key to Token is one-to-one - Key: CASSANDRA-1034 URL: https://issues.apache.org/jira/browse/CASSANDRA-1034 Project: Cassandra Issue Type: Bug Reporter: Stu Hood Assignee: Sylvain Lebresne Priority: Minor Fix For: 0.8 Attachments: 0001-Make-range-accept-both-Token-and-DecoratedKey.patch, 0002-LengthPartitioner.patch, 1034_v1.txt, Remove-assumption-that-key-and-token-are-one-to-one-v2.patch get_range_slices assumes that Tokens do not collide and converts a KeyRange to an AbstractBounds. For RandomPartitioner, this assumption isn't safe, and would lead to a very weird heisenberg. Converting AbstractBounds to use a DecoratedKey would solve this, because the byte[] key portion of the DecoratedKey can act as a tiebreaker. Alternatively, we could make DecoratedKey extend Token, and then use DecoratedKeys in places where collisions are unacceptable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1087039 - in /cassandra/trunk: drivers/py/cqlsh test/system/test_cql.py
Author: eevans
Date: Wed Mar 30 19:02:37 2011
New Revision: 1087039
URL: http://svn.apache.org/viewvc?rev=1087039view=rev
Log:
(dumb )cqlsh completion for PRIMARY KEY in CREATE
Patch by eevans
Modified:
cassandra/trunk/drivers/py/cqlsh
cassandra/trunk/test/system/test_cql.py
Modified: cassandra/trunk/drivers/py/cqlsh
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/py/cqlsh?rev=1087039r1=1087038r2=1087039view=diff
==
--- cassandra/trunk/drivers/py/cqlsh (original)
+++ cassandra/trunk/drivers/py/cqlsh Wed Mar 30 19:02:37 2011
@@ -129,6 +129,7 @@ class Shell(cmd.Cmd):
if words[1].upper() == 'COLUMNFAMILY':
types = startswith(CQLTYPES, text)
+keywords = startswith(('KEY', 'PRIMARY'), text.upper())
props = startswith((comparator,
comment,
row_cache_size,
@@ -144,7 +145,7 @@ class Shell(cmd.Cmd):
memtable_throughput_in_mb,
memtable_operations_in_millions,
replicate_on_write), text)
-return startswith(common, text.upper()) + types + props
+return startswith(common, text.upper()) + types + keywords + props
if words[1].upper() == 'KEYSPACE':
props = (replication_factor, strategy_options,
strategy_class)
Modified: cassandra/trunk/test/system/test_cql.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/test/system/test_cql.py?rev=1087039r1=1087038r2=1087039view=diff
==
--- cassandra/trunk/test/system/test_cql.py (original)
+++ cassandra/trunk/test/system/test_cql.py Wed Mar 30 19:02:37 2011
@@ -81,13 +81,13 @@ def load_sample(dbconn):
dbconn.execute(
BEGIN BATCH USING CONSISTENCY ONE
- UPDATE StandardLongA SET 1='1', 2='2', 3='3', 4='4' WHERE KEY='aa';
- UPDATE StandardLongA SET 5='5', 6='6', 7='8', 9='9' WHERE KEY='ab';
- UPDATE StandardLongA SET 9='9', 8='8', 7='7', 6='6' WHERE KEY='ac';
- UPDATE StandardLongA SET 5='5', 4='4', 3='3', 2='2' WHERE KEY='ad';
- UPDATE StandardLongA SET 1='1', 2='2', 3='3', 4='4' WHERE KEY='ae';
- UPDATE StandardLongA SET 1='1', 2='2', 3='3', 4='4' WHERE KEY='af';
- UPDATE StandardLongA SET 5='5', 6='6', 7='8', 9='9' WHERE KEY='ag';
+ UPDATE StandardLongA SET 1='1', 2='2', 3='3', 4='4' WHERE KEY='aa'
+ UPDATE StandardLongA SET 5='5', 6='6', 7='8', 9='9' WHERE KEY='ab'
+ UPDATE StandardLongA SET 9='9', 8='8', 7='7', 6='6' WHERE KEY='ac'
+ UPDATE StandardLongA SET 5='5', 4='4', 3='3', 2='2' WHERE KEY='ad'
+ UPDATE StandardLongA SET 1='1', 2='2', 3='3', 4='4' WHERE KEY='ae'
+ UPDATE StandardLongA SET 1='1', 2='2', 3='3', 4='4' WHERE KEY='af'
+ UPDATE StandardLongA SET 5='5', 6='6', 7='8', 9='9' WHERE KEY='ag'
APPLY BATCH
)
svn commit: r1087040 - in /cassandra/trunk/drivers/py/cql: connection.py decoders.py results.py
Author: eevans
Date: Wed Mar 30 19:02:41 2011
New Revision: 1087040
URL: http://svn.apache.org/viewvc?rev=1087040view=rev
Log:
decode keys according to type too
Patch by eevans
Modified:
cassandra/trunk/drivers/py/cql/connection.py
cassandra/trunk/drivers/py/cql/decoders.py
cassandra/trunk/drivers/py/cql/results.py
Modified: cassandra/trunk/drivers/py/cql/connection.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/py/cql/connection.py?rev=1087040r1=1087039r2=1087040view=diff
==
--- cassandra/trunk/drivers/py/cql/connection.py (original)
+++ cassandra/trunk/drivers/py/cql/connection.py Wed Mar 30 19:02:41 2011
@@ -97,6 +97,8 @@ class Connection(object):
cfresults[cf.name] = {comparator: cf.comparator_type}
cfresults[cf.name][default_validation_class] = \
cf.default_validation_class
+cfresults[cf.name][key_validation_class] = \
+ cf.key_validation_class
cfresults[cf.name][columns] = columns(cf.column_metadata)
return cfresults
Modified: cassandra/trunk/drivers/py/cql/decoders.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/py/cql/decoders.py?rev=1087040r1=1087039r2=1087040view=diff
==
--- cassandra/trunk/drivers/py/cql/decoders.py (original)
+++ cassandra/trunk/drivers/py/cql/decoders.py Wed Mar 30 19:02:41 2011
@@ -21,10 +21,16 @@ from marshal import unmarshal
class BaseDecoder(object):
def decode_column(self, keyspace, column_family, name, value):
raise NotImplementedError()
+
+def decode_key(self, keyspace, column_family, key):
+raise NotImplementedError()
class NoopDecoder(BaseDecoder):
def decode_column(self, keyspace, column_family, name, value):
return (name, value)
+
+def decode_key(self, keyspace, column_family, key):
+return key
class SchemaDecoder(BaseDecoder):
@@ -53,9 +59,18 @@ class SchemaDecoder(BaseDecoder):
else:
return cfam[default_validation_class]
return None
+
+def __keytype_for(self, keyspace, column_family, key):
+cfam = self.__get_column_family_def(keyspace, column_family)
+if cfam and cfam.has_key(key_validation_class):
+return cfam[key_validation_class]
+return None
def decode_column(self, keyspace, column_family, name, value):
comparator = self.__comparator_for(keyspace, column_family)
validator = self.__validator_for(keyspace, column_family, name)
return (unmarshal(name, comparator), unmarshal(value, validator))
+
+def decode_key(self, keyspace, column_family, key):
+return unmarshal(key, self.__keytype_for(keyspace, column_family, key))
Modified: cassandra/trunk/drivers/py/cql/results.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/py/cql/results.py?rev=1087040r1=1087039r2=1087040view=diff
==
--- cassandra/trunk/drivers/py/cql/results.py (original)
+++ cassandra/trunk/drivers/py/cql/results.py Wed Mar 30 19:02:41 2011
@@ -38,9 +38,16 @@ class RowsProxy(object):
class Row(object):
def __init__(self, key, columns, keyspace, cfam, decoder):
-self.key = key
+self._key = key
+self.keyspace = keyspace
+self.cfam = cfam
+self.decoder = decoder
self.columns = ColumnsProxy(columns, keyspace, cfam, decoder)
-
+
+def __get_key(self):
+return self.decoder.decode_key(self.keyspace, self.cfam, self._key)
+key = property(__get_key)
+
def __iter__(self):
return iter(self.columns)
@@ -88,4 +95,4 @@ class Column(object):
return Column(%s, %s) % (self.name, self.value)
def __repr__(self):
-return str(self)
\ No newline at end of file
+return str(self)
svn commit: r1087042 - in /cassandra/trunk/drivers: py/cql/cassandra/constants.py py/cql/cassandra/ttypes.py txpy/txcql/cassandra/constants.py txpy/txcql/cassandra/ttypes.py
Author: eevans
Date: Wed Mar 30 19:02:48 2011
New Revision: 1087042
URL: http://svn.apache.org/viewvc?rev=1087042view=rev
Log:
update generated code in python drivers
Patch by eevans
Modified:
cassandra/trunk/drivers/py/cql/cassandra/constants.py
cassandra/trunk/drivers/py/cql/cassandra/ttypes.py
cassandra/trunk/drivers/txpy/txcql/cassandra/constants.py
cassandra/trunk/drivers/txpy/txcql/cassandra/ttypes.py
Modified: cassandra/trunk/drivers/py/cql/cassandra/constants.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/py/cql/cassandra/constants.py?rev=1087042r1=1087041r2=1087042view=diff
==
--- cassandra/trunk/drivers/py/cql/cassandra/constants.py (original)
+++ cassandra/trunk/drivers/py/cql/cassandra/constants.py Wed Mar 30 19:02:48
2011
@@ -7,4 +7,4 @@
from thrift.Thrift import *
from ttypes import *
-VERSION = 19.4.0
+VERSION = 19.5.0
Modified: cassandra/trunk/drivers/py/cql/cassandra/ttypes.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/py/cql/cassandra/ttypes.py?rev=1087042r1=1087041r2=1087042view=diff
==
--- cassandra/trunk/drivers/py/cql/cassandra/ttypes.py (original)
+++ cassandra/trunk/drivers/py/cql/cassandra/ttypes.py Wed Mar 30 19:02:48 2011
@@ -2510,6 +2510,7 @@ class CfDef:
- memtable_operations_in_millions
- replicate_on_write
- merge_shards_chance
+ - key_validation_class
thrift_spec = (
@@ -2539,9 +2540,10 @@ class CfDef:
(23, TType.DOUBLE, 'memtable_operations_in_millions', None, None, ), # 23
(24, TType.BOOL, 'replicate_on_write', None, False, ), # 24
(25, TType.DOUBLE, 'merge_shards_chance', None, None, ), # 25
+(26, TType.STRING, 'key_validation_class', None, None, ), # 26
)
- def __init__(self, keyspace=None, name=None, column_type=thrift_spec[3][4],
comparator_type=thrift_spec[5][4], subcomparator_type=None, comment=None,
row_cache_size=thrift_spec[9][4], key_cache_size=thrift_spec[11][4],
read_repair_chance=thrift_spec[12][4], column_metadata=None,
gc_grace_seconds=None, default_validation_class=None, id=None,
min_compaction_threshold=None, max_compaction_threshold=None,
row_cache_save_period_in_seconds=None, key_cache_save_period_in_seconds=None,
memtable_flush_after_mins=None, memtable_throughput_in_mb=None,
memtable_operations_in_millions=None, replicate_on_write=thrift_spec[24][4],
merge_shards_chance=None,):
+ def __init__(self, keyspace=None, name=None, column_type=thrift_spec[3][4],
comparator_type=thrift_spec[5][4], subcomparator_type=None, comment=None,
row_cache_size=thrift_spec[9][4], key_cache_size=thrift_spec[11][4],
read_repair_chance=thrift_spec[12][4], column_metadata=None,
gc_grace_seconds=None, default_validation_class=None, id=None,
min_compaction_threshold=None, max_compaction_threshold=None,
row_cache_save_period_in_seconds=None, key_cache_save_period_in_seconds=None,
memtable_flush_after_mins=None, memtable_throughput_in_mb=None,
memtable_operations_in_millions=None, replicate_on_write=thrift_spec[24][4],
merge_shards_chance=None, key_validation_class=None,):
self.keyspace = keyspace
self.name = name
self.column_type = column_type
@@ -2564,6 +2566,7 @@ class CfDef:
self.memtable_operations_in_millions = memtable_operations_in_millions
self.replicate_on_write = replicate_on_write
self.merge_shards_chance = merge_shards_chance
+self.key_validation_class = key_validation_class
def read(self, iprot):
if iprot.__class__ == TBinaryProtocol.TBinaryProtocolAccelerated and
isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is
not None and fastbinary is not None:
@@ -2690,6 +2693,11 @@ class CfDef:
self.merge_shards_chance = iprot.readDouble();
else:
iprot.skip(ftype)
+ elif fid == 26:
+if ftype == TType.STRING:
+ self.key_validation_class = iprot.readString();
+else:
+ iprot.skip(ftype)
else:
iprot.skip(ftype)
iprot.readFieldEnd()
@@ -2791,6 +2799,10 @@ class CfDef:
oprot.writeFieldBegin('merge_shards_chance', TType.DOUBLE, 25)
oprot.writeDouble(self.merge_shards_chance)
oprot.writeFieldEnd()
+if self.key_validation_class != None:
+ oprot.writeFieldBegin('key_validation_class', TType.STRING, 26)
+ oprot.writeString(self.key_validation_class)
+ oprot.writeFieldEnd()
oprot.writeFieldStop()
oprot.writeStructEnd()
def validate(self):
Modified: cassandra/trunk/drivers/txpy/txcql/cassandra/constants.py
URL:
http://svn.apache.org/viewvc/cassandra/trunk/drivers/txpy/txcql/cassandra/constants.py?rev=1087042r1=1087041r2=1087042view=diff
==
--- cassandra/trunk/drivers/txpy/txcql/cassandra/constants.py (original)
+++
svn commit: r1087043 - /cassandra/trunk/test/system/test_cql.py
Author: eevans Date: Wed Mar 30 19:02:52 2011 New Revision: 1087043 URL: http://svn.apache.org/viewvc?rev=1087043view=rev Log: stubbed functional test-case for typed keys Patch by eevans Modified: cassandra/trunk/test/system/test_cql.py Modified: cassandra/trunk/test/system/test_cql.py URL: http://svn.apache.org/viewvc/cassandra/trunk/test/system/test_cql.py?rev=1087043r1=1087042r2=1087043view=diff == --- cassandra/trunk/test/system/test_cql.py (original) +++ cassandra/trunk/test/system/test_cql.py Wed Mar 30 19:02:52 2011 @@ -591,3 +591,13 @@ class TestCql(ThriftTester): , test_escaped_quotes) assert (len(r) == 1) and (len(r[0]) == 1), wrong number of results assert r[0][0].name == x\'and\'y + +def test_typed_keys(self): +using typed keys +conn = init() +r = conn.execute(SELECT * FROM StandardString1 WHERE KEY = ?, ka) +assert isinstance(r[0].key, unicode), \ +wrong key-type returned, expected unicode, got %s % type(r[0].key) + +# FIXME: The above is woefully inadequate, but the test config uses +# CollatingOrderPreservingPartitioner which only supports UTF8.
[jira] [Assigned] (CASSANDRA-2353) JMX call StorageService.Operations.getNaturalEndpoints returns an NPE
[ https://issues.apache.org/jira/browse/CASSANDRA-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-2353: - Assignee: Patricio Echague JMX call StorageService.Operations.getNaturalEndpoints returns an NPE - Key: CASSANDRA-2353 URL: https://issues.apache.org/jira/browse/CASSANDRA-2353 Project: Cassandra Issue Type: Bug Components: API Affects Versions: 0.7.0 Reporter: Karl Mueller Assignee: Patricio Echague Priority: Minor Fix For: 0.7.5 The JMX operation StorageService.Operations.getNaturalEndpoints in cassandra.db always returns an NPE. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1087046 - /cassandra/trunk/test/conf/log4j-server.properties
Author: eevans Date: Wed Mar 30 19:06:03 2011 New Revision: 1087046 URL: http://svn.apache.org/viewvc?rev=1087046view=rev Log: set level to DEBUG for tests (inadvertently changed to TRACE) Patch by eevans Modified: cassandra/trunk/test/conf/log4j-server.properties Modified: cassandra/trunk/test/conf/log4j-server.properties URL: http://svn.apache.org/viewvc/cassandra/trunk/test/conf/log4j-server.properties?rev=1087046r1=1087045r2=1087046view=diff == --- cassandra/trunk/test/conf/log4j-server.properties (original) +++ cassandra/trunk/test/conf/log4j-server.properties Wed Mar 30 19:06:03 2011 @@ -15,7 +15,7 @@ # limitations under the License. -log4j.rootLogger=TRACE,R +log4j.rootLogger=DEBUG,R # rolling log file (system.log log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
buildbot failure in ASF Buildbot on cassandra-trunk
The Buildbot has detected a new failure on builder cassandra-trunk while building ASF Buildbot. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/1203 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: isis_ubuntu Build Reason: scheduler Build Source Stamp: [branch cassandra/trunk] 1087043 Blamelist: eevans BUILD FAILED: failed compile sincerely, -The Buildbot
buildbot success in ASF Buildbot on cassandra-trunk
The Buildbot has detected a restored build on builder cassandra-trunk while building ASF Buildbot. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/1204 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: isis_ubuntu Build Reason: scheduler Build Source Stamp: [branch cassandra/trunk] 1087046 Blamelist: eevans Build succeeded! sincerely, -The Buildbot
svn commit: r1087060 - in /cassandra/trunk/doc/cql: CQL.html CQL.textile
Author: eevans Date: Wed Mar 30 19:56:53 2011 New Revision: 1087060 URL: http://svn.apache.org/viewvc?rev=1087060view=rev Log: update docs for typed keys Patch by eevans Modified: cassandra/trunk/doc/cql/CQL.html cassandra/trunk/doc/cql/CQL.textile Modified: cassandra/trunk/doc/cql/CQL.html URL: http://svn.apache.org/viewvc/cassandra/trunk/doc/cql/CQL.html?rev=1087060r1=1087059r2=1087060view=diff == --- cassandra/trunk/doc/cql/CQL.html (original) +++ cassandra/trunk/doc/cql/CQL.html Wed Mar 30 19:56:53 2011 @@ -1,4 +1,4 @@ -?xml version='1.0' encoding='utf-8' ?!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;html xmlns=http://www.w3.org/1999/xhtml;headmeta http-equiv=Content-Type content=text/html; charset=utf-8//headbodyh1 id=CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) v1.0.0/h1h2 id=TableofContentsTable of Contents/h2ol style=list-style: none;lia href=#CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) v1.0.0/aol style=list-style: none;lia href=#TableofContentsTable of Contents/a/lilia href=#USEUSE/a/lilia href=#SELECTSELECT/aol style=list-style: none;lia href=#SpecifyingColumnsSpecifying Columns/a/lilia href=#ColumnFamilyColumn Family/a/lilia href=#ConsistencyLevelConsistency Level/a/lilia href=#FilteringrowsFiltering rows/a/lilia href=#LimitsLimits/a/li/ol/lili a href=#UPDATEUPDATE/aol style=list-style: none;lia href=#ColumnFamily2Column Family/a/lilia href=#ConsistencyLevel2Consistency Level/a/lilia href=#SpecifyingColumnsandRowSpecifying Columns and Row/a/li/ol/lilia href=#DELETEDELETE/aol style=list-style: none;lia href=#SpecifyingColumns2Specifying Columns/a/lilia href=#ColumnFamily3Column Family/a/lilia href=#ConsistencyLevel3Consistency Level/a/lilia href=#deleterowsSpecifying Rows/a/li/ol/lilia href=#TRUNCATETRUNCATE/a/lilia href=#CREATEKEYSPACECREATE KEYSPACE/a/lilia href=#CREATECOLUMNFAMILYCREATE COLUMNFAMILY/aol style=list-style: none;lia href=#columntypesSpecifying Column Type (optional)/a/lilia href=#ColumnFamilyOptionsoptionalColumn Family Options (optional)/a/li/ol/lilia href=#CREATEINDEXCREATE INDEX/a/lilia href=#DROPDROP/a/lilia href=#Comm onIdiomsCommon Idioms/aol style=list-style: none;lia href=#consistencySpecifying Consistency/a/lilia href=#termsTerm specification/a/li/ol/li/ol/lilia href=#VersioningVersioning/a/lilia href=#ChangesChanges/a/li/olh2 id=USEUSE/h2piSynopsis:/i/pprecodeUSE lt;KEYSPACEgt;; +?xml version='1.0' encoding='utf-8' ?!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;html xmlns=http://www.w3.org/1999/xhtml;headmeta http-equiv=Content-Type content=text/html; charset=utf-8//headbodyh1 id=CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) v1.0.0/h1h2 id=TableofContentsTable of Contents/h2ol style=list-style: none;lia href=#CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) v1.0.0/aol style=list-style: none;lia href=#TableofContentsTable of Contents/a/lilia href=#USEUSE/a/lilia href=#SELECTSELECT/aol style=list-style: none;lia href=#SpecifyingColumnsSpecifying Columns/a/lilia href=#ColumnFamilyColumn Family/a/lilia href=#ConsistencyLevelConsistency Level/a/lilia href=#FilteringrowsFiltering rows/a/lilia href=#LimitsLimits/a/li/ol/lili a href=#UPDATEUPDATE/aol style=list-style: none;lia href=#ColumnFamily2Column Family/a/lilia href=#ConsistencyLevel2Consistency Level/a/lilia href=#SpecifyingColumnsandRowSpecifying Columns and Row/a/li/ol/lilia href=#DELETEDELETE/aol style=list-style: none;lia href=#SpecifyingColumns2Specifying Columns/a/lilia href=#ColumnFamily3Column Family/a/lilia href=#ConsistencyLevel3Consistency Level/a/lilia href=#deleterowsSpecifying Rows/a/li/ol/lilia href=#TRUNCATETRUNCATE/a/lilia href=#CREATEKEYSPACECREATE KEYSPACE/a/lilia href=#CREATECOLUMNFAMILYCREATE COLUMNFAMILY/aol style=list-style: none;lia href=#keytypesSpecifying Key Type/a/lilia href=#columntypesSpecifying Column Type (optional)/a/lilia href=#ColumnFamilyOptionsoptionalColumn Family Options (optional)/a/li/ol/lilia href=#CREATEINDEXCREATE INDEX/a/li lia href=#DROPDROP/a/lilia href=#CommonIdiomsCommon Idioms/aol style=list-style: none;lia href=#consistencySpecifying Consistency/a/lilia href=#termsTerm specification/a/li/ol/li/ol/lilia href=#VersioningVersioning/a/lilia href=#ChangesChanges/a/li/olh2 id=USEUSE/h2piSynopsis:/i/pprecodeUSE lt;KEYSPACEgt;; /code/prepA codeUSE/code statement consists of the codeUSE/code keyword, followed by a valid keyspace name. Its purpose is to assign the per-connection, current working keyspace. All subsequent keyspace-specific actions will be performed in the context of the supplied value./ph2 id=SELECTSELECT/h2piSynopsis:/i/pprecodeSELECT [FIRST N] [REVERSED] lt;SELECT EXPRgt; FROM lt;COLUMN
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013632#comment-13013632 ] Peter Schuller commented on CASSANDRA-1902: --- (isPageCaheMigrationEnabled is misspelled.) I tested some on 2.6.32 (it's what I have easily available on running physical h/w). I used stress.py as per the earlier benchmarks in this ticket, but after initial population I just used 'nodetool compact' to trigger compaction. (I tested with standard I/O because I happened to make the changes to the BSF (below), and I also realized I'm not sure where the pre-pop happens for the mmap case. Will have to revisit.) I added some logging and a sleep to BufferedSegmentedFile.tryPreserveFilePageCache() (I'll attach the patch to that file only, it's only for testing purposes). By doing 'sync ; echo 1 /proc/sys/vm/drop_caches' I artificially triggered the newly written file not being in page cache at all (essentially simulating compacting a sufficiently large sstable that 10% (or such) fits in ram). A compaction after letting: python contrib/py_stress/stress.py -S 3000 -n 10 -o read run for a while (~ 5 thousand progress), with the above drop cache / compaction, gave: WARN 22:06:56,723 about to preserve, please drop caches WARN 22:07:06,723 begin preserve WARN 22:07:34,299 end preserve So firstly, this shows WILLNEED being blocking (I guess I should have looked closer, or it's different due to the .32 kernel). During this time I observed iostat -x -k 1. There was an initial flurry of I/O with queue depth 100, but it then settled for a queue depth of around 5-6 or so for the remainder of the period. Here is a fairly typical second (no pretty graphs I'm afraid): Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 6.00 0.00 152.000.00 61716.00 0.00 812.05 5.05 33.13 6.16 93.60 Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: T Jake Luciani Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1902: --- Attachment: (was: CASSANDRA-1902-v6.patch) Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: T Jake Luciani Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1902: --- Attachment: CASSANDRA-1902-v6.patch fixed misspelling. Peter can you write a resume of your previous comment for the mere mortals? Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: T Jake Luciani Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1902: --- Assignee: Pavel Yaskevich (was: T Jake Luciani) Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: Pavel Yaskevich Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013646#comment-13013646 ] Peter Schuller commented on CASSANDRA-1902: --- Scratch the bit about mmap, I made a boo-boo grepping. I forgot to address the seeking though: So advising larger regions might help depending on I/O scheduling behavior in the sense that it may make it easier for the kernel/storage to do sequential reading of adjacent blocks. However, my main concern was the fundamentally required seeking resulting from a sparsely hot sstables. It is not necessarily a problem as such, as long as everyone is aware that the seeks will be happening and that it will potentially add significantly to the time required to complete compaction. The main outright danger would be if the I/O imposed by the WILLNEED is sufficiently detrimental to live reads to cause more problems (albeit during a briefer period perhaps) than the negative effects of compaction that is being addressed. Exactly how it will behave I'm not sure (lack of detailed kernel knowledge), but if the I/O implied by WILLNEED (which in the .32 case seemed to yield the ~ 5 queue depth) is roughly equal to what it would have been had we had ~5 threads doing the concurrent pre-heating in userland, I would expect a persistent queue depth of 5 to have a significant effect on live reads - particularly on single-disk systems and few-disk raid:s. Unless prioritization is going on, a lone live read should on average have to wait for ~ 5 reads to complete before it gets it's turn. For a more concurrent live read workload the impact should be less in terms of aggregate latency since the extra 5 queue depth implies by WILLNEED is less in relative terms. Actually, a complicating factor here which I just now noticed is that the kernel seems to be pulling in a lot more data than intended. The average request size claimed by iostat is roughly ~400 k. The normal stress.py read test on this data set (-S 3000) seems to generate more like ~50k. Depending on whether or not this is retained in page cache without expedited eviction, that's potentially bad. Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: Pavel Yaskevich Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013659#comment-13013659 ] Peter Schuller commented on CASSANDRA-1902: --- Regarding drop caches: Right, I don't remember whether the echo will block until eviction is complete or not (in cases where it is slow, it should be CPU bound though and not imply I/O). But I made sure that: (1) the echo terminated, (2) I got iostat running, (3) waited for flurry of I/O to complete that is generated by background operations on any modern machine, that you suddenly see when you do a *complete* buffer cache drop, and (4) saw it idle, and (5) only then Cassandra began the pre-population. So hopefully that part of the test should be kosher. Regarding mere mortals ;) Sorry. Is it the iostat stuff which is unclear? I'm looking at (in a monospaced font btw, for alignment...) the avgqu-sz column which indicates the average number of outstanding I/O requests for the sampling duration (1 second in this case). This is effectively the queue depth. There are usually two main interesting things about iostat -k -x 1 (-x being key). One is utilization, which shows the percentage of time there was *any* outstanding I/O request to the device. (But one has to interpret it in context; for example a RAID0 of 10 disks can be at 100% utilization yet be only 10% saturated.) The other is the average queue size, which is a more direct indication of how any concurrent requests are being serviced. In the case of the 10 disk RAID0, 100% utilization with an average queue size of 1 would mean roughly 10% saturation of underlying disks. 100% utilization with an average queue size of 5 would mean roughly 50% saturation of underlying disks. The other relevance of the average queue size is on latency. Disregarding potential relative prioritization going on, if the average number of outstanding requests is, say, 10 - any single request will typically have to wait for 10 other requests to be servied first. (But again that has to be interpreted in context; if you have 10 constitutent disks in a RAID0, that 10 is effectively 1 for latency purposes) So, when judging the expected effects on the latency (and throughput) of live reads, it's interesting to look at these values. In particular, consider the simple case of a single seek-bound serial reader. If the average queue depth is 5, this single reader would probably see a throughput roughly 1/5 of normal (Im assuming otherwise identical I/O in terms of size of requests). A practical example is something like a tar -czvf that is reading a lot of small files (fs meta data etc). So in that sense, a constant pressure of 5 outstanding requests will cause a very significant slow-down to the serial reader. On the other hand, if instead of having a serial read you have N number of concurrent readers - you would now rather expect a throughput more like N/(N + 5 - 1) of normal. As the concurrency of the interesting I/O increases, the 5 extra makes less of a difference. You tend to reach an interesting equilibrium here. Suppose you serve some amount of requests per second normally, and that it gives rise to an average queue depth of 0.5. Now add the constant background pressure of 5 requests. Assuming the reads (that normally gave rise to the 0.5 queue depth) are *independent* (I.e., added latency to one does not prevent the next one from coming in), what tends to happen is that you start accumulating outstanding requests until the number of concurrent requests is high enough that you reach the throughput you had before. Only instead of 0.5 average concurrency, you have something higher than 5. Whatever is required to drown out the extra 5 enough. Even if you are able to reach your desired throughput (requests per second) like this, it significantly adds to the average latency of each read. Not only will each read have to contend with the extra 5 background I/O operations always pending, they also have to compete with other concurrent live requests. Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: Pavel Yaskevich Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013665#comment-13013665 ] Peter Schuller commented on CASSANDRA-1902: --- So as an example. Suppose you have live read traffic that results in 0.5 average queue size. Assume for simplicity a single underlying disk that does exactly one request at a time and does no optimization for locality on disk. 0.5 average under those circumstances means you're effectively utilizing 50% of the I/O capacity - the disk is spending 50% of it's wall clock time servicing your reads. Suppose now that you add a background reader that does nothing but generate seek bound small reads, one after another. That, if left alone, would generate an average queue size of 1. Running these at once should roughly even out. They should get about 0.5 each. But suppose you run two background readers. In order for your live reads to get 50% of the time the disk spends doing reads, live reads must again have as many outstanding requests on average as the background readers. The background readers have 2 outstanding, so your live traffic must maintain two outstanding requests as well. This means that at any given moment you need a queue depth of 4, two of which are live traffic. So when a new live request comes in, again absent prioritization/re-ordering, it must now wait for 4 requests to complete before it can be serviced. Thus, request time becomes 5x instead of 1x (x being the time needed to perform a read) because it needs to do the read itself, in addition to 4 other reads. Now, in reality you have a queue depth that is preserved down onto a SAS/SATA bus, and you have disks that will optimize I/O (completely random seek bound I/O can get about twice as fast in terms of throughput (reqs/second) when queue depth starts to reach 32-64 requests - unless this has changed since a few years ago). With RAID you have multiple constituent drives which changes things. SSD:s complicate things too by being capable of actual real physical concurrent I/O. So the above example would never reflect reality in terms of concrete numbers, but the effect is there - the details are just different. Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: Pavel Yaskevich Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Page NodeProbe renamed to NodeTool on Cassandra Wiki
Dear wiki user, You have subscribed to a wiki page Cassandra Wiki for change notification. The page NodeTool has been renamed from NodeProbe by AlexisLeQuoc. The comment on this change is: nodeprobe is now nodetool.. http://wiki.apache.org/cassandra/NodeTool
[Cassandra Wiki] Update of NodeProbe by AlexisLeQuoc
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The NodeProbe page has been changed by AlexisLeQuoc. http://wiki.apache.org/cassandra/NodeProbe -- New page: You actually want to read about NodeTool.
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013680#comment-13013680 ] Pavel Yaskevich commented on CASSANDRA-1902: I got you point, it confirms one more time that making this an option was a good decision to make. Also making WILLNEED as a single call for range instead of the page seems to be a good idea too so kernel can decide what to do depending on situation (it still has a right to not take advice into account). Migrate cached pages during compaction --- Key: CASSANDRA-1902 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.7.1 Reporter: T Jake Luciani Assignee: Pavel Yaskevich Fix For: 0.7.5, 0.8 Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch Original Estimate: 32h Time Spent: 56h Remaining Estimate: 0h Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing. For example an active CF being compacted hurts reads since nothing is cached in the new SSTable. The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable. A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2279) Tombstones not collected post-repair
[
https://issues.apache.org/jira/browse/CASSANDRA-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013702#comment-13013702
]
Joaquin Casares commented on CASSANDRA-2279:
I just read this and was wondering if this may be the case:
http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds
GCGraceSeconds was set to 30 minutes to allow the repair to finish and I waited
35 before running the compaction on the above steps.
Tombstones not collected post-repair
Key: CASSANDRA-2279
URL: https://issues.apache.org/jira/browse/CASSANDRA-2279
Project: Cassandra
Issue Type: Bug
Components: Tools
Affects Versions: 0.6
Reporter: Joaquin Casares
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 0.6.13, 0.7.5
Attachments: RowIteration-unit-tests.patch,
fix-RowIteratorFactory.patch, nodeA.txt, nodeB.txt
The keys would only show up in sstables2json and look like this:
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:34am
=== /opt/cassandra/bin/sstable2json Queue-2527-Data.db -k
waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b
2011-02-23 07:24:43,710 INFO [org.apache.cassandra.config.DatabaseDescriptor]
- DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
2011-02-23 07:24:43,972 INFO [org.apache.cassandra.io.SSTableReader] -
Opening /opt/cassandra/storage/queue/data/Panama/Queue-2527-Data.db
{
waq:publicMessageIndexingWorkArea:PUM8a65ce95-9d35-4941-928c-dd5965e8b29b:
[]
}
(root@aps4):/opt/cassandra/storage/queue/data/Panama Wed Feb 23 07:24:44am
===
The steps that I took to reproduce it were:
Create a keyspace, column family, and a key
Delete the key on Node 1 using the cli (del cf['key'];)
Flush
Repair on a cluster with more than 1 node
Wait GCSeconds
Compact
And the empty row would appear on Node 2
However, when I was able to get rid of the empty rows, I was following these
steps on a single machine:
Create a keyspace, column family, and a key
Delete the key
Flush
Sample write (writing to some temporary key)
Deleting the attribute to that temporary key (not the entire key)
Flush
Compact
or these steps:
Create a keyspace, column family, and a key
Delete the key
Flush
Wait GCseconds
Compact
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2387) Make it possible for pig to understand packed data
[ https://issues.apache.org/jira/browse/CASSANDRA-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-2387: Attachment: 2387-v3_.patch Made a FBUtilities method to getInstance - essentially for AbstractTypes - getting the singleton. Still has some issues, but took care of the construct errors I was getting when it was using FBUtilities.construct in CassandraStorage. Make it possible for pig to understand packed data -- Key: CASSANDRA-2387 URL: https://issues.apache.org/jira/browse/CASSANDRA-2387 Project: Cassandra Issue Type: Improvement Reporter: Jeremy Hanna Assignee: Jeremy Hanna Labels: contrib, hadoop, pig Attachments: 2387-1.txt, 2387-v2.txt, loadstorecaster-patch.txt Packed values are throwing off pig. This ticket is to make it so pig can interpret packed values. Originally we thought we could just use a loadcaster. However, the only way we know how we can do it now is to get the schema through thrift and essentially perform the function of the loadcaster in the getNext method. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2387) Make it possible for pig to understand packed data
[ https://issues.apache.org/jira/browse/CASSANDRA-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-2387: Attachment: 2387-v3.txt Make it possible for pig to understand packed data -- Key: CASSANDRA-2387 URL: https://issues.apache.org/jira/browse/CASSANDRA-2387 Project: Cassandra Issue Type: Improvement Reporter: Jeremy Hanna Assignee: Jeremy Hanna Labels: contrib, hadoop, pig Attachments: 2387-1.txt, 2387-v2.txt, 2387-v3.txt, loadstorecaster-patch.txt Packed values are throwing off pig. This ticket is to make it so pig can interpret packed values. Originally we thought we could just use a loadcaster. However, the only way we know how we can do it now is to get the schema through thrift and essentially perform the function of the loadcaster in the getNext method. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1263) Push replication factor down to the replication strategy
[ https://issues.apache.org/jira/browse/CASSANDRA-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Hermes updated CASSANDRA-1263: -- Attachment: 1263-5.txt Updated schema-sample.txt. Push replication factor down to the replication strategy Key: CASSANDRA-1263 URL: https://issues.apache.org/jira/browse/CASSANDRA-1263 Project: Cassandra Issue Type: Task Components: Core Reporter: Jeremy Hanna Assignee: Jon Hermes Priority: Minor Fix For: 0.8 Attachments: 1263-2.txt, 1263-3.txt, 1263-4.txt, 1263-5.txt, 1263-incomplete.txt, 1263.txt Original Estimate: 8h Remaining Estimate: 8h Currently the replication factor is in the keyspace metadata. As we've added the datacenter shard strategy, the replication factor becomes more computed by the replication strategy. It seems reasonable to therefore push the replication factor for the keyspace down to the replication strategy so that it can be handled in one place. This adds on the work being done in CASSANDRA-1066 since that ticket will make the replication strategy a member variable of keyspace metadata instead of just a quasi singleton giving the replication strategy state for each keyspace. That makes it able to have the replication factor. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of FAQ_JP by MakiWatanabe
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Cassandra Wiki for
change notification.
The FAQ_JP page has been changed by MakiWatanabe.
The comment on this change is: Translate range_rp, a_long_is_exactly_8_bytes.
http://wiki.apache.org/cassandra/FAQ_JP?action=diffrev1=78rev2=79
--
Anchor(a_long_is_exactly_8_bytes)
== 挿入操作でInvalidRequestExeptionが発生し，次のようなメッセージが出力されます:A long is exactly 8
bytes ==
- You are propably using !LongType column sorter in your column family.
!LongType assumes that the numbers stored into column names are exactly 64bit
(8 bytes) long and in big endian format. Example code how to pack and unpack an
integer for storing into cassandra and unpacking it for php:
+
あなたは恐らくカラムファミリーの中で!LongTypeカラムソーターを使っているはずです。!LongTypeはカラムに格納される数値がぴったり64bit(8
byte)であり、またbig
endianであることを想定しています。以下のサンプルコードで、phpからcassandraにintegerをpackして格納する方法、そしてcassandraから読み出してunpackする方法について説明します。
{{{
/**
@@ -435, +435 @@
Anchor(range_rp)
== レンジスライスやシーケンシャルスキャンはなぜ期待通りの結果を返さないのでしょうか？ ==
- You're probably using the RandomPartitioner. This is the default because it
avoids hotspots, but it means your rows are ordered by the md5 of the row key
rather than lexicographically by the raw key bytes.
+
あなたは多分RandomPartitionerを使っているはずです。これはデフォルト設定で、データが特定のノードに偏ることを避けられます。一方で、RandomPartitionerを使った場合、行はあなたが指定するrow
keyの字句的順序ではなく、 row keyのmd5ハッシュ値の順で並びます。
+ ページングが目的なら、start keyとend keyを空に指定し、row
count引数を指定すれば実現できます。次のページを取得するには前のページで取得した最後のキーをstart keyに指定します。
+ 行の中であれば、カラムをカラム名の順序で取得できます。データを各行に配置する方法を工夫することで、行そのものが整列していなくても済む場合もあるでしょう。
- You '''can''' start out with a start key and end key of [empty] and use the
row count argument instead, if your goal is paging the rows. To get the next
page, start from the last key you got in the previous page.
-
- You can also use intra-row ordering of column names to get ordered results
'''within''' a row; with appropriate row 'bucketing,' you often don't need the
rows themselves to be ordered.
Anchor(unsubscribe)
[jira] [Created] (CASSANDRA-2406) Secondary index and index expression problems
Secondary index and index expression problems - Key: CASSANDRA-2406 URL: https://issues.apache.org/jira/browse/CASSANDRA-2406 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Environment: CentOS 5.5 (64bit), JDK 1.6.0_23 Reporter: Muga Nishizawa Fix For: 0.7.5 Attachments: create_table.cli When I iteratively get data with secondary index and index clause, result of data acquired by consistency level one is different from the one by consistency level quorum. The one by consistecy level one is correct result. But the one by consistecy level quorum is incorrect and is dropped by Cassandra. You can reproduce the bug by executing attached programs. - 1. Start Cassandra cluster. It consists of 3 cassandra nodes and distributes data by ByteOrderedPartitioner. Initial tokens of those nodes are [31, 32, 33]. - 2. Create keyspace and column family, according to create_table.cli, - 3. Execute secondary_index_insertv2.py, inserting a few hundred columns to cluster - 4. Execute secondary_index_checkv2.py and get data with secondary index and index clause iteratively. secondary_index_insertv2.py and secondary_index_checkv2.py require pycassa. You will be able to execute 4th secondary_index_checkv2.py script with following option so that you get data with consistency level one. % python secondary_index_checkv2.py -one On the other hand, to acquire data with consistency level quorum, you will need to use following option. % python secondary_index_checkv2.py -quorum You can check that result of data acquired by consistency level one is different from one by consistency level quorum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2406) Secondary index and index expression problems
[ https://issues.apache.org/jira/browse/CASSANDRA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muga Nishizawa updated CASSANDRA-2406: -- Attachment: create_table.cli Secondary index and index expression problems - Key: CASSANDRA-2406 URL: https://issues.apache.org/jira/browse/CASSANDRA-2406 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Environment: CentOS 5.5 (64bit), JDK 1.6.0_23 Reporter: Muga Nishizawa Fix For: 0.7.5 Attachments: create_table.cli When I iteratively get data with secondary index and index clause, result of data acquired by consistency level one is different from the one by consistency level quorum. The one by consistecy level one is correct result. But the one by consistecy level quorum is incorrect and is dropped by Cassandra. You can reproduce the bug by executing attached programs. - 1. Start Cassandra cluster. It consists of 3 cassandra nodes and distributes data by ByteOrderedPartitioner. Initial tokens of those nodes are [31, 32, 33]. - 2. Create keyspace and column family, according to create_table.cli, - 3. Execute secondary_index_insertv2.py, inserting a few hundred columns to cluster - 4. Execute secondary_index_checkv2.py and get data with secondary index and index clause iteratively. secondary_index_insertv2.py and secondary_index_checkv2.py require pycassa. You will be able to execute 4th secondary_index_checkv2.py script with following option so that you get data with consistency level one. % python secondary_index_checkv2.py -one On the other hand, to acquire data with consistency level quorum, you will need to use following option. % python secondary_index_checkv2.py -quorum You can check that result of data acquired by consistency level one is different from one by consistency level quorum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2406) Secondary index and index expression problems
[ https://issues.apache.org/jira/browse/CASSANDRA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muga Nishizawa updated CASSANDRA-2406: -- Attachment: secondary_index_insertv2.py Secondary index and index expression problems - Key: CASSANDRA-2406 URL: https://issues.apache.org/jira/browse/CASSANDRA-2406 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Environment: CentOS 5.5 (64bit), JDK 1.6.0_23 Reporter: Muga Nishizawa Fix For: 0.7.5 Attachments: create_table.cli, secondary_index_checkv2.py, secondary_index_insertv2.py When I iteratively get data with secondary index and index clause, result of data acquired by consistency level one is different from the one by consistency level quorum. The one by consistecy level one is correct result. But the one by consistecy level quorum is incorrect and is dropped by Cassandra. You can reproduce the bug by executing attached programs. - 1. Start Cassandra cluster. It consists of 3 cassandra nodes and distributes data by ByteOrderedPartitioner. Initial tokens of those nodes are [31, 32, 33]. - 2. Create keyspace and column family, according to create_table.cli, - 3. Execute secondary_index_insertv2.py, inserting a few hundred columns to cluster - 4. Execute secondary_index_checkv2.py and get data with secondary index and index clause iteratively. secondary_index_insertv2.py and secondary_index_checkv2.py require pycassa. You will be able to execute 4th secondary_index_checkv2.py script with following option so that you get data with consistency level one. % python secondary_index_checkv2.py -one On the other hand, to acquire data with consistency level quorum, you will need to use following option. % python secondary_index_checkv2.py -quorum You can check that result of data acquired by consistency level one is different from one by consistency level quorum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2406) Secondary index and index expression problems
[ https://issues.apache.org/jira/browse/CASSANDRA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muga Nishizawa updated CASSANDRA-2406: -- Attachment: secondary_index_checkv2.py Secondary index and index expression problems - Key: CASSANDRA-2406 URL: https://issues.apache.org/jira/browse/CASSANDRA-2406 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Environment: CentOS 5.5 (64bit), JDK 1.6.0_23 Reporter: Muga Nishizawa Fix For: 0.7.5 Attachments: create_table.cli, secondary_index_checkv2.py, secondary_index_insertv2.py When I iteratively get data with secondary index and index clause, result of data acquired by consistency level one is different from the one by consistency level quorum. The one by consistecy level one is correct result. But the one by consistecy level quorum is incorrect and is dropped by Cassandra. You can reproduce the bug by executing attached programs. - 1. Start Cassandra cluster. It consists of 3 cassandra nodes and distributes data by ByteOrderedPartitioner. Initial tokens of those nodes are [31, 32, 33]. - 2. Create keyspace and column family, according to create_table.cli, - 3. Execute secondary_index_insertv2.py, inserting a few hundred columns to cluster - 4. Execute secondary_index_checkv2.py and get data with secondary index and index clause iteratively. secondary_index_insertv2.py and secondary_index_checkv2.py require pycassa. You will be able to execute 4th secondary_index_checkv2.py script with following option so that you get data with consistency level one. % python secondary_index_checkv2.py -one On the other hand, to acquire data with consistency level quorum, you will need to use following option. % python secondary_index_checkv2.py -quorum You can check that result of data acquired by consistency level one is different from one by consistency level quorum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1969) Use BB for row cache - To Improve GC performance.
[ https://issues.apache.org/jira/browse/CASSANDRA-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013819#comment-13013819 ] Chris Goffinet commented on CASSANDRA-1969: --- Could we get a rebase on 1969-rollup-and-config on trunk? Use BB for row cache - To Improve GC performance. - Key: CASSANDRA-1969 URL: https://issues.apache.org/jira/browse/CASSANDRA-1969 Project: Cassandra Issue Type: Improvement Components: Core Environment: Linux and Mac Reporter: Vijay Assignee: Vijay Priority: Minor Attachments: 0001-Config-1969.txt, 0001-introduce-ICache-InstrumentingCache-IRowCacheProvider.txt, 0002-Update_existing-1965.txt, 0002-implement-SerializingCache.txt, 0002-implement-SerializingCacheV2.txt, 0003-New_Cache_Providers-1969.txt, 0003-add-ICache.isCopying-method.txt, 0004-Null-Check-and-duplicate-bb.txt, 0004-TestCase-1969.txt, 1969-rollup-and-config.txt, BB_Cache-1945.png, JMX-Cache-1945.png, Old_Cahce-1945.png, POC-0001-Config-1945.txt, POC-0002-Update_existing-1945.txt, POC-0003-New_Cache_Providers-1945.txt Java BB.allocateDirect() will allocate native memory out of the JVM and will help reducing the GC pressure in the JVM with a large Cache. From some of the basic tests it shows around 50% improvement than doing a normal Object cache. In addition this patch provide the users an option to choose BB.allocateDirect or store everything in the heap. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2407) Compaction thread should try to empty a bucket before moving on
Compaction thread should try to empty a bucket before moving on
---
Key: CASSANDRA-2407
URL: https://issues.apache.org/jira/browse/CASSANDRA-2407
Project: Cassandra
Issue Type: Improvement
Reporter: Stu Hood
Priority: Minor
As suggested by Aaron Morton
[(1)|https://issues.apache.org/jira/browse/CASSANDRA-2191?focusedCommentId=13010077page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13010077],
a compaction thread should attempt to empty a bucket before moving on to a
larger bucket. This would change the submitMinorIfNeeded {{for}} loop into a
while loop that regenerated the buckets and started from the bottom after each
successful compaction.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2191) Multithread across compaction buckets
[
https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013837#comment-13013837
]
Stu Hood commented on CASSANDRA-2191:
-
1. Added {{if (max min || max 0) return null;}} check
2. Switched to LinkedHashSet to preserve order
3. Fixed, thanks
4. The idea is that the throttling from CASSANDRA-2156 is a sufficient
preventative measure for this, but it will be important to enable it by
default, hopefully using the metrics from CASSANDRA-2171. I'd prefer not to
have two tunables, but it's worth discussing.
5. CompactionManagerMBean exposes getPendingTasks and getCompletedTasks with
entirely different meanings, so I'm not sure we'd gain anything by this
6. (see next comment)
7. Yes, it probably would, but I think that is an issue for a separate ticket.
Usually compacting the smallest bucket first (since the files are likely to be
hot in cache) is the biggest win (which we do): it will be very rare for higher
buckets to be more important
8. This would probably be a good idea, for example, if you have more than
{{max}} sstables in the minimum bucket and not enough active threads to
parallelize the bucket. Opened CASSANDRA-2407
9. I can't think of an easy way to do this, but if you can, I'm willing
Multithread across compaction buckets
-
Key: CASSANDRA-2191
URL: https://issues.apache.org/jira/browse/CASSANDRA-2191
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Stu Hood
Priority: Critical
Labels: compaction
Fix For: 0.8
Attachments: 0001-Add-a-compacting-set-to-sstabletracker.txt,
0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt,
0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt
This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and
reasoning are different enough to open a separate issue.
The problem with compactions currently is that they compact the set of
sstables that existed the moment the compaction started. This means that for
longer running compactions (even when running as fast as possible on the
hardware), a very large number of new sstables might be created in the
meantime. We have observed this proliferation of sstables killing performance
during major/high-bucketed compactions.
One approach would be to pause compactions in upper buckets (containing
larger files) when compactions in lower buckets become possible. While this
would likely solve the problem with read performance, it does not actually
help us perform compaction any faster, which is a reasonable requirement for
other situations.
Instead, we need to be able to perform any compactions that are currently
required in parallel, independent of what bucket they might be in.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2191) Multithread across compaction buckets
[ https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013838#comment-13013838 ] Stu Hood commented on CASSANDRA-2191: - Regarding 6.: there is an issue related to behaviour here: thank you for pointing it out. As implemented in this patch, triggering a major compaction will start it immediately with whatever sstables aren't already active in compaction: this means that triggering a major compaction while another compaction is running will not result in a major compaction. In the past, this wouldn't have been kosher, because major compactions were required to clean up tombstones. But now that we have the bloomfilter checking optimization for compaction, the new behaviour is probably sufficient: when there ''aren't'' any other compactions active, the major flag is an optimization that makes the bloomfilter checks unnecessary; when there ''are'' other compactions active, the bloomfilter check works as usual. Thoughts? Multithread across compaction buckets - Key: CASSANDRA-2191 URL: https://issues.apache.org/jira/browse/CASSANDRA-2191 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Priority: Critical Labels: compaction Fix For: 0.8 Attachments: 0001-Add-a-compacting-set-to-sstabletracker.txt, 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt, 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and reasoning are different enough to open a separate issue. The problem with compactions currently is that they compact the set of sstables that existed the moment the compaction started. This means that for longer running compactions (even when running as fast as possible on the hardware), a very large number of new sstables might be created in the meantime. We have observed this proliferation of sstables killing performance during major/high-bucketed compactions. One approach would be to pause compactions in upper buckets (containing larger files) when compactions in lower buckets become possible. While this would likely solve the problem with read performance, it does not actually help us perform compaction any faster, which is a reasonable requirement for other situations. Instead, we need to be able to perform any compactions that are currently required in parallel, independent of what bucket they might be in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
