date:20110804


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079250#comment-13079250
 ] 

Sylvain Lebresne commented on CASSANDRA-2991:
-

Wouldn't sstableloader (and the JMX bulkload call) essentially solve this ?

 Add a 'load new sstables' JMX/nodetool command
 --

 Key: CASSANDRA-2991
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2991
 Project: Cassandra
  Issue Type: New Feature
Reporter: Brandon Williams
Priority: Minor
 Fix For: 0.8.4


 Sometimes people have to create a new cluster to get around a problem and 
 need to copy sstables around.  It would be convenient to be able to trigger 
 this from nodetool or JMX instead of doing a restart of the node.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2991) Add a 'load new sstables' JMX/nodetool command

2011-08-04 Thread Brandon Williams (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079251#comment-13079251
 ] 

Brandon Williams commented on CASSANDRA-2991:
-

Almost.  The catch is you can't use sstableloader from an existing cluster 
member, so now you need an intermediate machine to load for you, plus it has to 
rebuild secondaries.

 Add a 'load new sstables' JMX/nodetool command
 --

 Key: CASSANDRA-2991
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2991
 Project: Cassandra
  Issue Type: New Feature
Reporter: Brandon Williams
Priority: Minor
 Fix For: 0.8.4


 Sometimes people have to create a new cluster to get around a problem and 
 need to copy sstables around.  It would be convenient to be able to trigger 
 this from nodetool or JMX instead of doing a restart of the node.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2964) IndexRangeSliceQuery results include index column even though it is not in SliceRange

2011-08-04 Thread Roland Gude (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079275#comment-13079275
 ] 

Roland Gude commented on CASSANDRA-2964:


great i will apply the patch and give it a try.

 IndexRangeSliceQuery results include index column even though it is not in 
 SliceRange
 -

 Key: CASSANDRA-2964
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2964
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.7
Reporter: Roland Gude
Assignee: Jonathan Ellis
 Fix For: 0.7.9

 Attachments: 2964-0.7.txt, TestIndexRangeSliceQuery.java, 
 cassandra.yaml


 When an IndexSlicwQuery is done the result contains the index column even 
 though it was not in the slice range.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079293#comment-13079293
 ] 

Pavel Yaskevich commented on CASSANDRA-1717:


After thinking about this for a while I think we should do checksum at the 
column level only which will give us better control over individual columns and 
does not seem to be a big overhead (instead of doing it at the column index 
level and relaying on digest). Checksum on the compressed block level is 
unnecessary because bitrot, for example, will be detected right on 
decompression or column deserialization time. Thoughts?

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2991) Add a 'load new sstables' JMX/nodetool command


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079303#comment-13079303
 ] 

Sylvain Lebresne commented on CASSANDRA-2991:
-

bq. you can't use sstableloader from an existing cluster member

Well, you can using the JMX bulkload call

bq. plus it has to rebuild secondaries

True. Though I do admit that I think sstableloader will be simpler in most case 
of moving sstables between clusters, especially if the destination cluster 
doesn't have the same number of node/replication factor (as you don't have to 
think which sstable should be copied where and avoid both cleanup and repair), 
and thus wonder if a 'load new sstables' command would be so useful. Another 
way to put it is that I think a 'load new sstables' command will be more hacky, 
in that you will have to run it on every nodes, know which sstable goes where 
and not forget the cleanup and repair in most case, so I would be in favor of 
not adding such hacky command unless it really makes a difference.

 Add a 'load new sstables' JMX/nodetool command
 --

 Key: CASSANDRA-2991
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2991
 Project: Cassandra
  Issue Type: New Feature
Reporter: Brandon Williams
Priority: Minor
 Fix For: 0.8.4


 Sometimes people have to create a new cluster to get around a problem and 
 need to copy sstables around.  It would be convenient to be able to trigger 
 this from nodetool or JMX instead of doing a restart of the node.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data

[
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079314#comment-13079314
]

Sylvain Lebresne commented on CASSANDRA-1717:
-

bq. checksum at the column level only which will give us better control over
individual columns and does not seem to be a big overhead

I agree that it is by far the simplest approach for non compressed data, but I,
for one, am a bit concerned by the overhead: 4 bytes per column is not
negligible. On some load, that could easily mean a 10-20% data size increase.
Basically I am concerned about people upgrading to 1.0 and want to make sure
that upgrading brings no surprise for them (and this even if they don't trust
compression yet, which would be perfectly reasonable). For that to be true, I
think that if we go with checksum at the column level we would need to make
that optional and off by default.

bq. Checksum on the compressed block level is unnecessary because bitrot, for
example, will be detected right on decompression

Not sure that's bulletproof. I don't think all compression algorithm ships with
a checksum (I don't know about snappy typically). When they don't, it's totally
possible for bitrot to corrupt compressed data without being a problem at
decompression nor at deserialization if you're unlucky (granted it is more
unlikely to go undetected that without compression but it is not good enough).
So either we check that snappy use checksumming and we only add support for
algorithm that does, or it is still useful.

Cassandra cannot detect corrupt-but-readable column data

Key: CASSANDRA-1717
URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
Fix For: 1.0

Attachments: checksums.txt

Most corruptions of on-disk data due to bitrot render the column (or row)
unreadable, so the data can be replaced by read repair or anti-entropy. But
if the corruption keeps column data readable we do not detect it, and if it
corrupts to a higher timestamp value can even resist being overwritten by
newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2500) Ruby dbi client (for CQL) that conforms to AR:ConnectionAdapter

2011-08-04 Thread Kelley Reynolds (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079333#comment-13079333
 ] 

Kelley Reynolds commented on CASSANDRA-2500:


Upon further investigation, I think that writing a straight DBI driver as the 
premier ruby CQL implementation is a bad idea. The largest problem comes from a 
mismatch in the assumptions the DBI code makes that does not hold true with one 
of Cassandra's most powerful features, column slicing. DBI assumes that the 
column metadata does not change with each row but with Cassandra it does with 
some frequency. Altering DBI to not make those assumptions is non-trivial and 
it's unlikely they'd be accepted back into the tree anyway. I think at this 
point the best course of action is to write something that behaves just like 
DBI but doesn't use that framework at all to get the full capability of CQL and 
then for a DBI driver, use that thing but disallow slicing or any other 
operations that break fundamental DBI assumptions.

 Ruby dbi client (for CQL) that conforms to AR:ConnectionAdapter
 ---

 Key: CASSANDRA-2500
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2500
 Project: Cassandra
  Issue Type: Task
  Components: API
Reporter: Jon Hermes
Assignee: Pavel Yaskevich
  Labels: cql
 Fix For: 0.8.4

 Attachments: 2500.txt, genthriftrb.txt, rbcql-0.0.0.tgz


 Create a ruby driver for CQL.
 Lacking something standard (such as py-dbapi), going with something common 
 instead -- RoR ActiveRecord Connection Adapter 
 (http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/AbstractAdapter.html).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079382#comment-13079382
 ] 

Pavel Yaskevich commented on CASSANDRA-1717:


Making a checksum optional and off by default sounds good to me.

bq. Not sure that's bulletproof...

That is why I mentioned that if we have checksum per column it will work as a 
protection from wrong decompression on the block level and spares us additional 
read and check, isn't it?

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2992) Cassandra don't start on Red Hat Linux due to hardcoded JAVA_HOME

2011-08-04 Thread Taras Puchko (JIRA)

Cassandra don't start on Red Hat Linux due to hardcoded JAVA_HOME
-

 Key: CASSANDRA-2992
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2992
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.3
 Environment: CentOS release 5.6
Reporter: Taras Puchko


On CentOS /etc/init.d/cassandra has
bq. export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/

While there is no such a directory on our server it was ok for 0.8.2, because 
/usr/sbin/cassandra checked the executable
{quote}
if [ -x $JAVA_HOME/bin/java ]; then
JAVA=$JAVA_HOME/bin/java
else
JAVA=`which java`
fi
{quote}

But 0.8.3 builds replaced the above code with one that doesn't check if 
JAVA_HOME is set correctly.
{quote}
if [ -n $JAVA_HOME ]; then
JAVA=$JAVA_HOME/bin/java
else
JAVA=java
fi
{quote}

That's why cassandra doesn't start anymore.


The correct fix would be to remove export JAVA_HOME from 
/etc/init.d/cassandra or set it only to correct path and only if it hasn't 
already been set.

It would also be nice to revert to [ -x $JAVA_HOME/bin/java ] in 
/usr/sbin/cassandra


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2992) Cassandra doesn't start on Red Hat Linux due to hardcoded JAVA_HOME

2011-08-04 Thread Taras Puchko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taras Puchko updated CASSANDRA-2992:


Summary: Cassandra doesn't start on Red Hat Linux due to hardcoded 
JAVA_HOME  (was: Cassandra don't start on Red Hat Linux due to hardcoded 
JAVA_HOME)

 Cassandra doesn't start on Red Hat Linux due to hardcoded JAVA_HOME
 ---

 Key: CASSANDRA-2992
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2992
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.3
 Environment: CentOS release 5.6
Reporter: Taras Puchko

 On CentOS /etc/init.d/cassandra has
 bq. export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/
 While there is no such a directory on our server it was ok for 0.8.2, because 
 /usr/sbin/cassandra checked the executable
 {quote}
 if [ -x $JAVA_HOME/bin/java ]; then
 JAVA=$JAVA_HOME/bin/java
 else
 JAVA=`which java`
 fi
 {quote}
 But 0.8.3 builds replaced the above code with one that doesn't check if 
 JAVA_HOME is set correctly.
 {quote}
 if [ -n $JAVA_HOME ]; then
 JAVA=$JAVA_HOME/bin/java
 else
 JAVA=java
 fi
 {quote}
 That's why cassandra doesn't start anymore.
 The correct fix would be to remove export JAVA_HOME from 
 /etc/init.d/cassandra or set it only to correct path and only if it hasn't 
 already been set.
 It would also be nice to revert to [ -x $JAVA_HOME/bin/java ] in 
 /usr/sbin/cassandra

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2500) Ruby dbi client (for CQL) that conforms to AR:ConnectionAdapter


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079392#comment-13079392
 ] 

Jonathan Ellis commented on CASSANDRA-2500:
---

did you have a look at how the python dbapi2 driver does this?

basically cursor.description will change as the columns in the row does.  for a 
normal db this would be fixed for the entire resultset but it's not much of a 
stretch to make it row-dependent.

 Ruby dbi client (for CQL) that conforms to AR:ConnectionAdapter
 ---

 Key: CASSANDRA-2500
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2500
 Project: Cassandra
  Issue Type: Task
  Components: API
Reporter: Jon Hermes
Assignee: Pavel Yaskevich
  Labels: cql
 Fix For: 0.8.4

 Attachments: 2500.txt, genthriftrb.txt, rbcql-0.0.0.tgz


 Create a ruby driver for CQL.
 Lacking something standard (such as py-dbapi), going with something common 
 instead -- RoR ActiveRecord Connection Adapter 
 (http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/AbstractAdapter.html).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2992) Cassandra doesn't start on Red Hat Linux due to hardcoded JAVA_HOME


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079394#comment-13079394
 ] 

Jonathan Ellis commented on CASSANDRA-2992:
---

Is this fixed by CASSANDRA-2785 ?

 Cassandra doesn't start on Red Hat Linux due to hardcoded JAVA_HOME
 ---

 Key: CASSANDRA-2992
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2992
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.3
 Environment: CentOS release 5.6
Reporter: Taras Puchko

 On CentOS /etc/init.d/cassandra has
 bq. export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/
 While there is no such a directory on our server it was ok for 0.8.2, because 
 /usr/sbin/cassandra checked the executable
 {quote}
 if [ -x $JAVA_HOME/bin/java ]; then
 JAVA=$JAVA_HOME/bin/java
 else
 JAVA=`which java`
 fi
 {quote}
 But 0.8.3 builds replaced the above code with one that doesn't check if 
 JAVA_HOME is set correctly.
 {quote}
 if [ -n $JAVA_HOME ]; then
 JAVA=$JAVA_HOME/bin/java
 else
 JAVA=java
 fi
 {quote}
 That's why cassandra doesn't start anymore.
 The correct fix would be to remove export JAVA_HOME from 
 /etc/init.d/cassandra or set it only to correct path and only if it hasn't 
 already been set.
 It would also be nice to revert to [ -x $JAVA_HOME/bin/java ] in 
 /usr/sbin/cassandra

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079401#comment-13079401
 ] 

Jason Rutherglen commented on CASSANDRA-1902:
-

I think we can revive this issue leveraging the [same] work Lucene has 
performed in this area at [1].  Apparently O_DIRECT is the flag that will work. 
 Perhaps we should open a new issue?

1. https://issues.apache.org/jira/browse/LUCENE-2500
2. 
https://builds.apache.org/job/Lucene-trunk/javadoc/all/org/apache/lucene/store/DirectIOLinuxDirectory.html

 Migrate cached pages during compaction 
 ---

 Key: CASSANDRA-1902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.7.1
Reporter: T Jake Luciani
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: 
 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 
 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 
 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, 
 CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, 
 CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, 
 CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, 
 CASSANDRA-1902-v9-trunk-rebased.patch, 
 CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, 
 CASSANDRA-1902-v9.patch

   Original Estimate: 32h
  Time Spent: 56h
  Remaining Estimate: 0h

 Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a 
 pre-compacted CF during the compaction process.  This is now important since 
 CASSANDRA-1470 caches effectively nothing.  
 For example an active CF being compacted hurts reads since nothing is cached 
 in the new SSTable. 
 The purpose of this ticket then is to make sure SOME data is cached from 
 active CFs. This can be done my monitoring which Old SSTables are in the page 
 cache and caching active rows in the New SStable.
 A simpler yet similar approach is described here: 
 http://insights.oetiker.ch/linux/fadvise/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction

2011-08-04 Thread T Jake Luciani (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079403#comment-13079403
 ] 

T Jake Luciani commented on CASSANDRA-1902:
---

Jason,

We already bypass the page cache (see CASSANDRA-1470)

This ticket is to migrate the hot dataset after compaction of the data

 Migrate cached pages during compaction 
 ---

 Key: CASSANDRA-1902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.7.1
Reporter: T Jake Luciani
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: 
 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 
 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 
 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, 
 CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, 
 CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, 
 CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, 
 CASSANDRA-1902-v9-trunk-rebased.patch, 
 CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, 
 CASSANDRA-1902-v9.patch

   Original Estimate: 32h
  Time Spent: 56h
  Remaining Estimate: 0h

 Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a 
 pre-compacted CF during the compaction process.  This is now important since 
 CASSANDRA-1470 caches effectively nothing.  
 For example an active CF being compacted hurts reads since nothing is cached 
 in the new SSTable. 
 The purpose of this ticket then is to make sure SOME data is cached from 
 active CFs. This can be done my monitoring which Old SSTables are in the page 
 cache and caching active rows in the New SStable.
 A simpler yet similar approach is described here: 
 http://insights.oetiker.ch/linux/fadvise/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2992) Cassandra doesn't start on Red Hat Linux due to hardcoded JAVA_HOME

2011-08-04 Thread Taras Puchko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079406#comment-13079406
 ] 

Taras Puchko commented on CASSANDRA-2992:
-

No, CASSANDRA-2785 actually caused this bug, since now invalid JAVA_HOME is not 
ignored:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/bin/cassandra?r1=1126728r2=1152241pathrev=1152241

But the real bug is setting JAVA_HOME to a fixed value in redhat specific 
scripts.

 Cassandra doesn't start on Red Hat Linux due to hardcoded JAVA_HOME
 ---

 Key: CASSANDRA-2992
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2992
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.3
 Environment: CentOS release 5.6
Reporter: Taras Puchko

 On CentOS /etc/init.d/cassandra has
 bq. export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/
 While there is no such a directory on our server it was ok for 0.8.2, because 
 /usr/sbin/cassandra checked the executable
 {quote}
 if [ -x $JAVA_HOME/bin/java ]; then
 JAVA=$JAVA_HOME/bin/java
 else
 JAVA=`which java`
 fi
 {quote}
 But 0.8.3 builds replaced the above code with one that doesn't check if 
 JAVA_HOME is set correctly.
 {quote}
 if [ -n $JAVA_HOME ]; then
 JAVA=$JAVA_HOME/bin/java
 else
 JAVA=java
 fi
 {quote}
 That's why cassandra doesn't start anymore.
 The correct fix would be to remove export JAVA_HOME from 
 /etc/init.d/cassandra or set it only to correct path and only if it hasn't 
 already been set.
 It would also be nice to revert to [ -x $JAVA_HOME/bin/java ] in 
 /usr/sbin/cassandra

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079424#comment-13079424
 ] 

Sylvain Lebresne commented on CASSANDRA-1717:
-

My bad, I read that as let's not use checksum for compression at all. 
Nevermind.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-1777) The describe_host API method is misleading in that it returns the interface associated with gossip traffic

[
https://issues.apache.org/jira/browse/CASSANDRA-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-1777:

Fix Version/s: (was: 1.0)
0.8.3

The describe_host API method is misleading in that it returns the interface
associated with gossip traffic
--

Key: CASSANDRA-1777
URL: https://issues.apache.org/jira/browse/CASSANDRA-1777
Project: Cassandra
Issue Type: Bug
Reporter: Nate McCall
Assignee: Brandon Williams
Fix For: 0.8.3

Attachments: 1777-v2.txt, 1777.txt

Original Estimate: 16h
Remaining Estimate: 16h

If the hardware is configured to use separate interfaces for thrift and
gossip, the gossip interface will be returned, given the results come out of
the ReplicationStrategy eventually.
I understand the approach, but given this is on the API, it effective
worthless in situations of host auto discovery via describe_ring from a
client. I actually see this as the primary use case of this method - why else
would I care about the gossip iface from the client perspective? It's current
form should be relegated to JMX only.
At the same time, we should add port information as well.
describe_splits probably has similar issues.
I see the potential cart-before-horse issues here and that this will probably
be non-trivial to fix, but I think give me a set of all the hosts to which I
can talk is pretty important from a client perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1153914 - /cassandra/branches/cassandra-0.8.3/

2011-08-04 Thread slebresne

Author: slebresne
Date: Thu Aug  4 16:06:44 2011
New Revision: 1153914

URL: http://svn.apache.org/viewvc?rev=1153914view=rev
Log:
Create branch to revert 2785 and retry 0.8.3

Added:
cassandra/branches/cassandra-0.8.3/   (props changed)
  - copied from r1153193, cassandra/branches/cassandra-0.8/

Propchange: cassandra/branches/cassandra-0.8.3/
--
--- svn:ignore (added)
+++ svn:ignore Thu Aug  4 16:06:44 2011
@@ -0,0 +1,8 @@
+.classpath
+.project
+.settings
+temp-testng-customsuite.xml
+build
+build.properties
+.idea
+out

Propchange: cassandra/branches/cassandra-0.8.3/
--
--- svn:mergeinfo (added)
+++ svn:mergeinfo Thu Aug  4 16:06:44 2011
@@ -0,0 +1,12 @@
+/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
+/cassandra/branches/cassandra-0.7:1026516-1151306
+/cassandra/branches/cassandra-0.7.0:1053690-1055654
+/cassandra/branches/cassandra-0.8:1090934-1125013,1125041
+/cassandra/branches/cassandra-0.8.0:1125021-1130369
+/cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689
+/cassandra/tags/cassandra-0.8.0-rc1:1102511-1125020
+/cassandra/trunk:1129049-1129050,1129065,1151625,1152416,1153189
+/incubator/cassandra/branches/cassandra-0.3:774578-796573
+/incubator/cassandra/branches/cassandra-0.4:810145-834239,834349-834350
+/incubator/cassandra/branches/cassandra-0.5:72-915439
+/incubator/cassandra/branches/cassandra-0.6:911237-922688

svn commit: r1153917 - in /cassandra/branches/cassandra-0.8.3: bin/cassandra conf/cassandra-env.sh debian/cassandra.postinst debian/init

2011-08-04 Thread slebresne

Author: slebresne
Date: Thu Aug  4 16:12:00 2011
New Revision: 1153917

URL: http://svn.apache.org/viewvc?rev=1153917view=rev
Log:
Reverting 2785

Modified:
cassandra/branches/cassandra-0.8.3/bin/cassandra
cassandra/branches/cassandra-0.8.3/conf/cassandra-env.sh
cassandra/branches/cassandra-0.8.3/debian/cassandra.postinst
cassandra/branches/cassandra-0.8.3/debian/init

Modified: cassandra/branches/cassandra-0.8.3/bin/cassandra
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8.3/bin/cassandra?rev=1153917r1=1153916r2=1153917view=diff
==
--- cassandra/branches/cassandra-0.8.3/bin/cassandra (original)
+++ cassandra/branches/cassandra-0.8.3/bin/cassandra Thu Aug  4 16:12:00 2011
@@ -83,10 +83,10 @@ elif [ -r $CASSANDRA_INCLUDE ]; then
 fi
 
 # Use JAVA_HOME if set, otherwise look for java in PATH
-if [ -n $JAVA_HOME ]; then
+if [ -x $JAVA_HOME/bin/java ]; then
 JAVA=$JAVA_HOME/bin/java
 else
-JAVA=java
+JAVA=`which java`
 fi
 
 if [ -z $CASSANDRA_CONF -o -z $CLASSPATH ]; then

Modified: cassandra/branches/cassandra-0.8.3/conf/cassandra-env.sh
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8.3/conf/cassandra-env.sh?rev=1153917r1=1153916r2=1153917view=diff
==
--- cassandra/branches/cassandra-0.8.3/conf/cassandra-env.sh (original)
+++ cassandra/branches/cassandra-0.8.3/conf/cassandra-env.sh Thu Aug  4 
16:12:00 2011
@@ -97,7 +97,7 @@ JMX_PORT=7199
 JVM_OPTS=$JVM_OPTS -ea
 
 # add the jamm javaagent
-check_openjdk=`${JAVA:-java} -version 21 | awk '{if (NR == 2) {print $1}}'`
+check_openjdk=$(java -version 21 | awk '{if (NR == 2) {print $1}}')
 if [ $check_openjdk != OpenJDK ]
 then
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.2.jar

Modified: cassandra/branches/cassandra-0.8.3/debian/cassandra.postinst
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8.3/debian/cassandra.postinst?rev=1153917r1=1153916r2=1153917view=diff
==
--- cassandra/branches/cassandra-0.8.3/debian/cassandra.postinst (original)
+++ cassandra/branches/cassandra-0.8.3/debian/cassandra.postinst Thu Aug  4 
16:12:00 2011
@@ -34,7 +34,7 @@ case $1 in
 cassandra
 fi
 
-if [ -n $2 ]  dpkg --compare-versions $2 le 0.6.4-2; then
+if [ -n $2 ]  dpkg --compare-versions $2 le 0.6.4-2; then
 chown -R cassandra: /var/lib/cassandra
 chown -R cassandra: /var/log/cassandra
 fi

Modified: cassandra/branches/cassandra-0.8.3/debian/init
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8.3/debian/init?rev=1153917r1=1153916r2=1153917view=diff
==
--- cassandra/branches/cassandra-0.8.3/debian/init (original)
+++ cassandra/branches/cassandra-0.8.3/debian/init Thu Aug  4 16:12:00 2011
@@ -30,15 +30,23 @@ JVM_SEARCH_DIRS=/usr/lib/jvm/java-6-ope
 [ -e /etc/cassandra/cassandra.yaml ] || exit 0
 [ -e /etc/cassandra/cassandra-env.sh ] || exit 0
 
+# Read Cassandra environment file.
+. /etc/cassandra/cassandra-env.sh
+
 # Read configuration variable file if it is present
 [ -r /etc/default/$NAME ]  . /etc/default/$NAME
 
+if [ -z $JVM_OPTS ]; then
+echo Initialization failed; \$JVM_OPTS not set! 2
+exit 3
+fi
+
 # If JAVA_HOME has not been set, try to determine it.
 if [ -z $JAVA_HOME ]; then
 # If java is in PATH, use a JAVA_HOME that corresponds to that. This is
 # both consistent with how the upstream startup script works, and how
 # Debian works (read: the use of alternatives to set a system JVM).
-if [ -n `which java` ]; then
+if [ -n `which java` ]; then
 java=`which java`
 # Dereference symlink(s)
 while true; do
@@ -59,15 +67,6 @@ if [ -z $JAVA_HOME ]; then
 done
 fi
 fi
-JAVA=$JAVA_HOME/bin/java
-
-# Read Cassandra environment file.
-. /etc/cassandra/cassandra-env.sh
-
-if [ -z $JVM_OPTS ]; then
-echo Initialization failed; \$JVM_OPTS not set! 2
-exit 3
-fi
 
 # Load the VERBOSE setting and other rcS variables
 . /lib/init/vars.sh

[jira] [Updated] (CASSANDRA-1777) The describe_host API method is misleading in that it returns the interface associated with gossip traffic

[
https://issues.apache.org/jira/browse/CASSANDRA-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-1777:

Fix Version/s: (was: 0.8.3)
0.8.4

The describe_host API method is misleading in that it returns the interface
associated with gossip traffic
--

Key: CASSANDRA-1777
URL: https://issues.apache.org/jira/browse/CASSANDRA-1777
Project: Cassandra
Issue Type: Bug
Reporter: Nate McCall
Assignee: Brandon Williams
Fix For: 0.8.4

Attachments: 1777-v2.txt, 1777.txt

Original Estimate: 16h
Remaining Estimate: 16h

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data

2011-08-04 Thread Ryan King (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079449#comment-13079449
 ] 

Ryan King commented on CASSANDRA-1717:
--

I think checksums per column would be way too much overhead. We already add a 
lot of overhead to all data stored in Cassandra, we should be careful about 
adding more.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (CASSANDRA-2785) should export JAVA variable in the bin/cassandra and use that in the cassandra-env.sh when check for the java version


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne reopened CASSANDRA-2785:
-


This is causing problems, see CASSANDRA-2992.

I have only reverted this patch on a specific branch so that it doesn't block 
0.8.3 release. However, I have not reverted yet on 0.8 branch (or trunk), so 
that it could be fixed incrementally.

As a side note, it would be nice to set the correct fix version and reviewer 
when marking an issue resolved.

 should export JAVA variable in the bin/cassandra and use that in the 
 cassandra-env.sh when check for the java version
 -

 Key: CASSANDRA-2785
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2785
 Project: Cassandra
  Issue Type: Bug
Reporter: Jackson Chung
Assignee: paul cannon
 Attachments: 0001-use-JAVA-in-cassandra-env.sh.patch.txt, 
 0002-fix-usage-of-bash-n-tests.patch.txt


 I forgot which jira we add this java -version check in the cassandra-env.sh 
 (for adding jamm to the javaagent), but we should probably use the variable 
 JAVA set in bin/cassandra (will need export) and use $JAVA instead of java 
 in the cassandra-env.sh
 In a situation where JAVA_HOME may have been properly set as the Sun's java 
 but the PATH still have the OpenJDK's java in front, the check will fail to 
 add the jamm.jar, even though the cassandra jvm is properly started via the 
 Sun's java.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079479#comment-13079479
 ] 

Pavel Yaskevich commented on CASSANDRA-1717:


That is why we want to let users to decide if they need that protection at all.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079489#comment-13079489
]

Jason Rutherglen commented on CASSANDRA-2915:
-

I looked at MessagingService which seems to be more [custom] asynchronous?

I think we could offer a Thrift API? What does CQL use?

I think we'd want to look towards making this [Lucene] play well / integrate
with CQL?

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

Secondary indexes (of type KEYS) suffer from a number of limitations in their
current form:
- Multiple IndexClauses only work when there is a subset of rows under the
highest clause
- One new column family is created per index this means 10 new CFs for 10
secondary indexes
This ticket will use the Lucene library to implement secondary indexes as one
index per CF, and utilize the Lucene query engine to handle multiple index
clauses. Also, by using the Lucene we get a highly optimized file format.
There are a few parallels we can draw between Cassandra and Lucene.
Lucene indexes segments in memory then flushes them to disk so we can sync
our memtable flushes to lucene flushes. Lucene also has optimize() which
correlates to our compaction process, so these can be sync'd as well.
We will also need to correlate column validators to Lucene tokenizers, so the
data can be stored properly, the big win in once this is done we can perform
complex queries within a column like wildcard searches.
The downside of this approach is we will need to read before write since
documents in Lucene are written as complete documents. For random workloads
with lot's of indexed columns this means we need to read the document from
the index, update it and write it back.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-2901:

Attachment: 0003-Fix-LCR.patch

The DefsTest and CliTest problem is because we don't ignore purged tombstone on
the first pass when computing the serializedSize. Attaching a small patch with
the fix. The patch also fixes a failure with StreamingTransferTest: in SSTII,
the columnPosition should be set for non file input, otherwise headerSiez()
returns the wrong value and the assertion in getColumnFamilyWithColumns is
triggered. This seems to fix all unit tests here.

The patch looks good, but each deserializer now get the full maxInMemorySize
instead of maxInMemorySize / nb(Deserializers). Was that intended ?

Allow taking advantage of multiple cores while compacting a single CF
-

Key: CASSANDRA-2901
URL: https://issues.apache.org/jira/browse/CASSANDRA-2901
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 0.8.4

Attachments:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt,
0002-parallel-compaction.txt, 0003-Fix-LCR.patch

Moved from CASSANDRA-1876:
There are five stages: read, deserialize, merge, serialize, and write. We
probably want to continue doing read+deserialize and serialize+write
together, or you waste a lot copying to/from buffers.
So, what I would suggest is: one thread per input sstable doing read +
deserialize (a row at a time). A thread pool (one per core?) merging
corresponding rows from each input sstable. One thread doing serialize +
writing the output (this has to wait for the merge threads to complete
in-order, obviously). This should take us from being CPU bound on SSDs (since
only one core is compacting) to being I/O bound.
This will require roughly 2x the memory, to allow the reader threads to work
ahead of the merge stage. (I.e. for each input sstable you will have up to
one row in a queue waiting to be merged, and the reader thread working on the
next.) Seems quite reasonable on that front. You'll also want a small queue
size for the serialize-merged-rows executor.
Multithreaded compaction should be either on or off. It doesn't make sense to
try to do things halfway (by doing the reads with a
threadpool whose size you can grow/shrink, for instance): we still have
compaction threads tuned to low priority, by default, so the impact on the
rest of the system won't be very different. Nor do we expect to have so many
input sstables that we lose a lot in context switching between reader threads.
IMO it's acceptable to punt completely on rows that are larger than memory,
and fall back to the old non-parallel code there. I don't see any sane way to
parallelize large-row compactions.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079497#comment-13079497
]

Sylvain Lebresne commented on CASSANDRA-2901:
-

Nevermind, LazilyCompactedRowTest seems broken with the patch above.

Allow taking advantage of multiple cores while compacting a single CF
-

Attachments:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt,
0002-parallel-compaction.txt, 0003-Fix-LCR.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of FrontPage by Nick Neuberger

2011-08-04 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The FrontPage page has been changed by Nick Neuberger:
http://wiki.apache.org/cassandra/FrontPage?action=diffrev1=63rev2=64

Comment:
adding section for administration applications.

   * [[http://www.datastax.com/docs|Datastax's Cassandra documentation]]
   * [[ClientOptions|Client options: ways to access Cassandra]] -- interfaces 
for Ruby, Python, Scala and more
   * [[IntegrationPoints]] -- list of ways Cassandra is integrated with other 
projects/products
+  * [[Administration Tools]] -- list of administrative tools to configure / 
admin your Cassandra instance
   * [[RunningCassandra|Running Cassandra]]
   * [[ArchitectureOverview|Architecture Overview]]
   * [[UseCases|Simple Use Cases and Solutions]] -- please help complete

[Cassandra Wiki] Update of Administration Tools by Nick Neuberger

2011-08-04 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The Administration Tools page has been changed by Nick Neuberger:
http://wiki.apache.org/cassandra/Administration%20Tools

New page:
= Cassandra Administration Tools =

Cassandra has built in tools for accessing Cassandra from the direct download 
such cassandra-cli and node-tool.

This page is to define any administration type of applications for Cassandra.


=== Cassandra Cluster Admin ===
 * [[https://github.com/sebgiroux/Cassandra-Cluster-Admin]]  Cassandra Cluster 
Admin is a GUI tool to help people administrate their Apache Cassandra cluster. 
 Similair to myPHPAdmin for remote MySQL administration.

[jira] [Commented] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079536#comment-13079536
 ] 

Jonathan Ellis commented on CASSANDRA-2901:
---

I added some debug logging that shows that it's actually including extra 
columns in the first pass.

[pass 1]
{noformat}
...
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,056 LazilyCompactedRow.java 
(line 225) added 16481 to serializedSize for 2ab319d0beba11e0fe8ebeead9cb
[the next are bogus]
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,056 LazilyCompactedRow.java 
(line 225) added 17075 to serializedSize for 2acf0640beba11e0fe8ebeead9cb
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,056 LazilyCompactedRow.java 
(line 225) added 17585 to serializedSize for 2af15b50beba11e0fe8ebeead9cb
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,056 LazilyCompactedRow.java 
(line 225) added 17596 to serializedSize for 2af8fc70beba11e0fe8ebeead9cb
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,057 LazilyCompactedRow.java 
(line 225) added 17493 to serializedSize for 2b0335a0beba11e0fe8ebeead9cb
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,057 LazilyCompactedRow.java 
(line 225) added 17493 to serializedSize for 2b200c70beba11e0fe8ebeead9cb
{noformat}

[pass 2]
{noformat}
...
DEBUG [CompactionExecutor:1] 2011-08-04 11:52:42,088 LazilyCompactedRow.java 
(line 225) added 16481 to serializedSize for 2ab319d0beba11e0fe8ebeead9cb
{noformat}

Baffling.

 Allow taking advantage of multiple cores while compacting a single CF
 -

 Key: CASSANDRA-2901
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2901
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.8.4

 Attachments: 
 0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt, 
 0002-parallel-compaction.txt, 0003-Fix-LCR.patch


 Moved from CASSANDRA-1876:
 There are five stages: read, deserialize, merge, serialize, and write. We 
 probably want to continue doing read+deserialize and serialize+write 
 together, or you waste a lot copying to/from buffers.
 So, what I would suggest is: one thread per input sstable doing read + 
 deserialize (a row at a time). A thread pool (one per core?) merging 
 corresponding rows from each input sstable. One thread doing serialize + 
 writing the output (this has to wait for the merge threads to complete 
 in-order, obviously). This should take us from being CPU bound on SSDs (since 
 only one core is compacting) to being I/O bound.
 This will require roughly 2x the memory, to allow the reader threads to work 
 ahead of the merge stage. (I.e. for each input sstable you will have up to 
 one row in a queue waiting to be merged, and the reader thread working on the 
 next.) Seems quite reasonable on that front.  You'll also want a small queue 
 size for the serialize-merged-rows executor.
 Multithreaded compaction should be either on or off. It doesn't make sense to 
 try to do things halfway (by doing the reads with a
 threadpool whose size you can grow/shrink, for instance): we still have 
 compaction threads tuned to low priority, by default, so the impact on the 
 rest of the system won't be very different. Nor do we expect to have so many 
 input sstables that we lose a lot in context switching between reader threads.
 IMO it's acceptable to punt completely on rows that are larger than memory, 
 and fall back to the old non-parallel code there. I don't see any sane way to 
 parallelize large-row compactions.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079541#comment-13079541
 ] 

Jonathan Ellis commented on CASSANDRA-1717:
---

I'd rather not add more configuration complexity for this.

What is the downside to doing it at column index level?  Feels like a good 
compromise between overhead and granularity to me.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of Administration Tools by NickBailey

2011-08-04 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The Administration Tools page has been changed by NickBailey:
http://wiki.apache.org/cassandra/Administration%20Tools?action=diffrev1=1rev2=2

  
  This page is to define any administration type of applications for 
Cassandra.
  
+ === OpsCenter ===
+ 
+ [[http://www.datastax.com/products/opscenter]]
+ 
+ OpsCenter is a tool for management and monitoring of a Cassandra cluster. It 
is provided by DataStax as a free download for non-production use. Production 
licenses for OpsCenter come with Cassandra support provided by DataStax as well.
+ 
+ Screenshots:
+ 
+ {{http://www.datastax.com/wp-content/uploads/2011/02/opscenter3_thumb.png}}
+ {{http://www.datastax.com/wp-content/uploads/2011/02/opscenter1_thumb.png}}
+ {{http://www.datastax.com/wp-content/uploads/2011/02/opscenter2_thumb.png}}
  
  === Cassandra Cluster Admin ===
   * [[https://github.com/sebgiroux/Cassandra-Cluster-Admin]]  Cassandra 
Cluster Admin is a GUI tool to help people administrate their Apache Cassandra 
cluster.  Similair to myPHPAdmin for remote MySQL administration.

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079556#comment-13079556
 ] 

Pavel Yaskevich commented on CASSANDRA-1717:


If we do that on the column index level won't that imply that we will checksum 
(and check) a row as a whole instead of a single column?

Can you please describe your idea about doing it at the column index in here 
just to make sure that we all are on the same page.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079561#comment-13079561
 ] 

Stu Hood commented on CASSANDRA-1717:
-

You should also consider that checksumming at the column index or column level 
will require separate checksums for the column index and row header. 
Checksumming at the block level gets you that in one go.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1154006 - /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/gms/Gossiper.java

2011-08-04 Thread brandonwilliams

Author: brandonwilliams
Date: Thu Aug  4 20:50:25 2011
New Revision: 1154006

URL: http://svn.apache.org/viewvc?rev=1154006view=rev
Log:
Avoid possible race in gossip to unreachable endpoints

Modified:

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/gms/Gossiper.java

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/gms/Gossiper.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/gms/Gossiper.java?rev=1154006r1=1154005r2=1154006view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/gms/Gossiper.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/gms/Gossiper.java
 Thu Aug  4 20:50:25 2011
@@ -443,6 +443,8 @@ public class Gossiper implements IFailur
 private boolean sendGossip(MessageProducer prod, SetInetAddress epSet)
 {
 int size = epSet.size();
+if (size  1)
+return false;
 /* Generate a random number from 0 - size */
 ListInetAddress liveEndpoints = new ArrayListInetAddress(epSet);
 int index = (size == 1) ? 0 : random.nextInt(size);

[jira] [Updated] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-2901:
--

Attachment: (was:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt)

Allow taking advantage of multiple cores while compacting a single CF
-

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-2901:
--

Attachment: (was: 0003-Fix-LCR.patch)

Allow taking advantage of multiple cores while compacting a single CF
-

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-2901:
--

Attachment: (was: 0002-parallel-compaction.txt)

Allow taking advantage of multiple cores while compacting a single CF
-

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-2901:
--

Attachment: 0002-parallel-compaction.txt
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt

Allow taking advantage of multiple cores while compacting a single CF
-

Attachments:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt,
0002-parallel-compaction.txt

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

2011-08-04 Thread Patricio Echague (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: CASSANDRA-2034-trunk-v7.patch

New v7 patch.

- When HH is enabled it waits for the min(RPC Timeout , all replicas's ACK)

The down side is that even for CL.ONE we wait for all the replicas to reply to 
decide if we need to write a hint or not.
So if a node is slow, request/mutation with CL.ONE will perform slightly slower 
when HH is enabled.

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: 2034-formatting.txt, CASSANDRA-2034-trunk-v2.patch, 
 CASSANDRA-2034-trunk-v3.patch, CASSANDRA-2034-trunk-v4.patch, 
 CASSANDRA-2034-trunk-v5.patch, CASSANDRA-2034-trunk-v6.patch, 
 CASSANDRA-2034-trunk-v7.patch, CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079590#comment-13079590
]

Jonathan Ellis commented on CASSANDRA-2901:
---

Found the bug: DeserializedColumnIterator needed to create a new iter on
reset(). (So, it was indeed a problem with mixed lazy/nonlazy iteration,
specifically a problem with eager deserialize that only showed up when it
needed to make multiple passes b/c of the presence of a lazy iterator.)

new patches up. Incorporated Sylvain's improvements as well, partly in 01 and
partly in 02.

Allow taking advantage of multiple cores while compacting a single CF
-

Attachments:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt,
0002-parallel-compaction.txt

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079608#comment-13079608
 ] 

Jonathan Ellis commented on CASSANDRA-1717:
---

what about this?

- add checksum-on-flush to SequentialWriter
- compressed reads will always check on uncompress
- uncompressed reads will check on repair
- and also on read repair digest mismatch retry


 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079615#comment-13079615
]

Jonathan Ellis commented on CASSANDRA-2901:
---

bq. each deserializer now get the full maxInMemorySize

Well, I changed the semantics of maxInMemorySize. So the constructor used
outside of tests looks like

{code}
this(type, getScanners(sstables), controller,
DatabaseDescriptor.getInMemoryCompactionLimit() / Iterables.size(sstables));
{code}

Allow taking advantage of multiple cores while compacting a single CF
-

Attachments:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt,
0002-parallel-compaction.txt

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079616#comment-13079616
 ] 

Pavel Yaskevich commented on CASSANDRA-1717:


This is a good idea but it has few complications:

 - buffer length should be store in order to be used by reader
 - reads should be aligned by that buffer length so we always read a whole 
checksummed chunk of the data which implies that we will potentially always 
need to read more data on each request

This seems to be a clear tradeoff between using additional space to store 
checksum for index + columns for each row v.s. doing more I/O...


 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079616#comment-13079616
 ] 

Pavel Yaskevich edited comment on CASSANDRA-1717 at 8/4/11 9:39 PM:


This is a good idea but it has few complications:

 - buffer length should be stored in order to be used by reader
 - reads should be aligned by that buffer length so we always read a whole 
checksummed chunk of the data which implies that we will potentially always 
need to read more data on each request

This seems to be a clear tradeoff between using additional space to store 
checksum for index + columns for each row v.s. doing more I/O...


  was (Author: xedin):
This is a good idea but it has few complications:

 - buffer length should be store in order to be used by reader
 - reads should be aligned by that buffer length so we always read a whole 
checksummed chunk of the data which implies that we will potentially always 
need to read more data on each request

This seems to be a clear tradeoff between using additional space to store 
checksum for index + columns for each row v.s. doing more I/O...

  
 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079618#comment-13079618
 ] 

Jonathan Ellis commented on CASSANDRA-2034:
---

So I proposed two mutually exclusive approaches here:

- return to the client normally after ConsistencyLevel is achieved, but after 
RpcTimeout we check the responseHandler write acks and write local hints for 
any missing targets
- add a separate executor here, with a blocking, capped queue. When we go to do 
a hint-after-failure we enqueue [and] wait for the write and then return 
success to the client

The difference can be summarized as: do we wait for all hints to be written 
before returning to the client?  If you do, then CL.ONE write latency becomes 
worst-of-N instead of best-of-N.  But, you are guaranteed that successful 
writes have been hinted (if necessary) so you do not have to repair unless 
there is hardware permadeath.  (Otherwise you would have to repair after power 
failure or crashes, too.)

I'm inclined to think that the first option is better, partly because writes 
are *fast* so worst-of-N really isn't that different from best-of-N.  Also, we 
could use CASSANDRA-2819 to reduce the default write timeout while still being 
conservative for reads (which might hit disk).

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: 2034-formatting.txt, CASSANDRA-2034-trunk-v2.patch, 
 CASSANDRA-2034-trunk-v3.patch, CASSANDRA-2034-trunk-v4.patch, 
 CASSANDRA-2034-trunk-v5.patch, CASSANDRA-2034-trunk-v6.patch, 
 CASSANDRA-2034-trunk-v7.patch, CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2993) Issues with parameters being escaped correctly in Python CQL

2011-08-04 Thread Blake Visin (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079629#comment-13079629
 ] 

Blake Visin commented on CASSANDRA-2993:


This is using pickle to serialise objects in python.

 Issues with parameters being escaped correctly in Python CQL
 

 Key: CASSANDRA-2993
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2993
 Project: Cassandra
  Issue Type: Bug
 Environment: Python CQL
Reporter: Blake Visin
  Labels: CQL, parameter, python

 When using parameterised queries in Python CQL strings are not being escaped 
 correctly.
 Query and Parameters:
 {code}
 'UPDATE sites SET :col = :val WHERE KEY = :site_id'
 {'col': 'feed_stats:1312493736688033024',
  'site_id': '899d15e8-bd4a-11e0-bc8c-001fe14cba06',
  'val': 
 (dp0\nS'1'\np1\n(lp2\nI1\naI2\naI3\naI4\nasS'0'\np3\n(lp4\nI1\naI2\naI3\naI4\nasS'3'\np5\n(lp6\nI1\naI2\naI3\naI4\nasS'2'\np7\n(lp8\nI1\naI2\naI3\naI4\nas.}
 {code}
 Query trying to be executed after processing parameters
 {code} 
 UPDATE sites SET 'feed_stats:1312493736688033024' = 
 '(dp0\nS''1''\np1\n(lp2\nI1\naI2\naI3\naI4\nasS''0''\np3\n(lp4\nI1\naI2\naI3\naI4\nasS''3''\np5\n(lp6\nI1\naI2\naI3\naI4\nasS''2''\np7\n(lp8\nI1\naI2\naI3\naI4\nas.'
  WHERE KEY = '899d15e8-bd4a-11e0-bc8c-001fe14cba06'
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2993) Issues with parameters being escaped correctly in Python CQL

2011-08-04 Thread Blake Visin (JIRA)

Issues with parameters being escaped correctly in Python CQL


 Key: CASSANDRA-2993
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2993
 Project: Cassandra
  Issue Type: Bug
 Environment: Python CQL
Reporter: Blake Visin


When using parameterised queries in Python CQL strings are not being escaped 
correctly.


Query and Parameters:
{code}
'UPDATE sites SET :col = :val WHERE KEY = :site_id'

{'col': 'feed_stats:1312493736688033024',
 'site_id': '899d15e8-bd4a-11e0-bc8c-001fe14cba06',
 'val': 
(dp0\nS'1'\np1\n(lp2\nI1\naI2\naI3\naI4\nasS'0'\np3\n(lp4\nI1\naI2\naI3\naI4\nasS'3'\np5\n(lp6\nI1\naI2\naI3\naI4\nasS'2'\np7\n(lp8\nI1\naI2\naI3\naI4\nas.}
{code}

Query trying to be executed after processing parameters
{code} 
UPDATE sites SET 'feed_stats:1312493736688033024' = 
'(dp0\nS''1''\np1\n(lp2\nI1\naI2\naI3\naI4\nasS''0''\np3\n(lp4\nI1\naI2\naI3\naI4\nasS''3''\np5\n(lp6\nI1\naI2\naI3\naI4\nasS''2''\np7\n(lp8\nI1\naI2\naI3\naI4\nas.'
 WHERE KEY = '899d15e8-bd4a-11e0-bc8c-001fe14cba06'

{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (CASSANDRA-2993) Issues with parameters being escaped correctly in Python CQL


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis reassigned CASSANDRA-2993:
-

Assignee: Tyler Hobbs

 Issues with parameters being escaped correctly in Python CQL
 

 Key: CASSANDRA-2993
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2993
 Project: Cassandra
  Issue Type: Bug
 Environment: Python CQL
Reporter: Blake Visin
Assignee: Tyler Hobbs
  Labels: CQL, parameter, python

 When using parameterised queries in Python CQL strings are not being escaped 
 correctly.
 Query and Parameters:
 {code}
 'UPDATE sites SET :col = :val WHERE KEY = :site_id'
 {'col': 'feed_stats:1312493736688033024',
  'site_id': '899d15e8-bd4a-11e0-bc8c-001fe14cba06',
  'val': 
 (dp0\nS'1'\np1\n(lp2\nI1\naI2\naI3\naI4\nasS'0'\np3\n(lp4\nI1\naI2\naI3\naI4\nasS'3'\np5\n(lp6\nI1\naI2\naI3\naI4\nasS'2'\np7\n(lp8\nI1\naI2\naI3\naI4\nas.}
 {code}
 Query trying to be executed after processing parameters
 {code} 
 UPDATE sites SET 'feed_stats:1312493736688033024' = 
 '(dp0\nS''1''\np1\n(lp2\nI1\naI2\naI3\naI4\nasS''0''\np3\n(lp4\nI1\naI2\naI3\naI4\nasS''3''\np5\n(lp6\nI1\naI2\naI3\naI4\nasS''2''\np7\n(lp8\nI1\naI2\naI3\naI4\nas.'
  WHERE KEY = '899d15e8-bd4a-11e0-bc8c-001fe14cba06'
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data

2011-08-04 Thread Todd Lipcon (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079632#comment-13079632
]

Todd Lipcon commented on CASSANDRA-1717:

xedin on IRC asked me to comment on this issue. For reference of what other
systems do: HDFS checksums every file in 512-byte chunks with a CRC32. It's
verified on write (by only the first DN in the pipeline) and on read (by the
client). If the client gets a checksum error while reading, it will report this
to the NN, and the NN will mark that block as corrupt, schedule another
replication, etc.

This is all transparent to the HBase layer since it's done at the FS layer. So,
HBase itself doesn't do any extra checksumming. If you compress your tables,
then you might get an extra layer of checksumming for free from gzip as someone
mentioned above.

For some interesting JIRAs on checksum performance, check out HADOOP-6148 and
various followups, as well as current work in progress HDFS-2080

Cassandra cannot detect corrupt-but-readable column data

Attachments: checksums.txt

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1717) Cassandra cannot detect corrupt-but-readable column data

2011-08-04 Thread Todd Lipcon (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079635#comment-13079635
 ] 

Todd Lipcon commented on CASSANDRA-1717:


BTW, we also scan all blocks verifying checksums in a background process, 
continuously, to catch bit-rot even for data that isn't getting read.

 Cassandra cannot detect corrupt-but-readable column data
 

 Key: CASSANDRA-1717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1717
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
 Fix For: 1.0

 Attachments: checksums.txt


 Most corruptions of on-disk data due to bitrot render the column (or row) 
 unreadable, so the data can be replaced by read repair or anti-entropy.  But 
 if the corruption keeps column data readable we do not detect it, and if it 
 corrupts to a higher timestamp value can even resist being overwritten by 
 newer values.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2994) OutOfBounds in CompressedSequentialWriter.flushData

OutOfBounds in CompressedSequentialWriter.flushData
---

 Key: CASSANDRA-2994
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2994
 Project: Cassandra
  Issue Type: Bug
Reporter: Stu Hood
Assignee: Pavel Yaskevich
 Fix For: 1.0


Near the beginning of a wide row test with CASSANDRA-47 compression enabled on 
a counter column family, I see the following exception:

{code:java} WARN [CompactionExecutor:5] 2011-08-04 21:50:14,558 FileUtils.java 
(line 95) Failed closing 
org.apache.cassandra.io.compress.CompressedSequentialWriter@28f01347
java.lang.IndexOutOfBoundsException
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
at 
org.apache.cassandra.io.compress.CompressedSequentialWriter.flushData(CompressedSequentialWriter.java:88)
at 
org.apache.cassandra.io.util.SequentialWriter.flushInternal(SequentialWriter.java:174)
at 
org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:150)
at 
org.apache.cassandra.io.util.SequentialWriter.close(SequentialWriter.java:283)
at 
org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:159)
at 
org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:91)
at 
org.apache.cassandra.io.sstable.SSTableWriter.cleanupIfNecessary(SSTableWriter.java:201)
at 
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:176)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:120)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:103)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [CompactionExecutor:5] 2011-08-04 21:50:14,561 
AbstractCassandraDaemon.java (line 146) Fatal exception in thread 
Thread[CompactionExecutor:5,1,main]
java.lang.IndexOutOfBoundsException
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
at 
org.apache.cassandra.io.compress.CompressedSequentialWriter.flushData(CompressedSequentialWriter.java:88)
at 
org.apache.cassandra.io.util.SequentialWriter.flushInternal(SequentialWriter.java:174)
at 
org.apache.cassandra.io.util.SequentialWriter.reBuffer(SequentialWriter.java:226)
at 
org.apache.cassandra.io.util.SequentialWriter.writeAtMost(SequentialWriter.java:117)
at 
org.apache.cassandra.io.util.SequentialWriter.write(SequentialWriter.java:101)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at 
org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:105)
at 
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:150)
at 
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:120)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:103)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2915) Lucene based Secondary Indexes

2011-08-04 Thread Todd Nine (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079653#comment-13079653
]

Todd Nine edited comment on CASSANDRA-2915 at 8/4/11 10:33 PM:
---

Hey guys. We're doing something similar in the hector JPA plugin.

Would using dynamic composites within cassandra alleviate the need for Lucene
documents? We're using this in secondary indexing and it gives us order by
semantics and AND (Union). The largest issue becomes iteration with OR
clauses, AND clauses can be compressed into a single column for efficient range
scans, we then use iterators to UNION the OR trees together with order clauses
in the composites. The caveat is that the user must define indexes with order
semantics up front. However this can easily be added to the existing secondary
indexing clauses.

was (Author: tnine):
Hey guys. We're doing something similar in the hector JPA plugin.

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2994) OutOfBounds in CompressedSequentialWriter.flushData


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079661#comment-13079661
 ] 

Stu Hood edited comment on CASSANDRA-2994 at 8/4/11 10:52 PM:
--

The workload was a wide row usecase, but based on the stack trace, the row 
hadn't reached a width to trigger LazilyCompactedRow.

  was (Author: stuhood):
The workload was a wide row usecase, but based on the stack trace, the row 
had reached a width to trigger LazilyCompactedRow.
  
 OutOfBounds in CompressedSequentialWriter.flushData
 ---

 Key: CASSANDRA-2994
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2994
 Project: Cassandra
  Issue Type: Bug
Reporter: Stu Hood
Assignee: Pavel Yaskevich
 Fix For: 1.0


 Near the beginning of a wide row test with CASSANDRA-47 compression enabled 
 on a counter column family, I see the following exception:
 {code:java} WARN [CompactionExecutor:5] 2011-08-04 21:50:14,558 
 FileUtils.java (line 95) Failed closing 
 org.apache.cassandra.io.compress.CompressedSequentialWriter@28f01347
 java.lang.IndexOutOfBoundsException
   at java.io.RandomAccessFile.writeBytes(Native Method)
   at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
   at 
 org.apache.cassandra.io.compress.CompressedSequentialWriter.flushData(CompressedSequentialWriter.java:88)
   at 
 org.apache.cassandra.io.util.SequentialWriter.flushInternal(SequentialWriter.java:174)
   at 
 org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:150)
   at 
 org.apache.cassandra.io.util.SequentialWriter.close(SequentialWriter.java:283)
   at 
 org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:159)
   at 
 org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:91)
   at 
 org.apache.cassandra.io.sstable.SSTableWriter.cleanupIfNecessary(SSTableWriter.java:201)
   at 
 org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:176)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:120)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:103)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 ERROR [CompactionExecutor:5] 2011-08-04 21:50:14,561 
 AbstractCassandraDaemon.java (line 146) Fatal exception in thread 
 Thread[CompactionExecutor:5,1,main]
 java.lang.IndexOutOfBoundsException
   at java.io.RandomAccessFile.writeBytes(Native Method)
   at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
   at 
 org.apache.cassandra.io.compress.CompressedSequentialWriter.flushData(CompressedSequentialWriter.java:88)
   at 
 org.apache.cassandra.io.util.SequentialWriter.flushInternal(SequentialWriter.java:174)
   at 
 org.apache.cassandra.io.util.SequentialWriter.reBuffer(SequentialWriter.java:226)
   at 
 org.apache.cassandra.io.util.SequentialWriter.writeAtMost(SequentialWriter.java:117)
   at 
 org.apache.cassandra.io.util.SequentialWriter.write(SequentialWriter.java:101)
   at java.io.DataOutputStream.write(DataOutputStream.java:90)
   at 
 org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:105)
   at 
 org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:150)
   at 
 org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:120)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:103)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079682#comment-13079682
]

Jason Rutherglen commented on CASSANDRA-2915:
-

bq. Would using dynamic composites within cassandra alleviate the need for
Lucene documents?

I think it is hard to duplicate the efficiency of Lucene for dis/conjunction
queries (OR / AND), especially with PFOR implemented (a CPU directed enhanced
system for decoding integers on todays microprocessors).

We can/will turn off scoring which further makes Lucene a straight query
execution engine, as opposed to a free text search engine. Range queries in
Lucene use a trie system which is highly effective.

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079684#comment-13079684
]

Jason Rutherglen commented on CASSANDRA-2915:
-

I think the open design question on this one is distributed search, and how a
distributed search client will know which Cassandra servers to send a query to.
Meaning, traditionally a query is sent to N servers whose responses are merged
and X results are returned. We can send a query to all servers however I think
we'd then have duplicate rows/documents returned. How does CQL handle this?

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

2011-08-04 Thread Todd Nine (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079695#comment-13079695
]

Todd Nine commented on CASSANDRA-2915:
--

I'm quite keen to contribute on this issue, as this will greatly enhance the
functionality of the hector-jpa project. If I can contribute any work, please
let me know.

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079703#comment-13079703
]

Stu Hood commented on CASSANDRA-2901:
-

I like the addition of close() to AbstractCompactedRow, but...

At first glance, I'm not convinced the complexity of this ticket is worth the
benefits:
* If compaction is falling behind during normal operation (aka, too many
sstables), the multithreaded compaction from CASSANDRA-2191 should kick in
appropriately, assuming you have reasonable compaction thresholds (for example,
you can increase parallelism by lowering the MAX_THRESHOLD)
* For repair-related compactions, it sounds like there are a few other efforts
underway to make the tasks less like a major compaction
* Major compactions would benefit from this ticket, but I think our endgame is
that major compactions will be a thing of the past

Finally, with CASSANDRA-1608, there should be more and easier available options
for parallelizing compaction, since there will be more, smaller files.

Allow taking advantage of multiple cores while compacting a single CF
-

Attachments:
0001-fix-tracker-getting-out-of-sync-with-underlying-data-s.txt,
0002-parallel-compaction.txt

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

2011-08-04 Thread T Jake Luciani (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079712#comment-13079712
]

T Jake Luciani commented on CASSANDRA-2915:
---

Todd: once CASSANDRA-2982 is done we can get started. I'm trying to focus on
that right now. In the meantime we need to think of how to link lucene
analyzers to column_metadata.

Jason: This currently works by executing the query locally if that does not
have enough results it moves on to the next node. since the ring is split we
know the range of keys to restrict the search to. this avoids dups

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079735#comment-13079735
]

Jason Rutherglen commented on CASSANDRA-2915:
-

bq. like getLuceneAnalyzer()

There won't always be a 1 to 1 mapping of a column to a field. For example in
Solr, there is copy field, which essentially creates a new field. Also
Analyzer is for any field, the right per-field class would be Tokenizer.

I strongly believe we need to have an interface that accepts a row and
essentially generates a Lucene Document. This should be the most
straightforward approach that enables just about anything, including using a
Solr schema at some point.

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

2011-08-04 Thread T Jake Luciani (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079745#comment-13079745
]

T Jake Luciani commented on CASSANDRA-2915:
---

bq. Will read after write be available? I.E if your mutation for the row key
returns to the client, then the row now has an entry in the Lucence index,
which can immediately be queried to return the results.

Yes. We can use a RAMDirectory() to keep writes real-time.

bq. What about durability, in the event cassandra crashes, will the Lucene
index retain these indexed values, or will they be lost if commit is not
invoked on the index?

When the memtable is flushed. we will merge the RAMDirectory index into the
FSDirectory index and call reopen().

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2915) Lucene based Secondary Indexes

[
https://issues.apache.org/jira/browse/CASSANDRA-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079776#comment-13079776
]

Jason Rutherglen commented on CASSANDRA-2915:
-

bq. Yes. We can use a RAMDirectory() to keep writes real-time.

LUCENE-3092 implemented NRTCachingDirectory which we can use for in RAM NRT
until LUCENE-2312 is completed.

Lucene based Secondary Indexes
--

Key: CASSANDRA-2915
URL: https://issues.apache.org/jira/browse/CASSANDRA-2915
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: T Jake Luciani
Labels: secondary_index
Fix For: 1.0

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2474) CQL support for compound columns