svn commit: r1407374 - in /cassandra/site: publish/download/index.html src/settings.py
Author: slebresne Date: Fri Nov 9 07:59:31 2012 New Revision: 1407374 URL: http://svn.apache.org/viewvc?rev=1407374view=rev Log: Update website for 1.2.0-beta2 release Modified: cassandra/site/publish/download/index.html cassandra/site/src/settings.py Modified: cassandra/site/publish/download/index.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/download/index.html?rev=1407374r1=1407373r2=1407374view=diff == --- cassandra/site/publish/download/index.html (original) +++ cassandra/site/publish/download/index.html Fri Nov 9 07:59:31 2012 @@ -107,16 +107,16 @@ h2 class=hdrDevelopment Cassandra Server Releases (not production ready)/h2 p - The latest development release is 1.2.0-beta1 (released on - 2012-09-24). + The latest development release is 1.2.0-beta2 (released on + 2012-11-09). /p ul li -a class=filename href=http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-bin.tar.gz;apache-cassandra-1.2.0-beta1-bin.tar.gz/a -[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-bin.tar.gz.asc;PGP/a] -[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-bin.tar.gz.md5;MD5/a] -[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-bin.tar.gz.sha1;SHA1/a] +a class=filename href=http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-bin.tar.gz;apache-cassandra-1.2.0-beta2-bin.tar.gz/a +[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-bin.tar.gz.asc;PGP/a] +[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-bin.tar.gz.md5;MD5/a] +[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-bin.tar.gz.sha1;SHA1/a] /li /ul @@ -195,10 +195,10 @@ li -a class=filename href=http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-src.tar.gz;apache-cassandra-1.2.0-beta1-src.tar.gz/a -[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-src.tar.gz.asc;PGP/a] -[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-src.tar.gz.md5;MD5/a] -[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta1-src.tar.gz.sha1;SHA1/a] +a class=filename href=http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-src.tar.gz;apache-cassandra-1.2.0-beta2-src.tar.gz/a +[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-src.tar.gz.asc;PGP/a] +[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-src.tar.gz.md5;MD5/a] +[a href=http://www.apache.org/dist/cassandra/1.2.0/apache-cassandra-1.2.0-beta2-src.tar.gz.sha1;SHA1/a] /li /ul Modified: cassandra/site/src/settings.py URL: http://svn.apache.org/viewvc/cassandra/site/src/settings.py?rev=1407374r1=1407373r2=1407374view=diff == --- cassandra/site/src/settings.py (original) +++ cassandra/site/src/settings.py Fri Nov 9 07:59:31 2012 @@ -100,8 +100,8 @@ class CassandraDef(object): veryoldstable_exists = True stable_version = '1.1.6' stable_release_date = '2012-10-15' -devel_version = '1.2.0-beta1' -devel_release_date = '2012-09-24' +devel_version = '1.2.0-beta2' +devel_release_date = '2012-11-09' devel_exists = True _apache_base_url = 'http://www.apache.org' _svn_base_url = 'https://svn.apache.org/repos/asf'
[jira] [Updated] (CASSANDRA-4914) Aggregate functions in CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-4914: Fix Version/s: (was: 1.2.0 beta 2) 1.2.1 Aggregate functions in CQL -- Key: CASSANDRA-4914 URL: https://issues.apache.org/jira/browse/CASSANDRA-4914 Project: Cassandra Issue Type: Bug Reporter: Vijay Assignee: Vijay Fix For: 1.2.1 The requirement is to do aggregation of data in Cassandra (Wide row of column values of int, double, float etc). With some basic agree gate functions like AVG, SUM, Mean, Min, Max, etc (for the columns within a row). Example: SELECT * FROM emp WHERE empID IN (130) ORDER BY deptID DESC; empid | deptid | first_name | last_name | salary ---+++---+ 130 | 3 | joe| doe | 10.1 130 | 2 | joe| doe |100 130 | 1 | joe| doe | 1e+03 SELECT sum(salary), empid FROM emp WHERE empID IN (130); sum(salary) | empid -+ 1110.1| 130 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4914) Aggregate functions in CQL
[
https://issues.apache.org/jira/browse/CASSANDRA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493857#comment-13493857
]
Sylvain Lebresne commented on CASSANDRA-4914:
-
I'm not necessarily opposed to the idea on principle, but unless we have some
fancy idea, this will only save networks between the client and the
coordinator, as internally we'll still have to pull all the data (though that's
not very different from what we do for count today). Meaning that if we do
that, we should be clear about that fact and that people should still go the
hadoop route to do large aggregations.
I'm also halfway convinced that it wouldn't be much harder to support custom
filter functions. I.e. to allow people to define some class having a method
along the line of:
{noformat}
public ResultSet filter(ResultSet rs);
{noformat}
and so that it might be worth to go that more general route right away and just
provide a number of default aggregation functions.
I'm also not sure it's wise to support this until we can properly page CQL
queries (i.e. I think this should depends on CASSANDRA-4415). Also, I think it
would be weird to introduce aggregation before we remove our current select
arbitrary limit (though I'm in favor of doing that sooner than later:
CASSANDRA-4918).
Lastly, aggregation might lose a bit of it's usefulness without a proper
support for DISTINCT. So overall my opinion would be: if we do do that, let's
push that to 1.3 and do that correctly.
Aggregate functions in CQL
--
Key: CASSANDRA-4914
URL: https://issues.apache.org/jira/browse/CASSANDRA-4914
Project: Cassandra
Issue Type: Bug
Reporter: Vijay
Assignee: Vijay
Fix For: 1.2.1
The requirement is to do aggregation of data in Cassandra (Wide row of column
values of int, double, float etc).
With some basic agree gate functions like AVG, SUM, Mean, Min, Max, etc (for
the columns within a row).
Example:
SELECT * FROM emp WHERE empID IN (130) ORDER BY deptID DESC;
empid | deptid | first_name | last_name | salary
---+++---+
130 | 3 | joe| doe | 10.1
130 | 2 | joe| doe |100
130 | 1 | joe| doe | 1e+03
SELECT sum(salary), empid FROM emp WHERE empID IN (130);
sum(salary) | empid
-+
1110.1| 130
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4938) Response from CREATE INDEX command comes faster than the creation of secondary index.
Krzysztof Cieslinski Cognitum created CASSANDRA-4938:
Summary: Response from CREATE INDEX command comes faster than the
creation of secondary index.
Key: CASSANDRA-4938
URL: https://issues.apache.org/jira/browse/CASSANDRA-4938
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.0 beta 1
Reporter: Krzysztof Cieslinski Cognitum
Response from CREATE INDEX command comes faster than the creation of secondary
index. So below code:
{code:xml}
CREATE INDEX ON tab(name);
SELECT * FROM tab WHERE name = 'Chris';
{code}
doesn't return any rows(of course, in column family tab, there are some
records with name value = 'Chris'..) and any errors ( i would expect
something like ??Bad Request: No indexed columns present in by-columns clause
with Equal operator??)
Inputing some timeout between those two commands resolves the problem, so:
{code:xml}
CREATE INDEX ON tab(name);
Sleep(timeout); // for column family with 2000 rows the timeout had to be set
for ~1 second
SELECT * FROM tab WHERE name = 'Chris';
{code}
will return all rows with values as specified.
I'm using single node cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4933) SimpleAuthority is incompatible with new Permissions and resources lists
[ https://issues.apache.org/jira/browse/CASSANDRA-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493999#comment-13493999 ] Jonathan Ellis commented on CASSANDRA-4933: --- That's right. SimpleAuthority is incompatible with new Permissions and resources lists -- Key: CASSANDRA-4933 URL: https://issues.apache.org/jira/browse/CASSANDRA-4933 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.6 Environment: Previously 1.1.0 + Authentication patch CASSANDRA-4155 (working). Currently: 1.1.6 (not working) Reporter: Michał Michalski Assignee: Michał Michalski Priority: Trivial Attachments: cassandra-1.1.6-issue-4933.txt Commit aba5a37650232dbf10b505c04b257f73b6c9b579 by Pavel Yaskevich introduced some significant changes in Permissions system. Except new permission types, also resource hierarchy has changed - previously creating a keyspace was requesting for for WRITE permission for /cassandra/keyspaces. Now it requests for CREATE permission for /cassandra/keyspaces/new-keyspace-name. This change brakes the SimpleAuthority code that relies on the length of the resource list which differs now - we cannot distinguish operations that modify keyspaces (perviously: resource list of length 2; currently: 3) from these ones that read it or so (resource list of length 3). I've prepared a patch that fixes it in the way that I understand it should work now (comment in 1.1.6's IAuthority.java is out of date and refers to old READ/WRITE permissions only). Yes, I know that SimpleAuth(enticator/ority) are deprecated, should not be used in production and so on, but even if they are unsufficient as a protection from external threats, they're still good enough in our case as a very basic protection from accidental changes made by developers ;) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Git Push Summary
Updated Branches: refs/heads/cassandra-1.2.0 [created] 686a17cf4
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494047#comment-13494047
]
Jonathan Ellis commented on CASSANDRA-4924:
---
bq. Statements like This CQL definition will store data in exactly the same
way than the thrift definition above
(http://www.datastax.com/dev/blog/thrift-to-cql3) Created the impression two
API's were using the storage engine in an equal but different way. Hiding table
definitions from one API because it does things the other will not understand
goes against those statements and that impression.
I'm not 100% sure what you mean by the first part, but the second is easy to
elaborate on:
* CQL3 can always make sense of a Thrift CF definition.
* The reverse is not true.
It's the difference between backwards compatibility and forward-compatibility.
This is pretty basic.
bq. We've been managing that on the client for the past 2.5 years with well
typed exceptions
It sounds like you're talking about (say) Hector circa Jan 2013. I'm talking
about Hector circa Oct 2012. The latter is not *forwards compatible* with what
we've done in CQL3: if we waved a wand and included all the metadata in CfDef,
the already-released versions of Hector can't deal with them and will in fact
do the Wrong Thing.
In other words: previously we've achieved forwards compatibility mostly by
luck, because we haven't had to change anything fundamental since 0.7. Now, we
have a situation where we can't achieve perfect forwards compatibility, but
what we can do is hide the incompatible parts from Hector-2012 so it doesn't
shoot itself in the foot.
Hector-2013 of course can use CQL3 introspection to inspect the schema, and can
then proceed to mangle it via Thrift RPC if it really wants to.
bq. While it's a little bit more work of supporting thrift
First, I want to repeat that it's less about saving work than it is about
salvaging what forwards compatibility we can.
For the cli specifically, we could indeed show that the table exists (by using
the CQL3 api) but then what?
# Try to cram it into the same format used by the others? That would lose
crucial information. Remember that people use this to back up schema, so
that would not be a friendly thing to do.
# Show the CQL3 definition? That fails the principle of least surprise, the
CLI can't actually interpret that syntax. If you want CQL3 syntax use cqlsh.
bq. We really can not live without it. If your need to insert to this CF from
thrift/hector having the CLI output show you the 'old school' way is the most
helpful.
While I still think it's wrongheaded to try to jam CQL3 into an old Hector app,
it is reasonable to add CQL3 introspection to the CLI on GET. (SET already
works.)
Pushed this to http://github.com/jbellis/cassandra/branches/4924.
{{COMPACT STORAGE}} works well, because this is thrift compatibility mode; PK
columns get packed into the comparator, and the last one gets turned into the
default validator.
{noformat}
cqlsh:keyspace1 create table testcompact (a text, b int, c int, primary key
(a, b)) with compact storage;
cqlsh:keyspace1 insert into testcompact (a, b, c) values ('k1', 1, 2);
[default@keyspace1] get testcompact['k1'];
= (column=1, value=2, timestamp=1352474787942000)
{noformat}
sparse CQL tables work less well because thrift/cli doesn't really have a way
to represent transposed (cql) columns within a (storage engine) row. So all
column values show as bytes, but at least you can see them:
{noformat}
cqlsh:keyspace1 create table testsparse (a text, b int, c int, primary key (a,
b));
cqlsh:keyspace1 insert into testsparse (a, b, c) values ('k1', 1, 2);
[default@keyspace1] get testsparse['k1'];
= (column=1:, value=, timestamp=135247473947)
= (column=1:c, value=0002, timestamp=135247473947)
{noformat}
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on
git commit: Update slf4j to a modern version
Updated Branches: refs/heads/cassandra-1.2.0 686a17cf4 - 03f8fec8c Update slf4j to a modern version Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03f8fec8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03f8fec8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03f8fec8 Branch: refs/heads/cassandra-1.2.0 Commit: 03f8fec8c4a8737c08f0f5e1a82a54c58811a699 Parents: 686a17c Author: Sylvain Lebresne sylv...@datastax.com Authored: Fri Nov 9 16:47:38 2012 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Fri Nov 9 16:48:31 2012 +0100 -- build.xml |4 ++-- lib/slf4j-api-1.6.1.jar | Bin 25496 - 0 bytes lib/slf4j-api-1.7.2.jar | Bin 0 - 26083 bytes lib/slf4j-log4j12-1.6.1.jar | Bin 9753 - 0 bytes lib/slf4j-log4j12-1.7.2.jar | Bin 0 - 8819 bytes 5 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/build.xml -- diff --git a/build.xml b/build.xml index a9a1b14..6d4d4a0 100644 --- a/build.xml +++ b/build.xml @@ -336,8 +336,8 @@ dependency groupId=commons-lang artifactId=commons-lang version=2.4/ dependency groupId=com.googlecode.concurrentlinkedhashmap artifactId=concurrentlinkedhashmap-lru version=1.3/ dependency groupId=org.antlr artifactId=antlr version=3.2/ - dependency groupId=org.slf4j artifactId=slf4j-api version=1.6.1/ - dependency groupId=org.slf4j artifactId=slf4j-log4j12 version=1.6.1/ + dependency groupId=org.slf4j artifactId=slf4j-api version=1.7.2/ + dependency groupId=org.slf4j artifactId=slf4j-log4j12 version=1.7.2/ dependency groupId=org.codehaus.jackson artifactId=jackson-core-asl version=1.4.0/ dependency groupId=org.codehaus.jackson artifactId=jackson-mapper-asl version=1.4.0/ dependency groupId=jline artifactId=jline version=1.0 http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-api-1.6.1.jar -- diff --git a/lib/slf4j-api-1.6.1.jar b/lib/slf4j-api-1.6.1.jar deleted file mode 100644 index 42e0ad0..000 Binary files a/lib/slf4j-api-1.6.1.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-api-1.7.2.jar -- diff --git a/lib/slf4j-api-1.7.2.jar b/lib/slf4j-api-1.7.2.jar new file mode 100644 index 000..73f38db Binary files /dev/null and b/lib/slf4j-api-1.7.2.jar differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-log4j12-1.6.1.jar -- diff --git a/lib/slf4j-log4j12-1.6.1.jar b/lib/slf4j-log4j12-1.6.1.jar deleted file mode 100644 index 873d119..000 Binary files a/lib/slf4j-log4j12-1.6.1.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-log4j12-1.7.2.jar -- diff --git a/lib/slf4j-log4j12-1.7.2.jar b/lib/slf4j-log4j12-1.7.2.jar new file mode 100644 index 000..37a85d7 Binary files /dev/null and b/lib/slf4j-log4j12-1.7.2.jar differ
git commit: Update slf4j to a modern version
Updated Branches: refs/heads/cassandra-1.2 686a17cf4 - 03f8fec8c Update slf4j to a modern version Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03f8fec8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03f8fec8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03f8fec8 Branch: refs/heads/cassandra-1.2 Commit: 03f8fec8c4a8737c08f0f5e1a82a54c58811a699 Parents: 686a17c Author: Sylvain Lebresne sylv...@datastax.com Authored: Fri Nov 9 16:47:38 2012 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Fri Nov 9 16:48:31 2012 +0100 -- build.xml |4 ++-- lib/slf4j-api-1.6.1.jar | Bin 25496 - 0 bytes lib/slf4j-api-1.7.2.jar | Bin 0 - 26083 bytes lib/slf4j-log4j12-1.6.1.jar | Bin 9753 - 0 bytes lib/slf4j-log4j12-1.7.2.jar | Bin 0 - 8819 bytes 5 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/build.xml -- diff --git a/build.xml b/build.xml index a9a1b14..6d4d4a0 100644 --- a/build.xml +++ b/build.xml @@ -336,8 +336,8 @@ dependency groupId=commons-lang artifactId=commons-lang version=2.4/ dependency groupId=com.googlecode.concurrentlinkedhashmap artifactId=concurrentlinkedhashmap-lru version=1.3/ dependency groupId=org.antlr artifactId=antlr version=3.2/ - dependency groupId=org.slf4j artifactId=slf4j-api version=1.6.1/ - dependency groupId=org.slf4j artifactId=slf4j-log4j12 version=1.6.1/ + dependency groupId=org.slf4j artifactId=slf4j-api version=1.7.2/ + dependency groupId=org.slf4j artifactId=slf4j-log4j12 version=1.7.2/ dependency groupId=org.codehaus.jackson artifactId=jackson-core-asl version=1.4.0/ dependency groupId=org.codehaus.jackson artifactId=jackson-mapper-asl version=1.4.0/ dependency groupId=jline artifactId=jline version=1.0 http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-api-1.6.1.jar -- diff --git a/lib/slf4j-api-1.6.1.jar b/lib/slf4j-api-1.6.1.jar deleted file mode 100644 index 42e0ad0..000 Binary files a/lib/slf4j-api-1.6.1.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-api-1.7.2.jar -- diff --git a/lib/slf4j-api-1.7.2.jar b/lib/slf4j-api-1.7.2.jar new file mode 100644 index 000..73f38db Binary files /dev/null and b/lib/slf4j-api-1.7.2.jar differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-log4j12-1.6.1.jar -- diff --git a/lib/slf4j-log4j12-1.6.1.jar b/lib/slf4j-log4j12-1.6.1.jar deleted file mode 100644 index 873d119..000 Binary files a/lib/slf4j-log4j12-1.6.1.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-log4j12-1.7.2.jar -- diff --git a/lib/slf4j-log4j12-1.7.2.jar b/lib/slf4j-log4j12-1.7.2.jar new file mode 100644 index 000..37a85d7 Binary files /dev/null and b/lib/slf4j-log4j12-1.7.2.jar differ
git commit: Update slf4j to a modern version
Updated Branches: refs/heads/trunk 686a17cf4 - 03f8fec8c Update slf4j to a modern version Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03f8fec8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03f8fec8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03f8fec8 Branch: refs/heads/trunk Commit: 03f8fec8c4a8737c08f0f5e1a82a54c58811a699 Parents: 686a17c Author: Sylvain Lebresne sylv...@datastax.com Authored: Fri Nov 9 16:47:38 2012 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Fri Nov 9 16:48:31 2012 +0100 -- build.xml |4 ++-- lib/slf4j-api-1.6.1.jar | Bin 25496 - 0 bytes lib/slf4j-api-1.7.2.jar | Bin 0 - 26083 bytes lib/slf4j-log4j12-1.6.1.jar | Bin 9753 - 0 bytes lib/slf4j-log4j12-1.7.2.jar | Bin 0 - 8819 bytes 5 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/build.xml -- diff --git a/build.xml b/build.xml index a9a1b14..6d4d4a0 100644 --- a/build.xml +++ b/build.xml @@ -336,8 +336,8 @@ dependency groupId=commons-lang artifactId=commons-lang version=2.4/ dependency groupId=com.googlecode.concurrentlinkedhashmap artifactId=concurrentlinkedhashmap-lru version=1.3/ dependency groupId=org.antlr artifactId=antlr version=3.2/ - dependency groupId=org.slf4j artifactId=slf4j-api version=1.6.1/ - dependency groupId=org.slf4j artifactId=slf4j-log4j12 version=1.6.1/ + dependency groupId=org.slf4j artifactId=slf4j-api version=1.7.2/ + dependency groupId=org.slf4j artifactId=slf4j-log4j12 version=1.7.2/ dependency groupId=org.codehaus.jackson artifactId=jackson-core-asl version=1.4.0/ dependency groupId=org.codehaus.jackson artifactId=jackson-mapper-asl version=1.4.0/ dependency groupId=jline artifactId=jline version=1.0 http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-api-1.6.1.jar -- diff --git a/lib/slf4j-api-1.6.1.jar b/lib/slf4j-api-1.6.1.jar deleted file mode 100644 index 42e0ad0..000 Binary files a/lib/slf4j-api-1.6.1.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-api-1.7.2.jar -- diff --git a/lib/slf4j-api-1.7.2.jar b/lib/slf4j-api-1.7.2.jar new file mode 100644 index 000..73f38db Binary files /dev/null and b/lib/slf4j-api-1.7.2.jar differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-log4j12-1.6.1.jar -- diff --git a/lib/slf4j-log4j12-1.6.1.jar b/lib/slf4j-log4j12-1.6.1.jar deleted file mode 100644 index 873d119..000 Binary files a/lib/slf4j-log4j12-1.6.1.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/03f8fec8/lib/slf4j-log4j12-1.7.2.jar -- diff --git a/lib/slf4j-log4j12-1.7.2.jar b/lib/slf4j-log4j12-1.7.2.jar new file mode 100644 index 000..37a85d7 Binary files /dev/null and b/lib/slf4j-log4j12-1.7.2.jar differ
[jira] [Commented] (CASSANDRA-4934) assertion error in offheap bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494082#comment-13494082
]
Jonathan Ellis commented on CASSANDRA-4934:
---
I think the problem is that compaction never bothered to mark overlapped
sstables (that it uses for tombstone purge checking) as referenced, so those
can get free'd out from under an ongoing compaction. Before OHBS this was not
a problem because the contents of those sstables was never referenced, only the
bloom filter, which the GC kept around as long as it was needed. But now we
need to manually manage that.
Patchset pushed to http://github.com/jbellis/cassandra/branches/4934
assertion error in offheap bloom filter
---
Key: CASSANDRA-4934
URL: https://issues.apache.org/jira/browse/CASSANDRA-4934
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 2
Reporter: Brandon Williams
Assignee: Vijay
Fix For: 1.2.0 rc1
Attachments: log.txt
Saw this while running the dtests:
{noformat}
INFO [CompactionExecutor:2] 2012-11-08 09:35:18,206 CompactionTask.java
(line 116) Compacting
[SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-9-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-11-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-10-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-13-Data.db')]
ERROR [CompactionExecutor:2] 2012-11-08 09:35:18,257 CassandraDaemon.java
(line 132) Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:134)
at org.apache.cassandra.io.util.Memory.getByte(Memory.java:104)
at org.apache.cassandra.utils.obs.OffHeapBitSet.get(OffHeapBitSet.java:60)
at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:71)
at
org.apache.cassandra.db.compaction.CompactionController.shouldPurge(CompactionController.java:106)
at
org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:64)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:95)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:151)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$8.computeNext(Iterators.java:734)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:179)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4934) assertion error in offheap bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4934:
--
Reviewer: vijay2...@yahoo.com
Assignee: Jonathan Ellis (was: Vijay)
assertion error in offheap bloom filter
---
Key: CASSANDRA-4934
URL: https://issues.apache.org/jira/browse/CASSANDRA-4934
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 2
Reporter: Brandon Williams
Assignee: Jonathan Ellis
Fix For: 1.2.0 rc1
Attachments: log.txt
Saw this while running the dtests:
{noformat}
INFO [CompactionExecutor:2] 2012-11-08 09:35:18,206 CompactionTask.java
(line 116) Compacting
[SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-9-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-11-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-10-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-13-Data.db')]
ERROR [CompactionExecutor:2] 2012-11-08 09:35:18,257 CassandraDaemon.java
(line 132) Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:134)
at org.apache.cassandra.io.util.Memory.getByte(Memory.java:104)
at org.apache.cassandra.utils.obs.OffHeapBitSet.get(OffHeapBitSet.java:60)
at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:71)
at
org.apache.cassandra.db.compaction.CompactionController.shouldPurge(CompactionController.java:106)
at
org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:64)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:95)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:151)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$8.computeNext(Iterators.java:734)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:179)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3974) Per-CF TTL
[ https://issues.apache.org/jira/browse/CASSANDRA-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494091#comment-13494091 ] Kirk True commented on CASSANDRA-3974: -- Pinging for feedback. Per-CF TTL -- Key: CASSANDRA-3974 URL: https://issues.apache.org/jira/browse/CASSANDRA-3974 Project: Cassandra Issue Type: New Feature Affects Versions: 1.2.0 beta 1 Reporter: Jonathan Ellis Assignee: Kirk True Priority: Minor Fix For: 1.2.0 rc1 Attachments: trunk-3974.txt, trunk-3974v2.txt, trunk-3974v3.txt, trunk-3974v4.txt, trunk-3974v5.txt, trunk-3974v6.txt Per-CF TTL would allow compaction optimizations (drop an entire sstable's worth of expired data) that we can't do with per-column. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3017) add a Message size limit
[ https://issues.apache.org/jira/browse/CASSANDRA-3017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494090#comment-13494090 ] Kirk True commented on CASSANDRA-3017: -- Pinging for feedback. add a Message size limit Key: CASSANDRA-3017 URL: https://issues.apache.org/jira/browse/CASSANDRA-3017 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Kirk True Priority: Minor Labels: lhf Attachments: 0001-use-the-thrift-max-message-size-for-inter-node-messa.patch, trunk-3017.txt We protect the server from allocating huge buffers for malformed message with the Thrift frame size (CASSANDRA-475). But we don't have similar protection for the inter-node Message objects. Adding this would be good to deal with malicious adversaries as well as a malfunctioning cluster participant. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4848) Expose black-listed directories via JMX
[
https://issues.apache.org/jira/browse/CASSANDRA-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated CASSANDRA-4848:
-
Priority: Minor (was: Major)
Labels: lhf (was: )
Expose black-listed directories via JMX
---
Key: CASSANDRA-4848
URL: https://issues.apache.org/jira/browse/CASSANDRA-4848
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: Kirk True
Assignee: Kirk True
Priority: Minor
Labels: lhf
JBOD support included failure modes (CASSANDRA-2118) that insert directories
on bad disks to the {{BlacklistedDirectories}} class. However, it doesn't
appear that the list of directories is exposed to administrators via JMX or
other means. So unless they're scouring the logs or being notified at the OS
level, they will remain unaware of the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4550) nodetool ring output should use hex not integers for tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494092#comment-13494092 ] Kirk True commented on CASSANDRA-4550: -- Pinging for feedback. nodetool ring output should use hex not integers for tokens --- Key: CASSANDRA-4550 URL: https://issues.apache.org/jira/browse/CASSANDRA-4550 Project: Cassandra Issue Type: Improvement Components: Tools Environment: Linux Reporter: Aaron Turner Assignee: Kirk True Priority: Trivial Labels: lhf Attachments: trunk-4550.txt The current output of nodetool ring prints start token values as base10 integers instead of hex. This is not very user friendly for a number of reasons: 1. Hides the fact that the values are 128bit 2. Values are not of a consistent length, while in hex padding with zero is generally accepted 3. When using the default random partitioner, having the values in hex makes it easier for users to determine which node(s) a given key resides on since md5 utilities like md5sum output hex. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3974) Per-CF TTL
[
https://issues.apache.org/jira/browse/CASSANDRA-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494137#comment-13494137
]
Sylvain Lebresne commented on CASSANDRA-3974:
-
I realize I'm reviewer on this one. I seem that remember that [~jbellis] wanted
to have a look at that but maybe I misunderstood that?
In any case, I had a look at that patch and that looks good to me overall. That
being, and that's not really a criticism of the patch, I do was slightly
surprised that we only need to modify {{ColumnFamily.addColumn(QueryPath,
...)}} and {{InsertStatement}} to make that work. Don't get me wrong, I do
think this is correct, but it does feel a bit fragile that some future internal
code could too easily add an ExpiringColumn though
ColumnFamily.addColumn(IColumn) and skip the global cf setting. I don't really
have any good solution to make it less fragile however, I'm just thinking out
loud. But that remark aside, again the patch does lgtm (aside from needing
rebase).
Per-CF TTL
--
Key: CASSANDRA-3974
URL: https://issues.apache.org/jira/browse/CASSANDRA-3974
Project: Cassandra
Issue Type: New Feature
Affects Versions: 1.2.0 beta 1
Reporter: Jonathan Ellis
Assignee: Kirk True
Priority: Minor
Fix For: 1.2.0 rc1
Attachments: trunk-3974.txt, trunk-3974v2.txt, trunk-3974v3.txt,
trunk-3974v4.txt, trunk-3974v5.txt, trunk-3974v6.txt
Per-CF TTL would allow compaction optimizations (drop an entire sstable's
worth of expired data) that we can't do with per-column.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4550) nodetool ring output should use hex not integers for tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494147#comment-13494147 ] Aaron Turner commented on CASSANDRA-4550: - I'm going to remove my support for this request since it seems to be more pain then it's worth. nodetool ring output should use hex not integers for tokens --- Key: CASSANDRA-4550 URL: https://issues.apache.org/jira/browse/CASSANDRA-4550 Project: Cassandra Issue Type: Improvement Components: Tools Environment: Linux Reporter: Aaron Turner Assignee: Kirk True Priority: Trivial Labels: lhf Attachments: trunk-4550.txt The current output of nodetool ring prints start token values as base10 integers instead of hex. This is not very user friendly for a number of reasons: 1. Hides the fact that the values are 128bit 2. Values are not of a consistent length, while in hex padding with zero is generally accepted 3. When using the default random partitioner, having the values in hex makes it easier for users to determine which node(s) a given key resides on since md5 utilities like md5sum output hex. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4939) Add debug logging to list filenames processed by o.a.c.db.Directories.migrateFile method
J.B. Langston created CASSANDRA-4939: Summary: Add debug logging to list filenames processed by o.a.c.db.Directories.migrateFile method Key: CASSANDRA-4939 URL: https://issues.apache.org/jira/browse/CASSANDRA-4939 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.6 Reporter: J.B. Langston Priority: Minor Customer is getting the following error when starting Cassandra: ERROR 20:20:06,635 Exception encountered during startup java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(Unknown Source) at org.apache.cassandra.db.Directories.migrateFile(Directories.java:556) at org.apache.cassandra.db.Directories.migrateSSTables(Directories.java:493) It looks like this is caused by an file with an unexpected name in one of his keyspace directories. However, because we don't log the name of the file as it is processed, it is hard to tell which file is causing it to choke. It would be good to add a logger.debug statement at the beginning of the method to list the file currently being processed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494166#comment-13494166 ] Jonathan Ellis commented on CASSANDRA-4937: --- bq. we need to minimize allocation rate that compression involves How much memory do you see CM using, prior to this change? bq. RAR skip cache harms performance in situation when reads are done in parallel with compaction But it's still the case that we absolutely want to keep compaction of cold data from thrashing the cache. What if we used the read metrics we collect to decide whether to skip cache instead of turning it off entirely? Feedback on the code: - I'm not sure how much can CRAR objectCache help if we trash the queue on reference release. This would be more useful if we (1) made it per-SSTR instead of global and (2) allowed CRAR to be used by more than one request. Either way, let's split this out to a separate patch -- think this should be 1.2-only. - CM.close is added but I don't see it called anywhere - If we're going to move CompressionMetadata off-heap, why not use a single Memory object instead of BLA? This should also be 1.2. ISTM this needs to split into three patches -- objectCache, off-heap CM, and cache skipping. CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494192#comment-13494192
]
Jason Brown commented on CASSANDRA-4924:
Hi Jonathan,
I see your point about not letting 2012 Hector (or astyanax :)) do the wrong
thing and bork any CQL3 data set it thinks it's safely using. I was thinking
skirting around the issue and exposing it thrift was OK, but now, thinking more
about it, I'm less convinced of that. Having the CLI be able to have access to
CQL3 data, however, I think is the safest route as far as a transition to CQL3.
I checked out your branch, and was able to see the testcompact table in the
cli, but not the testsparse table.
-Jason
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4934) assertion error in offheap bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494195#comment-13494195
]
Vijay commented on CASSANDRA-4934:
--
Hi Jonathan, If the above senario happens again then the code
{code}
while (true)
{
SetSSTableReader overlapped = getOverlappingSSTables(sstables);
if (SSTableReader.acquireReferences(overlapped))
return overlapped;
}
{code}
Wont we run into a spin lock because the aquireReference code will always
return false always?
Will it be better to just cancel the compaction and let the next catchup?
Honestly, i did think about this somehow missed the overlapping SST code path
earlier :)
assertion error in offheap bloom filter
---
Key: CASSANDRA-4934
URL: https://issues.apache.org/jira/browse/CASSANDRA-4934
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 2
Reporter: Brandon Williams
Assignee: Jonathan Ellis
Fix For: 1.2.0 rc1
Attachments: log.txt
Saw this while running the dtests:
{noformat}
INFO [CompactionExecutor:2] 2012-11-08 09:35:18,206 CompactionTask.java
(line 116) Compacting
[SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-9-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-11-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-10-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-13-Data.db')]
ERROR [CompactionExecutor:2] 2012-11-08 09:35:18,257 CassandraDaemon.java
(line 132) Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:134)
at org.apache.cassandra.io.util.Memory.getByte(Memory.java:104)
at org.apache.cassandra.utils.obs.OffHeapBitSet.get(OffHeapBitSet.java:60)
at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:71)
at
org.apache.cassandra.db.compaction.CompactionController.shouldPurge(CompactionController.java:106)
at
org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:64)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:95)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:151)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$8.computeNext(Iterators.java:734)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:179)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4934) assertion error in offheap bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494195#comment-13494195
]
Vijay edited comment on CASSANDRA-4934 at 11/9/12 6:43 PM:
---
+1 (ignore my first comment getOverlappingSSTables will return a diffrent set).
Honestly, i did think about this somehow missed the overlapping SST code path
earlier :)
was (Author: vijay2...@yahoo.com):
Hi Jonathan, If the above senario happens again then the code
{code}
while (true)
{
SetSSTableReader overlapped = getOverlappingSSTables(sstables);
if (SSTableReader.acquireReferences(overlapped))
return overlapped;
}
{code}
Wont we run into a spin lock because the aquireReference code will always
return false always?
Will it be better to just cancel the compaction and let the next catchup?
Honestly, i did think about this somehow missed the overlapping SST code path
earlier :)
assertion error in offheap bloom filter
---
Key: CASSANDRA-4934
URL: https://issues.apache.org/jira/browse/CASSANDRA-4934
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 2
Reporter: Brandon Williams
Assignee: Jonathan Ellis
Fix For: 1.2.0 rc1
Attachments: log.txt
Saw this while running the dtests:
{noformat}
INFO [CompactionExecutor:2] 2012-11-08 09:35:18,206 CompactionTask.java
(line 116) Compacting
[SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-9-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-11-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-10-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-13-Data.db')]
ERROR [CompactionExecutor:2] 2012-11-08 09:35:18,257 CassandraDaemon.java
(line 132) Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:134)
at org.apache.cassandra.io.util.Memory.getByte(Memory.java:104)
at org.apache.cassandra.utils.obs.OffHeapBitSet.get(OffHeapBitSet.java:60)
at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:71)
at
org.apache.cassandra.db.compaction.CompactionController.shouldPurge(CompactionController.java:106)
at
org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:64)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:95)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:151)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$8.computeNext(Iterators.java:734)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:179)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494210#comment-13494210
]
Jonathan Ellis commented on CASSANDRA-4924:
---
Odd, can you try with --debug on the cli?
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494212#comment-13494212 ] Pavel Yaskevich commented on CASSANDRA-4937: bq. How much memory do you see CM using, prior to this change? bq. I'm not sure how much can CRAR objectCache help if we trash the queue on reference release. This would be more useful if we (1) made it per-SSTR instead of global and (2) allowed CRAR to be used by more than one request. Either way, let's split this out to a separate patch – think this should be 1.2-only. It depends on the data size per-SSTable, but on each 1GB overhead is about 20MB, which would definitely be promoted to old gen and kept there. That is not the biggest problem, the problem with compression allocation rate is that we are opening one CRAR per row read (CompressedSegmentedFile.getSegment) which allocates 128KB (64KB for compression/decompression buffers) plus has additional small memory overhead of checksum buffer and fields, that is what objectCache here to solve because it was figured that even when up to 12 SSTables are involved per read we wouldn't have more then concurrent_reads CRARs per-SSTable in the cache. bq. RAR skip cache harms performance in situation when reads are done in parallel with compaction The problem with the code we have right now is that it doesn't actually skip blocks that compaction read, it drops *whole* file page cache after fixed internal (128MB) so when you have long running compactions (trottled or big data files, for example) normal reads would hit the cold data very frequently. The only thing is working correctly right now in terms of skiping cache is SW. I have done some benchmarks with and without skiping cache and it shows that page replacement done by kernel is much better then our skip suggestions via recent read latency histograms. bq. CM.close is added but I don't see it called anywhere It should be called on CRAR.deallocate(), sorry, I must have missed that when I was merging, I will fix that and update the patch. bq. If we're going to move CompressionMetadata off-heap, why not use a single Memory object instead of BLA? This should also be 1.2. Yes, that could be done as a single object, I just didn't want to remove historical paging change especially actual overhead of paging is very-very low :) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4936) Less than operator when comparing timeuuids behaves as less than equal.
[ https://issues.apache.org/jira/browse/CASSANDRA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494229#comment-13494229 ] Cesar Lopez-Nataren commented on CASSANDRA-4936: Jonathan: What's the easiest way to find that out? Less than operator when comparing timeuuids behaves as less than equal. --- Key: CASSANDRA-4936 URL: https://issues.apache.org/jira/browse/CASSANDRA-4936 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.6 Environment: Linux CentOS. Linux localhost.localdomain 2.6.18-308.16.1.el5 #1 SMP Tue Oct 2 22:01:37 EDT 2012 i686 i686 i386 GNU/Linux Reporter: Cesar Lopez-Nataren If we define the following column family using CQL3: CREATE TABLE useractivity ( user_id int, activity_id 'TimeUUIDType', data text, PRIMARY KEY (user_id, activity_id) ); Add some values to it. And then query it like: SELECT * FROM useractivity WHERE user_id = '3' AND activity_id '2012-11-07 18:18:22-0800' ORDER BY activity_id DESC LIMIT 1; the record with timeuuid '2012-11-07 18:18:22-0800' returns in the results. According to the documentation, on CQL3 the '' and '' operators are strict, meaning not inclusive, so this seems to be a bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494248#comment-13494248
]
Edward Capriolo commented on CASSANDRA-4924:
{quote}
While I still think it's wrongheaded to try to jam CQL3 into an old Hector app,
it is reasonable to add CQL3 introspection to the CLI on GET. (SET already
works.)
{quote}
I need to understand how the table is structured internally so I can deal with
it. Sure there are docs out there that explain the internals, but asking the
user to read the docs and reverse engineer the CQL table to a thrift table is
annoying and error prone. I agree that having half the information is bad in
ways, but having no information is much worse.
I am not trying to be wrongheaded, by jamming CQL3 into hector/thrift. In fact
I am trying to understand CQL3 more deeply. This learning and transition is not
aided by the missing CFDef. Even if semi-complete it is better then nothing.
Also we did say we are going to support thrift clients, and not providing the
old style schema is not much support. Even if I go for CQL3 100% I may have
some legacy apps that may need to use hector thrift for some time.
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494271#comment-13494271
]
Edward Capriolo commented on CASSANDRA-4924:
{quote}
I see your point about not letting 2012 Hector (or astyanax ) do the wrong
thing and bork any CQL3 data set it thinks it's safely using. I was thinking
skirting around the issue and exposing it thrift was OK, but now, thinking more
about it, I'm less convinced of that. Having the CLI be able to have access to
CQL3 data, however, I think is the safest route as far as a transition to CQL3.
{quote}
http://www.datastax.com/dev/blog/thrift-to-cql3
{quote}
But the thrift API is not going anywhere. Existing applications do not have to
upgrade to CQL3.
{quote}
We don't have to upgrade, but without some schema hints it becomes increasingly
more difficult to work with. This is more like a meet-me-half-way deal. To use
thrift I need some clue as to what the thrift schema looks like. validation
exceptions will help me with the rest.
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4940) Truncate doesn't clear row cache
Jeremiah Jordan created CASSANDRA-4940: -- Summary: Truncate doesn't clear row cache Key: CASSANDRA-4940 URL: https://issues.apache.org/jira/browse/CASSANDRA-4940 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.6 Reporter: Jeremiah Jordan Truncate doesn't clear the row cache. select * from table which skips the row cache returns no data, but selecting by key does. cqlsh:temp select v1..v3 from temp2 where k in (3,2,1); v1 | v2 | v3 ++ 16 | 17 | 18 12 | 13 | 14 8 | 9 | 10 cqlsh:temp truncate temp2; cqlsh:temp select v1..v3 from temp2 where k in (3,2,1); v1 | v2 | v3 ++ 16 | 17 | 18 12 | 13 | 14 8 | 9 | 10 cqlsh:temp select * from temp2; cqlsh:temp select v1..v3 from temp2 where k in (3,2,1); v1 | v2 | v3 ++ 16 | 17 | 18 12 | 13 | 14 8 | 9 | 10 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4813) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node.
[ https://issues.apache.org/jira/browse/CASSANDRA-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494315#comment-13494315 ] Yuki Morishita commented on CASSANDRA-4813: --- [~mkjellman] Just checking, did you use patched version for both Cassandra and hadoop job? Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node. - Key: CASSANDRA-4813 URL: https://issues.apache.org/jira/browse/CASSANDRA-4813 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Environment: I am using SLES 10 SP3, Java 6, 4 Cassandra + Hadoop nodes, 3 Hadoop only nodes (datanodes/tasktrackers), 1 namenode/jobtracker. The machines used are Six-Core AMD Opteron(tm) Processor 8431, 24 cores and 33 GB of RAM. I get the issue on both cassandra 1.1.3, 1.1.5 and I am using Hadoop 0.20.2. Reporter: Ralph Romanos Assignee: Yuki Morishita Priority: Minor Labels: Bulkoutputformat, Hadoop, SSTables Fix For: 1.2.0 rc1 Attachments: 4813.txt The issue occurs when streaming simultaneously SSTables from the same node to a cassandra cluster using SSTableloader. It seems to me that Cassandra cannot handle receiving simultaneously SSTables from the same node. However, when it receives simultaneously SSTables from two different nodes, everything works fine. As a consequence, when using BulkOutputFormat to generate SSTables and stream them to a cassandra cluster, I cannot use more than one reducer per node otherwise I get a java.io.EOFException in the tasktracker's logs and a java.io.IOException: Broken pipe in the Cassandra logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[3/6] git commit: acquire references on overlapped sstables patch by jbellis; reviewed by vijay for CASSANDRA-4934
acquire references on overlapped sstables
patch by jbellis; reviewed by vijay for CASSANDRA-4934
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f09a89f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f09a89f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f09a89f4
Branch: refs/heads/cassandra-1.2.0
Commit: f09a89f4cd13af2087fcc92f09f6cf1ee4785feb
Parents: e0a0eb7
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 10:06:18 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 15:17:11 2012 -0600
--
CHANGES.txt|3 +++
.../org/apache/cassandra/db/ColumnFamilyStore.java | 13 +
.../db/compaction/CompactionController.java| 14 +++---
.../cassandra/db/compaction/CompactionManager.java |1 +
.../cassandra/db/compaction/CompactionTask.java|2 ++
.../apache/cassandra/db/compaction/Scrubber.java |4
.../cassandra/streaming/IncomingStreamReader.java |4
7 files changed, 38 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba9e134..3b80885 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,9 +1,12 @@
1.2-rc1
+ * acquire references to overlapping sstables during compaction so bloom filter
+ doesn't get free'd prematurely (CASSANDRA-4934)
* Don't share slice query filter in CQL3 SelectStatement (CASSANDRA-4928)
* Separate tracing from Log4J (CASSANDRA-4861)
* Exclude gcable tombstones from merkle-tree computation (CASSANDRA-4905)
* Better printing of AbstractBounds for tracing (CASSANDRA-4931)
+
1.2-beta2
* fp rate of 1.0 disables BF entirely; LCS defaults to 1.0 (CASSANDRA-4876)
* off-heap bloom filters for row keys (CASSANDRA_4865)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 882a322..a91af8c 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -889,6 +889,19 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
return results;
}
+/**
+ * like getOverlappingSSTables, but acquires references before returning
+ */
+public SetSSTableReader
getAndReferenceOverlappingSSTables(CollectionSSTableReader sstables)
+{
+while (true)
+{
+SetSSTableReader overlapped = getOverlappingSSTables(sstables);
+if (SSTableReader.acquireReferences(overlapped))
+return overlapped;
+}
+}
+
/*
* Called after a BinaryMemtable flushes its in-memory data, or we add a
file
* via bootstrap. This information is cached in the ColumnFamilyStore.
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 3b04833..2c8ddba 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.db.compaction;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
+import java.util.Set;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,6 +45,7 @@ public class CompactionController
public final ColumnFamilyStore cfs;
private final DataTracker.SSTableIntervalTree overlappingTree;
+private final SetSSTableReader overlappingSSTables;
public final int gcBefore;
public final int mergeShardBefore;
@@ -66,7 +68,7 @@ public class CompactionController
{
this(cfs,
gcBefore,
-
DataTracker.buildIntervalTree(cfs.getOverlappingSSTables(sstables)));
+ cfs.getAndReferenceOverlappingSSTables(sstables));
}
/**
@@ -79,7 +81,7 @@ public class CompactionController
private CompactionController(ColumnFamilyStore cfs,
int gcBefore,
- DataTracker.SSTableIntervalTree
overlappingTree)
+ SetSSTableReader overlappingSSTables)
{
[5/6] git commit: cleanup
cleanup
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e0a0eb75
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e0a0eb75
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e0a0eb75
Branch: refs/heads/trunk
Commit: e0a0eb751f01b3deca3a1d02e77913b222453e3a
Parents: 03f8fec
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 09:52:21 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 09:52:21 2012 -0600
--
.../db/compaction/CompactionController.java| 10 +-
.../cassandra/db/compaction/CompactionManager.java |2 +-
.../apache/cassandra/db/compaction/Scrubber.java |2 +-
3 files changed, 11 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 1a9e08c..3b04833 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -69,7 +69,15 @@ public class CompactionController
DataTracker.buildIntervalTree(cfs.getOverlappingSSTables(sstables)));
}
-protected CompactionController(ColumnFamilyStore cfs,
+/**
+ * Constructor that subclasses may use when overriding shouldPurge to not
need overlappingTree
+ */
+protected CompactionController(ColumnFamilyStore cfs, int maxValue)
+{
+this(cfs, maxValue, null);
+}
+
+private CompactionController(ColumnFamilyStore cfs,
int gcBefore,
DataTracker.SSTableIntervalTree
overlappingTree)
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a93f392..4f16f6a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -900,7 +900,7 @@ public class CompactionManager implements
CompactionManagerMBean
{
public ValidationCompactionController(ColumnFamilyStore cfs, int
gcBefore)
{
-super(cfs, gcBefore, null);
+super(cfs, gcBefore);
}
@Override
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/Scrubber.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
index ec99059..3946c37 100644
--- a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
+++ b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
@@ -345,7 +345,7 @@ public class Scrubber implements Closeable
{
public ScrubController(ColumnFamilyStore cfs)
{
-super(cfs, Integer.MAX_VALUE,
DataTracker.buildIntervalTree(Collections.SSTableReaderemptyList()));
+super(cfs, Integer.MAX_VALUE);
}
@Override
[1/6] git commit: acquire references on overlapped sstables patch by jbellis; reviewed by vijay for CASSANDRA-4934
Updated Branches:
refs/heads/cassandra-1.2 03f8fec8c - f09a89f4c
refs/heads/cassandra-1.2.0 03f8fec8c - f09a89f4c
refs/heads/trunk 03f8fec8c - f09a89f4c
acquire references on overlapped sstables
patch by jbellis; reviewed by vijay for CASSANDRA-4934
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f09a89f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f09a89f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f09a89f4
Branch: refs/heads/trunk
Commit: f09a89f4cd13af2087fcc92f09f6cf1ee4785feb
Parents: e0a0eb7
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 10:06:18 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 15:17:11 2012 -0600
--
CHANGES.txt|3 +++
.../org/apache/cassandra/db/ColumnFamilyStore.java | 13 +
.../db/compaction/CompactionController.java| 14 +++---
.../cassandra/db/compaction/CompactionManager.java |1 +
.../cassandra/db/compaction/CompactionTask.java|2 ++
.../apache/cassandra/db/compaction/Scrubber.java |4
.../cassandra/streaming/IncomingStreamReader.java |4
7 files changed, 38 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba9e134..3b80885 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,9 +1,12 @@
1.2-rc1
+ * acquire references to overlapping sstables during compaction so bloom filter
+ doesn't get free'd prematurely (CASSANDRA-4934)
* Don't share slice query filter in CQL3 SelectStatement (CASSANDRA-4928)
* Separate tracing from Log4J (CASSANDRA-4861)
* Exclude gcable tombstones from merkle-tree computation (CASSANDRA-4905)
* Better printing of AbstractBounds for tracing (CASSANDRA-4931)
+
1.2-beta2
* fp rate of 1.0 disables BF entirely; LCS defaults to 1.0 (CASSANDRA-4876)
* off-heap bloom filters for row keys (CASSANDRA_4865)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 882a322..a91af8c 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -889,6 +889,19 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
return results;
}
+/**
+ * like getOverlappingSSTables, but acquires references before returning
+ */
+public SetSSTableReader
getAndReferenceOverlappingSSTables(CollectionSSTableReader sstables)
+{
+while (true)
+{
+SetSSTableReader overlapped = getOverlappingSSTables(sstables);
+if (SSTableReader.acquireReferences(overlapped))
+return overlapped;
+}
+}
+
/*
* Called after a BinaryMemtable flushes its in-memory data, or we add a
file
* via bootstrap. This information is cached in the ColumnFamilyStore.
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 3b04833..2c8ddba 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.db.compaction;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
+import java.util.Set;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,6 +45,7 @@ public class CompactionController
public final ColumnFamilyStore cfs;
private final DataTracker.SSTableIntervalTree overlappingTree;
+private final SetSSTableReader overlappingSSTables;
public final int gcBefore;
public final int mergeShardBefore;
@@ -66,7 +68,7 @@ public class CompactionController
{
this(cfs,
gcBefore,
-
DataTracker.buildIntervalTree(cfs.getOverlappingSSTables(sstables)));
+ cfs.getAndReferenceOverlappingSSTables(sstables));
}
/**
@@ -79,7 +81,7 @@ public class CompactionController
private CompactionController(ColumnFamilyStore cfs,
int gcBefore,
-
[2/6] git commit: acquire references on overlapped sstables patch by jbellis; reviewed by vijay for CASSANDRA-4934
acquire references on overlapped sstables
patch by jbellis; reviewed by vijay for CASSANDRA-4934
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f09a89f4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f09a89f4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f09a89f4
Branch: refs/heads/cassandra-1.2
Commit: f09a89f4cd13af2087fcc92f09f6cf1ee4785feb
Parents: e0a0eb7
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 10:06:18 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 15:17:11 2012 -0600
--
CHANGES.txt|3 +++
.../org/apache/cassandra/db/ColumnFamilyStore.java | 13 +
.../db/compaction/CompactionController.java| 14 +++---
.../cassandra/db/compaction/CompactionManager.java |1 +
.../cassandra/db/compaction/CompactionTask.java|2 ++
.../apache/cassandra/db/compaction/Scrubber.java |4
.../cassandra/streaming/IncomingStreamReader.java |4
7 files changed, 38 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba9e134..3b80885 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,9 +1,12 @@
1.2-rc1
+ * acquire references to overlapping sstables during compaction so bloom filter
+ doesn't get free'd prematurely (CASSANDRA-4934)
* Don't share slice query filter in CQL3 SelectStatement (CASSANDRA-4928)
* Separate tracing from Log4J (CASSANDRA-4861)
* Exclude gcable tombstones from merkle-tree computation (CASSANDRA-4905)
* Better printing of AbstractBounds for tracing (CASSANDRA-4931)
+
1.2-beta2
* fp rate of 1.0 disables BF entirely; LCS defaults to 1.0 (CASSANDRA-4876)
* off-heap bloom filters for row keys (CASSANDRA_4865)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 882a322..a91af8c 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -889,6 +889,19 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
return results;
}
+/**
+ * like getOverlappingSSTables, but acquires references before returning
+ */
+public SetSSTableReader
getAndReferenceOverlappingSSTables(CollectionSSTableReader sstables)
+{
+while (true)
+{
+SetSSTableReader overlapped = getOverlappingSSTables(sstables);
+if (SSTableReader.acquireReferences(overlapped))
+return overlapped;
+}
+}
+
/*
* Called after a BinaryMemtable flushes its in-memory data, or we add a
file
* via bootstrap. This information is cached in the ColumnFamilyStore.
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f09a89f4/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 3b04833..2c8ddba 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.db.compaction;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
+import java.util.Set;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,6 +45,7 @@ public class CompactionController
public final ColumnFamilyStore cfs;
private final DataTracker.SSTableIntervalTree overlappingTree;
+private final SetSSTableReader overlappingSSTables;
public final int gcBefore;
public final int mergeShardBefore;
@@ -66,7 +68,7 @@ public class CompactionController
{
this(cfs,
gcBefore,
-
DataTracker.buildIntervalTree(cfs.getOverlappingSSTables(sstables)));
+ cfs.getAndReferenceOverlappingSSTables(sstables));
}
/**
@@ -79,7 +81,7 @@ public class CompactionController
private CompactionController(ColumnFamilyStore cfs,
int gcBefore,
- DataTracker.SSTableIntervalTree
overlappingTree)
+ SetSSTableReader overlappingSSTables)
{
[4/6] git commit: cleanup
cleanup
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e0a0eb75
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e0a0eb75
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e0a0eb75
Branch: refs/heads/cassandra-1.2.0
Commit: e0a0eb751f01b3deca3a1d02e77913b222453e3a
Parents: 03f8fec
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 09:52:21 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 09:52:21 2012 -0600
--
.../db/compaction/CompactionController.java| 10 +-
.../cassandra/db/compaction/CompactionManager.java |2 +-
.../apache/cassandra/db/compaction/Scrubber.java |2 +-
3 files changed, 11 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 1a9e08c..3b04833 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -69,7 +69,15 @@ public class CompactionController
DataTracker.buildIntervalTree(cfs.getOverlappingSSTables(sstables)));
}
-protected CompactionController(ColumnFamilyStore cfs,
+/**
+ * Constructor that subclasses may use when overriding shouldPurge to not
need overlappingTree
+ */
+protected CompactionController(ColumnFamilyStore cfs, int maxValue)
+{
+this(cfs, maxValue, null);
+}
+
+private CompactionController(ColumnFamilyStore cfs,
int gcBefore,
DataTracker.SSTableIntervalTree
overlappingTree)
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a93f392..4f16f6a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -900,7 +900,7 @@ public class CompactionManager implements
CompactionManagerMBean
{
public ValidationCompactionController(ColumnFamilyStore cfs, int
gcBefore)
{
-super(cfs, gcBefore, null);
+super(cfs, gcBefore);
}
@Override
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/Scrubber.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
index ec99059..3946c37 100644
--- a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
+++ b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
@@ -345,7 +345,7 @@ public class Scrubber implements Closeable
{
public ScrubController(ColumnFamilyStore cfs)
{
-super(cfs, Integer.MAX_VALUE,
DataTracker.buildIntervalTree(Collections.SSTableReaderemptyList()));
+super(cfs, Integer.MAX_VALUE);
}
@Override
[6/6] git commit: cleanup
cleanup
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e0a0eb75
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e0a0eb75
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e0a0eb75
Branch: refs/heads/cassandra-1.2
Commit: e0a0eb751f01b3deca3a1d02e77913b222453e3a
Parents: 03f8fec
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 09:52:21 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 09:52:21 2012 -0600
--
.../db/compaction/CompactionController.java| 10 +-
.../cassandra/db/compaction/CompactionManager.java |2 +-
.../apache/cassandra/db/compaction/Scrubber.java |2 +-
3 files changed, 11 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index 1a9e08c..3b04833 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -69,7 +69,15 @@ public class CompactionController
DataTracker.buildIntervalTree(cfs.getOverlappingSSTables(sstables)));
}
-protected CompactionController(ColumnFamilyStore cfs,
+/**
+ * Constructor that subclasses may use when overriding shouldPurge to not
need overlappingTree
+ */
+protected CompactionController(ColumnFamilyStore cfs, int maxValue)
+{
+this(cfs, maxValue, null);
+}
+
+private CompactionController(ColumnFamilyStore cfs,
int gcBefore,
DataTracker.SSTableIntervalTree
overlappingTree)
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index a93f392..4f16f6a 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -900,7 +900,7 @@ public class CompactionManager implements
CompactionManagerMBean
{
public ValidationCompactionController(ColumnFamilyStore cfs, int
gcBefore)
{
-super(cfs, gcBefore, null);
+super(cfs, gcBefore);
}
@Override
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0a0eb75/src/java/org/apache/cassandra/db/compaction/Scrubber.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
index ec99059..3946c37 100644
--- a/src/java/org/apache/cassandra/db/compaction/Scrubber.java
+++ b/src/java/org/apache/cassandra/db/compaction/Scrubber.java
@@ -345,7 +345,7 @@ public class Scrubber implements Closeable
{
public ScrubController(ColumnFamilyStore cfs)
{
-super(cfs, Integer.MAX_VALUE,
DataTracker.buildIntervalTree(Collections.SSTableReaderemptyList()));
+super(cfs, Integer.MAX_VALUE);
}
@Override
[jira] [Resolved] (CASSANDRA-4934) assertion error in offheap bloom filter
[
https://issues.apache.org/jira/browse/CASSANDRA-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-4934.
---
Resolution: Fixed
committed
assertion error in offheap bloom filter
---
Key: CASSANDRA-4934
URL: https://issues.apache.org/jira/browse/CASSANDRA-4934
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0 beta 2
Reporter: Brandon Williams
Assignee: Jonathan Ellis
Fix For: 1.2.0 rc1
Attachments: log.txt
Saw this while running the dtests:
{noformat}
INFO [CompactionExecutor:2] 2012-11-08 09:35:18,206 CompactionTask.java
(line 116) Compacting
[SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-9-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-11-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-10-Data.db'),
SSTableReader(path='/tmp/dtest-n2D_fM/test/node2/data/system/local/system-local-ia-13-Data.db')]
ERROR [CompactionExecutor:2] 2012-11-08 09:35:18,257 CassandraDaemon.java
(line 132) Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:134)
at org.apache.cassandra.io.util.Memory.getByte(Memory.java:104)
at org.apache.cassandra.utils.obs.OffHeapBitSet.get(OffHeapBitSet.java:60)
at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:71)
at
org.apache.cassandra.db.compaction.CompactionController.shouldPurge(CompactionController.java:106)
at
org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:64)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:95)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:151)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$8.computeNext(Iterators.java:734)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:179)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4813) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node.
[ https://issues.apache.org/jira/browse/CASSANDRA-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494317#comment-13494317 ] Michael Kjellman commented on CASSANDRA-4813: - yes, actually the first time i didn't and then i realized i hadn't rebuilt the Cassandra nodes and only replaced the compiled jar in my hadoop job. I've tested a few times now just to make sure i'm not missing anything. Seems to die as soon as it tries to stream the first sstable to the nodes. Never progresses past 0% on the streaming and then throws the exception. Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node. - Key: CASSANDRA-4813 URL: https://issues.apache.org/jira/browse/CASSANDRA-4813 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Environment: I am using SLES 10 SP3, Java 6, 4 Cassandra + Hadoop nodes, 3 Hadoop only nodes (datanodes/tasktrackers), 1 namenode/jobtracker. The machines used are Six-Core AMD Opteron(tm) Processor 8431, 24 cores and 33 GB of RAM. I get the issue on both cassandra 1.1.3, 1.1.5 and I am using Hadoop 0.20.2. Reporter: Ralph Romanos Assignee: Yuki Morishita Priority: Minor Labels: Bulkoutputformat, Hadoop, SSTables Fix For: 1.2.0 rc1 Attachments: 4813.txt The issue occurs when streaming simultaneously SSTables from the same node to a cassandra cluster using SSTableloader. It seems to me that Cassandra cannot handle receiving simultaneously SSTables from the same node. However, when it receives simultaneously SSTables from two different nodes, everything works fine. As a consequence, when using BulkOutputFormat to generate SSTables and stream them to a cassandra cluster, I cannot use more than one reducer per node otherwise I get a java.io.EOFException in the tasktracker's logs and a java.io.IOException: Broken pipe in the Cassandra logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-4937: --- Attachment: CASSANDRA-4937.patch updated the patch with missing CRAR metadata.close() CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3974) Per-CF TTL
[ https://issues.apache.org/jira/browse/CASSANDRA-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494329#comment-13494329 ] Jonathan Ellis commented on CASSANDRA-3974: --- bq. it does feel a bit fragile that some future internal code could too easily add an ExpiringColumn though ColumnFamily.addColumn(IColumn) Tracing and unit tests (via Util.expiringColumn) already use this method. Should we make the constructors private and expose a factory method that requires the metadata? Per-CF TTL -- Key: CASSANDRA-3974 URL: https://issues.apache.org/jira/browse/CASSANDRA-3974 Project: Cassandra Issue Type: New Feature Affects Versions: 1.2.0 beta 1 Reporter: Jonathan Ellis Assignee: Kirk True Priority: Minor Fix For: 1.2.0 rc1 Attachments: trunk-3974.txt, trunk-3974v2.txt, trunk-3974v3.txt, trunk-3974v4.txt, trunk-3974v5.txt, trunk-3974v6.txt Per-CF TTL would allow compaction optimizations (drop an entire sstable's worth of expired data) that we can't do with per-column. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-4937: --- Attachment: (was: CASSANDRA-4937.patch) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
buildbot failure in ASF Buildbot on cassandra-trunk
The Buildbot has detected a new failure on builder cassandra-trunk while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/2065 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch trunk] f09a89f4cd13af2087fcc92f09f6cf1ee4785feb Blamelist: Jonathan Ellis jbel...@apache.org BUILD FAILED: failed shell sincerely, -The Buildbot
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494331#comment-13494331
]
Jason Brown commented on CASSANDRA-4924:
[~appodictic] Yes, I read that last week. By 'safest route' I meant how can we
see what this new thing is doing so I can (more or less) compare it with my
expectations about the old thing. I think once experienced c* folks get the
basic hang of the differences from thrift/CF model to CQL3, the need for the
intermediate 'cql3 to thrift' conversation may wane quickly. Note: I said
'may'; I'm still thinking about this.
If I can slightly reframe what Jonathan said, I think it's that existing apps
would only need to upgrade when they want to use CQL3 tables (or when thrift is
finally retired). Apps that do want to take advantage of CQL3 tables would need
to updated, whether that's getting a client that supports cql3 syntax or using
the updated protocol. I agree there's a rub when there's a mixed mode of
thrift-style CFs and cql3 tables in a keyspace or cluster, but I wonder if
that's a documentation issue, as Sylvain alluded to earlier. Of course, how
best to communicate that is up for grabs.
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4935) occasional TableTest failure
[
https://issues.apache.org/jira/browse/CASSANDRA-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuki Morishita updated CASSANDRA-4935:
--
Attachment: 4935.txt
Alright, I removed forceBlockingFlush and set TTL long enough to not get
tombstoned so we get consistent test result.
occasional TableTest failure
Key: CASSANDRA-4935
URL: https://issues.apache.org/jira/browse/CASSANDRA-4935
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Assignee: Yuki Morishita
Priority: Minor
Fix For: 1.2.0 rc1
Attachments: 4935.txt
The TableTest unit test fails somewhat rarely:
{noformat}
[junit] Testsuite: org.apache.cassandra.db.TableTest
[junit] Tests run: 11, Failures: 1, Errors: 0, Time elapsed: 4.339 sec
[junit]
[junit] Testcase:
testGetSliceWithExpiration(org.apache.cassandra.db.TableTest): FAILED
[junit] Columns
[636f6c31:false:4@1,636f6c32:true:4@1!1,636f6c33:false:4@1,])] is not
expected [col1,col2]
[junit] junit.framework.AssertionFailedError: Columns
[636f6c31:false:4@1,636f6c32:true:4@1!1,636f6c33:false:4@1,])] is not
expected [col1,col2]
[junit] at
org.apache.cassandra.db.TableTest.assertColumns(TableTest.java:574)
[junit] at
org.apache.cassandra.db.TableTest.assertColumns(TableTest.java:541)
[junit] at
org.apache.cassandra.db.TableTest$5.runMayThrow(TableTest.java:353)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at org.apache.cassandra.db.TableTest.reTest(TableTest.java:56)
[junit] at
org.apache.cassandra.db.TableTest.testGetSliceWithExpiration(TableTest.java:362)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494346#comment-13494346
]
Jason Brown commented on CASSANDRA-4924:
[~jbellis]Tried again with --debug, but no additional output.
{code}lgmac-jasbrown:jbellis-cassandra $ bin/cassandra-cli --debug
Connected to: Test Cluster on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.2.0-beta1-SNAPSHOT
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown] use jeb;
Authenticated to keyspace: jeb
[default@jeb] get testsparse['k1'];
testsparse not found in current keyspace.
{code}
The xyz not found in current keyspace gets printed from
CliCompiler.getColumnFamily() - will poke around a little more.
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494347#comment-13494347
]
Jason Brown commented on CASSANDRA-4924:
FWIW, CliSessionState.session state is only referenced in
CliClient.executeUseKeySpace() when a TException is caught :).
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[3/3] git commit: Fix rare race condition in getExpireTimeForEndpoint; document the behavior of a few other apparently-raceful Map calls patch by Yu Lin and jbellis; reviewed by slebresne for CASSANDR
Fix rare race condition in getExpireTimeForEndpoint; document the behavior of a
few other apparently-raceful Map calls
patch by Yu Lin and jbellis; reviewed by slebresne for CASSANDRA-4402
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f5e3ae67
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f5e3ae67
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f5e3ae67
Branch: refs/heads/cassandra-1.2
Commit: f5e3ae67c4a4fef6a541a8a06a9871810639a7ec
Parents: f09a89f
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 15:53:52 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:04:45 2012 -0600
--
CHANGES.txt |1 +
src/java/org/apache/cassandra/db/Table.java | 27 ++
src/java/org/apache/cassandra/gms/Gossiper.java | 50 ++---
3 files changed, 36 insertions(+), 42 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5e3ae67/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3b80885..f6769fe 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
doesn't get free'd prematurely (CASSANDRA-4934)
* Don't share slice query filter in CQL3 SelectStatement (CASSANDRA-4928)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5e3ae67/src/java/org/apache/cassandra/db/Table.java
--
diff --git a/src/java/org/apache/cassandra/db/Table.java
b/src/java/org/apache/cassandra/db/Table.java
index 85611de..18b7e4b 100644
--- a/src/java/org/apache/cassandra/db/Table.java
+++ b/src/java/org/apache/cassandra/db/Table.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.locks.ReentrantReadWriteLock;
@@ -71,7 +72,7 @@ public class Table
/* Table name. */
public final String name;
/* ColumnFamilyStore per column family */
-private final MapUUID, ColumnFamilyStore columnFamilyStores = new
ConcurrentHashMapUUID, ColumnFamilyStore();
+private final ConcurrentMapUUID, ColumnFamilyStore columnFamilyStores =
new ConcurrentHashMapUUID, ColumnFamilyStore();
private final Object[] indexLocks;
private volatile AbstractReplicationStrategy replicationStrategy;
@@ -319,19 +320,25 @@ public class Table
*/
public void initCf(UUID cfId, String cfName, boolean loadSSTables)
{
-if (columnFamilyStores.containsKey(cfId))
-{
-// this is the case when you reset local schema
-// just reload metadata
-ColumnFamilyStore cfs = columnFamilyStores.get(cfId);
-assert cfs.getColumnFamilyName().equals(cfName);
+ColumnFamilyStore cfs = columnFamilyStores.get(cfId);
-cfs.metadata.reload();
-cfs.reload();
+if (cfs == null)
+{
+// CFS being created for the first time, either on server startup
or new CF being added.
+// We don't worry about races here; startup is safe, and adding
multiple idential CFs
+// simultaneously is a don't do that scenario.
+ColumnFamilyStore oldCfs = columnFamilyStores.putIfAbsent(cfId,
ColumnFamilyStore.createColumnFamilyStore(this, cfName, loadSSTables));
+// CFS mbean instantiation will error out before we hit this, but
in case that changes...
+if (oldCfs != null)
+throw new IllegalStateException(added multiple mappings for
cf id + cfId);
}
else
{
-columnFamilyStores.put(cfId,
ColumnFamilyStore.createColumnFamilyStore(this, cfName, loadSSTables));
+// re-initializing an existing CF. This will happen if you
cleared the schema
+// on this node and it's getting repopulated from the rest of the
cluster.
+assert cfs.getColumnFamilyName().equals(cfName);
+cfs.metadata.reload();
+cfs.reload();
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5e3ae67/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index e49a6b3..5880210 100644
---
[1/3] git commit: Fix rare race condition in getExpireTimeForEndpoint; document the behavior of a few other apparently-raceful Map calls patch by Yu Lin and jbellis; reviewed by slebresne for CASSANDR
Updated Branches:
refs/heads/cassandra-1.2 f09a89f4c - f5e3ae67c
refs/heads/cassandra-1.2.0 f09a89f4c - f5e3ae67c
refs/heads/trunk f09a89f4c - f5e3ae67c
Fix rare race condition in getExpireTimeForEndpoint; document the behavior of a
few other apparently-raceful Map calls
patch by Yu Lin and jbellis; reviewed by slebresne for CASSANDRA-4402
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f5e3ae67
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f5e3ae67
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f5e3ae67
Branch: refs/heads/trunk
Commit: f5e3ae67c4a4fef6a541a8a06a9871810639a7ec
Parents: f09a89f
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 15:53:52 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:04:45 2012 -0600
--
CHANGES.txt |1 +
src/java/org/apache/cassandra/db/Table.java | 27 ++
src/java/org/apache/cassandra/gms/Gossiper.java | 50 ++---
3 files changed, 36 insertions(+), 42 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5e3ae67/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3b80885..f6769fe 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
doesn't get free'd prematurely (CASSANDRA-4934)
* Don't share slice query filter in CQL3 SelectStatement (CASSANDRA-4928)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5e3ae67/src/java/org/apache/cassandra/db/Table.java
--
diff --git a/src/java/org/apache/cassandra/db/Table.java
b/src/java/org/apache/cassandra/db/Table.java
index 85611de..18b7e4b 100644
--- a/src/java/org/apache/cassandra/db/Table.java
+++ b/src/java/org/apache/cassandra/db/Table.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.locks.ReentrantReadWriteLock;
@@ -71,7 +72,7 @@ public class Table
/* Table name. */
public final String name;
/* ColumnFamilyStore per column family */
-private final MapUUID, ColumnFamilyStore columnFamilyStores = new
ConcurrentHashMapUUID, ColumnFamilyStore();
+private final ConcurrentMapUUID, ColumnFamilyStore columnFamilyStores =
new ConcurrentHashMapUUID, ColumnFamilyStore();
private final Object[] indexLocks;
private volatile AbstractReplicationStrategy replicationStrategy;
@@ -319,19 +320,25 @@ public class Table
*/
public void initCf(UUID cfId, String cfName, boolean loadSSTables)
{
-if (columnFamilyStores.containsKey(cfId))
-{
-// this is the case when you reset local schema
-// just reload metadata
-ColumnFamilyStore cfs = columnFamilyStores.get(cfId);
-assert cfs.getColumnFamilyName().equals(cfName);
+ColumnFamilyStore cfs = columnFamilyStores.get(cfId);
-cfs.metadata.reload();
-cfs.reload();
+if (cfs == null)
+{
+// CFS being created for the first time, either on server startup
or new CF being added.
+// We don't worry about races here; startup is safe, and adding
multiple idential CFs
+// simultaneously is a don't do that scenario.
+ColumnFamilyStore oldCfs = columnFamilyStores.putIfAbsent(cfId,
ColumnFamilyStore.createColumnFamilyStore(this, cfName, loadSSTables));
+// CFS mbean instantiation will error out before we hit this, but
in case that changes...
+if (oldCfs != null)
+throw new IllegalStateException(added multiple mappings for
cf id + cfId);
}
else
{
-columnFamilyStores.put(cfId,
ColumnFamilyStore.createColumnFamilyStore(this, cfName, loadSSTables));
+// re-initializing an existing CF. This will happen if you
cleared the schema
+// on this node and it's getting repopulated from the rest of the
cluster.
+assert cfs.getColumnFamilyName().equals(cfName);
+cfs.metadata.reload();
+cfs.reload();
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5e3ae67/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git
[jira] [Updated] (CASSANDRA-4402) Atomicity violation bugs because of misusing concurrent collections
[
https://issues.apache.org/jira/browse/CASSANDRA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4402:
--
Component/s: Core
Priority: Trivial (was: Major)
Affects Version/s: (was: 1.1.1)
Labels: gossip (was: )
Atomicity violation bugs because of misusing concurrent collections
---
Key: CASSANDRA-4402
URL: https://issues.apache.org/jira/browse/CASSANDRA-4402
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Yu Lin
Assignee: Yu Lin
Priority: Trivial
Labels: gossip
Fix For: 1.2.0
Attachments: 4402-v2.txt, cassandra-1.1.1-4402.txt
Original Estimate: 504h
Remaining Estimate: 504h
My name is Yu Lin. I'm a Ph.D. student in the CS department at
UIUC. I'm currently doing research on mining Java concurrent library
misusages. I found some misusages of ConcurrentHashMap in Cassandra
1.1.1, which may result in potential atomicity violation bugs or harm
the performance.
The code below is a snapshot of the code in file
src/java/org/apache/cassandra/db/Table.java from line 348 to 369
L348if (columnFamilyStores.containsKey(cfId))
L349{
L350// this is the case when you reset local schema
L351// just reload metadata
L352ColumnFamilyStore cfs = columnFamilyStores.get(cfId);
L353assert cfs.getColumnFamilyName().equals(cfName);
...
L364}
L365else
L366{
L367columnFamilyStores.put(cfId,
ColumnFamilyStore.createColumnFamilyStore(this, cfName));
L368}
In the code above, an atomicity violation may occur between line 348
and 352. Suppose thread T1 executes line 348 and finds that the
concurrent hashmap columnFamilyStores contains the key
cfId. Before thread T1 executes line 352, another thread T2 removes
the cfId key from columnFamilyStores. Now thread T1 resumes
execution at line 352 and will get a null value for cfs. Then the
next line will throw a NullPointerException when invoking the method
on cfs.
Second, the snapshot above has another atomicity violation. Let's look
at lines 348 and 367. Suppose a thread T1 executes line 348 and finds
out the concurrent hashmap does not contain the key cfId. Before it
gets to execute line 367, another thread T2 puts a pair cfid, v in
the concurrent hashmap columnFamilyStores. Now thread T1 resumes
execution and it will overwrite the value written by thread T2. Thus,
the code no longer preserves the put-if-absent semantics.
I found some similar misusages in other files:
In src/java/org/apache/cassandra/gms/Gossiper.java, similar atomicity
violation may occur if thread T2 puts a value to map
endpointStateMap between lines 1094 and 1099, 1173 and 1178. Another
atomicity violation may occur if thread T2 removes the value on key
endpoint between lines 681 and 683.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4402) Atomicity violation bugs because of misusing concurrent collections
[
https://issues.apache.org/jira/browse/CASSANDRA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494352#comment-13494352
]
Jonathan Ellis commented on CASSANDRA-4402:
---
(with comments explaining the CASSANDRA-2963 situation)
Atomicity violation bugs because of misusing concurrent collections
---
Key: CASSANDRA-4402
URL: https://issues.apache.org/jira/browse/CASSANDRA-4402
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Yu Lin
Assignee: Yu Lin
Priority: Trivial
Labels: gossip
Fix For: 1.2.0
Attachments: 4402-v2.txt, cassandra-1.1.1-4402.txt
Original Estimate: 504h
Remaining Estimate: 504h
My name is Yu Lin. I'm a Ph.D. student in the CS department at
UIUC. I'm currently doing research on mining Java concurrent library
misusages. I found some misusages of ConcurrentHashMap in Cassandra
1.1.1, which may result in potential atomicity violation bugs or harm
the performance.
The code below is a snapshot of the code in file
src/java/org/apache/cassandra/db/Table.java from line 348 to 369
L348if (columnFamilyStores.containsKey(cfId))
L349{
L350// this is the case when you reset local schema
L351// just reload metadata
L352ColumnFamilyStore cfs = columnFamilyStores.get(cfId);
L353assert cfs.getColumnFamilyName().equals(cfName);
...
L364}
L365else
L366{
L367columnFamilyStores.put(cfId,
ColumnFamilyStore.createColumnFamilyStore(this, cfName));
L368}
In the code above, an atomicity violation may occur between line 348
and 352. Suppose thread T1 executes line 348 and finds that the
concurrent hashmap columnFamilyStores contains the key
cfId. Before thread T1 executes line 352, another thread T2 removes
the cfId key from columnFamilyStores. Now thread T1 resumes
execution at line 352 and will get a null value for cfs. Then the
next line will throw a NullPointerException when invoking the method
on cfs.
Second, the snapshot above has another atomicity violation. Let's look
at lines 348 and 367. Suppose a thread T1 executes line 348 and finds
out the concurrent hashmap does not contain the key cfId. Before it
gets to execute line 367, another thread T2 puts a pair cfid, v in
the concurrent hashmap columnFamilyStores. Now thread T1 resumes
execution and it will overwrite the value written by thread T2. Thus,
the code no longer preserves the put-if-absent semantics.
I found some similar misusages in other files:
In src/java/org/apache/cassandra/gms/Gossiper.java, similar atomicity
violation may occur if thread T2 puts a value to map
endpointStateMap between lines 1094 and 1099, 1173 and 1178. Another
atomicity violation may occur if thread T2 removes the value on key
endpoint between lines 681 and 683.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4935) occasional TableTest failure
[
https://issues.apache.org/jira/browse/CASSANDRA-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494360#comment-13494360
]
Jonathan Ellis commented on CASSANDRA-4935:
---
+1
occasional TableTest failure
Key: CASSANDRA-4935
URL: https://issues.apache.org/jira/browse/CASSANDRA-4935
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Assignee: Yuki Morishita
Priority: Minor
Fix For: 1.2.0 rc1
Attachments: 4935.txt
The TableTest unit test fails somewhat rarely:
{noformat}
[junit] Testsuite: org.apache.cassandra.db.TableTest
[junit] Tests run: 11, Failures: 1, Errors: 0, Time elapsed: 4.339 sec
[junit]
[junit] Testcase:
testGetSliceWithExpiration(org.apache.cassandra.db.TableTest): FAILED
[junit] Columns
[636f6c31:false:4@1,636f6c32:true:4@1!1,636f6c33:false:4@1,])] is not
expected [col1,col2]
[junit] junit.framework.AssertionFailedError: Columns
[636f6c31:false:4@1,636f6c32:true:4@1!1,636f6c33:false:4@1,])] is not
expected [col1,col2]
[junit] at
org.apache.cassandra.db.TableTest.assertColumns(TableTest.java:574)
[junit] at
org.apache.cassandra.db.TableTest.assertColumns(TableTest.java:541)
[junit] at
org.apache.cassandra.db.TableTest$5.runMayThrow(TableTest.java:353)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at org.apache.cassandra.db.TableTest.reTest(TableTest.java:56)
[junit] at
org.apache.cassandra.db.TableTest.testGetSliceWithExpiration(TableTest.java:362)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494347#comment-13494347
]
Jason Brown edited comment on CASSANDRA-4924 at 11/9/12 10:20 PM:
--
FWIW, CliSessionState.debug is only referenced in
CliClient.executeUseKeySpace() when a TException is caught :).
was (Author: jasobrown):
FWIW, CliSessionState.session state is only referenced in
CliClient.executeUseKeySpace() when a TException is caught :).
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494367#comment-13494367
]
Jonathan Ellis commented on CASSANDRA-4924:
---
Are you sure you did a clean build with my branch? I added code to use cql to
grab the extra CF definitions and merge them into what gets passed to the
compiler:
{code}
private IterableCfDef currentCfDefs()
{
return Iterables.concat(keyspacesMap.get(keySpace).cf_defs,
cql3KeyspacesMap.get(keySpace).values());
}
{code}
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: Stabilize TableTest
Updated Branches:
refs/heads/trunk f5e3ae67c - 880188384
Stabilize TableTest
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/88018838
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/88018838
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/88018838
Branch: refs/heads/trunk
Commit: 880188384f83ef673d5f02099be91e8da07a57b2
Parents: f5e3ae6
Author: Yuki Morishita yu...@apache.org
Authored: Fri Nov 9 16:29:43 2012 -0600
Committer: Yuki Morishita yu...@apache.org
Committed: Fri Nov 9 16:29:43 2012 -0600
--
test/unit/org/apache/cassandra/db/TableTest.java |3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/88018838/test/unit/org/apache/cassandra/db/TableTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/TableTest.java
b/test/unit/org/apache/cassandra/db/TableTest.java
index 4244e1c..20ab173 100644
--- a/test/unit/org/apache/cassandra/db/TableTest.java
+++ b/test/unit/org/apache/cassandra/db/TableTest.java
@@ -336,12 +336,11 @@ public class TableTest extends SchemaLoader
RowMutation rm = new RowMutation(Keyspace1, ROW.key);
ColumnFamily cf = ColumnFamily.create(Keyspace1, Standard1);
cf.addColumn(column(col1, val1, 1L));
-cf.addColumn(expiringColumn(col2, val2, 1L, 1));
+cf.addColumn(expiringColumn(col2, val2, 1L, 60)); // long enough
not to be tombstoned
cf.addColumn(column(col3, val3, 1L));
rm.add(cf);
rm.apply();
-cfStore.forceBlockingFlush();
Runnable verify = new WrappedRunnable()
{
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494380#comment-13494380
]
Jason Brown commented on CASSANDRA-4924:
Crap ... just realized I have my ~/.cassandra.in.sh baked to point to my usual
dev space. Lemme try that again :)
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[3/7] git commit: Stabilize TableTest
Stabilize TableTest
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dda7aee3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dda7aee3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dda7aee3
Branch: refs/heads/cassandra-1.2
Commit: dda7aee34f31ebbb039fb57c0b434291b1e72dfb
Parents: 792bd18
Author: Yuki Morishita yu...@apache.org
Authored: Fri Nov 9 16:29:43 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:55:25 2012 -0600
--
test/unit/org/apache/cassandra/db/TableTest.java |3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/dda7aee3/test/unit/org/apache/cassandra/db/TableTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/TableTest.java
b/test/unit/org/apache/cassandra/db/TableTest.java
index 4244e1c..20ab173 100644
--- a/test/unit/org/apache/cassandra/db/TableTest.java
+++ b/test/unit/org/apache/cassandra/db/TableTest.java
@@ -336,12 +336,11 @@ public class TableTest extends SchemaLoader
RowMutation rm = new RowMutation(Keyspace1, ROW.key);
ColumnFamily cf = ColumnFamily.create(Keyspace1, Standard1);
cf.addColumn(column(col1, val1, 1L));
-cf.addColumn(expiringColumn(col2, val2, 1L, 1));
+cf.addColumn(expiringColumn(col2, val2, 1L, 60)); // long enough
not to be tombstoned
cf.addColumn(column(col3, val3, 1L));
rm.add(cf);
rm.apply();
-cfStore.forceBlockingFlush();
Runnable verify = new WrappedRunnable()
{
[4/7] git commit: Stabilize TableTest
Stabilize TableTest
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dda7aee3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dda7aee3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dda7aee3
Branch: refs/heads/trunk
Commit: dda7aee34f31ebbb039fb57c0b434291b1e72dfb
Parents: 792bd18
Author: Yuki Morishita yu...@apache.org
Authored: Fri Nov 9 16:29:43 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:55:25 2012 -0600
--
test/unit/org/apache/cassandra/db/TableTest.java |3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/dda7aee3/test/unit/org/apache/cassandra/db/TableTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/TableTest.java
b/test/unit/org/apache/cassandra/db/TableTest.java
index 4244e1c..20ab173 100644
--- a/test/unit/org/apache/cassandra/db/TableTest.java
+++ b/test/unit/org/apache/cassandra/db/TableTest.java
@@ -336,12 +336,11 @@ public class TableTest extends SchemaLoader
RowMutation rm = new RowMutation(Keyspace1, ROW.key);
ColumnFamily cf = ColumnFamily.create(Keyspace1, Standard1);
cf.addColumn(column(col1, val1, 1L));
-cf.addColumn(expiringColumn(col2, val2, 1L, 1));
+cf.addColumn(expiringColumn(col2, val2, 1L, 60)); // long enough
not to be tombstoned
cf.addColumn(column(col3, val3, 1L));
rm.add(cf);
rm.apply();
-cfStore.forceBlockingFlush();
Runnable verify = new WrappedRunnable()
{
[5/7] git commit: fix initializeNodeUnsafe
fix initializeNodeUnsafe
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/792bd184
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/792bd184
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/792bd184
Branch: refs/heads/cassandra-1.2.0
Commit: 792bd184f57407d2853cd7233c2732eacc551761
Parents: f5e3ae6
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 16:54:59 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:54:59 2012 -0600
--
src/java/org/apache/cassandra/gms/Gossiper.java |9 +
1 files changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/792bd184/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index 5880210..1179815 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -389,7 +389,7 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
EndpointState epState = endpointStateMap.get(endpoint);
epState.updateTimestamp(); // make sure we don't evict it too soon
epState.getHeartBeatState().forceNewerGenerationUnsafe();
-epState.addApplicationState(ApplicationState.STATUS,
StorageService.instance.valueFactory.removedNonlocal(hostId,computeExpireTime()));
+epState.addApplicationState(ApplicationState.STATUS,
StorageService.instance.valueFactory.removedNonlocal(hostId,
computeExpireTime()));
logger.info(Completing removal of + endpoint);
endpointStateMap.put(endpoint, epState);
// ensure at least one gossip round occurs before returning
@@ -1090,9 +1090,10 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
public void initializeNodeUnsafe(InetAddress addr, UUID uuid, int
generationNbr)
{
HeartBeatState hbState = new HeartBeatState(generationNbr);
-EndpointState localState = new EndpointState(hbState);
-localState.markAlive();
-endpointStateMap.putIfAbsent(addr, localState);
+EndpointState newState = new EndpointState(hbState);
+newState.markAlive();
+EndpointState oldState = endpointStateMap.putIfAbsent(addr, newState);
+EndpointState localState = oldState == null ? newState : oldState;
// always add the version state
localState.addApplicationState(ApplicationState.NET_VERSION,
StorageService.instance.valueFactory.networkVersion());
[1/7] git commit: Merge branch 'cassandra-1.2' into trunk
Updated Branches: refs/heads/cassandra-1.2 f5e3ae67c - dda7aee34 refs/heads/cassandra-1.2.0 f5e3ae67c - dda7aee34 refs/heads/trunk 880188384 - fa8550f35 Merge branch 'cassandra-1.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fa8550f3 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fa8550f3 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fa8550f3 Branch: refs/heads/trunk Commit: fa8550f35552f16f4c268f7f39c63899f766ca35 Parents: 8801883 dda7aee Author: Jonathan Ellis jbel...@apache.org Authored: Fri Nov 9 16:55:39 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Fri Nov 9 16:55:39 2012 -0600 -- src/java/org/apache/cassandra/gms/Gossiper.java |9 + 1 files changed, 5 insertions(+), 4 deletions(-) --
[2/7] git commit: Stabilize TableTest
Stabilize TableTest
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dda7aee3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dda7aee3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dda7aee3
Branch: refs/heads/cassandra-1.2.0
Commit: dda7aee34f31ebbb039fb57c0b434291b1e72dfb
Parents: 792bd18
Author: Yuki Morishita yu...@apache.org
Authored: Fri Nov 9 16:29:43 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:55:25 2012 -0600
--
test/unit/org/apache/cassandra/db/TableTest.java |3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/dda7aee3/test/unit/org/apache/cassandra/db/TableTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/TableTest.java
b/test/unit/org/apache/cassandra/db/TableTest.java
index 4244e1c..20ab173 100644
--- a/test/unit/org/apache/cassandra/db/TableTest.java
+++ b/test/unit/org/apache/cassandra/db/TableTest.java
@@ -336,12 +336,11 @@ public class TableTest extends SchemaLoader
RowMutation rm = new RowMutation(Keyspace1, ROW.key);
ColumnFamily cf = ColumnFamily.create(Keyspace1, Standard1);
cf.addColumn(column(col1, val1, 1L));
-cf.addColumn(expiringColumn(col2, val2, 1L, 1));
+cf.addColumn(expiringColumn(col2, val2, 1L, 60)); // long enough
not to be tombstoned
cf.addColumn(column(col3, val3, 1L));
rm.add(cf);
rm.apply();
-cfStore.forceBlockingFlush();
Runnable verify = new WrappedRunnable()
{
[7/7] git commit: fix initializeNodeUnsafe
fix initializeNodeUnsafe
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/792bd184
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/792bd184
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/792bd184
Branch: refs/heads/cassandra-1.2
Commit: 792bd184f57407d2853cd7233c2732eacc551761
Parents: f5e3ae6
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 16:54:59 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:54:59 2012 -0600
--
src/java/org/apache/cassandra/gms/Gossiper.java |9 +
1 files changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/792bd184/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index 5880210..1179815 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -389,7 +389,7 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
EndpointState epState = endpointStateMap.get(endpoint);
epState.updateTimestamp(); // make sure we don't evict it too soon
epState.getHeartBeatState().forceNewerGenerationUnsafe();
-epState.addApplicationState(ApplicationState.STATUS,
StorageService.instance.valueFactory.removedNonlocal(hostId,computeExpireTime()));
+epState.addApplicationState(ApplicationState.STATUS,
StorageService.instance.valueFactory.removedNonlocal(hostId,
computeExpireTime()));
logger.info(Completing removal of + endpoint);
endpointStateMap.put(endpoint, epState);
// ensure at least one gossip round occurs before returning
@@ -1090,9 +1090,10 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
public void initializeNodeUnsafe(InetAddress addr, UUID uuid, int
generationNbr)
{
HeartBeatState hbState = new HeartBeatState(generationNbr);
-EndpointState localState = new EndpointState(hbState);
-localState.markAlive();
-endpointStateMap.putIfAbsent(addr, localState);
+EndpointState newState = new EndpointState(hbState);
+newState.markAlive();
+EndpointState oldState = endpointStateMap.putIfAbsent(addr, newState);
+EndpointState localState = oldState == null ? newState : oldState;
// always add the version state
localState.addApplicationState(ApplicationState.NET_VERSION,
StorageService.instance.valueFactory.networkVersion());
[6/7] git commit: fix initializeNodeUnsafe
fix initializeNodeUnsafe
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/792bd184
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/792bd184
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/792bd184
Branch: refs/heads/trunk
Commit: 792bd184f57407d2853cd7233c2732eacc551761
Parents: f5e3ae6
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 16:54:59 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 16:54:59 2012 -0600
--
src/java/org/apache/cassandra/gms/Gossiper.java |9 +
1 files changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/792bd184/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index 5880210..1179815 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -389,7 +389,7 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
EndpointState epState = endpointStateMap.get(endpoint);
epState.updateTimestamp(); // make sure we don't evict it too soon
epState.getHeartBeatState().forceNewerGenerationUnsafe();
-epState.addApplicationState(ApplicationState.STATUS,
StorageService.instance.valueFactory.removedNonlocal(hostId,computeExpireTime()));
+epState.addApplicationState(ApplicationState.STATUS,
StorageService.instance.valueFactory.removedNonlocal(hostId,
computeExpireTime()));
logger.info(Completing removal of + endpoint);
endpointStateMap.put(endpoint, epState);
// ensure at least one gossip round occurs before returning
@@ -1090,9 +1090,10 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
public void initializeNodeUnsafe(InetAddress addr, UUID uuid, int
generationNbr)
{
HeartBeatState hbState = new HeartBeatState(generationNbr);
-EndpointState localState = new EndpointState(hbState);
-localState.markAlive();
-endpointStateMap.putIfAbsent(addr, localState);
+EndpointState newState = new EndpointState(hbState);
+newState.markAlive();
+EndpointState oldState = endpointStateMap.putIfAbsent(addr, newState);
+EndpointState localState = oldState == null ? newState : oldState;
// always add the version state
localState.addApplicationState(ApplicationState.NET_VERSION,
StorageService.instance.valueFactory.networkVersion());
[jira] [Commented] (CASSANDRA-4813) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node.
[ https://issues.apache.org/jira/browse/CASSANDRA-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494393#comment-13494393 ] Michael Kjellman commented on CASSANDRA-4813: - [~yukim] I take it back. Ship It! Dependency issues on my end with the last patch, sorry for the false alarm. Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node. - Key: CASSANDRA-4813 URL: https://issues.apache.org/jira/browse/CASSANDRA-4813 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Environment: I am using SLES 10 SP3, Java 6, 4 Cassandra + Hadoop nodes, 3 Hadoop only nodes (datanodes/tasktrackers), 1 namenode/jobtracker. The machines used are Six-Core AMD Opteron(tm) Processor 8431, 24 cores and 33 GB of RAM. I get the issue on both cassandra 1.1.3, 1.1.5 and I am using Hadoop 0.20.2. Reporter: Ralph Romanos Assignee: Yuki Morishita Priority: Minor Labels: Bulkoutputformat, Hadoop, SSTables Fix For: 1.2.0 rc1 Attachments: 4813.txt The issue occurs when streaming simultaneously SSTables from the same node to a cassandra cluster using SSTableloader. It seems to me that Cassandra cannot handle receiving simultaneously SSTables from the same node. However, when it receives simultaneously SSTables from two different nodes, everything works fine. As a consequence, when using BulkOutputFormat to generate SSTables and stream them to a cassandra cluster, I cannot use more than one reducer per node otherwise I get a java.io.EOFException in the tasktracker's logs and a java.io.IOException: Broken pipe in the Cassandra logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494401#comment-13494401
]
Aleksey Yeschenko commented on CASSANDRA-4924:
--
I can confirm that CLI get works on Jonathan's branch:
[default@keyspace1] get testcompact['k1'];
= (column=1, value=2, timestamp=1352502050183000)
Returned 1 results.
Elapsed time: 19 msec(s).
[default@keyspace1] get testsparse['k1'];
= (column=1:, value=, timestamp=1352502075136000)
= (column=1:c, value=0002, timestamp=1352502075136000)
Returned 2 results.
Elapsed time: 13 msec(s).
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494404#comment-13494404
]
Jason Brown commented on CASSANDRA-4924:
Yup, now works for me, too (stoopid computer).
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4813) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node.
[ https://issues.apache.org/jira/browse/CASSANDRA-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-4813: Comment: was deleted (was: [~yukim] I take it back. Ship It! Dependency issues on my end with the last patch, sorry for the false alarm.) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node. - Key: CASSANDRA-4813 URL: https://issues.apache.org/jira/browse/CASSANDRA-4813 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Environment: I am using SLES 10 SP3, Java 6, 4 Cassandra + Hadoop nodes, 3 Hadoop only nodes (datanodes/tasktrackers), 1 namenode/jobtracker. The machines used are Six-Core AMD Opteron(tm) Processor 8431, 24 cores and 33 GB of RAM. I get the issue on both cassandra 1.1.3, 1.1.5 and I am using Hadoop 0.20.2. Reporter: Ralph Romanos Assignee: Yuki Morishita Priority: Minor Labels: Bulkoutputformat, Hadoop, SSTables Fix For: 1.2.0 rc1 Attachments: 4813.txt The issue occurs when streaming simultaneously SSTables from the same node to a cassandra cluster using SSTableloader. It seems to me that Cassandra cannot handle receiving simultaneously SSTables from the same node. However, when it receives simultaneously SSTables from two different nodes, everything works fine. As a consequence, when using BulkOutputFormat to generate SSTables and stream them to a cassandra cluster, I cannot use more than one reducer per node otherwise I get a java.io.EOFException in the tasktracker's logs and a java.io.IOException: Broken pipe in the Cassandra logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
buildbot success in ASF Buildbot on cassandra-trunk
The Buildbot has detected a restored build on builder cassandra-trunk while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/2068 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch trunk] fa8550f35552f16f4c268f7f39c63899f766ca35 Blamelist: Jonathan Ellis jbel...@apache.org,Yuki Morishita yu...@apache.org Build succeeded! sincerely, -The Buildbot
[jira] [Created] (CASSANDRA-4941) Move CompressionMetadata off-heap
Jonathan Ellis created CASSANDRA-4941: - Summary: Move CompressionMetadata off-heap Key: CASSANDRA-4941 URL: https://issues.apache.org/jira/browse/CASSANDRA-4941 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.0.0 Reporter: Jonathan Ellis Priority: Minor Fix For: 1.2.0 rc1 Breaking this out of CASSANDRA-4937. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4813) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node.
[ https://issues.apache.org/jira/browse/CASSANDRA-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494317#comment-13494317 ] Michael Kjellman edited comment on CASSANDRA-4813 at 11/9/12 11:13 PM: --- yes, actually the first time I tested I did forget to ensure both the Hadoop jar and Cassandra nodes had the newest patched version. I've tested a few times now just to make sure i'm not missing anything. Seems to die as soon as it tries to stream the first sstable to the nodes. Never progresses past 0% on the streaming and then throws the exception. applied patch to trunk when it was at commit f09a89f4cd13af2087fcc92f09f6cf1ee4785feb. i rebuilt the entire cluster, and ensured my maven dependencies were all set. Still reproduced the problem unfortunately (i actually thought it had been resolved but i just reproduced the java.io.IOException: Broken pipe again). MD5 (build/apache-cassandra-1.2.0-beta2-SNAPSHOT.jar) = 92d8ffacb3963116dd153a2c8c83fbe9 was (Author: mkjellman): yes, actually the first time i didn't and then i realized i hadn't rebuilt the Cassandra nodes and only replaced the compiled jar in my hadoop job. I've tested a few times now just to make sure i'm not missing anything. Seems to die as soon as it tries to stream the first sstable to the nodes. Never progresses past 0% on the streaming and then throws the exception. Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node. - Key: CASSANDRA-4813 URL: https://issues.apache.org/jira/browse/CASSANDRA-4813 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.0 Environment: I am using SLES 10 SP3, Java 6, 4 Cassandra + Hadoop nodes, 3 Hadoop only nodes (datanodes/tasktrackers), 1 namenode/jobtracker. The machines used are Six-Core AMD Opteron(tm) Processor 8431, 24 cores and 33 GB of RAM. I get the issue on both cassandra 1.1.3, 1.1.5 and I am using Hadoop 0.20.2. Reporter: Ralph Romanos Assignee: Yuki Morishita Priority: Minor Labels: Bulkoutputformat, Hadoop, SSTables Fix For: 1.2.0 rc1 Attachments: 4813.txt The issue occurs when streaming simultaneously SSTables from the same node to a cassandra cluster using SSTableloader. It seems to me that Cassandra cannot handle receiving simultaneously SSTables from the same node. However, when it receives simultaneously SSTables from two different nodes, everything works fine. As a consequence, when using BulkOutputFormat to generate SSTables and stream them to a cassandra cluster, I cannot use more than one reducer per node otherwise I get a java.io.EOFException in the tasktracker's logs and a java.io.IOException: Broken pipe in the Cassandra logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4924) Make CQL 3 data accessible via thrift.
[
https://issues.apache.org/jira/browse/CASSANDRA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494419#comment-13494419
]
Jason Brown commented on CASSANDRA-4924:
LGTM, +1
Make CQL 3 data accessible via thrift.
--
Key: CASSANDRA-4924
URL: https://issues.apache.org/jira/browse/CASSANDRA-4924
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: amorton
Assignee: Jonathan Ellis
Labels: cli, cql
Fix For: 1.2.0 rc1
Following the changes from CASSANDRA-4377 data created using CQL 3 is not
visible via the thrift interface.
This goes against the spirit of many comments by the project that the thrift
API is not going away. These statements and ones such as Internally, both
CQL3 and thrift use the same storage engine, so all future improvements to
this engine will impact both of them equally.
(http://www.datastax.com/dev/blog/thrift-to-cql3) and the CQL3 and thrift
examples given here
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts gave the
impression CQL 3 was a layer on top of the core storage engine. It now
appears to be an incompatible format change.
It makes it impossible to explain to existing using users how CQL 3 stores
it's data.
It also creates an all or nothing approach to trying CQL 3.
My request is to make all data written by CQL 3 readable via the thrift API.
An example of using the current 1.2 trunk is below:
{noformat}
cqlsh:cass_college CREATE TABLE UserTweets
... (
... tweet_idbigint,
... user_name text,
... bodytext,
... timestamp timestamp,
... PRIMARY KEY (user_name, tweet_id)
... );
cqlsh:cass_college INSERT INTO
... UserTweets
... (tweet_id, body, user_name, timestamp)
... VALUES
... (1, 'The Tweet', 'fred', 1352150816917);
cqlsh:cass_college
cqlsh:cass_college
cqlsh:cass_college select * from UserTweets;
user_name | tweet_id | body | timestamp
---+--+---+--
fred |1 | The Tweet | 2012-11-06 10:26:56+1300
{noformat}
and in the CLI
{noformat}
[default@cass_college] show schema;
create keyspace cass_college
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 3}
and durable_writes = true;
use cass_college;
[default@cass_college] list UserTweets;
UserTweets not found in current keyspace.
[default@cass_college]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4941) Move CompressionMetadata off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4941: -- Reviewer: xedin Assignee: Jonathan Ellis Move CompressionMetadata off-heap - Key: CASSANDRA-4941 URL: https://issues.apache.org/jira/browse/CASSANDRA-4941 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.0.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Priority: Minor Fix For: 1.2.0 rc1 Attachments: 4941.txt Breaking this out of CASSANDRA-4937. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[3/7] git commit: allow CLI to GET cql3 columnfamily data patch by jbellis; reviewed by Jason Brown and Aleksey Yeschenko for CASSANDRA-424
allow CLI to GET cql3 columnfamily data
patch by jbellis; reviewed by Jason Brown and Aleksey Yeschenko for
CASSANDRA-424
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5c81f4b9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5c81f4b9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5c81f4b9
Branch: refs/heads/cassandra-1.2
Commit: 5c81f4b928b3a94d317c2e3829a07447ea21ea77
Parents: fc5f877
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Nov 8 11:13:10 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 17:35:44 2012 -0600
--
CHANGES.txt|1 +
src/java/org/apache/cassandra/cli/CliClient.java | 118 +++
src/java/org/apache/cassandra/cli/CliCompiler.java |4 +-
src/java/org/apache/cassandra/cli/Cql3CfDef.java |5 -
4 files changed, 92 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c81f4b9/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index f6769fe..b73b14d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * allow CLI to GET cql3 columnfamily data (CASSANDRA-4924)
* Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
doesn't get free'd prematurely (CASSANDRA-4934)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c81f4b9/src/java/org/apache/cassandra/cli/CliClient.java
--
diff --git a/src/java/org/apache/cassandra/cli/CliClient.java
b/src/java/org/apache/cassandra/cli/CliClient.java
index 9594525..3060359 100644
--- a/src/java/org/apache/cassandra/cli/CliClient.java
+++ b/src/java/org/apache/cassandra/cli/CliClient.java
@@ -30,6 +30,7 @@ import java.util.*;
import com.google.common.base.Charsets;
import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
+import com.google.common.collect.Iterables;
import org.apache.commons.lang.StringUtils;
import org.antlr.runtime.tree.Tree;
@@ -148,6 +149,7 @@ public class CliClient
private String keySpace = null;
private String username = null;
private final MapString, KsDef keyspacesMap = new HashMapString,
KsDef();
+private final MapString, MapString, CfDef cql3KeyspacesMap = new
HashMapString, MapString, CfDef();
private final MapString, AbstractType? cfKeysComparators;
private ConsistencyLevel consistencyLevel = ConsistencyLevel.ONE;
private final CfAssumptions assumptions = new CfAssumptions();
@@ -317,13 +319,72 @@ public class CliClient
// Lazily lookup keyspace meta-data.
if (!(keyspacesMap.containsKey(keyspace)))
{
-keyspacesMap.put(keyspace,
thriftClient.describe_keyspace(keyspace));
+KsDef ksDef = thriftClient.describe_keyspace(keyspace);
+keyspacesMap.put(keyspace, ksDef);
+cql3KeyspacesMap.put(keyspace, loadCql3Defs(thriftClient, ksDef));
assumptions.replayAssumptions(keyspace);
}
return keyspacesMap.get(keyspace);
}
+public static MapString, CfDef loadCql3Defs(Cassandra.Client
thriftClient, KsDef thriftKs)
+{
+try
+{
+return loadCql3DefsUnchecked(thriftClient, thriftKs);
+}
+catch (Exception e)
+{
+throw new RuntimeException(e);
+}
+}
+
+public static MapString, CfDef loadCql3DefsUnchecked(Cassandra.Client
thriftClient, KsDef thriftKs) throws Exception
+{
+MapString, CfDef cql3Defs = new HashMapString, CfDef();
+
+String query = SELECT columnfamily_name, comparator,
default_validator, key_validator FROM system.schema_columnfamilies WHERE
keyspace_name='%s';
+String formatted = String.format(query, thriftKs.name);
+CqlResult result =
thriftClient.execute_cql3_query(ByteBufferUtil.bytes(formatted),
+ Compression.NONE,
+
ConsistencyLevel.ONE);
+outer:
+for (CqlRow row : result.rows)
+{
+Column rawName = row.columns.get(0);
+assert
ByteBufferUtil.string(ByteBuffer.wrap(rawName.getName())).equals(columnfamily_name);
+String name =
ByteBufferUtil.string(ByteBuffer.wrap(rawName.getValue()));
+
+Column rawComparator = row.columns.get(1);
+assert
ByteBufferUtil.string(ByteBuffer.wrap(rawComparator.getName())).equals(comparator);
+String comparator =
[4/7] git commit: allow CLI to GET cql3 columnfamily data patch by jbellis; reviewed by Jason Brown and Aleksey Yeschenko for CASSANDRA-424
allow CLI to GET cql3 columnfamily data
patch by jbellis; reviewed by Jason Brown and Aleksey Yeschenko for
CASSANDRA-424
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5c81f4b9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5c81f4b9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5c81f4b9
Branch: refs/heads/trunk
Commit: 5c81f4b928b3a94d317c2e3829a07447ea21ea77
Parents: fc5f877
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Nov 8 11:13:10 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 17:35:44 2012 -0600
--
CHANGES.txt|1 +
src/java/org/apache/cassandra/cli/CliClient.java | 118 +++
src/java/org/apache/cassandra/cli/CliCompiler.java |4 +-
src/java/org/apache/cassandra/cli/Cql3CfDef.java |5 -
4 files changed, 92 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c81f4b9/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index f6769fe..b73b14d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * allow CLI to GET cql3 columnfamily data (CASSANDRA-4924)
* Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
doesn't get free'd prematurely (CASSANDRA-4934)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c81f4b9/src/java/org/apache/cassandra/cli/CliClient.java
--
diff --git a/src/java/org/apache/cassandra/cli/CliClient.java
b/src/java/org/apache/cassandra/cli/CliClient.java
index 9594525..3060359 100644
--- a/src/java/org/apache/cassandra/cli/CliClient.java
+++ b/src/java/org/apache/cassandra/cli/CliClient.java
@@ -30,6 +30,7 @@ import java.util.*;
import com.google.common.base.Charsets;
import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
+import com.google.common.collect.Iterables;
import org.apache.commons.lang.StringUtils;
import org.antlr.runtime.tree.Tree;
@@ -148,6 +149,7 @@ public class CliClient
private String keySpace = null;
private String username = null;
private final MapString, KsDef keyspacesMap = new HashMapString,
KsDef();
+private final MapString, MapString, CfDef cql3KeyspacesMap = new
HashMapString, MapString, CfDef();
private final MapString, AbstractType? cfKeysComparators;
private ConsistencyLevel consistencyLevel = ConsistencyLevel.ONE;
private final CfAssumptions assumptions = new CfAssumptions();
@@ -317,13 +319,72 @@ public class CliClient
// Lazily lookup keyspace meta-data.
if (!(keyspacesMap.containsKey(keyspace)))
{
-keyspacesMap.put(keyspace,
thriftClient.describe_keyspace(keyspace));
+KsDef ksDef = thriftClient.describe_keyspace(keyspace);
+keyspacesMap.put(keyspace, ksDef);
+cql3KeyspacesMap.put(keyspace, loadCql3Defs(thriftClient, ksDef));
assumptions.replayAssumptions(keyspace);
}
return keyspacesMap.get(keyspace);
}
+public static MapString, CfDef loadCql3Defs(Cassandra.Client
thriftClient, KsDef thriftKs)
+{
+try
+{
+return loadCql3DefsUnchecked(thriftClient, thriftKs);
+}
+catch (Exception e)
+{
+throw new RuntimeException(e);
+}
+}
+
+public static MapString, CfDef loadCql3DefsUnchecked(Cassandra.Client
thriftClient, KsDef thriftKs) throws Exception
+{
+MapString, CfDef cql3Defs = new HashMapString, CfDef();
+
+String query = SELECT columnfamily_name, comparator,
default_validator, key_validator FROM system.schema_columnfamilies WHERE
keyspace_name='%s';
+String formatted = String.format(query, thriftKs.name);
+CqlResult result =
thriftClient.execute_cql3_query(ByteBufferUtil.bytes(formatted),
+ Compression.NONE,
+
ConsistencyLevel.ONE);
+outer:
+for (CqlRow row : result.rows)
+{
+Column rawName = row.columns.get(0);
+assert
ByteBufferUtil.string(ByteBuffer.wrap(rawName.getName())).equals(columnfamily_name);
+String name =
ByteBufferUtil.string(ByteBuffer.wrap(rawName.getValue()));
+
+Column rawComparator = row.columns.get(1);
+assert
ByteBufferUtil.string(ByteBuffer.wrap(rawComparator.getName())).equals(comparator);
+String comparator =
[6/7] git commit: switch defaultValidator to BytesType where CQL doesn't care
switch defaultValidator to BytesType where CQL doesn't care Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fc5f877f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fc5f877f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fc5f877f Branch: refs/heads/trunk Commit: fc5f877fb2c17e3003ba4b31817bd2ac10ee5f6a Parents: dda7aee Author: Jonathan Ellis jbel...@apache.org Authored: Thu Nov 8 13:17:12 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Fri Nov 9 17:34:15 2012 -0600 -- .../statements/CreateColumnFamilyStatement.java| 11 +++ 1 files changed, 3 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fc5f877f/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java b/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java index b8ea732..1f8ee7e 100644 --- a/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java @@ -34,16 +34,11 @@ import org.apache.cassandra.config.Schema; import org.apache.cassandra.cql3.*; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; +import org.apache.cassandra.db.marshal.*; import org.apache.cassandra.exceptions.ConfigurationException; import org.apache.cassandra.exceptions.RequestValidationException; import org.apache.cassandra.exceptions.UnauthorizedException; import org.apache.cassandra.db.ColumnFamilyType; -import org.apache.cassandra.db.marshal.AbstractType; -import org.apache.cassandra.db.marshal.CollectionType; -import org.apache.cassandra.db.marshal.ColumnToCollectionType; -import org.apache.cassandra.db.marshal.CompositeType; -import org.apache.cassandra.db.marshal.ReversedType; -import org.apache.cassandra.db.marshal.CounterColumnType; import org.apache.cassandra.exceptions.InvalidRequestException; import org.apache.cassandra.service.ClientState; import org.apache.cassandra.service.MigrationManager; @@ -279,7 +274,7 @@ public class CreateColumnFamilyStatement extends SchemaAlteringStatement throw new InvalidRequestException(String.format(COMPACT STORAGE with non-composite PRIMARY KEY require one column not part of the PRIMARY KEY (got: %s), StringUtils.join(stmt.columns.keySet(), , ))); // The only value we'll insert will be the empty one, so the default validator don't matter -stmt.defaultValidator = CFDefinition.definitionType; +stmt.defaultValidator = BytesType.instance; // We need to distinguish between // * I'm upgrading from thrift so the valueAlias is null // * I've define my table with only a PK (and the column value will be empty) @@ -303,7 +298,7 @@ public class CreateColumnFamilyStatement extends SchemaAlteringStatement // the actual validator don't matter much (except that we want to recognize counter CF as limitation apply to them). stmt.defaultValidator = !stmt.columns.isEmpty() (stmt.columns.values().iterator().next() instanceof CounterColumnType) ? CounterColumnType.instance -: CFDefinition.definitionType; +: BytesType.instance; }
[1/7] git commit: Merge branch 'cassandra-1.2' into trunk
Updated Branches: refs/heads/cassandra-1.2 dda7aee34 - 5c81f4b92 refs/heads/cassandra-1.2.0 dda7aee34 - 5c81f4b92 refs/heads/trunk fa8550f35 - d82c4408c Merge branch 'cassandra-1.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d82c4408 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d82c4408 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d82c4408 Branch: refs/heads/trunk Commit: d82c4408ce62cd815a065fffdd1b30564815c991 Parents: fa8550f 5c81f4b Author: Jonathan Ellis jbel...@apache.org Authored: Fri Nov 9 17:35:57 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Fri Nov 9 17:35:57 2012 -0600 -- CHANGES.txt|1 + src/java/org/apache/cassandra/cli/CliClient.java | 118 +++ src/java/org/apache/cassandra/cli/CliCompiler.java |4 +- src/java/org/apache/cassandra/cli/Cql3CfDef.java |5 - .../statements/CreateColumnFamilyStatement.java| 11 +- 5 files changed, 95 insertions(+), 44 deletions(-) --
[7/7] git commit: switch defaultValidator to BytesType where CQL doesn't care
switch defaultValidator to BytesType where CQL doesn't care Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fc5f877f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fc5f877f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fc5f877f Branch: refs/heads/cassandra-1.2 Commit: fc5f877fb2c17e3003ba4b31817bd2ac10ee5f6a Parents: dda7aee Author: Jonathan Ellis jbel...@apache.org Authored: Thu Nov 8 13:17:12 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Fri Nov 9 17:34:15 2012 -0600 -- .../statements/CreateColumnFamilyStatement.java| 11 +++ 1 files changed, 3 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fc5f877f/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java b/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java index b8ea732..1f8ee7e 100644 --- a/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java @@ -34,16 +34,11 @@ import org.apache.cassandra.config.Schema; import org.apache.cassandra.cql3.*; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; +import org.apache.cassandra.db.marshal.*; import org.apache.cassandra.exceptions.ConfigurationException; import org.apache.cassandra.exceptions.RequestValidationException; import org.apache.cassandra.exceptions.UnauthorizedException; import org.apache.cassandra.db.ColumnFamilyType; -import org.apache.cassandra.db.marshal.AbstractType; -import org.apache.cassandra.db.marshal.CollectionType; -import org.apache.cassandra.db.marshal.ColumnToCollectionType; -import org.apache.cassandra.db.marshal.CompositeType; -import org.apache.cassandra.db.marshal.ReversedType; -import org.apache.cassandra.db.marshal.CounterColumnType; import org.apache.cassandra.exceptions.InvalidRequestException; import org.apache.cassandra.service.ClientState; import org.apache.cassandra.service.MigrationManager; @@ -279,7 +274,7 @@ public class CreateColumnFamilyStatement extends SchemaAlteringStatement throw new InvalidRequestException(String.format(COMPACT STORAGE with non-composite PRIMARY KEY require one column not part of the PRIMARY KEY (got: %s), StringUtils.join(stmt.columns.keySet(), , ))); // The only value we'll insert will be the empty one, so the default validator don't matter -stmt.defaultValidator = CFDefinition.definitionType; +stmt.defaultValidator = BytesType.instance; // We need to distinguish between // * I'm upgrading from thrift so the valueAlias is null // * I've define my table with only a PK (and the column value will be empty) @@ -303,7 +298,7 @@ public class CreateColumnFamilyStatement extends SchemaAlteringStatement // the actual validator don't matter much (except that we want to recognize counter CF as limitation apply to them). stmt.defaultValidator = !stmt.columns.isEmpty() (stmt.columns.values().iterator().next() instanceof CounterColumnType) ? CounterColumnType.instance -: CFDefinition.definitionType; +: BytesType.instance; }
[5/7] git commit: switch defaultValidator to BytesType where CQL doesn't care
switch defaultValidator to BytesType where CQL doesn't care Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fc5f877f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fc5f877f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fc5f877f Branch: refs/heads/cassandra-1.2.0 Commit: fc5f877fb2c17e3003ba4b31817bd2ac10ee5f6a Parents: dda7aee Author: Jonathan Ellis jbel...@apache.org Authored: Thu Nov 8 13:17:12 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Fri Nov 9 17:34:15 2012 -0600 -- .../statements/CreateColumnFamilyStatement.java| 11 +++ 1 files changed, 3 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fc5f877f/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java b/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java index b8ea732..1f8ee7e 100644 --- a/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/CreateColumnFamilyStatement.java @@ -34,16 +34,11 @@ import org.apache.cassandra.config.Schema; import org.apache.cassandra.cql3.*; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; +import org.apache.cassandra.db.marshal.*; import org.apache.cassandra.exceptions.ConfigurationException; import org.apache.cassandra.exceptions.RequestValidationException; import org.apache.cassandra.exceptions.UnauthorizedException; import org.apache.cassandra.db.ColumnFamilyType; -import org.apache.cassandra.db.marshal.AbstractType; -import org.apache.cassandra.db.marshal.CollectionType; -import org.apache.cassandra.db.marshal.ColumnToCollectionType; -import org.apache.cassandra.db.marshal.CompositeType; -import org.apache.cassandra.db.marshal.ReversedType; -import org.apache.cassandra.db.marshal.CounterColumnType; import org.apache.cassandra.exceptions.InvalidRequestException; import org.apache.cassandra.service.ClientState; import org.apache.cassandra.service.MigrationManager; @@ -279,7 +274,7 @@ public class CreateColumnFamilyStatement extends SchemaAlteringStatement throw new InvalidRequestException(String.format(COMPACT STORAGE with non-composite PRIMARY KEY require one column not part of the PRIMARY KEY (got: %s), StringUtils.join(stmt.columns.keySet(), , ))); // The only value we'll insert will be the empty one, so the default validator don't matter -stmt.defaultValidator = CFDefinition.definitionType; +stmt.defaultValidator = BytesType.instance; // We need to distinguish between // * I'm upgrading from thrift so the valueAlias is null // * I've define my table with only a PK (and the column value will be empty) @@ -303,7 +298,7 @@ public class CreateColumnFamilyStatement extends SchemaAlteringStatement // the actual validator don't matter much (except that we want to recognize counter CF as limitation apply to them). stmt.defaultValidator = !stmt.columns.isEmpty() (stmt.columns.values().iterator().next() instanceof CounterColumnType) ? CounterColumnType.instance -: CFDefinition.definitionType; +: BytesType.instance; }
[2/7] git commit: allow CLI to GET cql3 columnfamily data patch by jbellis; reviewed by Jason Brown and Aleksey Yeschenko for CASSANDRA-424
allow CLI to GET cql3 columnfamily data
patch by jbellis; reviewed by Jason Brown and Aleksey Yeschenko for
CASSANDRA-424
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5c81f4b9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5c81f4b9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5c81f4b9
Branch: refs/heads/cassandra-1.2.0
Commit: 5c81f4b928b3a94d317c2e3829a07447ea21ea77
Parents: fc5f877
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Nov 8 11:13:10 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 17:35:44 2012 -0600
--
CHANGES.txt|1 +
src/java/org/apache/cassandra/cli/CliClient.java | 118 +++
src/java/org/apache/cassandra/cli/CliCompiler.java |4 +-
src/java/org/apache/cassandra/cli/Cql3CfDef.java |5 -
4 files changed, 92 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c81f4b9/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index f6769fe..b73b14d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * allow CLI to GET cql3 columnfamily data (CASSANDRA-4924)
* Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
doesn't get free'd prematurely (CASSANDRA-4934)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c81f4b9/src/java/org/apache/cassandra/cli/CliClient.java
--
diff --git a/src/java/org/apache/cassandra/cli/CliClient.java
b/src/java/org/apache/cassandra/cli/CliClient.java
index 9594525..3060359 100644
--- a/src/java/org/apache/cassandra/cli/CliClient.java
+++ b/src/java/org/apache/cassandra/cli/CliClient.java
@@ -30,6 +30,7 @@ import java.util.*;
import com.google.common.base.Charsets;
import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
+import com.google.common.collect.Iterables;
import org.apache.commons.lang.StringUtils;
import org.antlr.runtime.tree.Tree;
@@ -148,6 +149,7 @@ public class CliClient
private String keySpace = null;
private String username = null;
private final MapString, KsDef keyspacesMap = new HashMapString,
KsDef();
+private final MapString, MapString, CfDef cql3KeyspacesMap = new
HashMapString, MapString, CfDef();
private final MapString, AbstractType? cfKeysComparators;
private ConsistencyLevel consistencyLevel = ConsistencyLevel.ONE;
private final CfAssumptions assumptions = new CfAssumptions();
@@ -317,13 +319,72 @@ public class CliClient
// Lazily lookup keyspace meta-data.
if (!(keyspacesMap.containsKey(keyspace)))
{
-keyspacesMap.put(keyspace,
thriftClient.describe_keyspace(keyspace));
+KsDef ksDef = thriftClient.describe_keyspace(keyspace);
+keyspacesMap.put(keyspace, ksDef);
+cql3KeyspacesMap.put(keyspace, loadCql3Defs(thriftClient, ksDef));
assumptions.replayAssumptions(keyspace);
}
return keyspacesMap.get(keyspace);
}
+public static MapString, CfDef loadCql3Defs(Cassandra.Client
thriftClient, KsDef thriftKs)
+{
+try
+{
+return loadCql3DefsUnchecked(thriftClient, thriftKs);
+}
+catch (Exception e)
+{
+throw new RuntimeException(e);
+}
+}
+
+public static MapString, CfDef loadCql3DefsUnchecked(Cassandra.Client
thriftClient, KsDef thriftKs) throws Exception
+{
+MapString, CfDef cql3Defs = new HashMapString, CfDef();
+
+String query = SELECT columnfamily_name, comparator,
default_validator, key_validator FROM system.schema_columnfamilies WHERE
keyspace_name='%s';
+String formatted = String.format(query, thriftKs.name);
+CqlResult result =
thriftClient.execute_cql3_query(ByteBufferUtil.bytes(formatted),
+ Compression.NONE,
+
ConsistencyLevel.ONE);
+outer:
+for (CqlRow row : result.rows)
+{
+Column rawName = row.columns.get(0);
+assert
ByteBufferUtil.string(ByteBuffer.wrap(rawName.getName())).equals(columnfamily_name);
+String name =
ByteBufferUtil.string(ByteBuffer.wrap(rawName.getValue()));
+
+Column rawComparator = row.columns.get(1);
+assert
ByteBufferUtil.string(ByteBuffer.wrap(rawComparator.getName())).equals(comparator);
+String comparator =
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494441#comment-13494441 ] Pavel Yaskevich commented on CASSANDRA-4937: bq. Still don't like the global objectCache code. SSTR queue (no Map) would be better. Or, what about SSTR Threadlocal? The reason why it's done via queues associated with filenames is that we want to get rid of CRAR instances as soon as we done with individual SSTables and I don't want to involve SSTR there more then just cleaning queues upon termination. bq. Isn't the right fix then to only drop the recently-read blocks during sequential i/o? We can't because it would require (e.g. compaction) to know of how something is using files, it could remove cache from somebody's fit anyway and we would use too many system calls if we do that. CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4941) Move CompressionMetadata off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-4941: --- Issue Type: Sub-task (was: Improvement) Parent: CASSANDRA-4937 Move CompressionMetadata off-heap - Key: CASSANDRA-4941 URL: https://issues.apache.org/jira/browse/CASSANDRA-4941 Project: Cassandra Issue Type: Sub-task Components: Core Affects Versions: 1.0.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Priority: Minor Fix For: 1.2.0 rc1 Attachments: 4941.txt Breaking this out of CASSANDRA-4937. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494447#comment-13494447 ] Jonathan Ellis commented on CASSANDRA-4937: --- bq. The reason why it's done via queues associated with filenames is that we want to get rid of CRAR instances as soon as we done with individual SSTables Still feels like a hack to me... Will think about alternatives. bq. We can't because it would require (e.g. compaction) to know of how something is using files But isn't that the point, that we do know that compaction only* does sequential i/o? * not quite, but close enough for this purpose CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4936) Less than operator when comparing timeuuids behaves as less than equal.
[ https://issues.apache.org/jira/browse/CASSANDRA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494449#comment-13494449 ] Tyler Hobbs commented on CASSANDRA-4936: bq. Are you sure the UUIDs do not differ in their non-time-based component? (Note: I suggested that Cesar open this ticket.) It almost certainly is a problem with the non-time-based component. However, the intention of the user is pretty clear for this query: select anything where the timestamp component is less than the provided one. I've been handling this specially in pycassa for quite a while. If the comparator is TimeUUID, and a datetime/timestamp is passed as a slice end, it will create a TimeUUID with special non-timestamp components. For example, since slice ends are inclusive in the thrift api, when creating the TimeUUID for the slice 'finish', the highest possible TimeUUID with the given timestamp will be used. We could do something similar with CQL, it would just require passing along information about whether to create the highest or lowest TimeUUID representation for a given datestamp based on the comparison operator that's used. If details about what non-timestamp components pycassa uses would be useful, let me know. Less than operator when comparing timeuuids behaves as less than equal. --- Key: CASSANDRA-4936 URL: https://issues.apache.org/jira/browse/CASSANDRA-4936 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.6 Environment: Linux CentOS. Linux localhost.localdomain 2.6.18-308.16.1.el5 #1 SMP Tue Oct 2 22:01:37 EDT 2012 i686 i686 i386 GNU/Linux Reporter: Cesar Lopez-Nataren If we define the following column family using CQL3: CREATE TABLE useractivity ( user_id int, activity_id 'TimeUUIDType', data text, PRIMARY KEY (user_id, activity_id) ); Add some values to it. And then query it like: SELECT * FROM useractivity WHERE user_id = '3' AND activity_id '2012-11-07 18:18:22-0800' ORDER BY activity_id DESC LIMIT 1; the record with timeuuid '2012-11-07 18:18:22-0800' returns in the results. According to the documentation, on CQL3 the '' and '' operators are strict, meaning not inclusive, so this seems to be a bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4941) Move CompressionMetadata off-heap
[
https://issues.apache.org/jira/browse/CASSANDRA-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Yaskevich updated CASSANDRA-4941:
---
Attachment: CASSANDRA-4941-v2.patch
removed offset alignment in Memory for {set,get}Long operations and
BigLongArray file. Fixed CM.getChunksForSections() to use correct next chunk
offset.
Move CompressionMetadata off-heap
-
Key: CASSANDRA-4941
URL: https://issues.apache.org/jira/browse/CASSANDRA-4941
Project: Cassandra
Issue Type: Sub-task
Components: Core
Affects Versions: 1.0.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 1.2.0 rc1
Attachments: 4941.txt, CASSANDRA-4941-v2.patch
Breaking this out of CASSANDRA-4937.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494468#comment-13494468 ] Pavel Yaskevich commented on CASSANDRA-4937: bq. Still feels like a hack to me... Will think about alternatives. You are welcome to think how to do so without involving more then one file, I have exploited lhf possibility :) bq. But isn't that the point, that we do know that compaction only* does sequential i/o? Even so, we will need to expose RAR method to skip cache to SSTable iterators so they can hint RAR to hit OS to remove cache after they are done with the row for example, which, for example, would generate too many system calls if you have a small rows (or try to somehow batch them together which would require coordination between sstable iterators) as well we would have to teach other components, which want to skip cache, how to do so (e.g. streaming). That individual approach creates nothing but complexing because reads would still be suffering even if we change granularity of hints. CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4912) BulkOutputFormat should support Hadoop MultipleOutput
[ https://issues.apache.org/jira/browse/CASSANDRA-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494471#comment-13494471 ] Michael Kjellman commented on CASSANDRA-4912: - [~brandon.williams] If I patch BulkOutputFormat.java in a similar manner to CASSANDRA-4208 (line 40) this is what is causing the initial check of the config to pass but fail when the reducer is created. Still not sure why the behavior is different. BulkOutputFormat should support Hadoop MultipleOutput - Key: CASSANDRA-4912 URL: https://issues.apache.org/jira/browse/CASSANDRA-4912 Project: Cassandra Issue Type: New Feature Components: Hadoop Affects Versions: 1.2.0 beta 1 Reporter: Michael Kjellman Attachments: Example.java Much like CASSANDRA-4208 BOF should support outputting to Multiple Column Families. The current approach takken in the patch for COF results in only one stream being sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4848) Expose black-listed directories via JMX
[
https://issues.apache.org/jira/browse/CASSANDRA-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated CASSANDRA-4848:
-
Attachment: trunk-4848.txt
Expose black-listed directories via JMX
---
Key: CASSANDRA-4848
URL: https://issues.apache.org/jira/browse/CASSANDRA-4848
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: Kirk True
Assignee: Kirk True
Priority: Minor
Labels: lhf
Fix For: 1.2.0 rc1
Attachments: trunk-4848.txt
JBOD support included failure modes (CASSANDRA-2118) that insert directories
on bad disks to the {{BlacklistedDirectories}} class. However, it doesn't
appear that the list of directories is exposed to administrators via JMX or
other means. So unless they're scouring the logs or being notified at the OS
level, they will remain unaware of the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4848) Expose black-listed directories via JMX
[
https://issues.apache.org/jira/browse/CASSANDRA-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494490#comment-13494490
]
Kirk True commented on CASSANDRA-4848:
--
Added a BlacklistedDirectoriesMBean that allows for retrieving the set of
unreadable and unwritable directories.
Expose black-listed directories via JMX
---
Key: CASSANDRA-4848
URL: https://issues.apache.org/jira/browse/CASSANDRA-4848
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: Kirk True
Assignee: Kirk True
Priority: Minor
Labels: lhf
Fix For: 1.2.0 rc1
Attachments: trunk-4848.txt
JBOD support included failure modes (CASSANDRA-2118) that insert directories
on bad disks to the {{BlacklistedDirectories}} class. However, it doesn't
appear that the list of directories is exposed to administrators via JMX or
other means. So unless they're scouring the logs or being notified at the OS
level, they will remain unaware of the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4912) BulkOutputFormat should support Hadoop MultipleOutput
[ https://issues.apache.org/jira/browse/CASSANDRA-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494500#comment-13494500 ] Michael Kjellman commented on CASSANDRA-4912: - looks like OUTPUT_COLUMNFAMILY_CONFIG never gets set in ConfigHelper when a a new BulkRecordWriter is created. Difficult to figure out exactly what should/where the code should be setting mapreduce.output.basename in the job config. BulkOutputFormat should support Hadoop MultipleOutput - Key: CASSANDRA-4912 URL: https://issues.apache.org/jira/browse/CASSANDRA-4912 Project: Cassandra Issue Type: New Feature Components: Hadoop Affects Versions: 1.2.0 beta 1 Reporter: Michael Kjellman Attachments: Example.java Much like CASSANDRA-4208 BOF should support outputting to Multiple Column Families. The current approach takken in the patch for COF results in only one stream being sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4848) Expose black-listed directories via JMX
[
https://issues.apache.org/jira/browse/CASSANDRA-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-4848:
--
Reviewer: dbrosius
Expose black-listed directories via JMX
---
Key: CASSANDRA-4848
URL: https://issues.apache.org/jira/browse/CASSANDRA-4848
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2.0 beta 1
Reporter: Kirk True
Assignee: Kirk True
Priority: Minor
Labels: lhf
Fix For: 1.2.0 rc1
Attachments: trunk-4848.txt
JBOD support included failure modes (CASSANDRA-2118) that insert directories
on bad disks to the {{BlacklistedDirectories}} class. However, it doesn't
appear that the list of directories is exposed to administrators via JMX or
other means. So unless they're scouring the logs or being notified at the OS
level, they will remain unaware of the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4912) BulkOutputFormat should support Hadoop MultipleOutput
[ https://issues.apache.org/jira/browse/CASSANDRA-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494513#comment-13494513 ] Michael Kjellman commented on CASSANDRA-4912: - I think also another difference in behavior between CFOF and BOF is that when a new BulkRecordWriter(Configuration conf) is created it creates the directory for the sstables. It calls ConfigHelper here to get the name of the column family so it can create the directory. The only call to getOutputColumnFamily is RangeClient in CFOF. Normally, without MultipleOutputs the job config would include a setOutputColumnFamily(). I don't understand what calls setOutputColumnFamily when you add a new named MultipleOutput. I presume this is where the problem is. BulkOutputFormat should support Hadoop MultipleOutput - Key: CASSANDRA-4912 URL: https://issues.apache.org/jira/browse/CASSANDRA-4912 Project: Cassandra Issue Type: New Feature Components: Hadoop Affects Versions: 1.2.0 beta 1 Reporter: Michael Kjellman Attachments: Example.java Much like CASSANDRA-4208 BOF should support outputting to Multiple Column Families. The current approach takken in the patch for COF results in only one stream being sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[4/4] git commit: Move CompressionMetadata off-heap patch by jbellis and xedin for CASSANDRA-4937
Move CompressionMetadata off-heap
patch by jbellis and xedin for CASSANDRA-4937
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4549a98b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4549a98b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4549a98b
Branch: refs/heads/cassandra-1.2
Commit: 4549a98b4ef6c293c200cbd8e226aee2e574c2c0
Parents: 5c81f4b
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 20:47:27 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 20:47:27 2012 -0600
--
CHANGES.txt|1 +
.../cassandra/io/compress/CompressionMetadata.java | 35 ---
.../cassandra/io/util/CompressedSegmentedFile.java |2 +-
src/java/org/apache/cassandra/io/util/Memory.java | 12 +++
.../org/apache/cassandra/utils/BigLongArray.java | 77 ---
5 files changed, 35 insertions(+), 92 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4549a98b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b73b14d..43fc53a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * Move CompressionMetadata off-heap (CASSANDRA-4937)
* allow CLI to GET cql3 columnfamily data (CASSANDRA-4924)
* Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4549a98b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
--
diff --git a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
index 5587efc..81f99aa 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.io.compress;
import java.io.*;
import java.util.*;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.primitives.Longs;
import org.apache.cassandra.exceptions.ConfigurationException;
@@ -31,7 +32,7 @@ import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.CorruptSSTableException;
import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.util.FileUtils;
-import org.apache.cassandra.utils.BigLongArray;
+import org.apache.cassandra.io.util.Memory;
import org.apache.cassandra.utils.Pair;
/**
@@ -41,7 +42,7 @@ public class CompressionMetadata
{
public final long dataLength;
public final long compressedFileLength;
-private final BigLongArray chunkOffsets;
+private final Memory chunkOffsets;
public final String indexFilePath;
public final CompressionParameters parameters;
@@ -62,7 +63,7 @@ public class CompressionMetadata
return new
CompressionMetadata(desc.filenameFor(Component.COMPRESSION_INFO), new
File(dataFilePath).length());
}
-// This is package protected because of the tests.
+@VisibleForTesting
CompressionMetadata(String indexFilePath, long compressedLength)
{
this.indexFilePath = indexFilePath;
@@ -129,18 +130,18 @@ public class CompressionMetadata
*
* @return collection of the chunk offsets.
*/
-private BigLongArray readChunkOffsets(DataInput input)
+private Memory readChunkOffsets(DataInput input)
{
try
{
int chunkCount = input.readInt();
-BigLongArray offsets = new BigLongArray(chunkCount);
+Memory offsets = Memory.allocate(chunkCount * 8);
for (int i = 0; i chunkCount; i++)
{
try
{
-offsets.set(i, input.readLong());
+offsets.setLong(i * 8, input.readLong());
}
catch (EOFException e)
{
@@ -167,15 +168,15 @@ public class CompressionMetadata
public Chunk chunkFor(long position)
{
// position of the chunk
-int idx = (int) (position / parameters.chunkLength());
+int idx = 8 * (int) (position / parameters.chunkLength());
-if (idx = chunkOffsets.size)
+if (idx = chunkOffsets.size())
throw new CorruptSSTableException(new EOFException(),
indexFilePath);
-long chunkOffset = chunkOffsets.get(idx);
-long nextChunkOffset = (idx + 1 == chunkOffsets.size)
+long chunkOffset = chunkOffsets.getLong(idx);
+long nextChunkOffset = (idx + 8 == chunkOffsets.size())
[3/4] git commit: Move CompressionMetadata off-heap patch by jbellis and xedin for CASSANDRA-4937
Move CompressionMetadata off-heap
patch by jbellis and xedin for CASSANDRA-4937
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4549a98b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4549a98b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4549a98b
Branch: refs/heads/cassandra-1.2.0
Commit: 4549a98b4ef6c293c200cbd8e226aee2e574c2c0
Parents: 5c81f4b
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 20:47:27 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 20:47:27 2012 -0600
--
CHANGES.txt|1 +
.../cassandra/io/compress/CompressionMetadata.java | 35 ---
.../cassandra/io/util/CompressedSegmentedFile.java |2 +-
src/java/org/apache/cassandra/io/util/Memory.java | 12 +++
.../org/apache/cassandra/utils/BigLongArray.java | 77 ---
5 files changed, 35 insertions(+), 92 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4549a98b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b73b14d..43fc53a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * Move CompressionMetadata off-heap (CASSANDRA-4937)
* allow CLI to GET cql3 columnfamily data (CASSANDRA-4924)
* Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4549a98b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
--
diff --git a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
index 5587efc..81f99aa 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.io.compress;
import java.io.*;
import java.util.*;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.primitives.Longs;
import org.apache.cassandra.exceptions.ConfigurationException;
@@ -31,7 +32,7 @@ import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.CorruptSSTableException;
import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.util.FileUtils;
-import org.apache.cassandra.utils.BigLongArray;
+import org.apache.cassandra.io.util.Memory;
import org.apache.cassandra.utils.Pair;
/**
@@ -41,7 +42,7 @@ public class CompressionMetadata
{
public final long dataLength;
public final long compressedFileLength;
-private final BigLongArray chunkOffsets;
+private final Memory chunkOffsets;
public final String indexFilePath;
public final CompressionParameters parameters;
@@ -62,7 +63,7 @@ public class CompressionMetadata
return new
CompressionMetadata(desc.filenameFor(Component.COMPRESSION_INFO), new
File(dataFilePath).length());
}
-// This is package protected because of the tests.
+@VisibleForTesting
CompressionMetadata(String indexFilePath, long compressedLength)
{
this.indexFilePath = indexFilePath;
@@ -129,18 +130,18 @@ public class CompressionMetadata
*
* @return collection of the chunk offsets.
*/
-private BigLongArray readChunkOffsets(DataInput input)
+private Memory readChunkOffsets(DataInput input)
{
try
{
int chunkCount = input.readInt();
-BigLongArray offsets = new BigLongArray(chunkCount);
+Memory offsets = Memory.allocate(chunkCount * 8);
for (int i = 0; i chunkCount; i++)
{
try
{
-offsets.set(i, input.readLong());
+offsets.setLong(i * 8, input.readLong());
}
catch (EOFException e)
{
@@ -167,15 +168,15 @@ public class CompressionMetadata
public Chunk chunkFor(long position)
{
// position of the chunk
-int idx = (int) (position / parameters.chunkLength());
+int idx = 8 * (int) (position / parameters.chunkLength());
-if (idx = chunkOffsets.size)
+if (idx = chunkOffsets.size())
throw new CorruptSSTableException(new EOFException(),
indexFilePath);
-long chunkOffset = chunkOffsets.get(idx);
-long nextChunkOffset = (idx + 1 == chunkOffsets.size)
+long chunkOffset = chunkOffsets.getLong(idx);
+long nextChunkOffset = (idx + 8 == chunkOffsets.size())
[2/4] git commit: Move CompressionMetadata off-heap patch by jbellis and xedin for CASSANDRA-4937
Move CompressionMetadata off-heap
patch by jbellis and xedin for CASSANDRA-4937
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4549a98b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4549a98b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4549a98b
Branch: refs/heads/trunk
Commit: 4549a98b4ef6c293c200cbd8e226aee2e574c2c0
Parents: 5c81f4b
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Nov 9 20:47:27 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Nov 9 20:47:27 2012 -0600
--
CHANGES.txt|1 +
.../cassandra/io/compress/CompressionMetadata.java | 35 ---
.../cassandra/io/util/CompressedSegmentedFile.java |2 +-
src/java/org/apache/cassandra/io/util/Memory.java | 12 +++
.../org/apache/cassandra/utils/BigLongArray.java | 77 ---
5 files changed, 35 insertions(+), 92 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4549a98b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b73b14d..43fc53a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2-rc1
+ * Move CompressionMetadata off-heap (CASSANDRA-4937)
* allow CLI to GET cql3 columnfamily data (CASSANDRA-4924)
* Fix rare race condition in getExpireTimeForEndpoint (CASSANDRA-4402)
* acquire references to overlapping sstables during compaction so bloom filter
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4549a98b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
--
diff --git a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
index 5587efc..81f99aa 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.io.compress;
import java.io.*;
import java.util.*;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.primitives.Longs;
import org.apache.cassandra.exceptions.ConfigurationException;
@@ -31,7 +32,7 @@ import org.apache.cassandra.io.sstable.Component;
import org.apache.cassandra.io.sstable.CorruptSSTableException;
import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.util.FileUtils;
-import org.apache.cassandra.utils.BigLongArray;
+import org.apache.cassandra.io.util.Memory;
import org.apache.cassandra.utils.Pair;
/**
@@ -41,7 +42,7 @@ public class CompressionMetadata
{
public final long dataLength;
public final long compressedFileLength;
-private final BigLongArray chunkOffsets;
+private final Memory chunkOffsets;
public final String indexFilePath;
public final CompressionParameters parameters;
@@ -62,7 +63,7 @@ public class CompressionMetadata
return new
CompressionMetadata(desc.filenameFor(Component.COMPRESSION_INFO), new
File(dataFilePath).length());
}
-// This is package protected because of the tests.
+@VisibleForTesting
CompressionMetadata(String indexFilePath, long compressedLength)
{
this.indexFilePath = indexFilePath;
@@ -129,18 +130,18 @@ public class CompressionMetadata
*
* @return collection of the chunk offsets.
*/
-private BigLongArray readChunkOffsets(DataInput input)
+private Memory readChunkOffsets(DataInput input)
{
try
{
int chunkCount = input.readInt();
-BigLongArray offsets = new BigLongArray(chunkCount);
+Memory offsets = Memory.allocate(chunkCount * 8);
for (int i = 0; i chunkCount; i++)
{
try
{
-offsets.set(i, input.readLong());
+offsets.setLong(i * 8, input.readLong());
}
catch (EOFException e)
{
@@ -167,15 +168,15 @@ public class CompressionMetadata
public Chunk chunkFor(long position)
{
// position of the chunk
-int idx = (int) (position / parameters.chunkLength());
+int idx = 8 * (int) (position / parameters.chunkLength());
-if (idx = chunkOffsets.size)
+if (idx = chunkOffsets.size())
throw new CorruptSSTableException(new EOFException(),
indexFilePath);
-long chunkOffset = chunkOffsets.get(idx);
-long nextChunkOffset = (idx + 1 == chunkOffsets.size)
+long chunkOffset = chunkOffsets.getLong(idx);
+long nextChunkOffset = (idx + 8 == chunkOffsets.size())
[1/4] git commit: Merge branch 'cassandra-1.2' into trunk
Updated Branches: refs/heads/cassandra-1.2 5c81f4b92 - 4549a98b4 refs/heads/cassandra-1.2.0 5c81f4b92 - 4549a98b4 refs/heads/trunk d82c4408c - f9607c113 Merge branch 'cassandra-1.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9607c11 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9607c11 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9607c11 Branch: refs/heads/trunk Commit: f9607c11384e60505ede239ce409d3ff4b542a82 Parents: d82c440 4549a98 Author: Jonathan Ellis jbel...@apache.org Authored: Fri Nov 9 20:49:05 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Fri Nov 9 20:49:05 2012 -0600 -- CHANGES.txt|1 + .../cassandra/io/compress/CompressionMetadata.java | 35 --- .../cassandra/io/util/CompressedSegmentedFile.java |2 +- src/java/org/apache/cassandra/io/util/Memory.java | 12 +++ .../org/apache/cassandra/utils/BigLongArray.java | 77 --- 5 files changed, 35 insertions(+), 92 deletions(-) --
[jira] [Commented] (CASSANDRA-4912) BulkOutputFormat should support Hadoop MultipleOutput
[ https://issues.apache.org/jira/browse/CASSANDRA-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494542#comment-13494542 ] Michael Kjellman commented on CASSANDRA-4912: - okay so it looks like setting outputdir in the creation of the object is causing the problem. I moved setting outputdir into prepareWriter() and it looks like both sstables are created and streamed. [~brandon.williams] any reason the outputdir is created when the BulkRecordWriter object is created? BulkOutputFormat should support Hadoop MultipleOutput - Key: CASSANDRA-4912 URL: https://issues.apache.org/jira/browse/CASSANDRA-4912 Project: Cassandra Issue Type: New Feature Components: Hadoop Affects Versions: 1.2.0 beta 1 Reporter: Michael Kjellman Attachments: Example.java Much like CASSANDRA-4208 BOF should support outputting to Multiple Column Families. The current approach takken in the patch for COF results in only one stream being sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494543#comment-13494543 ] Jonathan Ellis commented on CASSANDRA-4937: --- Back up a second ... how are you opening multiple CRAR for the same sstable in a single request? Secondary index reads? CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4295) move checkaccess into statement.prepare
[ https://issues.apache.org/jira/browse/CASSANDRA-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko reassigned CASSANDRA-4295: Assignee: Aleksey Yeschenko (was: Pavel Yaskevich) move checkaccess into statement.prepare --- Key: CASSANDRA-4295 URL: https://issues.apache.org/jira/browse/CASSANDRA-4295 Project: Cassandra Issue Type: Improvement Components: API Affects Versions: 1.1.0 Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Priority: Minor Fix For: 1.2.0 Attachments: CASSANDRA-4295.patch there's no need to redo this every execution since the schema, tables, and users involved should all be immutable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4295) Implement caching of authorization results
[ https://issues.apache.org/jira/browse/CASSANDRA-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-4295: - Reviewer: jbellis (was: slebresne) Description: 1.2 will come with default IAuthority implementation that stores permissions in Cassandra, and each permission check will involve at least 1 Cassandra read. Some form of authorization result caching is very important for this scenario. (was: there's no need to redo this every execution since the schema, tables, and users involved should all be immutable) Priority: Major (was: Minor) Summary: Implement caching of authorization results (was: move checkaccess into statement.prepare) Implement caching of authorization results -- Key: CASSANDRA-4295 URL: https://issues.apache.org/jira/browse/CASSANDRA-4295 Project: Cassandra Issue Type: Improvement Components: API Affects Versions: 1.1.0 Reporter: Jonathan Ellis Assignee: Aleksey Yeschenko Fix For: 1.2.0 Attachments: CASSANDRA-4295.patch 1.2 will come with default IAuthority implementation that stores permissions in Cassandra, and each permission check will involve at least 1 Cassandra read. Some form of authorization result caching is very important for this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494545#comment-13494545 ] Pavel Yaskevich commented on CASSANDRA-4937: bq. You are welcome to think how to do so without involving more then one file, I have exploited lhf possibility That probably sounded confusing, I didn't mean to say that we are opening the same file multiple times per one request, I was just trying to say that I wanted to keep caching changes as local to CRAR as possible, without involving other files (or classes) in handling that cache, which allowed to preserve open/close calls without involving anything specific to CRAR into file handling. CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4937) CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap).
[ https://issues.apache.org/jira/browse/CASSANDRA-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13494549#comment-13494549 ] Jonathan Ellis commented on CASSANDRA-4937: --- But doesn't your caching only last for a single request? What is the point then if not opening the same file multiple times? CRAR improvements (object cache + CompressionMetadata chunk offset storage moved off-heap). --- Key: CASSANDRA-4937 URL: https://issues.apache.org/jira/browse/CASSANDRA-4937 Project: Cassandra Issue Type: Improvement Affects Versions: 1.1.6 Reporter: Pavel Yaskevich Assignee: Pavel Yaskevich Priority: Minor Fix For: 1.1.7 Attachments: CASSANDRA-4937.patch After good amount of testing on one of the clusters it was found that in order to improve read latency we need to minimize allocation rate that compression involves, that minimizes GC (as well as heap usage) and substantially decreases latency on read heavy workloads. I have also discovered that RAR skip cache harms performance in situation when reads are done in parallel with compaction working with relatively big SSTable files (few GB and more). The attached patch removes possibility to skip cache from compressed files (I can also add changes to RAR to remove skip cache functionality as a separate patch). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
