date:20140728

[jira] [Commented] (CASSANDRA-6977) attempting to create 10K column families fails with 100 node cluster

2014-07-28 Thread Michael Nelson (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075947#comment-14075947
]

Michael Nelson commented on CASSANDRA-6977:
---

This is a showstopper for a very large customer. They need the ability to
create new keyspaces as they add new customers. Their use case is
multi-tenancy, due to HIPPA and PCI, so that each customer is a separate
keyspace, keeping the data separate.

attempting to create 10K column families fails with 100 node cluster

Key: CASSANDRA-6977
URL: https://issues.apache.org/jira/browse/CASSANDRA-6977
Project: Cassandra
Issue Type: Bug
Environment: 100 nodes, Ubuntu 12.04.3 LTS, AWS m1.large instances
Reporter: Daniel Meyer
Assignee: Russ Hatch
Priority: Minor
Attachments: 100_nodes_all_data.png, all_data_5_nodes.png,
keyspace_create.py, logs.tar, tpstats.txt, visualvm_tracer_data.csv

During this test we are attempting to create a total of 1K keyspaces with 10
column families each to bring the total column families to 10K. With a 5
node cluster this operation can be completed; however, it fails with 100
nodes. Please see the two charts. For the 5 node case the time required to
create each keyspace and subsequent 10 column families increases linearly
until the number of keyspaces is 1K. For a 100 node cluster there is a
sudden increase in latency between 450 keyspaces and 550 keyspaces. The test
ends when the test script times out. After the test script times out it is
impossible to reconnect to the cluster with the datastax python driver
because it cannot connect to the host:
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
{'10.199.5.98': OperationTimedOut()}
It was found that running the following stress command does work from the
same machine the test script runs on.
cassandra-stress -d 10.199.5.98 -l 2 -e QUORUM -L3 -b -o INSERT
It should be noted that this test was initially done with DSE 4.0 and c*
version 2.0.5.24 and in that case it was not possible to run stress against
the cluster even locally on a node due to not finding the host.
Attached are system logs from one of the nodes, charts showing schema
creation latency for 5 and 100 node clusters and virtualvm tracer data for
cpu, memory, num_threads and gc runs, tpstat output and the test script.
The test script was on an m1.large aws instance outside of the cluster under
test.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (CASSANDRA-7575) Custom 2i validation

2014-07-28 Thread Sergio Bossa (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076044#comment-14076044
 ] 

Sergio Bossa commented on CASSANDRA-7575:
-

[~adelapena], following the review of your patch, I believe that while it works 
in practice, pulling the index searchers in SelectStatement#getRangeCommand and 
validating them that way is a bit odd, more specifically:
* SelectStatement#getRangeCommand may be called even if a 2i query is not 
present, so enforcing 2i validation there is a bit misleading and unexpected.
* SecondaryIndexSearcher#validate is called with the whole list of index 
expressions, which means each searcher implementation will have to go through 
the list to inspect each expression and decide if that specific expression was 
targeted for it and is wrong, or was just for another searcher.

I'd rather rework the patch in the following way:
* Add a SecondaryIndexManager#validateIndexSearchersForQuery method that works 
similarly to getIndexSearchersForQuery, but rather than just getting the index 
by each column, it also validates it against the proper column/expression by 
calling SecondaryIndexSearcher#validate(IndexExpression).
* Call SecondaryIndexManager#validateIndexSearchersForQuery from 
SelectStatement#RawStatement#validateSecondaryIndexSelections

That should improve encapsulation and responsibility placement and provide 
better 2i APIs.

Finally, I would add a few tests.

 Custom 2i validation
 

 Key: CASSANDRA-7575
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7575
 Project: Cassandra
  Issue Type: Improvement
  Components: API
Reporter: Andrés de la Peña
Assignee: Andrés de la Peña
Priority: Minor
  Labels: 2i, cql3, secondaryIndex, secondary_index, select
 Fix For: 2.1.0, 3.0

 Attachments: 2i_validation.patch


 There are several projects using custom secondary indexes as an extension 
 point to integrate C* with other systems such as Solr or Lucene. The usual 
 approach is to embed third party indexing queries in CQL clauses. 
 For example, [DSE 
 Search|http://www.datastax.com/what-we-offer/products-services/datastax-enterprise]
  embeds Solr syntax this way:
 {code}
 SELECT title FROM solr WHERE solr_query='title:natio*';
 {code}
 [Stratio platform|https://github.com/Stratio/stratio-cassandra] embeds custom 
 JSON syntax for searching in Lucene indexes:
 {code}
 SELECT * FROM tweets WHERE lucene='{
 filter : {
 type: range,
 field: time,
 lower: 2014/04/25,
 upper: 2014/04/1
 },
 query  : {
 type: phrase, 
 field: body, 
 values: [big, data]
 },
 sort  : {fields: [ {field:time, reverse:true} ] }
 }';
 {code}
 Tuplejump [Stargate|http://tuplejump.github.io/stargate/] also uses the 
 Stratio's open source JSON syntax:
 {code}
 SELECT name,company FROM PERSON WHERE stargate ='{
 filter: {
 type: range,
 field: company,
 lower: a,
 upper: p
 },
 sort:{
fields: [{field:name,reverse:true}]
 }
 }';
 {code}
 These syntaxes are validated by the corresponding 2i implementation. This 
 validation is done behind the StorageProxy command distribution. So, far as I 
 know, there is no way to give rich feedback about syntax errors to CQL users.
 I'm uploading a patch with some changes trying to improve this. I propose 
 adding an empty validation method to SecondaryIndexSearcher that can be 
 overridden by custom 2i implementations:
 {code}
 public void validate(ListIndexExpression clause) {}
 {code}
 And call it from SelectStatement#getRangeCommand:
 {code}
 ColumnFamilyStore cfs = 
 Keyspace.open(keyspace()).getColumnFamilyStore(columnFamily());
 for (SecondaryIndexSearcher searcher : 
 cfs.indexManager.getIndexSearchersForQuery(expressions))
 {
 try
 {
 searcher.validate(expressions);
 }
 catch (RuntimeException e)
 {
 String exceptionMessage = e.getMessage();
 if (exceptionMessage != null 
  !exceptionMessage.trim().isEmpty())
 throw new InvalidRequestException(
 Invalid index expression:  + e.getMessage());
 else
 throw new InvalidRequestException(
 Invalid index expression);
 }
 }
 {code}
 In this way C* allows custom 2i implementations to give feedback about syntax 
 errors.
 We are currently using these changes in a fork with no problems.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1

2014-07-28 Thread Marcus Eriksson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076068#comment-14076068
 ] 

Marcus Eriksson commented on CASSANDRA-7593:


no, they should not be empty

We are inserting a RangeTombstone with start='token' and end='token' (ie, 
delete the set for this row).

In 2.0 we only make the end have an EOC 
(https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/cql3/Sets.java#L234)
 while in 2.1 both do: 
https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/cql3/Sets.java#L252
 + 
https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/composites/AbstractComposite.java#L69

 Errors when upgrading through several versions to 2.1
 -

 Key: CASSANDRA-7593
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
 Project: Cassandra
  Issue Type: Bug
 Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
 Fix For: 2.1.0


 I'm seeing two different errors cropping up in the dtest which upgrades a 
 cluster through several versions.
 This is the more common error:
 {noformat}
 ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 - 
 Exception in thread Thread[GossipStage:10,5,main]
 java.lang.AssertionError: null
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
  ~[main/:na]
 at 
 org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
  ~[main/:na]
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
  ~[main/:na]
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
  ~[main/:na]
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
  ~[main/:na]
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
  ~[main/:na]
 at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345) 
 ~[main/:na]
 at 
 org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
  ~[main/:na]
 at 
 org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
  ~[main/:na]
 at 
 org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
  ~[main/:na]
 at 
 org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
  ~[main/:na]
 at 
 org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
  ~[main/:na]
 at 
 org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
  ~[main/:na]
 at 
 org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
  ~[main/:na]
 at 
 org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
  ~[main/:na]
 at 
 org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
  ~[main/:na]
 at 
 org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
  ~[main/:na]
 at 
 org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
  ~[main/:na]
 at 
 org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40) 
 ~[main/:na]
 at 
 org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
 ~[main/:na]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  ~[na:1.7.0_60]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  ~[na:1.7.0_60]
 at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
 {noformat}
 The same test sometimes fails with this exception instead:
 {noformat}
 ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168 
 - Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
 java.util.concurrent.RejectedExecutionException: Task 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9 
 rejected from 
 org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
  pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
 at 
 java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
  ~[na:1.7.0_60]
 at 
 java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) 
 ~[na:1.7.0_60]
 at

[jira] [Assigned] (CASSANDRA-7596) Don't swap min/max column names when mutating level or repairedAt

2014-07-28 Thread Marcus Eriksson (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson reassigned CASSANDRA-7596:
--

Assignee: Marcus Eriksson

 Don't swap min/max column names when mutating level or repairedAt
 -

 Key: CASSANDRA-7596
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7596
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
 Fix For: 2.1.0

 Attachments: 0001-dont-swap.patch


 Seems we swap min/max col names when mutating sstable metadata



--
This message was sent by Atlassian JIRA
(v6.2#6252)

git commit: Don't swap max/min column names when mutating sstable metadata.

2014-07-28 Thread marcuse

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1.0 6f15fe260 - ee62ae104


Don't swap max/min column names when mutating sstable metadata.

Patch by marcuse; reviewed by benedict for CASSANDRA-7596.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ee62ae10
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ee62ae10
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ee62ae10

Branch: refs/heads/cassandra-2.1.0
Commit: ee62ae104ee2c69d852b488f904b1854aa58aa2a
Parents: 6f15fe2
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Jul 28 12:48:24 2014 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Jul 28 12:48:24 2014 +0200

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/io/sstable/metadata/StatsMetadata.java  | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee62ae10/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 0a1ba51..c6aaef9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -14,6 +14,7 @@
  * Fix tracing of range slices and secondary index lookups that are local
to the coordinator (CASSANDRA-7599)
  * Set -Dcassandra.storagedir for all tool shell scripts (CASSANDRA-7587)
+ * Don't swap max/min col names when mutating sstable metadata (CASSANDRA-7596)
 Merged from 2.0:
  * Fix ReversedType(DateType) mapping to native protocol (CASSANDRA-7576)
  * Always merge ranges owned by a single node (CASSANDRA-6930)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee62ae10/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
index 900bd4e..a557b88 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
@@ -124,8 +124,8 @@ public class StatsMetadata extends MetadataComponent
  compressionRatio,
  estimatedTombstoneDropTime,
  newLevel,
- maxColumnNames,
  minColumnNames,
+ maxColumnNames,
  hasLegacyCounterShards,
  repairedAt);
 }
@@ -141,8 +141,8 @@ public class StatsMetadata extends MetadataComponent
  compressionRatio,
  estimatedTombstoneDropTime,
  sstableLevel,
- maxColumnNames,
  minColumnNames,
+ maxColumnNames,
  hasLegacyCounterShards,
  newRepairedAt);
 }

[1/2] git commit: Don't swap max/min column names when mutating sstable metadata.

2014-07-28 Thread marcuse

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 3744d7792 - 2236afb7a


Don't swap max/min column names when mutating sstable metadata.

Patch by marcuse; reviewed by benedict for CASSANDRA-7596.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ee62ae10
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ee62ae10
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ee62ae10

Branch: refs/heads/cassandra-2.1
Commit: ee62ae104ee2c69d852b488f904b1854aa58aa2a
Parents: 6f15fe2
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Jul 28 12:48:24 2014 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Jul 28 12:48:24 2014 +0200

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/io/sstable/metadata/StatsMetadata.java  | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee62ae10/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 0a1ba51..c6aaef9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -14,6 +14,7 @@
  * Fix tracing of range slices and secondary index lookups that are local
to the coordinator (CASSANDRA-7599)
  * Set -Dcassandra.storagedir for all tool shell scripts (CASSANDRA-7587)
+ * Don't swap max/min col names when mutating sstable metadata (CASSANDRA-7596)
 Merged from 2.0:
  * Fix ReversedType(DateType) mapping to native protocol (CASSANDRA-7576)
  * Always merge ranges owned by a single node (CASSANDRA-6930)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee62ae10/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
index 900bd4e..a557b88 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java
@@ -124,8 +124,8 @@ public class StatsMetadata extends MetadataComponent
  compressionRatio,
  estimatedTombstoneDropTime,
  newLevel,
- maxColumnNames,
  minColumnNames,
+ maxColumnNames,
  hasLegacyCounterShards,
  repairedAt);
 }
@@ -141,8 +141,8 @@ public class StatsMetadata extends MetadataComponent
  compressionRatio,
  estimatedTombstoneDropTime,
  sstableLevel,
- maxColumnNames,
  minColumnNames,
+ maxColumnNames,
  hasLegacyCounterShards,
  newRepairedAt);
 }

[2/2] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1

2014-07-28 Thread marcuse

Merge branch 'cassandra-2.1.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2236afb7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2236afb7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2236afb7

Branch: refs/heads/cassandra-2.1
Commit: 2236afb7a06725f9ceb13bab8c2180eb0d6134f5
Parents: 3744d77 ee62ae1
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Jul 28 12:49:06 2014 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Jul 28 12:49:06 2014 +0200

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/io/sstable/metadata/StatsMetadata.java  | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2236afb7/CHANGES.txt
--

[2/3] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1

2014-07-28 Thread marcuse

Merge branch 'cassandra-2.1.0' into cassandra-2.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2236afb7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2236afb7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2236afb7

Branch: refs/heads/trunk
Commit: 2236afb7a06725f9ceb13bab8c2180eb0d6134f5
Parents: 3744d77 ee62ae1
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Jul 28 12:49:06 2014 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Jul 28 12:49:06 2014 +0200

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/io/sstable/metadata/StatsMetadata.java  | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2236afb7/CHANGES.txt
--

[3/3] git commit: Merge branch 'cassandra-2.1' into trunk

2014-07-28 Thread marcuse

Merge branch 'cassandra-2.1' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0fd1a0bb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0fd1a0bb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0fd1a0bb

Branch: refs/heads/trunk
Commit: 0fd1a0bb47f66eaa29ce821aa4836c52b65e46e1
Parents: f3aa83b 2236afb
Author: Marcus Eriksson marc...@apache.org
Authored: Mon Jul 28 12:49:28 2014 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Mon Jul 28 12:49:28 2014 +0200

--
 CHANGES.txt  | 1 +
 .../org/apache/cassandra/io/sstable/metadata/StatsMetadata.java  | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0fd1a0bb/CHANGES.txt
--

81 matches

Mail list logo