[jira] [Resolved] (CASSANDRA-11314) Inconsistent select count(*)

2016-03-09 Thread Mircea Lemnaru (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mircea Lemnaru resolved CASSANDRA-11314.

Resolution: Invalid

> Inconsistent select count(*)
> 
>
> Key: CASSANDRA-11314
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11314
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: Ununtu 14.04 LTS
>Reporter: Mircea Lemnaru
>Assignee: Benjamin Lerer
> Attachments: testrun.log, vnodes_and_hosts
>
>
> Hello,
> I currently have this setup: 
> Cassandra 3.3 (Community edition downloaded from Datastax) installed on 3 
> nodes and I have created this table:
> CREATE TABLE billing.collected_data_day (
> collection_day int,
> timestamp timestamp,
> record_id uuid,
> dimensions map,
> entity_id text,
> measurements map,
> source_id text,
> PRIMARY KEY (collection_day, timestamp, record_id)
> ) WITH CLUSTERING ORDER BY (timestamp ASC, record_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> This table as you notice is partitioned by collection_day. This is because at 
> the end of the day we need to have fast access to all the data generated in a 
> day. collection day will be the x day from 1970
> In this table we have inserted roughly 12milion rows for testing purposes and 
> we did a simple count. As you can see the results vary ... 
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55341
> (1 rows)
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55372
> (1 rows)
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55300
> (1 rows)
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55300
> (1 rows)
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55300
> (1 rows)
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55303
> (1 rows)
> cqlsh:billing> select count(*) from collected_data_day where 
> collection_day=16462;
>  count
> ---
>  55374
> (1 rows)
> I am running the query from the seed node of the cassandra cluster. As you 
> can see most of the results are varying and I don't know the reason for this. 
> We are not writing anything into the cluster at this time , we are only 
> querying the cluster and only using this CQLSH.
> This is very similar to CASSANDRA-8940 but that is targeted for 2.1x
> Could it be that we are having the same issue in 3.3 ? 
> Please let me know what extra info I can provide



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11314) Inconsistent select count(*)

2016-03-09 Thread Mircea Lemnaru (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188869#comment-15188869
 ] 

Mircea Lemnaru commented on CASSANDRA-11314:


Hi [~blerer], I did some more tests over the last couple of days and found that 
there is not really a cassandra issue , it's rather something like a usage 
issue. Let me describe what I found ... 

So as I have mentioned before the cluster had 3 nodes node1 (seed node) 
,node2,node3 and my keyspace had replication factor = 2. Then I did more tests 
using the following scenario:

1. node1 UP , node2 DOWN, node3 DOWN , I tried to do the following query: 
select count(*) from collected_data_day where collection_day=16462 --> this 
query did not execute properly because node1 did not contain a replica of the 
row = 16462 this suggested to me that the data must be ONLY on nodes node2 and 
node3

2. node1 UP, node2 UP, node3 DOWN - executing the same query ( select count(*) 
from collected_data_day where collection_day=16462 ) 20 times consecutively 
resulted in exactly the same result ... COUNT = 55300 ... strange right ? 

3. node1 UP, node2 DOWN, node3 UP - same as 2 and I got COUNT = 55677 - every 
time , 20 times.

So this rang some bells , it seems that row = 16642 has 2 replicas = but the 
replicas are not identical , it seems that when the data was written the 
CONSISTENCY level was not properly set , in the sense that it was left default 
, and that is ONE if I am not mistaking.

I then proceeded to drop the table and reimport the data. This time I imported 
the data with CONSISTENCY = ALL and after this , all the counts were identical 
, no matter how many times I ran the query. 

So yeah ,this must have been the issue. I am guessing this can be closed as not 
a real issue / bug :)

PS: One thing that confused me all along was the COUNT values I got while the 
replicas were not identical ... if you remember , I got something like: 

Count (done on cassandra cluster): 55300
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55300
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55634
Count (done on cassandra cluster): 55530
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55342
Count (done on cassandra cluster): 55300
Count (done on cassandra cluster): 55630
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55297
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55352
Count (done on cassandra cluster): 55300
Count (done on cassandra cluster): 55677
Count (done on cassandra cluster): 55300

>From these results ... it's clear that 55300 came from counting the items from 
>node1 and that 55677 came from node3  but what about 55634, 55342, 55352 ? 
>Where were these results comming from ... ?

Anyway, I am glad this is not a bug but a configuration issue. Thank you.

> Inconsistent select count(*)
> 
>
> Key: CASSANDRA-11314
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11314
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: Ununtu 14.04 LTS
>Reporter: Mircea Lemnaru
>Assignee: Benjamin Lerer
> Attachments: testrun.log, vnodes_and_hosts
>
>
> Hello,
> I currently have this setup: 
> Cassandra 3.3 (Community edition downloaded from Datastax) installed on 3 
> nodes and I have created this table:
> CREATE TABLE billing.collected_data_day (
> collection_day int,
> timestamp timestamp,
> record_id uuid,
> dimensions map,
> entity_id text,
> measurements map,
> source_id text,
> PRIMARY KEY (collection_day, timestamp, record_id)
> ) WITH CLUSTERING ORDER BY (timestamp ASC, record_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> This table as you notice is partitioned by collection_day. This is because at 
> the end of the day we need to have fast access to all the data generated in a 
> day.

[jira] [Commented] (CASSANDRA-10331) Establish and implement canonical bulk reading workload(s)

2016-03-09 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188861#comment-15188861
 ] 

Stefania commented on CASSANDRA-10331:
--

In terms of establishing the canonical bulk reading workload, I suggest using 
the profiles available in [this 
benchmark|https://github.com/stef1927/cstar_bulk_read_test] with single 
clustering and row sizes of approximately 100 bytes, 1 kb and 10 kb. These 
correspond to the following user profiles:

* 
[bulk_read_value50_cluster1.yaml|https://github.com/stef1927/cstar_bulk_read_test/blob/master/bulk_read_value50_cluster1.yaml]
* 
[bulk_read_value500_cluster1.yaml|https://github.com/stef1927/cstar_bulk_read_test/blob/master/bulk_read_value500_cluster1.yaml]
* 
[bulk_read_value5k_cluster1.yaml|https://github.com/stef1927/cstar_bulk_read_test/blob/master/bulk_read_value5k_cluster1.yaml]

Here are sample cassandra-stress commands to generate the data and read it back:

{code}
cassandra-stress user profile=bulk_read_value50_cluster1.yaml ops\(insert=1\) 
n=5M -rate threads=25
cassandra-stress user profile=bulk_read_value50_cluster1.yaml 
ops\(all_columns_tr_query=1\) -rate threads=25
{code}



> Establish and implement canonical bulk reading workload(s)
> --
>
> Key: CASSANDRA-10331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10331
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ariel Weisberg
>Assignee: Stefania
> Fix For: 3.x
>
>
> Implement a client, use stress, or extend stress to a bulk reading workload 
> that is indicative of the performance we are trying to improve.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (CASSANDRA-10944) ERROR [CompactionExecutor] CassandraDaemon.java Exception in thread

2016-03-09 Thread Alexey Ivanchin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Ivanchin reopened CASSANDRA-10944:
-
Reproduced In: 3.3
Since Version: 3.0.1

ERROR [CompactionExecutor:2458] 2016-03-10 12:41:15,127 
CassandraDaemon.java:195 - Exception in thread 
Thread[CompactionExecutor:2458,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.db.rows.BufferCell.(BufferCell.java:49) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:88) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:83) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.purge(BufferCell.java:175) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.ComplexColumnData.lambda$purge$107(ComplexColumnData.java:165)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.ComplexColumnData$$Lambda$68/1224572667.apply(Unknown
 Source) ~[na:na]
at 
org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.ComplexColumnData.transformAndFilter(ComplexColumnData.java:170)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:165)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:43) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.BTreeRow.lambda$purge$102(BTreeRow.java:333) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.BTreeRow$$Lambda$67/1968133513.apply(Unknown 
Source) ~[na:na]
at 
org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:338) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:333) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:116) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
 ~[apache-cassandra-3.3.0.jar:3.3.0]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_51]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_51]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_51]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]


> ERROR [CompactionExecutor] CassandraDaemon.java  Exception in thread 
>

[jira] [Updated] (CASSANDRA-11301) Non-obsoleting compaction operations over compressed files can impose rate limit on normal reads

2016-03-09 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-11301:
-
Status: Patch Available  (was: In Progress)

> Non-obsoleting compaction operations over compressed files can impose rate 
> limit on normal reads
> 
>
> Key: CASSANDRA-11301
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11301
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Benedict
>Assignee: Stefania
> Fix For: 2.2.6
>
>
> Broken by CASSANDRA-9240; the rate limiting reader passes the ICompressedFile 
> interface to its parent, which uses this to attach an "owner" - which means 
> the reader gets recycled on close, i.e. pooled, for normal use. If the 
> compaction were to replace the sstable there would be no problem, which is 
> presumably why this hasn't been encountered frequently. However validation 
> compactions on long lived sstables would permit these rate limited readers to 
> accumulate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11301) Non-obsoleting compaction operations over compressed files can impose rate limit on normal reads

2016-03-09 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188838#comment-15188838
 ] 

Stefania commented on CASSANDRA-11301:
--

It turns out there is an expiry time of 512 milliseconds on the cache entries, 
so I've opted for the unit test.

Provided we don't stop with a breakpoint, the unit test fails without the patch 
and passes with the patch applied. I've also cleaned up a 
{{FileNotFoundException}} that was not thrown any longer. The patch should be 
committed to 2.2 only. I'm going to look for a reviewer on IRC.



> Non-obsoleting compaction operations over compressed files can impose rate 
> limit on normal reads
> 
>
> Key: CASSANDRA-11301
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11301
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Benedict
>Assignee: Stefania
> Fix For: 2.2.6
>
>
> Broken by CASSANDRA-9240; the rate limiting reader passes the ICompressedFile 
> interface to its parent, which uses this to attach an "owner" - which means 
> the reader gets recycled on close, i.e. pooled, for normal use. If the 
> compaction were to replace the sstable there would be no problem, which is 
> presumably why this hasn't been encountered frequently. However validation 
> compactions on long lived sstables would permit these rate limited readers to 
> accumulate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM

2016-03-09 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188755#comment-15188755
 ] 

Stefania commented on CASSANDRA-11320:
--

Thank you for your suggestion, we'll try to keep track of in-flight requests as 
well. Due to the load balancing policy, it may not be possible to know for sure 
which host is handling a request, but we can keep track of the total number of 
messages, or the number for each intended replica.

> Improve backoff policy for cqlsh COPY FROM
> --
>
> Key: CASSANDRA-11320
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11320
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
>
> Currently we have an exponential back-off policy in COPY FROM that kicks in 
> when timeouts are received. However there are two limitations:
> * it does not cover new requests and therefore we may not back-off 
> sufficiently to give time to an overloaded server to recover
> * the pause is performed in the receiving thread and therefore we may not 
> process server messages quickly enough
> There is a static throttling mechanism in rows per second from feeder to 
> worker processes (the INGESTRATE) but the feeder has no idea of the load of 
> each worker process. However it's easy to keep track of how many chunks a 
> worker process has yet to read by introducing a bounded semaphore.
> The idea is to move the back-off pauses to the worker processes main thread 
> so as to include all messages, new and retries, not just the retries that 
> timed out. The worker process will not read new chunks during the back-off 
> pauses, and the feeder process can then look at the number of pending chunks 
> before sending new chunks to a worker process.
> [~aholmber], [~aweisberg] what do you think?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10331) Establish and implement canonical bulk reading workload(s)

2016-03-09 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-10331:
-
Status: Patch Available  (was: In Progress)

> Establish and implement canonical bulk reading workload(s)
> --
>
> Key: CASSANDRA-10331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10331
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ariel Weisberg
>Assignee: Stefania
> Fix For: 3.x
>
>
> Implement a client, use stress, or extend stress to a bulk reading workload 
> that is indicative of the performance we are trying to improve.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10331) Establish and implement canonical bulk reading workload(s)

2016-03-09 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-10331:
-
Status: Ready to Commit  (was: Patch Available)

> Establish and implement canonical bulk reading workload(s)
> --
>
> Key: CASSANDRA-10331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10331
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ariel Weisberg
>Assignee: Stefania
> Fix For: 3.x
>
>
> Implement a client, use stress, or extend stress to a bulk reading workload 
> that is indicative of the performance we are trying to improve.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10331) Establish and implement canonical bulk reading workload(s)

2016-03-09 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188639#comment-15188639
 ] 

Stefania commented on CASSANDRA-10331:
--

Thank you for reviewing the latest changes.

Committing to 3.5 is fine with me. I'm not sure if this requires committing to 
3.0.5 as well, so I prepared all 3 patches and started CI on all of them:

||3.0||3.5||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/10331-3.0]|[patch|https://github.com/stef1927/cassandra/commits/10331-3.5]|[patch|https://github.com/stef1927/cassandra/commits/10331]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10331-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10331-3.5-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10331-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10331-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10331-3.5-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10331-dtest/]|

There is a conflict from 3.0 to 3.5 whilst the 3.5 patch applies cleanly to 
trunk. If you decide to commit only to 3.5 or trunk, then you should change 
CHANGES.txt. CI results are still pending.

> Establish and implement canonical bulk reading workload(s)
> --
>
> Key: CASSANDRA-10331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10331
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ariel Weisberg
>Assignee: Stefania
> Fix For: 3.x
>
>
> Implement a client, use stress, or extend stress to a bulk reading workload 
> that is indicative of the performance we are trying to improve.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false

2016-03-09 Thread Brandon Williams (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-11168:
-
Status: Patch Available  (was: Open)

> Hint Metrics are updated even if hinted_hand-offs=false
> ---
>
> Key: CASSANDRA-11168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11168
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anubhav Kale
>Assignee: Anubhav Kale
>Priority: Minor
> Attachments: 0001-Hinted-Handoff-Fix.patch, 
> 0001-Hinted-handoff-metrics.patch
>
>
> In our PROD logs, we noticed a lot of hint metrics even though we have 
> disabled hinted handoffs.
> The reason is StorageProxy.ShouldHint has an inverted if condition. 
> We should also wrap the if (hintWindowExpired) block in if 
> (DatabaseDescriptor.hintedHandoffEnabled()).
> The fix is easy, and I can provide a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11333) cqlsh: COPY FROM should check that explicit column names are valid

2016-03-09 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-11333:
-
Description: 
If an invalid column is specified in the COPY FROM command, then it fails 
without an appropriate error notification.

For example using this schema:

{code}
CREATE TABLE bulk_read.value500k_cluster1 (
pk int,
c1 int,
v1 text,
v2 text,
PRIMARY KEY (pk, c1)
);
{code}

and this COPY FROM command (note the third column name is wrong:

{code}
COPY bulk_read.value500k_cluster1 (pk, c1, vv, v2) FROM 'test.csv';
{code}

we get the following error:

{code}
Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
'v2'].
1 child process(es) died unexpectedly, aborting
Processed: 0 rows; Rate:   0 rows/s; Avg. rate:   0 rows/s
0 rows imported from 0 files in 0.109 seconds (0 skipped).
{code}

Running cqlsh with {{--debug}} reveals where the problem is:

{code}
Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
'v2'].
Traceback (most recent call last):
  File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", line 
2005, in run
self.inner_run(*self.make_params())
  File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", line 
2027, in make_params
is_counter = ("counter" in [table_meta.columns[name].cql_type for name in 
self.valid_columns])
{code}

The parent process should check that all column names are valid and output an 
appropriate error message rather than letting worker processes crash.

  was:
If an invalid column is specified in the COPY FROM command, then it fails 
without an appropriate error notification.

For example using this schema:

{code}
CREATE TABLE bulk_read.value500k_cluster1 (
pk int,
c1 int,
v1 text,
v2 text,
PRIMARY KEY (pk, c1)
);
{code}

and this COPY FROM command (note the third column name is wrong:

{code}
COPY bulk_read.value500k_cluster1 (pk, c1, vv, v2) FROM 'test.csv';
{code}

we get the following error:

{code}
Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
'v2'].
1 child process(es) died unexpectedly, aborting
Processed: 0 rows; Rate:   0 rows/s; Avg. rate:   0 rows/s
0 rows imported from 0 files in 0.109 seconds (0 skipped).
{code}

Running cqlsh with {{--debug}} reveals where the problem is:

{code}
Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
'v2'].
Traceback (most recent call last):
  File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", line 
2005, in run
self.inner_run(*self.make_params())
  File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", line 
2027, in make_params
is_counter = ("counter" in [table_meta.columns[name].cql_type for name in 
self.valid_columns])
{code}

The parent process should check that all column names are valid and output an 
appropriate error message rather than letting worker processes die.


> cqlsh: COPY FROM should check that explicit column names are valid
> --
>
> Key: CASSANDRA-11333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11333
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
>Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> If an invalid column is specified in the COPY FROM command, then it fails 
> without an appropriate error notification.
> For example using this schema:
> {code}
> CREATE TABLE bulk_read.value500k_cluster1 (
> pk int,
> c1 int,
> v1 text,
> v2 text,
> PRIMARY KEY (pk, c1)
> );
> {code}
> and this COPY FROM command (note the third column name is wrong:
> {code}
> COPY bulk_read.value500k_cluster1 (pk, c1, vv, v2) FROM 'test.csv';
> {code}
> we get the following error:
> {code}
> Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
> 'v2'].
> 1 child process(es) died unexpectedly, aborting
> Processed: 0 rows; Rate:   0 rows/s; Avg. rate:   0 rows/s
> 0 rows imported from 0 files in 0.109 seconds (0 skipped).
> {code}
> Running cqlsh with {{--debug}} reveals where the problem is:
> {code}
> Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
> 'v2'].
> Traceback (most recent call last):
>   File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", 
> line 2005, in run
> self.inner_run(*self.make_params())
>   File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", 
> line 2027, in make_params
> is_counter = ("counter" in [table_meta.columns[name].cql_type for name in 
> self.valid_columns])
> {code}
> The parent process should check that all column names are valid and output an 
> appropriate error

[jira] [Created] (CASSANDRA-11333) cqlsh: COPY FROM should check that explicit column names are valid

2016-03-09 Thread Stefania (JIRA) 
Stefania created CASSANDRA-11333:


 Summary: cqlsh: COPY FROM should check that explicit column names 
are valid
 Key: CASSANDRA-11333
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11333
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Stefania
Assignee: Stefania
Priority: Minor
 Fix For: 2.2.x, 3.0.x, 3.x


If an invalid column is specified in the COPY FROM command, then it fails 
without an appropriate error notification.

For example using this schema:

{code}
CREATE TABLE bulk_read.value500k_cluster1 (
pk int,
c1 int,
v1 text,
v2 text,
PRIMARY KEY (pk, c1)
);
{code}

and this COPY FROM command (note the third column name is wrong:

{code}
COPY bulk_read.value500k_cluster1 (pk, c1, vv, v2) FROM 'test.csv';
{code}

we get the following error:

{code}
Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
'v2'].
1 child process(es) died unexpectedly, aborting
Processed: 0 rows; Rate:   0 rows/s; Avg. rate:   0 rows/s
0 rows imported from 0 files in 0.109 seconds (0 skipped).
{code}

Running cqlsh with {{--debug}} reveals where the problem is:

{code}
Starting copy of bulk_read.value500k_cluster1 with columns ['pk', 'c1', 'vv', 
'v2'].
Traceback (most recent call last):
  File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", line 
2005, in run
self.inner_run(*self.make_params())
  File "/home/automaton/cassandra-src/bin/../pylib/cqlshlib/copyutil.py", line 
2027, in make_params
is_counter = ("counter" in [table_meta.columns[name].cql_type for name in 
self.valid_columns])
{code}

The parent process should check that all column names are valid and output an 
appropriate error message rather than letting worker processes die.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11274) cqlsh: interpret CQL type for formatting blob types

2016-03-09 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188458#comment-15188458
 ] 

Stefania commented on CASSANDRA-11274:
--

This is ready for commit, on trunk only:

|[patch|https://github.com/stef1927/cassandra/commits/11274]|[dtests 
results|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11274-dtest/]|

> cqlsh: interpret CQL type for formatting blob types
> ---
>
> Key: CASSANDRA-11274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11274
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
>
> During the development of CASSANDRA-11053 we have added changes to the cqlsh 
> formatting code so that we can format {{blob}} types correctly even if they 
> are represented as {{str}} rather than {{bytearray}}.
> At the moment we ensure {{blob}} are of type {{bytearray}} via the following 
> shortcut:
> {code}
> cassandra.cqltypes.BytesType.deserialize = staticmethod(lambda byts, 
> protocol_version: bytearray(byts))
> {code}
> After CASSANDRA-11053 is committed there will be a similar shortcut to 
> override the fast serializers implemented in cython. 
> Decoding the CQL type is safer in that it decouples cqlsh formatting from the 
> types returned by the driver deserializers but it is also unnecessary for 
> CASSANDRA-11053 performance goals and risky for older releases. 
> Therefore this ticket delivers this functionality but only on trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11274) cqlsh: interpret CQL type for formatting blob types

2016-03-09 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-11274:
-
Status: Ready to Commit  (was: Patch Available)

> cqlsh: interpret CQL type for formatting blob types
> ---
>
> Key: CASSANDRA-11274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11274
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
>
> During the development of CASSANDRA-11053 we have added changes to the cqlsh 
> formatting code so that we can format {{blob}} types correctly even if they 
> are represented as {{str}} rather than {{bytearray}}.
> At the moment we ensure {{blob}} are of type {{bytearray}} via the following 
> shortcut:
> {code}
> cassandra.cqltypes.BytesType.deserialize = staticmethod(lambda byts, 
> protocol_version: bytearray(byts))
> {code}
> After CASSANDRA-11053 is committed there will be a similar shortcut to 
> override the fast serializers implemented in cython. 
> Decoding the CQL type is safer in that it decouples cqlsh formatting from the 
> types returned by the driver deserializers but it is also unnecessary for 
> CASSANDRA-11053 performance goals and risky for older releases. 
> Therefore this ticket delivers this functionality but only on trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11327) Maintain a histogram of times when writes are blocked due to no available memory

2016-03-09 Thread Benedict (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188351#comment-15188351
 ] 

Benedict commented on CASSANDRA-11327:
--

No; they're about actually freeing the memory.  

The point of memtables is that they completely mask latency until you exceed 
write throughput by total system buffer capacity.  The idea being that the 
cluster should always be provisioned above that level, since it's for real-time 
service provision.  Any rate limit of the kind you describe would artificially 
introduce latency at all other times, i.e. when a healthy cluster would have 
none.  

Certainly there are schemes that are better than others, such as calculating 
the difference between allocation rate and flush rate, applying a rate limit 
when one exceeds the other, by an amount inversely proportional to the amount 
of free space (i.e. so that the latency adulteration only occurs as you 
approach overload).

Actually reclaiming space as flush progresses has the advantage of only 
introducing latency only when absolutely necessary, but also ensures progress 
to queries at the disk throughput limit of the cluster.

> Maintain a histogram of times when writes are blocked due to no available 
> memory
> 
>
> Key: CASSANDRA-11327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11327
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Ariel Weisberg
>
> I have a theory that part of the reason C* is so sensitive to timeouts during 
> saturating write load is that throughput is basically a sawtooth with valleys 
> at zero. This is something I have observed and it gets worse as you add 2i to 
> a table or do anything that decreases the throughput of flushing.
> I think the fix for this is to incrementally release memory pinned by 
> memtables and 2i during flushing instead of releasing it all at once. I know 
> that's not really possible, but we can fake it with memory accounting that 
> tracks how close to completion flushing is and releases permits for 
> additional memory. This will lead to a bit of a sawtooth in real memory 
> usage, but we can account for that so the peak footprint is the same.
> I think the end result of this change will be a sawtooth, but the valley of 
> the sawtooth will not be zero it will be the rate at which flushing 
> progresses. Optimizing the rate at which flushing progresses and it's 
> fairness with other work can then be tackled separately.
> Before we do this I think we should demonstrate that pinned memory due to 
> flushing is actually the issue by getting better visibility into the 
> distribution of instances of not having any memory by maintaining a histogram 
> of spans of time where no memory is available and a thread is blocked.
> [MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
>  should be a relatively straightforward entry point for this. The first 
> thread to block can mark the start of memory starvation and the last thread 
> out can mark the end. Have a periodic task that tracks the amount of time 
> spent blocked per interval of time and if it is greater than some threshold 
> log with more details, possibly at debug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11286) streaming socket never times out

2016-03-09 Thread Yuki Morishita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-11286:
---
Status: Ready to Commit  (was: Patch Available)

> streaming socket never times out
> 
>
> Key: CASSANDRA-11286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11286
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>
> While trying to reproduce CASSANDRA-8343 I was not able to trigger a 
> {{SocketTimeoutException}} by adding an artificial sleep longer than 
> {{streaming_socket_timeout_in_ms}}.
> After investigation, I detected two problems:
> * {{ReadableByteChannel}} creation via {{socket.getChannel()}}, as done in 
> {{ConnectionHandler.getReadChannel(socket)}}, does not respect 
> {{socket.setSoTimeout()}}, as explained in this [blog 
> post|https://technfun.wordpress.com/2009/01/29/networking-in-java-non-blocking-nio-blocking-nio-and-io/]
> ** bq. The only difference between “blocking NIO” and “NIO wrapped around IO” 
> is that you can’t use socket timeout with SocketChannels. Why ? Read a 
> javadoc for setSocketTimeout(). It says that this timeout is used only by 
> streams.
> * {{socketSoTimeout}} is never set on "follower" side, only on initiator side 
> via {{DefaultConnectionFactory.createConnection(peer)}}.
> This may cause streaming to hang indefinitely, as exemplified by 
> CASSANDRA-8621:
> bq. For the scenario that prompted this ticket, it appeared that the 
> streaming process was completely stalled. One side of the stream (the sender 
> side) had an exception that appeared to be a connection reset. The receiving 
> side appeared to think that the connection was still active, at least in 
> terms of the netstats reported by nodetool. We were unable to verify whether 
> this was specifically the case in terms of connected sockets due to the fact 
> that there were multiple streams for those peers, and there is no simple way 
> to correlate a specific stream to a tcp session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11286) streaming socket never times out

2016-03-09 Thread Yuki Morishita (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188344#comment-15188344
 ] 

Yuki Morishita commented on CASSANDRA-11286:


+1. Will commit.

> streaming socket never times out
> 
>
> Key: CASSANDRA-11286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11286
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>
> While trying to reproduce CASSANDRA-8343 I was not able to trigger a 
> {{SocketTimeoutException}} by adding an artificial sleep longer than 
> {{streaming_socket_timeout_in_ms}}.
> After investigation, I detected two problems:
> * {{ReadableByteChannel}} creation via {{socket.getChannel()}}, as done in 
> {{ConnectionHandler.getReadChannel(socket)}}, does not respect 
> {{socket.setSoTimeout()}}, as explained in this [blog 
> post|https://technfun.wordpress.com/2009/01/29/networking-in-java-non-blocking-nio-blocking-nio-and-io/]
> ** bq. The only difference between “blocking NIO” and “NIO wrapped around IO” 
> is that you can’t use socket timeout with SocketChannels. Why ? Read a 
> javadoc for setSocketTimeout(). It says that this timeout is used only by 
> streams.
> * {{socketSoTimeout}} is never set on "follower" side, only on initiator side 
> via {{DefaultConnectionFactory.createConnection(peer)}}.
> This may cause streaming to hang indefinitely, as exemplified by 
> CASSANDRA-8621:
> bq. For the scenario that prompted this ticket, it appeared that the 
> streaming process was completely stalled. One side of the stream (the sender 
> side) had an exception that appeared to be a connection reset. The receiving 
> side appeared to think that the connection was still active, at least in 
> terms of the netstats reported by nodetool. We were unable to verify whether 
> this was specifically the case in terms of connected sockets due to the fact 
> that there were multiple streams for those peers, and there is no simple way 
> to correlate a specific stream to a tcp session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11327) Maintain a histogram of times when writes are blocked due to no available memory

2016-03-09 Thread Ariel Weisberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188316#comment-15188316
 ] 

Ariel Weisberg commented on CASSANDRA-11327:


I'm not proposing reclaiming the space. I am just proposing easing backpressure 
as flushing progresses. It's an accounting change. The memory will still be 
fully committed until the memtable is completely flushed. Or is that an idea 
discussed in those threads? They all seem a bit orthogonal and are also focused 
on changing the data structures.

> Maintain a histogram of times when writes are blocked due to no available 
> memory
> 
>
> Key: CASSANDRA-11327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11327
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Ariel Weisberg
>
> I have a theory that part of the reason C* is so sensitive to timeouts during 
> saturating write load is that throughput is basically a sawtooth with valleys 
> at zero. This is something I have observed and it gets worse as you add 2i to 
> a table or do anything that decreases the throughput of flushing.
> I think the fix for this is to incrementally release memory pinned by 
> memtables and 2i during flushing instead of releasing it all at once. I know 
> that's not really possible, but we can fake it with memory accounting that 
> tracks how close to completion flushing is and releases permits for 
> additional memory. This will lead to a bit of a sawtooth in real memory 
> usage, but we can account for that so the peak footprint is the same.
> I think the end result of this change will be a sawtooth, but the valley of 
> the sawtooth will not be zero it will be the rate at which flushing 
> progresses. Optimizing the rate at which flushing progresses and it's 
> fairness with other work can then be tackled separately.
> Before we do this I think we should demonstrate that pinned memory due to 
> flushing is actually the issue by getting better visibility into the 
> distribution of instances of not having any memory by maintaining a histogram 
> of spans of time where no memory is available and a thread is blocked.
> [MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
>  should be a relatively straightforward entry point for this. The first 
> thread to block can mark the start of memory starvation and the last thread 
> out can mark the end. Have a periodic task that tracks the amount of time 
> spent blocked per interval of time and if it is greater than some threshold 
> log with more details, possibly at debug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11327) Maintain a histogram of times when writes are blocked due to no available memory

2016-03-09 Thread Benedict (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188309#comment-15188309
 ] 

Benedict commented on CASSANDRA-11327:
--

That line not only already supports it, it was intended to do this at the point 
of writing.  In fact I'm not sure why it isn't already.  

The call to {{parent.hasRoom().register()}} can simply be provided a 
{{TimerContext}}, i.e. {{parent.hasRoom().register(TimerContext)}}

As regards the incremental release of memory you're about two years late to 
that party - see the abandoned (but fully functioning at the time) branch 
[here|https://github.com/belliottsmith/cassandra/tree/6843-offheap.gc]. See the 
related discussion on CASSANDRA-6689, CASSANDRA-6694 and CASSANDRA-6843.  
Ultimately it was unpalatable to the project.  Possibly with thread-per-core a 
more palatable approach will be viable.




> Maintain a histogram of times when writes are blocked due to no available 
> memory
> 
>
> Key: CASSANDRA-11327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11327
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Ariel Weisberg
>
> I have a theory that part of the reason C* is so sensitive to timeouts during 
> saturating write load is that throughput is basically a sawtooth with valleys 
> at zero. This is something I have observed and it gets worse as you add 2i to 
> a table or do anything that decreases the throughput of flushing.
> I think the fix for this is to incrementally release memory pinned by 
> memtables and 2i during flushing instead of releasing it all at once. I know 
> that's not really possible, but we can fake it with memory accounting that 
> tracks how close to completion flushing is and releases permits for 
> additional memory. This will lead to a bit of a sawtooth in real memory 
> usage, but we can account for that so the peak footprint is the same.
> I think the end result of this change will be a sawtooth, but the valley of 
> the sawtooth will not be zero it will be the rate at which flushing 
> progresses. Optimizing the rate at which flushing progresses and it's 
> fairness with other work can then be tackled separately.
> Before we do this I think we should demonstrate that pinned memory due to 
> flushing is actually the issue by getting better visibility into the 
> distribution of instances of not having any memory by maintaining a histogram 
> of spans of time where no memory is available and a thread is blocked.
> [MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
>  should be a relatively straightforward entry point for this. The first 
> thread to block can mark the start of memory starvation and the last thread 
> out can mark the end. Have a periodic task that tracks the amount of time 
> spent blocked per interval of time and if it is greater than some threshold 
> log with more details, possibly at debug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false

2016-03-09 Thread Anubhav Kale (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anubhav Kale updated CASSANDRA-11168:
-
Attachment: 0001-Hinted-handoff-metrics.patch

> Hint Metrics are updated even if hinted_hand-offs=false
> ---
>
> Key: CASSANDRA-11168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11168
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anubhav Kale
>Assignee: Anubhav Kale
>Priority: Minor
> Attachments: 0001-Hinted-Handoff-Fix.patch, 
> 0001-Hinted-handoff-metrics.patch
>
>
> In our PROD logs, we noticed a lot of hint metrics even though we have 
> disabled hinted handoffs.
> The reason is StorageProxy.ShouldHint has an inverted if condition. 
> We should also wrap the if (hintWindowExpired) block in if 
> (DatabaseDescriptor.hintedHandoffEnabled()).
> The fix is easy, and I can provide a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false

2016-03-09 Thread Anubhav Kale (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188197#comment-15188197
 ] 

Anubhav Kale commented on CASSANDRA-11168:
--

Updated patch. Not really sure if its really necessary to be back-ported though.

> Hint Metrics are updated even if hinted_hand-offs=false
> ---
>
> Key: CASSANDRA-11168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11168
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anubhav Kale
>Assignee: Anubhav Kale
>Priority: Minor
> Attachments: 0001-Hinted-Handoff-Fix.patch, 
> 0001-Hinted-handoff-metrics.patch
>
>
> In our PROD logs, we noticed a lot of hint metrics even though we have 
> disabled hinted handoffs.
> The reason is StorageProxy.ShouldHint has an inverted if condition. 
> We should also wrap the if (hintWindowExpired) block in if 
> (DatabaseDescriptor.hintedHandoffEnabled()).
> The fix is easy, and I can provide a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11332) nodes connect to themselves when NTS is used

2016-03-09 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188098#comment-15188098
 ] 

Brandon Williams commented on CASSANDRA-11332:
--

I should note that all my nodes are in a single DC, and I reproduced this on 
2.1 head but it probably runs through all the versions.

> nodes connect to themselves when NTS is used
> 
>
> Key: CASSANDRA-11332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11332
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
> Fix For: 2.1.x
>
>
> I tested this with both the simple snitch and PFS.  It's quite easy to repro, 
> setup a cluster, start it.  Mine looks like this:
> {noformat}
> tcp0  0 10.208.8.123:48003  10.208.8.63:7000
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.8.63:40215   
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:9  10.208.35.225:7000  
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:33498  10.208.8.63:7000
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.35.225:52530 
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.35.225:53674 
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:40846  10.208.35.225:7000  
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.8.63:48880   
> ESTABLISHED 26254/java
> {noformat}
> No problems so far.  Now create a keyspace using NTS with an rf of 3, and 
> perform some writes.  Now it looks like this:
> {noformat}
> tcp0  0 10.208.8.123:48003  10.208.8.63:7000
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.123:35024  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:35024  10.208.8.123:7000   
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:47212  10.208.8.123:7000   
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.63:40215   
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:9  10.208.35.225:7000  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:33498  10.208.8.63:7000
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.35.225:52530 
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.35.225:53674 
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.123:47212  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:40846  10.208.35.225:7000  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.63:48880   
> ESTABLISHED 26254/java  
> {noformat}
> I can't think of any reason for a node to connect to itself, and this can 
> cause problems with PFS where you might only define the broadcast addresses, 
> but now you need the internal addresses too because the node will need to 
> look itself up when connecting to itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11332) nodes connect to themselves when NTS is used

2016-03-09 Thread Brandon Williams (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-11332:
-
Description: 
I tested this with both the simple snitch and PFS.  It's quite easy to repro, 
setup a cluster, start it.  Mine looks like this:

{noformat}
tcp0  0 10.208.8.123:48003  10.208.8.63:7000ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.8.63:40215   ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:9  10.208.35.225:7000  ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:33498  10.208.8.63:7000ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.35.225:52530 ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.35.225:53674 ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:40846  10.208.35.225:7000  ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.8.63:48880   ESTABLISHED 
26254/java
{noformat}

No problems so far.  Now create a keyspace using NTS with an rf of 3, and 
perform some writes.  Now it looks like this:

{noformat}
tcp0  0 10.208.8.123:48003  10.208.8.63:7000ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.123:35024  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:35024  10.208.8.123:7000   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:47212  10.208.8.123:7000   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.63:40215   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:9  10.208.35.225:7000  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:33498  10.208.8.63:7000ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.35.225:52530 ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.35.225:53674 ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.123:47212  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:40846  10.208.35.225:7000  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.63:48880   ESTABLISHED 
26254/java  
{noformat}

I can't think of any reason for a node to connect to itself, and this can cause 
problems with PFS where you might only define the broadcast addresses, but now 
you need the internal addresses too because the node will need to look itself 
up when connecting to itself.

  was:
I tested this with both the simple snitch and PFS.  It's quite easy to repro, 
setup a cluster, start it.  Mine looks like this:

{noformat}
tcp0  0 10.208.8.123:48003  10.208.8.63:7000ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.8.63:40215   ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:9  10.208.35.225:7000  ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:33498  10.208.8.63:7000ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.35.225:52530 ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.35.225:53674 ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:40846  10.208.35.225:7000  ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.8.63:48880   ESTABLISHED 
26254/java
{noformat}

No problems so far.  Now create a keyspace using NTS with an rf of 3, and 
perform some writes.  Now it looks like this:

{noformat}
tcp0  0 10.208.8.123:48003  10.208.8.63:7000ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.123:35024  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:35024  10.208.8.123:7000   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:47212  10.208.8.123:7000   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.63:40215   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:9  10.208.35.225:7000  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:33498  10.208.8.63:7000ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.35.225:52530 ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.35.225:53674 ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.123:47212  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:40846  10.208.35.225:7000  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.63:48880   ESTABLISHED 
26254/java  
{noformat}

I can't think of any reason for a node to connect to itself, and this can

[jira] [Created] (CASSANDRA-11332) nodes connect to themselves when NTS is used

2016-03-09 Thread Brandon Williams (JIRA) 
Brandon Williams created CASSANDRA-11332:


 Summary: nodes connect to themselves when NTS is used
 Key: CASSANDRA-11332
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11332
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Brandon Williams
 Fix For: 2.1.x


I tested this with both the simple snitch and PFS.  It's quite easy to repro, 
setup a cluster, start it.  Mine looks like this:

{noformat}
tcp0  0 10.208.8.123:48003  10.208.8.63:7000ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.8.63:40215   ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:9  10.208.35.225:7000  ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:33498  10.208.8.63:7000ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.35.225:52530 ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.35.225:53674 ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:40846  10.208.35.225:7000  ESTABLISHED 
26254/java
tcp0  0 10.208.8.123:7000   10.208.8.63:48880   ESTABLISHED 
26254/java
{noformat}

No problems so far.  Now create a keyspace using NTS with an rf of 3, and 
perform some writes.  Now it looks like this:

{noformat}
tcp0  0 10.208.8.123:48003  10.208.8.63:7000ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.123:35024  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:35024  10.208.8.123:7000   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:47212  10.208.8.123:7000   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.63:40215   ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:9  10.208.35.225:7000  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:33498  10.208.8.63:7000ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.35.225:52530 ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.35.225:53674 ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.123:47212  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:40846  10.208.35.225:7000  ESTABLISHED 
26254/java  
tcp0  0 10.208.8.123:7000   10.208.8.63:48880   ESTABLISHED 
26254/java  
{noformat}

I can't think of any reason for a node to connect to itself, and this can cause 
problems with PFS where you might only define the broadcast addresses, but now 
you need the internal addresses to because the node will need to look itself up 
when connecting to itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10990) Support streaming of older version sstables in 3.0

2016-03-09 Thread Paulo Motta (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188030#comment-15188030
 ] 

Paulo Motta commented on CASSANDRA-10990:
-

Updated patch and submitted tests.

||3.0||3.5||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:3.0-10990]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.5...pauloricardomg:3.5-10990]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-10990]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:10990]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10990-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.5-10990-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10990-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10990-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.5-10990-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10990-dtest/lastCompletedBuild/testReport/]|

note: the dtests are configured to use the [updated dtest 
branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:10990].

commit info: conflict on 3.5, merges cleanly to trunk.

> Support streaming of older version sstables in 3.0
> --
>
> Key: CASSANDRA-10990
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10990
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Jeremy Hanna
>Assignee: Paulo Motta
>
> In 2.0 we introduced support for streaming older versioned sstables 
> (CASSANDRA-5772).  In 3.0, because of the rewrite of the storage layer, this 
> became no longer supported.  So currently, while 3.0 can read sstables in the 
> 2.1/2.2 format, it cannot stream the older versioned sstables.  We should do 
> some work to make this still possible to be consistent with what 
> CASSANDRA-5772 provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11317) dtest failure in repair_tests.incremental_repair_test.TestIncRepair.sstable_repairedset_test

2016-03-09 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187923#comment-15187923
 ] 

Joel Knighton commented on CASSANDRA-11317:
---

Sure - I've pushed the change at 
[11317-2.2|https://github.com/jkni/cassandra/tree/11317-2.2]. It is a one-liner 
that sets clientMode to true.

Looking at the stack trace, it looks like this was probably broken by 
[CASSANDRA-10412], which inits DatabaseDescriptor directly in many offline 
tools. 

> dtest failure in 
> repair_tests.incremental_repair_test.TestIncRepair.sstable_repairedset_test
> 
>
> Key: CASSANDRA-11317
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11317
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jim Witschey
>  Labels: dtest
> Fix For: 2.2.x
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest/536/testReport/repair_tests.incremental_repair_test/TestIncRepair/sstable_repairedset_test
> Here's the failure and stack trace:
> {code}
> [' 0']
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-4WjpOf
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'start_rpc': 'true'}
> dtest: DEBUG: Repair timestamps are: [' 0', ' 0']
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File 
> "/home/automaton/cassandra-dtest/repair_tests/incremental_repair_test.py", 
> line 198, in sstable_repairedset_test
> self.assertGreaterEqual(len(uniquematches), 2, uniquematches)
>   File "/usr/lib/python2.7/unittest/case.py", line 948, in assertGreaterEqual
> self.fail(self._formatMessage(msg, standardMsg))
>   File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
> raise self.failureException(msg)
> "[' 0']\n >> begin captured logging << 
> \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-4WjpOf\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'start_rpc': 'true'}\ndtest: DEBUG: Repair timestamps are: [' 0', ' 
> 0']\n- >> end captured logging << -"
> {code}
> This has failed in this way on CassCI build cassandra-2.2_dtest 536-9.
> [~philipthompson] Could you have a first look at this? You had a recent look 
> at this test in CASSANDRA-11220.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

2016-03-09 Thread Joshua McKenzie (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187903#comment-15187903
 ] 

Joshua McKenzie commented on CASSANDRA-8844:


bq. where can I read more about that? Is there anything like a write-up or 
something? Or should I just read a handful of tickets with comments? 
No centralized store, more tribal knowledge and wisdom of experience. Searching 
through JIRA for tickets on those topics is your best bet (note: we moved to 
file-based hints for 3.0 as a particularly salient example of lessons learned)

bq. Can you help me to compile a list?
Don't have time right now unfortunately as I'm elbows deep in CDC 
implementation. :)

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it. 
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be configurable.
> - Logfiles should be named with a predictable naming schema, making it 
> triivial to process them in order.
> - Daemons should be able to checkpoint their work, and resume from where they 
> left off. This means they would have to leave some file artifact in the CDC 
> log's directory.
> - A sophisticated daemon should be able to be written that could 
>

[jira] [Commented] (CASSANDRA-11317) dtest failure in repair_tests.incremental_repair_test.TestIncRepair.sstable_repairedset_test

2016-03-09 Thread Yuki Morishita (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187873#comment-15187873
 ] 

Yuki Morishita commented on CASSANDRA-11317:


CASSANDRA-10132 is caused by BufferPool introduced in CASSANDRA-8897, thus 3.0+.
[~jkni] would you mind sharing your fix here?

> dtest failure in 
> repair_tests.incremental_repair_test.TestIncRepair.sstable_repairedset_test
> 
>
> Key: CASSANDRA-11317
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11317
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jim Witschey
>  Labels: dtest
> Fix For: 2.2.x
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest/536/testReport/repair_tests.incremental_repair_test/TestIncRepair/sstable_repairedset_test
> Here's the failure and stack trace:
> {code}
> [' 0']
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-4WjpOf
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'start_rpc': 'true'}
> dtest: DEBUG: Repair timestamps are: [' 0', ' 0']
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File 
> "/home/automaton/cassandra-dtest/repair_tests/incremental_repair_test.py", 
> line 198, in sstable_repairedset_test
> self.assertGreaterEqual(len(uniquematches), 2, uniquematches)
>   File "/usr/lib/python2.7/unittest/case.py", line 948, in assertGreaterEqual
> self.fail(self._formatMessage(msg, standardMsg))
>   File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
> raise self.failureException(msg)
> "[' 0']\n >> begin captured logging << 
> \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-4WjpOf\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'start_rpc': 'true'}\ndtest: DEBUG: Repair timestamps are: [' 0', ' 
> 0']\n- >> end captured logging << -"
> {code}
> This has failed in this way on CassCI build cassandra-2.2_dtest 536-9.
> [~philipthompson] Could you have a first look at this? You had a recent look 
> at this test in CASSANDRA-11220.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

2016-03-09 Thread Pavel Trukhanov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187867#comment-15187867
 ] 

Pavel Trukhanov commented on CASSANDRA-8844:


bq. Which is starting to sound an awful lot like storing CDC-data in a 
Cassandra table. While that's technically feasible, the lessons we've learned 
with both hints and batchlog should make us think really long and really hard 
before going down that road w/another feature that's basically going to double 
write mutation overhead for an enabled CF.


[~JoshuaMcKenzie] where can I read more about that? Is there anything like a 
write-up or something? Or should I just read a handful of tickets with 
comments? Can you help me to compile a list?

I'm looking forward to having this feature, although I think there's a core 
problem with the current design. I'm trying to learn more around all that and 
to read thoroughly and comprehend current design doc before making my comment.

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it. 
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be configurable.
> - Logfiles should be named with a predictable naming schema, making it 
> triivial

[jira] [Commented] (CASSANDRA-11331) Create Index IF NOT EXISTS throws error when index already exists

2016-03-09 Thread Philip Thompson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187856#comment-15187856
 ] 

Philip Thompson commented on CASSANDRA-11331:
-

This is not happening on 2.2.

> Create Index IF NOT EXISTS throws error when index already exists
> -
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.0.x, 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t (v2);
> CREATE INDEX IF NOT EXISTS ON t (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t_v2_idx_1 is a 
> duplicate of existing index t_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11331) Create Index IF NOT EXISTS throws error when index already exists

2016-03-09 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11331:

Fix Version/s: 3.0.x

> Create Index IF NOT EXISTS throws error when index already exists
> -
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.0.x, 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t (v2);
> CREATE INDEX IF NOT EXISTS ON t (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t_v2_idx_1 is a 
> duplicate of existing index t_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11331) Create Index IF NOT EXISTS not working on trunk

2016-03-09 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11331:

Reproduced In: 3.4, 3.0.4  (was: 3.4)

> Create Index IF NOT EXISTS not working on trunk
> ---
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t (v2);
> CREATE INDEX IF NOT EXISTS ON t (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t_v2_idx_1 is a 
> duplicate of existing index t_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11331) Create Index IF NOT EXISTS throws error when index already exists

2016-03-09 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11331:

Summary: Create Index IF NOT EXISTS throws error when index already exists  
(was: Create Index IF NOT EXISTS not working on trunk)

> Create Index IF NOT EXISTS throws error when index already exists
> -
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t (v2);
> CREATE INDEX IF NOT EXISTS ON t (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t_v2_idx_1 is a 
> duplicate of existing index t_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11331) Create Index IF NOT EXISTS not working on trunk

2016-03-09 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11331:

Description: 
While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
USE k;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
);
CREATE INDEX IF NOT EXISTS ON t (v2);
CREATE INDEX IF NOT EXISTS ON t (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t_v2_idx_1 is a 
duplicate of existing index t_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.

  was:
While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
USE k;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
);
CREATE INDEX IF NOT EXISTS ON t (v2);
CREATE INDEX IF NOT EXISTS ON t (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
duplicate of existing index t2_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.


> Create Index IF NOT EXISTS not working on trunk
> ---
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t (v2);
> CREATE INDEX IF NOT EXISTS ON t (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t_v2_idx_1 is a 
> duplicate of existing index t_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11331) Create Index IF NOT EXISTS not working on trunk

2016-03-09 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11331:

Description: 
While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
USE k;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
);
CREATE INDEX IF NOT EXISTS ON t (v2);
CREATE INDEX IF NOT EXISTS ON t (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
duplicate of existing index t2_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.

  was:
While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
USE k;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
);
CREATE INDEX IF NOT EXISTS ON t2 (v2);
CREATE INDEX IF NOT EXISTS ON t2 (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
duplicate of existing index t2_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.


> Create Index IF NOT EXISTS not working on trunk
> ---
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t (v2);
> CREATE INDEX IF NOT EXISTS ON t (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
> duplicate of existing index t2_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11331) Create Index IF NOT EXISTS not working on trunk

2016-03-09 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11331:

Description: 
While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
USE k;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
);
CREATE INDEX IF NOT EXISTS ON t2 (v2);
CREATE INDEX IF NOT EXISTS ON t2 (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
duplicate of existing index t2_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.

  was:
While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
)
CREATE INDEX IF NOT EXISTS ON t2 (v2);
CREATE INDEX IF NOT EXISTS ON t2 (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
duplicate of existing index t2_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.


> Create Index IF NOT EXISTS not working on trunk
> ---
>
> Key: CASSANDRA-11331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philip Thompson
> Fix For: 3.x
>
>
> While testing with trunk, I see that issuing the following queries throws an 
> InvalidRequest, despite being valid.
> {code}
> CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '1'}  AND durable_writes = true;
> USE k;
> CREATE TABLE k.t (
> id int PRIMARY KEY,
> v int,
> v2 int,
> v3 text
> );
> CREATE INDEX IF NOT EXISTS ON t2 (v2);
> CREATE INDEX IF NOT EXISTS ON t2 (v2);
> InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
> duplicate of existing index t2_v2_idx"
> {code}
> The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11331) Create Index IF NOT EXISTS not working on trunk

2016-03-09 Thread Philip Thompson (JIRA) 
Philip Thompson created CASSANDRA-11331:
---

 Summary: Create Index IF NOT EXISTS not working on trunk
 Key: CASSANDRA-11331
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11331
 Project: Cassandra
  Issue Type: Bug
  Components: CQL
Reporter: Philip Thompson
 Fix For: 3.x


While testing with trunk, I see that issuing the following queries throws an 
InvalidRequest, despite being valid.

{code}
CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;
CREATE TABLE k.t (
id int PRIMARY KEY,
v int,
v2 int,
v3 text
)
CREATE INDEX IF NOT EXISTS ON t2 (v2);
CREATE INDEX IF NOT EXISTS ON t2 (v2);
InvalidRequest: code=2200 [Invalid query] message="Index t2_v2_idx_1 is a 
duplicate of existing index t2_v2_idx"
{code}

The second {{CREATE INDEX IF NOT EXISTS}} should work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11289) dtest failure in secondary_indexes_test.TestSecondaryIndexes.test_only_coordinator_chooses_index_for_query

2016-03-09 Thread Russ Hatch (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch updated CASSANDRA-11289:
---
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

> dtest failure in 
> secondary_indexes_test.TestSecondaryIndexes.test_only_coordinator_chooses_index_for_query
> --
>
> Key: CASSANDRA-11289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11289
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Sam Tunnicliffe
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/297/testReport/secondary_indexes_test/TestSecondaryIndexes/test_only_coordinator_chooses_index_for_query
> Failed on CassCI build trunk_novnode_dtest #297
> Failing intermittently, error looks like this:
> {noformat}
> Expected to find between 1 and 200 trace events matching Executing read on 
> ks.cf using index b_index from 127.0.0.1, but actually found 0. (Full counts: 
> {'127.0.0.3': 1, '127.0.0.2': 1, '127.0.0.1': 0})
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11289) dtest failure in secondary_indexes_test.TestSecondaryIndexes.test_only_coordinator_chooses_index_for_query

2016-03-09 Thread Russ Hatch (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187829#comment-15187829
 ] 

Russ Hatch commented on CASSANDRA-11289:


pr merged by [~philipthompson]

> dtest failure in 
> secondary_indexes_test.TestSecondaryIndexes.test_only_coordinator_chooses_index_for_query
> --
>
> Key: CASSANDRA-11289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11289
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Sam Tunnicliffe
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/297/testReport/secondary_indexes_test/TestSecondaryIndexes/test_only_coordinator_chooses_index_for_query
> Failed on CassCI build trunk_novnode_dtest #297
> Failing intermittently, error looks like this:
> {noformat}
> Expected to find between 1 and 200 trace events matching Executing read on 
> ks.cf using index b_index from 127.0.0.1, but actually found 0. (Full counts: 
> {'127.0.0.3': 1, '127.0.0.2': 1, '127.0.0.1': 0})
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11289) dtest failure in secondary_indexes_test.TestSecondaryIndexes.test_only_coordinator_chooses_index_for_query

2016-03-09 Thread Russ Hatch (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch updated CASSANDRA-11289:
---
Status: Ready to Commit  (was: Patch Available)

> dtest failure in 
> secondary_indexes_test.TestSecondaryIndexes.test_only_coordinator_chooses_index_for_query
> --
>
> Key: CASSANDRA-11289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11289
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Sam Tunnicliffe
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/297/testReport/secondary_indexes_test/TestSecondaryIndexes/test_only_coordinator_chooses_index_for_query
> Failed on CassCI build trunk_novnode_dtest #297
> Failing intermittently, error looks like this:
> {noformat}
> Expected to find between 1 and 200 trace events matching Executing read on 
> ks.cf using index b_index from 127.0.0.1, but actually found 0. (Full counts: 
> {'127.0.0.3': 1, '127.0.0.2': 1, '127.0.0.1': 0})
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7276) Include keyspace and table names in logs where possible

2016-03-09 Thread Paulo Motta (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-7276:
---
Status: Open  (was: Patch Available)

> Include keyspace and table names in logs where possible
> ---
>
> Key: CASSANDRA-7276
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7276
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Priority: Minor
>  Labels: bootcamp, lhf
> Fix For: 2.1.x
>
> Attachments: 0001-Better-Logging-for-KS-and-CF.patch, 
> 0001-Logging-KS-and-CF-consistently.patch, 
> 0001-Logging-for-Keyspace-and-Tables.patch, 2.1-CASSANDRA-7276-v1.txt, 
> cassandra-2.1-7276-compaction.txt, cassandra-2.1-7276.txt, 
> cassandra-2.1.9-7276-v2.txt, cassandra-2.1.9-7276.txt
>
>
> Most error messages and stacktraces give you no clue as to what keyspace or 
> table was causing the problem.  For example:
> {noformat}
> ERROR [MutationStage:61648] 2014-05-20 12:05:45,145 CassandraDaemon.java 
> (line 198) Exception in thread Thread[MutationStage:61648,5,main]
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Unknown Source)
> at 
> org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:63)
> at 
> org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:72)
> at 
> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:98)
> at 
> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
> at 
> edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538)
> at 
> edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108)
> at 
> edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059)
> at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023)
> at 
> edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985)
> at 
> org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:328)
> at 
> org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:200)
> at org.apache.cassandra.db.Memtable.resolve(Memtable.java:226)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:173)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:893)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:368)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:333)
> at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:206)
> at 
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
> at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> {noformat}
> We should try to include info on the keyspace and column family in the error 
> messages or logs whenever possible.  This includes reads, writes, 
> compactions, flushes, repairs, and probably more.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7276) Include keyspace and table names in logs where possible

2016-03-09 Thread Paulo Motta (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187713#comment-15187713
 ] 

Paulo Motta edited comment on CASSANDRA-7276 at 3/9/16 7:40 PM:


Thanks and sorry for the delay.

I'm a bit worried that the repetition of log statements with the \[ks.cf\] 
prefix is error prone and cumbersome so may not be followed by other 
developers, so I think we can improve this by providing a 
{{ContextualizedLogger}} that will wrap a {{Logger}} object and provide 
utillity methods to automatically add the \[ks.cf\] prefix. This will allow us 
to change the prefix format if necessary and easily add new contextualized log 
statements types in the future.

I thought of something along those lines (these are just 
prototypes/suggestions):

{noformat}
public class ContextualizedLogger implements Logger
{
private final Logger logger;

public static ContextualizedLogger getLogger(Class clazz) {
return new ContextualizedLogger(LoggerFactory.getLogger(clazz));
}

private ContextualizedLogger(Logger logger)
{
this.logger = logger;
}

public void debug(ColumnFamilyStore cfs, String msg, Object... args)
{
debug(cfs.name, cfs.keyspace.getName(), msg, args);
}

public void debug(CFMetaData cfm, String msg, Object... args)
{
debug(cfm.ksName, cfm.cfName, msg, args);
}

public void debug(String ks, String table, String msg, Object... args)
{
logger.debug(extendMsg(ks, table, msg), args);
}

private String extendMsg(String ks, String table, String msg)
{
return String.format("[%s.%s] %s", ks, table, msg);
}

public String getName()
{
return logger.getName();
}

public boolean isTraceEnabled()
{
return logger.isTraceEnabled();
}

public void trace(String s)
{
logger.trace(s);
}

public boolean isTraceEnabled(Marker marker)
{
return logger.isTraceEnabled(marker);
}
}
{noformat}

So this, could be used like this, for instance:

{noformat}
public class ColumnFamilyStore implements ColumnFamilyStoreMBean
{
private static final ContextualizedLogger logger = 
ContextualizedLogger.getLogger(ColumnFamilyStore.class);

private void logFlush()
{
logger.debug(this, "Enqueuing flush: {}", String.format("%d (%.0f%%) 
on-heap, %d (%.0f%%) off-heap", onHeapTotal,
   onHeapRatio * 100, offHeapTotal, 
offHeapRatio * 100));
}
}
{noformat}

or

{noformat}
public class CompactionManager implements CompactionManagerMBean
{
private static final ContextualizedLogger logger = 
ContextualizedLogger.getLogger(CompactionManager.class);

public Future submitCacheWrite(final AutoSavingCache.Writer writer)
{
Runnable runnable = new Runnable()
{
public void run()
{
if (!AutoSavingCache.flushInProgress.add(writer.cacheType()))
{
CFMetaData cfm = writer.getCompactionInfo().getCFMetaData();
logger.debug(cfm, "Cache flushing was already in progress: 
skipping {}", writer.getCompactionInfo());
return;
}
 }
}
}
}
{noformat}

WDYT about this approach? If you agree please add support to this and use it on 
all log statements of {{ColumnFamilyStore}}, {{CompactionManager}}, 
{{Memtable}}, {{CompactionTask}} (and subclasses). You may need to update log 
statements that already mention ks/cf info to only show those only in the 
prefix, for example:
* {{Loading new SSTables and building secondary indexes for ks/cf: 
sstable_XYZ}} would become {{\[ks.cf\] Loading new SSTables and building 
secondary indexes: sstable_XYZ}}

We can also improve {{ContextualizedException}} to add the {{ks/cf}} info to 
the Exception message itself, so it will be automatically print when the stack 
trace is print by the uncaught exception handler. I modified the class to do 
this so you can reuse it on your next patch:
{noformat}
diff --git 
a/src/java/org/apache/cassandra/exceptions/ContextualizedException.java 
b/src/java/org/apache/cassandra/exceptions/ContextualizedException.java
index 68c3e54..9cc5a93 100644
--- a/src/java/org/apache/cassandra/exceptions/ContextualizedException.java
+++ b/src/java/org/apache/cassandra/exceptions/ContextualizedException.java
@@ -19,6 +19,9 @@ package org.apache.cassandra.exceptions;
 
 import java.util.List;
 import java.util.Optional;
+import java.util.stream.Collectors;
+
+import org.apache.commons.lang3.StringUtils;
 
 /**
  * This exception is used by MessageDeliveryTask to
@@ -49,4 +52,28 @@ public class ContextualizedException extends 
RuntimeException {
 this.keyspace = keyspace;
 this.tables = tables;
 }
+
+public String getPrettyKeyspaceAndTables()
+

[jira] [Commented] (CASSANDRA-11317) dtest failure in repair_tests.incremental_repair_test.TestIncRepair.sstable_repairedset_test

2016-03-09 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187756#comment-15187756
 ] 

Joel Knighton commented on CASSANDRA-11317:
---

It looks like this is because [CASSANDRA-10132] was only merged in 3.0+. Is 
there a reason it wasn't included in 2.2 [~yukim]?

Backporting this fix looks like it fixes the test (in a few local test runs).

> dtest failure in 
> repair_tests.incremental_repair_test.TestIncRepair.sstable_repairedset_test
> 
>
> Key: CASSANDRA-11317
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11317
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jim Witschey
>  Labels: dtest
> Fix For: 2.2.x
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest/536/testReport/repair_tests.incremental_repair_test/TestIncRepair/sstable_repairedset_test
> Here's the failure and stack trace:
> {code}
> [' 0']
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-4WjpOf
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'start_rpc': 'true'}
> dtest: DEBUG: Repair timestamps are: [' 0', ' 0']
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File 
> "/home/automaton/cassandra-dtest/repair_tests/incremental_repair_test.py", 
> line 198, in sstable_repairedset_test
> self.assertGreaterEqual(len(uniquematches), 2, uniquematches)
>   File "/usr/lib/python2.7/unittest/case.py", line 948, in assertGreaterEqual
> self.fail(self._formatMessage(msg, standardMsg))
>   File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
> raise self.failureException(msg)
> "[' 0']\n >> begin captured logging << 
> \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-4WjpOf\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'start_rpc': 'true'}\ndtest: DEBUG: Repair timestamps are: [' 0', ' 
> 0']\n- >> end captured logging << -"
> {code}
> This has failed in this way on CassCI build cassandra-2.2_dtest 536-9.
> [~philipthompson] Could you have a first look at this? You had a recent look 
> at this test in CASSANDRA-11220.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer

2016-03-09 Thread Patrick Brown (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187753#comment-15187753
 ] 

Patrick Brown commented on CASSANDRA-8723:
--

I am having similar issues on 2.2.5 after upgrading from 2.1.8, CentOS 7, 8 GB 
heap, 64 GB RAM on server.

Memory overcommits to ~100GB and then it is killed by system OOM killer.

Happy to provide more info, just let me know what is needed.

> Cassandra 2.1.2 Memory issue - java process memory usage continuously 
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jeff Liu
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously 
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill 
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919 
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar 
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M 
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log 
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M 
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
>  -Dlogback.configurationFile=logback.xml 
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir= 
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
>  -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof 
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log 
> org.apache.cassandra.service.CassandraDaemon
> {noformat}



--
This message was sent by Atlassian JIRA

[jira] [Updated] (CASSANDRA-11330) Enable sstabledump to be used on 2i tables

2016-03-09 Thread Yuki Morishita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-11330:
---
Reviewer: Yuki Morishita

> Enable sstabledump to be used on 2i tables
> --
>
> Key: CASSANDRA-11330
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11330
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
>
> It is sometimes useful to be able to inspect the sstables backing 2i tables, 
> which requires a small tweak to the way the partitioner is created.
> Although this is an improvement rather than a bugfix, I've marked it for 
> 3.0.x as it's really very non-invasive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM

2016-03-09 Thread Adam Holmberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187745#comment-15187745
 ] 

Adam Holmberg commented on CASSANDRA-11320:
---

I agree with the general approach. I now see that the current implementation 
only stalls for a specific type of error (server-side timeouts), and does not 
halt new requests on availability exceptions, etc. I think it might also be 
wise to limit the total concurrent requests in-flight (per host?). It could be 
a high number -- just something to prevent runaways if the client is not the 
bottleneck.

> Improve backoff policy for cqlsh COPY FROM
> --
>
> Key: CASSANDRA-11320
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11320
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
>
> Currently we have an exponential back-off policy in COPY FROM that kicks in 
> when timeouts are received. However there are two limitations:
> * it does not cover new requests and therefore we may not back-off 
> sufficiently to give time to an overloaded server to recover
> * the pause is performed in the receiving thread and therefore we may not 
> process server messages quickly enough
> There is a static throttling mechanism in rows per second from feeder to 
> worker processes (the INGESTRATE) but the feeder has no idea of the load of 
> each worker process. However it's easy to keep track of how many chunks a 
> worker process has yet to read by introducing a bounded semaphore.
> The idea is to move the back-off pauses to the worker processes main thread 
> so as to include all messages, new and retries, not just the retries that 
> timed out. The worker process will not read new chunks during the back-off 
> pauses, and the feeder process can then look at the number of pending chunks 
> before sending new chunks to a worker process.
> [~aholmber], [~aweisberg] what do you think?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11330) Enable sstabledump to be used on 2i tables

2016-03-09 Thread Sam Tunnicliffe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-11330:

Status: Patch Available  (was: Open)

The patch available 
[here|https://github.com/beobal/cassandra/commit/9c16f8f563d30d8d6ae894cfde8f5af1015a196d]
 also includes a fix to a minor typo in a method name. Causes a compilation 
error on merge to 3.5 because the method with the changed name has an 
additional callsite, but nothing more significant.


> Enable sstabledump to be used on 2i tables
> --
>
> Key: CASSANDRA-11330
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11330
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
>
> It is sometimes useful to be able to inspect the sstables backing 2i tables, 
> which requires a small tweak to the way the partitioner is created.
> Although this is an improvement rather than a bugfix, I've marked it for 
> 3.0.x as it's really very non-invasive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11127) index_summary_upgrade_test.py is failing

2016-03-09 Thread Jim Witschey (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187714#comment-15187714
 ] 

Jim Witschey commented on CASSANDRA-11127:
--

This also seems to fail hard on Windows:

http://cassci.datastax.com/job/trunk_dtest_win32/361/testReport/index_summary_upgrade_test/TestUpgradeIndexSummary/test_upgrade_index_summary/history/

> index_summary_upgrade_test.py is failing
> 
>
> Key: CASSANDRA-11127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11127
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Philip Thompson
>  Labels: dtest
> Fix For: 3.0.x
>
> Attachments: node1_debug.log
>
>
> index_summary_upgrade_test.py is failing on the cassandra-3.0 branch, when 
> run without vnodes. The exception I'm seeing on cassci is different than 
> locally. The cassci failure is 
> [here|http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/157/testReport/index_summary_upgrade_test/TestUpgradeIndexSummary/test_upgrade_index_summary/].
> Locally I see the following:
> {code}
> 'ERROR [SSTableBatchOpen:2] 2016-02-05 15:29:04,304 CassandraDaemon.java:195 
> - Exception in thread 
> Thread[SSTableBatchOpen:2,5,main]\njava.lang.AssertionError: Illegal bounds 
> [4..8); size: 4\n\tat 
> org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:339) 
> ~[main/:na]\n\tat org.apache.cassandra.io.util.Memory.getInt(Memory.java:292) 
> ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:146)
>  ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:151) 
> ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.format.SSTableReader.validateSummarySamplingLevel(SSTableReader.java:928)
>  ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:748)
>  ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:705)
>  ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:491)
>  ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
>  ~[main/:na]\n\tat 
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
>  ~[main/:na]\n\tat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_66]\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_66]\n\tat 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_66]\n\tat 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_66]\n\tat java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]']
> {code}
> Node log is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11330) Enable sstabledump to be used on 2i tables

2016-03-09 Thread Sam Tunnicliffe (JIRA) 
Sam Tunnicliffe created CASSANDRA-11330:
---

 Summary: Enable sstabledump to be used on 2i tables
 Key: CASSANDRA-11330
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11330
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Priority: Minor
 Fix For: 3.0.x, 3.x


It is sometimes useful to be able to inspect the sstables backing 2i tables, 
which requires a small tweak to the way the partitioner is created.
Although this is an improvement rather than a bugfix, I've marked it for 3.0.x 
as it's really very non-invasive.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7276) Include keyspace and table names in logs where possible

2016-03-09 Thread Paulo Motta (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187713#comment-15187713
 ] 

Paulo Motta commented on CASSANDRA-7276:


Thanks and sorry for the delay.

I'm a bit worried that the repetition of log statements with the \[ks.cf\] 
prefix is error prone and cumbersome so may not be followed by other 
developers, so I think we can improve this by providing a 
{{ContextualizedLogger}} that will wrap a {{Logger}} object and provide 
utillity methods to automatically add the \[ks.cf\] prefix. This will allow us 
to change the prefix format if necessary and easily add new contextualized log 
statements types in the future.

I thought of something along those lines (these are just 
prototypes/suggestions):

{noformat}
public class ContextualizedLogger implements Logger
{
private final Logger logger;

public static ContextualizedLogger getLogger(Class clazz) {
return new ContextualizedLogger(LoggerFactory.getLogger(clazz));
}

private ContextualizedLogger(Logger logger)
{
this.logger = logger;
}

public void debug(ColumnFamilyStore cfs, String msg, Object... args)
{
debug(cfs.name, cfs.keyspace.getName(), msg, args);
}

public void debug(CFMetaData cfm, String msg, Object... args)
{
debug(cfm.ksName, cfm.cfName, msg, args);
}

public void debug(String ks, String table, String msg, Object... args)
{
logger.debug(extendMsg(msg), extendArgs(ks, table, args));
}

private String extendMsg(String msg)
{
return String.format("[{}.{}] %s", msg);
}

private Object[] extendArgs(String ks, String table, Object[] args)
{
Object[] newArgs = new Object[args.length+2];
newArgs[0] = ks;
newArgs[1] = table;
System.arraycopy(args, 0, newArgs, 2, args.length);
return newArgs;
}

public String getName()
{
return logger.getName();
}

public boolean isTraceEnabled()
{
return logger.isTraceEnabled();
}

public void trace(String s)
{
logger.trace(s);
}

public boolean isTraceEnabled(Marker marker)
{
return logger.isTraceEnabled(marker);
}
}
{noformat}

So this, could be used like this, for instance:

{noformat}
public class ColumnFamilyStore implements ColumnFamilyStoreMBean
{
private static final ContextualizedLogger logger = 
ContextualizedLogger.getLogger(ColumnFamilyStore.class);

private void logFlush()
{
logger.debug(this, "Enqueuing flush: {}", String.format("%d (%.0f%%) 
on-heap, %d (%.0f%%) off-heap", onHeapTotal,
   onHeapRatio * 100, offHeapTotal, 
offHeapRatio * 100));
}
}
{noformat}

or

{noformat}
public class CompactionManager implements CompactionManagerMBean
{
private static final ContextualizedLogger logger = 
ContextualizedLogger.getLogger(CompactionManager.class);

public Future submitCacheWrite(final AutoSavingCache.Writer writer)
{
Runnable runnable = new Runnable()
{
public void run()
{
if (!AutoSavingCache.flushInProgress.add(writer.cacheType()))
{
CFMetaData cfm = writer.getCompactionInfo().getCFMetaData();
logger.debug(cfm, "Cache flushing was already in progress: 
skipping {}", writer.getCompactionInfo());
return;
}
 }
}
}
}
{noformat}

WDYT about this approach? If you agree please add support to this and use it on 
all log statements of {{ColumnFamilyStore}}, {{CompactionManager}}, 
{{Memtable}}, {{CompactionTask}} (and subclasses). You may need to update log 
statements that already mention ks/cf info to only show those only in the 
prefix, for example:
* {{Loading new SSTables and building secondary indexes for ks/cf: 
sstable_XYZ}} would become {{\[ks.cf\] Loading new SSTables and building 
secondary indexes: sstable_XYZ}}

We can also improve {{ContextualizedException}} to add the {{ks/cf}} info to 
the Exception message itself, so it will be automatically print when the stack 
trace is print by the uncaught exception handler. I modified the class to do 
this so you can reuse it on your next patch:
{noformat}
diff --git 
a/src/java/org/apache/cassandra/exceptions/ContextualizedException.java 
b/src/java/org/apache/cassandra/exceptions/ContextualizedException.java
index 68c3e54..9cc5a93 100644
--- a/src/java/org/apache/cassandra/exceptions/ContextualizedException.java
+++ b/src/java/org/apache/cassandra/exceptions/ContextualizedException.java
@@ -19,6 +19,9 @@ package org.apache.cassandra.exceptions;
 
 import java.util.List;
 import java.util.Optional;
+import java.util.stream.Collectors;
+
+import org.apache.commons.lang3.StringUtils;
 
 /**
  * This exception is used by MessageDeliveryTask to
@@

[jira] [Updated] (CASSANDRA-11329) Indexers are not informed when expired rows are encountered in compaction

2016-03-09 Thread Sam Tunnicliffe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-11329:

Reviewer: Sylvain Lebresne
  Status: Patch Available  (was: Open)

I've pushed a fix, which also modifies {{CassandraIndex}} to make {{removeRow}} 
a no-op when the index is on a primary key column. Without that, additional 
updates are applied to those indexes during compaction. Those additional 
updates are harmless duplicates, which are removed when the index cfs itself is 
compacted, but it's clearly more efficient to skip them in the first place.

||branch||testall||dtest||
|[11329-3.0|https://github.com/beobal/cassandra/tree/11329-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11329-3.0-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11329-3.0-dtest]|
|[11329-3.5|https://github.com/beobal/cassandra/tree/11329-3.5]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11329-3.5-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11329-3.5-dtest]|
|[11329-trunk|https://github.com/beobal/cassandra/tree/11329-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11329-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11329-trunk-dtest]|


> Indexers are not informed when expired rows are encountered in compaction
> -
>
> Key: CASSANDRA-11329
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11329
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.x, 3.x
>
>
> When rows are merged during a compaction, if the row resulting from that 
> merge is expired due to a row level ttl, registered indexes should be 
> notified. 
> Index implementers need to be aware that just because an expired row is 
> written to the new SSTable, it doesn't necessarily mean that the index should 
> purge its entry/entries for that row as there may still be be live data in 
> other SSTables. 
> That said, it should probably be the responsibility of the index 
> implementation to manage that, but at the moment the handling of an 
> {{onPrimaryKeyLivenessInfo}} event during compaction is a no-op and doesn't 
> cause the registered indexes to be notified. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11329) Indexers are not informed when expired rows are encountered in compaction

2016-03-09 Thread Sam Tunnicliffe (JIRA) 
Sam Tunnicliffe created CASSANDRA-11329:
---

 Summary: Indexers are not informed when expired rows are 
encountered in compaction
 Key: CASSANDRA-11329
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11329
 Project: Cassandra
  Issue Type: Bug
  Components: Compaction
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
 Fix For: 3.0.x, 3.x


When rows are merged during a compaction, if the row resulting from that merge 
is expired due to a row level ttl, registered indexes should be notified. 

Index implementers need to be aware that just because an expired row is written 
to the new SSTable, it doesn't necessarily mean that the index should purge its 
entry/entries for that row as there may still be be live data in other 
SSTables. 

That said, it should probably be the responsibility of the index implementation 
to manage that, but at the moment the handling of an 
{{onPrimaryKeyLivenessInfo}} event during compaction is a no-op and doesn't 
cause the registered indexes to be notified. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11328) dtest failure in ttl_test.TestTTL.remove_column_ttl_with_default_ttl_test

2016-03-09 Thread Jim Witschey (JIRA) 
Jim Witschey created CASSANDRA-11328:


 Summary: dtest failure in 
ttl_test.TestTTL.remove_column_ttl_with_default_ttl_test
 Key: CASSANDRA-11328
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11328
 Project: Cassandra
  Issue Type: Test
Reporter: Jim Witschey
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/trunk_dtest/1043/testReport/ttl_test/TestTTL/remove_column_ttl_with_default_ttl_test

This test started failing on build #1041, when a change to TTL behavior was 
introduced:

https://github.com/apache/cassandra/commit/e017f9494844234fa73848890347f59c622cea40

[~blerer] Looks like I failed to review this properly, sorry. Could you have a 
look at this, please?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-03-09 Thread yukim 
Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b808a180
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b808a180
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b808a180

Branch: refs/heads/cassandra-3.0
Commit: b808a1806448884d6c7aa20c1c0812b92c40e9a5
Parents: d0b7a02 f791589
Author: Yuki Morishita 
Authored: Wed Mar 9 11:42:58 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:42:58 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b808a180/CHANGES.txt
--
diff --cc CHANGES.txt
index e6c40aa,d7b0fc2..5efb1b0
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,40 -1,12 +1,41 @@@
 -2.2.6
 +3.0.5
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
   * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
   * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 - * Fix filtering on non-primary key columns for thrift static column families
 -   (CASSANDRA-6377)
   * Only log yaml config once, at startup (CASSANDRA-11217)
 - * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
   * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 +Merged from 2.1:
 + * InvalidateKeys should have a weak ref to key cache (CASSANDRA-11176)
 + * COPY FROM on large datasets: fix progress report and debug performance 
(CASSANDRA-11053)
 +
 +3.0.4
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * MV should only query complex columns included in the view (CASSANDRA-11069)
 + * Failed aggregate creation breaks server permanently (CASSANDRA-11064)
 + * Add sstabledump tool (CASSANDRA-7464)
 + * Introduce backpressure for hints (CASSANDRA-10972)
 + * Fix ClusteringPrefix not being able to read tombstone range boundaries 
(CASSANDRA-11158)
 + * Prevent logging in sandboxed state (CASSANDRA-11033)
 + * Disallow drop/alter operations of UDTs used by UDAs (CASSANDRA-10721)
 + * Add query time validation method on Index (CASSANDRA-11043)
 + * Avoid potential AssertionError in mixed version cluster (CASSANDRA-11128)
 + * Properly handle hinted handoff after topology changes (CASSANDRA-5902)
 + * AssertionError when listing sstable files on inconsistent disk state 
(CASSANDRA-11156)
 + * Fix wrong rack counting and invalid conditions check for TokenAllocation
 +   (CASSANDRA-11139)
 + * Avoid creating empty hint files (CASSANDRA-11090)
 + * Fix leak detection strong reference loop using weak reference 
(CASSANDRA-11120)
 + * Configurie BatchlogManager to stop delayed tasks on shutdown 
(CASSANDRA-11062)
 + * Hadoop integration is incompatible with Cassandra Driver 3.0.0 
(CASSANDRA-11001)
 + * Add dropped_columns to the list of schema table so it gets handled
 +   properly (CASSANDRA-11050)
 + * Fix NPE when using forceRepairRangeAsync without DC (CASSANDRA-11239)
 +Merged from 2.2:
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
   * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
   * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
   * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)

[02/10] cassandra git commit: Add missing newline at end of bin/cqlsh

2016-03-09 Thread yukim 
Add missing newline at end of bin/cqlsh


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f7915899
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f7915899
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f7915899

Branch: refs/heads/cassandra-3.0
Commit: f79158998b1aaecb237ebd236948c2be95e70cc9
Parents: 9b53d6a
Author: Joel Knighton 
Authored: Wed Mar 9 11:26:27 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:41:38 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9d4779a..d7b0fc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
  * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
  * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
  * Fix filtering on non-primary key columns for thrift static column families

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 518b986..82a4a53 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -23,4 +23,4 @@ for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
 done
 echo "No appropriate python interpreter found." >&2
-exit 1
\ No newline at end of file
+exit 1

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-03-09 Thread yukim 
Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b808a180
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b808a180
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b808a180

Branch: refs/heads/trunk
Commit: b808a1806448884d6c7aa20c1c0812b92c40e9a5
Parents: d0b7a02 f791589
Author: Yuki Morishita 
Authored: Wed Mar 9 11:42:58 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:42:58 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b808a180/CHANGES.txt
--
diff --cc CHANGES.txt
index e6c40aa,d7b0fc2..5efb1b0
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,40 -1,12 +1,41 @@@
 -2.2.6
 +3.0.5
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
   * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
   * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 - * Fix filtering on non-primary key columns for thrift static column families
 -   (CASSANDRA-6377)
   * Only log yaml config once, at startup (CASSANDRA-11217)
 - * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
   * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 +Merged from 2.1:
 + * InvalidateKeys should have a weak ref to key cache (CASSANDRA-11176)
 + * COPY FROM on large datasets: fix progress report and debug performance 
(CASSANDRA-11053)
 +
 +3.0.4
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * MV should only query complex columns included in the view (CASSANDRA-11069)
 + * Failed aggregate creation breaks server permanently (CASSANDRA-11064)
 + * Add sstabledump tool (CASSANDRA-7464)
 + * Introduce backpressure for hints (CASSANDRA-10972)
 + * Fix ClusteringPrefix not being able to read tombstone range boundaries 
(CASSANDRA-11158)
 + * Prevent logging in sandboxed state (CASSANDRA-11033)
 + * Disallow drop/alter operations of UDTs used by UDAs (CASSANDRA-10721)
 + * Add query time validation method on Index (CASSANDRA-11043)
 + * Avoid potential AssertionError in mixed version cluster (CASSANDRA-11128)
 + * Properly handle hinted handoff after topology changes (CASSANDRA-5902)
 + * AssertionError when listing sstable files on inconsistent disk state 
(CASSANDRA-11156)
 + * Fix wrong rack counting and invalid conditions check for TokenAllocation
 +   (CASSANDRA-11139)
 + * Avoid creating empty hint files (CASSANDRA-11090)
 + * Fix leak detection strong reference loop using weak reference 
(CASSANDRA-11120)
 + * Configurie BatchlogManager to stop delayed tasks on shutdown 
(CASSANDRA-11062)
 + * Hadoop integration is incompatible with Cassandra Driver 3.0.0 
(CASSANDRA-11001)
 + * Add dropped_columns to the list of schema table so it gets handled
 +   properly (CASSANDRA-11050)
 + * Fix NPE when using forceRepairRangeAsync without DC (CASSANDRA-11239)
 +Merged from 2.2:
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
   * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
   * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
   * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

2016-03-09 Thread yukim 
Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2333a92
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2333a92
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2333a92

Branch: refs/heads/trunk
Commit: d2333a9256c67306c617a237a5f465f913263702
Parents: e33d4be b808a18
Author: Yuki Morishita 
Authored: Wed Mar 9 11:43:37 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:43:37 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2333a92/CHANGES.txt
--
diff --cc CHANGES.txt
index 72ce859,5efb1b0..ff67a2b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -3,6 -3,8 +3,7 @@@ Merged from 3.0
   * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
   * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
  Merged from 2.2:
+  * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
 - * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
   * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
   * Only log yaml config once, at startup (CASSANDRA-11217)
   * Reference leak with parallel repairs on the same table (CASSANDRA-11215)

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

2016-03-09 Thread yukim 
Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b808a180
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b808a180
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b808a180

Branch: refs/heads/cassandra-3.5
Commit: b808a1806448884d6c7aa20c1c0812b92c40e9a5
Parents: d0b7a02 f791589
Author: Yuki Morishita 
Authored: Wed Mar 9 11:42:58 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:42:58 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b808a180/CHANGES.txt
--
diff --cc CHANGES.txt
index e6c40aa,d7b0fc2..5efb1b0
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,40 -1,12 +1,41 @@@
 -2.2.6
 +3.0.5
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
   * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
   * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
 - * Fix filtering on non-primary key columns for thrift static column families
 -   (CASSANDRA-6377)
   * Only log yaml config once, at startup (CASSANDRA-11217)
 - * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
   * Reference leak with parallel repairs on the same table (CASSANDRA-11215)
 +Merged from 2.1:
 + * InvalidateKeys should have a weak ref to key cache (CASSANDRA-11176)
 + * COPY FROM on large datasets: fix progress report and debug performance 
(CASSANDRA-11053)
 +
 +3.0.4
 + * Preserve order for preferred SSL cipher suites (CASSANDRA-11164)
 + * MV should only query complex columns included in the view (CASSANDRA-11069)
 + * Failed aggregate creation breaks server permanently (CASSANDRA-11064)
 + * Add sstabledump tool (CASSANDRA-7464)
 + * Introduce backpressure for hints (CASSANDRA-10972)
 + * Fix ClusteringPrefix not being able to read tombstone range boundaries 
(CASSANDRA-11158)
 + * Prevent logging in sandboxed state (CASSANDRA-11033)
 + * Disallow drop/alter operations of UDTs used by UDAs (CASSANDRA-10721)
 + * Add query time validation method on Index (CASSANDRA-11043)
 + * Avoid potential AssertionError in mixed version cluster (CASSANDRA-11128)
 + * Properly handle hinted handoff after topology changes (CASSANDRA-5902)
 + * AssertionError when listing sstable files on inconsistent disk state 
(CASSANDRA-11156)
 + * Fix wrong rack counting and invalid conditions check for TokenAllocation
 +   (CASSANDRA-11139)
 + * Avoid creating empty hint files (CASSANDRA-11090)
 + * Fix leak detection strong reference loop using weak reference 
(CASSANDRA-11120)
 + * Configurie BatchlogManager to stop delayed tasks on shutdown 
(CASSANDRA-11062)
 + * Hadoop integration is incompatible with Cassandra Driver 3.0.0 
(CASSANDRA-11001)
 + * Add dropped_columns to the list of schema table so it gets handled
 +   properly (CASSANDRA-11050)
 + * Fix NPE when using forceRepairRangeAsync without DC (CASSANDRA-11239)
 +Merged from 2.2:
 + * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
   * Range.compareTo() violates the contract of Comparable (CASSANDRA-11216)
   * Avoid NPE when serializing ErrorMessage with null message (CASSANDRA-11167)
   * Replacing an aggregate with a new version doesn't reset INITCOND 
(CASSANDRA-10840)

[jira] [Updated] (CASSANDRA-11325) Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf Ant plugin

2016-03-09 Thread Yuki Morishita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-11325:
---
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 2.2.x)
   (was: 3.x)
   3.5
   3.0.5
   2.2.6
   Status: Resolved  (was: Patch Available)

Thanks, +1 and committed in f79158998b1aaecb237ebd236948c2be95e70cc9.

> Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf 
> Ant plugin
> -
>
> Key: CASSANDRA-11325
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11325
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Joel Knighton
>Assignee: Joel Knighton
>Priority: Trivial
> Fix For: 2.2.6, 3.0.5, 3.5
>
>
> bin/cqlsh is missing a new line - when doing an artifact build, the fixcrlf 
> plugin will insert a newline at the end of bin/cqlsh. This change inserts the 
> newline and changes the mode on cqlsh to 644.
> This isn't a problem affecting release builds, since we explicitly set the 
> mode of files under bin to executable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[01/10] cassandra git commit: Add missing newline at end of bin/cqlsh

2016-03-09 Thread yukim 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 9b53d6a5b -> f79158998
  refs/heads/cassandra-3.0 d0b7a02b2 -> b808a1806
  refs/heads/cassandra-3.5 e33d4be66 -> d2333a925
  refs/heads/trunk dc480c794 -> 32b5f8fe4


Add missing newline at end of bin/cqlsh


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f7915899
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f7915899
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f7915899

Branch: refs/heads/cassandra-2.2
Commit: f79158998b1aaecb237ebd236948c2be95e70cc9
Parents: 9b53d6a
Author: Joel Knighton 
Authored: Wed Mar 9 11:26:27 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:41:38 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9d4779a..d7b0fc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
  * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
  * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
  * Fix filtering on non-primary key columns for thrift static column families

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 518b986..82a4a53 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -23,4 +23,4 @@ for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
 done
 echo "No appropriate python interpreter found." >&2
-exit 1
\ No newline at end of file
+exit 1

[04/10] cassandra git commit: Add missing newline at end of bin/cqlsh

2016-03-09 Thread yukim 
Add missing newline at end of bin/cqlsh


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f7915899
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f7915899
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f7915899

Branch: refs/heads/trunk
Commit: f79158998b1aaecb237ebd236948c2be95e70cc9
Parents: 9b53d6a
Author: Joel Knighton 
Authored: Wed Mar 9 11:26:27 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:41:38 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9d4779a..d7b0fc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
  * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
  * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
  * Fix filtering on non-primary key columns for thrift static column families

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 518b986..82a4a53 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -23,4 +23,4 @@ for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
 done
 echo "No appropriate python interpreter found." >&2
-exit 1
\ No newline at end of file
+exit 1

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

2016-03-09 Thread yukim 
Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d2333a92
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d2333a92
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d2333a92

Branch: refs/heads/cassandra-3.5
Commit: d2333a9256c67306c617a237a5f465f913263702
Parents: e33d4be b808a18
Author: Yuki Morishita 
Authored: Wed Mar 9 11:43:37 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:43:37 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d2333a92/CHANGES.txt
--
diff --cc CHANGES.txt
index 72ce859,5efb1b0..ff67a2b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -3,6 -3,8 +3,7 @@@ Merged from 3.0
   * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
   * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
  Merged from 2.2:
+  * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
 - * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
   * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
   * Only log yaml config once, at startup (CASSANDRA-11217)
   * Reference leak with parallel repairs on the same table (CASSANDRA-11215)

[10/10] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

2016-03-09 Thread yukim 
Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/32b5f8fe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/32b5f8fe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/32b5f8fe

Branch: refs/heads/trunk
Commit: 32b5f8fe4939fc560f783ed39e4d9ac69f086a11
Parents: dc480c7 d2333a9
Author: Yuki Morishita 
Authored: Wed Mar 9 11:43:44 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:43:44 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/32b5f8fe/CHANGES.txt
--

[03/10] cassandra git commit: Add missing newline at end of bin/cqlsh

2016-03-09 Thread yukim 
Add missing newline at end of bin/cqlsh


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f7915899
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f7915899
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f7915899

Branch: refs/heads/cassandra-3.5
Commit: f79158998b1aaecb237ebd236948c2be95e70cc9
Parents: 9b53d6a
Author: Joel Knighton 
Authored: Wed Mar 9 11:26:27 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Mar 9 11:41:38 2016 -0600

--
 CHANGES.txt | 1 +
 bin/cqlsh   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9d4779a..d7b0fc2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Add missing newline at end of bin/cqlsh (CASSANDRA-11325)
  * Fix AE in nodetool cfstats (backport CASSANDRA-10859) (CASSANDRA-11297)
  * Unresolved hostname leads to replace being ignored (CASSANDRA-11210)
  * Fix filtering on non-primary key columns for thrift static column families

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7915899/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 518b986..82a4a53 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -23,4 +23,4 @@ for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
 done
 echo "No appropriate python interpreter found." >&2
-exit 1
\ No newline at end of file
+exit 1

[jira] [Comment Edited] (CASSANDRA-11325) Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf Ant plugin

2016-03-09 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187457#comment-15187457
 ] 

Joel Knighton edited comment on CASSANDRA-11325 at 3/9/16 5:46 PM:
---

I've pushed a branch off 2.2 at 
[11325-2.2|https://github.com/jkni/cassandra/tree/11325-2.2]. It should merge 
forward cleanly.

Given that the fix is to add one newline character, I've opted not to start a 
CI run, but I can if the reviewer is concerned.


was (Author: jkni):
I've pushed a branch off 2.2 at 
[11325-2.2|https://github.com/jkni/cassandra/tree/11325-2.2].

Given that the fix is to add one newline character, I've opted not to start a 
CI run, but I can if the reviewer is concerned.

> Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf 
> Ant plugin
> -
>
> Key: CASSANDRA-11325
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11325
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Joel Knighton
>Assignee: Joel Knighton
>Priority: Trivial
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> bin/cqlsh is missing a new line - when doing an artifact build, the fixcrlf 
> plugin will insert a newline at the end of bin/cqlsh. This change inserts the 
> newline and changes the mode on cqlsh to 644.
> This isn't a problem affecting release builds, since we explicitly set the 
> mode of files under bin to executable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11325) Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf Ant plugin

2016-03-09 Thread Yuki Morishita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-11325:
---
Reviewer: Yuki Morishita

> Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf 
> Ant plugin
> -
>
> Key: CASSANDRA-11325
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11325
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Joel Knighton
>Assignee: Joel Knighton
>Priority: Trivial
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> bin/cqlsh is missing a new line - when doing an artifact build, the fixcrlf 
> plugin will insert a newline at the end of bin/cqlsh. This change inserts the 
> newline and changes the mode on cqlsh to 644.
> This isn't a problem affecting release builds, since we explicitly set the 
> mode of files under bin to executable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11327) Maintain a histogram of times when writes are blocked due to no available memory

2016-03-09 Thread Ariel Weisberg (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-11327:
---
Description: 
I have a theory that part of the reason C* is so sensitive to timeouts during 
saturating write load is that throughput is basically a sawtooth with valleys 
at zero. This is something I have observed and it gets worse as you add 2i to a 
table or do anything that decreases the throughput of flushing.

I think the fix for this is to incrementally release memory pinned by memtables 
and 2i during flushing instead of releasing it all at once. I know that's not 
really possible, but we can fake it with memory accounting that tracks how 
close to completion flushing is and releases permits for additional memory. 
This will lead to a bit of a sawtooth in real memory usage, but we can account 
for that so the peak footprint is the same.

I think the end result of this change will be a sawtooth, but the valley of the 
sawtooth will not be zero it will be the rate at which flushing progresses. 
Optimizing the rate at which flushing progresses and it's fairness with other 
work can then be tackled separately.

Before we do this I think we should demonstrate that pinned memory due to 
flushing is actually the issue by getting better visibility into the 
distribution of instances of not having any memory by maintaining a histogram 
of spans of time where no memory is available and a thread is blocked.

[MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
 should be a relatively straightforward entry point for this. The first thread 
to block can mark the start of memory starvation and the last thread out can 
mark the end. Have a periodic task that tracks the amount of time spent blocked 
per interval of time and if it is greater than some threshold log with more 
details, possibly at debug.

  was:
I have a theory that part of the reason C* is so sensitive to timeouts during 
saturating write load is that throughput is basically a sawtooth with valleys 
at zero. This is something I have observed and it gets worse as you add 2i to a 
table or do anything that decreases the throughput of flushing.

I think the fix for this is to incrementally release memory pinned by memtables 
and 2i during flushing instead of releasing it all at once. I know that's not 
really possible, but we can fake it with memory accounting that tracks how 
close to completion flushing is and releases permits for additional memory. 
This will lead to a bit of a sawtooth in real memory usage, but we can account 
for that so the peak footprint is the same.

I think the end result of this change will be a sawtooth, but the valley of the 
sawtooth will not be zero it will be the rate at which flushing progresses. 
Optimizing the rate at which flushing progresses and it's fairness with other 
work can then be tackled separately.

Before we do this I think we should demonstrate that pinned memory due to 
flushing is actually the issue by getting better visibility into the 
distribution of instances of not having any memory by maintaining a histogram 
of spans of time where no memory is available and a thread is blocked.

[MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
 should be a relative straightforward entry point for this. The first thread to 
block can mark the start of memory starvation and the last thread out can mark 
the end. Have a periodic task that tracks the amount of time spent blocked per 
interval of time and if it is greater than some threshold log with more 
details, possibly at debug.


> Maintain a histogram of times when writes are blocked due to no available 
> memory
> 
>
> Key: CASSANDRA-11327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11327
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Ariel Weisberg
>
> I have a theory that part of the reason C* is so sensitive to timeouts during 
> saturating write load is that throughput is basically a sawtooth with valleys 
> at zero. This is something I have observed and it gets worse as you add 2i to 
> a table or do anything that decreases the throughput of flushing.
> I think the fix for this is to incrementally release memory pinned by 
> memtables and 2i during flushing instead of releasing it all at once. I know 
> that's not really possible, but we can fake it with memory accounting that 
> tracks how close to completion flushing is and releases permits for 
> additional memory. This will lead to a bit of a sawtooth in real memory

[jira] [Updated] (CASSANDRA-11327) Maintain a histogram of times when writes are blocked due to no available memory

2016-03-09 Thread Ariel Weisberg (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-11327:
---
Description: 
I have a theory that part of the reason C* is so sensitive to timeouts during 
saturating write load is that throughput is basically a sawtooth with valleys 
at zero. This is something I have observed and it gets worse as you add 2i to a 
table or do anything that decreases the throughput of flushing.

I think the fix for this is to incrementally release memory pinned by memtables 
and 2i during flushing instead of releasing it all at once. I know that's not 
really possible, but we can fake it with memory accounting that tracks how 
close to completion flushing is and releases permits for additional memory. 
This will lead to a bit of a sawtooth in real memory usage, but we can account 
for that so the peak footprint is the same.

I think the end result of this change will be a sawtooth, but the valley of the 
sawtooth will not be zero it will be the rate at which flushing progresses. 
Optimizing the rate at which flushing progresses and it's fairness with other 
work can then be tackled separately.

Before we do this I think we should demonstrate that pinned memory due to 
flushing is actually the issue by getting better visibility into the 
distribution of instances of not having any memory by maintaining a histogram 
of spans of time where no memory is available and a thread is blocked.

[MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
 should be a relative straightforward entry point for this. The first thread to 
block can mark the start of memory starvation and the last thread out can mark 
the end. Have a periodic task that tracks the amount of time spent blocked per 
interval of time and if it is greater than some threshold log with more 
details, possibly at debug.

  was:
I have a theory that part of the reason C* is so sensitive to timeouts during 
saturating write load is that throughput is basically a sawtooth with valleys 
at zero. This is something I have observed and it gets worse as you add 2i to a 
table or do anything that decreases the throughput of flushing.

I think the fix for this is to incrementally release memory pinned by memtables 
and 2i during flushing instead of releasing it all at once. I know that's not 
really possible, but we can fake it with memory accounting that tracks how 
close to completion flushing is and releases permits for additional memory. 
This will lead to a bit of a sawtooth in real memory usage, but we can account 
for so that the peak footprint is the same.

I think the end result of this change will be a sawtooth, but the valley of the 
sawtooth will not be zero it will be the rate at which flushing progresses. 
Optimizing the rate at which flushing progresses and it's fairness with other 
work can then be tackled separately.

Before we do this I think we should demonstrate that pinned memory due to 
flushing is actually the issue by getting better visibility into the 
distribution of instances of not having any memory by maintaining a histogram 
of spans of time where no memory is available and a thread is blocked.

[MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
 should be a relative straightforward entry point for this. The first thread to 
block can mark the start of memory starvation and the last thread out can mark 
the end. Have a periodic task that tracks the amount of time spent blocked per 
interval of time and if it is greater than some threshold log with more 
details, possibly at debug.


> Maintain a histogram of times when writes are blocked due to no available 
> memory
> 
>
> Key: CASSANDRA-11327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11327
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Ariel Weisberg
>
> I have a theory that part of the reason C* is so sensitive to timeouts during 
> saturating write load is that throughput is basically a sawtooth with valleys 
> at zero. This is something I have observed and it gets worse as you add 2i to 
> a table or do anything that decreases the throughput of flushing.
> I think the fix for this is to incrementally release memory pinned by 
> memtables and 2i during flushing instead of releasing it all at once. I know 
> that's not really possible, but we can fake it with memory accounting that 
> tracks how close to completion flushing is and releases permits for 
> additional memory. This will lead to a bit of a sawtooth in real memory

[jira] [Comment Edited] (CASSANDRA-11325) Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf Ant plugin

2016-03-09 Thread Joel Knighton (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187457#comment-15187457
 ] 

Joel Knighton edited comment on CASSANDRA-11325 at 3/9/16 5:29 PM:
---

I've pushed a branch off 2.2 at 
[11325-2.2|https://github.com/jkni/cassandra/tree/11325-2.2].

Given that the fix is to add one newline character, I've opted not to start a 
CI run, but I can if the reviewer is concerned.


was (Author: jkni):
I've pushed a branch off 2.2 at 
[11325-2.2|https://github.com/jkni/cassandra/commit/53f3a32a8c6eff578fccf94953db9c6d932f274e].

Given that the fix is to add one newline character, I've opted not to start a 
CI run, but I can if the reviewer is concerned.

> Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf 
> Ant plugin
> -
>
> Key: CASSANDRA-11325
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11325
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Joel Knighton
>Assignee: Joel Knighton
>Priority: Trivial
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> bin/cqlsh is missing a new line - when doing an artifact build, the fixcrlf 
> plugin will insert a newline at the end of bin/cqlsh. This change inserts the 
> newline and changes the mode on cqlsh to 644.
> This isn't a problem affecting release builds, since we explicitly set the 
> mode of files under bin to executable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11325) Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf Ant plugin

2016-03-09 Thread Joel Knighton (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-11325:
--
Status: Patch Available  (was: Open)

I've pushed a branch off 2.2 at 
[11325-2.2|https://github.com/jkni/cassandra/commit/53f3a32a8c6eff578fccf94953db9c6d932f274e].

Given that the fix is to add one newline character, I've opted not to start a 
CI run, but I can if the reviewer is concerned.

> Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf 
> Ant plugin
> -
>
> Key: CASSANDRA-11325
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11325
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Joel Knighton
>Assignee: Joel Knighton
>Priority: Trivial
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> bin/cqlsh is missing a new line - when doing an artifact build, the fixcrlf 
> plugin will insert a newline at the end of bin/cqlsh. This change inserts the 
> newline and changes the mode on cqlsh to 644.
> This isn't a problem affecting release builds, since we explicitly set the 
> mode of files under bin to executable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11327) Maintain a histogram of times when writes are blocked due to no available memory

2016-03-09 Thread Ariel Weisberg (JIRA) 
Ariel Weisberg created CASSANDRA-11327:
--

 Summary: Maintain a histogram of times when writes are blocked due 
to no available memory
 Key: CASSANDRA-11327
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11327
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Ariel Weisberg


I have a theory that part of the reason C* is so sensitive to timeouts during 
saturating write load is that throughput is basically a sawtooth with valleys 
at zero. This is something I have observed and it gets worse as you add 2i to a 
table or do anything that decreases the throughput of flushing.

I think the fix for this is to incrementally release memory pinned by memtables 
and 2i during flushing instead of releasing it all at once. I know that's not 
really possible, but we can fake it with memory accounting that tracks how 
close to completion flushing is and releases permits for additional memory. 
This will lead to a bit of a sawtooth in real memory usage, but we can account 
for so that the peak footprint is the same.

I think the end result of this change will be a sawtooth, but the valley of the 
sawtooth will not be zero it will be the rate at which flushing progresses. 
Optimizing the rate at which flushing progresses and it's fairness with other 
work can then be tackled separately.

Before we do this I think we should demonstrate that pinned memory due to 
flushing is actually the issue by getting better visibility into the 
distribution of instances of not having any memory by maintaining a histogram 
of spans of time where no memory is available and a thread is blocked.

[MemtableAllocatr$SubPool.allocate(long)|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/utils/memory/MemtableAllocator.java#L186]
 should be a relative straightforward entry point for this. The first thread to 
block can mark the start of memory starvation and the last thread out can mark 
the end. Have a periodic task that tracks the amount of time spent blocked per 
interval of time and if it is greater than some threshold log with more 
details, possibly at debug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11326) dtest failure in thrift_tests.TestMutations.test_bad_calls

2016-03-09 Thread Jim Witschey (JIRA) 
Jim Witschey created CASSANDRA-11326:


 Summary: dtest failure in thrift_tests.TestMutations.test_bad_calls
 Key: CASSANDRA-11326
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11326
 Project: Cassandra
  Issue Type: Test
Reporter: Jim Witschey
Assignee: DS Test Eng


This test failed on two consecutive CassCI builds on trunk, but hasn't failed 
otherwise:

http://cassci.datastax.com/job/trunk_dtest/1041/testReport/thrift_tests/TestMutations/test_bad_calls

I can't see anything obvious in C* itself that would have caused this error, 
and when I ran it locally on the 2 C* SHAs on which the test failed on CassCI 
(615d0e15551cbb7e8f5100b33723562c31876889 and 
e017f9494844234fa73848890347f59c622cea40), it passed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11310) Allow filtering on clustering columns for queries without secondary indexes

2016-03-09 Thread Benjamin Lerer (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187376#comment-15187376
 ] 

Benjamin Lerer commented on CASSANDRA-11310:


{quote}I'm not sure how fast I can pull it out, but I'll do my best.{quote}

Take your time. There is no pressure :-). 
It is an improvement which means that the next release for it is 3.6 anyway 
(May).
 

> Allow filtering on clustering columns for queries without secondary indexes
> ---
>
> Key: CASSANDRA-11310
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11310
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Alex Petrov
>  Labels: doc-impacting
> Fix For: 3.x
>
>
> Since CASSANDRA-6377 queries without index filtering non-primary key columns 
> are fully supported.
> It makes sense to also support filtering on clustering-columns.
> {code}
> CREATE TABLE emp_table2 (
> empID int,
> firstname text,
> lastname text,
> b_mon text,
> b_day text,
> b_yr text,
> PRIMARY KEY (empID, b_yr, b_mon, b_day));
> SELECT b_mon,b_day,b_yr,firstname,lastname FROM emp_table2
> WHERE b_mon='oct' ALLOW FILTERING;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] cassandra git commit: Bump base.version

2016-03-09 Thread aleksey 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.5 a38a4e5e5 -> e33d4be66
  refs/heads/trunk 30784fd29 -> dc480c794


Bump base.version


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e33d4be6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e33d4be6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e33d4be6

Branch: refs/heads/cassandra-3.5
Commit: e33d4be66cd94ab6c297a51df80d0a41700302a9
Parents: a38a4e5
Author: Aleksey Yeschenko 
Authored: Wed Mar 9 16:37:07 2016 +
Committer: Aleksey Yeschenko 
Committed: Wed Mar 9 16:37:07 2016 +

--
 build.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e33d4be6/build.xml
--
diff --git a/build.xml b/build.xml
index 113b730..264a0be 100644
--- a/build.xml
+++ b/build.xml
@@ -25,7 +25,7 @@
 
 
 
-
+
 
 
 http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/>

[3/3] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

2016-03-09 Thread aleksey 
Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dc480c79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dc480c79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dc480c79

Branch: refs/heads/trunk
Commit: dc480c794e8ca49e29ceff6405149273456c984d
Parents: 30784fd e33d4be
Author: Aleksey Yeschenko 
Authored: Wed Mar 9 16:37:17 2016 +
Committer: Aleksey Yeschenko 
Committed: Wed Mar 9 16:37:35 2016 +

--
 build.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dc480c79/build.xml
--
diff --cc build.xml
index 113b730,264a0be..5dc1e06
--- a/build.xml
+++ b/build.xml
@@@ -25,7 -25,7 +25,7 @@@
  
  
  
- 
 -
++
  
  
  http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/>

[2/3] cassandra git commit: Bump base.version

2016-03-09 Thread aleksey 
Bump base.version


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e33d4be6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e33d4be6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e33d4be6

Branch: refs/heads/trunk
Commit: e33d4be66cd94ab6c297a51df80d0a41700302a9
Parents: a38a4e5
Author: Aleksey Yeschenko 
Authored: Wed Mar 9 16:37:07 2016 +
Committer: Aleksey Yeschenko 
Committed: Wed Mar 9 16:37:07 2016 +

--
 build.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e33d4be6/build.xml
--
diff --git a/build.xml b/build.xml
index 113b730..264a0be 100644
--- a/build.xml
+++ b/build.xml
@@ -25,7 +25,7 @@
 
 
 
-
+
 
 
 http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/>

[jira] [Commented] (CASSANDRA-11310) Allow filtering on clustering columns for queries without secondary indexes

2016-03-09 Thread Alex Petrov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187360#comment-15187360
 ] 

Alex Petrov commented on CASSANDRA-11310:
-

Thanks once again for the pointer! 

I somehow thought that it's possible to restrict everything by the bounds, 
which I thought of restricting on the both ends, but in the retrospect I 
understand that it was a rather bad idea.
I've implemented a "stupid" prototype that works for cases like {{SELECT * FROM 
%s WHERE b > 20 and c > 30 ALLOW FILTERING}}, so at least it's clear what to do 
now.

I'm not sure how fast I can pull it out, but I'll do my best.

> Allow filtering on clustering columns for queries without secondary indexes
> ---
>
> Key: CASSANDRA-11310
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11310
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Alex Petrov
>  Labels: doc-impacting
> Fix For: 3.x
>
>
> Since CASSANDRA-6377 queries without index filtering non-primary key columns 
> are fully supported.
> It makes sense to also support filtering on clustering-columns.
> {code}
> CREATE TABLE emp_table2 (
> empID int,
> firstname text,
> lastname text,
> b_mon text,
> b_day text,
> b_yr text,
> PRIMARY KEY (empID, b_yr, b_mon, b_day));
> SELECT b_mon,b_day,b_yr,firstname,lastname FROM emp_table2
> WHERE b_mon='oct' ALLOW FILTERING;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11325) Missing newline at end of bin/cqlsh causes it to be changed by the fixcrlf Ant plugin

2016-03-09 Thread Joel Knighton (JIRA) 
Joel Knighton created CASSANDRA-11325:
-

 Summary: Missing newline at end of bin/cqlsh causes it to be 
changed by the fixcrlf Ant plugin
 Key: CASSANDRA-11325
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11325
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Joel Knighton
Assignee: Joel Knighton
Priority: Trivial
 Fix For: 2.2.x, 3.0.x, 3.x


bin/cqlsh is missing a new line - when doing an artifact build, the fixcrlf 
plugin will insert a newline at the end of bin/cqlsh. This change inserts the 
newline and changes the mode on cqlsh to 644.

This isn't a problem affecting release builds, since we explicitly set the mode 
of files under bin to executable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11324) Make SSTABLESCRUB and related jobs multi-threaded / parallelize somehow

2016-03-09 Thread Thom Valley (JIRA) 
Thom Valley created CASSANDRA-11324:
---

 Summary: Make SSTABLESCRUB and related jobs multi-threaded  / 
parallelize somehow
 Key: CASSANDRA-11324
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11324
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Reporter: Thom Valley
Priority: Minor


SSTABLE scrub right now appears to be single threaded in offline mode (unsure 
about online).  When scrubbing a node, especially a well provisioned one, the 
node sits mostly idle for potentially hours while scrubbing.

Making better use of available resources through parallelization / 
multi-threading of the scrub process seems like a good idea.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7159) sstablemetadata command should print some more stuff

2016-03-09 Thread Joshua McKenzie (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-7159:
---
Status: Patch Available  (was: Open)

> sstablemetadata command should print some more stuff
> 
>
> Key: CASSANDRA-7159
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7159
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jeremiah Jordan
>Priority: Trivial
>  Labels: lhf
> Fix For: 2.1.x
>
> Attachments: 
> CASSANDRA-7159_-_sstablemetadata_command_should_print_some_more_stuff.patch
>
>
> It would be nice if the sstablemetadata command printed out some more of the 
> stuff we track.  Like the Min/Max column names and the min/max token in the 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11310) Allow filtering on clustering columns for queries without secondary indexes

2016-03-09 Thread Benjamin Lerer (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187256#comment-15187256
 ] 

Benjamin Lerer commented on CASSANDRA-11310:


{quote}I hope I'm not bugging you with questions too much.{quote}

Not at all. :-)

{quote}It looks like it all requires changes to MultiCBuilder, since at the 
moment only a single slice is allowed, and it always has to be the "last" one, 
as either buildBoundForSlice or buildBound is returned in 
PrimaryKeyRestrictionSet::boundsAsClustering. At least for the cases with 
slices over multiple columns, such as SELECT * FROM myTable WHERE b < 3 AND c 
<= 4.{quote}

{{MultiCBuilder}} is used to build the {{Clustering}} or the {{Slice.Bound}}. I 
do not think that you need to change it. The way the rows are selected based on 
their clustering columns should stay the same. What will change is the fact 
that the selected rows will be filtered out afterwards. What you need to do is 
to make sure that the columns that should be used for the filtering are not 
used for computing the {{Clustering}} set and the {{Slice.Bound}} set (and 
ideally that the clustering columns that are not used for the filtering are not 
used to build {{RowFilter.Expression}}).

Basically, it is a 2 step process: first we select the rows or range of rows 
that match the specifed {{Clustering}} or {{Slice.Bound}} sets, then we filter 
the selected rows using the remaining restrictions. The first part of the 
process rely on the  {{PrimaryKeyRestrictionSet::valuesAsClustering}} and 
{{PrimaryKeyRestrictionSet::boundsAsClustering methods}}. The second part of 
the process rely on the {{PrimaryKeyRestrictionSet::addRowFilterTo}} method.

> Allow filtering on clustering columns for queries without secondary indexes
> ---
>
> Key: CASSANDRA-11310
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11310
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Alex Petrov
>  Labels: doc-impacting
> Fix For: 3.x
>
>
> Since CASSANDRA-6377 queries without index filtering non-primary key columns 
> are fully supported.
> It makes sense to also support filtering on clustering-columns.
> {code}
> CREATE TABLE emp_table2 (
> empID int,
> firstname text,
> lastname text,
> b_mon text,
> b_day text,
> b_yr text,
> PRIMARY KEY (empID, b_yr, b_mon, b_day));
> SELECT b_mon,b_day,b_yr,firstname,lastname FROM emp_table2
> WHERE b_mon='oct' ALLOW FILTERING;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7190) Add schema to snapshot manifest

2016-03-09 Thread Aleksey Yeschenko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-7190:
-
Assignee: (was: Max Barnash)

> Add schema to snapshot manifest
> ---
>
> Key: CASSANDRA-7190
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7190
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Priority: Minor
> Fix For: 3.x
>
>
> followup from CASSANDRA-6326



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11323) When node runs out of commitlog space you get poor log information

2016-03-09 Thread T Jake Luciani (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-11323:
---
Labels: fallout  (was: )

> When node runs out of commitlog space you get poor log information
> --
>
> Key: CASSANDRA-11323
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11323
> Project: Cassandra
>  Issue Type: Bug
>Reporter: T Jake Luciani
>Priority: Trivial
>  Labels: fallout
>
> {code}
> ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 
> StorageService.java:470 - Stopping gossiper
> WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 
> StorageService.java:377 - Stopping gossip by operator request
> INFO  [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 Gossiper.java:1463 
> - Announcing shutdown
> {code}
> That's all you get when a node runs out of commit log space. 
> We should explicitly callout the fact the commitlog is out of disk.  I see 
> that in the commit log error handler but after it shuts down. So I think it's 
> never getting written before shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11323) When node runs out of commitlog space you get poor log information

2016-03-09 Thread T Jake Luciani (JIRA) 
T Jake Luciani created CASSANDRA-11323:
--

 Summary: When node runs out of commitlog space you get poor log 
information
 Key: CASSANDRA-11323
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11323
 Project: Cassandra
  Issue Type: Bug
Reporter: T Jake Luciani
Priority: Trivial


{code}
ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 
StorageService.java:470 - Stopping gossiper
WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 
StorageService.java:377 - Stopping gossip by operator request
INFO  [PERIODIC-COMMIT-LOG-SYNCER] 2016-03-08 20:27:33,899 Gossiper.java:1463 - 
Announcing shutdown
{code}

That's all you get when a node runs out of commit log space. 

We should explicitly callout the fact the commitlog is out of disk.  I see that 
in the commit log error handler but after it shuts down. So I think it's never 
getting written before shutdown.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10990) Support streaming of older version sstables in 3.0

2016-03-09 Thread Yuki Morishita (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187223#comment-15187223
 ] 

Yuki Morishita commented on CASSANDRA-10990:


I think patch looks good for me with the following little change:

* getTempBufferFile should throw IOException instead of RuntimeException when 
buffer file cannot be created, so stream won't be retried.
* RewindableDISP.reset should throw IOException instead of 
IllegalStateException so that it fits the InputStream's interface.

Can you create patch for 3.0 branch also, and run dtests?

> Support streaming of older version sstables in 3.0
> --
>
> Key: CASSANDRA-10990
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10990
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Jeremy Hanna
>Assignee: Paulo Motta
>
> In 2.0 we introduced support for streaming older versioned sstables 
> (CASSANDRA-5772).  In 3.0, because of the rewrite of the storage layer, this 
> became no longer supported.  So currently, while 3.0 can read sstables in the 
> 2.1/2.2 format, it cannot stream the older versioned sstables.  We should do 
> some work to make this still possible to be consistent with what 
> CASSANDRA-5772 provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11304) Stack overflow when querying 2ndary index

2016-03-09 Thread Sam Tunnicliffe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-11304:

Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Thanks, committed to 3.0 in {{d0b7a02b230f65a3ba2665a8327b6e6cc585dde9}} and 
merged with {{-s ours}} to 3.5 & trunk.

> Stack overflow when querying 2ndary index
> -
>
> Key: CASSANDRA-11304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11304
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, CQL
> Environment: 3 Node cluster / Ubuntu 14.04 / Cassandra 3.0.3
>Reporter: Job Tiel Groenestege
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.5
>
> Attachments: 11304-3.0.txt
>
>
> When reading data through a secondary index _select * from tableName where 
> secIndexField = 'foo'_  (from a Java application) I get the following 
> stacktrace on all nodes; after the query read fails. It happens repeatably 
> when  I rerun the same query:
> {quote}
> WARN  [SharedPool-Worker-8] 2016-03-04 13:26:28,041 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[SharedPool-Worker-8,5,main]: {}
> java.lang.StackOverflowError: null
> at 
> org.apache.cassandra.db.rows.BTreeRow$Builder.build(BTreeRow.java:653) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:436)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardIndexedReader.computeNext(SSTableIterator.java:266)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:153)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:340)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:219)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:428)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:288)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.index.internal.composites.CompositesSearcher$1.prepareNext(CompositesSearcher.java:128)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.index.internal.composites.CompositesSearcher$1.prepareNext(CompositesSearcher.java:133)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.index.internal.composites.CompositesSearcher$1.prepareNext(CompositesSearcher.java:133)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

2016-03-09 Thread samt 
Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a38a4e5e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a38a4e5e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a38a4e5e

Branch: refs/heads/cassandra-3.5
Commit: a38a4e5e548465efe99558951b5ea1271b604c1b
Parents: 96792f6 d0b7a02
Author: Sam Tunnicliffe 
Authored: Wed Mar 9 14:49:26 2016 +
Committer: Sam Tunnicliffe 
Committed: Wed Mar 9 14:49:26 2016 +

--

--

[6/6] cassandra git commit: Merge branch 'cassandra-3.5' into trunk

2016-03-09 Thread samt 
Merge branch 'cassandra-3.5' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/30784fd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/30784fd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/30784fd2

Branch: refs/heads/trunk
Commit: 30784fd294d268deb302662319fba4dcd76fa3e0
Parents: 9841cb7 a38a4e5
Author: Sam Tunnicliffe 
Authored: Wed Mar 9 14:58:19 2016 +
Committer: Sam Tunnicliffe 
Committed: Wed Mar 9 14:58:19 2016 +

--

--

[1/6] cassandra git commit: Replace recursion with iteration in CompositesSearcher

2016-03-09 Thread samt 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 f6af142f1 -> d0b7a02b2
  refs/heads/cassandra-3.5 96792f638 -> a38a4e5e5
  refs/heads/trunk 9841cb72b -> 30784fd29


Replace recursion with iteration in CompositesSearcher

Patch by Sam Tunnicliffe; reviewed by Stefania Alborghetti for
CASSANDRA-11304


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d0b7a02b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d0b7a02b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d0b7a02b

Branch: refs/heads/cassandra-3.0
Commit: d0b7a02b230f65a3ba2665a8327b6e6cc585dde9
Parents: f6af142
Author: Sam Tunnicliffe 
Authored: Tue Mar 8 12:21:14 2016 +
Committer: Sam Tunnicliffe 
Committed: Wed Mar 9 14:42:13 2016 +

--
 CHANGES.txt |   1 +
 .../internal/composites/CompositesSearcher.java | 115 ++-
 2 files changed, 60 insertions(+), 56 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d0b7a02b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 384e7ea..e6c40aa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
  * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
  * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 Merged from 2.2:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d0b7a02b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
--
diff --git 
a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
 
b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
index 765ae4d..135839b 100644
--- 
a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
+++ 
b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
@@ -95,70 +95,73 @@ public class CompositesSearcher extends 
CassandraIndexSearcher
 
 private boolean prepareNext()
 {
-if (next != null)
-return true;
-
-if (nextEntry == null)
+while (true)
 {
-if (!indexHits.hasNext())
-return false;
+if (next != null)
+return true;
 
-nextEntry = index.decodeEntry(indexKey, indexHits.next());
-}
+if (nextEntry == null)
+{
+if (!indexHits.hasNext())
+return false;
 
-// Gather all index hits belonging to the same partition and 
query the data for those hits.
-// TODO: it's much more efficient to do 1 read for all hits to 
the same partition than doing
-// 1 read per index hit. However, this basically mean 
materializing all hits for a partition
-// in memory so we should consider adding some paging 
mechanism. However, index hits should
-// be relatively small so it's much better than the previous 
code that was materializing all
-// *data* for a given partition.
-BTreeSet.Builder clusterings = 
BTreeSet.builder(index.baseCfs.getComparator());
-List entries = new ArrayList<>();
-DecoratedKey partitionKey = 
index.baseCfs.decorateKey(nextEntry.indexedKey);
-
-while (nextEntry != null && 
partitionKey.getKey().equals(nextEntry.indexedKey))
-{
-// We're queried a slice of the index, but some hits may 
not match some of the clustering column constraints
-if (isMatchingEntry(partitionKey, nextEntry, command))
+nextEntry = index.decodeEntry(indexKey, 
indexHits.next());
+}
+
+// Gather all index hits belonging to the same partition 
and query the data for those hits.
+// TODO: it's much more efficient to do 1 read for all 
hits to the same partition than doing
+// 1 read per index hit. However, this basically mean 
materializing all hits for a partition
+// in memory so we should consider adding some paging 
mechanism. However, index hits should
+// be relatively small so it's much better than the 
previous code that was materializing all
+// *data* for a given partition.
+BTreeSet.Builder

[2/6] cassandra git commit: Replace recursion with iteration in CompositesSearcher

2016-03-09 Thread samt 
Replace recursion with iteration in CompositesSearcher

Patch by Sam Tunnicliffe; reviewed by Stefania Alborghetti for
CASSANDRA-11304


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d0b7a02b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d0b7a02b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d0b7a02b

Branch: refs/heads/cassandra-3.5
Commit: d0b7a02b230f65a3ba2665a8327b6e6cc585dde9
Parents: f6af142
Author: Sam Tunnicliffe 
Authored: Tue Mar 8 12:21:14 2016 +
Committer: Sam Tunnicliffe 
Committed: Wed Mar 9 14:42:13 2016 +

--
 CHANGES.txt |   1 +
 .../internal/composites/CompositesSearcher.java | 115 ++-
 2 files changed, 60 insertions(+), 56 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d0b7a02b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 384e7ea..e6c40aa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
  * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
  * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 Merged from 2.2:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d0b7a02b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
--
diff --git 
a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
 
b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
index 765ae4d..135839b 100644
--- 
a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
+++ 
b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
@@ -95,70 +95,73 @@ public class CompositesSearcher extends 
CassandraIndexSearcher
 
 private boolean prepareNext()
 {
-if (next != null)
-return true;
-
-if (nextEntry == null)
+while (true)
 {
-if (!indexHits.hasNext())
-return false;
+if (next != null)
+return true;
 
-nextEntry = index.decodeEntry(indexKey, indexHits.next());
-}
+if (nextEntry == null)
+{
+if (!indexHits.hasNext())
+return false;
 
-// Gather all index hits belonging to the same partition and 
query the data for those hits.
-// TODO: it's much more efficient to do 1 read for all hits to 
the same partition than doing
-// 1 read per index hit. However, this basically mean 
materializing all hits for a partition
-// in memory so we should consider adding some paging 
mechanism. However, index hits should
-// be relatively small so it's much better than the previous 
code that was materializing all
-// *data* for a given partition.
-BTreeSet.Builder clusterings = 
BTreeSet.builder(index.baseCfs.getComparator());
-List entries = new ArrayList<>();
-DecoratedKey partitionKey = 
index.baseCfs.decorateKey(nextEntry.indexedKey);
-
-while (nextEntry != null && 
partitionKey.getKey().equals(nextEntry.indexedKey))
-{
-// We're queried a slice of the index, but some hits may 
not match some of the clustering column constraints
-if (isMatchingEntry(partitionKey, nextEntry, command))
+nextEntry = index.decodeEntry(indexKey, 
indexHits.next());
+}
+
+// Gather all index hits belonging to the same partition 
and query the data for those hits.
+// TODO: it's much more efficient to do 1 read for all 
hits to the same partition than doing
+// 1 read per index hit. However, this basically mean 
materializing all hits for a partition
+// in memory so we should consider adding some paging 
mechanism. However, index hits should
+// be relatively small so it's much better than the 
previous code that was materializing all
+// *data* for a given partition.
+BTreeSet.Builder clusterings = 
BTreeSet.builder(index.baseCfs.getComparator());
+List entries = new ArrayList<>();
+DecoratedKey partitionKey =

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

2016-03-09 Thread samt 
Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a38a4e5e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a38a4e5e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a38a4e5e

Branch: refs/heads/trunk
Commit: a38a4e5e548465efe99558951b5ea1271b604c1b
Parents: 96792f6 d0b7a02
Author: Sam Tunnicliffe 
Authored: Wed Mar 9 14:49:26 2016 +
Committer: Sam Tunnicliffe 
Committed: Wed Mar 9 14:49:26 2016 +

--

--

[3/6] cassandra git commit: Replace recursion with iteration in CompositesSearcher

2016-03-09 Thread samt 
Replace recursion with iteration in CompositesSearcher

Patch by Sam Tunnicliffe; reviewed by Stefania Alborghetti for
CASSANDRA-11304


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d0b7a02b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d0b7a02b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d0b7a02b

Branch: refs/heads/trunk
Commit: d0b7a02b230f65a3ba2665a8327b6e6cc585dde9
Parents: f6af142
Author: Sam Tunnicliffe 
Authored: Tue Mar 8 12:21:14 2016 +
Committer: Sam Tunnicliffe 
Committed: Wed Mar 9 14:42:13 2016 +

--
 CHANGES.txt |   1 +
 .../internal/composites/CompositesSearcher.java | 115 ++-
 2 files changed, 60 insertions(+), 56 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d0b7a02b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 384e7ea..e6c40aa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
  * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
  * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 Merged from 2.2:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d0b7a02b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
--
diff --git 
a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
 
b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
index 765ae4d..135839b 100644
--- 
a/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
+++ 
b/src/java/org/apache/cassandra/index/internal/composites/CompositesSearcher.java
@@ -95,70 +95,73 @@ public class CompositesSearcher extends 
CassandraIndexSearcher
 
 private boolean prepareNext()
 {
-if (next != null)
-return true;
-
-if (nextEntry == null)
+while (true)
 {
-if (!indexHits.hasNext())
-return false;
+if (next != null)
+return true;
 
-nextEntry = index.decodeEntry(indexKey, indexHits.next());
-}
+if (nextEntry == null)
+{
+if (!indexHits.hasNext())
+return false;
 
-// Gather all index hits belonging to the same partition and 
query the data for those hits.
-// TODO: it's much more efficient to do 1 read for all hits to 
the same partition than doing
-// 1 read per index hit. However, this basically mean 
materializing all hits for a partition
-// in memory so we should consider adding some paging 
mechanism. However, index hits should
-// be relatively small so it's much better than the previous 
code that was materializing all
-// *data* for a given partition.
-BTreeSet.Builder clusterings = 
BTreeSet.builder(index.baseCfs.getComparator());
-List entries = new ArrayList<>();
-DecoratedKey partitionKey = 
index.baseCfs.decorateKey(nextEntry.indexedKey);
-
-while (nextEntry != null && 
partitionKey.getKey().equals(nextEntry.indexedKey))
-{
-// We're queried a slice of the index, but some hits may 
not match some of the clustering column constraints
-if (isMatchingEntry(partitionKey, nextEntry, command))
+nextEntry = index.decodeEntry(indexKey, 
indexHits.next());
+}
+
+// Gather all index hits belonging to the same partition 
and query the data for those hits.
+// TODO: it's much more efficient to do 1 read for all 
hits to the same partition than doing
+// 1 read per index hit. However, this basically mean 
materializing all hits for a partition
+// in memory so we should consider adding some paging 
mechanism. However, index hits should
+// be relatively small so it's much better than the 
previous code that was materializing all
+// *data* for a given partition.
+BTreeSet.Builder clusterings = 
BTreeSet.builder(index.baseCfs.getComparator());
+List entries = new ArrayList<>();
+DecoratedKey partitionKey =

[jira] [Created] (CASSANDRA-11322) dtest failure in compaction_test.TestCompaction_with_LeveledCompactionStrategy.data_size_test

2016-03-09 Thread Jim Witschey (JIRA) 
Jim Witschey created CASSANDRA-11322:


 Summary: dtest failure in 
compaction_test.TestCompaction_with_LeveledCompactionStrategy.data_size_test
 Key: CASSANDRA-11322
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11322
 Project: Cassandra
  Issue Type: Test
Reporter: Jim Witschey
Assignee: DS Test Eng


This dtest has failed once:

http://cassci.datastax.com/job/cassandra-3.0_dtest/597/testReport/compaction_test/TestCompaction_with_LeveledCompactionStrategy/data_size_test

Here's the history for this test:

http://cassci.datastax.com/job/cassandra-3.0_dtest/597/testReport/compaction_test/TestCompaction_with_LeveledCompactionStrategy/data_size_test/history/

It failed at this line:

https://github.com/riptano/cassandra-dtest/blob/88a74d7/compaction_test.py#L86

Basically, it ran compaction over the default stress tables, but timed out 
waiting to see the line {{Compacted }} in the log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10331) Establish and implement canonical bulk reading workload(s)

2016-03-09 Thread T Jake Luciani (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187155#comment-15187155
 ] 

T Jake Luciani commented on CASSANDRA-10331:


+1 since this is stress I'm happy to commit this to 3.5 if you agree

> Establish and implement canonical bulk reading workload(s)
> --
>
> Key: CASSANDRA-10331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10331
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ariel Weisberg
>Assignee: Stefania
> Fix For: 3.x
>
>
> Implement a client, use stress, or extend stress to a bulk reading workload 
> that is indicative of the performance we are trying to improve.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11296) Run dtests with -Dcassandra.debugrefcount=true and increase checking frequency

2016-03-09 Thread Marcus Eriksson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187161#comment-15187161
 ] 

Marcus Eriksson commented on CASSANDRA-11296:
-

[~aweisberg] could you have a quick look at the new errors?

> Run dtests with -Dcassandra.debugrefcount=true and increase checking frequency
> --
>
> Key: CASSANDRA-11296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11296
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>
> We should run dtests with refcount debugging and check every second instead 
> of every 15 minutes 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10805) Additional Compaction Logging

2016-03-09 Thread Marcus Eriksson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-10805:

Status: Patch Available  (was: Open)

setting as patch available

> Additional Compaction Logging
> -
>
> Key: CASSANDRA-10805
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10805
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction, Observability
>Reporter: Carl Yeksigian
>Assignee: Carl Yeksigian
>Priority: Minor
> Fix For: 3.x
>
>
> Currently, viewing the results of past compactions requires parsing the log 
> and looking at the compaction history system table, which doesn't have 
> information about, for example, flushed sstables not previously compacted.
> This is a proposal to extend the information captured for compaction. 
> Initially, this would be done through a JMX call, but if it proves to be 
> useful and not much overhead, it might be a feature that could be enabled for 
> the compaction strategy all the time.
> Initial log information would include:
> - The compaction strategy type controlling each column family
> - The set of sstables included in each compaction strategy
> - Information about flushes and compactions, including times and all involved 
> sstables
> - Information about sstables, including generation, size, and tokens
> - Any additional metadata the strategy wishes to add to a compaction or an 
> sstable, like the level of an sstable or the type of compaction being 
> performed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11310) Allow filtering on clustering columns for queries without secondary indexes

2016-03-09 Thread Alex Petrov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187154#comment-15187154
 ] 

Alex Petrov edited comment on CASSANDRA-11310 at 3/9/16 2:31 PM:
-

I hope I'm not bugging you with questions too much. 
It looks like it all requires changes to {{MultiCBuilder}}, since at the moment 
only a single slice is allowed, and it always has to be the "last" one, as 
either {{buildBoundForSlice}} or {{buildBound}} is returned in 
{{PrimaryKeyRestrictionSet::boundsAsClustering}}. At least for the cases with 
slices over multiple columns, such as {{SELECT * FROM myTable WHERE b < 3 AND c 
<= 4}}.
Thank you for your help.


was (Author: ifesdjeen):
I hope I'm not bugging you with questions too much. 
It looks like it all requires changes to {{MultiCBuilder}}, since at the moment 
only a single slice is allowed, and it always has to be the "last" one, as 
either {{buildBoundForSlice}} or {{buildBound}} is returned in 
{{PrimaryKeyRestrictionSet::boundsAsClustering}}. At least for the cases with 
slices over multiple columns, such as `SELECT * FROM myTable WHERE b < 3 AND c 
<= 4`.
Thank you for your help.

> Allow filtering on clustering columns for queries without secondary indexes
> ---
>
> Key: CASSANDRA-11310
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11310
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Alex Petrov
>  Labels: doc-impacting
> Fix For: 3.x
>
>
> Since CASSANDRA-6377 queries without index filtering non-primary key columns 
> are fully supported.
> It makes sense to also support filtering on clustering-columns.
> {code}
> CREATE TABLE emp_table2 (
> empID int,
> firstname text,
> lastname text,
> b_mon text,
> b_day text,
> b_yr text,
> PRIMARY KEY (empID, b_yr, b_mon, b_day));
> SELECT b_mon,b_day,b_yr,firstname,lastname FROM emp_table2
> WHERE b_mon='oct' ALLOW FILTERING;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11310) Allow filtering on clustering columns for queries without secondary indexes

2016-03-09 Thread Alex Petrov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187154#comment-15187154
 ] 

Alex Petrov edited comment on CASSANDRA-11310 at 3/9/16 2:30 PM:
-

I hope I'm not bugging you with questions too much. 
It looks like it all requires changes to {MultiCBuilder}, since at the moment 
only a single slice is allowed, and it always has to be the "last" one, as 
either `buildBoundForSlice` or `buildBound` is returned in 
`PrimaryKeyRestrictionSet::boundsAsClustering`. At least for the cases with 
slices over multiple columns, such as `SELECT * FROM myTable WHERE b < 3 AND c 
<= 4`.
Thank you for your help.


was (Author: ifesdjeen):
I hope I'm not bugging you with questions too much. 
It looks like it all requires changes to `MultiCBuilder`, since at the moment 
only a single slice is allowed, and it always has to be the "last" one, as 
either `buildBoundForSlice` or `buildBound` is returned in 
`PrimaryKeyRestrictionSet::boundsAsClustering`. At least for the cases with 
slices over multiple columns, such as `SELECT * FROM myTable WHERE b < 3 AND c 
<= 4`.
Thank you for your help.

> Allow filtering on clustering columns for queries without secondary indexes
> ---
>
> Key: CASSANDRA-11310
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11310
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Alex Petrov
>  Labels: doc-impacting
> Fix For: 3.x
>
>
> Since CASSANDRA-6377 queries without index filtering non-primary key columns 
> are fully supported.
> It makes sense to also support filtering on clustering-columns.
> {code}
> CREATE TABLE emp_table2 (
> empID int,
> firstname text,
> lastname text,
> b_mon text,
> b_day text,
> b_yr text,
> PRIMARY KEY (empID, b_yr, b_mon, b_day));
> SELECT b_mon,b_day,b_yr,firstname,lastname FROM emp_table2
> WHERE b_mon='oct' ALLOW FILTERING;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11310) Allow filtering on clustering columns for queries without secondary indexes

2016-03-09 Thread Alex Petrov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187154#comment-15187154
 ] 

Alex Petrov commented on CASSANDRA-11310:
-

I hope I'm not bugging you with questions too much. 
It looks like it all requires changes to `MultiCBuilder`, since at the moment 
only a single slice is allowed, and it always has to be the "last" one, as 
either `buildBoundForSlice` or `buildBound` is returned in 
`PrimaryKeyRestrictionSet::boundsAsClustering`. At least for the cases with 
slices over multiple columns, such as `SELECT * FROM myTable WHERE b < 3 AND c 
<= 4`.
Thank you for your help.

> Allow filtering on clustering columns for queries without secondary indexes
> ---
>
> Key: CASSANDRA-11310
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11310
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Alex Petrov
>  Labels: doc-impacting
> Fix For: 3.x
>
>
> Since CASSANDRA-6377 queries without index filtering non-primary key columns 
> are fully supported.
> It makes sense to also support filtering on clustering-columns.
> {code}
> CREATE TABLE emp_table2 (
> empID int,
> firstname text,
> lastname text,
> b_mon text,
> b_day text,
> b_yr text,
> PRIMARY KEY (empID, b_yr, b_mon, b_day));
> SELECT b_mon,b_day,b_yr,firstname,lastname FROM emp_table2
> WHERE b_mon='oct' ALLOW FILTERING;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

  1   2   >