[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427938#comment-16427938
 ] 

Michael Shuler edited comment on CASSANDRA-14281 at 4/6/18 4:17 AM:


!bench2.png|width=700,height=400!

Converted the TIFF bench.png to PNG


was (Author: mshuler):
!bench2.png!

Converted the TIFF bench.png to PNG

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png, bench2.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-14281:
---
Attachment: bench2.png

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png, bench2.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427938#comment-16427938
 ] 

Michael Shuler commented on CASSANDRA-14281:


!bench2.png!

Converted the TIFF bench.png to PNG

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png, bench2.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-14281:
---
Attachment: (was: bench.png)

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-14281:
---
Attachment: bench.png

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png, bench.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-14281:
--
Attachment: bench.png

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427874#comment-16427874
 ] 

Chris Lohfink edited comment on CASSANDRA-14281 at 4/6/18 3:15 AM:
---

I dont see any issues with code and thats an improvement certainly. I tried 
cassandra-stress locally and

!bench.png|width=799,height=417!

didnt get an improvement but it maybe different with a faster host (tried 8-128 
threads in powers of 2)


was (Author: cnlwsu):
I dont see any issues with code and thats an improvement certainly. I tried 
cassandra-stress locally and

!image-2018-04-05-22-06-05-947.png|width=799,height=417!

didnt get an improvement but it maybe different with a faster host (tried 8-128 
threads in powers of 2)

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-14281:
--
Attachment: (was: image-2018-04-05-22-06-05-947.png)

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: bench.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13403) nodetool repair breaks SASI index

[Anonymous (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anonymous updated CASSANDRA-13403:
--
Status: Ready to Commit  (was: Patch Available)

> nodetool repair breaks SASI index
> -
>
> Key: CASSANDRA-13403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13403
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: 3.10
>Reporter: Igor Novgorodov
>Assignee: Alex Petrov
>Priority: Major
>  Labels: patch
> Attachments: 3_nodes_compaction.log, 4_nodes_compaction.log, 
> CASSANDRA-13403.patch, testSASIRepair.patch
>
>
> I've got table:
> {code}
> CREATE TABLE cservice.bulks_recipients (
> recipient text,
> bulk_id uuid,
> datetime_final timestamp,
> datetime_sent timestamp,
> request_id uuid,
> status int,
> PRIMARY KEY (recipient, bulk_id)
> ) WITH CLUSTERING ORDER BY (bulk_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX bulk_recipients_bulk_id ON cservice.bulks_recipients 
> (bulk_id) USING 'org.apache.cassandra.index.sasi.SASIIndex';
> {code}
> There are 11 rows in it:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> Let's query by index (all rows have the same *bulk_id*):
> {code}
> > select * from bulks_recipients where bulk_id = 
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;   
> >   
> ...
> (11 rows)
> {code}
> Ok, everything is fine.
> Now i'm doing *nodetool repair --partitioner-range --job-threads 4 --full* on 
> each node in cluster sequentially.
> After it finished:
> {code}
> > select * from bulks_recipients where bulk_id = 
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;
> ...
> (2 rows)
> {code}
> Only two rows.
> While the rows are actually there:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> If i issue an incremental repair on a random node, i can get like 7 rows 
> after index query.
> Dropping index and recreating it fixes the issue. Is it a bug or am i doing 
> the repair the wrong way?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427874#comment-16427874
 ] 

Chris Lohfink edited comment on CASSANDRA-14281 at 4/6/18 3:12 AM:
---

I dont see any issues with code and thats an improvement certainly. I tried 
cassandra-stress locally and

!image-2018-04-05-22-06-05-947.png|width=799,height=417!

didnt get an improvement but it maybe different with a faster host (tried 8-128 
threads in powers of 2)


was (Author: cnlwsu):
I dont see any issues with code and thats an improvement certainly. I tried 
cassandra-stress locally and

!image-2018-04-05-22-06-05-947.png|width=799,height=417!

didnt get an improvement but it maybe different with a faster host (tried 8-128 
threads in powers of 2)

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: image-2018-04-05-22-06-05-947.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427876#comment-16427876
 ] 

Chris Lohfink commented on CASSANDRA-14281:
---

I am +1 for code if that was a vague comment

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: image-2018-04-05-22-06-05-947.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427874#comment-16427874
 ] 

Chris Lohfink commented on CASSANDRA-14281:
---

I dont see any issues with code and thats an improvement certainly. I tried 
cassandra-stress locally and

!image-2018-04-05-22-06-05-947.png|width=799,height=417!

didnt get an improvement but it maybe different with a faster host (tried 8-128 
threads in powers of 2)

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: image-2018-04-05-22-06-05-947.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-14281:
--
Attachment: image-2018-04-05-22-06-05-947.png

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
> Attachments: image-2018-04-05-22-06-05-947.png
>
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: a map is the same size as it's keySet or values collection, so just use that

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 23787b5f8 -> 478c1a9fd


a map is the same size as it's keySet or values collection, so just use that


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/478c1a9f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/478c1a9f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/478c1a9f

Branch: refs/heads/trunk
Commit: 478c1a9fdf027af0f373f9e26521803ae8775fdb
Parents: 23787b5
Author: Dave Brosius 
Authored: Thu Apr 5 22:54:02 2018 -0400
Committer: Dave Brosius 
Committed: Thu Apr 5 22:54:02 2018 -0400

--
 src/java/org/apache/cassandra/hadoop/cql3/CqlRecordWriter.java | 2 +-
 src/java/org/apache/cassandra/tools/nodetool/Status.java   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/478c1a9f/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordWriter.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordWriter.java 
b/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordWriter.java
index d2a0d86..989d154 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordWriter.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordWriter.java
@@ -438,7 +438,7 @@ class CqlRecordWriter extends RecordWriter, Listhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/478c1a9f/src/java/org/apache/cassandra/tools/nodetool/Status.java
--
diff --git a/src/java/org/apache/cassandra/tools/nodetool/Status.java 
b/src/java/org/apache/cassandra/tools/nodetool/Status.java
index 10c7fdf..b2f9c19 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/Status.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/Status.java
@@ -91,7 +91,7 @@ public class Status extends NodeToolCmd
 SortedMap dcs = 
NodeTool.getOwnershipByDcWithPort(probe, resolveIp, tokensToEndpoints, 
ownerships);
 
 // More tokens than nodes (aka vnodes)?
-if (dcs.values().size() < tokensToEndpoints.keySet().size())
+if (dcs.size() < tokensToEndpoints.size())
 isTokenPerNode = false;
 
 findMaxAddressLengthWithPort(dcs);


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

[Dave Brosius (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Brosius updated CASSANDRA-14367:
-
Attachment: 14367.txt

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

[Dave Brosius (JIRA)]

Dave Brosius created CASSANDRA-14367:


 Summary: prefer Collections.singletonList to 
Arrays.asList(one_element)
 Key: CASSANDRA-14367
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Dave Brosius
Assignee: Dave Brosius
 Fix For: 4.x


small improvement, but Arrays.asList first creates an array, then wraps it with 
a collections instance, whereas Collections.singletonList just creates one 
small (one field) bean instance.

so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-10023) Emit a metric for number of local read and write calls

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves reassigned CASSANDRA-10023:


Assignee: Damien Stevenson  (was: Tushar Agrawal)

> Emit a metric for number of local read and write calls
> --
>
> Key: CASSANDRA-10023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10023
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Damien Stevenson
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: CASSANDRA-10023.patch
>
>
> Many C* drivers have feature to be replica aware and chose the co-ordinator 
> which is a replica. We should add a metric which tells us whether all calls 
> to the co-ordinator are replica aware.
> We have seen issues where client thinks they are replica aware when they 
> forget to add routing key at various places in the code. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: c* uses commons-lang3, not commons-lang

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 1fafcb019 -> 23787b5f8


c* uses commons-lang3, not commons-lang


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/23787b5f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/23787b5f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/23787b5f

Branch: refs/heads/trunk
Commit: 23787b5f8b42cf2cce430ead3445528941e39aac
Parents: 1fafcb0
Author: Dave Brosius 
Authored: Thu Apr 5 21:52:44 2018 -0400
Committer: Dave Brosius 
Committed: Thu Apr 5 21:52:44 2018 -0400

--
 src/java/org/apache/cassandra/cql3/Attributes.java   | 4 ++--
 src/java/org/apache/cassandra/cql3/BatchQueryOptions.java| 4 ++--
 src/java/org/apache/cassandra/cql3/QueryOptions.java | 4 ++--
 src/java/org/apache/cassandra/cql3/WhereClause.java  | 4 ++--
 .../org/apache/cassandra/cql3/conditions/ColumnCondition.java| 4 ++--
 .../org/apache/cassandra/cql3/conditions/ColumnConditions.java   | 4 ++--
 .../cassandra/cql3/restrictions/CustomIndexExpression.java   | 4 ++--
 .../apache/cassandra/cql3/restrictions/IndexRestrictions.java| 4 ++--
 .../cassandra/cql3/restrictions/MultiColumnRestriction.java  | 4 ++--
 .../org/apache/cassandra/cql3/restrictions/RestrictionSet.java   | 4 ++--
 .../cassandra/cql3/restrictions/RestrictionSetWrapper.java   | 4 ++--
 .../cassandra/cql3/restrictions/StatementRestrictions.java   | 4 ++--
 .../apache/cassandra/cql3/statements/AlterKeyspaceStatement.java | 4 ++--
 .../org/apache/cassandra/cql3/statements/AlterRoleStatement.java | 4 ++--
 .../cassandra/cql3/statements/AlterTableStatementColumn.java | 4 ++--
 .../org/apache/cassandra/cql3/statements/AlterTypeStatement.java | 4 ++--
 .../apache/cassandra/cql3/statements/AuthorizationStatement.java | 4 ++--
 src/java/org/apache/cassandra/cql3/statements/CFProperties.java  | 4 ++--
 src/java/org/apache/cassandra/cql3/statements/CFStatement.java   | 4 ++--
 .../org/apache/cassandra/cql3/statements/CQL3CasRequest.java | 4 ++--
 .../cassandra/cql3/statements/CreateAggregateStatement.java  | 4 ++--
 .../cassandra/cql3/statements/CreateFunctionStatement.java   | 4 ++--
 .../apache/cassandra/cql3/statements/CreateIndexStatement.java   | 4 ++--
 .../cassandra/cql3/statements/CreateKeyspaceStatement.java   | 4 ++--
 .../apache/cassandra/cql3/statements/CreateRoleStatement.java| 4 ++--
 .../apache/cassandra/cql3/statements/CreateTableStatement.java   | 4 ++--
 .../apache/cassandra/cql3/statements/CreateTriggerStatement.java | 4 ++--
 .../apache/cassandra/cql3/statements/CreateTypeStatement.java| 4 ++--
 .../apache/cassandra/cql3/statements/CreateViewStatement.java| 4 ++--
 .../org/apache/cassandra/cql3/statements/DeleteStatement.java| 4 ++--
 .../apache/cassandra/cql3/statements/DropAggregateStatement.java | 4 ++--
 .../apache/cassandra/cql3/statements/DropFunctionStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/DropIndexStatement.java | 4 ++--
 .../apache/cassandra/cql3/statements/DropKeyspaceStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/DropRoleStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/DropTableStatement.java | 4 ++--
 .../apache/cassandra/cql3/statements/DropTriggerStatement.java   | 4 ++--
 .../org/apache/cassandra/cql3/statements/DropTypeStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/DropViewStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/GrantRoleStatement.java | 4 ++--
 src/java/org/apache/cassandra/cql3/statements/IndexTarget.java   | 4 ++--
 .../cassandra/cql3/statements/ListPermissionsStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/ListRolesStatement.java | 4 ++--
 .../org/apache/cassandra/cql3/statements/ListUsersStatement.java | 4 ++--
 .../cql3/statements/PermissionsManagementStatement.java  | 4 ++--
 .../cassandra/cql3/statements/RevokePermissionsStatement.java| 4 ++--
 .../apache/cassandra/cql3/statements/RevokeRoleStatement.java| 4 ++--
 .../cassandra/cql3/statements/RoleManagementStatement.java   | 4 ++--
 .../org/apache/cassandra/cql3/statements/SelectStatement.java| 4 ++--
 .../org/apache/cassandra/cql3/statements/TruncateStatement.java  | 4 ++--
 .../org/apache/cassandra/cql3/statements/UpdateStatement.java| 4 ++--
 src/java/org/apache/cassandra/cql3/statements/UseStatement.java  | 4 ++--
 52 files changed, 104 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/23787b5f/src/java/org/apache/cassandra/cql3/Attributes.java
--
diff --git 

[jira] [Assigned] (CASSANDRA-14358) OutboundTcpConnection can hang for many minutes when nodes restart

[Vinay Chella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay Chella reassigned CASSANDRA-14358:


Assignee: Joseph Lynch

> OutboundTcpConnection can hang for many minutes when nodes restart
> --
>
> Key: CASSANDRA-14358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14358
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
> Environment: Cassandra 2.1.19 (also reproduced on 3.0.15), running 
> with {{internode_encryption: all}} and the EC2 multi region snitch on Linux 
> 4.13 within the same AWS region. Smallest cluster I've seen the problem on is 
> 12 nodes, reproduces more reliably on 40+ and 300 node clusters consistently 
> reproduce on at least one node in the cluster.
> So all the connections are SSL and we're connecting on the internal ip 
> addresses (not the public endpoint ones).
> Potentially relevant sysctls:
> {noformat}
> /proc/sys/net/ipv4/tcp_syn_retries = 2
> /proc/sys/net/ipv4/tcp_synack_retries = 5
> /proc/sys/net/ipv4/tcp_keepalive_time = 7200
> /proc/sys/net/ipv4/tcp_keepalive_probes = 9
> /proc/sys/net/ipv4/tcp_keepalive_intvl = 75
> /proc/sys/net/ipv4/tcp_retries2 = 15
> {noformat}
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Major
> Attachments: 10 Minute Partition.pdf
>
>
> I've been trying to debug nodes not being able to see each other during 
> longer (~5 minute+) Cassandra restarts in 3.0.x and 2.1.x which can 
> contribute to {{UnavailableExceptions}} during rolling restarts of 3.0.x and 
> 2.1.x clusters for us. I think I finally have a lead. It appears that prior 
> to trunk (with the awesome Netty refactor) we do not set socket connect 
> timeouts on SSL connections (in 2.1.x, 3.0.x, or 3.11.x) nor do we set 
> {{SO_TIMEOUT}} as far as I can tell on outbound connections either. I believe 
> that this means that we could potentially block forever on {{connect}} or 
> {{recv}} syscalls, and we could block forever on the SSL Handshake as well. I 
> think that the OS will protect us somewhat (and that may be what's causing 
> the eventual timeout) but I think that given the right network conditions our 
> {{OutboundTCPConnection}} threads can just be stuck never making any progress 
> until the OS intervenes.
> I have attached some logs of such a network partition during a rolling 
> restart where an old node in the cluster has a completely foobarred 
> {{OutboundTcpConnection}} for ~10 minutes before finally getting a 
> {{java.net.SocketException: Connection timed out (Write failed)}} and 
> immediately successfully reconnecting. I conclude that the old node is the 
> problem because the new node (the one that restarted) is sending ECHOs to the 
> old node, and the old node is sending ECHOs and REQUEST_RESPONSES to the new 
> node's ECHOs, but the new node is never getting the ECHO's. This appears, to 
> me, to indicate that the old node's {{OutboundTcpConnection}} thread is just 
> stuck and can't make any forward progress. By the time we could notice this 
> and slap TRACE logging on, the only thing we see is ~10 minutes later a 
> {{SocketException}} inside {{writeConnected}}'s flush and an immediate 
> recovery. It is interesting to me that the exception happens in 
> {{writeConnected}} and it's a _connection timeout_ (and since we see {{Write 
> failure}} I believe that this can't be a connection reset), because my 
> understanding is that we should have a fully handshaked SSL connection at 
> that point in the code.
> Current theory:
>  # "New" node restarts,  "Old" node calls 
> [newSocket|https://github.com/apache/cassandra/blob/6f30677b28dcbf82bcd0a291f3294ddf87dafaac/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L433]
>  # Old node starts [creating a 
> new|https://github.com/apache/cassandra/blob/6f30677b28dcbf82bcd0a291f3294ddf87dafaac/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java#L141]
>  SSL socket 
>  # SSLSocket calls 
> [createSocket|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/security/SSLFactory.java#L98],
>  which conveniently calls connect with a default timeout of "forever". We 
> could hang here forever until the OS kills us.
>  # If we continue, we get to 
> [writeConnected|https://github.com/apache/cassandra/blob/6f30677b28dcbf82bcd0a291f3294ddf87dafaac/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L263]
>  which eventually calls 
> [flush|https://github.com/apache/cassandra/blob/6f30677b28dcbf82bcd0a291f3294ddf87dafaac/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L341]
>  on the output stream and also can hang forever. I think the probability is 
> especially high when a node is restarting and is 

[jira] [Updated] (CASSANDRA-14297) Optional startup delay for peers should wait for count rather than percentage

[Joseph Lynch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Lynch updated CASSANDRA-14297:
-
Issue Type: Improvement  (was: Bug)

> Optional startup delay for peers should wait for count rather than percentage
> -
>
> Key: CASSANDRA-14297
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14297
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
>  Labels: PatchAvailable
>
> As I commented in CASSANDRA-13993, the current wait for functionality is a 
> great step in the right direction, but I don't think that the current setting 
> (70% of nodes in the cluster) is the right configuration option. First I 
> think this because 70% will not protect against errors as if you wait for 70% 
> of the cluster you could still very easily have {{UnavailableException}} or 
> {{ReadTimeoutException}} exceptions. This is because if you have even two 
> nodes down in different racks in a Cassandra cluster these exceptions are 
> possible (or with the default {{num_tokens}} setting of 256 it is basically 
> guaranteed). Second I think this option is not easy for operators to set, the 
> only setting I could think of that would "just work" is 100%.
> I proposed in that ticket instead of having `block_for_peers_percentage` 
> defaulting to 70%, we instead have `block_for_peers` as a count of nodes that 
> are allowed to be down before the starting node makes itself available as a 
> coordinator. Of course, we would still have the timeout to limit startup time 
> and deal with really extreme situations (whole datacenters down etc).
> I started working on a patch for this change [on 
> github|https://github.com/jasobrown/cassandra/compare/13993...jolynch:13993], 
> and am happy to finish it up with unit tests and such if someone can 
> review/commit it (maybe [~aweisberg]?).
> I think the short version of my proposal is we replace:
> {noformat}
> block_for_peers_percentage: 
> {noformat}
> with either
> {noformat}
> block_for_peers: 
> {noformat}
> or, if we want to do even better imo and enable advanced operators to finely 
> tune this behavior (while still having good defaults that work for almost 
> everyone):
> {noformat}
> block_for_peers_local_dc:  
> block_for_peers_each_dc: 
> block_for_peers_all_dcs: 
> {noformat}
> For example if an operator knows that they must be available at 
> {{LOCAL_QUORUM}} they would set {{block_for_peers_local_dc=1}}, if they use 
> {{EACH_QUOURM}} they would set {{block_for_peers_local_dc=1}}, if they use 
> {{QUORUM}} (RF=3, dcs=2) they would set {{block_for_peers_all_dcs=2}}. 
> Naturally everything would of course have a timeout to prevent startup taking 
> too long.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-10023) Emit a metric for number of local read and write calls

[Damien Stevenson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427736#comment-16427736
 ] 

Damien Stevenson commented on CASSANDRA-10023:
--

Looks like this ticket has gone quiet. I volunteer to work on it if no one has 
any objections.

> Emit a metric for number of local read and write calls
> --
>
> Key: CASSANDRA-10023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10023
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Tushar Agrawal
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: CASSANDRA-10023.patch
>
>
> Many C* drivers have feature to be replica aware and chose the co-ordinator 
> which is a replica. We should add a metric which tells us whether all calls 
> to the co-ordinator are replica aware.
> We have seen issues where client thinks they are replica aware when they 
> forget to add routing key at various places in the code. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13889) cfstats should take sorting and limit parameters

[Jon Haddad (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Haddad updated CASSANDRA-13889:
---
Reviewer: Jon Haddad

> cfstats should take sorting and limit parameters
> 
>
> Key: CASSANDRA-13889
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13889
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jon Haddad
>Assignee: Patrick Bannister
>Priority: Major
> Fix For: 4.0
>
> Attachments: 13889-trunk.txt, sample_output_normal.txt, 
> sample_output_sorted.txt, sample_output_sorted_top3.txt
>
>
> When looking at a problematic node I'm not familiar with, one of the first 
> things I do is check cfstats to identify the tables with the most reads, 
> writes, and data.  This is fine as long as there aren't a lot of tables but 
> once it goes above a dozen it's quite difficult.  cfstats should allow me to 
> sort the results and limit to top K tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14023) add_dc_after_mv_network_replication_test - materialized_views_test.TestMaterializedViews fails due to invalid datacenter

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14023:

Status: Ready to Commit  (was: Patch Available)

> add_dc_after_mv_network_replication_test - 
> materialized_views_test.TestMaterializedViews fails due to invalid datacenter
> 
>
> Key: CASSANDRA-14023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14023
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Marcus Eriksson
>Priority: Major
>
> add_dc_after_mv_network_replication_test - 
> materialized_views_test.TestMaterializedViews always fails due to:
>  message="Unrecognized strategy option {dc2} passed to NetworkTopologyStrategy 
> for keyspace ks">



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14023) add_dc_after_mv_network_replication_test - materialized_views_test.TestMaterializedViews fails due to invalid datacenter

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427673#comment-16427673
 ] 

Paulo Motta commented on CASSANDRA-14023:
-

Sorry for the delay. LGTM, thanks!

> add_dc_after_mv_network_replication_test - 
> materialized_views_test.TestMaterializedViews fails due to invalid datacenter
> 
>
> Key: CASSANDRA-14023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14023
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Marcus Eriksson
>Priority: Major
>
> add_dc_after_mv_network_replication_test - 
> materialized_views_test.TestMaterializedViews always fails due to:
>  message="Unrecognized strategy option {dc2} passed to NetworkTopologyStrategy 
> for keyspace ks">



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14366) Add prepared statement cache stats to nodetool info

[Jon Haddad (JIRA)]

Jon Haddad created CASSANDRA-14366:
--

 Summary: Add prepared statement cache stats to nodetool info
 Key: CASSANDRA-14366
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14366
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jon Haddad
 Fix For: 4.x


We expose some cache stats in {{nodetool info}}, let's add prepared statement 
cache stats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13665) nodetool clientlist

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-13665:

Fix Version/s: 4.0

> nodetool clientlist
> ---
>
> Key: CASSANDRA-13665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13665
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jon Haddad
>Assignee: Chris Lohfink
>Priority: Major
> Fix For: 4.0
>
>
> There should exist a nodetool command that lists each client connection. 
> Ideally it would display the following:
>  * host
>  * protocol version
>  * user logged in as
>  * current keyspace
>  * total queries executed
>  * ssl connections



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427583#comment-16427583
 ] 

Aleksey Yeschenko commented on CASSANDRA-13993:
---

The out-of-range problem, however, feels a bit silly. We shouldn't have padding 
just to avoid going out of ordinal bounds - we should handle ordinals that are 
outside of our known range robustly instead.

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to successfully communicate 
> with a peer after a bounce:
> - marking the peer as 'alive' (state that is held in gossip). This affects 
> the unavailable exceptions
> - having both open outbound and inbound connections open and ready to each 
> peer. This affects timeouts.
> Details of each of these mechanisms are described in the comments below.
> This ticket proposes adding a mechanism, optional and configurable, to delay 
> opening the client native protocol port until some percentage of the peers in 
> the cluster is marked alive and connected to/from. Thus while we potentially 
> slow down startup (delay opening the client port), we alleviate the chance 
> that queries made by clients don't hit transient unavailable/timeout 
> exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14176) Cannot export & import data containing no-break space (U+00A0)

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14176:

Reviewer: Paulo Motta

Good catch, would you mind adding a regression test to the 
[cqlsh_copy_tests|https://github.com/apache/cassandra-dtest/blob/master/cqlsh_tests/cqlsh_copy_tests.py]
 suite reproducing the problem?

> Cannot export & import data containing no-break space (U+00A0)
> --
>
> Key: CASSANDRA-14176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14176
> Project: Cassandra
>  Issue Type: Bug
>  Components: Libraries
>Reporter: Marcel Dopita
>Assignee: Marcel Dopita
>Priority: Major
> Attachments: fix.patch
>
>
> We were unable to export and then import the same data to Cassandra - like 
> line breaks or the no-break space (U+00A0).
> Adding v.decode() to copyutil.py fixed most characters like line breaks etc.
> Only after using the included patch, the character U+00A0 was correctly 
> stored in Cassandra and successfully (verifiable) exported & imported & 
> exported.
>  
> {code:java}
> diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
> index 7f97b49..883c957 100644
> --- a/pylib/cqlshlib/copyutil.py
> +++ b/pylib/cqlshlib/copyutil.py
> @@ -1871,7 +1871,7 @@ class ImportConversion(object):
>  return bytearray.fromhex(v[2:])
>  
>  def convert_text(v, **_):
> -    return v
> +    return v.decode('string_escape')
>  
>  def convert_uuid(v, **_):
>  return UUID(v)
> diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
> index 803ea63..79eb691 100644
> --- a/pylib/cqlshlib/formatting.py
> +++ b/pylib/cqlshlib/formatting.py
> @@ -33,7 +33,7 @@ from util import UTC
>  
>  is_win = platform.system() == 'Windows'
>  
> -unicode_controlchars_re = re.compile(r'[\x00-\x31\x7f-\xa0]')
> +unicode_controlchars_re = re.compile(r'[\x00-\x31]')
>  controlchars_re = re.compile(r'[\x00-\x31\x7f-\xff]')
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14176) Cannot export & import data containing no-break space (U+00A0)

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14176:

Status: Open  (was: Patch Available)

> Cannot export & import data containing no-break space (U+00A0)
> --
>
> Key: CASSANDRA-14176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14176
> Project: Cassandra
>  Issue Type: Bug
>  Components: Libraries
>Reporter: Marcel Dopita
>Assignee: Marcel Dopita
>Priority: Major
> Attachments: fix.patch
>
>
> We were unable to export and then import the same data to Cassandra - like 
> line breaks or the no-break space (U+00A0).
> Adding v.decode() to copyutil.py fixed most characters like line breaks etc.
> Only after using the included patch, the character U+00A0 was correctly 
> stored in Cassandra and successfully (verifiable) exported & imported & 
> exported.
>  
> {code:java}
> diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
> index 7f97b49..883c957 100644
> --- a/pylib/cqlshlib/copyutil.py
> +++ b/pylib/cqlshlib/copyutil.py
> @@ -1871,7 +1871,7 @@ class ImportConversion(object):
>  return bytearray.fromhex(v[2:])
>  
>  def convert_text(v, **_):
> -    return v
> +    return v.decode('string_escape')
>  
>  def convert_uuid(v, **_):
>  return UUID(v)
> diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
> index 803ea63..79eb691 100644
> --- a/pylib/cqlshlib/formatting.py
> +++ b/pylib/cqlshlib/formatting.py
> @@ -33,7 +33,7 @@ from util import UTC
>  
>  is_win = platform.system() == 'Windows'
>  
> -unicode_controlchars_re = re.compile(r'[\x00-\x31\x7f-\xa0]')
> +unicode_controlchars_re = re.compile(r'[\x00-\x31]')
>  controlchars_re = re.compile(r'[\x00-\x31\x7f-\xff]')
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427566#comment-16427566
 ] 

Aleksey Yeschenko commented on CASSANDRA-13993:
---

Disregard my last comment here, I was wrong, by a big margin. My apologies.

As implemented currently, we are going to send PINGs potentially to 3.11/3.0 - 
unless we switch to gating by version, which we do sometimes. And if you pick a 
verb after {{UNUSED_5}}, it would error out on 3.11/3.0 side. So, again, unless 
we gate by version (on which - see below*), we need to pick an ordinal that is 
within the range of 3.0/3.11 - so one of {{UNUSED_1..5}} verbs.

The latest still supported release is 2.2, which has only 3 {{UNUSED}} verbs. 
To be super paranoid and maxmimise the # of available {{UNUSED}} verbs in case 
of bad things happening that would force us to introduce new verbs in old 
versions - which is very unlikely to happen, but did happen before, we should 
use one of {{UNUSED_4}} or {{UNUSED_5}} verbs here, in my opinion.

But not inserting a verb before {{UNUSED_1}} like it is now - it's essentially 
taking up {{UNUSED_1}} verb, but confusing things between 4.0 and 3.0/3.11, 
where everything would slide by one and might introduce mistakes.

* So I was thinking about a major upgrade bounce scenario. Think the first ever 
node to upgrade to 4.0 in a cluster of 3.0 nodes - will send out pings to every 
node, but receive no pongs, correct? So every node until a threshold will have 
a significantly longer bounce. Do we care about this case?

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to successfully communicate 
> with a peer after a bounce:
> - marking the peer as 'alive' (state that is held in gossip). This affects 
> the unavailable exceptions
> - having both open outbound and inbound connections open and ready to each 
> peer. This affects timeouts.
> Details of each of these mechanisms are described in the comments below.
> This ticket proposes adding a mechanism, optional and configurable, to delay 
> opening the client native protocol port until some percentage of the peers in 
> the cluster is marked alive and connected to/from. Thus while we potentially 
> slow down startup (delay opening the client port), we alleviate the chance 
> that queries made by clients don't hit transient unavailable/timeout 
> exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14227) Extend maximum expiration date

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427544#comment-16427544
 ] 

Paulo Motta commented on CASSANDRA-14227:
-

It would be great to get this in for 4.0. The simplest approach would be to 
change the {{localDeletionTime}} representation to {{long}} as proposed by 
[~VincentWhite], but we need to verify the impact of this on heap for TTL and 
non-TTL workloads.

Alternatives include using a unsigned int32 to represent {{localDeletionTime}} 
and/or using a later start epoch (maybe 2000 or so) - but this would require 
some considerate work to maintain backward compatibility and would difficult 
interoperability with other systems - even though it's an internal structure.

[~VincentWhite] would you be willing to work on this and maybe perform some 
stress tests to check impact on heap and GC of changing the 
{{localDeletionTime}} representation from {{int}} to {{long}} ?

> Extend maximum expiration date
> --
>
> Key: CASSANDRA-14227
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14227
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Paulo Motta
>Priority: Major
>
> The maximum expiration timestamp that can be represented by the storage 
> engine is
> 2038-01-19T03:14:06+00:00 due to the encoding of {{localExpirationTime}} as 
> an int32.
> On CASSANDRA-14092 we added an overflow policy which rejects requests with 
> expiration above the maximum date as a temporary measure, but we should 
> remove this limitation by updating the storage engine to support at least the 
> maximum allowed TTL of 20 years.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14358) OutboundTcpConnection can hang for many minutes when nodes restart

[Joseph Lynch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427531#comment-16427531
 ] 

Joseph Lynch commented on CASSANDRA-14358:
--

A quick update, AWS has confirmed that this type of half open partition is very 
much possible when using VPC/security groups as they do their own [connection 
tracking|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking].
 If the sg doesn't know about the flow because the "timeout" (which is very 
non-specific as to how long it is) occurs after close, the sg will blackhole 
packets. We're following up to see if they could have the more useful behavior 
of sending resets instead of dropping, but I imagine they'll answer that it's 
for security reasons.

I'm going to spend time seeing if the proposed workaround of setting 
{{/proc/sys/net/ipv4/tcp_retries2 = 5 }} fixes the issue satisfactorily, and if 
so I think that's a reasonable workaround for pre-netty, and post-netty we can 
use the reproduction steps above to see if Cassandra is now resilient to this 
kind of half-open partition. [~aweisberg] [~jasobrown] [~djoshi3] what do you 
guys think?

> OutboundTcpConnection can hang for many minutes when nodes restart
> --
>
> Key: CASSANDRA-14358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14358
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
> Environment: Cassandra 2.1.19 (also reproduced on 3.0.15), running 
> with {{internode_encryption: all}} and the EC2 multi region snitch on Linux 
> 4.13 within the same AWS region. Smallest cluster I've seen the problem on is 
> 12 nodes, reproduces more reliably on 40+ and 300 node clusters consistently 
> reproduce on at least one node in the cluster.
> So all the connections are SSL and we're connecting on the internal ip 
> addresses (not the public endpoint ones).
> Potentially relevant sysctls:
> {noformat}
> /proc/sys/net/ipv4/tcp_syn_retries = 2
> /proc/sys/net/ipv4/tcp_synack_retries = 5
> /proc/sys/net/ipv4/tcp_keepalive_time = 7200
> /proc/sys/net/ipv4/tcp_keepalive_probes = 9
> /proc/sys/net/ipv4/tcp_keepalive_intvl = 75
> /proc/sys/net/ipv4/tcp_retries2 = 15
> {noformat}
>Reporter: Joseph Lynch
>Priority: Major
> Attachments: 10 Minute Partition.pdf
>
>
> I've been trying to debug nodes not being able to see each other during 
> longer (~5 minute+) Cassandra restarts in 3.0.x and 2.1.x which can 
> contribute to {{UnavailableExceptions}} during rolling restarts of 3.0.x and 
> 2.1.x clusters for us. I think I finally have a lead. It appears that prior 
> to trunk (with the awesome Netty refactor) we do not set socket connect 
> timeouts on SSL connections (in 2.1.x, 3.0.x, or 3.11.x) nor do we set 
> {{SO_TIMEOUT}} as far as I can tell on outbound connections either. I believe 
> that this means that we could potentially block forever on {{connect}} or 
> {{recv}} syscalls, and we could block forever on the SSL Handshake as well. I 
> think that the OS will protect us somewhat (and that may be what's causing 
> the eventual timeout) but I think that given the right network conditions our 
> {{OutboundTCPConnection}} threads can just be stuck never making any progress 
> until the OS intervenes.
> I have attached some logs of such a network partition during a rolling 
> restart where an old node in the cluster has a completely foobarred 
> {{OutboundTcpConnection}} for ~10 minutes before finally getting a 
> {{java.net.SocketException: Connection timed out (Write failed)}} and 
> immediately successfully reconnecting. I conclude that the old node is the 
> problem because the new node (the one that restarted) is sending ECHOs to the 
> old node, and the old node is sending ECHOs and REQUEST_RESPONSES to the new 
> node's ECHOs, but the new node is never getting the ECHO's. This appears, to 
> me, to indicate that the old node's {{OutboundTcpConnection}} thread is just 
> stuck and can't make any forward progress. By the time we could notice this 
> and slap TRACE logging on, the only thing we see is ~10 minutes later a 
> {{SocketException}} inside {{writeConnected}}'s flush and an immediate 
> recovery. It is interesting to me that the exception happens in 
> {{writeConnected}} and it's a _connection timeout_ (and since we see {{Write 
> failure}} I believe that this can't be a connection reset), because my 
> understanding is that we should have a fully handshaked SSL connection at 
> that point in the code.
> Current theory:
>  # "New" node restarts,  "Old" node calls 
> [newSocket|https://github.com/apache/cassandra/blob/6f30677b28dcbf82bcd0a291f3294ddf87dafaac/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L433]
>  # Old node 

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Vinay Chella (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427318#comment-16427318
 ] 

Vinay Chella commented on CASSANDRA-12151:
--

Thank you [~spo...@gmail.com]

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Joseph Lynch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427312#comment-16427312
 ] 

Joseph Lynch commented on CASSANDRA-14346:
--

[~spo...@gmail.com]
 Thanks for the feedback, let me try to address your concerns. If you have time 
can you comment specifically in the design doc so that I make sure we address 
the outstanding concerns (keeping track of points in a jira is very hard, 
keeping track in gdoc comments is easier for me)?
{quote}There are several aspect to consider for coming up with a working repair 
schedule:

number of tables (from a single table per cluster to hundreds of tables)
{quote}
I don't think this is an issue with the design. We currently use this design to 
repair hundreds of clusters that vary between a few large tables and thousands 
of various size tables. Our distributed design makes continuous progress and 
gets the job done. We also provide a path forward in the document for highly 
[concurrent 
repair|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#bookmark=id.xn6852786lv8]
 which further helps.
{quote}priority in repairing tables (some tables should be repaired more often, 
others never at all)
 data size per table (large table should not block repairs for smaller more 
important ones)
{quote}
I think cluster sharding is the better way to fix this (and I believe in trunk 
you can run multiple Cassandra clusters on the same machine now because of the 
port refactor). You want to isolate critical workloads from non critical 
workloads for lots of reasons aside from repair. I don't see any reason why 
multiple schedules with table filters couldn't achieve this, but I question if 
that's the right level of abstraction to solve it at (i.e. I think cluster 
sharding is a much better solution). Do you have any proposals for how to 
achieve this kind of coordination without a central coordinator? I'll think on 
it but if you think it's important I encourage you to contribute to the design.
{quote}predictable cluster load (try to schedule repairs off hours)

sustainable repair intensity (repair sessions should not leak into peak hours)
{quote}
I address this in the 
[design|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#bookmark=id.mykcdt32qw7i],
 and give a path forward. For what it's worth I think we disagree generally 
with doing something less frequently because it hurts; do it more so that you 
actually fix it (for example when we started running repair continuously we 
realized how important appropriately auto-sizing subrange are to preventing 
impact on the cluster, and now that's fixed and we run repair continuously 
without any impact to the cluster).
{quote}different gc_grace periods (plan intervals for each table so we can 
tolerate missing a repair run)
{quote}
I also address this in the 
[design|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#bookmark=id.j1wvnhf0scav].
 Once repair scheduling is part of Cassandra the 
{{only_purge_repaired_tombstones}} option becomes much more attractive imo.
{quote}Repair schedules, which will take these aspects into account, require a 
certain flexibility and some more careful configuration. Tools, such as reaper, 
allow you to put together such plans already. Looking at the configuration 
options described in the design document, I'd probably still want to use such 
an external tool. That would be mostly due to the use of delays instead of 
recurring repair times and the way you'd have to configure repairs on table 
level, which probably gets a bit "messy" fast when you have a lot of tables. 
The lack of any reporting doesn't help either to further tune these config 
options afterwards.
{quote}
We pretty strongly disagree that advanced scheduling is actually required. 
[Adaptive|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#bookmark=id.x9qx96jfyivi]
 subrange as proposed in the design and eventually making repair much cheaper 
(via incremental and continuous+incremental, and making it FADV_DONTNEED so you 
don't blow the OS cache) are in our opinions better places to put the 
complexity than in the scheduler (since schedulers are comparatively harder).

Regarding the table by table config, as stated [in the 
document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit#bookmark=id.fhyfqyylq2p3]
 an explicit goal is to have almost no configuration. As a production datapoint 
I think we have thousands of tables and I think we set overrides for maybe a 
few dozen of them.

We've tried to produce a minimally viable integration into Cassandra so that 
the 90% use cases (and even the 10% huge scale users such as us at Netflix) can 
have eventual consistency.
{quote}I think the intention is to keep the scope of this ticket to "integrated 

[jira] [Commented] (CASSANDRA-13426) Make all DDL statements idempotent and not dependent on global state

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427253#comment-16427253
 ] 

Aleksey Yeschenko commented on CASSANDRA-13426:
---

3.5 months.

Need to address review feedback and rebase on top of most recent trunk, and it 
can be shipped.

> Make all DDL statements idempotent and not dependent on global state
> 
>
> Key: CASSANDRA-13426
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13426
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Distributed Metadata
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 4.0
>
>
> A follow-up to CASSANDRA-9425 and a pre-requisite for CASSANDRA-10699.
> It's necessary for the latter to be able to apply any DDL statement several 
> times without side-effects. As part of the ticket I think we should also 
> clean up validation logic and our error texts. One example is varying 
> treatment of missing keyspace for DROP TABLE/INDEX/etc. statements with IF 
> EXISTS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13426) Make all DDL statements idempotent and not dependent on global state

[Nikolay Mankov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427251#comment-16427251
 ] 

Nikolay Mankov commented on CASSANDRA-13426:


Hello everyone. Could you please give quick update about the status of this 
issue. Last one was 5 months ago. 

This issue is the only requirement for CASSANDRA-10699 to be finished and from 
the comments above it looks like there are no blockers to finalize both.

Thank you 

> Make all DDL statements idempotent and not dependent on global state
> 
>
> Key: CASSANDRA-13426
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13426
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Distributed Metadata
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 4.0
>
>
> A follow-up to CASSANDRA-9425 and a pre-requisite for CASSANDRA-10699.
> It's necessary for the latter to be able to apply any DDL statement several 
> times without side-effects. As part of the ticket I think we should also 
> clean up validation logic and our error texts. One example is varying 
> treatment of missing keyspace for DROP TABLE/INDEX/etc. statements with IF 
> EXISTS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427243#comment-16427243
 ] 

Blake Eggleston commented on CASSANDRA-14346:
-

bq. I'll spare you any of my thoughts about how we should coordinate and 
execute repairs differently in a post CASSANDRA-9143 world

I'd really like to hear about this if you want to open another jira or dev list 
thread when you have the time.

bq. From my impression, 3rd party projects are probably also a better place to 
quickly iterate on these kind of problems.

Because it would be tied to the release cycle of C*, or some other reason? What 
about a sub project that could move independently? Sort of like dtests. 
Something that could release independently of the database and work with 
multiple C* versions?

> Scheduled Repair in Cassandra
> -
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Repair
>Reporter: Joseph Lynch
>Priority: Major
>  Labels: CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes 
> sense given that it is necessary to give our users eventual consistency. Most 
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked 
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), 
> which we spoke about last year at NGCC. Given the positive feedback at NGCC 
> we focussed on getting it production ready and have now been using it in 
> production to repair hundreds of clusters, tens of thousands of nodes, and 
> petabytes of data for the past six months. Also based on feedback at NGCC we 
> have invested effort in figuring out how to integrate this natively into 
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our 
> implementation into Cassandra, and have created a [design 
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
>  showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would 
> be greatly appreciated about the interface or v1 implementation features. I 
> have tried to call out in the document features which we explicitly consider 
> future work (as well as a path forward to implement them in the future) 
> because I would very much like to get this done before the 4.0 merge window 
> closes, and to do that I think aggressively pruning scope is going to be a 
> necessity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14286) IndexOutOfBoundsException with SELECT JSON using IN and ORDER BY

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427144#comment-16427144
 ] 

Benjamin Lerer commented on CASSANDRA-14286:


I just noticed while merging a problem with the trunk patch. In trunk 
{{Selection}} is immutable but the patch is breaking that contract.

Sorry, for noticing that problem only now.

> IndexOutOfBoundsException with SELECT JSON using IN and ORDER BY
> 
>
> Key: CASSANDRA-14286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14286
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: Kubernetes cluster using cassandra:3.11.1 Docker image.
>Reporter: Szymon Acedański
>Assignee: Francisco Fernandez
>Priority: Major
> Attachments: orderbug-traceback.txt
>
>
> When running the following code:
> {code}
> public class CassandraJsonOrderingBug {
> public static void main(String[] args) {
> Session session = CassandraFactory.getSession();
> session.execute("CREATE TABLE thebug ( PRIMARY KEY (a, b), a INT, b 
> INT)");
> try {
> session.execute("INSERT INTO thebug (a, b) VALUES (20, 30)");
> session.execute("INSERT INTO thebug (a, b) VALUES (100, 200)");
> Statement statement = new SimpleStatement("SELECT JSON a, b FROM 
> thebug WHERE a IN (20, 100) ORDER BY b");
> statement.setFetchSize(Integer.MAX_VALUE);
> for (Row w: session.execute(statement)) {
> System.out.println(w.toString());
> }
> } finally {
> session.execute("DROP TABLE thebug");
> }
> }
> }
> {code}
> The following exception is thrown server-side:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>   at java.util.Collections$SingletonList.get(Collections.java:4815) 
> ~[na:1.8.0_151]
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement$SingleColumnComparator.compare(SelectStatement.java:1297)
>  ~[apache-cassandra-3.11.1.jar:3.11.1]
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement$SingleColumnComparator.compare(SelectStatement.java:1284)
>  ~[apache-cassandra-3.11.1.jar:3.11.1]
>   at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) 
> ~[na:1.8.0_151]
>   at java.util.TimSort.sort(TimSort.java:220) ~[na:1.8.0_151]
>   at java.util.Arrays.sort(Arrays.java:1512) ~[na:1.8.0_151]
>   at java.util.ArrayList.sort(ArrayList.java:1460) ~[na:1.8.0_151]
>   at java.util.Collections.sort(Collections.java:175) ~[na:1.8.0_151]
> {noformat}
> (full traceback attached)
> The accessed index is the index of the sorted column in the SELECT JSON 
> fields list.
> Similarly, if the select clause is changed to
> SELECT JSON b, a FROM thebug WHERE a IN (20, 100) ORDER BY b
> then the query finishes, but the output is sorted incorrectly (by textual 
> JSON representation):
> {noformat}
> Row[{"b": 200, "a": 100}]
> Row[{"b": 30, "a": 20}]
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Burman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Burman updated CASSANDRA-14281:
---
Status: Patch Available  (was: In Progress)

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13698) Reinstate or get rid of unit tests with multiple compaction strategies

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427050#comment-16427050
 ] 

Paulo Motta commented on CASSANDRA-13698:
-

Sorry for the delay here, the ticket was not marked as patch available so it 
felt through the cracks.

The changes look great, thanks! Can you rebase the trunk patch and provide 
3.0/3.11 patches?

> Reinstate or get rid of unit tests with multiple compaction strategies
> --
>
> Key: CASSANDRA-13698
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13698
> Project: Cassandra
>  Issue Type: Test
>  Components: Testing
>Reporter: Paulo Motta
>Assignee: Lerh Chuan Low
>Priority: Minor
>  Labels: lhf
>
> At some point there were (anti-)compaction tests with multiple compaction 
> strategy classes, but now it's only tested with {{STCS}}:
> * 
> [AnticompactionTest|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/test/unit/org/apache/cassandra/db/compaction/AntiCompactionTest.java#L247]
> * 
> [CompactionsTest|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/test/unit/org/apache/cassandra/db/compaction/CompactionsTest.java#L85]
> We should either reinstate these tests or decide they are not important and 
> remove the unused parameter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426980#comment-16426980
 ] 

ASF GitHub Bot commented on CASSANDRA-14281:


GitHub user burmanm opened a pull request:

https://github.com/apache/cassandra/pull/217

CASSANDRA-14281

Remove locking from DEHR and use CAS update operation in rescaling to 
improve performance

Add method to combine two Snapshots

Use LongAdder[] instead of AtomicLongArray to reduce contention

Read metrics from children LatencyMetrics instances when needed, instead of 
writing to parent instances each time.

Add benchmark for LatencyMetrics and Reservoir

When child is released, it replicates previous status to parents

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/burmanm/cassandra latency_metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cassandra/pull/217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #217


commit 98ad2320e6c28bcfd7afc6a7f2bf25b640bc957a
Author: Michael Burman 
Date:   2018-03-01T12:59:53Z

Remove locking from DEHR and use CAS update operation in rescaling to 
improve performance

Add method to combine two Snapshots

Use LongAdder[] instead of AtomicLongArray to reduce contention

Read metrics from children LatencyMetrics instances when needed, instead of 
writing to parent instances each time.

Add benchmark for LatencyMetrics and Reservoir

When child is released, it replicates previous status to parents




> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Burman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426982#comment-16426982
 ] 

Michael Burman commented on CASSANDRA-14281:


https://github.com/apache/cassandra/pull/217

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing

[Michael Burman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426970#comment-16426970
 ] 

Michael Burman commented on CASSANDRA-14281:


{code:java}
trunk:

op rate : 44,168 op/s [WRITE: 44,168 op/s]
partition rate : 44,168 pk/s [WRITE: 44,168 pk/s]
row rate : 44,168 row/s [WRITE: 44,168 row/s]
latency mean : 2.9 ms [WRITE: 2.9 ms]
latency median : 2.0 ms [WRITE: 2.0 ms]
latency 95th percentile : 7.0 ms [WRITE: 7.0 ms]
latency 99th percentile : 11.8 ms [WRITE: 11.8 ms]
latency 99.9th percentile : 115.8 ms [WRITE: 115.8 ms]
latency max : 191.9 ms [WRITE: 191.9 ms]

latency_writes branch:

op rate : 46,633 op/s [WRITE: 46,633 op/s]
partition rate : 46,633 pk/s [WRITE: 46,633 pk/s]
row rate : 46,633 row/s [WRITE: 46,633 row/s]
latency mean : 2.7 ms [WRITE: 2.7 ms]
latency median : 1.9 ms [WRITE: 1.9 ms]
latency 95th percentile : 6.3 ms [WRITE: 6.3 ms]
latency 99th percentile : 11.1 ms [WRITE: 11.1 ms]
latency 99.9th percentile : 112.1 ms [WRITE: 112.1 ms]
latency max : 212.5 ms [WRITE: 212.5 ms]
{code}

> Improve LatencyMetrics performance by reducing write path processing
> 
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
>
> Currently for each write/read/rangequery/CAS touching the CFS we write a 
> latency metric which takes a lot of processing time (up to 66% of the total 
> processing time if the update was empty). 
> The way latencies are recorded is to use both a dropwizard "Timer" as well as 
> "Counter". Latter is used for totalLatency and the previous is decaying 
> metric for rates and certain percentile metrics. We then replicate all of 
> these CFS writes to the KeyspaceMetrics and globalWriteLatencies. 
> Instead of doing this on the write phase we should merge the metrics when 
> they're read. This is much less common occurrence and thus we save a lot of 
> CPU time in total. This also speeds up the write path.
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each 
> update operation, which causes a contention if there are more than one thread 
> updating the histogram. This impacts scalability when using larger machines. 
> We should make it lock-free as much as possible and also avoid a single 
> CAS-update from blocking all the concurrent threads from making an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426915#comment-16426915
 ] 

Stefan Podkowinski edited comment on CASSANDRA-12151 at 4/5/18 1:25 PM:


I was just giving the BinLogger a try and it looks like the way the bin log 
files are created, should make it possible to archive them before they get 
deleted. The most simple way would probably to run a rsync job for pulling the 
latest chronicle bin logs from the nodes. The timestamp based naming of the 
files should make this trivial.

Another option would be to allow the user to configure an archival script that 
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin 
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?

I'm going to write a custom {{IAuditLogger}} implementation tomorrow and will 
get back with some more feedback.


was (Author: spo...@gmail.com):
I was just giving the BinLogger a try and it looks like the way the bin log 
files are created, should make it possible to archive them before they get 
deleted. The most simple way would probably to run a rsync job for pulling the 
latest chronicle bin logs from the nodes. The timestamp based naming of the 
files should make this trivial.

Another option would be to allow the user to configure an archival script that 
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin 
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?

I'm going to write a custom implementation tomorrow and will get back with some 
more feedback.

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13891) fromJson(null) throws java.lang.NullPointerException on Cassandra

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown resolved CASSANDRA-13891.
-
   Resolution: Fixed
Fix Version/s: 3.0.17
   2.2.13
Reproduced In: 3.11.0, 3.9  (was: 3.9, 3.11.0)

+1 to backporting to 2.2 and 3.0. committed as sha 
{{2e5e11d66d41038bee8d2f81eb013f735d233def}}.

> fromJson(null) throws java.lang.NullPointerException on Cassandra
> -
>
> Key: CASSANDRA-13891
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13891
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: Cassandra 3.11
>Reporter: Marcel Villet
>Assignee: Edward Ribeiro
>Priority: Minor
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-13891.patch
>
>
> Basically, {{fromJson}} throws a {{java.lang.NullPointerException}} when NULL 
> is passed, instead of just returning a NULL itself. Say I create a UDT and a 
> table as follows:
> {code:java}
> create type type1
> (
> id int,
> name text
> );
> create table table1
> (
> id int,
> t FROZEN,
> primary key (id)
> );{code}
> And then try and insert a row as such:
> {{insert into table1 (id, t) VALUES (1, fromJson(null));}}
> I get the error: {{java.lang.NullPointerException}}
> This works as expected: {{insert into table1 (id, t) VALUES (1, null);}}
> Programmatically, one does not always know when a UDT will be null, hence me 
> expecting {{fromJson}} to just return NULL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426915#comment-16426915
 ] 

Stefan Podkowinski commented on CASSANDRA-12151:


I was just giving the BinLogger a try and it looks like the way the bin log 
files are created, should make it possible to archive them before they get 
deleted. The most simple way would probably to run a rsync job for pulling the 
latest chronicle bin logs from the nodes. The timestamp based naming of the 
files should make this trivial.

Another option would be to allow the user to configure an archival script that 
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin 
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?

I'm going to write a custom implementation tomorrow and will get back with some 
more feedback.

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[03/10] cassandra git commit: CQL fromJson(null) throws NullPointerException

[jasobrown]

CQL fromJson(null) throws NullPointerException

patch by Edward Ribeiro; reviewed by jasobrown for CASSANDRA-13891


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e5e11d6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e5e11d6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e5e11d6

Branch: refs/heads/cassandra-3.11
Commit: 2e5e11d66d41038bee8d2f81eb013f735d233def
Parents: ac77e5e
Author: Edward Ribeiro 
Authored: Wed Apr 4 14:18:01 2018 -0300
Committer: Jason Brown 
Committed: Thu Apr 5 06:10:59 2018 -0700

--
 CHANGES.txt   | 1 +
 .../org/apache/cassandra/cql3/functions/FunctionCall.java | 2 ++
 .../apache/cassandra/cql3/validation/entities/JsonTest.java   | 7 +++
 3 files changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4828517..527975c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.13
+ * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
  * Fix query pager DEBUG log leak causing hit in paged reads throughput 
(CASSANDRA-14318)
  * Backport circleci yaml (CASSANDRA-14240)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--
diff --git a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java 
b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
index 4f53c98..323f1bb 100644
--- a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
+++ b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
@@ -98,6 +98,8 @@ public class FunctionCall extends Term.NonTerminal
 
 private static Term.Terminal makeTerminal(Function fun, ByteBuffer result, 
int version) throws InvalidRequestException
 {
+if (result == null)
+return null;
 if (!(fun.returnType() instanceof CollectionType))
 return new Constants.Value(result);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 0f29928..46cdd52 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@ -247,6 +247,13 @@ public class JsonTest extends CQLTester
 
 // handle nulls
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, asciival FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, frozenmapval) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, frozenmapval FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, udtval) VALUES (?, fromJson(?))", 0, null);
+assertRows(execute("SELECT k, udtval FROM %s WHERE k = ?", 0), row(0, 
null));
 
 //  ascii 
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
"\"ascii text\"");


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/41f3b96f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/41f3b96f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/41f3b96f

Branch: refs/heads/cassandra-3.0
Commit: 41f3b96f81e08af8a95124b9ab50bad03067aab0
Parents: 2d2b1a7 2e5e11d
Author: Jason Brown 
Authored: Thu Apr 5 06:16:13 2018 -0700
Committer: Jason Brown 
Committed: Thu Apr 5 06:20:57 2018 -0700

--
 .../org/apache/cassandra/cql3/functions/FunctionCall.java |  2 ++
 .../cassandra/cql3/validation/entities/JsonTest.java  | 10 +-
 2 files changed, 11 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/41f3b96f/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/41f3b96f/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --cc test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 94b2087,46cdd52..62c6226
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@@ -39,7 -38,7 +39,8 @@@ import java.util.*
  import java.util.concurrent.*;
  import static org.junit.Assert.fail;
  
--public class JsonTest extends CQLTester
++public class
++JsonTest extends CQLTester
  {
  @BeforeClass
  public static void setUp()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1fafcb01
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1fafcb01
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1fafcb01

Branch: refs/heads/trunk
Commit: 1fafcb0190fec34ca38ca6294dd7919b0fe5aacc
Parents: 946aaa7 ad52c95
Author: Jason Brown 
Authored: Thu Apr 5 06:21:53 2018 -0700
Committer: Jason Brown 
Committed: Thu Apr 5 06:21:53 2018 -0700

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ad52c95a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ad52c95a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ad52c95a

Branch: refs/heads/trunk
Commit: ad52c95ae2797155a88a8b0c8d16d8fb07509dd6
Parents: 28bd6c2 41f3b96
Author: Jason Brown 
Authored: Thu Apr 5 06:21:27 2018 -0700
Committer: Jason Brown 
Committed: Thu Apr 5 06:21:27 2018 -0700

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/41f3b96f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/41f3b96f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/41f3b96f

Branch: refs/heads/trunk
Commit: 41f3b96f81e08af8a95124b9ab50bad03067aab0
Parents: 2d2b1a7 2e5e11d
Author: Jason Brown 
Authored: Thu Apr 5 06:16:13 2018 -0700
Committer: Jason Brown 
Committed: Thu Apr 5 06:20:57 2018 -0700

--
 .../org/apache/cassandra/cql3/functions/FunctionCall.java |  2 ++
 .../cassandra/cql3/validation/entities/JsonTest.java  | 10 +-
 2 files changed, 11 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/41f3b96f/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/41f3b96f/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --cc test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 94b2087,46cdd52..62c6226
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@@ -39,7 -38,7 +39,8 @@@ import java.util.*
  import java.util.concurrent.*;
  import static org.junit.Assert.fail;
  
--public class JsonTest extends CQLTester
++public class
++JsonTest extends CQLTester
  {
  @BeforeClass
  public static void setUp()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ad52c95a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ad52c95a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ad52c95a

Branch: refs/heads/cassandra-3.11
Commit: ad52c95ae2797155a88a8b0c8d16d8fb07509dd6
Parents: 28bd6c2 41f3b96
Author: Jason Brown 
Authored: Thu Apr 5 06:21:27 2018 -0700
Committer: Jason Brown 
Committed: Thu Apr 5 06:21:27 2018 -0700

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[02/10] cassandra git commit: CQL fromJson(null) throws NullPointerException

[jasobrown]

CQL fromJson(null) throws NullPointerException

patch by Edward Ribeiro; reviewed by jasobrown for CASSANDRA-13891


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e5e11d6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e5e11d6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e5e11d6

Branch: refs/heads/cassandra-3.0
Commit: 2e5e11d66d41038bee8d2f81eb013f735d233def
Parents: ac77e5e
Author: Edward Ribeiro 
Authored: Wed Apr 4 14:18:01 2018 -0300
Committer: Jason Brown 
Committed: Thu Apr 5 06:10:59 2018 -0700

--
 CHANGES.txt   | 1 +
 .../org/apache/cassandra/cql3/functions/FunctionCall.java | 2 ++
 .../apache/cassandra/cql3/validation/entities/JsonTest.java   | 7 +++
 3 files changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4828517..527975c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.13
+ * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
  * Fix query pager DEBUG log leak causing hit in paged reads throughput 
(CASSANDRA-14318)
  * Backport circleci yaml (CASSANDRA-14240)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--
diff --git a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java 
b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
index 4f53c98..323f1bb 100644
--- a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
+++ b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
@@ -98,6 +98,8 @@ public class FunctionCall extends Term.NonTerminal
 
 private static Term.Terminal makeTerminal(Function fun, ByteBuffer result, 
int version) throws InvalidRequestException
 {
+if (result == null)
+return null;
 if (!(fun.returnType() instanceof CollectionType))
 return new Constants.Value(result);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 0f29928..46cdd52 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@ -247,6 +247,13 @@ public class JsonTest extends CQLTester
 
 // handle nulls
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, asciival FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, frozenmapval) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, frozenmapval FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, udtval) VALUES (?, fromJson(?))", 0, null);
+assertRows(execute("SELECT k, udtval FROM %s WHERE k = ?", 0), row(0, 
null));
 
 //  ascii 
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
"\"ascii text\"");


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[01/10] cassandra git commit: CQL fromJson(null) throws NullPointerException

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 ac77e5e77 -> 2e5e11d66
  refs/heads/cassandra-3.0 2d2b1a71f -> 41f3b96f8
  refs/heads/cassandra-3.11 28bd6c2a0 -> ad52c95ae
  refs/heads/trunk 946aaa7b0 -> 1fafcb019


CQL fromJson(null) throws NullPointerException

patch by Edward Ribeiro; reviewed by jasobrown for CASSANDRA-13891


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e5e11d6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e5e11d6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e5e11d6

Branch: refs/heads/cassandra-2.2
Commit: 2e5e11d66d41038bee8d2f81eb013f735d233def
Parents: ac77e5e
Author: Edward Ribeiro 
Authored: Wed Apr 4 14:18:01 2018 -0300
Committer: Jason Brown 
Committed: Thu Apr 5 06:10:59 2018 -0700

--
 CHANGES.txt   | 1 +
 .../org/apache/cassandra/cql3/functions/FunctionCall.java | 2 ++
 .../apache/cassandra/cql3/validation/entities/JsonTest.java   | 7 +++
 3 files changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4828517..527975c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.13
+ * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
  * Fix query pager DEBUG log leak causing hit in paged reads throughput 
(CASSANDRA-14318)
  * Backport circleci yaml (CASSANDRA-14240)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--
diff --git a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java 
b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
index 4f53c98..323f1bb 100644
--- a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
+++ b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
@@ -98,6 +98,8 @@ public class FunctionCall extends Term.NonTerminal
 
 private static Term.Terminal makeTerminal(Function fun, ByteBuffer result, 
int version) throws InvalidRequestException
 {
+if (result == null)
+return null;
 if (!(fun.returnType() instanceof CollectionType))
 return new Constants.Value(result);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 0f29928..46cdd52 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@ -247,6 +247,13 @@ public class JsonTest extends CQLTester
 
 // handle nulls
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, asciival FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, frozenmapval) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, frozenmapval FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, udtval) VALUES (?, fromJson(?))", 0, null);
+assertRows(execute("SELECT k, udtval FROM %s WHERE k = ?", 0), row(0, 
null));
 
 //  ascii 
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
"\"ascii text\"");


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[04/10] cassandra git commit: CQL fromJson(null) throws NullPointerException

[jasobrown]

CQL fromJson(null) throws NullPointerException

patch by Edward Ribeiro; reviewed by jasobrown for CASSANDRA-13891


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e5e11d6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e5e11d6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e5e11d6

Branch: refs/heads/trunk
Commit: 2e5e11d66d41038bee8d2f81eb013f735d233def
Parents: ac77e5e
Author: Edward Ribeiro 
Authored: Wed Apr 4 14:18:01 2018 -0300
Committer: Jason Brown 
Committed: Thu Apr 5 06:10:59 2018 -0700

--
 CHANGES.txt   | 1 +
 .../org/apache/cassandra/cql3/functions/FunctionCall.java | 2 ++
 .../apache/cassandra/cql3/validation/entities/JsonTest.java   | 7 +++
 3 files changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4828517..527975c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.13
+ * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
  * Fix query pager DEBUG log leak causing hit in paged reads throughput 
(CASSANDRA-14318)
  * Backport circleci yaml (CASSANDRA-14240)
 Merged from 2.1:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--
diff --git a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java 
b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
index 4f53c98..323f1bb 100644
--- a/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
+++ b/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
@@ -98,6 +98,8 @@ public class FunctionCall extends Term.NonTerminal
 
 private static Term.Terminal makeTerminal(Function fun, ByteBuffer result, 
int version) throws InvalidRequestException
 {
+if (result == null)
+return null;
 if (!(fun.returnType() instanceof CollectionType))
 return new Constants.Value(result);
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e5e11d6/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java 
b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 0f29928..46cdd52 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@ -247,6 +247,13 @@ public class JsonTest extends CQLTester
 
 // handle nulls
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, asciival FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, frozenmapval) VALUES (?, fromJson(?))", 0, 
null);
+assertRows(execute("SELECT k, frozenmapval FROM %s WHERE k = ?", 0), 
row(0, null));
+
+execute("INSERT INTO %s (k, udtval) VALUES (?, fromJson(?))", 0, null);
+assertRows(execute("SELECT k, udtval FROM %s WHERE k = ?", 0), row(0, 
null));
 
 //  ascii 
 execute("INSERT INTO %s (k, asciival) VALUES (?, fromJson(?))", 0, 
"\"ascii text\"");


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/41f3b96f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/41f3b96f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/41f3b96f

Branch: refs/heads/cassandra-3.11
Commit: 41f3b96f81e08af8a95124b9ab50bad03067aab0
Parents: 2d2b1a7 2e5e11d
Author: Jason Brown 
Authored: Thu Apr 5 06:16:13 2018 -0700
Committer: Jason Brown 
Committed: Thu Apr 5 06:20:57 2018 -0700

--
 .../org/apache/cassandra/cql3/functions/FunctionCall.java |  2 ++
 .../cassandra/cql3/validation/entities/JsonTest.java  | 10 +-
 2 files changed, 11 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/41f3b96f/src/java/org/apache/cassandra/cql3/functions/FunctionCall.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/41f3b96f/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
--
diff --cc test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
index 94b2087,46cdd52..62c6226
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/JsonTest.java
@@@ -39,7 -38,7 +39,8 @@@ import java.util.*
  import java.util.concurrent.*;
  import static org.junit.Assert.fail;
  
--public class JsonTest extends CQLTester
++public class
++JsonTest extends CQLTester
  {
  @BeforeClass
  public static void setUp()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13653) Create meaningful toString() methods

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown resolved CASSANDRA-13653.
-
   Resolution: Fixed
Fix Version/s: 4.0

Thanks for addressing [~djoshi3]'s last comment. LGTM, as well. committed as 
sha {{946aaa7b06f2ccd697e31dcc15c29468da523311}}

> Create meaningful toString() methods
> 
>
> Key: CASSANDRA-13653
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13653
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jeff Jirsa
>Assignee: Jogesh Anand
>Priority: Trivial
>  Labels: lhf, low-hanging-fruit
> Fix For: 4.0
>
>
> True low-hanging fruit, good for a first-time contributor:
> There are a lot of classes without meaningful {{toString()}} implementations. 
> Some of these would be very nice to have for investigating bug reports.
> Some good places to start: 
> - CQL3 statements (UpdateStatement, DeleteStatement, etc), QueryOptions, and 
> Restrictions
> Some packages not to worry about: 
> - Deep internals that don't already have them 
> (org.apache.cassandra.db.rows/partitions/etc)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Add meaningful toString() impls

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/trunk d3bd26eb3 -> 946aaa7b0


Add meaningful toString() impls

patch by Jogesh Anand; reviewed by Dinesh Joshi for CASSANDRA-13653


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/946aaa7b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/946aaa7b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/946aaa7b

Branch: refs/heads/trunk
Commit: 946aaa7b06f2ccd697e31dcc15c29468da523311
Parents: d3bd26e
Author: djanand 
Authored: Sat Mar 31 17:53:09 2018 -0500
Committer: Jason Brown 
Committed: Thu Apr 5 05:19:22 2018 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/cql3/Attributes.java   |  8 
 .../cassandra/cql3/BatchQueryOptions.java   |  8 
 .../org/apache/cassandra/cql3/QueryOptions.java |  8 
 .../org/apache/cassandra/cql3/WhereClause.java  |  8 
 .../cql3/conditions/ColumnCondition.java| 20 
 .../cql3/conditions/ColumnConditions.java   |  8 
 .../restrictions/CustomIndexExpression.java |  8 
 .../cql3/restrictions/IndexRestrictions.java|  8 
 .../restrictions/MultiColumnRestriction.java|  8 
 .../cql3/restrictions/RestrictionSet.java   |  8 
 .../restrictions/RestrictionSetWrapper.java |  8 
 .../restrictions/StatementRestrictions.java |  8 
 .../cql3/statements/AlterKeyspaceStatement.java |  8 
 .../cql3/statements/AlterRoleStatement.java |  8 
 .../statements/AlterTableStatementColumn.java   |  8 
 .../cql3/statements/AlterTypeStatement.java |  8 
 .../cql3/statements/AuthorizationStatement.java |  8 
 .../cassandra/cql3/statements/CFProperties.java |  8 
 .../cassandra/cql3/statements/CFStatement.java  |  8 
 .../cql3/statements/CQL3CasRequest.java |  8 
 .../statements/CreateAggregateStatement.java|  8 
 .../statements/CreateFunctionStatement.java |  8 
 .../cql3/statements/CreateIndexStatement.java   |  8 
 .../statements/CreateKeyspaceStatement.java |  8 
 .../cql3/statements/CreateRoleStatement.java|  8 
 .../cql3/statements/CreateTableStatement.java   |  8 
 .../cql3/statements/CreateTriggerStatement.java |  8 
 .../cql3/statements/CreateTypeStatement.java|  8 
 .../cql3/statements/CreateViewStatement.java|  8 
 .../cql3/statements/DeleteStatement.java|  8 
 .../cql3/statements/DropAggregateStatement.java |  8 
 .../cql3/statements/DropFunctionStatement.java  |  8 
 .../cql3/statements/DropIndexStatement.java |  8 
 .../cql3/statements/DropKeyspaceStatement.java  |  8 
 .../cql3/statements/DropRoleStatement.java  |  8 
 .../cql3/statements/DropTableStatement.java |  8 
 .../cql3/statements/DropTriggerStatement.java   |  8 
 .../cql3/statements/DropTypeStatement.java  |  8 
 .../cql3/statements/DropViewStatement.java  |  8 
 .../cql3/statements/GrantRoleStatement.java |  8 
 .../cassandra/cql3/statements/IndexTarget.java  |  8 
 .../statements/ListPermissionsStatement.java|  8 
 .../cql3/statements/ListRolesStatement.java |  8 
 .../cql3/statements/ListUsersStatement.java |  8 
 .../PermissionsManagementStatement.java |  8 
 .../statements/RevokePermissionsStatement.java  |  8 
 .../cql3/statements/RevokeRoleStatement.java|  8 
 .../statements/RoleManagementStatement.java |  8 
 .../cql3/statements/SelectStatement.java|  8 
 .../cql3/statements/TruncateStatement.java  |  8 
 .../cql3/statements/UpdateStatement.java|  8 
 .../cassandra/cql3/statements/UseStatement.java |  8 
 53 files changed, 429 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/946aaa7b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4fd76fc..b3617c3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Add meaningful toString() impls (CASSANDRA-13653)
  * Add sstableloader option to accept target keyspace name (CASSANDRA-13884)
  * Move processing of EchoMessage response to gossip stage (CASSANDRA-13713)
  * Add coordinator write metric per CF (CASSANDRA-14232)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/946aaa7b/src/java/org/apache/cassandra/cql3/Attributes.java
--
diff --git 

[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426773#comment-16426773
 ] 

Stefan Podkowinski commented on CASSANDRA-14346:


If we keep the scope of this ticket to schedule repairs in Cassandra, we should 
really talk a bit more about the different requirements users have and how 
using the described solution in practice would look like.

There are several aspect to consider for coming up with a working repair 
schedule:
 * number of tables (from a single table per cluster to hundreds of tables)
 * priority in repairing tables (some tables should be repaired more often, 
others never at all)
 * data size per table (large table should not block repairs for smaller more 
important ones)
 * predictable cluster load (try to schedule repairs off hours)
 * sustainable repair intensity (repair sessions should not leak into peak 
hours)
 * different gc_grace periods (plan intervals for each table so we can tolerate 
missing a repair run)

Repair schedules, which will take these aspects into account, require a certain 
flexibility and some more careful configuration. Tools, such as reaper, allow 
you to put together such plans already. Looking at the configuration options 
described in the design document, I'd probably still want to use such an 
external tool. That would be mostly due to the use of delays instead of 
recurring repair times and the way you'd have to configure repairs on table 
level, which probably gets a bit "messy" fast when you have a lot of tables. 
The lack of any reporting doesn't help either to further tune these config 
options afterwards.

I think the intention is to keep the scope of this ticket to "integrated repair 
scheduling and execution", so I'll spare you any of my thoughts about how we 
should coordinate and execute repairs differently in a post CASSANDRA-9143 
world. But if we want to solve scheduling on top of our existing repair 
implementation, we have to make sure that we can compete with existing 3rd 
party solutions.

So far it was already suggested to move on incrementally. But then we also have 
to think about how improvements could be implemented on top of the proposed 
solution. I'd assume that optimizations would be easier to implement in 
external tools or sidecars that communicates via an IPC interface, compared to 
a baked in solution, which is using the yaml config, table properties, or has 
to deal with upgrade paths. From my impression, 3rd party projects are probably 
also a better place to quickly iterate on these kind of problems.

> Scheduled Repair in Cassandra
> -
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Repair
>Reporter: Joseph Lynch
>Priority: Major
>  Labels: CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes 
> sense given that it is necessary to give our users eventual consistency. Most 
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked 
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), 
> which we spoke about last year at NGCC. Given the positive feedback at NGCC 
> we focussed on getting it production ready and have now been using it in 
> production to repair hundreds of clusters, tens of thousands of nodes, and 
> petabytes of data for the past six months. Also based on feedback at NGCC we 
> have invested effort in figuring out how to integrate this natively into 
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our 
> implementation into Cassandra, and have created a [design 
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
>  showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would 
> be greatly appreciated about the interface or v1 implementation features. I 
> have tried to call out in the document features which we explicitly consider 
> future work (as well as a path forward to implement them in the future) 
> because I would very much like to get this done before the 4.0 merge window 
> closes, and to do that I think aggressively pruning scope is going to be a 
> necessity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: