date:20200512



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15667:
--
Fix Version/s: (was: 4.0)
   4.0-alpha

> StreamResultFuture check for completeness is inconsistent, leading to races
> ---
>
> Key: CASSANDRA-15667
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15667
> Project: Cassandra
>  Issue Type: Bug
>  Components: Legacy/Streaming and Messaging
>Reporter: Sergio Bossa
>Assignee: Massimiliano Tomassi
>Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: log_bootstrap_resumable
>
>
> {{StreamResultFuture#maybeComplete()}} uses 
> {{StreamCoordinator#hasActiveSessions()}} to determine if all sessions are 
> completed, but then accesses each session state via 
> {{StreamCoordinator#getAllSessionInfo()}}: this is inconsistent, as the 
> former relies on the actual {{StreamSession}} state, while the latter on the 
> {{SessionInfo}} state, and the two are concurrently updated with no 
> coordination whatsoever.
> This leads to races, i.e. apparent in some dtest spurious failures, such as 
> {{TestBootstrap.resumable_bootstrap_test}} in CASSANDRA-15614 cc 
> [~e.dimitrova].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15663) DESCRIBE KEYSPACE does not properly quote table names



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15663:
--
Fix Version/s: (was: 4.0)
   4.0-beta

> DESCRIBE KEYSPACE does not properly quote table names
> -
>
> Key: CASSANDRA-15663
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15663
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL/Syntax
>Reporter: Oskar Liljeblad
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 3.11.x, 4.0-beta
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> How to reproduce (3.11.6) - cqlsh:
> {code}
> CREATE KEYSPACE test1 WITH replication = \{'class': 'SimpleStrategy', 
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test1."default" (id text PRIMARY KEY, data text, etag text);
> DESCRIBE KEYSPACE test1;
> {code}
> Output will be:
> {code}
> CREATE TABLE test1.default (
>  id text PRIMARY KEY,
>  data text,
>  etag text
> ) WITH [..]
> {code}
> Output should be:
> {code}
> CREATE TABLE test1."default" (
>  id text PRIMARY KEY,
>  data text,
>  etag text
> ) WITH [..]
> {code}
>  If you try to run {{CREATE TABLE test1.default [..]}} you will get an error 
> SyntaxException: line 1:19 no viable alternative at input 'default' (CREATE 
> TABLE test1.[default]...)
> Oskar Liljeblad
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14825) Expose table schema for drivers



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14825:
--
Fix Version/s: (was: 4.0)
   4.0-alpha

> Expose table schema for drivers
> ---
>
> Key: CASSANDRA-14825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14825
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/CQL
>Reporter: Chris Lohfink
>Assignee: Robert Stupp
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 4.0-alpha
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Currently the drivers recreate the CQL for the tables by putting together the 
> system table values. This is very difficult to keep up to date and buggy 
> enough that its only even supported in Java and Python drivers. Cassandra 
> already has some limited output available for snapshots that we could provide 
> in a virtual table or new query that the drivers can fetch. This can greatly 
> reduce the complexity of drivers while also reducing bugs like 
> CASSANDRA-14822 as the underlying schema and properties change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15663) DESCRIBE KEYSPACE does not properly quote table names



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15663:
--
Fix Version/s: (was: 4.0-beta)
   4.0-alpha

> DESCRIBE KEYSPACE does not properly quote table names
> -
>
> Key: CASSANDRA-15663
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15663
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL/Syntax
>Reporter: Oskar Liljeblad
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 3.11.x, 4.0-alpha
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> How to reproduce (3.11.6) - cqlsh:
> {code}
> CREATE KEYSPACE test1 WITH replication = \{'class': 'SimpleStrategy', 
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test1."default" (id text PRIMARY KEY, data text, etag text);
> DESCRIBE KEYSPACE test1;
> {code}
> Output will be:
> {code}
> CREATE TABLE test1.default (
>  id text PRIMARY KEY,
>  data text,
>  etag text
> ) WITH [..]
> {code}
> Output should be:
> {code}
> CREATE TABLE test1."default" (
>  id text PRIMARY KEY,
>  data text,
>  etag text
> ) WITH [..]
> {code}
>  If you try to run {{CREATE TABLE test1.default [..]}} you will get an error 
> SyntaxException: line 1:19 no viable alternative at input 'default' (CREATE 
> TABLE test1.[default]...)
> Oskar Liljeblad
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14557) Consider adding default and required keyspace replication options

2020-05-12 Thread Alex Petrov (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105521#comment-17105521
 ] 

Alex Petrov commented on CASSANDRA-14557:
-

Wanted to commit this since it looked like this is a complete patch, so rebased 
and triggered a CI.

I have two comments:

  * I'm not sure if changing replication factor of system keyspaces is expected 
by anyone who sees this feature. Maybe we should have a discussion about this 
before the change.
  * There's a dtest failure that looks related, please check it out before we 
proceed with committing: 
https://circleci.com/gh/ifesdjeen/cassandra/2122#tests/containers/13 more 
failures here: 
https://circleci.com/gh/ifesdjeen/cassandra/2123#tests/containers/32

 Rebased branch: https://github.com/ifesdjeen/cassandra/tree/14557-trunk
All CI results: https://circleci.com/gh/ifesdjeen/cassandra/tree/14557-trunk

In general, I suggest putting ticket into "ready to commit" state so that 
committers could pick notice it, or ping some committer directly.

> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local/Config
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Low
>  Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.x
>
> Attachments: 14557-trunk.patch
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right 
> now - the system_auth table for example is created with RF=1 (to take into 
> account single node setups afaict from CASSANDRA-5112), and a user can 
> further create a keyspace with RF=1 posing availability and streaming risks 
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
>  # {{default_keyspace_rf}} (default: 1) - If replication factors are not 
> specified, use this number.
>  # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from 
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
>  * Provide defaults for new keyspaces created with SimpleStrategy or 
> NetworkTopologyStrategy (CASSANDRA-14303)
>  * Make the automatic token [allocation 
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
>  interface more intuitive allowing easy use of the new token allocation 
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply 
> don’t set the setting. For backwards compatibility the default remains 1 and 
> C* would create with RF=1, and would default to current behavior of allowing 
> any RF on keyspaces.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15472) Read failure due to exception from metrics-core dependency



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15472:
--
Fix Version/s: (was: 4.0)
   4.0-beta

> Read failure due to exception from metrics-core dependency
> --
>
> Key: CASSANDRA-15472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15472
> Project: Cassandra
>  Issue Type: Bug
>  Components: Dependencies
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> Stacktrace
> {code:java}
> Uncaught exception on thread Thread[SharedPool-Worker-27,5,main]: {}
> java.util.NoSuchElementException: null
>   at 
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2053)
>  ~[na:1.8.0_222]
>   at 
> com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:102)
>  ~[metrics-core-2.2.0.jar:na]
>   at 
> com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
>  ~[metrics-core-2.2.0.jar:na]
>   at com.yammer.metrics.core.Histogram.update(Histogram.java:110) 
> ~[metrics-core-2.2.0.jar:na]
>   at com.yammer.metrics.core.Timer.update(Timer.java:198) 
> ~[metrics-core-2.2.0.jar:na]
>   at com.yammer.metrics.core.Timer.update(Timer.java:76) 
> ~[metrics-core-2.2.0.jar:na]
>   at 
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:108) 
> ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at 
> org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:114) 
> ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1897)
>  ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:353) 
> ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at 
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
>  ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at 
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47) 
> ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_222]
>   at 
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>  ~[nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [nf-cassandra-2.1.19.10.jar:2.1.19.10]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]
> {code}
> This [issue|https://github.com/dropwizard/metrics/issues/1278] has been 
> [fixed|https://github.com/dropwizard/metrics/pull/1436] in 
> [v4.0.6|https://github.com/dropwizard/metrics/releases/tag/v4.0.6].
> This is observed on a 2.1.19 cluster, but this would impact pretty much any 
> version of C* since we depend on lower versions of metrics-core that do not 
> have the fix.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15393) Add byte array backed cells



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15393:
--
Fix Version/s: (was: 4.0)
   4.0-beta

> Add byte array backed cells
> ---
>
> Key: CASSANDRA-15393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15393
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0-beta
>
>
> We currently materialize all values as on heap byte buffers. Byte buffers 
> have a fairly high overhead given how frequently they’re used, and on the 
> compaction and local read path we don’t do anything that needs them. Use of 
> byte buffer methods only happens on the coordinator. Using cells that are 
> backed by byte arrays instead in these situations reduces compaction and read 
> garbage up to 22% in many cases.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15369) Fake row deletions and range tombstones, causing digest mismatch and sstable growth



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15369:
--
Fix Version/s: (was: 4.0)
   4.0-beta

> Fake row deletions and range tombstones, causing digest mismatch and sstable 
> growth
> ---
>
> Key: CASSANDRA-15369
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15369
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Coordination, Local/Memtable, Local/SSTable
>Reporter: Benedict Elliott Smith
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>
> As assessed in CASSANDRA-15363, we generate fake row deletions and fake 
> tombstone markers under various circumstances:
>  * If we perform a clustering key query (or select a compact column):
>  * Serving from a {{Memtable}}, we will generate fake row deletions
>  * Serving from an sstable, we will generate fake row tombstone markers
>  * If we perform a slice query, we will generate only fake row tombstone 
> markers for any range tombstone that begins or ends outside of the limit of 
> the requested slice
>  * If we perform a multi-slice or IN query, this will occur for each 
> slice/clustering
> Unfortunately, these different behaviours can lead to very different data 
> stored in sstables until a full repair is run.  When we read-repair, we only 
> send these fake deletions or range tombstones.  A fake row deletion, 
> clustering RT and slice RT, each produces a different digest.  So for each 
> single point lookup we can produce a digest mismatch twice, and until a full 
> repair is run we can encounter an unlimited number of digest mismatches 
> across different overlapping queries.
> Relatedly, this seems a more problematic variant of our atomicity failures 
> caused by our monotonic reads, since RTs can have an atomic effect across (up 
> to) the entire partition, whereas the propagation may happen on an 
> arbitrarily small portion.  If the RT exists on only one node, this could 
> plausibly lead to fairly problematic scenario if that node fails before the 
> range can be repaired. 
> At the very least, this behaviour can lead to an almost unlimited amount of 
> extraneous data being stored until the range is repaired and compaction 
> happens to overwrite the sub-range RTs and row deletions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15805) Potential duplicate rows on 2.X->3.X upgrade when multi-rows range tombstones interacts with collection tombstones

2020-05-12 Thread Sylvain Lebresne (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105536#comment-17105536
 ] 

Sylvain Lebresne commented on CASSANDRA-15805:
--

To understand why this happens, let me write down the atoms that the example of 
the description generates on 2.X (using a simplified representation that I hope 
is clear enough):
{noformat}
atom1: RT([A:_, A:X:b:_])@1, // beginning of all 'A' rows to beginning of 
A:X's b column
atom2: Cell(A:X:)@2, // row marker for A:X
atom3: Cell(A:X:a=foo)@2,// value of a in A:X
atom4: RT([A:X:b:_, A:X:b:!])@3, // collection tombstone for b in A:X's
atom5: RT([A:X:b:!, A:!])@1, // remainder of covering RT, from end of b in 
A:X to end of all 'A' rows
atom6: Cell(A:X:c=bar)@2 // value of c in A:X
{noformat}
Those atoms are deserialized into {{LegacyCell}} and {{LegacyRangeTombstone}} 
on 3.X as:
{noformat}
atom1: RT(Bound(INCL_START_BOUND(A), 
collection=null)-Bound(EXCL_END_BOUND(A:B), collection=null), deletedAt=1, 
localDeletion=1589204864)
atom2: LegacyCell(REGULAR, name=Cellname(clustering=A:X, column=null, 
collElt=null), v=, ts=2, ldt=2147483647, ttl=0)
atom3: LegacyCell(REGULAR, name=Cellname(clustering=A:X, column=a, 
collElt=null), v=foo, ts=2, ldt=2147483647, ttl=0)
atom4: RT(Bound(INCL_START_BOUND(A:X), collection=b)-Bound(INCL_END_BOUND(A:X), 
collection=b), deletedAt=3, localDeletion=1589204864)
atom5: RT(Bound(EXCL_START_BOUND(A:X), 
collection=null)-Bound(INCL_END_BOUND(A), collection=null), deletedAt=1, 
localDeletion=1589204864)
atom6: LegacyCell(REGULAR, name=Cellname(clustering=A:X, column=c, 
collElt=null), v=bar, ts=2, ldt=2147483647, ttl=0)
{noformat}

I'll point out that those are a direct translation of the 2.X atoms except for 
{{atom1}} and {{atom5}} that are slightly different:
* instead of {{atom1}} stopping at the beginning of the row {{b}} column, it 
extends to the end of the row.
* and instead of {{atom5}} staring after that {{b}} column, it starts after the 
row. Do note however that the order of atoms is still the one above, so that 
atom is effectively out-of-order.

The reason for those differences is the logic [at the beginning of 
{{LegacyLayout.RangeTombstone}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1883],
 whose comment is trying to explain, but is basically due to the legacy layer 
having to map all 2.X RTs into either a 3.X range tombstone (so one over 
multiple rows), a row tombstone or a collection one.

Anyway, as mentioned above, the problem is that {{atom5}} is out of order.  
What currently happens is that when {{atom5}} is encountered by 
{{UnfilteredDeserialized.OldFormatDeserializer}}, it will be passed to the 
{{CellGrouper}} currently grouping the row, and will end up in the 
[{{CellGrouper#addGenericTombstone}} 
method|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/LegacyLayout.java#L1544].
  But, because that atom starts strictly after the row being grouped, the 
method returns {{false}} and the row is generated a first time. Later, we get 
{{atom6}} which restarts the row with the value of column {{c}}, after which it 
is generated a second time.


> Potential duplicate rows on 2.X->3.X upgrade when multi-rows range tombstones 
> interacts with collection tombstones
> --
>
> Key: CASSANDRA-15805
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15805
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Coordination, Local/SSTable
>Reporter: Sylvain Lebresne
>Priority: Normal
>
> The legacy reading code ({{LegacyLayout}} and 
> {{UnfilteredDeserializer.OldFormatDeserializer}}) does not handle correctly 
> the case where a range tombstone covering multiple rows interacts with a 
> collection tombstone.
> A simple example of this problem is if one runs on 2.X:
> {noformat}
> CREATE TABLE t (
>   k int,
>   c1 text,
>   c2 text,
>   a text,
>   b set,
>   c text,
>   PRIMARY KEY((k), c1, c2)
> );
> // Delete all rows where c1 is 'A'
> DELETE FROM t USING TIMESTAMP 1 WHERE k = 0 AND c1 = 'A';
> // Inserts a row covered by that previous range tombstone
> INSERT INTO t(k, c1, c2, a, b, c) VALUES (0, 'A', 'X', 'foo', {'whatever'}, 
> 'bar') USING TIMESTAMP 2;
> // Delete the collection of that previously inserted row
> DELETE b FROM t USING TIMESTAMP 3 WHERE k = 0 AND c1 = 'A' and c2 = 'X';
> {noformat}
> If the following is ran on 2.X (with everything either flushed in the same 
> table or compacted together), then this will result in the inserted row being 
> duplicated (one part containing the {{a}} column, the other the {{c}} one).
> I will no

[jira] [Updated] (CASSANDRA-15805) Potential duplicate rows on 2.X->3.X upgrade when multi-rows range tombstones interacts with collection tombstones

2020-05-12 Thread Sylvain Lebresne (Jira)



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-15805:
-
 Bug Category: Parent values: Correctness(12982)Level 1 values: Recoverable 
Corruption / Loss(12986)
   Complexity: Normal
Discovered By: User Report
 Severity: Critical
 Assignee: Sylvain Lebresne
   Status: Open  (was: Triage Needed)

> Potential duplicate rows on 2.X->3.X upgrade when multi-rows range tombstones 
> interacts with collection tombstones
> --
>
> Key: CASSANDRA-15805
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15805
> Project: Cassandra
>  Issue Type: Bug
>  Components: Consistency/Coordination, Local/SSTable
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
>Priority: Normal
>
> The legacy reading code ({{LegacyLayout}} and 
> {{UnfilteredDeserializer.OldFormatDeserializer}}) does not handle correctly 
> the case where a range tombstone covering multiple rows interacts with a 
> collection tombstone.
> A simple example of this problem is if one runs on 2.X:
> {noformat}
> CREATE TABLE t (
>   k int,
>   c1 text,
>   c2 text,
>   a text,
>   b set,
>   c text,
>   PRIMARY KEY((k), c1, c2)
> );
> // Delete all rows where c1 is 'A'
> DELETE FROM t USING TIMESTAMP 1 WHERE k = 0 AND c1 = 'A';
> // Inserts a row covered by that previous range tombstone
> INSERT INTO t(k, c1, c2, a, b, c) VALUES (0, 'A', 'X', 'foo', {'whatever'}, 
> 'bar') USING TIMESTAMP 2;
> // Delete the collection of that previously inserted row
> DELETE b FROM t USING TIMESTAMP 3 WHERE k = 0 AND c1 = 'A' and c2 = 'X';
> {noformat}
> If the following is ran on 2.X (with everything either flushed in the same 
> table or compacted together), then this will result in the inserted row being 
> duplicated (one part containing the {{a}} column, the other the {{c}} one).
> I will note that this is _not_ a duplicate of CASSANDRA-15789 and this 
> reproduce even with the fix to {{LegacyLayout}} of this ticket. That said, 
> the additional code added to CASSANDRA-15789 to force merging duplicated rows 
> if they are produced _will_ end up fixing this as a consequence (assuming 
> there is no variation of this problem that leads to other visible issues than 
> duplicated rows). That said, I "think" we'd still rather fix the source of 
> the issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15299:
--
Fix Version/s: (was: 4.0-beta)
   4.0-alpha

> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol 
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it 
> introduced checksumming/CRC32 to request and response bodies. It’s an 
> important step forward, but it doesn’t cover the entire stream. In 
> particular, the message header is not covered by a checksum or a crc, which 
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of 
> computing the CRC32 on the bytes written on the wire - losing the properties 
> of the CRC32. In some cases, due to this sequencing, attempting to decompress 
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order, 
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for 
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since 
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression 
> operate on individual message bodies rather than frames of multiple complete 
> messages. In reality, this has several important additional downsides. To 
> name a couple:
> # For compression, we are getting poor compression ratios for smaller 
> messages - when operating on tiny sequences of bytes. In reality, for most 
> small requests and responses we are discarding the compressed value as it’d 
> be smaller than the uncompressed one - incurring both redundant allocations 
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages. 
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the 
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we 
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode 
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed, 
> and that we do it before native protocol v5 graduates from beta. See 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
>  and 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15146) Transitional TLS server configuration options are overly complex



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15146:
--
Fix Version/s: (was: 4.0-beta)
   (was: 4.0)
   4.0-alpha

> Transitional TLS server configuration options are overly complex
> 
>
> Key: CASSANDRA-15146
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15146
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/Encryption, Local/Config
>Reporter: Joey Lynch
>Assignee: M Carlise
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> It appears as part of the port from transitional client TLS to transitional 
> server TLS in CASSANDRA-10404 (the ability to switch a cluster to using 
> {{internode_encryption}} without listening on two ports and without downtime) 
> we carried the {{enabled}} setting over from the client implementation. I 
> believe that the {{enabled}} option is redundant to {{internode_encryption}} 
> and {{optional}} and it should therefore be removed prior to the 4.0 release 
> where we will have to start respecting that interface. 
> Current trunk yaml:
> {noformat}
> server_encryption_options:
>   
> # set to true for allowing secure incoming connections
>   
> enabled: false
>   
> # If enabled and optional are both set to true, encrypted and unencrypted 
> connections are handled on the storage_port
> optional: false   
>   
>   
>   
> 
> # if enabled, will open up an encrypted listening socket on 
> ssl_storage_port. Should be used
> # during upgrade to 4.0; otherwise, set to false. 
>   
> enable_legacy_ssl_storage_port: false 
>   
> # on outbound connections, determine which type of peers to securely 
> connect to. 'enabled' must be set to true.
> internode_encryption: none
>   
> keystore: conf/.keystore  
>   
> keystore_password: cassandra  
>   
> truststore: conf/.truststore  
>   
> truststore_password: cassandra
> {noformat}
> I propose we eliminate {{enabled}} and just use {{optional}} and 
> {{internode_encryption}} to determine the listener setup. I also propose we 
> change the default of {{optional}} to true. We could also re-name 
> {{optional}} since it's a new option but I think it's good to stay consistent 
> with the client and use {{optional}}.
> ||optional||internode_encryption||description||
> |true|none|(default) No encryption is used but if a server reaches out with 
> it we'll use it|
> |false|dc|Encryption is required for inter-dc communication, but not intra-dc|
> |false|all|Encryption is required for all communication|
> |false|none|We only listen for unencrypted connections|
> |true|dc|Encryption is used for inter-dc communication but is not required|
> |true|all|Encryption is used for all communication but is not required|
> From these states it is clear when we should be accepting TLS connections 
> (all except for false and none) as well as when we must enforce it.
> To transition without downtime from an un-encrypted cluster to an encrypted 
> cluster the user would do the following:
> 1. After adding valid truststores, change {{internode_encryption}} to the 
> desired level of encryption (recommended {{all}}) and restart Cassandra
>  2. Change {{optional=false}} and restart Cassandra to enforce #1
> If {{optional}} defaulted to {{false}} as it does right now we'd need a third 
> restart to first change {{optional}} to {{true}}, which given my 
> understanding of the OptionalSslHandler isn't really relevant.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14973) Bring v5 driver out of beta, introduce v6 before 4.0 release is cut



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14973:
--
Fix Version/s: (was: 4.0-rc)
   (was: 4.0)
   4.0-beta

> Bring v5 driver out of beta, introduce v6 before 4.0 release is cut
> ---
>
> Key: CASSANDRA-14973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14973
> Project: Cassandra
>  Issue Type: Task
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Urgent
>  Labels: protocolv5
> Fix For: 4.0-beta
>
>
> In http://issues.apache.org/jira/browse/CASSANDRA-12142, we’ve introduced 
> Beta flag for v5 protocol. However, up till now, v5 is in beta both in 
> [Cassandra|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/transport/ProtocolVersion.java#L46]
>  and in 
> [java-driver|https://github.com/datastax/java-driver/blob/3.x/driver-core/src/main/java/com/datastax/driver/core/ProtocolVersion.java#L35].
>  
> Before the final 4.0 release is cut, we need to bring v5 out of beta and 
> finalise native protocol spec, and start bringing all new changes to v6 
> protocol, which will be in beta.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-15806) C* 4.0 is missing a way to identify transient SSTables on disk

sequoyha pelletier created CASSANDRA-15806:
--

 Summary: C* 4.0 is missing a way to identify transient SSTables on 
disk
 Key: CASSANDRA-15806
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15806
 Project: Cassandra
  Issue Type: Bug
  Components: Tool/sstable
Reporter: sequoyha pelletier
Assignee: sequoyha pelletier


Currently, there is no way to identify SSTables that were created as transient 
replicated data. Even thought the feature is experimental we should open that 
up for those that want to experiment. This seems to be an oversight. I have 
added the missing line of code to the SStableMetadataViewer and will attach a 
patch shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14848) When upgrading 3.11.3->4.0 using SSL 4.0 nodes does not connect to old non seed nodes



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14848:
--
Fix Version/s: (was: 4.0)
   4.0-beta

> When upgrading 3.11.3->4.0 using SSL 4.0 nodes does not connect to old non 
> seed nodes
> -
>
> Key: CASSANDRA-14848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14848
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Internode
>Reporter: Tommy Stendahl
>Assignee: Tommy Stendahl
>Priority: Urgent
>  Labels: security
> Fix For: 4.0-beta
>
>
> When upgrading from 3.11.3 to 4.0 with server encryption enabled the new 4.0 
> node only connects to 3.11.3 seed node, there are no connection established 
> to non-seed nodes on the old version.
> I have four nodes, *.242 is upgraded to 4.0, *.243 and *.244 are 3.11.3 
> non-seed and *.246 are 3.11.3 seed. After starting the 4.0 node I get this 
> nodetool status on the different nodes:
> {noformat}
> *.242
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.216.193.242 1017.77 KiB 256 75,1% 7d278e14-d549-42f3-840d-77cfd852fbf4 
> RAC1
> DN 10.216.193.243 743.32 KiB 256 74,8% 5586243a-ca74-4125-8e7e-09e82e23c4e5 
> RAC1
> DN 10.216.193.244 711.54 KiB 256 75,2% c155e262-b898-4e86-9e1d-d4d0f97e88f6 
> RAC1
> UN 10.216.193.246 659.81 KiB 256 74,9% 502dd00f-fc02-4024-b65f-b98ba3808291 
> RAC1
> *.243 and *.244
> -- Address Load Tokens Owns (effective) Host ID Rack
> DN 10.216.193.242 657.4 KiB 256 75,1% 7d278e14-d549-42f3-840d-77cfd852fbf4 
> RAC1
> UN 10.216.193.243 471 KiB 256 74,8% 5586243a-ca74-4125-8e7e-09e82e23c4e5 RAC1
> UN 10.216.193.244 471.71 KiB 256 75,2% c155e262-b898-4e86-9e1d-d4d0f97e88f6 
> RAC1
> UN 10.216.193.246 388.54 KiB 256 74,9% 502dd00f-fc02-4024-b65f-b98ba3808291 
> RAC1
> *.246
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.216.193.242 657.4 KiB 256 75,1% 7d278e14-d549-42f3-840d-77cfd852fbf4 
> RAC1
> UN 10.216.193.243 471 KiB 256 74,8% 5586243a-ca74-4125-8e7e-09e82e23c4e5 RAC1
> UN 10.216.193.244 471.71 KiB 256 75,2% c155e262-b898-4e86-9e1d-d4d0f97e88f6 
> RAC1
> UN 10.216.193.246 388.54 KiB 256 74,9% 502dd00f-fc02-4024-b65f-b98ba3808291 
> RAC1
> {noformat}
>  
> I have built 4.0 with wire tracing activated and in my config the 
> storage_port=12700 and ssl_storage_port=12701. In the log I can see that the 
> 4.0 node start to connect to the 3.11.3 seed node on the storage_port but 
> quickly switch to the ssl_storage_port, but when connecting to the non-seed 
> nodes it never switch to the ssl_storage_port.
> {noformat}
> >grep 193.246 system.log | grep Outbound
> 2018-10-25T10:57:36.799+0200 [MessagingService-NettyOutbound-Thread-4-1] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0x2f0e5e55] CONNECT: 
> /10.216.193.246:12700
> 2018-10-25T10:57:36.902+0200 [MessagingService-NettyOutbound-Thread-4-2] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0x9e81f62c] CONNECT: 
> /10.216.193.246:12701
> 2018-10-25T10:57:36.905+0200 [MessagingService-NettyOutbound-Thread-4-2] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0x9e81f62c, 
> L:/10.216.193.242:37252 - R:10.216.193.246/10.216.193.246:12701] ACTIVE
> 2018-10-25T10:57:36.906+0200 [MessagingService-NettyOutbound-Thread-4-2] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0x9e81f62c, 
> L:/10.216.193.242:37252 - R:10.216.193.246/10.216.193.246:12701] WRITE: 8B
> >grep 193.243 system.log | grep Outbound
> 2018-10-25T10:57:38.438+0200 [MessagingService-NettyOutbound-Thread-4-3] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0xd8f1d6c4] CONNECT: 
> /10.216.193.243:12700
> 2018-10-25T10:57:38.540+0200 [MessagingService-NettyOutbound-Thread-4-4] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0xfde6cc9f] CONNECT: 
> /10.216.193.243:12700
> 2018-10-25T10:57:38.694+0200 [MessagingService-NettyOutbound-Thread-4-5] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0x7e87fc4e] CONNECT: 
> /10.216.193.243:12700
> 2018-10-25T10:57:38.741+0200 [MessagingService-NettyOutbound-Thread-4-7] INFO 
> i.n.u.internal.logging.Slf4JLogger:101 info [id: 0x39395296] CONNECT: 
> /10.216.193.243:12700{noformat}
>  
> When I had the dbug log activated and started the 4.0 node I can see that it 
> switch port for *.246 but not for *.243 and *.244.
> {noformat}
> >grep DEBUG system.log| grep OutboundMessagingConnection | grep 
> >maybeUpdateConnectionId
> 2018-10-25T13:12:56.095+0200 [ScheduledFastTasks:1] DEBUG 
> o.a.c.n.a.OutboundMessagingConnection:314 maybeUpdateConnectionId changing 
> connectionId to 10.216.193.246:12701 (GOSSIP), with a different port for 
> secure communication, because peer version is 11
> 2018-10-25T13:12:58.100+

[jira] [Updated] (CASSANDRA-14764) Test Messaging Refactor with: 12 Node Breaking Point, compression=none, encryption=none, coalescing=off



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14764:
--
Fix Version/s: (was: 4.0-rc)
   4.0-beta

> Test Messaging Refactor with: 12 Node Breaking Point, compression=none, 
> encryption=none, coalescing=off
> ---
>
> Key: CASSANDRA-14764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14764
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Legacy/Streaming and Messaging
>Reporter: Joey Lynch
>Assignee: Vinay Chella
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: i-03341e1c52de6ea3e-after-queue-change.svg, 
> i-07cd92e844d66d801-after-queue-bound.svg, i-07cd92e844d66d801-hint-play.svg, 
> i-07cd92e844d66d801-uninlined-with-jvm-methods.svg, ttop.txt
>
>
> *Setup:*
>  * Cassandra: 12 (2*6) node i3.xlarge AWS instance (4 cpu cores, 30GB ram) 
> running cassandra trunk off of jasobrown/14503 jdd7ec5a2 (Jasons patched 
> internode messaging branch) vs the same footprint running 3.0.17
>  * Two datacenters with 100ms latency between them
>  * No compression, encryption, or coalescing turned on
> *Test #1:*
> ndbench sent 1.5k QPS at a coordinator level to one datacenter (RF=3*2 = 6 so 
> 3k global replica QPS) of 4kb single partition BATCH mutations at LOCAL_ONE. 
> This represents about 250 QPS per coordinator in the first datacenter or 60 
> QPS per core. The goal was to observe P99 write and read latencies under 
> various QPS.
> *Result:*
> The good news is since the CASSANDRA-14503 changes, instead of keeping the 
> mutations on heap we put the message into hints instead and don't run out of 
> memory. The bad news is that the {{MessagingService-NettyOutbound-Thread's}} 
> would occasionally enter a degraded state where they would just spin on a 
> core. I've attached flame graphs showing the CPU state as [~jasobrown] 
> applied fixes to the {{OutboundMessagingConnection}} class.
>  *Follow Ups:*
> [~jasobrown] has committed a number of fixes onto his 
> {{jasobrown/14503-collab}} branch including:
> 1. Limiting the amount of time spent dequeuing messages if they are expired 
> (previously if messages entered the queue faster than we could dequeue them 
> we'd just inifinte loop on the consumer side)
> 2. Don't call {{dequeueMessages}} from within {{dequeueMessages}} created 
> callbacks.
> We're continuing to use CPU flamegraphs to figure out where we're looping and 
> fixing bugs as we find them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14761) Rename speculative_retry to match additional_write_policy



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14761:
--
Fix Version/s: (was: 4.0)
   4.0-alpha

> Rename speculative_retry to match additional_write_policy
> -
>
> Key: CASSANDRA-14761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14761
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/Core
>Reporter: Ariel Weisberg
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> It's not really speculative. This commit is where it was last named and shows 
> what to update 
> https://github.com/aweisberg/cassandra/commit/e1df8e977d942a1b0da7c2a7554149c781d0e6c3



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15806) C* 4.0 is missing a way to identify transient SSTables on disk



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sequoyha pelletier updated CASSANDRA-15806:
---
Attachment: 15806-4.0.txt

> C* 4.0 is missing a way to identify transient SSTables on disk
> --
>
> Key: CASSANDRA-15806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15806
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tool/sstable
>Reporter: sequoyha pelletier
>Assignee: sequoyha pelletier
>Priority: Normal
> Attachments: 15806-4.0.txt
>
>
> Currently, there is no way to identify SSTables that were created as 
> transient replicated data. Even thought the feature is experimental we should 
> open that up for those that want to experiment. This seems to be an 
> oversight. I have added the missing line of code to the SStableMetadataViewer 
> and will attach a patch shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14888) Several mbeans are not unregistered when dropping a keyspace and table



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14888:
--
Fix Version/s: (was: 4.0-rc)
   (was: 4.0)
   4.0-beta

> Several mbeans are not unregistered when dropping a keyspace and table
> --
>
> Key: CASSANDRA-14888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability/Metrics
>Reporter: Ariel Weisberg
>Assignee: Alex Deparvu
>Priority: Urgent
>  Labels: patch-available
> Fix For: 4.0-beta
>
> Attachments: CASSANDRA-14888.patch
>
>
> CasCommit, CasPrepare, CasPropose, ReadRepairRequests, 
> ShortReadProtectionRequests, AntiCompactionTime, BytesValidated, 
> PartitionsValidated, RepairPrepareTime, RepairSyncTime, 
> RepairedDataInconsistencies, ViewLockAcquireTime, ViewReadTime, 
> WriteFailedIdealCL
> Basically for 3 years people haven't known what they are doing because the 
> entire thing is kind of obscure. Fix it and also add a dtest that detects if 
> any mbeans are left behind after dropping a table and keyspace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15806) C* 4.0 is missing a way to identify transient SSTables on disk



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sequoyha pelletier updated CASSANDRA-15806:
---
 Bug Category: Parent values: Correctness(12982)
   Complexity: Low Hanging Fruit
Discovered By: User Report
Fix Version/s: 4.0
 Severity: Normal
   Status: Open  (was: Triage Needed)

> C* 4.0 is missing a way to identify transient SSTables on disk
> --
>
> Key: CASSANDRA-15806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15806
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tool/sstable
>Reporter: sequoyha pelletier
>Assignee: sequoyha pelletier
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15806-4.0.txt
>
>
> Currently, there is no way to identify SSTables that were created as 
> transient replicated data. Even thought the feature is experimental we should 
> open that up for those that want to experiment. This seems to be an 
> oversight. I have added the missing line of code to the SStableMetadataViewer 
> and will attach a patch shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14747) Test Messaging Refactor with: 200 node, compression=none, encryption=none, coalescing=off



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14747:
--
Fix Version/s: (was: 4.0-rc)
   4.0-beta

> Test Messaging Refactor with: 200 node, compression=none, encryption=none, 
> coalescing=off 
> --
>
> Key: CASSANDRA-14747
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14747
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Legacy/Testing
>Reporter: Joey Lynch
>Assignee: Joey Lynch
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: 3.0.17-QPS.png, 4.0.1-QPS.png, 
> 4.0.11-after-jolynch-tweaks.svg, 4.0.12-after-unconditional-flush.svg, 
> 4.0.15-after-sndbuf-fix.svg, 4.0.7-before-my-changes.svg, 
> 4.0_errors_showing_heap_pressure.txt, 
> 4.0_heap_histogram_showing_many_MessageOuts.txt, 
> i-0ed2acd2dfacab7c1-after-looping-fixes.svg, 
> trunk_14503_v2_cpuflamegraph.svg, trunk_vs_3.0.17_latency_under_load.png, 
> ttop_NettyOutbound-Thread_spinning.txt, 
> useast1c-i-0e1ddfe8b2f769060-mutation-flame.svg, 
> useast1e-i-08635fa1631601538_flamegraph_96node.svg, 
> useast1e-i-08635fa1631601538_ttop_netty_outbound_threads_96nodes, 
> useast1e-i-08635fa1631601538_uninlinedcpuflamegraph.0_96node_60sec_profile.svg
>
>
> Tracks evaluating a 200 node cluster with all internode settings off (no 
> compression, no encryption, no coalescing).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15806) C* 4.0 is missing a way to identify transient SSTables on disk



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sequoyha pelletier updated CASSANDRA-15806:
---
Test and Documentation Plan: 
Tested agains sstables to make sure it worked:
{noformat}
sequoyha.pelletier@spelletier-rmbp15:[~/cassandra-trunk/tools/bin]
510-Mon May 11 15:16
-->$ ./sstablemetadata 
../../data/data/daintree/products-2080916511eab425f9ae993c7b2c/na-7-big-Statistics.db
 
SSTable: 
/Users/sequoyha.pelletier/cassandra-trunk/data/data/daintree/products-2080916511eab425f9ae993c7b2c/na-7-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.01
Minimum timestamp: 1588980982231688 (05/08/2020 16:36:22)
Maximum timestamp: 1588982281330936 (05/08/2020 16:58:01)
SSTable min local deletion time: 2147483647 (no tombstones)
SSTable max local deletion time: 2147483647 (no tombstones)
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.6960054067248149
TTL min: 0
TTL max: 0
First token: -9223366104813810737 (cac75b42-a598-4d11-b497-80ffd39da575)
Last token: 9223335123199673029 (21dd4443-55b8-4592-a3f5-d76e9e35014e)
minClusteringValues: []
maxClusteringValues: []
Estimated droppable tombstones: 0.0
SSTable Level: 0
Repaired at: 0
Pending repair: --
Replay positions covered: {CommitLogPosition(segmentId=1588977222779, 
position=494)=CommitLogPosition(segmentId=1588977222788, position=10467779), 
CommitLogPosition(segmentId=1589235043721, 
position=494)=CommitLogPosition(segmentId=1589235043721, position=494)}
totalColumnsSet: 5242880
totalRows: 1048576
Estimated tombstone drop times: 
   Drop Time | Count  (%)  Histogram 
   Percentiles
   50th      0 
   75th      0 
   95th      0 
   98th      0 
   99th      0 
   Min       0 
   Max       0 
Partition Size: 
   Size (bytes) | Count   (%)  Histogram 
   103 (103 B)  |  27198 (  2) O
   124 (124 B)  | 663446 ( 63) OO
   149 (149 B)  | 347860 ( 33) OOO.
   179 (179 B)  |  10061 (  0) .
   215 (215 B)  |     11 (  0) 
   Percentiles
   50th      124 (124 B)
   75th      149 (149 B)
   95th      149 (149 B)
   98th      149 (149 B)
   99th      149 (149 B)
   Min       87 (87 B)
   Max       215 (215 B)
Column Count: 
   Columns | Count    (%)  Histogram 
   5       | 1048576 (100) OO
   Percentiles
   50th      5
   75th      5
   95th      5
   98th      5
   99th      5
   Min       5
   Max       5
Estimated cardinality: 1029815
EncodingStats minTTL: 0
EncodingStats minLocalDeletionTime: 144288 (09/21/2015 17:00:00)
EncodingStats minTimestamp: 1588980982231688 (05/08/2020 16:36:22)
KeyType: org.apache.cassandra.db.marshal.UUIDType
ClusteringTypes: []
StaticColumns: 
RegularColumns: author:org.apache.cassandra.db.marshal.UTF8Type, 
isbn:org.apache.cassandra.db.marshal.UTF8Type, 
publisher:org.apache.cassandra.db.marshal.UTF8Type, 
category:org.apache.cassandra.db.marshal.UTF8Type, 
product_name:org.apache.cassandra.db.marshal.UTF8Type
Is Transient: false{noformat}
 Status: Patch Available  (was: Open)

> C* 4.0 is missing a way to identify transient SSTables on disk
> --
>
> Key: CASSANDRA-15806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15806
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tool/sstable
>Reporter: sequoyha pelletier
>Assignee: sequoyha pelletier
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15806-4.0.txt
>
>
> Currently, there is no way to identify SSTables that were created as 
> transient replicated data. Even thought the feature is experimental we should 
> open that up for those that want to experiment. This seems to be an 
> oversight. I have added the missing line of code to the SStableMetadataViewer 
> and will attach a patch shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15806) C* 4.0 is missing a way to identify transient/non-transient SSTables on disk



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sequoyha pelletier updated CASSANDRA-15806:
---
Summary: C* 4.0 is missing a way to identify transient/non-transient 
SSTables on disk  (was: C* 4.0 is missing a way to identify transient SSTables 
on disk)

> C* 4.0 is missing a way to identify transient/non-transient SSTables on disk
> 
>
> Key: CASSANDRA-15806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15806
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tool/sstable
>Reporter: sequoyha pelletier
>Assignee: sequoyha pelletier
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15806-4.0.txt
>
>
> Currently, there is no way to identify SSTables that were created as 
> transient replicated data. Even thought the feature is experimental we should 
> open that up for those that want to experiment. This seems to be an 
> oversight. I have added the missing line of code to the SStableMetadataViewer 
> and will attach a patch shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14688) Update protocol spec and class level doc with protocol checksumming details



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-14688:
--
Fix Version/s: (was: 4.0-beta)
   (was: 4.0)
   4.0-rc

> Update protocol spec and class level doc with protocol checksumming details
> ---
>
> Key: CASSANDRA-14688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14688
> Project: Cassandra
>  Issue Type: Task
>  Components: Legacy/Documentation and Website
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-rc
>
>
> CASSANDRA-13304 provides an option to add checksumming to the frame body of 
> native protocol messages. The native protocol spec needs to be updated to 
> reflect this ASAP. We should also verify that the javadoc comments describing 
> the on-wire format in 
> {{o.a.c.transport.frame.checksum.ChecksummingTransformer}} are up to date.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15806) C* 4.0 is missing a way to identify transient/non-transient SSTables on disk



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sequoyha pelletier updated CASSANDRA-15806:
---
Test and Documentation Plan: 
Ran a simple test against sstable to make sure it worked:
{noformat}
sequoyha.pelletier@spelletier-rmbp15:[~/cassandra-trunk/tools/bin]
510-Mon May 11 15:16

 LocalDeletionTime: 144288 (09/21/2015 17:00:00)
EncodingStats minTimestamp: 1588980982231688 (05/08/2020 16:36:22)
...
RegularColumns: author:org.apache.cassandra.db.marshal.UTF8Type, 
isbn:org.apache.cassandra.db.marshal.UTF8Type, 
publisher:org.apache.cassandra.db.marshal.UTF8Type, 
category:org.apache.cassandra.db.marshal.UTF8Type, 
product_name:org.apache.cassandra.db.marshal.UTF8Type
Is Transient: false{noformat}

The isTransient is a boolean value so false is enough to verify the information 
is showing.

  was:
Tested agains sstables to make sure it worked:
{noformat}
sequoyha.pelletier@spelletier-rmbp15:[~/cassandra-trunk/tools/bin]
510-Mon May 11 15:16
-->$ ./sstablemetadata 
../../data/data/daintree/products-2080916511eab425f9ae993c7b2c/na-7-big-Statistics.db
 
SSTable: 
/Users/sequoyha.pelletier/cassandra-trunk/data/data/daintree/products-2080916511eab425f9ae993c7b2c/na-7-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.01
Minimum timestamp: 1588980982231688 (05/08/2020 16:36:22)
Maximum timestamp: 1588982281330936 (05/08/2020 16:58:01)
SSTable min local deletion time: 2147483647 (no tombstones)
SSTable max local deletion time: 2147483647 (no tombstones)
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.6960054067248149
TTL min: 0
TTL max: 0
First token: -9223366104813810737 (cac75b42-a598-4d11-b497-80ffd39da575)
Last token: 9223335123199673029 (21dd4443-55b8-4592-a3f5-d76e9e35014e)
minClusteringValues: []
maxClusteringValues: []
Estimated droppable tombstones: 0.0
SSTable Level: 0
Repaired at: 0
Pending repair: --
Replay positions covered: {CommitLogPosition(segmentId=1588977222779, 
position=494)=CommitLogPosition(segmentId=1588977222788, position=10467779), 
CommitLogPosition(segmentId=1589235043721, 
position=494)=CommitLogPosition(segmentId=1589235043721, position=494)}
totalColumnsSet: 5242880
totalRows: 1048576
Estimated tombstone drop times: 
   Drop Time | Count  (%)  Histogram 
   Percentiles
   50th      0 
   75th      0 
   95th      0 
   98th      0 
   99th      0 
   Min       0 
   Max       0 
Partition Size: 
   Size (bytes) | Count   (%)  Histogram 
   103 (103 B)  |  27198 (  2) O
   124 (124 B)  | 663446 ( 63) OO
   149 (149 B)  | 347860 ( 33) OOO.
   179 (179 B)  |  10061 (  0) .
   215 (215 B)  |     11 (  0) 
   Percentiles
   50th      124 (124 B)
   75th      149 (149 B)
   95th      149 (149 B)
   98th      149 (149 B)
   99th      149 (149 B)
   Min       87 (87 B)
   Max       215 (215 B)
Column Count: 
   Columns | Count    (%)  Histogram 
   5       | 1048576 (100) OO
   Percentiles
   50th      5
   75th      5
   95th      5
   98th      5
   99th      5
   Min       5
   Max       5
Estimated cardinality: 1029815
EncodingStats minTTL: 0
EncodingStats minLocalDeletionTime: 144288 (09/21/2015 17:00:00)
EncodingStats minTimestamp: 1588980982231688 (05/08/2020 16:36:22)
KeyType: org.apache.cassandra.db.marshal.UUIDType
ClusteringTypes: []
StaticColumns: 
RegularColumns: author:org.apache.cassandra.db.marshal.UTF8Type, 
isbn:org.apache.cassandra.db.marshal.UTF8Type, 
publisher:org.apache.cassandra.db.marshal.UTF8Type, 
category:org.apache.cassandra.db.marshal.UTF8Type, 
product_name:org.apache.cassandra.db.marshal.UTF8Type
Is Transient: false{noformat}


> C* 4.0 is missing a way to identify transient/non-transient SSTables on disk
> 
>
> Key: CASSANDRA-15806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15806
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tool/sstable
>Reporter: sequoyha pelletier
>Assignee: sequoyha pelletier
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15806-4.0.txt
>
>
> Currently, there is no way to identify SSTables that were created as 
> transient replicated data. Even thought the feature is experimental we should 
> open that up for those that want to experiment. This seems to be an 
> oversight. I have added the missing line of code to the SStableMetadataViewer 
> and will attach a patch shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15762) Update Bundled Python Driver to the latest release



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15762:
--
Fix Version/s: (was: 4.0-rc)
   4.0-beta

> Update Bundled Python Driver to the latest release
> --
>
> Key: CASSANDRA-15762
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15762
> Project: Cassandra
>  Issue Type: Task
>  Components: Tool/cqlsh
>Reporter: Alan Boudreault
>Priority: Normal
> Fix For: 4.0-beta
>
>
> This is not urgent for now since there are some missing 4.x features that 
> need to be implemented on the driver (checksum). This ticket is to make sure 
> we update the bundled driver before GA.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-15234) Standardise config and JVM parameters

2020-05-12 Thread Ekaterina Dimitrova (Jira)

[
https://issues.apache.org/jira/browse/CASSANDRA-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086118#comment-17086118
]

Ekaterina Dimitrova edited comment on CASSANDRA-15234 at 5/12/20, 4:23 PM:
---

Done but with one caveat which was caught thankfully with the DTests. Unit
tests were even on 100% green at some runs. (And my unit test was not covering
the problem conversion explained below, I should add it!)
I am new to snakeyaml so I might be wrong, I would appreciate if someone who
has the experience proves me wrong.
>From the DTest failures I realized that turning into Strings the values from
>the YAML we hit one issue. To me it looks like we already have it actually
>with Integer and String parameters, just they are not many so it didn't bite
>us up to now. Not sure whether this was considered when the Config class was
>built or not.

So the issue with int parameters in Config in my solution is:
Currently in trunk, when we parse to int variable, If a parameter is commented
in the cassandra.yaml, this is observed as null and it will raise an exception
in some cases. (i.e. commitlog_sync_period_in_ms) But if it is not commented,
just not assigned a value, then this is considered as 0 (as we parse to int).
The situation is different with String, Integer. They will be in both cases
null. No differentiator. But currently we don't have many parameters being
Integer and String and this wasn't a problem.
Now with the units attached to the parameters' values in the cassandra.yaml
file, we need to parse strings with snakeyaml and then convert to the proper
int values in Config.

A way to cope with this issue is to make the users use "". In this way, if we
have a parameter not commented, but not also having an assigned value in
cassandra.yaml, we can identify it by checking a string for being empty. If the
parameter is commented or not presented, it will be null. But this would be one
more new thing for the users to consider. That's why I came back to snakeyaml
again and tried to think of a different way utilizing it. Unfortunately, up to
now I didn't find any different solution. Also, the links to the examples in
the official snakeyaml documentation are broken. Not sure whether there they
had some interesting stuff. Maybe something with customized constructor but
also, I try to make as less as possible disruptive changes in the codebase in
order not to mess up something with the Config which might be extremely
unpleasant.

I will be happy to hear an experienced opinion here. Does anyone know a better
solution? Or are we ok to introduce "" in the yaml for empty
parameters/Strings recognition?

[Part1|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-2]
(This part was already revised by [~dcapwell] but I paste the explanation again
here for completeness).
- New cassandra.yaml ---> currently cassandra-new.yaml in the branch as I
wanted to test the backward compatibility on the old version (will have to
rename properly also these yaml files with version probably)
reorganization of the file renames to make the names more consistent
- Backward compatibility with the old names - my approach is:
Cassandra comes with the new version of the yaml to ensure any new builds will
be using it
During load of the yaml file I check whether old names are identified(means
someone upgraded to 4 and wants to still use the old yaml). If this is the
case, we load the values from the old yaml to the new variables (the already
renamed parameters). Also, there is a warning to the user that there is new
cassandra.yaml version and he/she can consider an update.

[Part2|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3]
- units added to the values in the cassandra.yaml file. Parsing and unit
conversion in place. One unit test added, should be further developed.

[DTests|https://github.com/ekaterinadimitrova2/cassandra-dtest] updated
accordingly for v.4

[Part3|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-5]
- SysProperties class, accessors, hope I didn't miss anything... most of them
are actually just using System.getProperty method.
TO DO: Extract the parser/converter in a separate class from Config

[~mck]? [~dcapwell]? [~benedict]? Anyone else any thoughts here?

NOTE: The branches are not fully cleaned, this is still work in progress but we
need to take a decision on the String issue first, I think.

was (Author: e.dimitrova):
Done but with one caveat which was caught thankfully with the DTests. Unit
tests were even on 100% green at some runs. (And my unit test was not covering
the problem conversion explained below, I should add it!)
I am new to snakeyaml so I might be wrong, I would appreciate if someone who
has the experience proves me wrong.
>From the DTest failures I realized that tur

[jira] [Updated] (CASSANDRA-13606) Improve handling of 2i initialization failures



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-13606:
--
Fix Version/s: (was: 4.0)
   4.0-beta

> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures 
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be 
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic 
> could be more complex than just an index build, hence it shouldn't be made 
> recoverable via a simple rebuild, and could cause the index to be fully 
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads, 
> writes, or both. 
> * Providing a proper method to recover, distinct from index rebuilds.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-8272) 2ndary indexes can return stale data



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-8272:
-
Fix Version/s: (was: 4.0)
   4.0-beta

> 2ndary indexes can return stale data
> 
>
> Key: CASSANDRA-8272
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8272
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/2i Index
>Reporter: Sylvain Lebresne
>Assignee: Andres de la Peña
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 3.0.x, 4.0-beta
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When replica return 2ndary index results, it's possible for a single replica 
> to return a stale result and that result will be sent back to the user, 
> potentially failing the CL contract.
> For instance, consider 3 replicas A, B and C, and the following situation:
> {noformat}
> CREATE TABLE test (k int PRIMARY KEY, v text);
> CREATE INDEX ON test(v);
> INSERT INTO test(k, v) VALUES (0, 'foo');
> {noformat}
> with every replica up to date. Now, suppose that the following queries are 
> done at {{QUORUM}}:
> {noformat}
> UPDATE test SET v = 'bar' WHERE k = 0;
> SELECT * FROM test WHERE v = 'foo';
> {noformat}
> then, if A and B acknowledge the insert but C respond to the read before 
> having applied the insert, then the now stale result will be returned (since 
> C will return it and A or B will return nothing).
> A potential solution would be that when we read a tombstone in the index (and 
> provided we make the index inherit the gcGrace of it's parent CF), instead of 
> skipping that tombstone, we'd insert in the result a corresponding range 
> tombstone.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15772) DOC - Improve tpstats documentation



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15772:
--
Fix Version/s: (was: 4.0-alpha)
   4.0

> DOC - Improve tpstats documentation
> ---
>
> Key: CASSANDRA-15772
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15772
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation/Website
>Reporter: Mark Curtis
>Priority: Normal
> Fix For: 4.0
>
>
> h2. Background
> The docs for {{nodetool tpstats}} gives the user a good overview of the 
> correct command format, but lack an explanation of the message types.
> As of Cassandra4.0 and CASSANDRA-15066 the Message types have expanded a lot 
> so this becomes quite overwhelming (particularly for new users).
> I'd propose we offer a simple explanation of these grouping them into logical 
> groups (see CASSANDRA-15771 which I opened for the formatting output). 
> The github code for the {{Verb.java}} class does give the user some idea of 
> what these are but not everyone will go looking for that.
> h2. Example
> Code for ref
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/Verb.java]
> I tried to sort some of the tpstats output into categories and made a first 
> attempt at an explanation here:
> {code:java}
> BatchesBATCH_REMOVE_REQ 
> BatchesBATCH_REMOVE_RSP 
> BatchesBATCH_STORE_REQ  
> BatchesBATCH_STORE_RSP
> - These are batches, I'd assume the store / removes are as we add and process 
> them?
>  
> Counters   COUNTER_MUTATION_REQ 
> Counters   COUNTER_MUTATION_RSP 
> - Counter writes
> Deprecated INTERNAL_RSP  
> Deprecated REQUEST_RSP   
> - Some of these are shown as deprecated in the code. Not sure what we should 
> add here
>  
> Gossip ECHO_REQ 
> Gossip ECHO_RSP 
> Gossip GOSSIP_DIGEST_ACK
> Gossip GOSSIP_DIGEST_ACK2   
> Gossip GOSSIP_DIGEST_SYN
> Gossip GOSSIP_SHUTDOWN  
> Gossip PING_REQ 
> Gossip PING_RSP
> - Gossip, probably we'd refer to a more detailed page on gossip from here
>   
> Hints  HINT_REQ 
> Hints  HINT_RSP
> - Hints!  
>   
> LWTs   PAXOS_COMMIT_REQ 
> LWTs   PAXOS_COMMIT_RSP 
> LWTs   PAXOS_PREPARE_REQ
> LWTs   PAXOS_PREPARE_RSP
> LWTs   PAXOS_PROPOSE_REQ
> LWTs   PAXOS_PROPOSE_RSP
> - Paxos phases request / response
> 
> Migration  SCHEMA_PULL_REQ  
> Migration  SCHEMA_PULL_RSP  
> Migration  SCHEMA_PUSH_REQ  
> Migration  SCHEMA_PUSH_RSP  
> Migration  SCHEMA_VERSION_REQ   
> Migration  SCHEMA_VERSION_RSP  
> - Schema migration messaging (i.e. adding new tables)
> Misc   REPLICATION_DONE_REQ 
> Misc   REPLICATION_DONE_RSP 
> Misc   SNAPSHOT_MSG 
> Misc   SNAPSHOT_REQ 
> Misc   SNAPSHOT_RSP
> - Unsure of these, could the snapshots be for simply nodetool snapshot 
> requests like on truncates? 
> 
> Reads  RANGE_REQ
> Reads  RANGE_RSP
> Reads  READ_REPAIR_REQ  
> Reads  READ_REPAIR_RSP  
> Reads  READ_REQ 
> Reads  READ_RSP 
> - All types of read messages 
>
> Repair CLEANUP_MSG  
> Repair FAILED_SESSION_MSG   
> Repair FAILURE_RSP  
> Repair FINALIZE_COMMIT_MSG  
> Repair FINALIZE_PROMISE_MSG 
> Repair FINALIZE_PROPOSE_MSG 
> Repair PREPARE_CONSISTENT_REQ   
> Repair PREPARE_CONSISTENT_RSP   
> Repair PREPARE_MSG  
> Repair REPAIR_RSP   
> Repair STATUS_REQ   
> Repair STATUS_RSP   
> Repair SYNC_REQ 
> Repair SYNC_RSP 
> Repair VALIDATION_REQ   
> Repair VALIDATION_RSP   
> Repair ASYMMETRIC_SYNC_REQ 
> - Repairs, and phases, I think we'd need a bit more explanation here. 
> Validation, Sync etc ok. But unsure on these other types
>  
> Writes MUTATION_REQ 
> Writes MUTATION_RSP 
> Writes TRUNCATE_REQ 
> Writes TRUNCATE_RSP 
> - All types of write messages{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15772) DOC - Improve tpstats documentation

2020-05-12 Thread Mark Curtis (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105574#comment-17105574
 ] 

Mark Curtis commented on CASSANDRA-15772:
-

[~jmeredithco] not at all, it shouldn't stop the beta. Cheers.

> DOC - Improve tpstats documentation
> ---
>
> Key: CASSANDRA-15772
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15772
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation/Website
>Reporter: Mark Curtis
>Priority: Normal
> Fix For: 4.0
>
>
> h2. Background
> The docs for {{nodetool tpstats}} gives the user a good overview of the 
> correct command format, but lack an explanation of the message types.
> As of Cassandra4.0 and CASSANDRA-15066 the Message types have expanded a lot 
> so this becomes quite overwhelming (particularly for new users).
> I'd propose we offer a simple explanation of these grouping them into logical 
> groups (see CASSANDRA-15771 which I opened for the formatting output). 
> The github code for the {{Verb.java}} class does give the user some idea of 
> what these are but not everyone will go looking for that.
> h2. Example
> Code for ref
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/Verb.java]
> I tried to sort some of the tpstats output into categories and made a first 
> attempt at an explanation here:
> {code:java}
> BatchesBATCH_REMOVE_REQ 
> BatchesBATCH_REMOVE_RSP 
> BatchesBATCH_STORE_REQ  
> BatchesBATCH_STORE_RSP
> - These are batches, I'd assume the store / removes are as we add and process 
> them?
>  
> Counters   COUNTER_MUTATION_REQ 
> Counters   COUNTER_MUTATION_RSP 
> - Counter writes
> Deprecated INTERNAL_RSP  
> Deprecated REQUEST_RSP   
> - Some of these are shown as deprecated in the code. Not sure what we should 
> add here
>  
> Gossip ECHO_REQ 
> Gossip ECHO_RSP 
> Gossip GOSSIP_DIGEST_ACK
> Gossip GOSSIP_DIGEST_ACK2   
> Gossip GOSSIP_DIGEST_SYN
> Gossip GOSSIP_SHUTDOWN  
> Gossip PING_REQ 
> Gossip PING_RSP
> - Gossip, probably we'd refer to a more detailed page on gossip from here
>   
> Hints  HINT_REQ 
> Hints  HINT_RSP
> - Hints!  
>   
> LWTs   PAXOS_COMMIT_REQ 
> LWTs   PAXOS_COMMIT_RSP 
> LWTs   PAXOS_PREPARE_REQ
> LWTs   PAXOS_PREPARE_RSP
> LWTs   PAXOS_PROPOSE_REQ
> LWTs   PAXOS_PROPOSE_RSP
> - Paxos phases request / response
> 
> Migration  SCHEMA_PULL_REQ  
> Migration  SCHEMA_PULL_RSP  
> Migration  SCHEMA_PUSH_REQ  
> Migration  SCHEMA_PUSH_RSP  
> Migration  SCHEMA_VERSION_REQ   
> Migration  SCHEMA_VERSION_RSP  
> - Schema migration messaging (i.e. adding new tables)
> Misc   REPLICATION_DONE_REQ 
> Misc   REPLICATION_DONE_RSP 
> Misc   SNAPSHOT_MSG 
> Misc   SNAPSHOT_REQ 
> Misc   SNAPSHOT_RSP
> - Unsure of these, could the snapshots be for simply nodetool snapshot 
> requests like on truncates? 
> 
> Reads  RANGE_REQ
> Reads  RANGE_RSP
> Reads  READ_REPAIR_REQ  
> Reads  READ_REPAIR_RSP  
> Reads  READ_REQ 
> Reads  READ_RSP 
> - All types of read messages 
>
> Repair CLEANUP_MSG  
> Repair FAILED_SESSION_MSG   
> Repair FAILURE_RSP  
> Repair FINALIZE_COMMIT_MSG  
> Repair FINALIZE_PROMISE_MSG 
> Repair FINALIZE_PROPOSE_MSG 
> Repair PREPARE_CONSISTENT_REQ   
> Repair PREPARE_CONSISTENT_RSP   
> Repair PREPARE_MSG  
> Repair REPAIR_RSP   
> Repair STATUS_REQ   
> Repair STATUS_RSP   
> Repair SYNC_REQ 
> Repair SYNC_RSP 
> Repair VALIDATION_REQ   
> Repair VALIDATION_RSP   
> Repair ASYMMETRIC_SYNC_REQ 
> - Repairs, and phases, I think we'd need a bit more explanation here. 
> Validation, Sync etc ok. But unsure on these other types
>  
> Writes MUTATION_REQ 
> Writes MUTATION_RSP 
> Writes TRUNCATE_REQ 
> Writes TRUNCATE_RSP 
> - All types of write messages{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15234) Standardise config and JVM parameters



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh McKenzie updated CASSANDRA-15234:
--
Fix Version/s: (was: 4.0-beta)
   (was: 4.0)
   4.0-alpha

> Standardise config and JVM parameters
> -
>
> Key: CASSANDRA-15234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15234
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Config
>Reporter: Benedict Elliott Smith
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> We have a bunch of inconsistent names and config patterns in the codebase, 
> both from the yams and JVM properties.  It would be nice to standardise the 
> naming (such as otc_ vs internode_) as well as the provision of values with 
> units - while maintaining perpetual backwards compatibility with the old 
> parameter names, of course.
> For temporal units, I would propose parsing strings with suffixes of:
> {{code}}
> u|micros(econds?)?
> ms|millis(econds?)?
> s(econds?)?
> m(inutes?)?
> h(ours?)?
> d(ays?)?
> mo(nths?)?
> {{code}}
> For rate units, I would propose parsing any of the standard {{B/s, KiB/s, 
> MiB/s, GiB/s, TiB/s}}.
> Perhaps for avoiding ambiguity we could not accept bauds {{bs, Mbps}} or 
> powers of 1000 such as {{KB/s}}, given these are regularly used for either 
> their old or new definition e.g. {{KiB/s}}, or we could support them and 
> simply log the value in bytes/s.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads

2020-05-12 Thread Alex Petrov (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105653#comment-17105653
 ] 

Alex Petrov commented on CASSANDRA-15778:
-

I've taken a short look and wanted to precaution against ignoring or even 
purging values from the hidden compact column, since there are scenarios when 
it can contain legitimate data. 

I've been able to reproduce an issue with a similar, albeit not exactly same 
stacktrace (leaving only the part from sstable iterator downward):

{code}
ERROR 18:08:11 Uncaught exception on thread Thread[SharedPool-Worker-2,10,main]
java.lang.RuntimeException: org.apache.cassandra.serializers.MarshalException: 
EmptyType only accept empty values
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_171]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
 ~[main/:na]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
 [main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[main/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.apache.cassandra.serializers.MarshalException: EmptyType only 
accept empty values
at 
org.apache.cassandra.serializers.EmptySerializer.validate(EmptySerializer.java:42)
 ~[main/:na]
at 
org.apache.cassandra.db.marshal.AbstractType.validate(AbstractType.java:159) 
~[main/:na]
at 
org.apache.cassandra.db.marshal.AbstractType.validateIfFixedSize(AbstractType.java:390)
 ~[main/:na]
at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1457)
 ~[main/:na]
at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1377)
 ~[main/:na]
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:542)
 ~[main/:na]
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:519)
 ~[main/:na]
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:336)
 ~[main/:na]
at 
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:140)
 ~[main/:na]
at 
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172)
 ~[main/:na]
at 
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336)
 ~
{code}

In short, this is a table created on 2.2 side with: {{CREATE TABLE table_0 (key 
text, value text, PRIMARY KEY (key, value)) WITH COMPACT STORAGE;}}

If you write data into this table with thrift ({{ThriftClient#batch_mutate}}), 
you will end up with data in the hidden column, which looks like this on the 
2.1 side:

{code}
select * from ks1.table_0;

 key   | value
---+---
 key22 |  key1
 key22 |  key4
 key11 |  key1
 key11 |  key4

(4 rows)
{code}

And like this in sstabledump:

{code}
[
{"key": "key22",
 "cells": [["key1","76616c756531",1589306163593],
   ["key4","76616c756534",1589306163594]]},
{"key": "key11",
 "cells": [["key1","76616c756531",1589306163593],
   ["key4","76616c756534",1589306163594]]}
]
{code}

Of course, on 3.x side you wouldn't be able to see the values.

One theory is that stack trace that Sumanth has posted above could have been a 
result of a read-repair, which has propagated the value to 3.0 sstable, since 
normally a 3.0 node wouldn't even start with such table. I'll continue 
tomorrow, will create a mixed mode cluster, and will see if a read-repair 
propagates this write and corrupts 3.0 sstable. 

Not saying that this all couldn't have been a result of some weird corruption 
resulting from anything else, but primary point of writing this is not to get 
to the bottom of the issue but caution against changing behaviour of 
{{EmptyType}} at least until we know what is going on. 

> CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
> -
>
> Key: CASSANDRA-15778
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15778
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction, Local/SSTable
>Reporter: Sumanth Pasupuleti
>Assignee: David Capwell
>Priority: Normal
> Fix For: 3.0.x
>
>
> Below is the exception with stack trace. This is

[jira] [Commented] (CASSANDRA-15375) Remove BackPressureStrategy



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105672#comment-17105672
 ] 

Jordan West commented on CASSANDRA-15375:
-

Sorry to come in late on this when there is already a patch but shouldn't we 
deprecate {{RateBasedBackPressure}} before removing it? I understand folks here 
haven't seen it in the wild but that doesn't mean someone isn't using it. 

> Remove BackPressureStrategy
> ---
>
> Key: CASSANDRA-15375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15375
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Observability/Logging
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Low
> Fix For: 4.0
>
>
> This is odd:
> {{INFO [main] 2019-10-25 10:33:07,985 DatabaseDescriptor.java:803 - 
> Back-pressure is disabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}
> When I saw that, I wasn't sure if back pressure was actually disabled, or if 
> I was really using {{RateBasedBackPressure.}}
> This should change to output either:
> {{Back-pressure is disabled}}
> {{or}}
> {{Back-pressure is enabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}{{}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15211) Remove BaseIterator.stopChild



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jordan West updated CASSANDRA-15211:

Fix Version/s: 4.0-beta

> Remove BaseIterator.stopChild
> -
>
> Key: CASSANDRA-15211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15211
> Project: Cassandra
>  Issue Type: Task
>  Components: CQL/Interpreter
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 3.0.x, 3.11.x, 4.0-beta
>
>
> This concept was introduced in CASSANDRA-13482, and while it logically 
> resolves the bug, the “more correct” solution is to attach the functions of 
> the extending iterator to the iterator they are being absorbed into.  
> CASSANDRA-13482 introduced a slight error in its evaluation of exit 
> conditions that cropped up in CASSANDRA-14812, that could be avoided with 
> this formulation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15566) Repair coordinator can hang under some cases



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-15566:
--
Fix Version/s: (was: 4.0-beta)
   4.x

> Repair coordinator can hang under some cases
> 
>
> Key: CASSANDRA-15566
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15566
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Consistency/Repair
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.x
>
>
> Repair coordination makes a few assumptions about message delivery which 
> cause it to hang forever when those assumptions don’t hold true: fire and 
> forget will not get rejected (participate has an issue and rejects the 
> message), and a very delayed message will one day be seen (messaging can be 
> dropped under load or when failure detector thinks a node is bad but is just 
> GCing).
> Given this and the desire to have better observability with repair (see 
> CASSANDRA-15399), coordination should be changed into a request/response 
> pattern (with retries) and polling (validation status and MerkleTree 
> sending).  This would allow the coordinator to detect changes in state (it 
> was known participate was working on validation, but it no longer knows about 
> the validation task), and to be able to recover from ephemeral issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15566) Repair coordinator can hang under some cases



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105683#comment-17105683
 ] 

David Capwell commented on CASSANDRA-15566:
---

Moved out of 4.0 since these are not regressions but improvements, and these 
improvements can be done in a minor release.  The 4.0 timeline should flesh out 
a plan for how to fix and validate these changes, but should not be made in the 
4.0 timeline.

> Repair coordinator can hang under some cases
> 
>
> Key: CASSANDRA-15566
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15566
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Consistency/Repair
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.x
>
>
> Repair coordination makes a few assumptions about message delivery which 
> cause it to hang forever when those assumptions don’t hold true: fire and 
> forget will not get rejected (participate has an issue and rejects the 
> message), and a very delayed message will one day be seen (messaging can be 
> dropped under load or when failure detector thinks a node is bad but is just 
> GCing).
> Given this and the desire to have better observability with repair (see 
> CASSANDRA-15399), coordination should be changed into a request/response 
> pattern (with retries) and polling (validation status and MerkleTree 
> sending).  This would allow the coordinator to detect changes in state (it 
> was known participate was working on validation, but it no longer knows about 
> the validation task), and to be able to recover from ephemeral issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15533) Don't allocate unneeded MergeIterator in OnDiskToken#iterator



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105687#comment-17105687
 ] 

Jordan West commented on CASSANDRA-15533:
-

Since CASSANDRA-15392 is merged and this has existed for a while my preference 
would be to wait until  the subsequent release. The improvement is minor if 
15392 isn't merged and if it is merged only affects the changes if SASI is 
enabled. 

> Don't allocate unneeded MergeIterator in OnDiskToken#iterator 
> --
>
> Key: CASSANDRA-15533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15533
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/SASI
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0, 4.0-rc
>
>
> When reviewing CASSANDRA-15392 it became apparent that the MergeIterator 
> allocated by OnDiskToken#iterator is rarely necessary and so we should avoid 
> allocating one when not needed and skip the MergeIterator pool when needed 
> because its unlikely to be sized correctly. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-15533) Don't allocate unneeded MergeIterator in OnDiskToken#iterator



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105687#comment-17105687
 ] 

Jordan West edited comment on CASSANDRA-15533 at 5/12/20, 7:27 PM:
---

Since CASSANDRA-15392 is not merged and this has existed for a while my 
preference would be to wait until  the subsequent release. The improvement is 
minor if 15392 isn't merged and if it is merged only affects the changes if 
SASI is enabled. 


was (Author: jrwest):
Since CASSANDRA-15392 is merged and this has existed for a while my preference 
would be to wait until  the subsequent release. The improvement is minor if 
15392 isn't merged and if it is merged only affects the changes if SASI is 
enabled. 

> Don't allocate unneeded MergeIterator in OnDiskToken#iterator 
> --
>
> Key: CASSANDRA-15533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15533
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/SASI
>Reporter: Jordan West
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0, 4.0-rc
>
>
> When reviewing CASSANDRA-15392 it became apparent that the MergeIterator 
> allocated by OnDiskToken#iterator is rarely necessary and so we should avoid 
> allocating one when not needed and skip the MergeIterator pool when needed 
> because its unlikely to be sized correctly. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads

2020-05-12 Thread Alex Petrov (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105653#comment-17105653
 ] 

Alex Petrov edited comment on CASSANDRA-15778 at 5/12/20, 7:33 PM:
---

I've taken a short look and wanted to precaution against ignoring or even 
purging values from the hidden compact column, since there are scenarios when 
it can contain legitimate data. 

I've been able to reproduce an issue with a similar, albeit not exactly same 
stacktrace (leaving only the part from sstable iterator downward):

{code}
ERROR 18:08:11 Uncaught exception on thread Thread[SharedPool-Worker-2,10,main]
java.lang.RuntimeException: org.apache.cassandra.serializers.MarshalException: 
EmptyType only accept empty values
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_171]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
 ~[main/:na]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
 [main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[main/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.apache.cassandra.serializers.MarshalException: EmptyType only 
accept empty values
at 
org.apache.cassandra.serializers.EmptySerializer.validate(EmptySerializer.java:42)
 ~[main/:na]
at 
org.apache.cassandra.db.marshal.AbstractType.validate(AbstractType.java:159) 
~[main/:na]
at 
org.apache.cassandra.db.marshal.AbstractType.validateIfFixedSize(AbstractType.java:390)
 ~[main/:na]
at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1457)
 ~[main/:na]
at 
org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1377)
 ~[main/:na]
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:542)
 ~[main/:na]
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:519)
 ~[main/:na]
at 
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:336)
 ~[main/:na]
at 
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:140)
 ~[main/:na]
at 
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172)
 ~[main/:na]
at 
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336)
 ~
{code}

In short, this is a table created on 2.2 side with: {{CREATE TABLE table_0 (key 
text, value text, PRIMARY KEY (key, value)) WITH COMPACT STORAGE;}}

If you write data into this table with thrift ({{ThriftClient#batch_mutate}}), 
you will end up with data in the hidden column, which looks like this on the 
2.1 side:

{code}
select * from ks1.table_0;

 key   | value
---+---
 key22 |  key1
 key22 |  key4
 key11 |  key1
 key11 |  key4

(4 rows)
{code}

And like this in sstabledump:

{code}
[
{"key": "key22",
 "cells": [["key1","76616c756531",1589306163593],
   ["key4","76616c756534",1589306163594]]},
{"key": "key11",
 "cells": [["key1","76616c756531",1589306163593],
   ["key4","76616c756534",1589306163594]]}
]
{code}

Of course, on 3.x side you wouldn't be able to see the values.

UPDATE: Was able to reproduce it: 

{code}
java.lang.RuntimeException: 
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
/Users/ifesdjeen/foss/java/apache-cassandra/data/data/ks1/table_0-05609b80948511eabfa891431c623cd5/md-2-big-Data.db
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_171]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
 ~[main/:na]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
 [main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[main/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
/Users/ifesdjeen/foss/java/apache-cassandra/data/data/ks1/table_0-05609b80948511eabfa891431c623cd5/md-2-big-Data.db
at 
org.apache.cassandra.db.columnit

[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta

2020-05-12 Thread Alexandre Dutra (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105688#comment-17105688
 ] 

Alexandre Dutra commented on CASSANDRA-15299:
-

Hi, as we have progressed on the driver-side implementation of this new 
feature, we feel that this should be an opt-in protocol feature.

Firstly, we think that not all users will be interested in enabling 
checksumming. Making it mandatory may keep them away from v5, potentially 
decreasing its adoption rate.

Secondly, we believe that without proper driver support, protocol v5 will not 
gain momentum. We are strongly committed to providing full support for 
checksumming at least in the Java driver, but other drivers might not be able 
to catch up, both those maintained by DataStax and those that are not, such as 
gocql:

 * Java 3.x, Java 4.x, Python and C# are currently on par with protocol v5 
specs (the latter as of 
[today|https://groups.google.com/a/lists.datastax.com/d/msg/csharp-driver-user/V8hOVUb-84A/RSwauBgjAgAJ]).
 For these drivers, it's a race against time that starts now: they _must_ 
implement protocol checksumming before Cassandra 4.0 GA, or they will lose 
access to all the v5 features implemented so far, especially keyspace-per-query 
– which is a nice usability improvement that many users are waiting for.
 * NodeJS, C++ and PHP could also be ready for Cassandra 4.0 (most protocol v5 
features are already there, only not exposed), but having to implement 
checksumming makes it unlikely that these drivers will be able to catch up.
 * Other DataStax drivers (Ruby) will take twice as much time to implement v5 
if they need to include checksumming; it's unlikely that they would be ready 
this year.
 * gocql: not expected to catch up with protocol checksumming any time soon.

Making checksumming opt-in would allow many drivers to be ready for Cassandra 
4.0 GA, and others to catch up quickly. We feel this would greatly contribute 
to promoting Cassandra 4.0's adoption.

> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol 
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it 
> introduced checksumming/CRC32 to request and response bodies. It’s an 
> important step forward, but it doesn’t cover the entire stream. In 
> particular, the message header is not covered by a checksum or a crc, which 
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of 
> computing the CRC32 on the bytes written on the wire - losing the properties 
> of the CRC32. In some cases, due to this sequencing, attempting to decompress 
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order, 
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for 
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since 
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression 
> operate on individual message bodies rather than frames of multiple complete 
> messages. In reality, this has several important additional downsides. To 
> name a couple:
> # For compression, we are getting poor compression ratios for smaller 
> messages - when operating on tiny sequences of bytes. In reality, for most 
> small requests and responses we are discarding the compressed value as it’d 
> be smaller than the uncompressed one - incurring both redundant allocations 
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages. 
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the 
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we 
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode 
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed, 
> and that we do it before native protocol v5 graduates from beta. See 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/Fram

[jira] [Commented] (CASSANDRA-14939) fix some operational holes in incremental repair

2020-05-12 Thread Sankalp Kohli (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105760#comment-17105760
 ] 

Sankalp Kohli commented on CASSANDRA-14939:
---

I think we should do this as part of 4.0-beta as it is important to users who 
will use IR in 4.0. IR has major changes in 4.0 and we hope lot more users will 
use this in 4.0!!

 

cc [~jwest]

> fix some operational holes in incremental repair
> 
>
> Key: CASSANDRA-14939
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14939
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Incremental repair has a few operational rough spots that make it more 
> difficult to fully automate and operate at scale than it should be.
> * Visibility into whether pending repair data exists for a given token range.
> * Ability to force promotion/demotion of data for completed sessions instead 
> of waiting for compaction.
> * Get the most recent repairedAt timestamp for a given token range.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15375) Remove BackPressureStrategy

2020-05-12 Thread Benedict Elliott Smith (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105768#comment-17105768
 ] 

Benedict Elliott Smith commented on CASSANDRA-15375:


Not everything has to go through a deprecation process, no.  Since this feature 
was never advertised by any means, it cannot be taken to be supported.  Since 
we also don't think it's _useful_, ignoring the property will leave the 
database approximately as functional as it was before.  Such a user, if they 
exist, will still have a working database after upgrade.



> Remove BackPressureStrategy
> ---
>
> Key: CASSANDRA-15375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15375
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Observability/Logging
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Low
> Fix For: 4.0
>
>
> This is odd:
> {{INFO [main] 2019-10-25 10:33:07,985 DatabaseDescriptor.java:803 - 
> Back-pressure is disabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}
> When I saw that, I wasn't sure if back pressure was actually disabled, or if 
> I was really using {{RateBasedBackPressure.}}
> This should change to output either:
> {{Back-pressure is disabled}}
> {{or}}
> {{Back-pressure is enabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}{{}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15375) Remove BackPressureStrategy



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105780#comment-17105780
 ] 

Jordan West commented on CASSANDRA-15375:
-

While I don't totally agree with your bar for what constitutes an official 
public API, my biggest concern was that since we weren't deprecating it, if 
there was a user of it, there would be an exception when their configuration 
was read on startup. Looking closer at your patch, it does look like this has 
been accounted for by indeed marking the fields {{@Deprecated}} in {{Config}} 
(as an aside I also agree with the comment that we should note them as such in 
cassandra.yaml). So while they will take no effect, they won't harm anyone who 
had it set in their config either. 

> Remove BackPressureStrategy
> ---
>
> Key: CASSANDRA-15375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15375
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Observability/Logging
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Low
> Fix For: 4.0
>
>
> This is odd:
> {{INFO [main] 2019-10-25 10:33:07,985 DatabaseDescriptor.java:803 - 
> Back-pressure is disabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}
> When I saw that, I wasn't sure if back pressure was actually disabled, or if 
> I was really using {{RateBasedBackPressure.}}
> This should change to output either:
> {{Back-pressure is disabled}}
> {{or}}
> {{Back-pressure is enabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}{{}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15375) Remove BackPressureStrategy



 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jordan West updated CASSANDRA-15375:

Fix Version/s: 4.0-beta

> Remove BackPressureStrategy
> ---
>
> Key: CASSANDRA-15375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15375
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Observability/Logging
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Low
> Fix For: 4.0, 4.0-beta
>
>
> This is odd:
> {{INFO [main] 2019-10-25 10:33:07,985 DatabaseDescriptor.java:803 - 
> Back-pressure is disabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}
> When I saw that, I wasn't sure if back pressure was actually disabled, or if 
> I was really using {{RateBasedBackPressure.}}
> This should change to output either:
> {{Back-pressure is disabled}}
> {{or}}
> {{Back-pressure is enabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}{{}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-15807) Support multiple keyspaces in Cassandra-Diff

2020-05-12 Thread Yifan Cai (Jira)

Yifan Cai created CASSANDRA-15807:
-

 Summary: Support multiple keyspaces in Cassandra-Diff
 Key: CASSANDRA-15807
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15807
 Project: Cassandra
  Issue Type: Improvement
  Components: Tool/diff
Reporter: Yifan Cai
Assignee: Yifan Cai


Adding the support to run diff comparison of tables across multiple keyspaces 
to avoid bringing up multiple spark jobs and give a better view of data diffs 
in the whole cluster. 
Cassandra-diff currently only support compare multiple tables under one single 
keyspace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15375) Remove BackPressureStrategy

2020-05-12 Thread Benedict Elliott Smith (Jira)



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105783#comment-17105783
 ] 

Benedict Elliott Smith commented on CASSANDRA-15375:


FWIW, we haven't actually always been great about deprecating old parameters in 
config files.  I'm pretty sure every version has had some config parameter 
vanish without deprecation (which is not an endorsement of that, of course)

Separately, I hadn't actually realised it was in the yaml, I had been searching 
for the wrong text string.  That could certainly be taken to be an 
advertisement of the feature.  I still don't think deprecation is warranted, 
given the earlier discussions about the feature itself, but my prior position 
was taken on faulty information.  I'd be happy to defer to somebody with a 
stronger opinion.

> Remove BackPressureStrategy
> ---
>
> Key: CASSANDRA-15375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15375
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Observability/Logging
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Low
> Fix For: 4.0, 4.0-beta
>
>
> This is odd:
> {{INFO [main] 2019-10-25 10:33:07,985 DatabaseDescriptor.java:803 - 
> Back-pressure is disabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}
> When I saw that, I wasn't sure if back pressure was actually disabled, or if 
> I was really using {{RateBasedBackPressure.}}
> This should change to output either:
> {{Back-pressure is disabled}}
> {{or}}
> {{Back-pressure is enabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}{{}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14939) fix some operational holes in incremental repair



[ 
https://issues.apache.org/jira/browse/CASSANDRA-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105790#comment-17105790
 ] 

Jordan West commented on CASSANDRA-14939:
-

[~kohlisankalp] I don't personally have enough background to comment on this 
ticket. With fresh eyes I saw it as a potential candidate for a follow up 
release since it hasn't been worked on for over one year and looks to be a set 
of improvements on an existing feature. I defer to those involved with the 
ticket and/or the community if its better to start a conversation on the 
mailing list. 

> fix some operational holes in incremental repair
> 
>
> Key: CASSANDRA-14939
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14939
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Incremental repair has a few operational rough spots that make it more 
> difficult to fully automate and operate at scale than it should be.
> * Visibility into whether pending repair data exists for a given token range.
> * Ability to force promotion/demotion of data for completed sessions instead 
> of waiting for compaction.
> * Get the most recent repairedAt timestamp for a given token range.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15375) Remove BackPressureStrategy



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105791#comment-17105791
 ] 

Jordan West commented on CASSANDRA-15375:
-

I'd be +1 on this if we update cassandra.yaml with a deprecation notice -- 
continuing with code removal as it is in the patch now. 

> Remove BackPressureStrategy
> ---
>
> Key: CASSANDRA-15375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15375
> Project: Cassandra
>  Issue Type: Bug
>  Components: Messaging/Client, Observability/Logging
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Low
> Fix For: 4.0, 4.0-beta
>
>
> This is odd:
> {{INFO [main] 2019-10-25 10:33:07,985 DatabaseDescriptor.java:803 - 
> Back-pressure is disabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}
> When I saw that, I wasn't sure if back pressure was actually disabled, or if 
> I was really using {{RateBasedBackPressure.}}
> This should change to output either:
> {{Back-pressure is disabled}}
> {{or}}
> {{Back-pressure is enabled with strategy 
> org.apache.cassandra.net.RateBasedBackPressure\{high_ratio=0.9, factor=5, 
> flow=FAST}.}}{{}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15778) CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105793#comment-17105793
 ] 

David Capwell commented on CASSANDRA-15778:
---

Thanks [~ifesdjeen] for taking a look at this.  Most of the context is not in 
JIRA and the patch I posted was so Sumanth and I could test out if the value 
field being empty was the issue and if there were other issues; *no one should 
apply that patch to production*.

The current state is that the value field silently corrupted the data because

* CASSANDRA-15373 is 3.0.20 and not 3.0.19 (test cluster)
* CASSANDRA-15790 is needed to not write data and instead fail.  We currently 
write out data that has no logic to read, so corrupts the stream.

Action items so far are to come up with a resolution to this case.  
[~bdeggleston] has the idea that we should migrate the hidden column out of 
VOID and to BLOB (2.1 logic), this will make the data readable via Thrift but 
not CQL.

[~ifesdjeen] about your reproduction, skimming the method (and CassandraServer 
version) you sent I don't see us injecting anything, are you saying 
CassandraDaemon, ThriftClient, or client code added this data?  If ThriftClient 
or CassandraDaemon do you know where?

> CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads
> -
>
> Key: CASSANDRA-15778
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15778
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction, Local/SSTable
>Reporter: Sumanth Pasupuleti
>Assignee: David Capwell
>Priority: Normal
> Fix For: 3.0.x
>
>
> Below is the exception with stack trace. This issue is consistently 
> reproduce-able.
> {code:java}
> ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 
> 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception 
> on thread 
> Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException:
>  Corrupted: 
> /mnt/data/cassandra/data//  at 
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at 
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadR

[jira] [Commented] (CASSANDRA-15799) CorruptSSTableException when compacting a 3.0 format sstable that was originally created as an outcome of 2.1 sstable upgrade



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105808#comment-17105808
 ] 

David Capwell commented on CASSANDRA-15799:
---

[~sumanth.pasupuleti] can you try out 
https://github.com/dcapwell/cassandra/tree/poc/CASSANDRA-15799_defensive_checks

> CorruptSSTableException when compacting a 3.0 format sstable that was 
> originally created as an outcome of 2.1 sstable upgrade
> -
>
> Key: CASSANDRA-15799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15799
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction, Local/SSTable
>Reporter: Sumanth Pasupuleti
>Assignee: David Capwell
>Priority: Normal
> Fix For: 3.0.x
>
>
> Below is the exception with stack trace. This issue is reproduce-able.
> {code:java}
> DEBUG [CompactionExecutor:10] 2020-05-07 19:33:34,268 CompactionTask.java:158 
> - Compacting (a3ea9fc0-9099-11ea-933f-c5e852f71338) 
> [/mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db:level=0, ]
> ERROR [CompactionExecutor:10] 2020-05-07 19:33:34,275 
> CassandraDaemon.java:208 - Exception in thread 
> Thread[CompactionExecutor:10,1,RMI Runtime]
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
> /mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:105)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:534)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:394)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:165)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>  ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:675)
>  ~[nf-cassandra-3.0.19.8

[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta



[ 
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105827#comment-17105827
 ] 

David Capwell commented on CASSANDRA-15299:
---

bq. Firstly, we think that not all users will be interested in enabling 
checksumming

Curious why you make that statement, I would hope users don't want silent 
corruption (for the same reason all tables should be compressed, we should 
improve the format to not couple checksumming with compression).  Since this is 
also between client and server, other than performance, how would the user even 
know about this without looking closer into the details?

> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol 
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
>  Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it 
> introduced checksumming/CRC32 to request and response bodies. It’s an 
> important step forward, but it doesn’t cover the entire stream. In 
> particular, the message header is not covered by a checksum or a crc, which 
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of 
> computing the CRC32 on the bytes written on the wire - losing the properties 
> of the CRC32. In some cases, due to this sequencing, attempting to decompress 
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order, 
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for 
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since 
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression 
> operate on individual message bodies rather than frames of multiple complete 
> messages. In reality, this has several important additional downsides. To 
> name a couple:
> # For compression, we are getting poor compression ratios for smaller 
> messages - when operating on tiny sequences of bytes. In reality, for most 
> small requests and responses we are discarding the compressed value as it’d 
> be smaller than the uncompressed one - incurring both redundant allocations 
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages. 
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the 
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we 
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode 
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed, 
> and that we do it before native protocol v5 graduates from beta. See 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
>  and 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15776) python dtest regression caused by CASSANDRA-15637