[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Vinay Chella (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446557#comment-16446557
 ] 

Vinay Chella commented on CASSANDRA-12151:
--

Thanks, [~jasobrown], for reviewing it and making changes, sure I will take 
care of tests. 

{quote}
 I've asked Vinay to add some nodetool commands for enabling/disabling auditlog 
as well (or, at least, to seriously think about it).
{quote}
I agree, will add nodetool commands for enabling/disabling auditlog to help 
improve the usability at node level. 

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14389) Resolve local address binding in 4.0

[Dinesh Joshi (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446553#comment-16446553
 ] 

Dinesh Joshi commented on CASSANDRA-14389:
--

I have incorporated the changes from your commit as well as added tests for 
\{{MessagingService#getPreferredRemoteAddr}}. Also squashed all commit into one.

> Resolve local address binding in 4.0
> 
>
> Key: CASSANDRA-14389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14389
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Dinesh Joshi
>Priority: Minor
> Fix For: 4.x
>
>
> CASSANDRA-8457/CASSANDRA-12229 introduced a regression against 
> CASSANDRA-12673. This was discovered with CASSANDRA-14362 and moved here for 
> resolution independent of that ticket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13985) Support restricting reads and writes to specific datacenters on a per user basis

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446467#comment-16446467
 ] 

Blake Eggleston commented on CASSANDRA-13985:
-

got all of the tests passing: 
https://circleci.com/workflow-run/7c339d36-2dcd-4a38-9020-bb629b4880a0

> Support restricting reads and writes to specific datacenters on a per user 
> basis
> 
>
> Key: CASSANDRA-13985
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13985
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0
>
>
> There are a few use cases where it makes sense to restrict the operations a 
> given user can perform in specific data centers. The obvious use case is the 
> production/analytics datacenter configuration. You don’t want the production 
> user to be reading/or writing to the analytics datacenter, and you don’t want 
> the analytics user to be reading from the production datacenter.
> Although we expect users to get this right on that application level, we 
> should also be able to enforce this at the database level. The first approach 
> that comes to mind would be to support an optional DC parameter when granting 
> select and modify permissions to roles. Something like {{GRANT SELECT ON 
> some_keyspace TO that_user IN DC dc1}}, statements that omit the dc would 
> implicitly be granting permission to all dcs. However, I’m not married to 
> this approach.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9608) Support Java 9

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446421#comment-16446421
 ] 

Michael Shuler commented on CASSANDRA-9608:
---

JDK 10 EOLs in Sept 2018 and JDK 11 is intended to be an LTS release. I don't 
think work on JDK 10 is very valuable, but perhaps JDK 11 makes more sense.

http://www.oracle.com/technetwork/java/javase/eol-135779.html

> Support Java 9
> --
>
> Key: CASSANDRA-9608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9608
> Project: Cassandra
>  Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> This ticket is intended to group all issues found to support Java 9 in the 
> future.
> From what I've found out so far:
> * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved. 
> It can be easily solved using this patch:
> {code}
> - artifactId="cobertura"/>
> + artifactId="cobertura">
> +  
> +
> {code}
> * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods 
> {{monitorEnter}} + {{monitorExit}}. These methods are used by 
> {{o.a.c.utils.concurrent.Locks}} which is only used by 
> {{o.a.c.db.AtomicBTreeColumns}}.
> I don't mind to start working on this yet since Java 9 is in a too early 
> development phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9608) Support Java 9

[Kamil (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446403#comment-16446403
 ] 

Kamil commented on CASSANDRA-9608:
--

JDK 10 is already released. I think that moving forward with this issue is 
rather important

> Support Java 9
> --
>
> Key: CASSANDRA-9608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9608
> Project: Cassandra
>  Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> This ticket is intended to group all issues found to support Java 9 in the 
> future.
> From what I've found out so far:
> * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved. 
> It can be easily solved using this patch:
> {code}
> - artifactId="cobertura"/>
> + artifactId="cobertura">
> +  
> +
> {code}
> * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods 
> {{monitorEnter}} + {{monitorExit}}. These methods are used by 
> {{o.a.c.utils.concurrent.Locks}} which is only used by 
> {{o.a.c.db.AtomicBTreeColumns}}.
> I don't mind to start working on this yet since Java 9 is in a too early 
> development phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14409) Transient Replication: Support ring changes when transient replication is in use (add/remove node, change RF, add/remove DC)

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14409:
---
Description: 
The additional state transitions that transient replication introduces require 
streaming and nodetool cleanup to behave differently. We already have code that 
does the streaming, but in some cases we shouldn't stream any data and in 
others when we stream to receive data we have to make sure we stream from a 
full replica and not a transient replica.

Transitioning from not replicated to transiently replicated means that a node 
must stay pending until the next incremental repair completes at which point 
the data for that range is known to be available at full replicas.

Transitioning from transiently replicated to fully replicated requires 
streaming from a full replica and is identical to how we stream from not 
replicated to replicated. The transition must be managed so the transient 
replica is not read from as a full replica until streaming completes. It can be 
used immediately for a write quorum.

Transitioning from fully replicated to transiently replicated requires cleanup 
to remove repaired data from the transiently replicated range to reclaim space. 
It can be used immediately for a write quorum.

Transitioning from transiently replicated to not replicated requires cleanup to 
be run to remove the formerly transiently replicated data.

nodetool move, removenode, cleanup, decommission, and rebuild need to handle 
these issues as does bootstrap.

Update web site, documentation, NEWS.txt with a description of the steps for 
doing common operations. Add/remove DC, Add/remove node(s), replace node, 
change RF. 

  was:
The additional state transitions that transient replication introduces require 
streaming and nodetool cleanup to behave differently. We already have code that 
does the streaming, but in some cases we shouldn't stream any data and in 
others when we stream to receive data we have to make sure we stream from a 
full replica and not a transient replica.

Transitioning from not replicated to transiently replicated means that a node 
must stay pending until the next incremental repair completes at which point 
the data for that range is known to be available at full replicas.

Transitioning from transiently replicated to fully replicated requires 
streaming from a full replica and is identical to how we stream from not 
replicated to replicated. The transition must be managed so the transient 
replica is not read from as a full replica until streaming completes. It can be 
used immediately for a write quorum.

Transitioning from fully replicated to transiently replicated requires cleanup 
to remove repaired data from the transiently replicated range to reclaim space. 
It can be used immediately for a write quorum.

Transitioning from transiently replicated to not replicated requires cleanup to 
be run to remove the formerly transiently replicated data.

nodetool move, removenode, cleanup, decommission,  need to handle these issues 
as does bootstrap.

Update web site, documentation, NEWS.txt with a description of the steps for 
doing common operations. Add/remove DC, Add/remove node(s), replace node, 
change RF. 


> Transient Replication: Support ring changes when transient replication is in 
> use (add/remove node, change RF, add/remove DC)
> 
>
> Key: CASSANDRA-14409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14409
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Coordination, Core, Documentation and Website
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> The additional state transitions that transient replication introduces 
> require streaming and nodetool cleanup to behave differently. We already have 
> code that does the streaming, but in some cases we shouldn't stream any data 
> and in others when we stream to receive data we have to make sure we stream 
> from a full replica and not a transient replica.
> Transitioning from not replicated to transiently replicated means that a node 
> must stay pending until the next incremental repair completes at which point 
> the data for that range is known to be available at full replicas.
> Transitioning from transiently replicated to fully replicated requires 
> streaming from a full replica and is identical to how we stream from not 
> replicated to replicated. The transition must be managed so the transient 
> replica is not read from as a full replica until streaming completes. It can 
> be used immediately for a write quorum.
> Transitioning from fully replicated to transiently replicated requires 
> cleanup 

[jira] [Commented] (CASSANDRA-14389) Resolve local address binding in 4.0

[Dinesh Joshi (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446396#comment-16446396
 ] 

Dinesh Joshi commented on CASSANDRA-14389:
--

Hi [~jasobrown], thank you for reviewing. I have incorporated your changes. I 
also added a test for {{StreamSession}}.

> Resolve local address binding in 4.0
> 
>
> Key: CASSANDRA-14389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14389
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Dinesh Joshi
>Priority: Minor
> Fix For: 4.x
>
>
> CASSANDRA-8457/CASSANDRA-12229 introduced a regression against 
> CASSANDRA-12673. This was discovered with CASSANDRA-14362 and moved here for 
> resolution independent of that ticket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14405) Transient Replication: Metadata refactor

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14405:
---
Description: 
Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i.

Block the creation of MV or 2i in keyspaces using transient replication.

Block the creation/alteration of keyspaces using transient replication if the 
experimental flag is not set.

Update web site, CQL spec, and any other documentation for the new syntax.


  was:
Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i.

Block the creation of MV or 2i in keyspaces using transient replication.

Update web site, CQL spec, and any other documentation for the new syntax.



> Transient Replication: Metadata refactor
> 
>
> Key: CASSANDRA-14405
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14405
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Distributed Metadata, Documentation and Website
>Reporter: Ariel Weisberg
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> Add support to CQL and NTS for configuring keyspaces to have transient 
> replicas.
> Add syntax allowing a keyspace using NTS to declare some replicas in each DC 
> as transient.
> Implement metadata internal to the DB so that it's possible to identify what 
> replicas are transient for a given token or range.
> Introduce Replica which is an InetAddressAndPort and a boolean indicating 
> whether the replica is transient. ReplicatedRange which is a wrapper around a 
> Range that indicates if the range is transient.
> Block altering of keyspaces to use transient replication if they already 
> contain MVs or 2i.
> Block the creation of MV or 2i in keyspaces using transient replication.
> Block the creation/alteration of keyspaces using transient replication if the 
> experimental flag is not set.
> Update web site, CQL spec, and any other documentation for the new syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14409) Transient Replication: Support ring changes when transient replication is in use (add/remove node, change RF, add/remove DC)

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14409:
---
Description: 
The additional state transitions that transient replication introduces require 
streaming and nodetool cleanup to behave differently. We already have code that 
does the streaming, but in some cases we shouldn't stream any data and in 
others when we stream to receive data we have to make sure we stream from a 
full replica and not a transient replica.

Transitioning from not replicated to transiently replicated means that a node 
must stay pending until the next incremental repair completes at which point 
the data for that range is known to be available at full replicas.

Transitioning from transiently replicated to fully replicated requires 
streaming from a full replica and is identical to how we stream from not 
replicated to replicated. The transition must be managed so the transient 
replica is not read from as a full replica until streaming completes. It can be 
used immediately for a write quorum.

Transitioning from fully replicated to transiently replicated requires cleanup 
to remove repaired data from the transiently replicated range to reclaim space. 
It can be used immediately for a write quorum.

Transitioning from transiently replicated to not replicated requires cleanup to 
be run to remove the formerly transiently replicated data.

nodetool move, removenode, cleanup, decommission,  need to handle these issues 
as does bootstrap.

Update web site, documentation, NEWS.txt with a description of the steps for 
doing common operations. Add/remove DC, Add/remove node(s), replace node, 
change RF. 

  was:
The additional state transitions that transient replication introduces require 
streaming and nodetool cleanup to behave differently. We already have code that 
does the streaming, but in some cases we shouldn't stream any data and in 
others when we stream to receive data we have to make sure we stream from a 
full replica and not a transient replica.

Transitioning from not replicated to transiently replicated means that a node 
must stay pending until the next incremental repair completes at which point 
the data for that range is known to be available at full replicas.

Transitioning from transiently replicated to fully replicated requires 
streaming from a full replica and is identical to how we stream from not 
replicated to replicated. The transition must be managed so the transient 
replica is not read from as a full replica until streaming completes. It can be 
used immediately for a write quorum.

Transitioning from fully replicated to transiently replicated requires cleanup 
to remove repaired data from the transiently replicated range to reclaim space. 
It can be used immediately for a write quorum.

Transitioning from transiently replicated to not replicated requires cleanup to 
be run to remove the formerly transiently replicated data.

nodetool move, removenode, cleanup, decommission,  need to handle these issues 
as does bootstrap.



> Transient Replication: Support ring changes when transient replication is in 
> use (add/remove node, change RF, add/remove DC)
> 
>
> Key: CASSANDRA-14409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14409
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Coordination, Core, Documentation and Website
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> The additional state transitions that transient replication introduces 
> require streaming and nodetool cleanup to behave differently. We already have 
> code that does the streaming, but in some cases we shouldn't stream any data 
> and in others when we stream to receive data we have to make sure we stream 
> from a full replica and not a transient replica.
> Transitioning from not replicated to transiently replicated means that a node 
> must stay pending until the next incremental repair completes at which point 
> the data for that range is known to be available at full replicas.
> Transitioning from transiently replicated to fully replicated requires 
> streaming from a full replica and is identical to how we stream from not 
> replicated to replicated. The transition must be managed so the transient 
> replica is not read from as a full replica until streaming completes. It can 
> be used immediately for a write quorum.
> Transitioning from fully replicated to transiently replicated requires 
> cleanup to remove repaired data from the transiently replicated range to 
> reclaim space. It can be used immediately for a write quorum.
> Transitioning from transiently replicated to 

[jira] [Updated] (CASSANDRA-14405) Transient Replication: Metadata refactor

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14405:
---
Description: 
Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i.

Block the creation of MV or 2i in keyspaces using transient replication.

Update web site, CQL spec, and any other documentation for the new syntax.


  was:
Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i.

Block the creation of MV or 2i in keyspaces using transient replication.



> Transient Replication: Metadata refactor
> 
>
> Key: CASSANDRA-14405
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14405
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Distributed Metadata, Documentation and Website
>Reporter: Ariel Weisberg
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> Add support to CQL and NTS for configuring keyspaces to have transient 
> replicas.
> Add syntax allowing a keyspace using NTS to declare some replicas in each DC 
> as transient.
> Implement metadata internal to the DB so that it's possible to identify what 
> replicas are transient for a given token or range.
> Introduce Replica which is an InetAddressAndPort and a boolean indicating 
> whether the replica is transient. ReplicatedRange which is a wrapper around a 
> Range that indicates if the range is transient.
> Block altering of keyspaces to use transient replication if they already 
> contain MVs or 2i.
> Block the creation of MV or 2i in keyspaces using transient replication.
> Update web site, CQL spec, and any other documentation for the new syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14409) Transient Replication: Support ring changes when transient replication is in use (add/remove node, change RF, add/remove DC)

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14409:
---
Component/s: Documentation and Website

> Transient Replication: Support ring changes when transient replication is in 
> use (add/remove node, change RF, add/remove DC)
> 
>
> Key: CASSANDRA-14409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14409
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Coordination, Core, Documentation and Website
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> The additional state transitions that transient replication introduces 
> require streaming and nodetool cleanup to behave differently. We already have 
> code that does the streaming, but in some cases we shouldn't stream any data 
> and in others when we stream to receive data we have to make sure we stream 
> from a full replica and not a transient replica.
> Transitioning from not replicated to transiently replicated means that a node 
> must stay pending until the next incremental repair completes at which point 
> the data for that range is known to be available at full replicas.
> Transitioning from transiently replicated to fully replicated requires 
> streaming from a full replica and is identical to how we stream from not 
> replicated to replicated. The transition must be managed so the transient 
> replica is not read from as a full replica until streaming completes. It can 
> be used immediately for a write quorum.
> Transitioning from fully replicated to transiently replicated requires 
> cleanup to remove repaired data from the transiently replicated range to 
> reclaim space. It can be used immediately for a write quorum.
> Transitioning from transiently replicated to not replicated requires cleanup 
> to be run to remove the formerly transiently replicated data.
> nodetool move, removenode, cleanup, decommission,  need to handle these 
> issues as does bootstrap.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13442) Support a means of strongly consistent highly available replication with tunable storage requirements

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg resolved CASSANDRA-13442.

Resolution: Duplicate

> Support a means of strongly consistent highly available replication with 
> tunable storage requirements
> -
>
> Key: CASSANDRA-13442
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13442
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction, Coordination, Distributed Metadata, Local 
> Write-Read Paths
>Reporter: Ariel Weisberg
>Priority: Major
>
> Replication factors like RF=2 can't provide strong consistency and 
> availability because if a single node is lost it's impossible to reach a 
> quorum of replicas. Stepping up to RF=3 will allow you to lose a node and 
> still achieve quorum for reads and writes, but requires committing additional 
> storage.
> The requirement of a quorum for writes/reads doesn't seem to be something 
> that can be relaxed without additional constraints on queries, but it seems 
> like it should be possible to relax the requirement that 3 full copies of the 
> entire data set are kept. What is actually required is a covering data set 
> for the range and we should be able to achieve a covering data set and high 
> availability without having three full copies. 
> After a repair we know that some subset of the data set is fully replicated. 
> At that point we don't have to read from a quorum of nodes for the repaired 
> data. It is sufficient to read from a single node for the repaired data and a 
> quorum of nodes for the unrepaired data.
> One way to exploit this would be to have N replicas, say the last N replicas 
> (where N varies with RF) in the preference list, delete all repaired data 
> after a repair completes. Subsequent quorum reads will be able to retrieve 
> the repaired data from any of the two full replicas and the unrepaired data 
> from a quorum read of any replica including the "transient" replicas.
> Configuration for something like this in NTS might be something similar to { 
> DC1="3-1", DC2="3-2" } where the first value is the replication factor used 
> for consistency and the second values is the number of transient replicas. If 
> you specify { DC1=3, DC2=3 } then the number of transient replicas defaults 
> to 0 and you get the same behavior you have today.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14405) Transient Replication: Metadata refactor

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14405:
---
Description: 
Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i.

Block the creation of MV or 2i in keyspaces using transient replication.


  was:
Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i

Block the creation of MV or 2i in keyspaces using transient replication



> Transient Replication: Metadata refactor
> 
>
> Key: CASSANDRA-14405
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14405
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Distributed Metadata, Documentation and Website
>Reporter: Ariel Weisberg
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> Add support to CQL and NTS for configuring keyspaces to have transient 
> replicas.
> Add syntax allowing a keyspace using NTS to declare some replicas in each DC 
> as transient.
> Implement metadata internal to the DB so that it's possible to identify what 
> replicas are transient for a given token or range.
> Introduce Replica which is an InetAddressAndPort and a boolean indicating 
> whether the replica is transient. ReplicatedRange which is a wrapper around a 
> Range that indicates if the range is transient.
> Block altering of keyspaces to use transient replication if they already 
> contain MVs or 2i.
> Block the creation of MV or 2i in keyspaces using transient replication.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14407) Transient Replication: Add support for correct reads when transient replication is in use

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14407:
---
Description: 
Digest reads should never be sent to transient replicas.

Mismatches with results from transient replicas shouldn't trigger read repair.

Read repair should never attempt to repair a transient replica.

Reads should always include at least one full replica. They should also prefer 
transient replicas where possible.

Range scans must ensure the entire scanned range performs replica selection 
that satisfies the requirement that every range scanned includes one full 
replica.


  was:
Digest reads should never be sent to transient replicas

Mismatches with results from transient replicas shouldn't trigger read repair.

Read repair should never attempt to repair a transient replica.

Reads should always include at least one full replica. They should also prefer 
transient replicas where possible.

Range scans must ensure the entire scanned range performs replica selection 
that satisfies the requirement that every range scanned includes one full 
replica.



> Transient Replication: Add support for correct reads when transient 
> replication is in use
> -
>
> Key: CASSANDRA-14407
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14407
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Coordination
>Reporter: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> Digest reads should never be sent to transient replicas.
> Mismatches with results from transient replicas shouldn't trigger read repair.
> Read repair should never attempt to repair a transient replica.
> Reads should always include at least one full replica. They should also 
> prefer transient replicas where possible.
> Range scans must ensure the entire scanned range performs replica selection 
> that satisfies the requirement that every range scanned includes one full 
> replica.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14408) Transient Replication: Incremental & Validation repair handling of transient replicas

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14408:
---
Fix Version/s: 4.0

> Transient Replication: Incremental & Validation repair handling of transient 
> replicas
> -
>
> Key: CASSANDRA-14408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14408
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Repair
>Reporter: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> At transient replicas anti-compaction shouldn't output any data for transient 
> ranges as the data will be dropped after repair.
> Transient replicas should also never have data streamed to them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14409) Transient Replication: Support ring changes when transient replication is in use (add/remove node, change RF, add/remove DC)

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14409:
--

 Summary: Transient Replication: Support ring changes when 
transient replication is in use (add/remove node, change RF, add/remove DC)
 Key: CASSANDRA-14409
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14409
 Project: Cassandra
  Issue Type: Sub-task
  Components: Coordination, Core
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
 Fix For: 4.0


The additional state transitions that transient replication introduces require 
streaming and nodetool cleanup to behave differently. We already have code that 
does the streaming, but in some cases we shouldn't stream any data and in 
others when we stream to receive data we have to make sure we stream from a 
full replica and not a transient replica.

Transitioning from not replicated to transiently replicated means that a node 
must stay pending until the next incremental repair completes at which point 
the data for that range is known to be available at full replicas.

Transitioning from transiently replicated to fully replicated requires 
streaming from a full replica and is identical to how we stream from not 
replicated to replicated. The transition must be managed so the transient 
replica is not read from as a full replica until streaming completes. It can be 
used immediately for a write quorum.

Transitioning from fully replicated to transiently replicated requires cleanup 
to remove repaired data from the transiently replicated range to reclaim space. 
It can be used immediately for a write quorum.

Transitioning from transiently replicated to not replicated requires cleanup to 
be run to remove the formerly transiently replicated data.

nodetool move, removenode, cleanup, decommission,  need to handle these issues 
as does bootstrap.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14408) Transient Replication: Incremental & Validation repair handling of transient replicas

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14408:
--

 Summary: Transient Replication: Incremental & Validation repair 
handling of transient replicas
 Key: CASSANDRA-14408
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14408
 Project: Cassandra
  Issue Type: Sub-task
  Components: Repair
Reporter: Ariel Weisberg


At transient replicas anti-compaction shouldn't output any data for transient 
ranges as the data will be dropped after repair.

Transient replicas should also never have data streamed to them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14407) Transient Replication: Add support for correct reads when transient replication is in use

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14407:
--

 Summary: Transient Replication: Add support for correct reads when 
transient replication is in use
 Key: CASSANDRA-14407
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14407
 Project: Cassandra
  Issue Type: Sub-task
  Components: Coordination
Reporter: Ariel Weisberg
 Fix For: 4.0


Digest reads should never be sent to transient replicas

Mismatches with results from transient replicas shouldn't trigger read repair.

Read repair should never attempt to repair a transient replica.

Reads should always include at least one full replica. They should also prefer 
transient replicas where possible.

Range scans must ensure the entire scanned range performs replica selection 
that satisfies the requirement that every range scanned includes one full 
replica.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14406) Transient Replication: Implement cheap quorum write optimizations

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14406:
--

 Summary: Transient Replication: Implement cheap quorum write 
optimizations
 Key: CASSANDRA-14406
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14406
 Project: Cassandra
  Issue Type: Sub-task
  Components: Coordination
Reporter: Ariel Weisberg


Writes should never be sent to transient replicas unless necessary to satisfy 
the requested consistency level. Such as RF not being sufficient for strong 
consistency or not enough full replicas marked as alive.

If a write doesn't receive sufficient responses in time additional replicas 
should be sent the write similar to Rapid Read Protection.

Hints should never be written for a transient replica.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14405) Transient Replication: Metadata refactor

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14405:
--

 Summary: Transient Replication: Metadata refactor
 Key: CASSANDRA-14405
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14405
 Project: Cassandra
  Issue Type: Sub-task
  Components: Core, Distributed Metadata, Documentation and Website
Reporter: Ariel Weisberg
Assignee: Blake Eggleston
 Fix For: 4.0


Add support to CQL and NTS for configuring keyspaces to have transient replicas.

Add syntax allowing a keyspace using NTS to declare some replicas in each DC as 
transient.

Implement metadata internal to the DB so that it's possible to identify what 
replicas are transient for a given token or range.

Introduce Replica which is an InetAddressAndPort and a boolean indicating 
whether the replica is transient. ReplicatedRange which is a wrapper around a 
Range that indicates if the range is transient.

Block altering of keyspaces to use transient replication if they already 
contain MVs or 2i

Block the creation of MV or 2i in keyspaces using transient replication




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14404) Transient Replication & Cheap Quorums: Decouple storage requirements from consensus group size using incremental repair

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-14404:
--

 Summary: Transient Replication & Cheap Quorums: Decouple storage 
requirements from consensus group size using incremental repair
 Key: CASSANDRA-14404
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14404
 Project: Cassandra
  Issue Type: New Feature
  Components: Coordination, Core, CQL, Distributed Metadata, Hints, 
Local Write-Read Paths, Materialized Views, Repair, Secondary Indexes, Testing, 
Tools
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
 Fix For: 4.0


Transient Replication is an implementation of [Witness 
Replicas|http://www2.cs.uh.edu/~paris/MYPAPERS/Icdcs86.pdf 
(https://www.google.com/url?sa=t=j==s=web=1=rja=8=0ahUKEwi834a%E2%80%948HaAhWCneAKHdj8DzAQFggpMAA=http%3A%2F%2Fwww2.cs.uh.edu%2F~paris%2FMYPAPERS%2FIcdcs86.pdf=AOvVaw0GfCaaAtdzHiM65du1-qeI)]
 that leverages incremental repair to make full replicas consistent with 
transient replicas that don't store the entire data set. Witness replicas are 
used in real world systems such as Megastore and Spanner to increase 
availability inexpensively without having to commit to more full copies of the 
database. Transient replicas implement functionality similar to upgradable and 
temporary replicas from the paper.

With transient replication the replication factor is increased beyond the 
desired level of data redundancy by adding replicas that only store data when 
sufficient full replicas are unavailable to store the data. These replicas are 
called transient replicas. When incremental repair runs transient replicas 
stream any data they have received to full replicas and once the data is fully 
replicated it is dropped at the transient replicas.

Cheap quorums are a further set of optimizations on the write path to avoid 
writing to transient replicas unless sufficient full replicas are available as 
well as optimizations on the read path to prefer reading from transient 
replicas. When writing at quorum to a table configured to use transient 
replication the quorum will always prefer available full replicas over 
transient replicas so that transient replicas don't have to process writes. 
Rapid write protection (similar to rapid read protection) reduces tail latency 
when full replicas are temporarily late to respond by sending writes to 
additional replicas if necessary.

Transient replicas can generally service reads faster because they don't have 
do anything beyond bloom filter checks if they have no data. With vnodes and 
larger size clusters they will not have a large quantity of data even in 
failure cases where transient replicas start to serve a steady amount of write 
traffic for some of their transiently replicated ranges.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14389) Resolve local address binding in 4.0

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446288#comment-16446288
 ] 

Jason Brown commented on CASSANDRA-14389:
-

@Dniesh, This looks pretty good. However, I wonder if we can dump all the 
places where we naively plumb the 'connecting' address though. I took a pass at 
it here:

||14389||
|[branch|https://github.com/jasobrown/cassandra/tree/14389]|
|[utests  
dtests|https://circleci.com/gh/jasobrown/workflows/cassandra/tree/14389]|
||

The only change of interest is in 
{{MessagingService#getPreferredRemoteAddr()}}, I check into 
{{SystemKeyspace.getPreferredIP(to)}} if there was no {{OutboundMessagingPool}} 
set up in {[MessagingService#channelManagers}}. wdyt?

> Resolve local address binding in 4.0
> 
>
> Key: CASSANDRA-14389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14389
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Dinesh Joshi
>Priority: Minor
> Fix For: 4.x
>
>
> CASSANDRA-8457/CASSANDRA-12229 introduced a regression against 
> CASSANDRA-12673. This was discovered with CASSANDRA-14362 and moved here for 
> resolution independent of that ticket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14389) Resolve local address binding in 4.0

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446288#comment-16446288
 ] 

Jason Brown edited comment on CASSANDRA-14389 at 4/20/18 8:02 PM:
--

[~djoshi3], This looks pretty good. However, I wonder if we can dump all the 
places where we naively plumb the 'connecting' address though. I took a pass at 
it here:

||14389||
|[branch|https://github.com/jasobrown/cassandra/tree/14389]|
|[utests  
dtests|https://circleci.com/gh/jasobrown/workflows/cassandra/tree/14389]|
||

The only change of interest is in 
{{MessagingService#getPreferredRemoteAddr()}}, I check into 
{{SystemKeyspace.getPreferredIP(to)}} if there was no {{OutboundMessagingPool}} 
set up in {[MessagingService#channelManagers}}. wdyt?


was (Author: jasobrown):
@Dniesh, This looks pretty good. However, I wonder if we can dump all the 
places where we naively plumb the 'connecting' address though. I took a pass at 
it here:

||14389||
|[branch|https://github.com/jasobrown/cassandra/tree/14389]|
|[utests  
dtests|https://circleci.com/gh/jasobrown/workflows/cassandra/tree/14389]|
||

The only change of interest is in 
{{MessagingService#getPreferredRemoteAddr()}}, I check into 
{{SystemKeyspace.getPreferredIP(to)}} if there was no {{OutboundMessagingPool}} 
set up in {[MessagingService#channelManagers}}. wdyt?

> Resolve local address binding in 4.0
> 
>
> Key: CASSANDRA-14389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14389
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Dinesh Joshi
>Priority: Minor
> Fix For: 4.x
>
>
> CASSANDRA-8457/CASSANDRA-12229 introduced a regression against 
> CASSANDRA-12673. This was discovered with CASSANDRA-14362 and moved here for 
> resolution independent of that ticket.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446271#comment-16446271
 ] 

Jason Brown edited comment on CASSANDRA-12151 at 4/20/18 7:46 PM:
--

Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of 
review here, I'll take it the rest of the way.

[~vinaykumarcse] and I spoke offline about some things, and on the whole we are 
in the right direction. As I started reviewing the patch, the whitespace 
inconsistencies nagged at me, so I went ahead and fixed those, as well as some 
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and 
no longer a singleton; a volatile reference is maintained in 
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}} 
to {{AuditLogEntry}} as they seemed more apropos there. Code available 
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151]. 
(Note: i didn't run the tests, so you may need to fix those up if necessary)
 
There are some things wrt the interaction of FQL and AuditLogging that I still 
need to work out in my head, and I've asked Vinay to add some nodetool commands 
for enabling/disabling auditlog as well (or, at least, to seriously think about 
it). However, I wanted to post that first batch of changes to get that out of 
the way.


was (Author: jasobrown):
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of 
review here, I'll take it the rest of the way.

[~vinaykumarcse] and I spoke offline about some things, and on the whole we are 
in the right direction. As I started reviewing the patch, the whitespace 
inconsistencies nagged at me, so I went ahead and fixed those, as well as some 
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and 
no longer a singleton; a volatile reference is maintained in 
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}} 
to {{AuditLogEntry}} as they seemed more apropos there. Code available 
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
 
There are some things wrt the interaction of FQL and AuditLogging that I still 
need to work out in my head, and I've asked Vinay to add some nodetool commands 
for enabling/disabling auditlog as well (or, at least, to seriously think about 
it). However, I wanted to post that first batch of changes to get that out of 
the way.

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446271#comment-16446271
 ] 

Jason Brown edited comment on CASSANDRA-12151 at 4/20/18 7:45 PM:
--

Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of 
review here, I'll take it the rest of the way.

[~vinaykumarcse] and I spoke offline about some things, and on the whole we are 
in the right direction. As I started reviewing the patch, the whitespace 
inconsistencies nagged at me, so I went ahead and fixed those, as well as some 
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and 
no longer a singleton; a volatile reference is maintained in 
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}} 
to {{AuditLogEntry}} as they seemed more apropos there. Code available 
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
 
There are some things wrt the interaction of FQL and AuditLogging that I still 
need to work out in my head, and I've asked Vinay to add some nodetool commands 
for enabling/disabling auditlog as well (or, at least, to seriously think about 
it). However, I wanted to post that first batch of changes to get that out of 
the way.


was (Author: jasobrown):
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of 
review here, I'll take it the rest of the way.

[~vinaykumarcse] and I spoke offline about some things, and on the whole we are 
in the right direction. As I started reviewing the patch, the whitespace 
inconsistencies nagged at me, so I went ahead and fixed those, as well as some 
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and 
no longer a singleton; a volatile reference is maintained in 
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}} 
to {{AuditLogEntry}} as they seemed more apropos there. Code available 
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
 


> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446271#comment-16446271
 ] 

Jason Brown commented on CASSANDRA-12151:
-

Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of 
review here, I'll take it the rest of the way.

[~vinaykumarcse] and I spoke offline about some things, and on the whole we are 
in the right direction. As I started reviewing the patch, the whitespace 
inconsistencies nagged at me, so I went ahead and fixed those, as well as some 
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and 
no longer a singleton; a volatile reference is maintained in 
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}} 
to {{AuditLogEntry}} as they seemed more apropos there. Code available 
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
 


> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14381) nodetool listsnapshots is missing local system keyspace snapshots

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14381:
---
   Resolution: Fixed
Fix Version/s: 4.0
   Status: Resolved  (was: Ready to Commit)

> nodetool listsnapshots is missing local system keyspace snapshots
> -
>
> Key: CASSANDRA-14381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14381
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: MacOs 10.12.5
> Java 1.8.0_144
> Cassandra 3.11.2 (brew install)
>Reporter: Cyril Scetbon
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> The output of *nodetool listsnapshots* is inconsistent with the snapshots 
> created :
> {code:java}
> $ nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
> $ nodetool snapshot -t tag1 --table local system
> Requested creating snapshot(s) for [system] with snapshot name [tag1] and 
> options {skipFlush=false}
> Snapshot directory: tag1
> $ nodetool snapshot -t tag2 --table local system
> Requested creating snapshot(s) for [system] with snapshot name [tag2] and 
> options {skipFlush=false}
> Snapshot directory: tag2
> $ nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
> $ ls 
> /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/
> tag1 tag2{code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14381) nodetool listsnapshots is missing local system keyspace snapshots

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445931#comment-16445931
 ] 

Ariel Weisberg commented on CASSANDRA-14381:


Committed as 
[e0524c099e0b59cccfd1a9cb9424147cfd001a32|https://github.com/apache/cassandra/commit/e0524c099e0b59cccfd1a9cb9424147cfd001a32].
 Thanks!

> nodetool listsnapshots is missing local system keyspace snapshots
> -
>
> Key: CASSANDRA-14381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14381
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: MacOs 10.12.5
> Java 1.8.0_144
> Cassandra 3.11.2 (brew install)
>Reporter: Cyril Scetbon
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> The output of *nodetool listsnapshots* is inconsistent with the snapshots 
> created :
> {code:java}
> $ nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
> $ nodetool snapshot -t tag1 --table local system
> Requested creating snapshot(s) for [system] with snapshot name [tag1] and 
> options {skipFlush=false}
> Snapshot directory: tag1
> $ nodetool snapshot -t tag2 --table local system
> Requested creating snapshot(s) for [system] with snapshot name [tag2] and 
> options {skipFlush=false}
> Snapshot directory: tag2
> $ nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
> $ ls 
> /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/
> tag1 tag2{code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: nodetool listsnapshots is missing local system keyspace snapshots

[aweisberg]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 4af23348e -> e0524c099


nodetool listsnapshots is missing local system keyspace snapshots

Patch by Ariel Weisberg; Reviewed by Jay Zhuang for CASSANDRA-14381


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e0524c09
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e0524c09
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e0524c09

Branch: refs/heads/trunk
Commit: e0524c099e0b59cccfd1a9cb9424147cfd001a32
Parents: 4af2334
Author: Ariel Weisberg 
Authored: Tue Apr 17 12:40:51 2018 -0400
Committer: Ariel Weisberg 
Committed: Fri Apr 20 11:51:05 2018 -0400

--
 CHANGES.txt   | 1 +
 src/java/org/apache/cassandra/service/StorageService.java | 3 ---
 2 files changed, 1 insertion(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0524c09/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 5902305..0e15a8d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * nodetool listsnapshots is missing local system keyspace snapshots 
(CASSANDRA-14381)
  * Remove StreamCoordinator.streamExecutor thread pool (CASSANDRA-14402)
  * Rename nodetool --with-port to --print-port to disambiguate from --port 
(CASSANDRA-14392)
  * Client TOPOLOGY_CHANGE messages have wrong port. (CASSANDRA-14398)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e0524c09/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java 
b/src/java/org/apache/cassandra/service/StorageService.java
index 4f62dd5..de5d62b 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -3484,9 +3484,6 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 Map snapshotMap = new HashMap<>();
 for (Keyspace keyspace : Keyspace.all())
 {
-if (SchemaConstants.isLocalSystemKeyspace(keyspace.getName()))
-continue;
-
 for (ColumnFamilyStore cfStore : keyspace.getColumnFamilyStores())
 {
 for (Map.Entry 
snapshotDetail : cfStore.getSnapshotDetails().entrySet())


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14381) nodetool listsnapshots is missing local system keyspace snapshots

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-14381:
---
Summary: nodetool listsnapshots is missing local system keyspace snapshots  
(was: nodetool listsnapshots is missing snapshots)

> nodetool listsnapshots is missing local system keyspace snapshots
> -
>
> Key: CASSANDRA-14381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14381
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: MacOs 10.12.5
> Java 1.8.0_144
> Cassandra 3.11.2 (brew install)
>Reporter: Cyril Scetbon
>Assignee: Ariel Weisberg
>Priority: Major
>
> The output of *nodetool listsnapshots* is inconsistent with the snapshots 
> created :
> {code:java}
> $ nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
> $ nodetool snapshot -t tag1 --table local system
> Requested creating snapshot(s) for [system] with snapshot name [tag1] and 
> options {skipFlush=false}
> Snapshot directory: tag1
> $ nodetool snapshot -t tag2 --table local system
> Requested creating snapshot(s) for [system] with snapshot name [tag2] and 
> options {skipFlush=false}
> Snapshot directory: tag2
> $ nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
> $ ls 
> /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/
> tag1 tag2{code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14403) Inconsistent query results for different sort orders (ORDER BY)

[Mahmut Arslan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445849#comment-16445849
 ] 

Mahmut Arslan commented on CASSANDRA-14403:
---

I tried with QUORUM and ALL, same results. 

> Inconsistent query results for different sort orders (ORDER BY)
> ---
>
> Key: CASSANDRA-14403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14403
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, CQL
> Environment: Cassandra 3.5 on Windows Server 2016. Cluster is 
> geo-replicated with 120 machines in 3 datacenters  (40 in each). Replication 
> factor for the table is 3. 
>Reporter: Mahmut Arslan
>Priority: Major
>
> Cassandra returns inconsistent  (different) results for the following queries 
> which only have sort orders specified in ORDER BY clause different. Table is 
> activities_v2 which has PK as (userid, lastmodified, activityid). Clustering 
> key lastmodified has clustering order DESC. Data types for key columns  are 
> (userId, Text), (lastModified, Timestamp), (activityid, UUID). 
> Query "select activityid, lastmodified from activity_feed.activities_v2 where 
> userid = 'userid' and  (lastmodified, activityid) >= 
> ('2018-04-01T07:29:52.611Z', e3afc72e-c41c-3e01-5397-0d972f6ced71) and 
> (lastmodified, activityid) < ('2018-06-01T07:29:52.611Z', 
> ----) ORDER BY lastmodified ASC, activityid 
> DESC;" returns 81 rows as seen below.
> This query returns 
>  activityid | lastmodified
> --+-
>  a5358a6d-669d-3acd-221b-43e62e9adae0 | 2018-04-01 07:29:52.83+
>  11f3bc30-4522-dcb1-8dea-a883e2e9514d | 2018-04-01 07:29:53.189000+
>  cd430200-de20-a614-7269-e9d7528e9cf9 | 2018-04-01 07:29:53.392000+
>  c78a3000-8c9d-c6ad-1df5-c275c6410175 | 2018-04-01 07:29:53.611000+
>  e93cde33-e2e0-a2f4-0083-4a1314e83364 | 2018-04-01 07:29:53.845000+
>  6289441d-2485-6345-5c0a-a06f6b571fab | 2018-04-01 07:31:53.524000+
>  8b4e8d1c-91c6-c549-a548-d87a297deb9c | 2018-04-01 07:31:53.758000+
>  abe73ae9-0d7a-bcb5-13dc-b61f555c7e8a | 2018-04-01 07:31:53.993000+
>  c386e284-c2ee-d511-6122-d6d410e95e34 | 2018-04-01 07:40:26.443000+
>  54b4c120-042d-1e27-ebd9-85824160c9ea | 2018-04-01 07:40:26.443000+
>  c386e284-c2ee-d511-6122-d6d410e95e34 | 2018-04-01 07:40:26.662000+
>  544f224e-9ba1-d4a2-5554-c1a34bbf5fa4 | 2018-04-01 07:40:28.914000+
>  b90ea728-5fb2-6047-d8ed-f64e6268a45e | 2018-04-01 07:40:28.914000+
>  963b308c-4c37-39df-4a0b-829026f3d054 | 2018-04-01 07:40:28.914000+
>  963b308c-4c37-39df-4a0b-829026f3d054 | 2018-04-01 07:40:29.414000+
>  b90ea728-5fb2-6047-d8ed-f64e6268a45e | 2018-04-01 07:40:29.726000+
>  586dac57-fa58-9243-de1e-ceefb297dcd9 | 2018-04-06 21:22:33.50+
> 
> (81 rows)
> When sort order changed in ORDER BY clause as "select activityid, 
> lastmodified from activity_feed.activities_v2 where userid = 'userid' and  
> (lastmodified, activityid) >= ('2018-04-01T07:29:52.611Z', 
> e3afc72e-c41c-3e01-5397-0d972f6ced71) and (lastmodified, activityid) < 
> ('2018-06-01T07:29:52.611Z', ----) ORDER BY 
> lastmodified DESC, activityid ASC;", query returns 142 rows as shown below.
>  
> activityid | lastmodified
> --+-
>  3e2ab8b7-5619-79bf-c992-73d98a1ea9f6 | 2018-04-20 00:54:55.908000+
>  3073f667-5c9f-4bb7-99fb-5e84c219ac91 | 2018-04-20 00:23:49.899000+
>  a3e375df-85a5-4e4f-bd74-73743efe1836 | 2018-04-20 00:23:49.805000+
>  3f91d870-34de-434a-9329-909d66cae0b4 | 2018-04-20 00:23:43.915000+
>  24179d47-4637-4c8d-a818-1b2b1752f790 | 2018-04-20 00:23:05.445000+
>  15bf9cb7-4e67-4cf6-818b-3713215dda32 | 2018-04-20 00:23:05.257000+
>  46c0a730-58c9-42f5-adfd-60261446b9e3 | 2018-04-20 00:23:02.788000+
>  24ecbe40-1990-4a43-a503-e6bf1ce537fb | 2018-04-19 23:44:15.339000+
>  833a2351-7fb7-2723-36d9-932f07a9bf1b | 2018-04-19 23:41:57.131000+
>  9158316d-022f-4150-94e4-229cbff777dc | 2018-04-19 23:41:00.129000+
>  848d5c2e-af22-4235-a6b8-4dcab83f44e5 | 2018-04-19 23:40:38.377000+
>  5646edc6-2f91-4e41-96b2-224bd74c7244 | 2018-04-19 23:37:42.294000+
>  e81c24ff-8b6f-49b3-bdea-d1e178126716 | 2018-04-19 23:27:27.524000+
>  85d137ce-d88b-4d4a-bace-c85ffdccc137 | 2018-04-19 23:20:45.682000+
>  c5c09a5d-4ffa-4a80-91b8-c44af30f0741 | 2018-04-19 23:20:07.587000+
>  c3fc0958-5efb-4555-acca-07e7755cf8cf | 2018-04-19 23:19:55.368000+
> ...
> (142 rows)
>  
> In both cases, queries ran with Local_Quorum. 
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CASSANDRA-13426) Make all DDL statements idempotent and not dependent on global state

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445765#comment-16445765
 ] 

Aleksey Yeschenko commented on CASSANDRA-13426:
---

Alright, this should be more or less it. Addressed everything, only commenting 
on points not addressed in code:

bq. TableMetadata/TableParams/Views / Maybe a bit subjective, but I find the 
naming of "unbuild" unintuitive. How about something like 
builderFrom/makeBuilder/asBuilder/toBuilder

I actually like it, as it’s a direct antonym of ‘build’ and does the exact 
opposite of what build() does. Couldn’t find a better antonym verb, and not a 
fan of any of the suggestions above, so I’ll be leaveing it as is.

bq. TableMetadata / toDebugString is unused

It’s just a helper method for debugging. We have a few similar ones, none of 
them used by code. It’s by design.

bq. Keyspaces / get(String name) is unused

It isn’t, but no harm in leaving it be. Similar container classes all have it.

bq. KeyspaceDiff / A comment explaining why function diffs need to be handled 
differently could be useful (I'm assuming it's because the filtering to 
distinguish UDFs/UDAs makes it slightly more expensive than the other types of 
diff).

Not sure what you mean. Why they are separate fields? Because from schema 
perspective they are very different categories, and it’s just more helpful for 
consumers of of KeyspaceDiff to have them as separate fields than as one.

bq. Functions / can Filter.test use isAggregate rather than instanceof?

It could, but test for UDF would still have to use instanceof. So I’d rather 
keep instanceof there for UDA for consistency.

bq. CompressionParams / outstanding TODO on setCrcCheckChance

Removing it is beyond this JIRA (although it didn’t stop quite a few other 
things from being included). For our purposes it’s harmless-ish, as it doesn’t 
count for hashCode() and equals() purposes and doesn’t mess with schema. 
Ultimately that field should be removed, so I left another comment. 

bq. SetType/ListType/MapType/DynamicCompositeType / getInstance can be 
simplified to just return the result of computeIfAbsent. There's an unchecked 
warning, but I'm not sure that's any different from the existing one. 
ReversedType and CompositeType impls already do this.

The extra check is there to avoid contention on the hot path. Updated 
ReversedType and CompositeType to also try a get() first.

bq. UDAggregate / When reconstructing from schema tables and the function can't 
be reconstructed for whatever reason - we preserve the old behaviour with a 
dummy, broken function for UDFs but not for UDAs. These now trigger an assert. 
Is this an issue?

A UDA has no body of its own, it’s just a combination of UDFs. As such, when we 
try to assemble a UDA from rows on disk, and one or more UDFs are missing, we 
treat this as an error and abort. An analogy: deserializing table metadata from 
schema tables with a column that references a non-existent UDT. A UDF OTOH can 
be correct, from schema perspective, but fail to compile for a variety of 
reasons (think internals having changed on upgrade). In that case we still want 
to start up, so we manufacture a broken function stub. But, again, for UDA we 
shouldn’t. That said, I updated the code to throw a more verbose and helpful 
exception.

bq. UFTest / line #114 seems to have a typo - s/string1/string

Not a typo. Body needs to be different or else the UDF won’t be updated.

> Make all DDL statements idempotent and not dependent on global state
> 
>
> Key: CASSANDRA-13426
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13426
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Distributed Metadata
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 4.0
>
>
> A follow-up to CASSANDRA-9425 and a pre-requisite for CASSANDRA-10699.
> It's necessary for the latter to be able to apply any DDL statement several 
> times without side-effects. As part of the ticket I think we should also 
> clean up validation logic and our error texts. One example is varying 
> treatment of missing keyspace for DROP TABLE/INDEX/etc. statements with IF 
> EXISTS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13426) Make all DDL statements idempotent and not dependent on global state

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13426:
--
Status: Patch Available  (was: In Progress)

> Make all DDL statements idempotent and not dependent on global state
> 
>
> Key: CASSANDRA-13426
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13426
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Distributed Metadata
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>Priority: Major
> Fix For: 4.0
>
>
> A follow-up to CASSANDRA-9425 and a pre-requisite for CASSANDRA-10699.
> It's necessary for the latter to be able to apply any DDL statement several 
> times without side-effects. As part of the ticket I think we should also 
> clean up validation logic and our error texts. One example is varying 
> treatment of missing keyspace for DROP TABLE/INDEX/etc. statements with IF 
> EXISTS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445687#comment-16445687
 ] 

Stefan Podkowinski commented on CASSANDRA-13971:


I've now rebased my patch on trunk, which required to resolve some conflicts 
related to CASSANDRA-14222, CASSANDRA-14314. The dtest is now migrated to 
Python3 and pytest as well. Vault was also upgrade to 0.10.0, but that was 
trivial.

I've now attached a script {{start_vault_ssl.sh}} to the ticket. It will 
bootstrap a local PKI enabled vault instance and print the matching 
cassandra.yaml config. My suggestion would be to run it on the conf folder, 
append the output to the yaml and manually enable encryption afterwards. Hope 
this makes testing even easier.

As already mentioned, the trunk patch doesn't contain any new libraries or 
third party code. It's all HTTP REST communication with any Vault server over 
the network. Nothing added in lib.


> Automatic certificate management using Vault
> 
>
> Key: CASSANDRA-13971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>  Labels: security
> Fix For: 4.x
>
> Attachments: start_vault_ssl.sh
>
>
> We've been adding security features during the last years to enable users to 
> secure their clusters, if they are willing to use them and do so correctly. 
> Some features are powerful and easy to work with, such as role based 
> authorization. Other features that require to manage a local keystore are 
> rather painful to deal with. Think about setting up SSL..
> To be fair, keystore related issues and certificate handling hasn't been 
> invented by us. We're just following Java standards there. But that doesn't 
> mean that we absolutely have to, if there are better options. I'd like to 
> give it a shoot and find out if we can automate certificate/key handling 
> (PKI) by using external APIs. In this case, the implementation will be based 
> on [Vault|https://vaultproject.io]. But certificate management services 
> offered by cloud providers may also be able to handle the use-case and I 
> intend to create a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13971) Automatic certificate management using Vault

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13971:
---
Attachment: start_vault_ssl.sh

> Automatic certificate management using Vault
> 
>
> Key: CASSANDRA-13971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>  Labels: security
> Fix For: 4.x
>
> Attachments: start_vault_ssl.sh
>
>
> We've been adding security features during the last years to enable users to 
> secure their clusters, if they are willing to use them and do so correctly. 
> Some features are powerful and easy to work with, such as role based 
> authorization. Other features that require to manage a local keystore are 
> rather painful to deal with. Think about setting up SSL..
> To be fair, keystore related issues and certificate handling hasn't been 
> invented by us. We're just following Java standards there. But that doesn't 
> mean that we absolutely have to, if there are better options. I'd like to 
> give it a shoot and find out if we can automate certificate/key handling 
> (PKI) by using external APIs. In this case, the implementation will be based 
> on [Vault|https://vaultproject.io]. But certificate management services 
> offered by cloud providers may also be able to handle the use-case and I 
> intend to create a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14402) Remove StreamCoordinator.streamExecutor thread pool

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14402:

   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

committed as sha {{4af23348ecd6fc8dbef44ac5ebdb6ae60d599283}}

> Remove StreamCoordinator.streamExecutor thread pool
> ---
>
> Key: CASSANDRA-14402
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14402
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> {{StreamCoordinator.streamExecutor}} was previously introduced to initiate 
> stream connections on a separate thread from the session invocation logic. 
> With CASSANDRA-12229 streaming now uses non-blocking IO, and connection 
> establishment is asynchronous via netty. Thus, the thread pool in 
> {{StreamCoordinator}} is unneeded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Remove StreamCoordinator.streamExecutor thread pool

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/trunk a8be43e45 -> 4af23348e


Remove StreamCoordinator.streamExecutor thread pool

patch by jasobrown; reviewed by Dinesh Joshi for CASANDRA-14402


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4af23348
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4af23348
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4af23348

Branch: refs/heads/trunk
Commit: 4af23348ecd6fc8dbef44ac5ebdb6ae60d599283
Parents: a8be43e
Author: Jason Brown 
Authored: Thu Apr 19 05:33:34 2018 -0700
Committer: Jason Brown 
Committed: Fri Apr 20 05:05:51 2018 -0700

--
 CHANGES.txt |  1 +
 .../cassandra/streaming/StreamCoordinator.java  | 30 
 2 files changed, 7 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4af23348/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7d9769c..5902305 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Remove StreamCoordinator.streamExecutor thread pool (CASSANDRA-14402)
  * Rename nodetool --with-port to --print-port to disambiguate from --port 
(CASSANDRA-14392)
  * Client TOPOLOGY_CHANGE messages have wrong port. (CASSANDRA-14398)
  * Add ability to load new SSTables from a separate directory (CASSANDRA-6719)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4af23348/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
--
diff --git a/src/java/org/apache/cassandra/streaming/StreamCoordinator.java 
b/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
index 139488d..6b92dfe 100644
--- a/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
+++ b/src/java/org/apache/cassandra/streaming/StreamCoordinator.java
@@ -22,9 +22,7 @@ import java.util.*;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor;
 import org.apache.cassandra.locator.InetAddressAndPort;
-import org.apache.cassandra.utils.FBUtilities;
 
 /**
  * {@link StreamCoordinator} is a helper class that abstracts away maintaining 
multiple
@@ -37,15 +35,9 @@ public class StreamCoordinator
 {
 private static final Logger logger = 
LoggerFactory.getLogger(StreamCoordinator.class);
 
-/**
- * Executor strictly for establishing the initial connections. Once we're 
connected to the other end the rest of the
- * streaming is handled directly by the {@link StreamingMessageSender}'s 
incoming and outgoing threads.
- */
-private static final DebuggableThreadPoolExecutor streamExecutor = 
DebuggableThreadPoolExecutor.createWithFixedPoolSize("StreamConnectionEstablisher",
-   
 
FBUtilities.getAvailableProcessors());
 private final boolean connectSequentially;
 
-private Map peerSessions = new 
HashMap<>();
+private final Map peerSessions = 
new HashMap<>();
 private final StreamOperation streamOperation;
 private final int connectionsPerHost;
 private StreamConnectionFactory factory;
@@ -144,7 +136,7 @@ public class StreamCoordinator
 {
 StreamSession next = sessionsToConnect.next();
 logger.debug("Connecting next session {} with {}.", next.planId(), 
next.peer.toString());
-streamExecutor.execute(new StreamSessionConnector(next));
+startSession(next);
 }
 else
 logger.debug("Finished connecting all sessions");
@@ -259,20 +251,10 @@ public class StreamCoordinator
 return pendingRepair;
 }
 
-private static class StreamSessionConnector implements Runnable
+private void startSession(StreamSession session)
 {
-private final StreamSession session;
-public StreamSessionConnector(StreamSession session)
-{
-this.session = session;
-}
-
-@Override
-public void run()
-{
-session.start();
-logger.info("[Stream #{}, ID#{}] Beginning stream session with 
{}", session.planId(), session.sessionIndex(), session.peer);
-}
+session.start();
+logger.info("[Stream #{}, ID#{}] Beginning stream session with {}", 
session.planId(), session.sessionIndex(), session.peer);
 }
 
 private class HostStreamingData
@@ -316,7 +298,7 @@ public class StreamCoordinator
 {
 for (StreamSession session : 

[jira] [Commented] (CASSANDRA-14400) Subrange repair doesn't always mark as repaired

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445435#comment-16445435
 ] 

Kurt Greaves commented on CASSANDRA-14400:
--

Ah that sounds like the issue. I'll test that out.

> Subrange repair doesn't always mark as repaired
> ---
>
> Key: CASSANDRA-14400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14400
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Priority: Major
>
> So was just messing around with subrange repair on trunk and found that if I 
> generated an SSTable with a single token and then tried to repair that 
> SSTable using subrange repairs it wouldn't get marked as repaired.
>  
>  Before repair:
> {code:java}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> Repair command:
> {code}
> ccm node1 nodetool "repair -st -9223362383595311663 -et -9223362383595311661 
> aoeu"
> [2018-04-19 05:44:42,806] Starting repair command #7 
> (c23f76c0-4394-11e8-8f20-3b8ee110d005), repairing keyspace aoeu with repair 
> options (parallelism: parallel, primary range: false, incremental: true, job 
> threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind: 
> NONE, # of ranges: 1, pull repair: false, force repair: false, optimise 
> streams: false)
> [2018-04-19 05:44:42,843] Repair session c242d220-4394-11e8-8f20-3b8ee110d005 
> for range [(-9223362383595311663,-9223362383595311661]] finished (progress: 
> 20%)
> [2018-04-19 05:44:43,139] Repair completed successfully
> [2018-04-19 05:44:43,140] Repair command #7 finished in 0 seconds
> {code}
> After repair SSTable hasn't changed and sstablemetadata outputs:
> {code}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> And parent_repair_history states that the repair is complete/range was 
> successful:
> {code}
> select * from system_distributed.parent_repair_history where 
> parent_id=862395e0-4394-11e8-8f20-3b8ee110d005 ;
>  parent_id| columnfamily_names | 
> exception_message | exception_stacktrace | finished_at | 
> keyspace_name | options   
>   
>   
>  | requested_ranges   
>  | started_at  | successful_ranges
> --++---+--+-+---++-+-+-
>  862395e0-4394-11e8-8f20-3b8ee110d005 |   {'aoeu'} |  
> null | null | 2018-04-19 05:43:14.578000+ |  aoeu 
> | {'dataCenters': '', 'forceRepair': 'false', 'hosts': '', 'incremental': 
> 'true', 'jobThreads': '1', 'optimiseStreams': 'false', 'parallelism': 
> 'parallel', 'previewKind': 'NONE', 'primaryRange': 'false', 'pullRepair': 
> 'false', 'sub_range_repair': 'true', 'trace': 'false'} | 
> {'(-9223362383595311663,-9223362383595311661]'} | 2018-04-19 
> 05:43:01.952000+ | {'(-9223362383595311663,-9223362383595311661]'}
> {code}
> Subrange repairs seem to work fine over large ranges and set {{Repaired at}} 
> as expected, but I haven't figured out why it works for a large range versus 
> a small range so far.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14400) Subrange repair doesn't always mark as repaired

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445436#comment-16445436
 ] 

Kurt Greaves commented on CASSANDRA-14400:
--

Ah that sounds like the issue. I'll test that out.

> Subrange repair doesn't always mark as repaired
> ---
>
> Key: CASSANDRA-14400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14400
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Priority: Major
>
> So was just messing around with subrange repair on trunk and found that if I 
> generated an SSTable with a single token and then tried to repair that 
> SSTable using subrange repairs it wouldn't get marked as repaired.
>  
>  Before repair:
> {code:java}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> Repair command:
> {code}
> ccm node1 nodetool "repair -st -9223362383595311663 -et -9223362383595311661 
> aoeu"
> [2018-04-19 05:44:42,806] Starting repair command #7 
> (c23f76c0-4394-11e8-8f20-3b8ee110d005), repairing keyspace aoeu with repair 
> options (parallelism: parallel, primary range: false, incremental: true, job 
> threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind: 
> NONE, # of ranges: 1, pull repair: false, force repair: false, optimise 
> streams: false)
> [2018-04-19 05:44:42,843] Repair session c242d220-4394-11e8-8f20-3b8ee110d005 
> for range [(-9223362383595311663,-9223362383595311661]] finished (progress: 
> 20%)
> [2018-04-19 05:44:43,139] Repair completed successfully
> [2018-04-19 05:44:43,140] Repair command #7 finished in 0 seconds
> {code}
> After repair SSTable hasn't changed and sstablemetadata outputs:
> {code}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> And parent_repair_history states that the repair is complete/range was 
> successful:
> {code}
> select * from system_distributed.parent_repair_history where 
> parent_id=862395e0-4394-11e8-8f20-3b8ee110d005 ;
>  parent_id| columnfamily_names | 
> exception_message | exception_stacktrace | finished_at | 
> keyspace_name | options   
>   
>   
>  | requested_ranges   
>  | started_at  | successful_ranges
> --++---+--+-+---++-+-+-
>  862395e0-4394-11e8-8f20-3b8ee110d005 |   {'aoeu'} |  
> null | null | 2018-04-19 05:43:14.578000+ |  aoeu 
> | {'dataCenters': '', 'forceRepair': 'false', 'hosts': '', 'incremental': 
> 'true', 'jobThreads': '1', 'optimiseStreams': 'false', 'parallelism': 
> 'parallel', 'previewKind': 'NONE', 'primaryRange': 'false', 'pullRepair': 
> 'false', 'sub_range_repair': 'true', 'trace': 'false'} | 
> {'(-9223362383595311663,-9223362383595311661]'} | 2018-04-19 
> 05:43:01.952000+ | {'(-9223362383595311663,-9223362383595311661]'}
> {code}
> Subrange repairs seem to work fine over large ranges and set {{Repaired at}} 
> as expected, but I haven't figured out why it works for a large range versus 
> a small range so far.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-14400) Subrange repair doesn't always mark as repaired

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-14400:
-
Comment: was deleted

(was: Ah that sounds like the issue. I'll test that out.)

> Subrange repair doesn't always mark as repaired
> ---
>
> Key: CASSANDRA-14400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14400
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Priority: Major
>
> So was just messing around with subrange repair on trunk and found that if I 
> generated an SSTable with a single token and then tried to repair that 
> SSTable using subrange repairs it wouldn't get marked as repaired.
>  
>  Before repair:
> {code:java}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> Repair command:
> {code}
> ccm node1 nodetool "repair -st -9223362383595311663 -et -9223362383595311661 
> aoeu"
> [2018-04-19 05:44:42,806] Starting repair command #7 
> (c23f76c0-4394-11e8-8f20-3b8ee110d005), repairing keyspace aoeu with repair 
> options (parallelism: parallel, primary range: false, incremental: true, job 
> threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind: 
> NONE, # of ranges: 1, pull repair: false, force repair: false, optimise 
> streams: false)
> [2018-04-19 05:44:42,843] Repair session c242d220-4394-11e8-8f20-3b8ee110d005 
> for range [(-9223362383595311663,-9223362383595311661]] finished (progress: 
> 20%)
> [2018-04-19 05:44:43,139] Repair completed successfully
> [2018-04-19 05:44:43,140] Repair command #7 finished in 0 seconds
> {code}
> After repair SSTable hasn't changed and sstablemetadata outputs:
> {code}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> And parent_repair_history states that the repair is complete/range was 
> successful:
> {code}
> select * from system_distributed.parent_repair_history where 
> parent_id=862395e0-4394-11e8-8f20-3b8ee110d005 ;
>  parent_id| columnfamily_names | 
> exception_message | exception_stacktrace | finished_at | 
> keyspace_name | options   
>   
>   
>  | requested_ranges   
>  | started_at  | successful_ranges
> --++---+--+-+---++-+-+-
>  862395e0-4394-11e8-8f20-3b8ee110d005 |   {'aoeu'} |  
> null | null | 2018-04-19 05:43:14.578000+ |  aoeu 
> | {'dataCenters': '', 'forceRepair': 'false', 'hosts': '', 'incremental': 
> 'true', 'jobThreads': '1', 'optimiseStreams': 'false', 'parallelism': 
> 'parallel', 'previewKind': 'NONE', 'primaryRange': 'false', 'pullRepair': 
> 'false', 'sub_range_repair': 'true', 'trace': 'false'} | 
> {'(-9223362383595311663,-9223362383595311661]'} | 2018-04-19 
> 05:43:01.952000+ | {'(-9223362383595311663,-9223362383595311661]'}
> {code}
> Subrange repairs seem to work fine over large ranges and set {{Repaired at}} 
> as expected, but I haven't figured out why it works for a large range versus 
> a small range so far.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445431#comment-16445431
 ] 

Kurt Greaves commented on CASSANDRA-14346:
--

{quote}What I'm assuming Kurt meant is the fact that we rely on an unbroken jmx 
connection on the repair client side.
{quote}
This, although I'll note I'm not yet super familiar with all the repair 
improvements in 4.0, but I was still under the impression that we're really 
relying on stable jmx connections because we don't yet properly handle every 
failure case in a repair and also detect running repairs. I really haven't 
tested this on trunk though so I can't say that with absolute confidence.

But in general, I'm saying that any improvements in that domain should be our 
first step for this task and worry about the scheduling afterwards. This way we 
can ensure that stuff gets into 4.0 and that third party developers can also 
utilise it rather than everything being tied into the scheduling patch.

 bq. If so we can definitely do the splitting for incremental as well. We like 
splitting up the token ranges into similarly sized pieces because it makes the 
timeout logic much easier to reason about (long running repairs are super 
annoying to tell if they are stuck or not).
This was what I was getting at, but if you're doing subrange in trunk 
incremental is the default so this optimisation is already possible.

bq. but you really shouldn't do subrange incremental repairs unless you have a 
really good reason, since you'll do a lot of additional anti-compaction. 
Anyway, as long as you're running incremental repair regularly, you should be 
able to repair full token ranges in less than 30 min.
Yeah, this is a downside I'm mildly concerned about, however as noted if you 
repair regularly (which with scheduling why not?) this problem goes away. 
However it's mostly going to be the initial repair that's problematic, but 
there are options there (could be done as full repair). It's also worth noting 
that CASSANDRA-10540 would mostly fix this problem 


> Scheduled Repair in Cassandra
> -
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Repair
>Reporter: Joseph Lynch
>Priority: Major
>  Labels: CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes 
> sense given that it is necessary to give our users eventual consistency. Most 
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked 
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), 
> which we spoke about last year at NGCC. Given the positive feedback at NGCC 
> we focussed on getting it production ready and have now been using it in 
> production to repair hundreds of clusters, tens of thousands of nodes, and 
> petabytes of data for the past six months. Also based on feedback at NGCC we 
> have invested effort in figuring out how to integrate this natively into 
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our 
> implementation into Cassandra, and have created a [design 
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
>  showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would 
> be greatly appreciated about the interface or v1 implementation features. I 
> have tried to call out in the document features which we explicitly consider 
> future work (as well as a path forward to implement them in the future) 
> because I would very much like to get this done before the 4.0 merge window 
> closes, and to do that I think aggressively pruning scope is going to be a 
> necessity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-10540) RangeAwareCompaction

[Lerh Chuan Low (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445374#comment-16445374
 ] 

Lerh Chuan Low edited comment on CASSANDRA-10540 at 4/20/18 6:38 AM:
-

Hey Marcus, thanks for getting back so quickly :)

The big motivation for us is repairs...because when vnodes are turned on, every 
SSTable has many vnodes in them...so when a (incremental) repair happens, the 
range it is interested in gets anticompacted out. After that the next range 
gets anticompacted out and so on. RACS solves that big pain. 

Besides making the SSTables per read much nicer, it's like a LCS on steroids. I 
think there are other benefits of maintaining each SSTable to one token 
range...but I can't quite remember any more off the top of my head. 

So I am hoping it doesn't come to grouping the vnodesunless it's a last 
resort. 

Currently it looks like you create a RACS for each of the 
repaired/unrepaired/pending repairs set, and each RACS keeps track of the 
compaction strategies it is in charge of (which are all of the same class). The 
CS instances are lazily initiated (so that's a win right there) until needed. 
It seems to be that the reason why we want so many CS instances is so that each 
of them can keep track of their own SSTables (which all belong to that single 
token range). 

How about if RACS doesn't instantiate the individual CS instances? It keeps 
track of all the SSTables in the CF like other CS instances - just that the 
logic on which SSTables to involve in a compaction is done in RACS. So we can 
make RACS check L0 and if there are none, L1 would involve grouping the 
SSTables by range and then calling the next background task for the 
underlying/wrapped CS instances and submitting them. 

In this way, the downside is calculating the grouping each time we ask for the 
next background task. We could also store it in memory in the form of a 
manifest like in LCS? So an array with the SSTables in each of them - beats 
having 256 instances but we're still going to have a 256 sized array in memory, 
I guess. It just seems so starkingly similar to a LCS restricted to just L0 and 
L1. 

A final thought: Is the memory footprint actually significant enough for us to 
want to not bite the bullet and further group the vnodes because the gains from 
having each SSTable as a single range is a lot, simple is a feature and RACS is 
customizable? 

Please excuse my ignorance if none of those suggestions made sense/worked, 
still not very confident with the code base..
  
(Btw also feel free to let me know if you would like a hand with anything)


was (Author: lerh low):
Hey Marcus, thanks for getting back so quickly :)

The big motivation for us is repairs...because when vnodes are turned on, every 
SSTable has many vnodes in them...so when a (incremental) repair happens, the 
range it is interested in gets anticompacted out. After that the next range 
gets anticompacted out and so on. RACS solves that big pain. 

Besides making the SSTables per read much nicer, it's like a LCS on steroids. I 
think there are other benefits of maintaining each SSTable to one token 
range...but I can't quite remember any more off the top of my head. 

So I am hoping it doesn't come to grouping the vnodesunless it's a last 
resort. 

Currently it looks like you create a RACS for each of the 
repaired/unrepaired/pending repairs set, and each RACS keeps track of the 
compaction strategies it is in charge of (which are all of the same class). The 
CS instances are lazily initiated (so that's a win right there) until needed. 
It seems to be that the reason why we want so many CS instances is so that each 
of them can keep track of their own SSTables (which all belong to that single 
token range). 

How about if RACS doesn't instantiate the individual CS instances? It keeps 
track of all the SSTables in the CF like other CS instances - just that the 
logic on which SSTables to involve in a compaction is done in RACS. So we can 
make RACS check L0 and if there are none, L1 would involve grouping the 
SSTables by range and then calling the next background task for the 
underlying/wrapped CS instances and submitting them. 

In this way, the downside is calculating the grouping each time we ask for the 
next background task. We could also store it in memory in the form of a 
manifest like in LCS? So an array with the SSTables in each of them - beats 
having 256 instances but we're still going to have a 256 sized array in memory, 
I guess. It just seems so starkingly similar to a LCS restricted to just L0 and 
L1. 

A final thought: Is the memory footprint actually significant enough for us to 
want to not bite the bullet and further group the vnodes because the gains from 
having each SSTable as a single range is a lot, simple is a feature and RACS is 
customizable? 

Please excuse my ignorance if 

[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction

[Lerh Chuan Low (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445374#comment-16445374
 ] 

Lerh Chuan Low commented on CASSANDRA-10540:


Hey Marcus, thanks for getting back so quickly :)

The big motivation for us is repairs...because when vnodes are turned on, every 
SSTable has many vnodes in them...so when a (incremental) repair happens, the 
range it is interested in gets anticompacted out. After that the next range 
gets anticompacted out and so on. RACS solves that big pain. 

Besides making the SSTables per read much nicer, it's like a LCS on steroids. I 
think there are other benefits of maintaining each SSTable to one token 
range...but I can't quite remember any more off the top of my head. 

So I am hoping it doesn't come to grouping the vnodesunless it's a last 
resort. 

Currently it looks like you create a RACS for each of the 
repaired/unrepaired/pending repairs set, and each RACS keeps track of the 
compaction strategies it is in charge of (which are all of the same class). The 
CS instances are lazily initiated (so that's a win right there) until needed. 
It seems to be that the reason why we want so many CS instances is so that each 
of them can keep track of their own SSTables (which all belong to that single 
token range). 

How about if RACS doesn't instantiate the individual CS instances? It keeps 
track of all the SSTables in the CF like other CS instances - just that the 
logic on which SSTables to involve in a compaction is done in RACS. So we can 
make RACS check L0 and if there are none, L1 would involve grouping the 
SSTables by range and then calling the next background task for the 
underlying/wrapped CS instances and submitting them. 

In this way, the downside is calculating the grouping each time we ask for the 
next background task. We could also store it in memory in the form of a 
manifest like in LCS? So an array with the SSTables in each of them - beats 
having 256 instances but we're still going to have a 256 sized array in memory, 
I guess. It just seems so starkingly similar to a LCS restricted to just L0 and 
L1. 

A final thought: Is the memory footprint actually significant enough for us to 
want to not bite the bullet and further group the vnodes because the gains from 
having each SSTable as a single range is a lot, simple is a feature and RACS is 
customizable? 

Please excuse my ignorance if none of those suggestions made sense/worked, 
still not very confident with the code base..
 

> RangeAwareCompaction
> 
>
> Key: CASSANDRA-10540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10540
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
>  Labels: compaction, lcs, vnodes
> Fix For: 4.x
>
>
> Broken out from CASSANDRA-6696, we should split sstables based on ranges 
> during compaction.
> Requirements;
> * dont create tiny sstables - keep them bunched together until a single vnode 
> is big enough (configurable how big that is)
> * make it possible to run existing compaction strategies on the per-range 
> sstables
> We should probably add a global compaction strategy parameter that states 
> whether this should be enabled or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org