[jira] [Assigned] (CASSANDRA-14347) Read payload metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Chella reassigned CASSANDRA-14347: Assignee: Sumanth Pasupuleti > Read payload metrics > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > > We currently have MutationSizeHistogram that gives an idea of write payloads. > This JIRA is about adding similar capability to reads, which can benefit us > with the following > * Histogram of payload sizes at Prepared Statement level > * Count of queries that meet vs do not meet SLO w.r.t. payload size > We could also log queries that result in payload exceeding SLO. This can > prove useful in fire-fighting an incident, and can help resolve the issue > since its easy to narrow down to the specific culprit prepared statement. > Read payload metrics could potentially be derived using network statistics, > however, it is difficult to isolate payloads due to reads, given that there > could be a bunch of other operations in C* that can amount to observed > network traffic (like repairs, streaming, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Description: We currently have MutationSizeHistogram that gives an idea of write payloads. This JIRA is about adding similar capability to reads, which can benefit us with the following * Histogram of payload sizes at Prepared Statement level * Count of queries that meet vs do not meet SLO w.r.t. payload size We could also log queries that result in payload exceeding SLO. This can prove useful in fire-fighting an incident, and can help resolve the issue since its easy to narrow down to the specific culprit prepared statement. Read payload metrics could potentially be derived using network statistics, however, it is difficult to isolate payloads due to reads, given that there could be a bunch of other operations in C* that can amount to observed network traffic (like repairs, streaming, etc). was: We currently have MutationSizeHistogram that gives an idea of write payloads. This JIRA is about adding similar capability to reads, which can benefit us with the following * Histogram of payload sizes at Prepared Statement level * Count of queries that meet vs do not meet SLO w.r.t. payload size We could also log queries that result in payload exceeding SLO. Read payload metrics could potentially be derived using network statistics, however, it is difficult to isolate payloads due to reads, given that there could be a bunch of other operations in C* that can amount to observed network traffic (like repairs, streaming, etc). > Read payload metrics > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > > We currently have MutationSizeHistogram that gives an idea of write payloads. > This JIRA is about adding similar capability to reads, which can benefit us > with the following > * Histogram of payload sizes at Prepared Statement level > * Count of queries that meet vs do not meet SLO w.r.t. payload size > We could also log queries that result in payload exceeding SLO. This can > prove useful in fire-fighting an incident, and can help resolve the issue > since its easy to narrow down to the specific culprit prepared statement. > Read payload metrics could potentially be derived using network statistics, > however, it is difficult to isolate payloads due to reads, given that there > could be a bunch of other operations in C* that can amount to observed > network traffic (like repairs, streaming, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Summary: Read payload metrics (was: Read payload metrics/logging) > Read payload metrics > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > > We currently have MutationSizeHistogram that gives an idea of write payloads. > This JIRA is about adding similar capability to reads, which can benefit us > with the following > * Histogram of payload sizes at Prepared Statement level > * Count of queries that meet vs do not meet SLO w.r.t. payload size > We could also log queries that result in payload exceeding SLO. > Read payload metrics could potentially be derived using network statistics, > however, it is difficult to isolate payloads due to reads, given that there > could be a bunch of other operations in C* that can amount to observed > network traffic (like repairs, streaming, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics/logging
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Description: We currently have MutationSizeHistogram that gives an idea of write payloads. This JIRA is about adding similar capability to reads, which can benefit us with the following * Histogram of payload sizes at Prepared Statement level * Count of queries that meet vs do not meet SLO w.r.t. payload size We could also log queries that result in payload exceeding SLO. Read payload metrics could potentially be derived using network statistics, however, it is difficult to isolate payloads due to reads, given that there could be a bunch of other operations in C* that can amount to observed network traffic (like repairs, streaming, etc). was: We currently have MutationSizeHistogram that gives an idea of write payloads. This JIRA is about adding similar capability to reads, which can benefit us with the following * Histogram of payload sizes at Prepared Statement level * Count of queries that meet vs do not meet SLO w.r.t. payload size * > Read payload metrics/logging > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > > We currently have MutationSizeHistogram that gives an idea of write payloads. > This JIRA is about adding similar capability to reads, which can benefit us > with the following > * Histogram of payload sizes at Prepared Statement level > * Count of queries that meet vs do not meet SLO w.r.t. payload size > We could also log queries that result in payload exceeding SLO. > Read payload metrics could potentially be derived using network statistics, > however, it is difficult to isolate payloads due to reads, given that there > could be a bunch of other operations in C* that can amount to observed > network traffic (like repairs, streaming, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics/logging
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Description: We currently have MutationSizeHistogram that gives an idea of write payloads. This JIRA is about adding similar capability to reads, which can benefit us with the following * Histogram of payload sizes at Prepared Statement level * Count of queries that meet vs do not meet SLO w.r.t. payload size * > Read payload metrics/logging > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > > We currently have MutationSizeHistogram that gives an idea of write payloads. > This JIRA is about adding similar capability to reads, which can benefit us > with the following > * Histogram of payload sizes at Prepared Statement level > * Count of queries that meet vs do not meet SLO w.r.t. payload size > * -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics/logging
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Component/s: (was: Repair) > Read payload metrics/logging > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics/logging
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Description: (was: There have been many attempts to automate repair in Cassandra, which makes sense given that it is necessary to give our users eventual consistency. Most recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked for ways to solve this problem. At Netflix we've built a scheduled repair service within Priam (our sidecar), which we spoke about last year at NGCC. Given the positive feedback at NGCC we focussed on getting it production ready and have now been using it in production to repair hundreds of clusters, tens of thousands of nodes, and petabytes of data for the past six months. Also based on feedback at NGCC we have invested effort in figuring out how to integrate this natively into Cassandra rather than open sourcing it as an external service (e.g. in Priam). As such, [~vinaykumarcse] and I would like to re-work and merge our implementation into Cassandra, and have created a [design document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] showing how we plan to make it happen, including the the user interface. As we work on the code migration from Priam to Cassandra, any feedback would be greatly appreciated about the interface or v1 implementation features. I have tried to call out in the document features which we explicitly consider future work (as well as a path forward to implement them in the future) because I would very much like to get this done before the 4.0 merge window closes, and to do that I think aggressively pruning scope is going to be a necessity.) > Read payload metrics/logging > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14347) Read payload metrics/logging
[ https://issues.apache.org/jira/browse/CASSANDRA-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumanth Pasupuleti updated CASSANDRA-14347: --- Labels: (was: CommunityFeedbackRequested) > Read payload metrics/logging > > > Key: CASSANDRA-14347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14347) Read payload metrics/logging
Sumanth Pasupuleti created CASSANDRA-14347: -- Summary: Read payload metrics/logging Key: CASSANDRA-14347 URL: https://issues.apache.org/jira/browse/CASSANDRA-14347 Project: Cassandra Issue Type: Improvement Components: Repair Reporter: Sumanth Pasupuleti Fix For: 4.0 There have been many attempts to automate repair in Cassandra, which makes sense given that it is necessary to give our users eventual consistency. Most recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked for ways to solve this problem. At Netflix we've built a scheduled repair service within Priam (our sidecar), which we spoke about last year at NGCC. Given the positive feedback at NGCC we focussed on getting it production ready and have now been using it in production to repair hundreds of clusters, tens of thousands of nodes, and petabytes of data for the past six months. Also based on feedback at NGCC we have invested effort in figuring out how to integrate this natively into Cassandra rather than open sourcing it as an external service (e.g. in Priam). As such, [~vinaykumarcse] and I would like to re-work and merge our implementation into Cassandra, and have created a [design document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] showing how we plan to make it happen, including the the user interface. As we work on the code migration from Priam to Cassandra, any feedback would be greatly appreciated about the interface or v1 implementation features. I have tried to call out in the document features which we explicitly consider future work (as well as a path forward to implement them in the future) because I would very much like to get this done before the 4.0 merge window closes, and to do that I think aggressively pruning scope is going to be a necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11181) Add broadcast_rpc_address to system.local
[ https://issues.apache.org/jira/browse/CASSANDRA-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416843#comment-16416843 ] ASF GitHub Bot commented on CASSANDRA-11181: GitHub user changliuau opened a pull request: https://github.com/apache/cassandra/pull/213 Add broadcast_rpc_address to system.local for CASSANDRA-11181 https://issues.apache.org/jira/browse/CASSANDRA-11181 Add broadcast_rpc_address to system.local You can merge this pull request into a Git repository by running: $ git pull https://github.com/changliuau/cassandra CASSANDRA-11181-add-broadcast-rpc-to-system-local Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra/pull/213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #213 > Add broadcast_rpc_address to system.local > - > > Key: CASSANDRA-11181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11181 > Project: Cassandra > Issue Type: Improvement > Components: Distributed Metadata >Reporter: Nick Bailey >Priority: Major > Labels: client-impacting > Fix For: 4.x > > > Right now it's impossible to get the broadcast_rpc_address of the node you > are connected to via the drivers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11559) Enhance node representation
[
https://issues.apache.org/jira/browse/CASSANDRA-11559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416572#comment-16416572
]
Alex Lourie commented on CASSANDRA-11559:
-
[~bdeggleston] I totally agree on VirtualEndpoint vs Endpoint.
There are indeed not that many changes except for using this changed object
where appropriate and initiating all the internal structures as required to
make it work.
Wrt to changing the Cassandra _behaviour_ - clearly that was not the purpose of
this ticket. This one is more of an infrastructure work to make the actual use
of it later. For example, https://issues.apache.org/jira/browse/CASSANDRA-12344
is going to rely on this one for solving node replacement issues when the
replacement node has same IP. But I'm also sure there are a lot of other
improvements that can be thought of further on, for instance, move
endpointStatesMap fields into this new Endpoint object, to make it easier for
Gossiper to handle this information.
I'm not yet overly familiar with the whole Cassandra to make other
suggestions,:) but would totally appreciate if there are any. If there's an
interest in improving other things that I don't know about (which is not that
hard), then I'd love to hear it and would definitely open tickets for any
suggestions.
Thanks.
> Enhance node representation
> ---
>
> Key: CASSANDRA-11559
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11559
> Project: Cassandra
> Issue Type: Sub-task
> Components: Distributed Metadata
>Reporter: Paulo Motta
>Assignee: Alex Lourie
>Priority: Minor
>
> We currently represent nodes as {{InetAddress}} objects on {{TokenMetadata}},
> what causes difficulties when replacing a node with the same address (see
> CASSANDRA-8523 and CASSANDRA-9244).
> Since CASSANDRA-4120 we index hosts by {{UUID}} in gossip, so I think it's
> time to move that representation to {{TokenMetadata}}.
> I propose representing nodes as {{InetAddress, UUID}} pairs on
> {{TokenMetadata}}, encapsulated in a {{VirtualNode}} interface, so it will
> backward compatible with the current representation, while still allowing us
> to enhance it in the future with additional metadata (and improved vnode
> handling) if needed.
> This change will probably affect interfaces of internal classes like
> {{TokenMetadata}} and {{AbstractReplicationStrategy}}, so I'd like to hear
> from integrators and other developers if it's possible to change these
> without major hassle or if we need to wait until 4.0.
> Besides updating {{TokenMetadata}} and {{AbstractReplicationStrategy}} (and
> subclasses), we will also need to replace all {{InetAddress}} uses with
> {{VirtualNode.getEndpoint()}} calls on {{StorageService}} and related classes
> and tests. We would probably already be able to replace some
> {{TokenMetadata.getHostId(InetAddress endpoint)}} calls with
> {{VirtualNode.getHostId()}}.
> While we will still be dealing with {{InetAddress}} on {{StorageService}} in
> this initial stage, in the future I think we should pass {{VirtualNode}}
> instances around and only translate from {{VirtualNode}} to {{InetAddress}}
> in the network layer.
> Public interfaces like {{IEndpointSnitch}} will not be affected by this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13529) cassandra-stress light-weight transaction support
[ https://issues.apache.org/jira/browse/CASSANDRA-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416534#comment-16416534 ] Jaydeepkumar Chovatia commented on CASSANDRA-13529: --- Hi [~djoshi3] I've incorporated your review comments, please find updated path details here: || trunk || | [patch | https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:CASSANDRA-13529?expand=1] | | [utest | https://circleci.com/gh/jaydeepkumar1984/cassandra/61] | > cassandra-stress light-weight transaction support > - > > Key: CASSANDRA-13529 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13529 > Project: Cassandra > Issue Type: New Feature > Components: Stress >Reporter: Jaydeepkumar Chovatia >Assignee: Jaydeepkumar Chovatia >Priority: Minor > Fix For: 4.x > > Attachments: 13529.txt, lwttest.yaml > > > It would be nice to have a light-weight transaction support in > cassandra-stress. > Although currently in cassandra-stress we can achieve light-weight > transaction partially by using static conditions like "IF col1 != null" or > "IF not EXIST". > If would be ideal to have full fledged light-weight transaction support like > "IF col1 = ? and col2 = ?". One way to implement is to read values from > Cassandra and use that in the condition so it will execute all the paxos > phases in Cassandra. > Please find git link for the patch: > https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:13529-trunk?expand=1 > ||trunk| > |[branch|https://github.com/jaydeepkumar1984/cassandra/tree/13529-trunk]| > |[utests|https://circleci.com/gh/jaydeepkumar1984/cassandra/8]| -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14297) Optional startup delay for peers should wait for count rather than percentage
[
https://issues.apache.org/jira/browse/CASSANDRA-14297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Lynch updated CASSANDRA-14297:
-
Labels: PatchAvailable (was: )
> Optional startup delay for peers should wait for count rather than percentage
> -
>
> Key: CASSANDRA-14297
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14297
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle
>Reporter: Joseph Lynch
>Priority: Minor
> Labels: PatchAvailable
>
> As I commented in CASSANDRA-13993, the current wait for functionality is a
> great step in the right direction, but I don't think that the current setting
> (70% of nodes in the cluster) is the right configuration option. First I
> think this because 70% will not protect against errors as if you wait for 70%
> of the cluster you could still very easily have {{UnavailableException}} or
> {{ReadTimeoutException}} exceptions. This is because if you have even two
> nodes down in different racks in a Cassandra cluster these exceptions are
> possible (or with the default {{num_tokens}} setting of 256 it is basically
> guaranteed). Second I think this option is not easy for operators to set, the
> only setting I could think of that would "just work" is 100%.
> I proposed in that ticket instead of having `block_for_peers_percentage`
> defaulting to 70%, we instead have `block_for_peers` as a count of nodes that
> are allowed to be down before the starting node makes itself available as a
> coordinator. Of course, we would still have the timeout to limit startup time
> and deal with really extreme situations (whole datacenters down etc).
> I started working on a patch for this change [on
> github|https://github.com/jasobrown/cassandra/compare/13993...jolynch:13993],
> and am happy to finish it up with unit tests and such if someone can
> review/commit it (maybe [~aweisberg]?).
> I think the short version of my proposal is we replace:
> {noformat}
> block_for_peers_percentage:
> {noformat}
> with either
> {noformat}
> block_for_peers:
> {noformat}
> or, if we want to do even better imo and enable advanced operators to finely
> tune this behavior (while still having good defaults that work for almost
> everyone):
> {noformat}
> block_for_peers_local_dc:
> block_for_peers_each_dc:
> block_for_peers_all_dcs:
> {noformat}
> For example if an operator knows that they must be available at
> {{LOCAL_QUORUM}} they would set {{block_for_peers_local_dc=1}}, if they use
> {{EACH_QUOURM}} they would set {{block_for_peers_local_dc=1}}, if they use
> {{QUORUM}} (RF=3, dcs=2) they would set {{block_for_peers_all_dcs=2}}.
> Naturally everything would of course have a timeout to prevent startup taking
> too long.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416498#comment-16416498 ] Joseph Lynch commented on CASSANDRA-14346: -- I uploaded a pdf just for folks that can't / don't want to access the gdoc. If possible, please make feature or design comments on the google doc itself rather than the comments so we can make sure we're addressing all the feedback. > Scheduled Repair in Cassandra > - > > Key: CASSANDRA-14346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14346 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Joseph Lynch >Priority: Major > Labels: CommunityFeedbackRequested > Fix For: 4.0 > > Attachments: ScheduledRepairV1_20180327.pdf > > > There have been many attempts to automate repair in Cassandra, which makes > sense given that it is necessary to give our users eventual consistency. Most > recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked > for ways to solve this problem. > At Netflix we've built a scheduled repair service within Priam (our sidecar), > which we spoke about last year at NGCC. Given the positive feedback at NGCC > we focussed on getting it production ready and have now been using it in > production to repair hundreds of clusters, tens of thousands of nodes, and > petabytes of data for the past six months. Also based on feedback at NGCC we > have invested effort in figuring out how to integrate this natively into > Cassandra rather than open sourcing it as an external service (e.g. in Priam). > As such, [~vinaykumarcse] and I would like to re-work and merge our > implementation into Cassandra, and have created a [design > document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] > showing how we plan to make it happen, including the the user interface. > As we work on the code migration from Priam to Cassandra, any feedback would > be greatly appreciated about the interface or v1 implementation features. I > have tried to call out in the document features which we explicitly consider > future work (as well as a path forward to implement them in the future) > because I would very much like to get this done before the 4.0 merge window > closes, and to do that I think aggressively pruning scope is going to be a > necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14346) Scheduled Repair in Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Lynch updated CASSANDRA-14346: - Attachment: (was: Public Discussion_ Scheduled Repair Design.pdf) > Scheduled Repair in Cassandra > - > > Key: CASSANDRA-14346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14346 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Joseph Lynch >Priority: Major > Labels: CommunityFeedbackRequested > Fix For: 4.0 > > Attachments: ScheduledRepairV1_20180327.pdf > > > There have been many attempts to automate repair in Cassandra, which makes > sense given that it is necessary to give our users eventual consistency. Most > recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked > for ways to solve this problem. > At Netflix we've built a scheduled repair service within Priam (our sidecar), > which we spoke about last year at NGCC. Given the positive feedback at NGCC > we focussed on getting it production ready and have now been using it in > production to repair hundreds of clusters, tens of thousands of nodes, and > petabytes of data for the past six months. Also based on feedback at NGCC we > have invested effort in figuring out how to integrate this natively into > Cassandra rather than open sourcing it as an external service (e.g. in Priam). > As such, [~vinaykumarcse] and I would like to re-work and merge our > implementation into Cassandra, and have created a [design > document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] > showing how we plan to make it happen, including the the user interface. > As we work on the code migration from Priam to Cassandra, any feedback would > be greatly appreciated about the interface or v1 implementation features. I > have tried to call out in the document features which we explicitly consider > future work (as well as a path forward to implement them in the future) > because I would very much like to get this done before the 4.0 merge window > closes, and to do that I think aggressively pruning scope is going to be a > necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14346) Scheduled Repair in Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Lynch updated CASSANDRA-14346: - Attachment: ScheduledRepairV1_20180327.pdf > Scheduled Repair in Cassandra > - > > Key: CASSANDRA-14346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14346 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Joseph Lynch >Priority: Major > Labels: CommunityFeedbackRequested > Fix For: 4.0 > > Attachments: ScheduledRepairV1_20180327.pdf > > > There have been many attempts to automate repair in Cassandra, which makes > sense given that it is necessary to give our users eventual consistency. Most > recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked > for ways to solve this problem. > At Netflix we've built a scheduled repair service within Priam (our sidecar), > which we spoke about last year at NGCC. Given the positive feedback at NGCC > we focussed on getting it production ready and have now been using it in > production to repair hundreds of clusters, tens of thousands of nodes, and > petabytes of data for the past six months. Also based on feedback at NGCC we > have invested effort in figuring out how to integrate this natively into > Cassandra rather than open sourcing it as an external service (e.g. in Priam). > As such, [~vinaykumarcse] and I would like to re-work and merge our > implementation into Cassandra, and have created a [design > document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] > showing how we plan to make it happen, including the the user interface. > As we work on the code migration from Priam to Cassandra, any feedback would > be greatly appreciated about the interface or v1 implementation features. I > have tried to call out in the document features which we explicitly consider > future work (as well as a path forward to implement them in the future) > because I would very much like to get this done before the 4.0 merge window > closes, and to do that I think aggressively pruning scope is going to be a > necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14346) Scheduled Repair in Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Lynch updated CASSANDRA-14346: - Attachment: Public Discussion_ Scheduled Repair Design.pdf > Scheduled Repair in Cassandra > - > > Key: CASSANDRA-14346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14346 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Joseph Lynch >Priority: Major > Labels: CommunityFeedbackRequested > Fix For: 4.0 > > Attachments: Public Discussion_ Scheduled Repair Design.pdf > > > There have been many attempts to automate repair in Cassandra, which makes > sense given that it is necessary to give our users eventual consistency. Most > recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked > for ways to solve this problem. > At Netflix we've built a scheduled repair service within Priam (our sidecar), > which we spoke about last year at NGCC. Given the positive feedback at NGCC > we focussed on getting it production ready and have now been using it in > production to repair hundreds of clusters, tens of thousands of nodes, and > petabytes of data for the past six months. Also based on feedback at NGCC we > have invested effort in figuring out how to integrate this natively into > Cassandra rather than open sourcing it as an external service (e.g. in Priam). > As such, [~vinaykumarcse] and I would like to re-work and merge our > implementation into Cassandra, and have created a [design > document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] > showing how we plan to make it happen, including the the user interface. > As we work on the code migration from Priam to Cassandra, any feedback would > be greatly appreciated about the interface or v1 implementation features. I > have tried to call out in the document features which we explicitly consider > future work (as well as a path forward to implement them in the future) > because I would very much like to get this done before the 4.0 merge window > closes, and to do that I think aggressively pruning scope is going to be a > necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14346) Scheduled Repair in Cassandra
Joseph Lynch created CASSANDRA-14346: Summary: Scheduled Repair in Cassandra Key: CASSANDRA-14346 URL: https://issues.apache.org/jira/browse/CASSANDRA-14346 Project: Cassandra Issue Type: Improvement Components: Repair Reporter: Joseph Lynch Fix For: 4.0 There have been many attempts to automate repair in Cassandra, which makes sense given that it is necessary to give our users eventual consistency. Most recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked for ways to solve this problem. At Netflix we've built a scheduled repair service within Priam (our sidecar), which we spoke about last year at NGCC. Given the positive feedback at NGCC we focussed on getting it production ready and have now been using it in production to repair hundreds of clusters, tens of thousands of nodes, and petabytes of data for the past six months. Also based on feedback at NGCC we have invested effort in figuring out how to integrate this natively into Cassandra rather than open sourcing it as an external service (e.g. in Priam). As such, [~vinaykumarcse] and I would like to re-work and merge our implementation into Cassandra, and have created a [design document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] showing how we plan to make it happen, including the the user interface. As we work on the code migration from Priam to Cassandra, any feedback would be greatly appreciated about the interface or v1 implementation features. I have tried to call out in the document features which we explicitly consider future work (as well as a path forward to implement them in the future) because I would very much like to get this done before the 4.0 merge window closes, and to do that I think aggressively pruning scope is going to be a necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13047) Point cqlsh help to the new doc
[ https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416364#comment-16416364 ] Jon Haddad commented on CASSANDRA-13047: [~ptbannister], Yes, I think we still need this. It looks like a small enough patch, the same patch should work on 3.0 and up. I'll assign to you. > Point cqlsh help to the new doc > --- > > Key: CASSANDRA-13047 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13047 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Priority: Major > Fix For: 4.0 > > > Cqlsh still points to the "old" textile CQL doc, but that's not really > maintain anymore now that we have the new doc (which include everything the > old doc had and more). We should update cqlsh to point to the new doc so we > can remove the old one completely. > Any takers? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13047) Point cqlsh help to the new doc
[ https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad reassigned CASSANDRA-13047: -- Assignee: Patrick Bannister > Point cqlsh help to the new doc > --- > > Key: CASSANDRA-13047 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13047 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Assignee: Patrick Bannister >Priority: Major > Fix For: 4.0 > > > Cqlsh still points to the "old" textile CQL doc, but that's not really > maintain anymore now that we have the new doc (which include everything the > old doc had and more). We should update cqlsh to point to the new doc so we > can remove the old one completely. > Any takers? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13884) sstableloader option to accept target keyspace name
[
https://issues.apache.org/jira/browse/CASSANDRA-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Zhuang updated CASSANDRA-13884:
---
Status: Patch Available (was: Open)
> sstableloader option to accept target keyspace name
> ---
>
> Key: CASSANDRA-13884
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13884
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Often as part of backup people store entire {{data}} directory. When they see
> some corruption in data then they would like to restore data in same cluster
> (for large clusters 200 nodes) but with different keyspace name.
> Currently {{sstableloader}} uses parent folder as {{keyspace}}, it would be
> nice to have an option to specify target keyspace name as part of
> {{sstableloader}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13884) sstableloader option to accept target keyspace name
[
https://issues.apache.org/jira/browse/CASSANDRA-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Zhuang updated CASSANDRA-13884:
---
Status: Ready to Commit (was: Patch Available)
> sstableloader option to accept target keyspace name
> ---
>
> Key: CASSANDRA-13884
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13884
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Often as part of backup people store entire {{data}} directory. When they see
> some corruption in data then they would like to restore data in same cluster
> (for large clusters 200 nodes) but with different keyspace name.
> Currently {{sstableloader}} uses parent folder as {{keyspace}}, it would be
> nice to have an option to specify target keyspace name as part of
> {{sstableloader}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13884) sstableloader option to accept target keyspace name
[
https://issues.apache.org/jira/browse/CASSANDRA-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416282#comment-16416282
]
Jay Zhuang commented on CASSANDRA-13884:
+1
I added a check for empty target keyspace: [Check if target keyspace is empty
and print error
message|https://github.com/cooldoger/cassandra/commit/9536d99b86dd37984f056cc525618e9a2e391a75]
|dTest|
|[519|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/519/]
|
> sstableloader option to accept target keyspace name
> ---
>
> Key: CASSANDRA-13884
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13884
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Often as part of backup people store entire {{data}} directory. When they see
> some corruption in data then they would like to restore data in same cluster
> (for large clusters 200 nodes) but with different keyspace name.
> Currently {{sstableloader}} uses parent folder as {{keyspace}}, it would be
> nice to have an option to specify target keyspace name as part of
> {{sstableloader}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14318) Debug logging can create massive performance issues
[
https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415923#comment-16415923
]
Alexander Dejanovski commented on CASSANDRA-14318:
--
For the record, the same tests on 3.11.2 didn't show any notable performance
difference between debug on and off :
Cassandra 3.11.2 debug on :
{noformat}
Results:
Op rate : 18 777 op/s [read_event_1: 3 165 op/s, read_event_2: 3 109 op/s,
read_event_3: 12 562 op/s]
Partition rate : 6 215 pk/s [read_event_1: 3 165 pk/s, read_event_2: 3 109
pk/s, read_event_3: 0 pk/s]
Row rate : 6 215 row/s [read_event_1: 3 165 row/s, read_event_2: 3 109 row/s,
read_event_3: 0 row/s]
Latency mean : 6,7 ms [read_event_1: 6,7 ms, read_event_2: 6,7 ms,
read_event_3: 6,6 ms]
Latency median : 5,0 ms [read_event_1: 5,0 ms, read_event_2: 5,0 ms,
read_event_3: 4,9 ms]
Latency 95th percentile : 15,6 ms [read_event_1: 15,5 ms, read_event_2: 15,9
ms, read_event_3: 15,5 ms]
Latency 99th percentile : 43,3 ms [read_event_1: 42,7 ms, read_event_2: 44,2
ms, read_event_3: 43,2 ms]
Latency 99.9th percentile : 82,0 ms [read_event_1: 80,3 ms, read_event_2: 82,4
ms, read_event_3: 82,1 ms]
Latency max : 272,4 ms [read_event_1: 272,4 ms, read_event_2: 268,7 ms,
read_event_3: 245,1 ms]
Total partitions : 330 970 [read_event_1: 165 386, read_event_2: 165 584,
read_event_3: 0]
Total errors : 0 [read_event_1: 0, read_event_2: 0, read_event_3: 0]
Total GC count : 42
Total GC memory : 13,102 GiB
Total GC time : 1,8 seconds
Avg GC time : 42,4 ms
StdDev GC time : 1,3 ms
Total operation time : 00:00:53{noformat}
Cassandra 3.11.2 debug off :
{noformat}
Results:
Op rate : 18 853 op/s [read_event_1: 3 138 op/s, read_event_2: 3 137 op/s,
read_event_3: 12 578 op/s]
Partition rate : 6 275 pk/s [read_event_1: 3 138 pk/s, read_event_2: 3 137
pk/s, read_event_3: 0 pk/s]
Row rate : 6 275 row/s [read_event_1: 3 138 row/s, read_event_2: 3 137 row/s,
read_event_3: 0 row/s]
Latency mean : 6,7 ms [read_event_1: 6,7 ms, read_event_2: 6,7 ms,
read_event_3: 6,7 ms]
Latency median : 5,0 ms [read_event_1: 5,1 ms, read_event_2: 5,1 ms,
read_event_3: 5,0 ms]
Latency 95th percentile : 15,5 ms [read_event_1: 15,5 ms, read_event_2: 15,6
ms, read_event_3: 15,4 ms]
Latency 99th percentile : 39,9 ms [read_event_1: 41,0 ms, read_event_2: 39,6
ms, read_event_3: 39,6 ms]
Latency 99.9th percentile : 73,3 ms [read_event_1: 73,4 ms, read_event_2: 71,6
ms, read_event_3: 73,6 ms]
Latency max : 367,0 ms [read_event_1: 240,5 ms, read_event_2: 250,3 ms,
read_event_3: 367,0 ms]
Total partitions : 332 852 [read_event_1: 166 447, read_event_2: 166 405,
read_event_3: 0]
Total errors : 0 [read_event_1: 0, read_event_2: 0, read_event_3: 0]
Total GC count : 46
Total GC memory : 14,024 GiB
Total GC time : 2,0 seconds
Avg GC time : 42,7 ms
StdDev GC time : 3,9 ms
Total operation time : 00:00:53{noformat}
The improvement over 2.2 is nice though :)
> Debug logging can create massive performance issues
> ---
>
> Key: CASSANDRA-14318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14318
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alexander Dejanovski
>Assignee: Alexander Dejanovski
>Priority: Major
> Labels: lhf, performance
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: cassandra-2.2-debug.yaml, debuglogging.png, flame22
> nodebug sjk svg.png, flame22-nodebug-sjk.svg, flame22-sjk.svg,
> flame_graph_snapshot.png
>
>
> Debug logging can involve in many cases (especially very low latency ones) a
> very important overhead on the read path in 2.2 as we've seen when upgrading
> clusters from 2.0 to 2.2.
> The performance impact was especially noticeable on the client side metrics,
> where p99 could go up to 10 times higher, while ClientRequest metrics
> recorded by Cassandra didn't show any overhead.
> Below shows latencies recorded on the client side with debug logging on
> first, and then without it :
> !debuglogging.png!
> We generated a flame graph before turning off debug logging that shows the
> read call stack is dominated by debug logging :
> !flame_graph_snapshot.png!
> I've attached the original flame graph for exploration.
> Once disabled, the new flame graph shows that the read call stack gets
> extremely thin, which is further confirmed by client recorded metrics :
> !flame22 nodebug sjk svg.png!
> The query pager code has been reworked since 3.0 and it looks like
> log.debug() calls are gone there, but for 2.2 users and to prevent such
> issues to appear with default settings, I really think debug logging should
> be disabled by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (CASSANDRA-14318) Debug logging can create massive performance issues
[
https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415846#comment-16415846
]
Alexander Dejanovski edited comment on CASSANDRA-14318 at 3/27/18 4:16 PM:
---
[~jjirsa]: apparently the ReadCallback class already logs at TRACE and not
DEBUG on the latest 2.2.
I've created the fix that downgrades debug logging to trace logging in the
query pager classes, and here are the results :
debug on - no fix :
{noformat}
Results:
op rate : 6681 [read_event_1:1109, read_event_2:1119, read_event_3:4452]
partition rate : 6681 [read_event_1:1109, read_event_2:1119, read_event_3:4452]
row rate : 6681 [read_event_1:1109, read_event_2:1119, read_event_3:4452]
latency mean : 19,1 [read_event_1:15,4, read_event_2:15,4, read_event_3:21,0]
latency median : 15,6 [read_event_1:14,2, read_event_2:14,0, read_event_3:16,3]
latency 95th percentile : 39,1 [read_event_1:28,4, read_event_2:28,6,
read_event_3:44,2]
latency 99th percentile : 75,6 [read_event_1:52,9, read_event_2:53,6,
read_event_3:87,7]
latency 99.9th percentile : 315,7 [read_event_1:101,0, read_event_2:110,1,
read_event_3:361,1]
latency max : 609,1 [read_event_1:319,6, read_event_2:315,9, read_event_3:609,1]
Total partitions : 993050 [read_event_1:164882, read_event_2:166381,
read_event_3:661787]
Total errors : 0 [read_event_1:0, read_event_2:0, read_event_3:0]
total gc count : 189
total gc mb : 56464
total gc time (s) : 7
avg gc time(ms) : 37
stdev gc time(ms) : 8
Total operation time : 00:02:28{noformat}
debug off - no fix :
{noformat}
Results:
op rate : 12655 [read_event_1:2141, read_event_2:2093, read_event_3:8422]
partition rate : 12655 [read_event_1:2141, read_event_2:2093, read_event_3:8422]
row rate : 12655 [read_event_1:2141, read_event_2:2093, read_event_3:8422]
latency mean : 10,1 [read_event_1:10,1, read_event_2:10,1, read_event_3:10,1]
latency median : 9,2 [read_event_1:9,2, read_event_2:9,2, read_event_3:9,3]
latency 95th percentile : 15,2 [read_event_1:15,8, read_event_2:15,9,
read_event_3:15,7]
latency 99th percentile : 29,3 [read_event_1:44,5, read_event_2:45,1,
read_event_3:41,3]
latency 99.9th percentile : 52,7 [read_event_1:67,9, read_event_2:66,9,
read_event_3:67,1]
latency max : 268,0 [read_event_1:257,1, read_event_2:263,3, read_event_3:268,0]
Total partitions : 983056 [read_event_1:166311, read_event_2:162570,
read_event_3:654175]
Total errors : 0 [read_event_1:0, read_event_2:0, read_event_3:0]
total gc count : 100
total gc mb : 31529
total gc time (s) : 4
avg gc time(ms) : 37
stdev gc time(ms) : 5
Total operation time : 00:01:17{noformat}
debug on - with fix :
{noformat}
Results:
op rate : 12289 [read_event_1:2058, read_event_2:2051, read_event_3:8181]
partition rate : 12289 [read_event_1:2058, read_event_2:2051, read_event_3:8181]
row rate : 12289 [read_event_1:2058, read_event_2:2051, read_event_3:8181]
latency mean : 10,4 [read_event_1:10,4, read_event_2:10,4, read_event_3:10,4]
latency median : 9,4 [read_event_1:9,4, read_event_2:9,4, read_event_3:9,4]
latency 95th percentile : 16,3 [read_event_1:16,8, read_event_2:17,3,
read_event_3:16,2]
latency 99th percentile : 36,6 [read_event_1:44,3, read_event_2:46,6,
read_event_3:37,2]
latency 99.9th percentile : 62,2 [read_event_1:78,0, read_event_2:77,1,
read_event_3:80,8]
latency max : 251,2 [read_event_1:246,9, read_event_2:249,9, read_event_3:251,2]
Total partitions : 100 [read_event_1:167422, read_event_2:166861,
read_event_3:665717]
Total errors : 0 [read_event_1:0, read_event_2:0, read_event_3:0]
total gc count : 102
total gc mb : 31843
total gc time (s) : 4
avg gc time(ms) : 38
stdev gc time(ms) : 6
Total operation time : 00:01:21{noformat}
So we have similar performance with debug logging off and with the fix and
debug on.
The difference in throughput is pretty massive as we roughly get *twice the
read throughput* with the fix.
Latencies without the fix and with the fix :
p95 : 35ms -> 16ms
p99 : 75ms -> 36ms
I've ran all tests several times, alternating with and without the fix to make
sure caches were not making a difference, and results were consistent with
what's pasted above.
It's been running on a single node using an i3.xlarge instance for Cassandra
and another i3.large for running cassandra-stress.
*One pretty interesting thing to note* is that when I tested with the
predefined mode of cassandra-stress, no paging occurred and the performance
difference was not noticeable. This is due to the fact that the predefined mode
generates COMPACT STORAGE tables, which involve a different read path
(apparently). I think anyone performing benchmarks for Cassandra changes should
be aware that the predefined mode isn't relevant and that a user defined test
should be used (maybe we should create one that would be used as standard
benchmark).
Here's the one I used :
[jira] [Commented] (CASSANDRA-14318) Debug logging can create massive performance issues
[
https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415846#comment-16415846
]
Alexander Dejanovski commented on CASSANDRA-14318:
--
[~jjirsa]: apparently the ReadCallback class already logs at TRACE and not
DEBUG on the latest 2.2.
I've created the fix that downgrades debug logging to trace logging in the
query pager classes, and here are the results :
debug on - no fix :
{noformat}
Results:
op rate : 6681 [read_event_1:1109, read_event_2:1119, read_event_3:4452]
partition rate : 6681 [read_event_1:1109, read_event_2:1119, read_event_3:4452]
row rate : 6681 [read_event_1:1109, read_event_2:1119, read_event_3:4452]
latency mean : 19,1 [read_event_1:15,4, read_event_2:15,4, read_event_3:21,0]
latency median : 15,6 [read_event_1:14,2, read_event_2:14,0, read_event_3:16,3]
latency 95th percentile : 39,1 [read_event_1:28,4, read_event_2:28,6,
read_event_3:44,2]
latency 99th percentile : 75,6 [read_event_1:52,9, read_event_2:53,6,
read_event_3:87,7]
latency 99.9th percentile : 315,7 [read_event_1:101,0, read_event_2:110,1,
read_event_3:361,1]
latency max : 609,1 [read_event_1:319,6, read_event_2:315,9, read_event_3:609,1]
Total partitions : 993050 [read_event_1:164882, read_event_2:166381,
read_event_3:661787]
Total errors : 0 [read_event_1:0, read_event_2:0, read_event_3:0]
total gc count : 189
total gc mb : 56464
total gc time (s) : 7
avg gc time(ms) : 37
stdev gc time(ms) : 8
Total operation time : 00:02:28{noformat}
debug off - no fix :
{noformat}
Results:
op rate : 12655 [read_event_1:2141, read_event_2:2093, read_event_3:8422]
partition rate : 12655 [read_event_1:2141, read_event_2:2093, read_event_3:8422]
row rate : 12655 [read_event_1:2141, read_event_2:2093, read_event_3:8422]
latency mean : 10,1 [read_event_1:10,1, read_event_2:10,1, read_event_3:10,1]
latency median : 9,2 [read_event_1:9,2, read_event_2:9,2, read_event_3:9,3]
latency 95th percentile : 15,2 [read_event_1:15,8, read_event_2:15,9,
read_event_3:15,7]
latency 99th percentile : 29,3 [read_event_1:44,5, read_event_2:45,1,
read_event_3:41,3]
latency 99.9th percentile : 52,7 [read_event_1:67,9, read_event_2:66,9,
read_event_3:67,1]
latency max : 268,0 [read_event_1:257,1, read_event_2:263,3, read_event_3:268,0]
Total partitions : 983056 [read_event_1:166311, read_event_2:162570,
read_event_3:654175]
Total errors : 0 [read_event_1:0, read_event_2:0, read_event_3:0]
total gc count : 100
total gc mb : 31529
total gc time (s) : 4
avg gc time(ms) : 37
stdev gc time(ms) : 5
Total operation time : 00:01:17{noformat}
debug on - with fix :
{noformat}
Results:
op rate : 12289 [read_event_1:2058, read_event_2:2051, read_event_3:8181]
partition rate : 12289 [read_event_1:2058, read_event_2:2051, read_event_3:8181]
row rate : 12289 [read_event_1:2058, read_event_2:2051, read_event_3:8181]
latency mean : 10,4 [read_event_1:10,4, read_event_2:10,4, read_event_3:10,4]
latency median : 9,4 [read_event_1:9,4, read_event_2:9,4, read_event_3:9,4]
latency 95th percentile : 16,3 [read_event_1:16,8, read_event_2:17,3,
read_event_3:16,2]
latency 99th percentile : 36,6 [read_event_1:44,3, read_event_2:46,6,
read_event_3:37,2]
latency 99.9th percentile : 62,2 [read_event_1:78,0, read_event_2:77,1,
read_event_3:80,8]
latency max : 251,2 [read_event_1:246,9, read_event_2:249,9, read_event_3:251,2]
Total partitions : 100 [read_event_1:167422, read_event_2:166861,
read_event_3:665717]
Total errors : 0 [read_event_1:0, read_event_2:0, read_event_3:0]
total gc count : 102
total gc mb : 31843
total gc time (s) : 4
avg gc time(ms) : 38
stdev gc time(ms) : 6
Total operation time : 00:01:21{noformat}
So we have similar performance with debug logging off and with the fix and
debug on.
The difference in throughput is pretty massive as we roughly get *twice the
read throughput* with the fix.
Latencies without the fix and with the fix :
p95 : 35ms -> 16ms
p99 : 75ms -> 36ms
I've ran all tests several times, alternating with and without the fix to make
sure caches were not making a difference, and results were consistent with
what's pasted above.
It's been running on a single node using an i3.xlarge instance for Cassandra
and another i3.large for running cassandra-stress.
*One pretty interesting thing to note* is that when I tested with the
predefined mode of cassandra-stress, no paging occurred and the performance
difference was not noticeable. This is due to the fact that the predefined mode
generates COMPACT STORAGE tables, which involve a different read path
(apparently). I think anyone performing benchmarks for Cassandra changes should
be aware that the predefined mode isn't relevant and that a user defined test
should be used (maybe we should create one that would be used as standard
benchmark).
Here's the one I used : [^cassandra-2.2-debug.yaml]
With the following commands for
[jira] [Updated] (CASSANDRA-14318) Debug logging can create massive performance issues
[ https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Dejanovski updated CASSANDRA-14318: - Attachment: cassandra-2.2-debug.yaml > Debug logging can create massive performance issues > --- > > Key: CASSANDRA-14318 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14318 > Project: Cassandra > Issue Type: Bug >Reporter: Alexander Dejanovski >Assignee: Alexander Dejanovski >Priority: Major > Labels: lhf, performance > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x > > Attachments: cassandra-2.2-debug.yaml, debuglogging.png, flame22 > nodebug sjk svg.png, flame22-nodebug-sjk.svg, flame22-sjk.svg, > flame_graph_snapshot.png > > > Debug logging can involve in many cases (especially very low latency ones) a > very important overhead on the read path in 2.2 as we've seen when upgrading > clusters from 2.0 to 2.2. > The performance impact was especially noticeable on the client side metrics, > where p99 could go up to 10 times higher, while ClientRequest metrics > recorded by Cassandra didn't show any overhead. > Below shows latencies recorded on the client side with debug logging on > first, and then without it : > !debuglogging.png! > We generated a flame graph before turning off debug logging that shows the > read call stack is dominated by debug logging : > !flame_graph_snapshot.png! > I've attached the original flame graph for exploration. > Once disabled, the new flame graph shows that the read call stack gets > extremely thin, which is further confirmed by client recorded metrics : > !flame22 nodebug sjk svg.png! > The query pager code has been reworked since 3.0 and it looks like > log.debug() calls are gone there, but for 2.2 users and to prevent such > issues to appear with default settings, I really think debug logging should > be disabled by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11559) Enhance node representation
[
https://issues.apache.org/jira/browse/CASSANDRA-11559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415840#comment-16415840
]
Blake Eggleston commented on CASSANDRA-11559:
-
I've taken a look at the patch, but didn't find anywhere that it really changes
the behavior of Cassandra, aside from renaming {{InetAddressAndPort}}. Am I
missing something? Also, not sure about the name of {{VirtualEndpoint}}. What's
virtual about it? Maybe just {{Endpoint}}
> Enhance node representation
> ---
>
> Key: CASSANDRA-11559
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11559
> Project: Cassandra
> Issue Type: Sub-task
> Components: Distributed Metadata
>Reporter: Paulo Motta
>Assignee: Alex Lourie
>Priority: Minor
>
> We currently represent nodes as {{InetAddress}} objects on {{TokenMetadata}},
> what causes difficulties when replacing a node with the same address (see
> CASSANDRA-8523 and CASSANDRA-9244).
> Since CASSANDRA-4120 we index hosts by {{UUID}} in gossip, so I think it's
> time to move that representation to {{TokenMetadata}}.
> I propose representing nodes as {{InetAddress, UUID}} pairs on
> {{TokenMetadata}}, encapsulated in a {{VirtualNode}} interface, so it will
> backward compatible with the current representation, while still allowing us
> to enhance it in the future with additional metadata (and improved vnode
> handling) if needed.
> This change will probably affect interfaces of internal classes like
> {{TokenMetadata}} and {{AbstractReplicationStrategy}}, so I'd like to hear
> from integrators and other developers if it's possible to change these
> without major hassle or if we need to wait until 4.0.
> Besides updating {{TokenMetadata}} and {{AbstractReplicationStrategy}} (and
> subclasses), we will also need to replace all {{InetAddress}} uses with
> {{VirtualNode.getEndpoint()}} calls on {{StorageService}} and related classes
> and tests. We would probably already be able to replace some
> {{TokenMetadata.getHostId(InetAddress endpoint)}} calls with
> {{VirtualNode.getHostId()}}.
> While we will still be dealing with {{InetAddress}} on {{StorageService}} in
> this initial stage, in the future I think we should pass {{VirtualNode}}
> instances around and only translate from {{VirtualNode}} to {{InetAddress}}
> in the network layer.
> Public interfaces like {{IEndpointSnitch}} will not be affected by this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14345) Empty partition keys allowed in MV, but not in normal table
[
https://issues.apache.org/jira/browse/CASSANDRA-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415605#comment-16415605
]
Duarte Nunes commented on CASSANDRA-14345:
--
This seems to be related how composites are serialized (as-is), versus how
multi-component are serialized ( ).
It's not clear to me whether the issue is at the MV level, or whether empty,
non-null keys should be allowed for normal tables too.
> Empty partition keys allowed in MV, but not in normal table
> ---
>
> Key: CASSANDRA-14345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14345
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Duarte Nunes
>Priority: Major
> Labels: materializedviews
>
> Given the following table:
>
> {code:java}
> cqlsh> create keyspace ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> cqlsh> create table t (p text, c int, v text, primary key (p));
> cqlsh> use ks;
> {code}
> The following fails:
> {code:java}
> cqlsh:ks> insert into t (p, c, v) values ('', 2, '');
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Key
> may not be empty"{code}
> However, MVs don't appear to have this restriction:
> {code:java}
> create materialized view mv as select * from t where v is not null and p is
> not null and c is not null primary key (v, p);
> insert into t (p, c, v) values ('a', 2, '');
> select * from mv;
> v | p | c
> ---+---+---
>| a | 2
> {code}
> I think the behavior should be made consistent, if nothing else because
> querying the MV for the empty key is impossible:
> {code:java}
> cqlsh:ks> select * from mv where v = '';
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Key
> may not be empty"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14345) Empty partition keys allowed in MV, but not in normal table
[
https://issues.apache.org/jira/browse/CASSANDRA-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415529#comment-16415529
]
Duarte Nunes commented on CASSANDRA-14345:
--
It's also puzzling that I can have an empty partition key if it's multicolumn:
{code:sql}
create table t (a text, b text, c text, primary key ((a, b)))
cqlsh:ks> insert into t (a, b, c) values ('', '', '');
cqlsh:ks> select * from t;
a | b | c
---+---+---
| |
{code}
> Empty partition keys allowed in MV, but not in normal table
> ---
>
> Key: CASSANDRA-14345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14345
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Duarte Nunes
>Priority: Major
> Labels: materializedviews
>
> Given the following table:
>
> {code:java}
> cqlsh> create keyspace ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> cqlsh> create table t (p text, c int, v text, primary key (p));
> cqlsh> use ks;
> {code}
> The following fails:
> {code:java}
> cqlsh:ks> insert into t (p, c, v) values ('', 2, '');
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Key
> may not be empty"{code}
> However, MVs don't appear to have this restriction:
> {code:java}
> create materialized view mv as select * from t where v is not null and p is
> not null and c is not null primary key (v, p);
> insert into t (p, c, v) values ('a', 2, '');
> select * from mv;
> v | p | c
> ---+---+---
>| a | 2
> {code}
> I think the behavior should be made consistent, if nothing else because
> querying the MV for the empty key is impossible:
> {code:java}
> cqlsh:ks> select * from mv where v = '';
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Key
> may not be empty"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14345) Empty partition keys allowed in MV, but not in normal table
[
https://issues.apache.org/jira/browse/CASSANDRA-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duarte Nunes updated CASSANDRA-14345:
-
Description:
Given the following table:
{code:java}
cqlsh> create keyspace ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
cqlsh> create table t (p text, c int, v text, primary key (p));
cqlsh> use ks;
{code}
The following fails:
{code:java}
cqlsh:ks> insert into t (p, c, v) values ('', 2, '');
InvalidRequest: Error from server: code=2200 [Invalid query] message="Key may
not be empty"{code}
However, MVs don't appear to have this restriction:
{code:java}
create materialized view mv as select * from t where v is not null and p is not
null and c is not null primary key (v, p);
insert into t (p, c, v) values ('a', 2, '');
select * from mv;
v | p | c
---+---+---
| a | 2
{code}
I think the behavior should be made consistent, if nothing else because
querying the MV for the empty key is impossible:
{code:java}
cqlsh:ks> select * from mv where v = '';
InvalidRequest: Error from server: code=2200 [Invalid query] message="Key may
not be empty"{code}
was:
Given the following table:
{code:java}
cqlsh> create keyspace ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
cqlsh> create table t (p text, c int, v text, primary key (p));
cqlsh> use ks;
{code}
The following fails:
{code:java}
cqlsh:ks> insert into t (p, c, v) values ('', 2, '');
InvalidRequest: Error from server: code=2200 [Invalid query] message="Key may
not be empty"{code}
However, MVs don't appear to have this restriction:
{code:java}
create materialized view mv as select * from t where v is not null and p is not
null and c is not null primary key (v, p);
insert into t (p, c, v) values ('a', 2, '');
select * from mv;
v | p | c
---+---+---
| a | 2
{code}
I guess this is because an empty value can't be distinguished from null at the
protocol level, but this distinction can be made internally.
I think the behavior should be made consistent, if nothing else because
querying the MV for the empty key is impossible:
{code:java}
cqlsh:ks> select * from mv where v = '';
InvalidRequest: Error from server: code=2200 [Invalid query] message="Key may
not be empty"{code}
> Empty partition keys allowed in MV, but not in normal table
> ---
>
> Key: CASSANDRA-14345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14345
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Duarte Nunes
>Priority: Major
> Labels: materializedviews
>
> Given the following table:
>
> {code:java}
> cqlsh> create keyspace ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> cqlsh> create table t (p text, c int, v text, primary key (p));
> cqlsh> use ks;
> {code}
> The following fails:
> {code:java}
> cqlsh:ks> insert into t (p, c, v) values ('', 2, '');
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Key
> may not be empty"{code}
> However, MVs don't appear to have this restriction:
> {code:java}
> create materialized view mv as select * from t where v is not null and p is
> not null and c is not null primary key (v, p);
> insert into t (p, c, v) values ('a', 2, '');
> select * from mv;
> v | p | c
> ---+---+---
>| a | 2
> {code}
> I think the behavior should be made consistent, if nothing else because
> querying the MV for the empty key is impossible:
> {code:java}
> cqlsh:ks> select * from mv where v = '';
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Key
> may not be empty"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13529) cassandra-stress light-weight transaction support
[
https://issues.apache.org/jira/browse/CASSANDRA-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415113#comment-16415113
]
Dinesh Joshi commented on CASSANDRA-13529:
--
Hi [~chovatia.jayd...@gmail.com], looks nice. Some more things we can improve
on -
# # {{CASQuery}} - {{bindArgs, fillRandom, random, randomBuffer}} are not
used. Please remove.
# {{PartitionGenerator}} - {{partitionKey}} and {{clusteringComponents}} - use
getters instead of making fields public and return immutable copies of the
lists.
# {{CASQuery::prepareCASDynamicConditionsReadStatement}} - on line 138 instead
of forced typecasting use {{Math.toIntExact}}
# {{CASQuery::prepareCASDynamicConditionsReadStatement}} - on line 135 {{new
HashMap()}} is unchecked assignment. This should be {{new HashMap<>()}}.
# {{keysIndex = new ArrayList();}} use {{<>}} operator
# Please add a sample profile in {{tools}} directory and update the in-tree
docs ({{eg. doc/tools/cassandra_stress.rst}})
Nits:
# {{CASQuery - PartitionIterator, Random}} - unused imports. Please remove.
# {{CASQuery::prepareCASDynamicConditionsReadStatement}} -
{{modificationStatement}} do you actually need to initialize this to {{null}}?
# {{QueryUtil::dynamicConditionsExist}} - typo in comment - `differenciate` ->
`differentiate`.
> cassandra-stress light-weight transaction support
> -
>
> Key: CASSANDRA-13529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13529
> Project: Cassandra
> Issue Type: New Feature
> Components: Stress
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
> Attachments: 13529.txt, lwttest.yaml
>
>
> It would be nice to have a light-weight transaction support in
> cassandra-stress.
> Although currently in cassandra-stress we can achieve light-weight
> transaction partially by using static conditions like "IF col1 != null" or
> "IF not EXIST".
> If would be ideal to have full fledged light-weight transaction support like
> "IF col1 = ? and col2 = ?". One way to implement is to read values from
> Cassandra and use that in the condition so it will execute all the paxos
> phases in Cassandra.
> Please find git link for the patch:
> https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:13529-trunk?expand=1
> ||trunk|
> |[branch|https://github.com/jaydeepkumar1984/cassandra/tree/13529-trunk]|
> |[utests|https://circleci.com/gh/jaydeepkumar1984/cassandra/8]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
