[jira] [Commented] (CASSANDRA-15045) Fix index summary redistribution compaction cancellation issues
[ https://issues.apache.org/jira/browse/CASSANDRA-15045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797703#comment-16797703 ] Jordan West commented on CASSANDRA-15045: - +1 > Fix index summary redistribution compaction cancellation issues > --- > > Key: CASSANDRA-15045 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15045 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x > > > We can't cancel ongoing index summary redistributions currently due to > {{CompactionInfo}} returning null for {{getTableMetadata/getCFMetaData}} > [here|https://github.com/apache/cassandra/blob/67d613204fa4fb9584f11ec9886a0e7a0d622e92/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1814] > for index summary redistributions > CASSANDRA-14935 also introduced a bug where we track the wrong sstables for > index summary redistributions -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Add Ubuntu 18.10 Java 11 dockerfile
This is an automated email from the ASF dual-hosted git repository. spod pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new 59169b2 Add Ubuntu 18.10 Java 11 dockerfile 59169b2 is described below commit 59169b24af35314eb0439961687cc5435008d949 Author: Stefan Podkowinski AuthorDate: Thu Jan 3 08:57:21 2019 +0100 Add Ubuntu 18.10 Java 11 dockerfile --- docker/testing/README.md | 35 +++ docker/testing/ubuntu1810_j11.docker | 107 + .../testing/ubuntu1810_j11_w_dependencies.docker | 37 +++ 3 files changed, 179 insertions(+) diff --git a/docker/testing/README.md b/docker/testing/README.md new file mode 100644 index 000..99a88e9 --- /dev/null +++ b/docker/testing/README.md @@ -0,0 +1,35 @@ +# Docker CI Testing + +Docker files in this directory are used to build images used by CircleCI. These are directly referenced in the `circle.yml` after publishing to dockerhub. There are two types of images: + +* Base image for Linux distribution to use for testing (e.g. `ubuntu1810_j11.docker`) +* Caching image that contains git sources, maven and ccm dependencies + +## Building Images + +Build images from the parent directory using the following commands. Change tag (`-t`) as needed (prefix and current date): + +Base image: + +`docker build -t spod/cassandra-testing-ubuntu1810:20180128 -t spod/cassandra-testing-ubuntu1810:latest -f testing/ubuntu1810_j11.docker testing/` + +Caching image: + +`docker build --no-cache -t spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 -t spod/cassandra-testing-ubuntu1810-java11-w-dependencies:latest -f testing/ubuntu1810_j11_w_dependencies.docker testing/` + +Please make sure to always tag also by date, so we can go back to that version in case anything breaks after the next update! + +## Publishing Images + +We are using Docker Hub for storing published images. See [Quick Start Guide](https://docs.docker.com/docker-hub/) on how to setup docker. + +Push both image references: + +``` +docker push spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 +docker push spod/cassandra-testing-ubuntu1810-java11-w-dependencies:latest +``` + +## Updating circleci.yml + +Check if the correct image is used in `.circleci/config-2_1.yml` by looking for the `- image:` value. It should either be set to the date dervived tag created above, or `:latest`. diff --git a/docker/testing/ubuntu1810_j11.docker b/docker/testing/ubuntu1810_j11.docker new file mode 100644 index 000..328cd31 --- /dev/null +++ b/docker/testing/ubuntu1810_j11.docker @@ -0,0 +1,107 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# base things off the latest LTS Ubuntu Release (18.04) +FROM ubuntu:18.10 +MAINTAINER Stefan Podkowinski + +# install our python dependencies and some other stuff we need +# libev4 libev-dev are for the python driver + +RUN export DEBIAN_FRONTEND=noninteractive && \ +apt-get update && \ +apt-get install -y --no-install-recommends software-properties-common apt-utils vim + +RUN export DEBIAN_FRONTEND=noninteractive && \ +apt-get update && \ +apt-get install -y git-core python2.7 python3.6 python3.6-venv python3.6-dev python3-pip net-tools libev4 libev-dev wget gcc + +# solves warning: "jemalloc shared library could not be preloaded to speed up memory allocations" +RUN export DEBIAN_FRONTEND=noninteractive && \ +apt-get update && \ +apt-get install -y --no-install-recommends libjemalloc2 + +# install dumb-init as minimal init system +RUN export DEBIAN_FRONTEND=noninteractive && \ +apt-get update && \ +apt-get install -y dumb-init + +# generate locales for the standard en_US.UTF8 value we use for testing +RUN export DEBIAN_FRONTEND=noninteractive && \ +apt-get update && \ +apt-get install -y locales && \ +locale-gen en_US.UTF-8 + +# as we only need the requirements.txt file from the dtest repo, let's just get it from GitHub as a raw asset +# so we can avoid needing to clone the entire repo just to get this file +ADD https://raw.githubusercontent.com/apache/cassandra-dtest/master/requirements.txt /opt +RUN chmod 0644 /opt/requirements.txt + +# now setup python via virtualenv with all of the python dependencies we need according to requirements.txt +RUN pip3 install virtualenv +RUN pip3 install
[jira] [Updated] (CASSANDRA-15056) SimpleClient should pass connection properties as options
[ https://issues.apache.org/jira/browse/CASSANDRA-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15056: Severity: Low Complexity: Low Hanging Fruit Discovered By: User Report Bug Category: Parent values: Availability(12983) Reviewers: Sam Tunnicliffe Component/s: Test/unit Messaging/Client Status: Open (was: Triage) > SimpleClient should pass connection properties as options > - > > Key: CASSANDRA-15056 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15056 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client, Test/unit >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > As of today. SimpleClient does not pass CHECKSUM or COMPRESSION as options to > the server. In order for these parameters to take effect on the server side, > they should be passed as options. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15056) SimpleClient should pass connection properties as options
[ https://issues.apache.org/jira/browse/CASSANDRA-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15056: Status: Review In Progress (was: Patch Available) > SimpleClient should pass connection properties as options > - > > Key: CASSANDRA-15056 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15056 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client, Test/unit >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > As of today. SimpleClient does not pass CHECKSUM or COMPRESSION as options to > the server. In order for these parameters to take effect on the server side, > they should be passed as options. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15056) SimpleClient should pass connection properties as options
[ https://issues.apache.org/jira/browse/CASSANDRA-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15056: Test and Documentation Plan: No existing usages are affected by this. Unit and dtest runs to confirm no regressions from fix. Status: Patch Available (was: Open) > SimpleClient should pass connection properties as options > - > > Key: CASSANDRA-15056 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15056 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client, Test/unit >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > As of today. SimpleClient does not pass CHECKSUM or COMPRESSION as options to > the server. In order for these parameters to take effect on the server side, > they should be passed as options. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14607) Explore optimizations in AbstractBtreePartiton, java 11 variant
[ https://issues.apache.org/jira/browse/CASSANDRA-14607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-14607: Reviewers: Benedict > Explore optimizations in AbstractBtreePartiton, java 11 variant > --- > > Key: CASSANDRA-14607 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14607 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Jason Brown >Assignee: Blake Eggleston >Priority: Normal > Labels: Java11 > Fix For: 4.0 > > > In CASSANDRA-9608, we discussed some way to optimize the java 11 > implementation of {{AbstractBTreePartition}}. This ticket serves that > purpose, as well as a "note to selves" to ensure the java 11 version does not > have a performance regression. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14607) Explore optimizations in AbstractBtreePartiton, java 11 variant
[ https://issues.apache.org/jira/browse/CASSANDRA-14607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-14607: Test and Documentation Plan: CIrcleCI run looks good: https://circleci.com/workflow-run/2680b5b1-e7a3-4ef8-92ae-ba238ea6a1bc all j11 failures are also failing in trunk Status: Patch Available (was: Open) |[trunk|https://github.com/bdeggleston/cassandra/tree/14607-trunk-2]|[circle| https://circleci.com/workflow-run/2680b5b1-e7a3-4ef8-92ae-ba238ea6a1bc]| > Explore optimizations in AbstractBtreePartiton, java 11 variant > --- > > Key: CASSANDRA-14607 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14607 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Jason Brown >Assignee: Blake Eggleston >Priority: Normal > Labels: Java11 > Fix For: 4.0 > > > In CASSANDRA-9608, we discussed some way to optimize the java 11 > implementation of {{AbstractBTreePartition}}. This ticket serves that > purpose, as well as a "note to selves" to ensure the java 11 version does not > have a performance regression. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15058) Avoid double counting read latencies for digest queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15058: Status: Ready to Commit (was: Review In Progress) +1 > Avoid double counting read latencies for digest queries > --- > > Key: CASSANDRA-15058 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15058 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We are closing the {{UnfilteredPartitionIterator}} wrapped with > {{withMetricsRecording}} twice when we get digest requests - closing it calls > {{onClose}} and that makes the metrics update twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15058) Avoid double counting read latencies for digest queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15058: Reviewers: Sam Tunnicliffe Status: Review In Progress (was: Patch Available) > Avoid double counting read latencies for digest queries > --- > > Key: CASSANDRA-15058 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15058 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We are closing the {{UnfilteredPartitionIterator}} wrapped with > {{withMetricsRecording}} twice when we get digest requests - closing it calls > {{onClose}} and that makes the metrics update twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15019) Repaired data tracking isn't working for range queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15019: Status: Review In Progress (was: Patch Available) > Repaired data tracking isn't working for range queries > -- > > Key: CASSANDRA-15019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15019 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Coordination, Consistency/Repair, Test/dtest >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0 > > Attachments: RepairDigestTrackingTest.java > > > CASSANDRA-14145 introduced optional tracking of the repaired dataset used to > construct a read response. If enabled, each replica computes a digest for the > repaired portion of the data, which the coordinator compares in order to > detect divergence between replicas. This isn't working correctly for range > reads, as the ReadCommand instance that the DataResolver is intialized with > does not have the tracking flag set. This has been undetected up until now as > the dtest which should verify it also has a bug in that when the relevant > range query is issued the test expectations are being incorrectly set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15019) Repaired data tracking isn't working for range queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15019: Status: Ready to Commit (was: Review In Progress) > Repaired data tracking isn't working for range queries > -- > > Key: CASSANDRA-15019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15019 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Coordination, Consistency/Repair, Test/dtest >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0 > > Attachments: RepairDigestTrackingTest.java > > > CASSANDRA-14145 introduced optional tracking of the repaired dataset used to > construct a read response. If enabled, each replica computes a digest for the > repaired portion of the data, which the coordinator compares in order to > detect divergence between replicas. This isn't working correctly for range > reads, as the ReadCommand instance that the DataResolver is intialized with > does not have the tracking flag set. This has been undetected up until now as > the dtest which should verify it also has a bug in that when the relevant > range query is issued the test expectations are being incorrectly set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15019) Repaired data tracking isn't working for range queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15019: Since Version: 4.0 Status: Resolved (was: Ready to Commit) Thanks for the review and for adding the in-jvm dtest. I've included that in the final patch along with checking the correct config setting and committed to trunk in {{99ce007c5beb7988ce83fb1443a1e0ca259264cc}} Re: the copy methods & constructor. I agree that doing something more radical is probably justified now that transient replication has landed (it hadn't when the original patch was written). I believe though that the removal of the {{REPAIRED_DATA_TRACKING}} header is imminent via another in-flight patch so I've made a note to revisit this if/when that lands. > Repaired data tracking isn't working for range queries > -- > > Key: CASSANDRA-15019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15019 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Coordination, Consistency/Repair, Test/dtest >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0 > > Attachments: RepairDigestTrackingTest.java > > > CASSANDRA-14145 introduced optional tracking of the repaired dataset used to > construct a read response. If enabled, each replica computes a digest for the > repaired portion of the data, which the coordinator compares in order to > detect divergence between replicas. This isn't working correctly for range > reads, as the ReadCommand instance that the DataResolver is intialized with > does not have the tracking flag set. This has been undetected up until now as > the dtest which should verify it also has a bug in that when the relevant > range query is issued the test expectations are being incorrectly set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Correctly set repaired data tracking flag on range commands
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 99ce007 Correctly set repaired data tracking flag on range commands 99ce007 is described below commit 99ce007c5beb7988ce83fb1443a1e0ca259264cc Author: Sam Tunnicliffe AuthorDate: Tue Feb 12 15:21:25 2019 + Correctly set repaired data tracking flag on range commands Patch by Sam Tunnicliffe; reviewed by Jordan West for CASSANDRA-15019 --- CHANGES.txt| 1 + .../org/apache/cassandra/service/StorageProxy.java | 16 ++- .../distributed/test/RepairDigestTrackingTest.java | 152 + 3 files changed, 162 insertions(+), 7 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 2254452..b511c18 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0 + * Set repaired data tracking flag on range reads if enabled (CASSANDRA-15019) * Calculate pending ranges for BOOTSTRAP_REPLACE correctly (CASSANDRA-14802) * Make TableCQLHelper reuse the single quote pattern (CASSANDRA-15033) * Add Zstd compressor (CASSANDRA-14482) diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java b/src/java/org/apache/cassandra/service/StorageProxy.java index 439d8cf..ce78674 100644 --- a/src/java/org/apache/cassandra/service/StorageProxy.java +++ b/src/java/org/apache/cassandra/service/StorageProxy.java @@ -2047,6 +2047,15 @@ public class StorageProxy implements StorageProxyMBean private SingleRangeResponse query(ReplicaPlan.ForRangeRead replicaPlan, boolean isFirst) { PartitionRangeReadCommand rangeCommand = command.forSubRange(replicaPlan.range(), isFirst); +// If enabled, request repaired data tracking info from full replicas but +// only if there are multiple full replicas to compare results from +if (DatabaseDescriptor.getRepairedDataTrackingForRangeReadsEnabled() +&& replicaPlan.contacts().filter(Replica::isFull).size() > 1) +{ +command.trackRepairedStatus(); +rangeCommand.trackRepairedStatus(); +} + ReplicaPlan.SharedForRangeRead sharedReplicaPlan = ReplicaPlan.shared(replicaPlan); ReadRepair readRepair = ReadRepair.create(command, sharedReplicaPlan, queryStartNanoTime); @@ -2055,13 +2064,6 @@ public class StorageProxy implements StorageProxyMBean ReadCallback handler = new ReadCallback<>(resolver, rangeCommand, sharedReplicaPlan, queryStartNanoTime); -// If enabled, request repaired data tracking info from full replicas but -// only if there are multiple full replicas to compare results from -if (DatabaseDescriptor.getRepairedDataTrackingForPartitionReadsEnabled() -&& replicaPlan.contacts().filter(Replica::isFull).size() > 1) -{ -command.trackRepairedStatus(); -} if (replicaPlan.contacts().size() == 1 && replicaPlan.contacts().get(0).isSelf()) { diff --git a/test/distributed/org/apache/cassandra/distributed/test/RepairDigestTrackingTest.java b/test/distributed/org/apache/cassandra/distributed/test/RepairDigestTrackingTest.java new file mode 100644 index 000..a987ea3 --- /dev/null +++ b/test/distributed/org/apache/cassandra/distributed/test/RepairDigestTrackingTest.java @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.distributed.test; + +import java.io.IOException; +import java.io.Serializable; +import java.util.EnumSet; +import java.util.Map; +import java.util.Set; + +import org.junit.Assert; +import org.junit.Test; + +import org.apache.cassandra.db.ConsistencyLevel; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.distributed.Cluster; +import org.apache.cassandra.io.sstable.Descriptor; +import org.apache.cassandra.io.sstable.format.SSTableReader; +import
[jira] [Updated] (CASSANDRA-14802) calculatePendingRanges assigns more pending ranges than necessary
[ https://issues.apache.org/jira/browse/CASSANDRA-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14802: Fix Version/s: (was: 4.x) 4.0 Since Version: 2.1.0 Status: Resolved (was: Ready to Commit) Thanks, committed to trunk in {{4f53bc87261b470adf292fdf37ed4e81bb6f8704}} > calculatePendingRanges assigns more pending ranges than necessary > -- > > Key: CASSANDRA-14802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14802 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination, Legacy/Distributed Metadata >Reporter: Benedict >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0 > > > This might be a good thing, but should probably be configurable, and made > consistent. Presently, in a number of circumstances where there are multiple > range movements, {{calculatePendingRanges}} will assign a pending range to a > node that will not ultimately own it. If done consistently, this might make > range movements resilient to node failures / aborted range movements, since > all nodes will be receiving all ranges they might own under any incomplete > range ownership movements. But done inconsistently it seems only to reduce > availability in the cluster, by potentially increasing the number of pending > nodes unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14802) calculatePendingRanges assigns more pending ranges than necessary
[ https://issues.apache.org/jira/browse/CASSANDRA-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14802: Status: Ready to Commit (was: Review In Progress) > calculatePendingRanges assigns more pending ranges than necessary > -- > > Key: CASSANDRA-14802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14802 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination, Legacy/Distributed Metadata >Reporter: Benedict >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.x > > > This might be a good thing, but should probably be configurable, and made > consistent. Presently, in a number of circumstances where there are multiple > range movements, {{calculatePendingRanges}} will assign a pending range to a > node that will not ultimately own it. If done consistently, this might make > range movements resilient to node failures / aborted range movements, since > all nodes will be receiving all ranges they might own under any incomplete > range ownership movements. But done inconsistently it seems only to reduce > availability in the cluster, by potentially increasing the number of pending > nodes unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14802) calculatePendingRanges assigns more pending ranges than necessary
[ https://issues.apache.org/jira/browse/CASSANDRA-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14802: Test and Documentation Plan: Unit test included Status: Patch Available (was: In Progress) > calculatePendingRanges assigns more pending ranges than necessary > -- > > Key: CASSANDRA-14802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14802 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination, Legacy/Distributed Metadata >Reporter: Benedict >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.x > > > This might be a good thing, but should probably be configurable, and made > consistent. Presently, in a number of circumstances where there are multiple > range movements, {{calculatePendingRanges}} will assign a pending range to a > node that will not ultimately own it. If done consistently, this might make > range movements resilient to node failures / aborted range movements, since > all nodes will be receiving all ranges they might own under any incomplete > range ownership movements. But done inconsistently it seems only to reduce > availability in the cluster, by potentially increasing the number of pending > nodes unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14802) calculatePendingRanges assigns more pending ranges than necessary
[ https://issues.apache.org/jira/browse/CASSANDRA-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14802: Status: Review In Progress (was: Patch Available) > calculatePendingRanges assigns more pending ranges than necessary > -- > > Key: CASSANDRA-14802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14802 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination, Legacy/Distributed Metadata >Reporter: Benedict >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.x > > > This might be a good thing, but should probably be configurable, and made > consistent. Presently, in a number of circumstances where there are multiple > range movements, {{calculatePendingRanges}} will assign a pending range to a > node that will not ultimately own it. If done consistently, this might make > range movements resilient to node failures / aborted range movements, since > all nodes will be receiving all ranges they might own under any incomplete > range ownership movements. But done inconsistently it seems only to reduce > availability in the cluster, by potentially increasing the number of pending > nodes unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14802) calculatePendingRanges assigns more pending ranges than necessary
[ https://issues.apache.org/jira/browse/CASSANDRA-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe reassigned CASSANDRA-14802: --- Assignee: Sam Tunnicliffe > calculatePendingRanges assigns more pending ranges than necessary > -- > > Key: CASSANDRA-14802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14802 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination, Legacy/Distributed Metadata >Reporter: Benedict >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.x > > > This might be a good thing, but should probably be configurable, and made > consistent. Presently, in a number of circumstances where there are multiple > range movements, {{calculatePendingRanges}} will assign a pending range to a > node that will not ultimately own it. If done consistently, this might make > range movements resilient to node failures / aborted range movements, since > all nodes will be receiving all ranges they might own under any incomplete > range ownership movements. But done inconsistently it seems only to reduce > availability in the cluster, by potentially increasing the number of pending > nodes unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14802) calculatePendingRanges assigns more pending ranges than necessary
[ https://issues.apache.org/jira/browse/CASSANDRA-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14802: Status: In Progress (was: Patch Available) > calculatePendingRanges assigns more pending ranges than necessary > -- > > Key: CASSANDRA-14802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14802 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination, Legacy/Distributed Metadata >Reporter: Benedict >Priority: Normal > Fix For: 4.x > > > This might be a good thing, but should probably be configurable, and made > consistent. Presently, in a number of circumstances where there are multiple > range movements, {{calculatePendingRanges}} will assign a pending range to a > node that will not ultimately own it. If done consistently, this might make > range movements resilient to node failures / aborted range movements, since > all nodes will be receiving all ranges they might own under any incomplete > range ownership movements. But done inconsistently it seems only to reduce > availability in the cluster, by potentially increasing the number of pending > nodes unnecessarily. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Bootstrap replace produce correct pending range calculations
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 4f53bc8 Bootstrap replace produce correct pending range calculations 4f53bc8 is described below commit 4f53bc87261b470adf292fdf37ed4e81bb6f8704 Author: Sam Tunnicliffe AuthorDate: Wed Feb 27 12:33:32 2019 + Bootstrap replace produce correct pending range calculations Patch by Sam Tunnicliffe; reviewed by Benedict Elliott Smith for CASSANDRA-14802 --- CHANGES.txt| 1 + .../apache/cassandra/locator/TokenMetadata.java| 16 +- .../cassandra/locator/PendingRangesTest.java | 260 + 3 files changed, 271 insertions(+), 6 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 7e1b5eb..2254452 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0 + * Calculate pending ranges for BOOTSTRAP_REPLACE correctly (CASSANDRA-14802) * Make TableCQLHelper reuse the single quote pattern (CASSANDRA-15033) * Add Zstd compressor (CASSANDRA-14482) * Fix IR prepare anti-compaction race (CASSANDRA-15027) diff --git a/src/java/org/apache/cassandra/locator/TokenMetadata.java b/src/java/org/apache/cassandra/locator/TokenMetadata.java index cb189c8..c16538b 100644 --- a/src/java/org/apache/cassandra/locator/TokenMetadata.java +++ b/src/java/org/apache/cassandra/locator/TokenMetadata.java @@ -906,19 +906,23 @@ public class TokenMetadata // At this stage newPendingRanges has been updated according to leave operations. We can // now continue the calculation by checking bootstrapping nodes. -// For each of the bootstrapping nodes, simply add and remove them one by one to -// allLeftMetadata and check in between what their ranges would be. +// For each of the bootstrapping nodes, simply add to the allLeftMetadata and check what their +// ranges would be. We actually need to clone allLeftMetadata each time as resetting its state +// after getting the new pending ranges is not as simple as just removing the bootstrapping +// endpoint. If the bootstrapping endpoint constitutes a replacement, removing it after checking +// the newly pending ranges means there are now fewer endpoints that there were originally and +// causes its next neighbour to take over its primary range which affects the next RF endpoints +// in the ring. Multimap bootstrapAddresses = bootstrapTokens.inverse(); for (InetAddressAndPort endpoint : bootstrapAddresses.keySet()) { Collection tokens = bootstrapAddresses.get(endpoint); - -allLeftMetadata.updateNormalTokens(tokens, endpoint); -for (Replica replica : strategy.getAddressReplicas(allLeftMetadata, endpoint)) +TokenMetadata cloned = allLeftMetadata.cloneOnlyTokenMap(); +cloned.updateNormalTokens(tokens, endpoint); +for (Replica replica : strategy.getAddressReplicas(cloned, endpoint)) { newPendingRanges.addPendingRange(replica.range(), replica); } -allLeftMetadata.removeEndpoint(endpoint); } // At this stage newPendingRanges has been updated according to leaving and bootstrapping nodes. diff --git a/test/unit/org/apache/cassandra/locator/PendingRangesTest.java b/test/unit/org/apache/cassandra/locator/PendingRangesTest.java new file mode 100644 index 000..48bf546 --- /dev/null +++ b/test/unit/org/apache/cassandra/locator/PendingRangesTest.java @@ -0,0 +1,260 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.locator; + +import java.net.UnknownHostException; +import java.util.Collections; + +import com.google.common.collect.*; +import org.junit.BeforeClass; +import org.junit.Test; + + +import org.apache.cassandra.config.DatabaseDescriptor; +import org.apache.cassandra.dht.Murmur3Partitioner; +import org.apache.cassandra.dht.Token; + +import static org.junit.Assert.assertEquals; +import static
[jira] [Commented] (CASSANDRA-15045) Fix index summary redistribution compaction cancellation issues
[ https://issues.apache.org/jira/browse/CASSANDRA-15045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797021#comment-16797021 ] Marcus Eriksson commented on CASSANDRA-15045: - Pushed a new commit to the trunk branch to rename the variable Also pushed the {{closeAll}} change to all branches and made sure we don't error-log {{CompactionInterruptedException}} bq. As usual, I want to double check its ok to change the semantics in each of the versions the change is proposed in. Are their any cases where it should be an option whether or not global compactions are cancelled? My initial inclination is no but wanted to bring it up explicitly regardless. It has been possible to cancel index summary redistributions before as well - but only explicitly (like {{nodetool stop INDEX_SUMMARY}}) but not automatically when starting anticompaction or truncating for example, so I think it should be safe. My biggest worry was that since explicit stops like above are much more rare than the automatic ones we would find some other issue (like transaction not getting closed properly) > Fix index summary redistribution compaction cancellation issues > --- > > Key: CASSANDRA-15045 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15045 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x > > > We can't cancel ongoing index summary redistributions currently due to > {{CompactionInfo}} returning null for {{getTableMetadata/getCFMetaData}} > [here|https://github.com/apache/cassandra/blob/67d613204fa4fb9584f11ec9886a0e7a0d622e92/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1814] > for index summary redistributions > CASSANDRA-14935 also introduced a bug where we track the wrong sstables for > index summary redistributions -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15058) Avoid double counting read latencies for digest queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796953#comment-16796953 ] Marcus Eriksson edited comment on CASSANDRA-15058 at 3/20/19 9:34 AM: -- attaching patch to not close the iterator after digesting it; [15058-3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...krummas:marcuse/15058-3.0] [15058-3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...krummas:marcuse/15058-3.11] [15058-trunk|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/15058-trunk] https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.0 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.11 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-trunk was (Author: krummas): attaching patch to not close the iterator after digesting it; [15058-3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...krummas:marcuse/15058-3.0] [15058-3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...krummas:marcuse/15058-3.11] [15058-trunk|https://github.com/apache/cassandra/compare/cassandra-trunk...krummas:marcuse/15058-trunk] https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.0 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.11 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-trunk > Avoid double counting read latencies for digest queries > --- > > Key: CASSANDRA-15058 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15058 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We are closing the {{UnfilteredPartitionIterator}} wrapped with > {{withMetricsRecording}} twice when we get digest requests - closing it calls > {{onClose}} and that makes the metrics update twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15058) Avoid double counting read latencies for digest queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796953#comment-16796953 ] Marcus Eriksson edited comment on CASSANDRA-15058 at 3/20/19 9:33 AM: -- attaching patch to not close the iterator after digesting it; [15058-3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...krummas:marcuse/15058-3.0] [15058-3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...krummas:marcuse/15058-3.11] [15058-trunk|https://github.com/apache/cassandra/compare/cassandra-trunk...krummas:marcuse/15058-trunk] https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.0 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.11 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-trunk was (Author: krummas): attaching patch to not close the iterator after digesting it; https://github.com/krummas/cassandra/commits/marcuse/15058-3.0 https://github.com/krummas/cassandra/commits/marcuse/15058-3.11 https://github.com/krummas/cassandra/commits/marcuse/15058-trunk https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.0 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.11 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-trunk > Avoid double counting read latencies for digest queries > --- > > Key: CASSANDRA-15058 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15058 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We are closing the {{UnfilteredPartitionIterator}} wrapped with > {{withMetricsRecording}} twice when we get digest requests - closing it calls > {{onClose}} and that makes the metrics update twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15058) Avoid double counting read latencies for digest queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15058: Test and Documentation Plan: A distributed in-jvm test was added to make sure we don't overcount Status: Patch Available (was: Open) attaching patch to not close the iterator after digesting it; https://github.com/krummas/cassandra/commits/marcuse/15058-3.0 https://github.com/krummas/cassandra/commits/marcuse/15058-3.11 https://github.com/krummas/cassandra/commits/marcuse/15058-trunk https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.0 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-3.11 https://circleci.com/gh/krummas/workflows/cassandra/tree/marcuse%2F15058-trunk > Avoid double counting read latencies for digest queries > --- > > Key: CASSANDRA-15058 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15058 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We are closing the {{UnfilteredPartitionIterator}} wrapped with > {{withMetricsRecording}} twice when we get digest requests - closing it calls > {{onClose}} and that makes the metrics update twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15058) Avoid double counting read latencies for digest queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15058: Severity: Normal Complexity: Low Hanging Fruit Discovered By: Workload Replay Bug Category: Parent values: Correctness(12982)Level 1 values: Semantic Failure(12988) Status: Open (was: Triage) > Avoid double counting read latencies for digest queries > --- > > Key: CASSANDRA-15058 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15058 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We are closing the {{UnfilteredPartitionIterator}} wrapped with > {{withMetricsRecording}} twice when we get digest requests - closing it calls > {{onClose}} and that makes the metrics update twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org