[jira] [Commented] (CASSANDRA-14747) Evaluate 200 node, compression=none, encryption=none, coalescing=off
[ https://issues.apache.org/jira/browse/CASSANDRA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615321#comment-16615321 ] Jeff Jirsa commented on CASSANDRA-14747: Thanks so much for running this! > Evaluate 200 node, compression=none, encryption=none, coalescing=off > - > > Key: CASSANDRA-14747 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14747 > Project: Cassandra > Issue Type: Sub-task >Reporter: Joseph Lynch >Assignee: Joseph Lynch >Priority: Major > Attachments: 3.0.17-QPS.png, 4.0.1-QPS.png, > 4.0_errors_showing_heap_pressure.txt, > 4.0_heap_histogram_showing_many_MessageOuts.txt, > useast1c-i-0e1ddfe8b2f769060-mutation-flame.svg > > > Tracks evaluating a 200 node cluster with all internode settings off (no > compression, no encryption, no coalescing). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14747) Evaluate 200 node, compression=none, encryption=none, coalescing=off
[ https://issues.apache.org/jira/browse/CASSANDRA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] C. Scott Andreas reassigned CASSANDRA-14747: Assignee: Joseph Lynch (was: joseph) > Evaluate 200 node, compression=none, encryption=none, coalescing=off > - > > Key: CASSANDRA-14747 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14747 > Project: Cassandra > Issue Type: Sub-task >Reporter: Joseph Lynch >Assignee: Joseph Lynch >Priority: Major > Attachments: 3.0.17-QPS.png, 4.0.1-QPS.png, > 4.0_errors_showing_heap_pressure.txt, > 4.0_heap_histogram_showing_many_MessageOuts.txt, > useast1c-i-0e1ddfe8b2f769060-mutation-flame.svg > > > Tracks evaluating a 200 node cluster with all internode settings off (no > compression, no encryption, no coalescing). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14747) Evaluate 200 node, compression=none, encryption=none, coalescing=off
[ https://issues.apache.org/jira/browse/CASSANDRA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] C. Scott Andreas reassigned CASSANDRA-14747: Assignee: joseph > Evaluate 200 node, compression=none, encryption=none, coalescing=off > - > > Key: CASSANDRA-14747 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14747 > Project: Cassandra > Issue Type: Sub-task >Reporter: Joseph Lynch >Assignee: joseph >Priority: Major > Attachments: 3.0.17-QPS.png, 4.0.1-QPS.png, > 4.0_errors_showing_heap_pressure.txt, > 4.0_heap_histogram_showing_many_MessageOuts.txt, > useast1c-i-0e1ddfe8b2f769060-mutation-flame.svg > > > Tracks evaluating a 200 node cluster with all internode settings off (no > compression, no encryption, no coalescing). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14685) Incremental repair 4.0 : SSTables remain locked forever if the coordinator dies during streaming
[
https://issues.apache.org/jira/browse/CASSANDRA-14685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615143#comment-16615143
]
Blake Eggleston commented on CASSANDRA-14685:
-
[~adejanovski] I took another look at the steps the reproduce, and this is
almost definitely working as expected... you wouldn't know it from the docs and
error messages though. I opened CASSANDRA-14753 to improve docs and error
messages related to this.
> Incremental repair 4.0 : SSTables remain locked forever if the coordinator
> dies during streaming
> -
>
> Key: CASSANDRA-14685
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14685
> Project: Cassandra
> Issue Type: Bug
> Components: Repair
>Reporter: Alexander Dejanovski
>Assignee: Jason Brown
>Priority: Critical
>
> The changes in CASSANDRA-9143 modified the way incremental repair performs by
> applying the following sequence of events :
> * Anticompaction is executed on all replicas for all SSTables overlapping
> the repaired ranges
> * Anticompacted SSTables are then marked as "Pending repair" and cannot be
> compacted anymore, nor part of another repair session
> * Merkle trees are generated and compared
> * Streaming takes place if needed
> * Anticompaction is committed and "pending repair" table are marked as
> repaired if it succeeded, or they are released if the repair session failed.
> If the repair coordinator dies during the streaming phase, *the SSTables on
> the replicas will remain in "pending repair" state and will never be eligible
> for repair or compaction*, even after all the nodes in the cluster are
> restarted.
> Steps to reproduce (I've used Jason's 13938 branch that fixes streaming
> errors) :
> {noformat}
> ccm create inc-repair-issue -v github:jasobrown/13938 -n 3
> # Allow jmx access and remove all rpc_ settings in yaml
> for f in ~/.ccm/inc-repair-issue/node*/conf/cassandra-env.sh;
> do
> sed -i'' -e
> 's/com.sun.management.jmxremote.authenticate=true/com.sun.management.jmxremote.authenticate=false/g'
> $f
> done
> for f in ~/.ccm/inc-repair-issue/node*/conf/cassandra.yaml;
> do
> grep -v "rpc_" $f > ${f}.tmp
> cat ${f}.tmp > $f
> done
> ccm start
> {noformat}
> I used [tlp-stress|https://github.com/thelastpickle/tlp-stress] to generate a
> few 10s of MBs of data (killed it after some time). Obviously
> cassandra-stress works as well :
> {noformat}
> bin/tlp-stress run BasicTimeSeries -i 1M -p 1M -t 2 --rate 5000
> --replication "{'class':'SimpleStrategy', 'replication_factor':2}"
> --compaction "{'class': 'SizeTieredCompactionStrategy'}" --host
> 127.0.0.1
> {noformat}
> Flush and delete all SSTables in node1 :
> {noformat}
> ccm node1 nodetool flush
> ccm node1 stop
> rm -f ~/.ccm/inc-repair-issue/node1/data0/tlp_stress/sensor*/*.*
> ccm node1 start{noformat}
> Then throttle streaming throughput to 1MB/s so we have time to take node1
> down during the streaming phase and run repair:
> {noformat}
> ccm node1 nodetool setstreamthroughput 1
> ccm node2 nodetool setstreamthroughput 1
> ccm node3 nodetool setstreamthroughput 1
> ccm node1 nodetool repair tlp_stress
> {noformat}
> Once streaming starts, shut down node1 and start it again :
> {noformat}
> ccm node1 stop
> ccm node1 start
> {noformat}
> Run repair again :
> {noformat}
> ccm node1 nodetool repair tlp_stress
> {noformat}
> The command will return very quickly, showing that it skipped all sstables :
> {noformat}
> [2018-08-31 19:05:16,292] Repair completed successfully
> [2018-08-31 19:05:16,292] Repair command #1 finished in 2 seconds
> $ ccm node1 nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
>Rack
> UN 127.0.0.1 228,64 KiB 256 ?
> 437dc9cd-b1a1-41a5-961e-cfc99763e29f rack1
> UN 127.0.0.2 60,09 MiB 256 ?
> fbcbbdbb-e32a-4716-8230-8ca59aa93e62 rack1
> UN 127.0.0.3 57,59 MiB 256 ?
> a0b1bcc6-0fad-405a-b0bf-180a0ca31dd0 rack1
> {noformat}
> sstablemetadata will then show that nodes 2 and 3 have SSTables still in
> "pending repair" state :
> {noformat}
> ~/.ccm/repository/gitCOLONtrunk/tools/bin/sstablemetadata na-4-big-Data.db |
> grep repair
> SSTable:
> /Users/adejanovski/.ccm/inc-repair-4.0/node2/data0/tlp_stress/sensor_data-b7375660ad3111e8a0e59357ff9c9bda/na-4-big
> Pending repair: 3844a400-ad33-11e8-b5a7-6b8dd8f31b62
> {noformat}
> Restarting these nodes wouldn't help either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Created] (CASSANDRA-14753) Document incremental repair session timeouts and repair_admin usage
Blake Eggleston created CASSANDRA-14753: --- Summary: Document incremental repair session timeouts and repair_admin usage Key: CASSANDRA-14753 URL: https://issues.apache.org/jira/browse/CASSANDRA-14753 Project: Cassandra Issue Type: Task Reporter: Blake Eggleston Assignee: Blake Eggleston Fix For: 4.0 As seen in CASSANDRA-14685, the behavior of incremental repair sessions with failed streams is not obvious and appears to be a bug (although it's working as expected). The incremental repair documentation should be updated to describe what happens if an incremental repair session fails mid-stream, the session timeouts, and how and when to use nodetool repair_admin. The sstable acquisition error should also be updated to mention this as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14145) Detecting data resurrection during read
[ https://issues.apache.org/jira/browse/CASSANDRA-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614897#comment-16614897 ] ASF GitHub Bot commented on CASSANDRA-14145: GitHub user beobal opened a pull request: https://github.com/apache/cassandra-dtest/pull/37 Add tests for CASSANDRA-14145 * Enable read time repaired data tracking for all tests by default * Add some specific tests for mismatching/matching repaired digests You can merge this pull request into a Git repository by running: $ git pull https://github.com/beobal/cassandra-dtest 14145 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra-dtest/pull/37.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #37 commit a6d673df641800e708f8ad593644e63fb175fb39 Author: Marcus Eriksson Date: 2018-08-23T18:40:43Z Add intial tests for CASSANDRA-14145 Enable read time repaired data tracking for all tests by default > Detecting data resurrection during read > > > Key: CASSANDRA-14145 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14145 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Sam Tunnicliffe >Priority: Minor > Labels: pull-request-available > Fix For: 4.0 > > > We have seen several bugs in which deleted data gets resurrected. We should > try to see if we can detect this on the read path and possibly fix it. Here > are a few examples which brought back data > A replica lost an sstable on startup which caused one replica to lose the > tombstone and not the data. This tombstone was past gc grace which means this > could resurrect data. We can detect such invalid states by looking at other > replicas. > If we are running incremental repair, Cassandra will keep repaired and > non-repaired data separate. Every-time incremental repair will run, it will > move the data from non-repaired to repaired. Repaired data across all > replicas should be 100% consistent. > Here is an example of how we can detect and mitigate the issue in most cases. > Say we have 3 machines, A,B and C. All these machines will have data split > b/w repaired and non-repaired. > 1. Machine A due to some bug bring backs data D. This data D is in repaired > dataset. All other replicas will have data D and tombstone T > 2. Read for data D comes from application which involve replicas A and B. The > data being read involves data which is in repaired state. A will respond > back to co-ordinator with data D and B will send nothing as tombstone is past > gc grace. This will cause digest mismatch. > 3. This patch will only kick in when there is a digest mismatch. Co-ordinator > will ask both replicas to send back all data like we do today but with this > patch, replicas will respond back what data it is returning is coming from > repaired vs non-repaired. If data coming from repaired does not match, we > know there is a something wrong!! At this time, co-ordinator cannot determine > if replica A has resurrected some data or replica B has lost some data. We > can still log error in the logs saying we hit an invalid state. > 4. Besides the log, we can take this further and even correct the response to > the query. After logging an invalid state, we can ask replica A and B (and > also C if alive) to send back all data for this including gcable tombstones. > If any machine returns a tombstone which is after this data, we know we > cannot return this data. This way we can avoid returning data which has been > deleted. > Some Challenges with this > 1. When data will be moved from non-repaired to repaired, there could be a > race here. We can look at which incremental repairs have promoted things on > which replica to avoid false positives. > 2. If the third replica is down and live replica does not have any tombstone, > we wont be able to break the tie in deciding whether data was actually > deleted or resurrected. > 3. If the read is for latest data only, we wont be able to detect it as the > read will be served from non-repaired data. > 4. If the replica where we lose a tombstone is the last replica to compact > the tombstone, we wont be able to decide if data is coming back or rest of > the replicas has lost that data. But we will still detect something is wrong. > 5. We wont affect 99.9% of the read queries as we only do extra work during > digest mismatch. > 6. CL.ONE reads will not be able to detect this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail:
[jira] [Updated] (CASSANDRA-14145) Detecting data resurrection during read
[ https://issues.apache.org/jira/browse/CASSANDRA-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRA-14145: --- Labels: pull-request-available (was: ) > Detecting data resurrection during read > > > Key: CASSANDRA-14145 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14145 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Sam Tunnicliffe >Priority: Minor > Labels: pull-request-available > Fix For: 4.0 > > > We have seen several bugs in which deleted data gets resurrected. We should > try to see if we can detect this on the read path and possibly fix it. Here > are a few examples which brought back data > A replica lost an sstable on startup which caused one replica to lose the > tombstone and not the data. This tombstone was past gc grace which means this > could resurrect data. We can detect such invalid states by looking at other > replicas. > If we are running incremental repair, Cassandra will keep repaired and > non-repaired data separate. Every-time incremental repair will run, it will > move the data from non-repaired to repaired. Repaired data across all > replicas should be 100% consistent. > Here is an example of how we can detect and mitigate the issue in most cases. > Say we have 3 machines, A,B and C. All these machines will have data split > b/w repaired and non-repaired. > 1. Machine A due to some bug bring backs data D. This data D is in repaired > dataset. All other replicas will have data D and tombstone T > 2. Read for data D comes from application which involve replicas A and B. The > data being read involves data which is in repaired state. A will respond > back to co-ordinator with data D and B will send nothing as tombstone is past > gc grace. This will cause digest mismatch. > 3. This patch will only kick in when there is a digest mismatch. Co-ordinator > will ask both replicas to send back all data like we do today but with this > patch, replicas will respond back what data it is returning is coming from > repaired vs non-repaired. If data coming from repaired does not match, we > know there is a something wrong!! At this time, co-ordinator cannot determine > if replica A has resurrected some data or replica B has lost some data. We > can still log error in the logs saying we hit an invalid state. > 4. Besides the log, we can take this further and even correct the response to > the query. After logging an invalid state, we can ask replica A and B (and > also C if alive) to send back all data for this including gcable tombstones. > If any machine returns a tombstone which is after this data, we know we > cannot return this data. This way we can avoid returning data which has been > deleted. > Some Challenges with this > 1. When data will be moved from non-repaired to repaired, there could be a > race here. We can look at which incremental repairs have promoted things on > which replica to avoid false positives. > 2. If the third replica is down and live replica does not have any tombstone, > we wont be able to break the tie in deciding whether data was actually > deleted or resurrected. > 3. If the read is for latest data only, we wont be able to detect it as the > read will be served from non-repaired data. > 4. If the replica where we lose a tombstone is the last replica to compact > the tombstone, we wont be able to decide if data is coming back or rest of > the replicas has lost that data. But we will still detect something is wrong. > 5. We wont affect 99.9% of the read queries as we only do extra work during > digest mismatch. > 6. CL.ONE reads will not be able to detect this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-14373:
Reviewers: Marcus Eriksson
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14749) Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
[
https://issues.apache.org/jira/browse/CASSANDRA-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613870#comment-16613870
]
Benedict edited comment on CASSANDRA-14749 at 9/14/18 12:34 PM:
-FWIW, I've been planning to file a ticket to audit the non-use of
{{getDroppedColumn}} also, off the back of this, and perhaps we should combine
the two efforts. I'm sure most uses of {{getColumn}} should have a
corresponding use of {{getDroppedColumn}} and I'm sure there have been other
oversights than this. We could keep two separate endeavours though.-
At least in trunk, it seems dropped columns have been considered in all places
that we invoke {{getColumn}}, besides in test cases
was (Author: benedict):
FWIW, I've been planning to file a ticket to audit the non-use of
{{getDroppedColumn}} also, off the back of this, and perhaps we should combine
the two efforts. I'm sure most uses of {{getColumn}} should have a
corresponding use of {{getDroppedColumn}} and I'm sure there have been other
oversights than this. We could keep two separate endeavours though.
> Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
> --
>
> Key: CASSANDRA-14749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14749
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Major
> Fix For: 3.0.18
>
>
> Similar to CASSANDRA-14568, if a 2.1 node sends a response to a 3.0 node
> containing a deletion for a dropped collection column, instead of deleting
> the collection, we will delete the row containing the collection.
>
> This is an admittedly unlikely cluster state but, during such a state, a
> great deal of data loss could happen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: ninja fix bad merges for unit test
Repository: cassandra
Updated Branches:
refs/heads/trunk d44059bfc -> 261e75f19
ninja fix bad merges for unit test
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/261e75f1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/261e75f1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/261e75f1
Branch: refs/heads/trunk
Commit: 261e75f19f5348ea2b4e8f23fda00be3bf7c0436
Parents: d44059b
Author: Benedict Elliott Smith
Authored: Fri Sep 14 11:45:22 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:48:20 2018 +0100
--
.../service/WriteResponseHandlerTest.java | 1 +
.../WriteResponseHandlerTransientTest.java | 10 +-
.../reads/DataResolverTransientTest.java| 227 ---
3 files changed, 7 insertions(+), 231 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/261e75f1/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
--
diff --git
a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
index a3f13c2..2c186ba 100644
--- a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
+++ b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
@@ -214,6 +214,7 @@ public class WriteResponseHandlerTest
@Test
public void failedIdealCLIncrementsStat() throws Throwable
{
+
ks.metric.idealCLWriteLatency.totalLatency.dec(ks.metric.idealCLWriteLatency.totalLatency.getCount());
AbstractWriteResponseHandler awr =
createWriteResponseHandler(ConsistencyLevel.LOCAL_QUORUM,
ConsistencyLevel.EACH_QUORUM);
//Succeed in local DC
http://git-wip-us.apache.org/repos/asf/cassandra/blob/261e75f1/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
--
diff --git
a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
index b6c95dd..d31d3f1 100644
---
a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
+++
b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
@@ -24,6 +24,7 @@ import java.util.Set;
import java.util.UUID;
import java.util.function.Predicate;
+import com.google.common.collect.Iterables;
import com.google.common.collect.Sets;
import org.apache.cassandra.dht.Murmur3Partitioner;
@@ -153,7 +154,8 @@ public class WriteResponseHandlerTransientTest
ReplicaLayout.ForTokenWrite layout = new
ReplicaLayout.ForTokenWrite(natural, pending);
ReplicaPlan.ForTokenWrite replicaPlan = ReplicaPlans.forWrite(ks,
ConsistencyLevel.QUORUM, layout, layout, ReplicaPlans.writeAll);
-Assert.assertEquals(EndpointsForRange.of(full(EP4), trans(EP6)),
replicaPlan.pending());
+
Assert.assertTrue(Iterables.elementsEqual(EndpointsForRange.of(full(EP4),
trans(EP6)),
+ replicaPlan.pending()));
}
private static ReplicaPlan.ForTokenWrite expected(EndpointsForToken
natural, EndpointsForToken selected)
@@ -171,9 +173,9 @@ public class WriteResponseHandlerTransientTest
private static void assertSpeculationReplicas(ReplicaPlan.ForTokenWrite
expected, EndpointsForToken replicas, Predicate
livePredicate)
{
ReplicaPlan.ForTokenWrite actual = getSpeculationContext(replicas,
livePredicate);
-Assert.assertEquals(expected.pending(), actual.pending());
-Assert.assertEquals(expected.live(), actual.live());
-Assert.assertEquals(expected.contacts(), actual.contacts());
+Assert.assertTrue(Iterables.elementsEqual(expected.pending(),
actual.pending()));
+Assert.assertTrue(Iterables.elementsEqual(expected.live(),
actual.live()));
+Assert.assertTrue(Iterables.elementsEqual(expected.contacts(),
actual.contacts()));
}
private static Predicate dead(InetAddressAndPort...
endpoints)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/261e75f1/test/unit/org/apache/cassandra/service/reads/DataResolverTransientTest.java
--
diff --git
a/test/unit/org/apache/cassandra/service/reads/DataResolverTransientTest.java
b/test/unit/org/apache/cassandra/service/reads/DataResolverTransientTest.java
deleted file mode 100644
index 456cec4..000
---
a/test/unit/org/apache/cassandra/service/reads/DataResolverTransientTest.java
+++ /dev/null
@@ -1,227 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more
[jira] [Updated] (CASSANDRA-14749) Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
[ https://issues.apache.org/jira/browse/CASSANDRA-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14749: - Resolution: Fixed Status: Resolved (was: Ready to Commit) Thanks. Committed as [06c55f779ae68de98cce531e0b78be5716849003|https://github.com/apache/cassandra/commit/06c55f779ae68de98cce531e0b78be5716849003]. > Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows > -- > > Key: CASSANDRA-14749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14749 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict >Priority: Major > Fix For: 3.0.18 > > > Similar to CASSANDRA-14568, if a 2.1 node sends a response to a 3.0 node > containing a deletion for a dropped collection column, instead of deleting > the collection, we will delete the row containing the collection. > > This is an admittedly unlikely cluster state but, during such a state, a > great deal of data loss could happen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict updated CASSANDRA-14568:
-
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thanks. Committed as
[68dbeb34c9404ee3cd7db00cc112e27c9a4b1f6f|https://github.com/apache/cassandra/commit/68dbeb34c9404ee3cd7db00cc112e27c9a4b1f6f].
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.11.3, 3.0.17
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[1/9] cassandra git commit: Static deletions are corrupted in 3.0 -> 2.{1, 2} messages (again)
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 9be437064 -> 06c55f779
refs/heads/cassandra-3.11 2725d5035 -> 5d5588204
refs/heads/trunk 047bcd7ad -> d44059bfc
Static deletions are corrupted in 3.0 -> 2.{1,2} messages (again)
The prior fix was incorrect; it turns out serialization to 2.{1,2} nodes was
broken as well,
not just deserialization from 2.{1,2} nodes.
patch by Benedict; reviewed by Aleksey and Sylvain for CASSANDRA-14568
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68dbeb34
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68dbeb34
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68dbeb34
Branch: refs/heads/cassandra-3.0
Commit: 68dbeb34c9404ee3cd7db00cc112e27c9a4b1f6f
Parents: 9be4370
Author: Benedict Elliott Smith
Authored: Wed Aug 15 18:38:07 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:16:50 2018 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/db/LegacyLayout.java | 34 ++--
.../apache/cassandra/db/LegacyLayoutTest.java | 1 -
3 files changed, 26 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/68dbeb34/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12c16b7..037e2a8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/68dbeb34/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index e0f66e3..2115c7d 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -25,7 +25,6 @@ import java.security.MessageDigest;
import java.util.*;
import org.apache.cassandra.cql3.SuperColumnCompatibility;
-import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.utils.AbstractIterator;
import com.google.common.collect.Iterators;
import com.google.common.collect.Lists;
@@ -46,6 +45,7 @@ import org.apache.cassandra.utils.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static com.google.common.collect.Iterables.all;
import static org.apache.cassandra.utils.ByteBufferUtil.bytes;
/**
@@ -201,16 +201,19 @@ public abstract class LegacyLayout
List components = CompositeType.splitName(bound);
byte eoc = CompositeType.lastEOC(bound);
+// if the bound we have decoded is static, 2.2 format requires there
to be N empty clusterings
+assert !isStatic ||
+(components.size() >= clusteringSize
+&& all(components.subList(0, clusteringSize),
ByteBufferUtil.EMPTY_BYTE_BUFFER::equals));
// There can be more components than the clustering size only in the
case this is the bound of a collection
// range tombstone. In which case, there is exactly one more
component, and that component is the name of the
// collection being selected/deleted.
-assert components.size() <= clusteringSize ||
(!metadata.isCompactTable() && components.size() == clusteringSize + 1);
-
ColumnDefinition collectionName = null;
-if (components.size() > (isStatic ? 0 : clusteringSize))
+if (components.size() > clusteringSize)
{
+assert clusteringSize + 1 == components.size() &&
!metadata.isCompactTable();
// pop the collection name from the back of the list of clusterings
-ByteBuffer collectionNameBytes = components.remove(isStatic ? 0 :
clusteringSize);
+ByteBuffer collectionNameBytes = components.remove(clusteringSize);
collectionName = metadata.getColumnDefinition(collectionNameBytes);
}
@@ -799,12 +802,18 @@ public abstract class LegacyLayout
if (!delTime.isLive())
{
Clustering clustering = row.clustering();
+boolean isStatic = clustering == Clustering.STATIC_CLUSTERING;
+assert isStatic == col.isStatic();
-Slice.Bound startBound =
Slice.Bound.inclusiveStartOf(clustering);
-Slice.Bound endBound = Slice.Bound.inclusiveEndOf(clustering);
+
[5/9] cassandra git commit: Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
patch by Benedict; reviewed by Sylvain Lebresne and Aleksey Yeschenko for
CASSANDRA-14749
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/06c55f77
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/06c55f77
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/06c55f77
Branch: refs/heads/cassandra-3.11
Commit: 06c55f779ae68de98cce531e0b78be5716849003
Parents: 68dbeb3
Author: Benedict Elliott Smith
Authored: Thu Sep 13 17:09:58 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:17:59 2018 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/db/LegacyLayout.java | 5 +++
.../apache/cassandra/db/LegacyLayoutTest.java | 39
3 files changed, 45 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 037e2a8..195c97c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix corrupted collection deletions for dropped columns in 3.0 <-> 2.{1,2}
messages (CASSANDRA-14568)
* Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 2115c7d..ecc507e 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -215,6 +215,11 @@ public abstract class LegacyLayout
// pop the collection name from the back of the list of clusterings
ByteBuffer collectionNameBytes = components.remove(clusteringSize);
collectionName = metadata.getColumnDefinition(collectionNameBytes);
+if (collectionName == null) {
+collectionName =
metadata.getDroppedColumnDefinition(collectionNameBytes, isStatic);
+if (collectionName == null)
+throw new RuntimeException("Unknown collection column " +
UTF8Type.instance.getString(collectionNameBytes) + " during deserialization");
+}
}
boolean isInclusive;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
index ce818c0..2bed35e 100644
--- a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
+++ b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
@@ -252,4 +252,43 @@ public class LegacyLayoutTest
}
}
+@Test
+public void testStaticRangeTombstoneRoundTripDroppedColumn() throws
Throwable
+{
+// this variant of the bug deletes a row with the same clustering key
value as the name of the static collection
+QueryProcessor.executeInternal(String.format("CREATE TABLE
\"%s\".legacy_static_rt_rt_dc (pk int, ck1 text, v int, s set, primary
key (pk, ck1))", KEYSPACE));
+Keyspace keyspace = Keyspace.open(KEYSPACE);
+CFMetaData table =
keyspace.getColumnFamilyStore("legacy_static_rt_rt_dc").metadata;
+ColumnDefinition v = table.getColumnDefinition(new
ColumnIdentifier("v", false));
+ColumnDefinition bug = table.getColumnDefinition(new
ColumnIdentifier("s", false));
+
+Row.Builder builder;
+builder = BTreeRow.unsortedBuilder(0);
+builder.newRow(new Clustering(UTF8Serializer.instance.serialize("a")));
+builder.addCell(BufferCell.live(table, v, 0L,
Int32Serializer.instance.serialize(1), null));
+builder.addComplexDeletion(bug, new DeletionTime(1L, 1));
+Row row = builder.build();
+
+DecoratedKey pk = table.decorateKey(ByteBufferUtil.bytes(1));
+PartitionUpdate upd = PartitionUpdate.singleRowUpdate(table, pk, row);
+
+// we need to perform the round trip in two parts here, with a column
drop inbetween
+try (RowIterator before =
FilteredRows.filter(upd.unfilteredIterator(), FBUtilities.nowInSeconds());
+ DataOutputBuffer serialized21 = new DataOutputBuffer())
+{
+
[3/9] cassandra git commit: Static deletions are corrupted in 3.0 -> 2.{1, 2} messages (again)
Static deletions are corrupted in 3.0 -> 2.{1,2} messages (again)
The prior fix was incorrect; it turns out serialization to 2.{1,2} nodes was
broken as well,
not just deserialization from 2.{1,2} nodes.
patch by Benedict; reviewed by Aleksey and Sylvain for CASSANDRA-14568
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68dbeb34
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68dbeb34
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68dbeb34
Branch: refs/heads/trunk
Commit: 68dbeb34c9404ee3cd7db00cc112e27c9a4b1f6f
Parents: 9be4370
Author: Benedict Elliott Smith
Authored: Wed Aug 15 18:38:07 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:16:50 2018 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/db/LegacyLayout.java | 34 ++--
.../apache/cassandra/db/LegacyLayoutTest.java | 1 -
3 files changed, 26 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/68dbeb34/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12c16b7..037e2a8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/68dbeb34/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index e0f66e3..2115c7d 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -25,7 +25,6 @@ import java.security.MessageDigest;
import java.util.*;
import org.apache.cassandra.cql3.SuperColumnCompatibility;
-import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.utils.AbstractIterator;
import com.google.common.collect.Iterators;
import com.google.common.collect.Lists;
@@ -46,6 +45,7 @@ import org.apache.cassandra.utils.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static com.google.common.collect.Iterables.all;
import static org.apache.cassandra.utils.ByteBufferUtil.bytes;
/**
@@ -201,16 +201,19 @@ public abstract class LegacyLayout
List components = CompositeType.splitName(bound);
byte eoc = CompositeType.lastEOC(bound);
+// if the bound we have decoded is static, 2.2 format requires there
to be N empty clusterings
+assert !isStatic ||
+(components.size() >= clusteringSize
+&& all(components.subList(0, clusteringSize),
ByteBufferUtil.EMPTY_BYTE_BUFFER::equals));
// There can be more components than the clustering size only in the
case this is the bound of a collection
// range tombstone. In which case, there is exactly one more
component, and that component is the name of the
// collection being selected/deleted.
-assert components.size() <= clusteringSize ||
(!metadata.isCompactTable() && components.size() == clusteringSize + 1);
-
ColumnDefinition collectionName = null;
-if (components.size() > (isStatic ? 0 : clusteringSize))
+if (components.size() > clusteringSize)
{
+assert clusteringSize + 1 == components.size() &&
!metadata.isCompactTable();
// pop the collection name from the back of the list of clusterings
-ByteBuffer collectionNameBytes = components.remove(isStatic ? 0 :
clusteringSize);
+ByteBuffer collectionNameBytes = components.remove(clusteringSize);
collectionName = metadata.getColumnDefinition(collectionNameBytes);
}
@@ -799,12 +802,18 @@ public abstract class LegacyLayout
if (!delTime.isLive())
{
Clustering clustering = row.clustering();
+boolean isStatic = clustering == Clustering.STATIC_CLUSTERING;
+assert isStatic == col.isStatic();
-Slice.Bound startBound =
Slice.Bound.inclusiveStartOf(clustering);
-Slice.Bound endBound = Slice.Bound.inclusiveEndOf(clustering);
+Slice.Bound startBound = isStatic
+? LegacyDeletionInfo.staticBound(metadata, true)
+: Slice.Bound.inclusiveStartOf(clustering);
+
[6/9] cassandra git commit: Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
patch by Benedict; reviewed by Sylvain Lebresne and Aleksey Yeschenko for
CASSANDRA-14749
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/06c55f77
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/06c55f77
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/06c55f77
Branch: refs/heads/trunk
Commit: 06c55f779ae68de98cce531e0b78be5716849003
Parents: 68dbeb3
Author: Benedict Elliott Smith
Authored: Thu Sep 13 17:09:58 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:17:59 2018 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/db/LegacyLayout.java | 5 +++
.../apache/cassandra/db/LegacyLayoutTest.java | 39
3 files changed, 45 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 037e2a8..195c97c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix corrupted collection deletions for dropped columns in 3.0 <-> 2.{1,2}
messages (CASSANDRA-14568)
* Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 2115c7d..ecc507e 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -215,6 +215,11 @@ public abstract class LegacyLayout
// pop the collection name from the back of the list of clusterings
ByteBuffer collectionNameBytes = components.remove(clusteringSize);
collectionName = metadata.getColumnDefinition(collectionNameBytes);
+if (collectionName == null) {
+collectionName =
metadata.getDroppedColumnDefinition(collectionNameBytes, isStatic);
+if (collectionName == null)
+throw new RuntimeException("Unknown collection column " +
UTF8Type.instance.getString(collectionNameBytes) + " during deserialization");
+}
}
boolean isInclusive;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
index ce818c0..2bed35e 100644
--- a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
+++ b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
@@ -252,4 +252,43 @@ public class LegacyLayoutTest
}
}
+@Test
+public void testStaticRangeTombstoneRoundTripDroppedColumn() throws
Throwable
+{
+// this variant of the bug deletes a row with the same clustering key
value as the name of the static collection
+QueryProcessor.executeInternal(String.format("CREATE TABLE
\"%s\".legacy_static_rt_rt_dc (pk int, ck1 text, v int, s set, primary
key (pk, ck1))", KEYSPACE));
+Keyspace keyspace = Keyspace.open(KEYSPACE);
+CFMetaData table =
keyspace.getColumnFamilyStore("legacy_static_rt_rt_dc").metadata;
+ColumnDefinition v = table.getColumnDefinition(new
ColumnIdentifier("v", false));
+ColumnDefinition bug = table.getColumnDefinition(new
ColumnIdentifier("s", false));
+
+Row.Builder builder;
+builder = BTreeRow.unsortedBuilder(0);
+builder.newRow(new Clustering(UTF8Serializer.instance.serialize("a")));
+builder.addCell(BufferCell.live(table, v, 0L,
Int32Serializer.instance.serialize(1), null));
+builder.addComplexDeletion(bug, new DeletionTime(1L, 1));
+Row row = builder.build();
+
+DecoratedKey pk = table.decorateKey(ByteBufferUtil.bytes(1));
+PartitionUpdate upd = PartitionUpdate.singleRowUpdate(table, pk, row);
+
+// we need to perform the round trip in two parts here, with a column
drop inbetween
+try (RowIterator before =
FilteredRows.filter(upd.unfilteredIterator(), FBUtilities.nowInSeconds());
+ DataOutputBuffer serialized21 = new DataOutputBuffer())
+{
+
[8/9] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5d558820
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5d558820
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5d558820
Branch: refs/heads/trunk
Commit: 5d558820486bbca6783da7797bcc472aad3b7dd3
Parents: 2725d50 06c55f7
Author: Benedict Elliott Smith
Authored: Fri Sep 14 11:24:41 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:24:41 2018 +0100
--
CHANGES.txt | 2 +
.../org/apache/cassandra/db/LegacyLayout.java | 38 +++
.../apache/cassandra/db/LegacyLayoutTest.java | 40 +++-
3 files changed, 71 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5d558820/CHANGES.txt
--
diff --cc CHANGES.txt
index 4f2a649,195c97c..f04cae1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,5 -1,6 +1,7 @@@
-3.0.18
+3.11.4
+Merged from 3.0:
+ * Fix corrupted collection deletions for dropped columns in 3.0 <-> 2.{1,2}
messages (CASSANDRA-14568)
+ * Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5d558820/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index eed4113,ecc507e..c697a8d
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -799,12 -807,18 +808,18 @@@ public abstract class LegacyLayou
if (!delTime.isLive())
{
Clustering clustering = row.clustering();
+ boolean isStatic = clustering == Clustering.STATIC_CLUSTERING;
+ assert isStatic == col.isStatic();
- ClusteringBound startBound =
ClusteringBound.inclusiveStartOf(clustering);
- ClusteringBound endBound =
ClusteringBound.inclusiveEndOf(clustering);
-Slice.Bound startBound = isStatic
++ClusteringBound startBound = isStatic
+ ? LegacyDeletionInfo.staticBound(metadata, true)
-: Slice.Bound.inclusiveStartOf(clustering);
-Slice.Bound endBound = isStatic
++: ClusteringBound.inclusiveStartOf(clustering);
++ClusteringBound endBound = isStatic
+ ? LegacyDeletionInfo.staticBound(metadata, false)
-: Slice.Bound.inclusiveEndOf(clustering);
++: ClusteringBound.inclusiveEndOf(clustering);
- LegacyLayout.LegacyBound start = new
LegacyLayout.LegacyBound(startBound, col.isStatic(), col);
- LegacyLayout.LegacyBound end = new
LegacyLayout.LegacyBound(endBound, col.isStatic(), col);
+ LegacyLayout.LegacyBound start = new
LegacyLayout.LegacyBound(startBound, isStatic, col);
+ LegacyLayout.LegacyBound end = new
LegacyLayout.LegacyBound(endBound, isStatic, col);
deletions.add(start, end, delTime.markedForDeleteAt(),
delTime.localDeletionTime());
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5d558820/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
--
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[9/9] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d44059bf Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d44059bf Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d44059bf Branch: refs/heads/trunk Commit: d44059bfc6aabc88a370a84b082a9dbaa5560d0e Parents: 047bcd7 5d55882 Author: Benedict Elliott Smith Authored: Fri Sep 14 11:25:27 2018 +0100 Committer: Benedict Elliott Smith Committed: Fri Sep 14 11:25:27 2018 +0100 -- CHANGES.txt | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44059bf/CHANGES.txt -- diff --cc CHANGES.txt index 3c3,f04cae1..76b7b8b --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,317 -1,7 +1,319 @@@ +4.0 + * Fail query on transient replica if coordinator only expects full data (CASSANDRA-14704) + * Remove mentions of transient replication from repair path (CASSANDRA-14698) + * Fix handleRepairStatusChangedNotification to remove first then add (CASSANDRA-14720) + * Allow transient node to serve as a repair coordinator (CASSANDRA-14693) + * DecayingEstimatedHistogramReservoir.EstimatedHistogramReservoirSnapshot returns wrong value for size() and incorrectly calculates count (CASSANDRA-14696) + * AbstractReplicaCollection equals and hash code should throw due to conflict between order sensitive/insensitive uses (CASSANDRA-14700) + * Detect inconsistencies in repaired data on the read path (CASSANDRA-14145) + * Add checksumming to the native protocol (CASSANDRA-13304) + * Make AuthCache more easily extendable (CASSANDRA-14662) + * Extend RolesCache to include detailed role info (CASSANDRA-14497) + * Add fqltool compare (CASSANDRA-14619) + * Add fqltool replay (CASSANDRA-14618) + * Log keyspace in full query log (CASSANDRA-14656) + * Transient Replication and Cheap Quorums (CASSANDRA-14404) + * Log server-generated timestamp and nowInSeconds used by queries in FQL (CASSANDRA-14675) + * Add diagnostic events for read repairs (CASSANDRA-14668) + * Use consistent nowInSeconds and timestamps values within a request (CASSANDRA-14671) + * Add sampler for query time and expose with nodetool (CASSANDRA-14436) + * Clean up Message.Request implementations (CASSANDRA-14677) + * Disable old native protocol versions on demand (CASANDRA-14659) + * Allow specifying now-in-seconds in native protocol (CASSANDRA-14664) + * Improve BTree build performance by avoiding data copy (CASSANDRA-9989) + * Make monotonic read / read repair configurable (CASSANDRA-14635) + * Refactor CompactionStrategyManager (CASSANDRA-14621) + * Flush netty client messages immediately by default (CASSANDRA-13651) + * Improve read repair blocking behavior (CASSANDRA-10726) + * Add a virtual table to expose settings (CASSANDRA-14573) + * Fix up chunk cache handling of metrics (CASSANDRA-14628) + * Extend IAuthenticator to accept peer SSL certificates (CASSANDRA-14652) + * Incomplete handling of exceptions when decoding incoming messages (CASSANDRA-14574) + * Add diagnostic events for user audit logging (CASSANDRA-13668) + * Allow retrieving diagnostic events via JMX (CASSANDRA-14435) + * Add base classes for diagnostic events (CASSANDRA-13457) + * Clear view system metadata when dropping keyspace (CASSANDRA-14646) + * Allocate ReentrantLock on-demand in java11 AtomicBTreePartitionerBase (CASSANDRA-14637) + * Make all existing virtual tables use LocalPartitioner (CASSANDRA-14640) + * Revert 4.0 GC alg back to CMS (CASANDRA-14636) + * Remove hardcoded java11 jvm args in idea workspace files (CASSANDRA-14627) + * Update netty to 4.1.128 (CASSANDRA-14633) + * Add a virtual table to expose thread pools (CASSANDRA-14523) + * Add a virtual table to expose caches (CASSANDRA-14538, CASSANDRA-14626) + * Fix toDate function for timestamp arguments (CASSANDRA-14502) + * Revert running dtests by default in circleci (CASSANDRA-14614) + * Stream entire SSTables when possible (CASSANDRA-14556) + * Cell reconciliation should not depend on nowInSec (CASSANDRA-14592) + * Add experimental support for Java 11 (CASSANDRA-9608) + * Make PeriodicCommitLogService.blockWhenSyncLagsNanos configurable (CASSANDRA-14580) + * Improve logging in MessageInHandler's constructor (CASSANDRA-14576) + * Set broadcast address in internode messaging handshake (CASSANDRA-14579) + * Wait for schema agreement prior to building MVs (CASSANDRA-14571) + * Make all DDL statements idempotent and not dependent on global state (CASSANDRA-13426) + * Bump the hints messaging version to match the current one (CASSANDRA-14536) + * OffsetAwareConfigurationLoader doesn't set ssl storage port causing bind errors in CircleCI
[4/9] cassandra git commit: Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
Collection Deletions for Dropped Columns in 2.1/3.0 mixed-mode can delete rows
patch by Benedict; reviewed by Sylvain Lebresne and Aleksey Yeschenko for
CASSANDRA-14749
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/06c55f77
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/06c55f77
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/06c55f77
Branch: refs/heads/cassandra-3.0
Commit: 06c55f779ae68de98cce531e0b78be5716849003
Parents: 68dbeb3
Author: Benedict Elliott Smith
Authored: Thu Sep 13 17:09:58 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:17:59 2018 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/db/LegacyLayout.java | 5 +++
.../apache/cassandra/db/LegacyLayoutTest.java | 39
3 files changed, 45 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 037e2a8..195c97c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix corrupted collection deletions for dropped columns in 3.0 <-> 2.{1,2}
messages (CASSANDRA-14568)
* Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 2115c7d..ecc507e 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -215,6 +215,11 @@ public abstract class LegacyLayout
// pop the collection name from the back of the list of clusterings
ByteBuffer collectionNameBytes = components.remove(clusteringSize);
collectionName = metadata.getColumnDefinition(collectionNameBytes);
+if (collectionName == null) {
+collectionName =
metadata.getDroppedColumnDefinition(collectionNameBytes, isStatic);
+if (collectionName == null)
+throw new RuntimeException("Unknown collection column " +
UTF8Type.instance.getString(collectionNameBytes) + " during deserialization");
+}
}
boolean isInclusive;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06c55f77/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
index ce818c0..2bed35e 100644
--- a/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
+++ b/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
@@ -252,4 +252,43 @@ public class LegacyLayoutTest
}
}
+@Test
+public void testStaticRangeTombstoneRoundTripDroppedColumn() throws
Throwable
+{
+// this variant of the bug deletes a row with the same clustering key
value as the name of the static collection
+QueryProcessor.executeInternal(String.format("CREATE TABLE
\"%s\".legacy_static_rt_rt_dc (pk int, ck1 text, v int, s set, primary
key (pk, ck1))", KEYSPACE));
+Keyspace keyspace = Keyspace.open(KEYSPACE);
+CFMetaData table =
keyspace.getColumnFamilyStore("legacy_static_rt_rt_dc").metadata;
+ColumnDefinition v = table.getColumnDefinition(new
ColumnIdentifier("v", false));
+ColumnDefinition bug = table.getColumnDefinition(new
ColumnIdentifier("s", false));
+
+Row.Builder builder;
+builder = BTreeRow.unsortedBuilder(0);
+builder.newRow(new Clustering(UTF8Serializer.instance.serialize("a")));
+builder.addCell(BufferCell.live(table, v, 0L,
Int32Serializer.instance.serialize(1), null));
+builder.addComplexDeletion(bug, new DeletionTime(1L, 1));
+Row row = builder.build();
+
+DecoratedKey pk = table.decorateKey(ByteBufferUtil.bytes(1));
+PartitionUpdate upd = PartitionUpdate.singleRowUpdate(table, pk, row);
+
+// we need to perform the round trip in two parts here, with a column
drop inbetween
+try (RowIterator before =
FilteredRows.filter(upd.unfilteredIterator(), FBUtilities.nowInSeconds());
+ DataOutputBuffer serialized21 = new DataOutputBuffer())
+{
+
[7/9] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5d558820
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5d558820
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5d558820
Branch: refs/heads/cassandra-3.11
Commit: 5d558820486bbca6783da7797bcc472aad3b7dd3
Parents: 2725d50 06c55f7
Author: Benedict Elliott Smith
Authored: Fri Sep 14 11:24:41 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:24:41 2018 +0100
--
CHANGES.txt | 2 +
.../org/apache/cassandra/db/LegacyLayout.java | 38 +++
.../apache/cassandra/db/LegacyLayoutTest.java | 40 +++-
3 files changed, 71 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5d558820/CHANGES.txt
--
diff --cc CHANGES.txt
index 4f2a649,195c97c..f04cae1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,5 -1,6 +1,7 @@@
-3.0.18
+3.11.4
+Merged from 3.0:
+ * Fix corrupted collection deletions for dropped columns in 3.0 <-> 2.{1,2}
messages (CASSANDRA-14568)
+ * Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5d558820/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index eed4113,ecc507e..c697a8d
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -799,12 -807,18 +808,18 @@@ public abstract class LegacyLayou
if (!delTime.isLive())
{
Clustering clustering = row.clustering();
+ boolean isStatic = clustering == Clustering.STATIC_CLUSTERING;
+ assert isStatic == col.isStatic();
- ClusteringBound startBound =
ClusteringBound.inclusiveStartOf(clustering);
- ClusteringBound endBound =
ClusteringBound.inclusiveEndOf(clustering);
-Slice.Bound startBound = isStatic
++ClusteringBound startBound = isStatic
+ ? LegacyDeletionInfo.staticBound(metadata, true)
-: Slice.Bound.inclusiveStartOf(clustering);
-Slice.Bound endBound = isStatic
++: ClusteringBound.inclusiveStartOf(clustering);
++ClusteringBound endBound = isStatic
+ ? LegacyDeletionInfo.staticBound(metadata, false)
-: Slice.Bound.inclusiveEndOf(clustering);
++: ClusteringBound.inclusiveEndOf(clustering);
- LegacyLayout.LegacyBound start = new
LegacyLayout.LegacyBound(startBound, col.isStatic(), col);
- LegacyLayout.LegacyBound end = new
LegacyLayout.LegacyBound(endBound, col.isStatic(), col);
+ LegacyLayout.LegacyBound start = new
LegacyLayout.LegacyBound(startBound, isStatic, col);
+ LegacyLayout.LegacyBound end = new
LegacyLayout.LegacyBound(endBound, isStatic, col);
deletions.add(start, end, delTime.markedForDeleteAt(),
delTime.localDeletionTime());
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5d558820/test/unit/org/apache/cassandra/db/LegacyLayoutTest.java
--
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/9] cassandra git commit: Static deletions are corrupted in 3.0 -> 2.{1, 2} messages (again)
Static deletions are corrupted in 3.0 -> 2.{1,2} messages (again)
The prior fix was incorrect; it turns out serialization to 2.{1,2} nodes was
broken as well,
not just deserialization from 2.{1,2} nodes.
patch by Benedict; reviewed by Aleksey and Sylvain for CASSANDRA-14568
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68dbeb34
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68dbeb34
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68dbeb34
Branch: refs/heads/cassandra-3.11
Commit: 68dbeb34c9404ee3cd7db00cc112e27c9a4b1f6f
Parents: 9be4370
Author: Benedict Elliott Smith
Authored: Wed Aug 15 18:38:07 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 11:16:50 2018 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/db/LegacyLayout.java | 34 ++--
.../apache/cassandra/db/LegacyLayoutTest.java | 1 -
3 files changed, 26 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/68dbeb34/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 12c16b7..037e2a8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Fix corrupted static collection deletions in 3.0 <-> 2.{1,2} messages
(CASSANDRA-14568)
* Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/68dbeb34/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index e0f66e3..2115c7d 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -25,7 +25,6 @@ import java.security.MessageDigest;
import java.util.*;
import org.apache.cassandra.cql3.SuperColumnCompatibility;
-import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.utils.AbstractIterator;
import com.google.common.collect.Iterators;
import com.google.common.collect.Lists;
@@ -46,6 +45,7 @@ import org.apache.cassandra.utils.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static com.google.common.collect.Iterables.all;
import static org.apache.cassandra.utils.ByteBufferUtil.bytes;
/**
@@ -201,16 +201,19 @@ public abstract class LegacyLayout
List components = CompositeType.splitName(bound);
byte eoc = CompositeType.lastEOC(bound);
+// if the bound we have decoded is static, 2.2 format requires there
to be N empty clusterings
+assert !isStatic ||
+(components.size() >= clusteringSize
+&& all(components.subList(0, clusteringSize),
ByteBufferUtil.EMPTY_BYTE_BUFFER::equals));
// There can be more components than the clustering size only in the
case this is the bound of a collection
// range tombstone. In which case, there is exactly one more
component, and that component is the name of the
// collection being selected/deleted.
-assert components.size() <= clusteringSize ||
(!metadata.isCompactTable() && components.size() == clusteringSize + 1);
-
ColumnDefinition collectionName = null;
-if (components.size() > (isStatic ? 0 : clusteringSize))
+if (components.size() > clusteringSize)
{
+assert clusteringSize + 1 == components.size() &&
!metadata.isCompactTable();
// pop the collection name from the back of the list of clusterings
-ByteBuffer collectionNameBytes = components.remove(isStatic ? 0 :
clusteringSize);
+ByteBuffer collectionNameBytes = components.remove(clusteringSize);
collectionName = metadata.getColumnDefinition(collectionNameBytes);
}
@@ -799,12 +802,18 @@ public abstract class LegacyLayout
if (!delTime.isLive())
{
Clustering clustering = row.clustering();
+boolean isStatic = clustering == Clustering.STATIC_CLUSTERING;
+assert isStatic == col.isStatic();
-Slice.Bound startBound =
Slice.Bound.inclusiveStartOf(clustering);
-Slice.Bound endBound = Slice.Bound.inclusiveEndOf(clustering);
+Slice.Bound startBound = isStatic
+? LegacyDeletionInfo.staticBound(metadata, true)
+:
[jira] [Updated] (CASSANDRA-14750) Missing check for receiving digest response from transient node
[ https://issues.apache.org/jira/browse/CASSANDRA-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRA-14750: --- Labels: pull-request-available (was: ) > Missing check for receiving digest response from transient node > --- > > Key: CASSANDRA-14750 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14750 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Major > Labels: pull-request-available > > Range reads do not check for transient nodes returning a request. Read > Command also currently allows a combination of transient node and digest > query. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14750) Missing check for receiving digest response from transient node
[ https://issues.apache.org/jira/browse/CASSANDRA-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-14750: Status: Patch Available (was: Open) > Missing check for receiving digest response from transient node > --- > > Key: CASSANDRA-14750 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14750 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Major > > Range reads do not check for transient nodes returning a request. Read > Command also currently allows a combination of transient node and digest > query. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14750) Missing check for receiving digest response from transient node
[ https://issues.apache.org/jira/browse/CASSANDRA-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14750: - Reviewer: Benedict > Missing check for receiving digest response from transient node > --- > > Key: CASSANDRA-14750 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14750 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Major > > Range reads do not check for transient nodes returning a request. Read > Command also currently allows a combination of transient node and digest > query. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/4] cassandra git commit: ReplicaPlan/Layout refactor follow-up/completion
ReplicaPlan/Layout refactor follow-up/completion
Finish much of the work to clarify endpoint selection
that was begun in Transient Replication (CASSANDRA-14404)
Also fixes:
- commitPaxos was incorrectly selecting only live nodes,
when needed to include down
- We were not writing to pending transient replicas
- On write, we were not hinting to full nodes with transient
replication
- rr.maybeSendAdditional{Reads,Writes} would only consult the
same node we may have speculated a read to
- transient->full movements mishandled consistency level upgrade by
retaining the 'full' pending variant, which increased CL requirement;
instead, the 'natural' replica is upgraded to 'full' for writes
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for
CASSANDRA-14705
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/047bcd7a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/047bcd7a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/047bcd7a
Branch: refs/heads/trunk
Commit: 047bcd7ad171d6a4aa89128c5e6c6ed5f012b1c0
Parents: 05dbb3e
Author: Benedict Elliott Smith
Authored: Fri Sep 7 11:41:28 2018 +0100
Committer: Benedict Elliott Smith
Committed: Fri Sep 14 10:14:37 2018 +0100
--
.../cassandra/batchlog/BatchlogManager.java | 64 ++-
.../apache/cassandra/db/ConsistencyLevel.java | 26 +-
.../apache/cassandra/gms/FailureDetector.java | 4 +
.../apache/cassandra/hints/HintsService.java| 4 +-
.../locator/AbstractReplicaCollection.java | 67 +--
.../locator/AbstractReplicationStrategy.java| 20 +-
.../org/apache/cassandra/locator/Endpoints.java | 59 +--
.../cassandra/locator/EndpointsForRange.java| 2 +-
.../cassandra/locator/EndpointsForToken.java| 2 +-
.../cassandra/locator/RangesAtEndpoint.java | 2 +-
.../cassandra/locator/ReplicaCollection.java| 27 +-
.../apache/cassandra/locator/ReplicaLayout.java | 435 ---
.../apache/cassandra/locator/ReplicaPlan.java | 240 ++
.../apache/cassandra/locator/ReplicaPlans.java | 295 +
.../cassandra/locator/SystemReplicas.java | 12 +-
.../apache/cassandra/locator/TokenMetadata.java | 8 +-
.../apache/cassandra/net/IAsyncCallback.java| 10 -
.../service/AbstractWriteResponseHandler.java | 50 +--
.../cassandra/service/ActiveRepairService.java | 2 +-
.../service/BatchlogResponseHandler.java| 15 +-
.../DatacenterSyncWriteResponseHandler.java | 12 +-
.../service/DatacenterWriteResponseHandler.java | 10 +-
.../apache/cassandra/service/StorageProxy.java | 239 +-
.../cassandra/service/StorageService.java | 82 +---
.../cassandra/service/WriteResponseHandler.java | 16 +-
.../service/reads/AbstractReadExecutor.java | 78 ++--
.../cassandra/service/reads/DataResolver.java | 26 +-
.../cassandra/service/reads/DigestResolver.java | 22 +-
.../cassandra/service/reads/ReadCallback.java | 62 ++-
.../service/reads/ResponseResolver.java | 22 +-
.../reads/ShortReadPartitionsProtection.java| 19 +-
.../reads/repair/AbstractReadRepair.java| 42 +-
.../reads/repair/BlockingPartitionRepair.java | 29 +-
.../reads/repair/BlockingReadRepair.java| 25 +-
.../service/reads/repair/NoopReadRepair.java| 11 +-
.../repair/PartitionIteratorMergeListener.java | 16 +-
.../reads/repair/ReadOnlyReadRepair.java| 13 +-
.../service/reads/repair/ReadRepair.java| 17 +-
.../reads/repair/ReadRepairDiagnostics.java | 11 +-
.../service/reads/repair/ReadRepairEvent.java | 2 +-
.../reads/repair/ReadRepairStrategy.java| 11 +-
.../reads/repair/RowIteratorMergeListener.java | 21 +-
.../locator/ReplicaCollectionTest.java | 89 ++--
.../service/WriteResponseHandlerTest.java | 4 +-
.../WriteResponseHandlerTransientTest.java | 71 ++-
.../service/reads/DataResolverTest.java | 21 +-
.../reads/DataResolverTransientTest.java| 227 ++
.../service/reads/DigestResolverTest.java | 5 +-
.../service/reads/ReadExecutorTest.java | 7 +-
.../reads/repair/AbstractReadRepairTest.java| 32 +-
.../reads/repair/BlockingReadRepairTest.java| 36 +-
.../DiagEventsBlockingReadRepairTest.java | 26 +-
.../reads/repair/InstrumentedReadRepair.java| 4 +-
.../reads/repair/ReadOnlyReadRepairTest.java| 24 +-
.../service/reads/repair/ReadRepairTest.java| 12 +-
.../reads/repair/TestableReadRepair.java| 18 +-
56 files changed, 1655 insertions(+), 1051 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/047bcd7a/src/java/org/apache/cassandra/batchlog/BatchlogManager.java
[2/4] cassandra git commit: ReplicaPlan/Layout refactor follow-up/completion
http://git-wip-us.apache.org/repos/asf/cassandra/blob/047bcd7a/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
--
diff --git
a/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
b/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
index 75885ae..c296cba 100644
--- a/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
+++ b/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java
@@ -21,8 +21,9 @@ import java.util.concurrent.TimeUnit;
import com.google.common.base.Preconditions;
-import org.apache.cassandra.locator.ReplicaLayout;
-
+import com.google.common.base.Predicates;
+import org.apache.cassandra.locator.ReplicaPlan;
+import org.apache.cassandra.locator.ReplicaPlans;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -50,7 +51,6 @@ import org.apache.cassandra.tracing.TraceState;
import org.apache.cassandra.tracing.Tracing;
import static com.google.common.collect.Iterables.all;
-import static com.google.common.collect.Iterables.tryFind;
/**
* Sends a read request to the replicas needed to satisfy a given
ConsistencyLevel.
@@ -65,24 +65,25 @@ public abstract class AbstractReadExecutor
private static final Logger logger =
LoggerFactory.getLogger(AbstractReadExecutor.class);
protected final ReadCommand command;
-private final ReplicaLayout.ForToken replicaLayout;
-protected final ReadRepair
readRepair;
-protected final DigestResolver
digestResolver;
-protected final ReadCallback
handler;
+private final ReplicaPlan.SharedForTokenRead replicaPlan;
+protected final ReadRepair
readRepair;
+protected final DigestResolver digestResolver;
+protected final ReadCallback
handler;
protected final TraceState traceState;
protected final ColumnFamilyStore cfs;
protected final long queryStartNanoTime;
private final int initialDataRequestCount;
protected volatile PartitionIterator result = null;
-AbstractReadExecutor(ColumnFamilyStore cfs, ReadCommand command,
ReplicaLayout.ForToken replicaLayout, int initialDataRequestCount, long
queryStartNanoTime)
+AbstractReadExecutor(ColumnFamilyStore cfs, ReadCommand command,
ReplicaPlan.ForTokenRead replicaPlan, int initialDataRequestCount, long
queryStartNanoTime)
{
this.command = command;
-this.replicaLayout = replicaLayout;
+this.replicaPlan = ReplicaPlan.shared(replicaPlan);
this.initialDataRequestCount = initialDataRequestCount;
-this.readRepair = ReadRepair.create(command, replicaLayout,
queryStartNanoTime);
-this.digestResolver = new DigestResolver<>(command, replicaLayout,
queryStartNanoTime);
-this.handler = new ReadCallback<>(digestResolver,
replicaLayout.consistencyLevel().blockFor(replicaLayout.keyspace()), command,
replicaLayout, queryStartNanoTime);
+// the ReadRepair and DigestResolver both need to see our updated
+this.readRepair = ReadRepair.create(command, this.replicaPlan,
queryStartNanoTime);
+this.digestResolver = new DigestResolver<>(command, this.replicaPlan,
queryStartNanoTime);
+this.handler = new ReadCallback<>(digestResolver, command,
this.replicaPlan, queryStartNanoTime);
this.cfs = cfs;
this.traceState = Tracing.instance.get();
this.queryStartNanoTime = queryStartNanoTime;
@@ -93,7 +94,7 @@ public abstract class AbstractReadExecutor
// TODO: we need this when talking with pre-3.0 nodes. So if we
preserve the digest format moving forward, we can get rid of this once
// we stop being compatible with pre-3.0 nodes.
int digestVersion = MessagingService.current_version;
-for (Replica replica : replicaLayout.selected())
+for (Replica replica : replicaPlan.contacts())
digestVersion = Math.min(digestVersion,
MessagingService.instance().getVersion(replica.endpoint()));
command.setDigestVersion(digestVersion);
}
@@ -168,7 +169,7 @@ public abstract class AbstractReadExecutor
*/
public void executeAsync()
{
-EndpointsForToken selected = replicaLayout().selected();
+EndpointsForToken selected = replicaPlan().contacts();
EndpointsForToken fullDataRequests = selected.filter(Replica::isFull,
initialDataRequestCount);
makeFullDataRequests(fullDataRequests);
makeTransientDataRequests(selected.filter(Replica::isTransient));
@@ -184,30 +185,25 @@ public abstract class AbstractReadExecutor
ColumnFamilyStore cfs =
keyspace.getColumnFamilyStore(command.metadata().id);
SpeculativeRetryPolicy retry = cfs.metadata().params.speculativeRetry;
-// Endpoints for Token
-ReplicaLayout.ForToken replicaLayout = ReplicaLayout.forRead(keyspace,
command.partitionKey().getToken(), consistencyLevel, retry);
+
[1/4] cassandra git commit: ReplicaPlan/Layout refactor follow-up/completion
Repository: cassandra
Updated Branches:
refs/heads/trunk 05dbb3e0a -> 047bcd7ad
http://git-wip-us.apache.org/repos/asf/cassandra/blob/047bcd7a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
--
diff --git
a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
index cf1e06a..a3f13c2 100644
--- a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
+++ b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTest.java
@@ -27,7 +27,7 @@ import java.util.concurrent.TimeUnit;
import com.google.common.base.Predicates;
import org.apache.cassandra.dht.Murmur3Partitioner;
import org.apache.cassandra.locator.EndpointsForToken;
-import org.apache.cassandra.locator.ReplicaLayout;
+import org.apache.cassandra.locator.ReplicaPlans;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -236,7 +236,7 @@ public class WriteResponseHandlerTest
private static AbstractWriteResponseHandler
createWriteResponseHandler(ConsistencyLevel cl, ConsistencyLevel ideal, long
queryStartTime)
{
-return
ks.getReplicationStrategy().getWriteResponseHandler(ReplicaLayout.forWriteWithDownNodes(ks,
cl, targets.token(), targets, pending),
+return
ks.getReplicationStrategy().getWriteResponseHandler(ReplicaPlans.forWrite(ks,
cl, targets, pending, Predicates.alwaysTrue(), ReplicaPlans.writeAll),
null,
WriteType.SIMPLE, queryStartTime, ideal);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/047bcd7a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
--
diff --git
a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
index c19e65e..b6c95dd 100644
---
a/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
+++
b/test/unit/org/apache/cassandra/service/WriteResponseHandlerTransientTest.java
@@ -24,7 +24,6 @@ import java.util.Set;
import java.util.UUID;
import java.util.function.Predicate;
-import com.google.common.base.Predicates;
import com.google.common.collect.Sets;
import org.apache.cassandra.dht.Murmur3Partitioner;
@@ -32,10 +31,10 @@ import org.apache.cassandra.dht.Token;
import org.apache.cassandra.locator.EndpointsForToken;
import org.apache.cassandra.locator.ReplicaLayout;
import org.apache.cassandra.locator.EndpointsForRange;
-import org.apache.cassandra.locator.ReplicaUtils;
+import org.apache.cassandra.locator.ReplicaPlan;
+import org.apache.cassandra.locator.ReplicaPlans;
import org.junit.Assert;
import org.junit.BeforeClass;
-import org.junit.Ignore;
import org.junit.Test;
import org.apache.cassandra.SchemaLoader;
@@ -54,7 +53,6 @@ import org.apache.cassandra.utils.ByteBufferUtil;
import static org.apache.cassandra.locator.Replica.fullReplica;
import static org.apache.cassandra.locator.Replica.transientReplica;
-import static org.apache.cassandra.locator.ReplicaUtils.FULL_RANGE;
import static org.apache.cassandra.locator.ReplicaUtils.full;
import static org.apache.cassandra.locator.ReplicaUtils.trans;
@@ -147,32 +145,35 @@ public class WriteResponseHandlerTransientTest
dummy =
DatabaseDescriptor.getPartitioner().getToken(ByteBufferUtil.bytes(0));
}
-@Ignore("Throws unavailable for quorum as written")
@Test
public void checkPendingReplicasAreNotFiltered()
{
-EndpointsForToken natural = EndpointsForToken.of(dummy.getToken(),
full(EP1), full(EP2), trans(EP3));
-EndpointsForToken pending = EndpointsForToken.of(dummy.getToken(),
full(EP4), full(EP5), trans(EP6));
-ReplicaLayout.ForToken replicaLayout = ReplicaLayout.forWrite(ks,
ConsistencyLevel.QUORUM, dummy.getToken(), 2, natural, pending,
Predicates.alwaysTrue());
+EndpointsForToken natural = EndpointsForToken.of(dummy.getToken(),
full(EP1), full(EP2), trans(EP3), full(EP5));
+EndpointsForToken pending = EndpointsForToken.of(dummy.getToken(),
full(EP4), trans(EP6));
+ReplicaLayout.ForTokenWrite layout = new
ReplicaLayout.ForTokenWrite(natural, pending);
+ReplicaPlan.ForTokenWrite replicaPlan = ReplicaPlans.forWrite(ks,
ConsistencyLevel.QUORUM, layout, layout, ReplicaPlans.writeAll);
-Assert.assertEquals(EndpointsForRange.of(full(EP4), full(EP5),
trans(EP6)), replicaLayout.pending());
+Assert.assertEquals(EndpointsForRange.of(full(EP4), trans(EP6)),
replicaPlan.pending());
}
-private static ReplicaLayout.ForToken expected(EndpointsForToken all,
EndpointsForToken selected)
+private static ReplicaPlan.ForTokenWrite expected(EndpointsForToken
natural,
[3/4] cassandra git commit: ReplicaPlan/Layout refactor follow-up/completion
http://git-wip-us.apache.org/repos/asf/cassandra/blob/047bcd7a/src/java/org/apache/cassandra/locator/ReplicaPlans.java
--
diff --git a/src/java/org/apache/cassandra/locator/ReplicaPlans.java
b/src/java/org/apache/cassandra/locator/ReplicaPlans.java
new file mode 100644
index 000..25f42c3
--- /dev/null
+++ b/src/java/org/apache/cassandra/locator/ReplicaPlans.java
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.locator;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ConsistencyLevel;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.PartitionPosition;
+import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.UnavailableException;
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.service.reads.AlwaysSpeculativeRetryPolicy;
+import org.apache.cassandra.service.reads.SpeculativeRetryPolicy;
+import org.apache.cassandra.utils.FBUtilities;
+
+import java.util.Collection;
+import java.util.function.Predicate;
+
+import static com.google.common.collect.Iterables.any;
+import static com.google.common.collect.Iterables.filter;
+import static com.google.common.collect.Iterables.limit;
+
+public class ReplicaPlans
+{
+
+/**
+ * Construct a ReplicaPlan for writing to exactly one node, with CL.ONE.
This node is *assumed* to be alive.
+ */
+public static ReplicaPlan.ForTokenWrite forSingleReplicaWrite(Keyspace
keyspace, Token token, Replica replica)
+{
+EndpointsForToken one = EndpointsForToken.of(token, replica);
+EndpointsForToken empty = EndpointsForToken.empty(token);
+return new ReplicaPlan.ForTokenWrite(keyspace, ConsistencyLevel.ONE,
empty, one, one, one);
+}
+
+/**
+ * A forwarding counter write is always sent to a single owning
coordinator for the range, by the original coordinator
+ * (if it is not itself an owner)
+ */
+public static ReplicaPlan.ForTokenWrite forForwardingCounterWrite(Keyspace
keyspace, Token token, Replica replica)
+{
+return forSingleReplicaWrite(keyspace, token, replica);
+}
+
+/**
+ * Requires that the provided endpoints are alive. Converts them to their
relevant system replicas.
+ * Note that the liveAndDown collection and live are equal to the provided
endpoints.
+ *
+ * The semantics are a bit weird, in that CL=ONE iff we have one node
provided, and otherwise is equal to TWO.
+ * How these CL were chosen, and why we drop the CL if only one live node
is available, are both unclear.
+ */
+public static ReplicaPlan.ForTokenWrite forBatchlogWrite(Keyspace
keyspace, Collection endpoints) throws UnavailableException
+{
+// A single case we write not for range or token, but multiple
mutations to many tokens
+Token token = DatabaseDescriptor.getPartitioner().getMinimumToken();
+
+ReplicaLayout.ForTokenWrite liveAndDown = ReplicaLayout.forTokenWrite(
+SystemReplicas.getSystemReplicas(endpoints).forToken(token),
+EndpointsForToken.empty(token)
+);
+ConsistencyLevel consistencyLevel = liveAndDown.all().size() == 1 ?
ConsistencyLevel.ONE : ConsistencyLevel.TWO;
+
+// assume that we have already been given live endpoints, and skip
applying the failure detector
+return forWrite(keyspace, consistencyLevel, liveAndDown, liveAndDown,
writeAll);
+}
+
+public static ReplicaPlan.ForTokenWrite forWrite(Keyspace keyspace,
ConsistencyLevel consistencyLevel, Token token, Selector selector) throws
UnavailableException
+{
+return forWrite(keyspace, consistencyLevel,
ReplicaLayout.forTokenWriteLiveAndDown(keyspace, token), selector);
+}
+
+@VisibleForTesting
+public static ReplicaPlan.ForTokenWrite forWrite(Keyspace keyspace,
ConsistencyLevel consistencyLevel, EndpointsForToken natural, EndpointsForToken
pending,
[jira] [Commented] (CASSANDRA-14568) Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
[
https://issues.apache.org/jira/browse/CASSANDRA-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614518#comment-16614518
]
Sylvain Lebresne commented on CASSANDRA-14568:
--
+1 on this one.
> Static collection deletions are corrupted in 3.0 -> 2.{1,2} messages
>
>
> Key: CASSANDRA-14568
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14568
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Fix For: 3.0.17, 3.11.3
>
>
> In 2.1 and 2.2, row and complex deletions were represented as range
> tombstones. LegacyLayout is our compatibility layer, that translates the
> relevant RT patterns in 2.1/2.2 to row/complex deletions in 3.0, and vice
> versa. Unfortunately, it does not handle the special case of static row
> deletions, they are treated as regular row deletions. Since static rows are
> themselves never directly deleted, the only issue is with collection
> deletions.
> Collection deletions in 2.1/2.2 were encoded as a range tombstone, consisting
> of a sequence of the clustering keys’ data for the affected row, followed by
> the bytes representing the name of the collection column. STATIC_CLUSTERING
> contains zero clusterings, so by treating the deletion as for a regular row,
> zero clusterings are written to precede the column name of the erased
> collection, so the column name is written at position zero.
> This can exhibit itself in at least two ways:
> # If the type of your first clustering key is a variable width type, new
> deletes will begin appearing covering the clustering key represented by the
> column name.
> ** If you have multiple clustering keys, you will receive a RT covering all
> those rows with a matching first clustering key.
> ** This RT will be valid as far as the system is concerned, and go
> undetected unless there are outside data quality checks in place.
> # Otherwise, an invalid size of data will be written to the clustering and
> sent over the network to the 2.1 node.
> ** The 2.1/2.2 node will handle this just fine, even though the record is
> junk. Since it is a deletion covering impossible data, there will be no
> user-API visible effect. But if received as a write from a 3.0 node, it will
> dutifully persist the junk record.
> ** The 3.0 node that originally sent this junk, may later coordinate a read
> of the partition, and will notice a digest mismatch, read-repair and
> serialize the junk to disk
> ** The sstable containing this record is now corrupt; the deserialization
> expects fixed-width data, but it encounters too many (or too few) bytes, and
> is now at an incorrect position to read its structural information
> ** (Alternatively when the 2.1 node is upgraded this will occur on eventual
> compaction)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14752) serializers/BooleanSerializer.java is using static bytebuffers which may cause problem for subsequent operations
[
https://issues.apache.org/jira/browse/CASSANDRA-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Barala updated CASSANDRA-14752:
-
Attachment: patch
> serializers/BooleanSerializer.java is using static bytebuffers which may
> cause problem for subsequent operations
>
>
> Key: CASSANDRA-14752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14752
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Varun Barala
>Priority: Major
> Attachments: patch
>
>
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/serializers/BooleanSerializer.java#L26]
> It has two static Bytebuffer variables:-
> {code:java}
> private static final ByteBuffer TRUE = ByteBuffer.wrap(new byte[]{1});
> private static final ByteBuffer FALSE = ByteBuffer.wrap(new byte[]{0});{code}
> What will happen if the position of these Bytebuffers is being changed by
> some other operations? It'll affect other subsequent operations. IMO Using
> static is not a good idea here.
> A potential place where it can become problematic:
> [https://github.com/apache/cassandra/blob/cassandra-2.1.13/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java#L243]
> Since we are calling *`.remaining()`* It may give wrong results _i.e 0_ if
> these Bytebuffers have been used previously.
> Solution:
>
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/serializers/BooleanSerializer.java#L42]
> Every time we return new bytebuffer object. Please do let me know If there
> is a better way. I'd like to contribute. Thanks!!
> {code:java}
> public ByteBuffer serialize(Boolean value)
> {
> return (value == null) ? ByteBufferUtil.EMPTY_BYTE_BUFFER
> : value ? ByteBuffer.wrap(new byte[] {1}) : ByteBuffer.wrap(new byte[] {0});
> // false
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14752) serializers/BooleanSerializer.java is using static bytebuffers which may cause problem for subsequent operations
[
https://issues.apache.org/jira/browse/CASSANDRA-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Barala updated CASSANDRA-14752:
-
Summary: serializers/BooleanSerializer.java is using static bytebuffers
which may cause problem for subsequent operations (was:
serializers/BooleanSerializer.java is using static bytebuffers which may cause
problem for subsequent oeprations)
> serializers/BooleanSerializer.java is using static bytebuffers which may
> cause problem for subsequent operations
>
>
> Key: CASSANDRA-14752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14752
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Varun Barala
>Priority: Major
>
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/serializers/BooleanSerializer.java#L26]
> It has two static Bytebuffer variables:-
> {code:java}
> private static final ByteBuffer TRUE = ByteBuffer.wrap(new byte[]{1});
> private static final ByteBuffer FALSE = ByteBuffer.wrap(new byte[]{0});{code}
> What will happen if the position of these Bytebuffers is being changed by
> some other operations? It'll affect other subsequent operations. IMO Using
> static is not a good idea here.
> A potential place where it can become problematic:
> [https://github.com/apache/cassandra/blob/cassandra-2.1.13/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java#L243]
> Since we are calling *`.remaining()`* It may give wrong results _i.e 0_ if
> these Bytebuffers have been used previously.
> Solution:
>
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/serializers/BooleanSerializer.java#L42]
> Every time we return new bytebuffer object. Please do let me know If there
> is a better way. I'd like to contribute. Thanks!!
> {code:java}
> public ByteBuffer serialize(Boolean value)
> {
> return (value == null) ? ByteBufferUtil.EMPTY_BYTE_BUFFER
> : value ? ByteBuffer.wrap(new byte[] {1}) : ByteBuffer.wrap(new byte[] {0});
> // false
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14752) serializers/BooleanSerializer.java is using static bytebuffers which may cause problem for subsequent oeprations
Varun Barala created CASSANDRA-14752:
Summary: serializers/BooleanSerializer.java is using static
bytebuffers which may cause problem for subsequent oeprations
Key: CASSANDRA-14752
URL: https://issues.apache.org/jira/browse/CASSANDRA-14752
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Varun Barala
[https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/serializers/BooleanSerializer.java#L26]
It has two static Bytebuffer variables:-
{code:java}
private static final ByteBuffer TRUE = ByteBuffer.wrap(new byte[]{1});
private static final ByteBuffer FALSE = ByteBuffer.wrap(new byte[]{0});{code}
What will happen if the position of these Bytebuffers is being changed by some
other operations? It'll affect other subsequent operations. IMO Using static is
not a good idea here.
A potential place where it can become problematic:
[https://github.com/apache/cassandra/blob/cassandra-2.1.13/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java#L243]
Since we are calling *`.remaining()`* It may give wrong results _i.e 0_ if
these Bytebuffers have been used previously.
Solution:
[https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/serializers/BooleanSerializer.java#L42]
Every time we return new bytebuffer object. Please do let me know If there is
a better way. I'd like to contribute. Thanks!!
{code:java}
public ByteBuffer serialize(Boolean value)
{
return (value == null) ? ByteBufferUtil.EMPTY_BYTE_BUFFER
: value ? ByteBuffer.wrap(new byte[] {1}) : ByteBuffer.wrap(new byte[] {0}); //
false
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14657) Handle failures in upgradesstables/cleanup/relocate
[
https://issues.apache.org/jira/browse/CASSANDRA-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-14657:
Resolution: Fixed
Fix Version/s: (was: 3.11.x)
(was: 4.x)
(was: 3.0.x)
4.0
3.11.4
3.0.18
Status: Resolved (was: Patch Available)
committed as {{9be437064f5348fe7f8fc6665b747ad751699f49}} to 3.0 and merged up,
thanks!
all test results:
3.0: https://circleci.com/workflow-run/315d4f3d-18b7-4af2-aa46-ef6a2a57d37e
3.11: https://circleci.com/workflow-run/b14718fb-474f-4a90-bf70-98af8df49820
trunk: https://circleci.com/workflow-run/ac121b2d-0f81-4198-b20c-3fd5c3bf1567
> Handle failures in upgradesstables/cleanup/relocate
> ---
>
> Key: CASSANDRA-14657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14657
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 3.0.18, 3.11.4, 4.0
>
>
> If a compaction in {{parallelAllSSTableOperation}} throws exception, all
> current transactions are closed, this can make us close a transaction that
> has not yet finished (since we can run many of these compactions in
> parallel). This causes this error:
> {code}
> java.lang.IllegalStateException: Cannot prepare to commit unless IN_PROGRESS;
> state is ABORTED
> {code}
> and this can get the leveled manifest (if running LCS) in a bad state causing
> this error message:
> {code}
> Could not acquire references for compacting SSTables ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[1/6] cassandra git commit: Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc)
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 21ec39a9d -> 9be437064
refs/heads/cassandra-3.11 0aca1b9df -> 2725d5035
refs/heads/trunk c2c43f814 -> 05dbb3e0a
Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc)
Patch by marcuse; reviewed by Benedict Elliott Smith for CASSANDRA-14657
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9be43706
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9be43706
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9be43706
Branch: refs/heads/cassandra-3.0
Commit: 9be437064f5348fe7f8fc6665b747ad751699f49
Parents: 21ec39a
Author: Marcus Eriksson
Authored: Mon Aug 20 11:06:03 2018 +0200
Committer: Marcus Eriksson
Committed: Fri Sep 14 08:42:28 2018 +0200
--
CHANGES.txt | 1 +
.../cassandra/db/compaction/CompactionManager.java| 14 +++---
2 files changed, 12 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9be43706/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d55ddb6..12c16b7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
* sstableloader should use discovered broadcast address to connect
intra-cluster (CASSANDRA-14522)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9be43706/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index f033bf2..5813f32 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -292,6 +292,7 @@ public class CompactionManager implements
CompactionManagerMBean
private AllSSTableOpStatus parallelAllSSTableOperation(final
ColumnFamilyStore cfs, final OneSSTableOperation operation, int jobs,
OperationType operationType) throws ExecutionException, InterruptedException
{
List transactions = new ArrayList<>();
+List> futures = new ArrayList<>();
try (LifecycleTransaction compacting =
cfs.markAllCompacting(operationType))
{
if (compacting == null)
@@ -304,8 +305,6 @@ public class CompactionManager implements
CompactionManagerMBean
return AllSSTableOpStatus.SUCCESSFUL;
}
-List> futures = new ArrayList<>();
-
for (final SSTableReader sstable : sstables)
{
final LifecycleTransaction txn =
compacting.split(singleton(sstable));
@@ -337,9 +336,18 @@ public class CompactionManager implements
CompactionManagerMBean
}
finally
{
+// wait on any unfinished futures to make sure we don't close an
ongoing transaction
+try
+{
+FBUtilities.waitOnFutures(futures);
+}
+catch (Throwable t)
+{
+ // these are handled/logged in CompactionExecutor#afterExecute
+}
Throwable fail = Throwables.close(null, transactions);
if (fail != null)
-logger.error("Failed to cleanup lifecycle transactions {}",
fail);
+logger.error("Failed to cleanup lifecycle transactions", fail);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2725d503 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2725d503 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2725d503 Branch: refs/heads/cassandra-3.11 Commit: 2725d5035d149e35f267029d10db6b5471041fce Parents: 0aca1b9 9be4370 Author: Marcus Eriksson Authored: Fri Sep 14 08:43:29 2018 +0200 Committer: Marcus Eriksson Committed: Fri Sep 14 08:43:29 2018 +0200 -- CHANGES.txt | 1 + .../cassandra/db/compaction/CompactionManager.java| 14 +++--- 2 files changed, 12 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2725d503/CHANGES.txt -- diff --cc CHANGES.txt index 06fc935,12c16b7..4f2a649 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ -3.0.18 +3.11.4 +Merged from 3.0: + * Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc) (CASSANDRA-14657) * Improve TokenMetaData cache populating performance avoid long locking (CASSANDRA-14660) * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638) * sstableloader should use discovered broadcast address to connect intra-cluster (CASSANDRA-14522) http://git-wip-us.apache.org/repos/asf/cassandra/blob/2725d503/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/05dbb3e0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/05dbb3e0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/05dbb3e0 Branch: refs/heads/trunk Commit: 05dbb3e0ac3e261e66a70afb2b0a00ab5947b2ad Parents: c2c43f8 2725d50 Author: Marcus Eriksson Authored: Fri Sep 14 08:43:54 2018 +0200 Committer: Marcus Eriksson Committed: Fri Sep 14 08:43:54 2018 +0200 -- CHANGES.txt | 1 + .../cassandra/db/compaction/CompactionManager.java | 12 ++-- 2 files changed, 11 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/05dbb3e0/CHANGES.txt -- diff --cc CHANGES.txt index a30bec7,4f2a649..3c3 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,317 -1,6 +1,318 @@@ +4.0 + * Fail query on transient replica if coordinator only expects full data (CASSANDRA-14704) + * Remove mentions of transient replication from repair path (CASSANDRA-14698) + * Fix handleRepairStatusChangedNotification to remove first then add (CASSANDRA-14720) + * Allow transient node to serve as a repair coordinator (CASSANDRA-14693) + * DecayingEstimatedHistogramReservoir.EstimatedHistogramReservoirSnapshot returns wrong value for size() and incorrectly calculates count (CASSANDRA-14696) + * AbstractReplicaCollection equals and hash code should throw due to conflict between order sensitive/insensitive uses (CASSANDRA-14700) + * Detect inconsistencies in repaired data on the read path (CASSANDRA-14145) + * Add checksumming to the native protocol (CASSANDRA-13304) + * Make AuthCache more easily extendable (CASSANDRA-14662) + * Extend RolesCache to include detailed role info (CASSANDRA-14497) + * Add fqltool compare (CASSANDRA-14619) + * Add fqltool replay (CASSANDRA-14618) + * Log keyspace in full query log (CASSANDRA-14656) + * Transient Replication and Cheap Quorums (CASSANDRA-14404) + * Log server-generated timestamp and nowInSeconds used by queries in FQL (CASSANDRA-14675) + * Add diagnostic events for read repairs (CASSANDRA-14668) + * Use consistent nowInSeconds and timestamps values within a request (CASSANDRA-14671) + * Add sampler for query time and expose with nodetool (CASSANDRA-14436) + * Clean up Message.Request implementations (CASSANDRA-14677) + * Disable old native protocol versions on demand (CASANDRA-14659) + * Allow specifying now-in-seconds in native protocol (CASSANDRA-14664) + * Improve BTree build performance by avoiding data copy (CASSANDRA-9989) + * Make monotonic read / read repair configurable (CASSANDRA-14635) + * Refactor CompactionStrategyManager (CASSANDRA-14621) + * Flush netty client messages immediately by default (CASSANDRA-13651) + * Improve read repair blocking behavior (CASSANDRA-10726) + * Add a virtual table to expose settings (CASSANDRA-14573) + * Fix up chunk cache handling of metrics (CASSANDRA-14628) + * Extend IAuthenticator to accept peer SSL certificates (CASSANDRA-14652) + * Incomplete handling of exceptions when decoding incoming messages (CASSANDRA-14574) + * Add diagnostic events for user audit logging (CASSANDRA-13668) + * Allow retrieving diagnostic events via JMX (CASSANDRA-14435) + * Add base classes for diagnostic events (CASSANDRA-13457) + * Clear view system metadata when dropping keyspace (CASSANDRA-14646) + * Allocate ReentrantLock on-demand in java11 AtomicBTreePartitionerBase (CASSANDRA-14637) + * Make all existing virtual tables use LocalPartitioner (CASSANDRA-14640) + * Revert 4.0 GC alg back to CMS (CASANDRA-14636) + * Remove hardcoded java11 jvm args in idea workspace files (CASSANDRA-14627) + * Update netty to 4.1.128 (CASSANDRA-14633) + * Add a virtual table to expose thread pools (CASSANDRA-14523) + * Add a virtual table to expose caches (CASSANDRA-14538, CASSANDRA-14626) + * Fix toDate function for timestamp arguments (CASSANDRA-14502) + * Revert running dtests by default in circleci (CASSANDRA-14614) + * Stream entire SSTables when possible (CASSANDRA-14556) + * Cell reconciliation should not depend on nowInSec (CASSANDRA-14592) + * Add experimental support for Java 11 (CASSANDRA-9608) + * Make PeriodicCommitLogService.blockWhenSyncLagsNanos configurable (CASSANDRA-14580) + * Improve logging in MessageInHandler's constructor (CASSANDRA-14576) + * Set broadcast address in internode messaging handshake (CASSANDRA-14579) + * Wait for schema agreement prior to building MVs (CASSANDRA-14571) + * Make all DDL statements idempotent and not dependent on global state (CASSANDRA-13426) + * Bump the hints messaging version to match the current one
[3/6] cassandra git commit: Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc)
Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc)
Patch by marcuse; reviewed by Benedict Elliott Smith for CASSANDRA-14657
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9be43706
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9be43706
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9be43706
Branch: refs/heads/trunk
Commit: 9be437064f5348fe7f8fc6665b747ad751699f49
Parents: 21ec39a
Author: Marcus Eriksson
Authored: Mon Aug 20 11:06:03 2018 +0200
Committer: Marcus Eriksson
Committed: Fri Sep 14 08:42:28 2018 +0200
--
CHANGES.txt | 1 +
.../cassandra/db/compaction/CompactionManager.java| 14 +++---
2 files changed, 12 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9be43706/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d55ddb6..12c16b7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
* sstableloader should use discovered broadcast address to connect
intra-cluster (CASSANDRA-14522)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9be43706/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index f033bf2..5813f32 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -292,6 +292,7 @@ public class CompactionManager implements
CompactionManagerMBean
private AllSSTableOpStatus parallelAllSSTableOperation(final
ColumnFamilyStore cfs, final OneSSTableOperation operation, int jobs,
OperationType operationType) throws ExecutionException, InterruptedException
{
List transactions = new ArrayList<>();
+List> futures = new ArrayList<>();
try (LifecycleTransaction compacting =
cfs.markAllCompacting(operationType))
{
if (compacting == null)
@@ -304,8 +305,6 @@ public class CompactionManager implements
CompactionManagerMBean
return AllSSTableOpStatus.SUCCESSFUL;
}
-List> futures = new ArrayList<>();
-
for (final SSTableReader sstable : sstables)
{
final LifecycleTransaction txn =
compacting.split(singleton(sstable));
@@ -337,9 +336,18 @@ public class CompactionManager implements
CompactionManagerMBean
}
finally
{
+// wait on any unfinished futures to make sure we don't close an
ongoing transaction
+try
+{
+FBUtilities.waitOnFutures(futures);
+}
+catch (Throwable t)
+{
+ // these are handled/logged in CompactionExecutor#afterExecute
+}
Throwable fail = Throwables.close(null, transactions);
if (fail != null)
-logger.error("Failed to cleanup lifecycle transactions {}",
fail);
+logger.error("Failed to cleanup lifecycle transactions", fail);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/6] cassandra git commit: Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc)
Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc)
Patch by marcuse; reviewed by Benedict Elliott Smith for CASSANDRA-14657
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9be43706
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9be43706
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9be43706
Branch: refs/heads/cassandra-3.11
Commit: 9be437064f5348fe7f8fc6665b747ad751699f49
Parents: 21ec39a
Author: Marcus Eriksson
Authored: Mon Aug 20 11:06:03 2018 +0200
Committer: Marcus Eriksson
Committed: Fri Sep 14 08:42:28 2018 +0200
--
CHANGES.txt | 1 +
.../cassandra/db/compaction/CompactionManager.java| 14 +++---
2 files changed, 12 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9be43706/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d55ddb6..12c16b7 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.18
+ * Handle failures in parallelAllSSTableOperation
(cleanup/upgradesstables/etc) (CASSANDRA-14657)
* Improve TokenMetaData cache populating performance avoid long locking
(CASSANDRA-14660)
* Fix static column order for SELECT * wildcard queries (CASSANDRA-14638)
* sstableloader should use discovered broadcast address to connect
intra-cluster (CASSANDRA-14522)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9be43706/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index f033bf2..5813f32 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -292,6 +292,7 @@ public class CompactionManager implements
CompactionManagerMBean
private AllSSTableOpStatus parallelAllSSTableOperation(final
ColumnFamilyStore cfs, final OneSSTableOperation operation, int jobs,
OperationType operationType) throws ExecutionException, InterruptedException
{
List transactions = new ArrayList<>();
+List> futures = new ArrayList<>();
try (LifecycleTransaction compacting =
cfs.markAllCompacting(operationType))
{
if (compacting == null)
@@ -304,8 +305,6 @@ public class CompactionManager implements
CompactionManagerMBean
return AllSSTableOpStatus.SUCCESSFUL;
}
-List> futures = new ArrayList<>();
-
for (final SSTableReader sstable : sstables)
{
final LifecycleTransaction txn =
compacting.split(singleton(sstable));
@@ -337,9 +336,18 @@ public class CompactionManager implements
CompactionManagerMBean
}
finally
{
+// wait on any unfinished futures to make sure we don't close an
ongoing transaction
+try
+{
+FBUtilities.waitOnFutures(futures);
+}
+catch (Throwable t)
+{
+ // these are handled/logged in CompactionExecutor#afterExecute
+}
Throwable fail = Throwables.close(null, transactions);
if (fail != null)
-logger.error("Failed to cleanup lifecycle transactions {}",
fail);
+logger.error("Failed to cleanup lifecycle transactions", fail);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2725d503 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2725d503 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2725d503 Branch: refs/heads/trunk Commit: 2725d5035d149e35f267029d10db6b5471041fce Parents: 0aca1b9 9be4370 Author: Marcus Eriksson Authored: Fri Sep 14 08:43:29 2018 +0200 Committer: Marcus Eriksson Committed: Fri Sep 14 08:43:29 2018 +0200 -- CHANGES.txt | 1 + .../cassandra/db/compaction/CompactionManager.java| 14 +++--- 2 files changed, 12 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2725d503/CHANGES.txt -- diff --cc CHANGES.txt index 06fc935,12c16b7..4f2a649 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ -3.0.18 +3.11.4 +Merged from 3.0: + * Handle failures in parallelAllSSTableOperation (cleanup/upgradesstables/etc) (CASSANDRA-14657) * Improve TokenMetaData cache populating performance avoid long locking (CASSANDRA-14660) * Fix static column order for SELECT * wildcard queries (CASSANDRA-14638) * sstableloader should use discovered broadcast address to connect intra-cluster (CASSANDRA-14522) http://git-wip-us.apache.org/repos/asf/cassandra/blob/2725d503/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
